Generate a robots.txt file to control how search engines crawl your store. Copy and upload to your site root.
A robots.txt file sits at the root of your website and tells search engine crawlers (like Googlebot) which paths they may or may not crawl. The generator lets you pick user-agents, allow or disallow specific folders, and add your sitemap URL, then outputs a ready-to-upload robots.txt. You place the file at yourstore.com/robots.txt so crawlers read it first.
An apparel store wants Google to index products but skip the cart and account pages. The generator produces: User-agent: * / Disallow: /cart/ / Disallow: /account/ / Allow: / / Sitemap: https://yourstore.com/sitemap.xml. They upload this as robots.txt at their domain root.
It must sit at the root of your domain, so it is reachable at yourstore.com/robots.txt. A file placed in a subfolder will not be read by crawlers.
No. Disallow only asks crawlers not to crawl a path; a blocked URL can still appear in results if other sites link to it. To keep a page out of search, use a noindex meta tag on the page instead.
Generally yes, because pages like cart, checkout, and login add no search value and waste crawl budget. Keep product, category, and content pages open so they can rank.
Many hosted store platforms generate one automatically, so check yourstore.com/robots.txt before adding your own. If it already lists your sitemap and blocks admin paths, you may not need changes.
Online stores on Commmerce ship with a correct robots.txt and auto-updating sitemap.
See Online Store
