What robots.txt controls — and what it doesn't
Robots.txt manages crawling, not indexing. A disallowed URL can still appear in search results (without a snippet) if other pages link to it, because Google never crawled it to see a noindex tag.
That distinction is where most of the damage happens, so it's the first thing we check.
- Disallow controls crawling; to keep a page out of the index use noindex — which requires the page to stay crawlable
- Never disallow a URL you also want deindexed: the crawler can't read the noindex tag if it's blocked
- Don't block CSS/JS — it breaks how Google renders and understands your pages
- A single accidental 'Disallow: /' can remove crawl access to the whole site
- Point to your XML sitemap with a Sitemap: line
Sensible crawl directives
Use robots.txt to keep crawlers out of low-value areas — internal search results, cart and checkout, admin, endless filter combinations — so their time goes to pages that matter.
Test every rule before it ships, and confirm environment blocks are set the way you intend.
- Block internal search result pages and infinite filter/sort parameter URLs
- Keep admin, cart, and account areas out of crawl paths
- Test rules with the robots.txt Tester / URL Inspection before deploying
- Remember rules are directives crawlers choose to honor — not a security control
- Keep staging blocked, but confirm the block is removed at launch (a classic migration bug)
More on technical seo
Frequently asked questions
Should I block a page in robots.txt or use noindex?
If you want it gone from search, use noindex and leave it crawlable so Google can read the tag. Use a robots.txt disallow when you simply don't want crawlers spending time there and don't mind the URL being occasionally listed.
Is robots.txt a way to hide private content?
No. It's a public file and doesn't stop access — it only asks well-behaved crawlers not to fetch listed paths. Private data needs authentication or server-side protection, not robots.txt.
My staging site got indexed — why?
Usually the disallow or password protection was missing, or the block was carried over to production and forgotten. We audit both sides so the wrong environment isn't exposed and the right one isn't blocked.