robots.txt: controlling how crawlers access your site
robots.txt is a small text file at the root of a site that tells crawlers which parts they are allowed to access. It is a simple, powerful, and easily misused tool — one wrong line can quietly hide an entire site from search.
Last updated
How robots.txt works
The file lives at the site root and uses a plain syntax to allow or disallow crawler access to paths. Well-behaved crawlers read it first and respect its rules, so it is the standard way to keep bots out of areas that should not be crawled — admin paths, search-result pages, staging directories.
It is a directive to crawlers, not a security measure. A disallowed path is still reachable by anyone with the URL; robots.txt only asks compliant bots not to crawl it. Sensitive content needs real access control, not a robots rule.
The classic mistakes
The most damaging error is disallowing more than intended — a broad rule that blocks the whole site, often left over from a staging environment and never removed at launch. A site can vanish from search entirely because one line was too broad.
A subtler confusion is expecting robots.txt to remove a page from search. Blocking crawling does not deindex a page; if it is already indexed or linked elsewhere, it can still appear, just without a proper description. Removal needs a different tool.
Crawl control versus index control
It helps to separate two jobs. robots.txt controls crawling — whether a bot fetches a page. A noindex directive controls indexing — whether a fetched page appears in results. They are different levers, and using the wrong one causes exactly the confusion above.
Notably, if you block a page in robots.txt, an engine cannot see a noindex on it, because it never crawls the page to read the tag. To reliably keep a page out of results, you generally allow crawling and apply noindex, rather than blocking it.
How Kirti approaches this
Kirti is being built to check robots.txt as part of its technical audit — catching overly broad disallows and the crawl-versus-index confusion — and to flag anything that may be hiding pages a brand actually wants found, for the team to review.
The agent is in active development and not yet generally available; robots.txt behaves the same across sites, so this guidance holds whatever platform you run.