XML sitemaps: helping search engines crawl your site
An XML sitemap is a file that lists the pages of a site you want search engines to know about. It is a discovery aid — a way of handing an engine a map so it does not have to find every page by following links alone.
Last updated
What a sitemap is for
A sitemap is a machine-readable list of URLs, sometimes with extra hints like when a page last changed. Its job is to help engines discover pages efficiently, which matters most for large sites, new sites with few links, or pages that are not well linked internally.
It does not force anything to be indexed. A sitemap is a suggestion of what exists and is worth crawling, not a command. Engines still decide, page by page, what to index based on quality and relevance.
What belongs in it
A sitemap should list the canonical, indexable pages you actually want in search — the real content — and leave out duplicates, redirected URLs, error pages, and anything blocked from indexing. Including junk sends mixed signals and wastes crawl attention.
Keeping it accurate matters. A sitemap full of dead or non-canonical URLs erodes trust in the file; one that stays current with the site’s real, wanted pages is a reliable aid. For large sites, it is generated automatically rather than maintained by hand.
Submitting and its limits
A sitemap is usually referenced in robots.txt and submitted through a search engine’s webmaster tools, where you can also see how many of its URLs were indexed. That index gap is a useful diagnostic — a big shortfall points to quality or technical problems.
The limit to keep in mind is that a sitemap aids discovery, not ranking. It helps engines find pages; it does nothing to make a poor page rank. It is plumbing, valuable when discovery is the bottleneck and irrelevant when it is not.
How Kirti approaches this
Kirti is being built to check a brand’s sitemap during its technical audit — confirming it lists the right canonical pages and flagging stale or wrong entries — and to surface index-coverage gaps for the team to act on.
The agent is in active development and not yet generally available; the sitemap conventions here apply to any site regardless of how it is generated.