Search engines map the vast expanse of the internet using automated software programmes called crawlers, bots, or spiders. Rather than merely viewing a website the way a human browser does, a crawler actively downloads the raw HTML code and resources of a page. This parsing process allows the search engine to read the written content, analyse the overall layout, and identify the underlying code structure.

To navigate from page to page, crawlers start with a curated seed list of known URLs and systematically follow the hyperlinks embedded on those pages to discover new content. To make this process more efficient, crawlers also read XML sitemaps, which are structured directories automatically generated by most content management systems to list all valuable pages on a site.