How do search engines work?

Welcome! I'm really excited to dive into the inner workings of search engines with you.

Before we unpack the heavy engineering behind how billions of pages are found and sorted in milliseconds, I'd love to know what brings you to learning about search engines today, and how technical of an explanation you're looking for.

I'm a software developer wanting a deeply technical architectural deep dive.

It's awesome to connect with another developer! Since you're looking for a deeply technical architectural deep dive, we can skip the usual high-level hand-waving and look directly at how these systems handle massive scale.

The High-Level Architecture

At its core, a search engine is a massive, highly distributed information retrieval system. Instead of searching the live web in real time when a user inputs a query (which would be catastrophically slow), the system splits the problem into an asynchronous, pipeline-based architecture consisting of three main phases.

Search engine data flow diagram illustrating web crawling, raw document storage, indexing into an inverted index, and query processing.

This visualization outlines the macro-architecture. Let's break down these core subsystems:

: An asynchronous, distributed agent that traverses the web graph, fetches raw HTML, and writes it to a raw document store.
: An offline processing pipeline that parses the raw documents, extracts metadata, builds a structured schema, and writes to an inverted index.
Query & Serving Engine: The low-latency runtime that handles incoming user queries, evaluates relevance across the index, applies ranking algorithms, and serves the results in milliseconds.

To kick things off, let's zoom in on the discovery phase and see how a distributed crawler actually manages billions of URLs without crashing.