Before a large language model can answer questions about your documents, those documents must be processed. This is the job of the ingestion pipeline. Its goal is to convert unstructured raw data, like PDFs and HTML files, into a structured, indexed set of vectors that a retrieval model can search efficiently.

This process is deterministic. For a given document and configuration, the output should always be the same. This repeatable data preparation flow creates the ground truth for your entire RAG system. If the foundation is weak, retrieval quality suffers, no matter how sophisticated the downstream components are.

The first critical step is breaking down large documents into smaller, manageable pieces called chunks. This ensures each piece fits within the context window of your embedding model and creates focused, semantically dense units for retrieval.

Two primary strategies dominate this task: recursive character splitting and token-based splitting. Recursive character splitting attempts to preserve semantic boundaries. It tries to split text along a hierarchy of separators, typically starting with double newlines (paragraphs), then single newlines, spaces, and finally, individual characters. This method is effective at keeping related sentences together, which can be crucial for context.

Token-based splitting is more direct. It uses the language model's own tokenizer to slice the text into chunks of a predefined token length. This gives you precise control over the size of each chunk, guaranteeing that none will exceed the model's input limit. The main drawback is its disregard for semantic structure; it can easily split a sentence in half, potentially separating a cause from its effect or a subject from its verb.

The choice between these methods depends on your content and goals. For prose-heavy documents where sentence integrity is key, recursive splitting is often preferred. For code or structured data where fixed blocks are more natural, token-based splitting can be more reliable.

Once you've chosen a splitting method, you must define the chunk size and overlap. These parameters directly control the trade-off between context fragmentation and retrieval noise.

Chunk size, typically between 256 and 512 tokens, determines how much information is contained in each vector. Smaller chunks create very specific, targeted vectors, which can improve the signal-to-noise ratio during retrieval. If a user asks a highly specific question, a small, focused chunk is more likely to be a perfect match. The downside is context fragmentation: a single idea might be split across multiple chunks, and the best answer may require information from all of them.

Larger chunks capture more context, reducing the risk of splitting a complete thought. However, they can introduce noise. If a user's query matches only a single sentence within a large chunk, the rest of the chunk's content acts as noise, potentially diluting the relevance score and making it harder for the retriever to identify the best result.

Chunk overlap helps mitigate the problem of context fragmentation. By having adjacent chunks share some text, typically 10–20% of the chunk size, you reduce the chance of a hard cutoff in the middle of a crucial sentence or idea. For example, if a sentence is split exactly at the boundary between two chunks, overlap ensures that the full sentence exists in at least one of them. It acts as a buffer, stitching together related concepts that fall on either side of a split.

Raw documents are messy. PDFs contain headers, footers, and page numbers. HTML is filled with navigation bars, ads, and boilerplate text. A robust ingestion pipeline must normalize this content to extract only the core information.

Normalization involves using parsers tailored to specific file types (like PDF or HTML) to strip away this extraneous content. During this phase, it's critical to preserve useful metadata. Information like the source document name, page number, or section header provides valuable context for the LLM and allows you to include citations in the final answer. This metadata should be attached to each chunk created from the document.

After normalization, a data cleaning layer can further refine the text. This might involve removing special characters, fixing encoding issues, or filtering out irrelevant sections based on patterns. The cleaner the text that goes into the embedding model, the higher the quality of the resulting vectors.

Finally, generating embeddings for thousands of chunks is computationally intensive. Sending each chunk to an embedding model one by one is inefficient. A better strategy is to use batching, where you group chunks together and send them to the model in a single API call. This significantly speeds up the embedding process by leveraging parallel processing capabilities of the underlying hardware.

With a clean, chunked, and vectorized knowledge base, the ingestion phase is complete. The stage is now set for the retriever to find the right information when a user asks a question.