GuideIntermediate

Production Rag And Document Ingestion

This guide builds the data plumbing of a production RAG system on top of real Reservo documents — thirteen documents across three raw formats: markdown, HTML, and a deliberately "dirty" `.txt` file that simulates a PDF/OCR extraction — and wires it up to an agent as just another tool. It covers the full ingestion pipeline: parsing each raw format, cleaning real artifacts (repeated headers/footers, broken hyphenation, page breaks), chunking with actual criteria, and attaching the metadata needed to cite the source later. Those chunks get indexed with a BM25 index built from scratch ($0, deterministic, always labeled as real lexical retrieval — the same algorithm behind Elasticsearch/OpenSearch — and never as semantic search), and search is exposed as the `search_docs(query, k)` tool that the agent calls from inside its own loop: agentic retrieval, not a monolithic single-pass RAG pipeline, with the agent deciding when to search and combining that search with Reservo's business tools in the same turn. The guide also covers incremental, idempotent ingestion (re-ingesting changed documents without duplicating them, detecting additions/changes/deletions by content hash), evaluating retrieval quality with a fixed ground-truth query set and form-based metrics — recall@k, precision@k, never a semantic judge — and operating the system: citing sources, handling zero results without hallucinating, filtering irrelevant chunks, and estimating pipeline cost. All the engineering actually runs on Python 3.14 and `numpy`; the LLM's decision is treated as a concept. It doesn't teach embedding theory or build a vector database — that's named as the next step in AI Engineering.