Eval-Driven RAG for Technical Documents
Built an independent RAG evaluation prototype with 12 synthetic-adversarial documents, 22 labeled queries, and three lexical retrieval baselines, separating retrieval scoring from deterministic answer evaluation.
Figure 02Source-backed implementation
Two evaluations. One inspectable corpus.
Shared document preparation feeds retrieval comparison and a separate BM25-backed answer evaluation.
- Corpus 12 source documents
Synthetic-adversarial technical Markdown.
- Ingestion Normalize & chunk
Character-based boundaries. Defaults: 700-character target, 120-character overlap.
- Shared input Chunk records
Text, document and chunk IDs, section index, and character offsets.
22 labeled queries supply questions, evidence spans, and expected answers to the two paths below. Both paths read the shared chunk records.
Retrieval comparison
Queries + chunks three lexical methods
- Keyword overlap
- BM25
- TF-IDF / cosine
Hit@k & chunk recall
Relevant chunks are matched from labeled document IDs and evidence text spans.
Answer evaluation
Queries + chunks BM25 retrieval, top-3 by default
- Deterministic control Sentence selection
Select retrieved sentences or return an insufficient-evidence response.
- Optional · API key required OpenAI generation
Prompt with the question and retrieved context.
Expected-information overlap & supporting-ID membership
Heuristic proxies, separate from retrieval scores.
I built an independent RAG evaluation prototype to inspect retrieval quality separately from answer selection. The public source connects a labeled dataset, deterministic chunking, three lexical retrieval baselines, and distinct generation and evaluation stages.
The standalone repository is public here: rag-eval-system
Evaluation Scope
The public dataset contains 12 synthetic-adversarial technical documents and 22 labeled queries. The examples exercise repeated terminology, section-sensitive evidence, multi-span answers, and unanswerable questions. This small, purpose-built dataset supports targeted failure inspection rather than conclusions about production-wide performance.
Engineering Decision
The engineering decision was to build the evaluation loop before optimizing generation. Separating evidence retrieval from answer selection makes it possible to ask which stage needs improvement and inspect the scoring logic behind that judgment.
Document Preparation
The ingestion code normalizes whitespace and builds overlapping, character-based chunks. Each record retains its document ID, chunk ID, section index, and character offsets. The defaults target 700 characters with 120-character overlap, with paragraph boundaries preferred when available.
Trade-off: deterministic boundaries make the preparation stage inspectable, but character counts do not model token budgets or guarantee that a complete evidence span stays in one chunk. The retrieval evaluation’s span-matching logic makes that boundary choice consequential.
Retrieval Baselines
The comparison script runs keyword overlap, BM25, and TF-IDF/cosine against the same chunks and labeled queries. Holding preparation constant helps separate ranking behavior from changes to the corpus or chunker. The answer-evaluation pipeline uses BM25 explicitly; it does not select a retriever automatically from comparison results.
The implementation names the TF-IDF path embedding; it uses term-frequency vectors and cosine similarity. All three baselines are lexical, so a neural semantic retriever would be a separate comparison.
Generation and Evaluation
The answer-evaluation function defaults to a deterministic sentence-selection generator over retrieved chunks. This control keeps answer selection inspectable without model variability. The comparison script attempts both this control and the optional OpenAI-backed generator, recording an LLM-path error if API configuration is missing. The README’s reported observations concern the deterministic control.
What the Checks Measure
| Check | Meaning and boundary |
|---|---|
| Retrieval | Hit@k and chunk recall compare retrieved IDs with relevant chunk IDs inferred from labeled document IDs and evidence-text containment. Boundary: the relevant set depends on how labeled spans match chunk boundaries; chunk coverage alone does not establish answer completeness. |
| Answer text | Apply a threshold to overlap with longer words in the expected answer. Boundary: vocabulary overlap is a heuristic proxy; an incorrect answer can share the expected words. |
| Source IDs | Check that a non-empty set of reported supporting IDs belongs to the retrieved set. Boundary: ID membership does not verify the answer’s statements. The optional LLM path fills these IDs from retrieved chunks, rather than validating semantic support. |
Separating these checks is useful for failure inspection. Full semantic correctness and statement-level evidence support require additional evaluation beyond this implementation.
Public Evidence
The inspected public snapshot is a4526f9. Available evidence includes:
- raw synthetic-adversarial documents
- 22 labeled evaluation records
- retrieval implementations
- generation implementations
- evaluation scripts
At this snapshot, data/processed/ and evals/reports/ contain only placeholders; generated chunks and evaluation reports are absent despite the README’s publication wording. The inspectable evidence is source code and input data. Benchmark results have not been independently reproduced for this case study, so no numerical performance results are presented.
Engineering Approach
This project shows how I approach AI systems work when reliability matters:
- evaluation is part of system design, not a reporting step
- retrieval quality and generation quality should be separated
- engineering tradeoffs should be measurable, not implied
- public project artifacts should include failure analysis, not just polished outputs
The contribution is an inspectable evaluation workflow. Next steps are to expand corpus coverage, add a neural retrieval comparison, and evaluate the optional LLM path with reproducible reports while retaining the deterministic control.