RAG Compliance and Incident Intelligence Assistant
Answers questions over UK health and safety guidance with a citation on every statement. An answer that cannot be cited becomes a refusal.
The problem
An LLM that paraphrases regulation from memory is dangerous where compliance matters. An answer is only useful if it cites the exact clause it came from. This project is built around that constraint: groundedness over fluency, with retrieval quality measured in CI, not assumed.
What I built
Hybrid retrieval
Dense vectors plus a BM25 implementation written from scratch (about 40 lines), fused with reciprocal rank fusion.
Cite or refuse
Low retrieval confidence triggers a refusal instead of a guess, and every citation marker is checked against the retrieved sources.
An evaluation harness
A 16-question golden set in GitHub Actions compares vector-only, BM25-only and hybrid retrieval, and fails the build if quality or citation rate regresses.
How it works
- IngestionParagraph-aware chunking with section labels and stable chunk ids
- IndexEmbeddings in SQLite with exact vector search, plus BM25
- RetrievalHybrid search fused with reciprocal rank fusion
- GenerationGrounded answer with inline citations; OpenAI, Ollama or an offline extractive mode
- VerificationCitation markers checked against sources; cite or refuse
- EvaluationGolden-set harness in CI with a comparison table
What I found
- Hybrid retrieval hits the correct source on 100% of questions (MRR 0.938) and 100% of answers carry valid citations.
- A real finding: BM25 alone wins on this corpus, because compliance questions reuse the regulations' exact vocabulary. That is the kind of trade-off the harness exists to catch.
- The whole system runs and is tested in CI with zero credentials, through a deterministic offline mode.
Limits and honest notes
- The evaluation set is 16 questions, enough to catch regressions but small for broad claims.
- The vector store is SQLite with exact search, behind a pgvector-shaped interface, which suits this corpus size.