RAG Compliance & Incident Intelligence Assistant
A question-answering system over UK health and safety guidance, where every answer carries citations back to the source section, and an answer that cannot be cited becomes a refusal. The differentiator is the evaluation harness: retrieval quality is measured as a CI quality gate, not assumed.
Why this project
I have spent the last three years working in security and compliance roles alongside my studies, and I have seen how much time goes into looking up what the regulations actually require. An LLM that paraphrases regulation from memory is dangerous in that setting; an answer is only useful if it cites the exact clause it came from. That constraint, groundedness over fluency, is what this project is built around.
What it does
- Ingests and chunks UK health and safety documents with section-aware splitting, so every chunk can be cited as "document, section"
- Retrieves with hybrid search: dense vectors plus a BM25 implementation written from scratch (~40 lines), fused with reciprocal rank fusion
- Enforces two guardrails a compliance setting cannot skip: low retrieval confidence triggers a refusal instead of a guess, and every citation marker in the answer is verified against the retrieved sources; an uncited answer becomes a refusal
- Runs on pluggable providers: OpenAI or Ollama for generation, with a deterministic offline mode that lets the entire system run and be tested in CI with zero credentials
- Evaluates every change against a 16-question golden set in GitHub Actions, failing the build if retrieval quality or citation rate regresses
Architecture
The evaluation harness
Most RAG demos show a chat box. This one ships a comparison table: the same 16-question golden set run against vector-only, BM25-only and hybrid retrieval. Current results: hybrid hits the correct source on 100% of questions (MRR 0.938) and 100% of answers carry valid citations. It also produced a real finding: BM25 alone wins on this corpus because compliance questions reuse the regulations' exact vocabulary, which is precisely the kind of trade-off the harness exists to catch.
Skills demonstrated
- RAG architecture: chunking, hybrid retrieval, rank fusion, citation grounding
- LLM evaluation as an engineering discipline: golden sets and CI quality gates
- Guardrail design: refusal policies and citation verification, tested with deliberately misbehaving model stubs
- Backend engineering: FastAPI, provider abstractions, Docker
- Domain knowledge from three years in UK security and compliance work