RAG Meets Public Health – Development and Evaluation of a Hybrid System for Automated Fact-Checking

Aim and Research Question(s)

This thesis develops and evaluates a hybrid Retrieval-Augmented Generation (RAG) + Natural Language Inference (NLI) pipeline for fact-checking measles claims, grounded in the WHO infodemic management framework [1] and framed as decision support, not autonomous verification.

• How does augmenting a local open-weight LLM with retrieved evidence and NLI filtering affect claim verification, and when does retrieval help or hurt?
• To what extent can a hybrid retrieval model reduce hallucination while improving transparency and scalability?

Background

Health misinformation spreads faster than official corrections, and measles is a prime example: a disease declared eliminated in several regions has resurged across Europe, driven by persistent vaccine-related myths on social media. With a basic reproduction number of 12–18, measles needs roughly 95% vaccination coverage for herd immunity [2]. Manual fact-checking is the gold standard but cannot scale, which motivates automated triage.

Methods

A four-layer pipeline was evaluated in a 3×2×2 design (evidence × LLM × abstain), yielding twelve conditions. all-MiniLM-L6-v2 embeddings index a 723-passage WHO, PubMed, CDC and ECDC corpus with FAISS retrieval; a DeBERTa-v3-small cross-encoder labels each claim–passage pair; Qwen3 8B and Mistral 7B run locally via Ollama, with no cloud API dependency. 150 measles claims were manually verified, hallucination screened on a 78-item sample.

Results and Discussion

Retrieval degraded both models in every configuration — not for lack of evidence: relevant passages were retrieved for 67.3% of claims, yet the Baseline still won on those. The failure is integration, not access. Allowing NOT_ENOUGH_EVIDENCE raised macro-F1 in five of six comparisons; Qwen3 Baseline + Abstain reached 73.0% selective accuracy at 91.3% coverage. MISLEADING stayed the weakest class, though deferral lifted its F1 from 0.299 to 0.680; the NLI verifier added little signal and the hallucination check was unreliable (κ = 0.03).

Conclusion

Retrieval did not improve performance; abstention proved decisive, shifting errors towards safe deferral. Given the MISLEADING failures and weak NLI signal, the system is best positioned as human-in-the-loop triage support with mandatory review of REFUTED and MISLEADING flags, not an autonomous fact-checker. Limits: a small corpus in which no pairwise comparison reached significance, a single annotator, and a three-class taxonomy that cannot represent framing-based misinformation. Next steps: a biomedically fine-tuned NLI model and a broader dataset.

References