Q-CARE reframes RAG evaluation around two operational principles: query coverage and claim verifiability. Rather than relying on a single gold answer, it decomposes a user query into sub-queries and a model response into atomic claims, then measures how well retrieved chunks cover the sub-queries and whether each claim is verifiable from the retrieved evidence. This makes the evaluation applicable across close-ended fact queries and open-ended explanatory requests and removes dependence on reference answers.
What Sets It Apart
- Query-agnostic, reference-free scoring: a single decomposition handles both factoid and explanatory queries, so you can evaluate systems without curated gold answers. This enables consistent diagnostics across diverse QA tasks.
- Two-sided pipeline: retrieval is evaluated with coverage-aware metrics (C-Prec@k, C-nDCG@k) that reward chunks covering distinct sub-queries; generation is evaluated at claim level with Completeness, Conciseness, and Verifiableness scores that reflect whether the answer makes the right claims and ties them to evidence.
- Benchmarked artifact: the dataset ships 800 queries (half close-ended, half open-ended) drawn from eight established QA sources, two retrievers (BM25 and ANCE) with top-30 chunks inlined, and answers from eight RAG systems (proprietary and open-weight). Everything needed to run the Q-CARE pipeline is included, so no corpus download or external API keys are necessary.
Key Contents and Usage
- Data: 800 queries (100 per source dataset), per-query top-30 retrieved chunks for BM25 and ANCE, model answers (16 per query covering 8 systems × 2 retrievers), and gold answers/chunks carried for reference-only comparisons.
- Metrics: C-Prec@k and C-nDCG@k for retrievers; Completeness, Conciseness, and Verifiableness for generators.
- Practicality: evaluation runs locally against an open-weight backbone; the repository provides the evaluation pipeline and prompts so you can score arbitrary (query, answer, chunks) triples.
Who It's For (and Tradeoffs)
Great fit if you need systematic, fine-grained diagnostics of RAG pipelines across heterogeneous query types—researchers benchmarking retrievers and generators, or teams diagnosing hallucination and evidence coverage.
Look elsewhere if you require large-scale web-crawled corpora integrated into the benchmark (Q-CARE ships inlined chunks and is sized for research use), or if commercial licensing on NewsQA or Reddit-derived ELI5 content prevents your target use case: those portions carry additional terms beyond the benchmark's CC BY-SA 4.0.
Practical Notes
- License: annotations and collection under CC BY-SA 4.0; evaluation code under MIT.
- Known constraints: two dataset portions (NewsQA-derived chunks and ELI5 Reddit content) have upstream terms that may restrict non-research use.
- Reproducibility: the repo includes JSON/Parquet testbed files and a run script that computes both retriever and generator metrics from a single query decomposition.
