Why this matters
Retrieval systems often succeed by finding a single document that contains the answer; SynthComp is engineered to break that shortcut. By design it forces systems to assemble a set of necessary supporting chunks across multiple documents, exposing weaknesses in retriever recall, multi-step query planning, and agent orchestration that aggregate metrics can hide.
What Sets It Apart
- Synthetic, adversarially filtered composition: questions are generated by an LLM composer over sampled subgraphs and passed through gates (solvability, surface-leak, broad-search shortcut, k-of-k ablation) to ensure every annotated supporting chunk is necessary. So what: evaluation emphasizes evidence-assembly rather than single-document hits.
- Fixed dense index with reproducible embeddings: the benchmark ships a prebuilt index using Qwen3-Embedding-8B. So what: comparisons focus on retrieval strategy and agent behavior rather than embedding training variability.
- Two language variants and fine-grained challenge types: English and Russian sets (395 questions each) cover Bridge, Star, Container, and Temporal mechanisms. So what: enables cross-lingual and mechanism-specific analysis of retrieval failures.
- Evaluation-first distribution: preview files on Hugging Face are encrypted and the repository includes scripts and a custom FAISS searcher tailored to the BrowseComp-Plus harness. So what: running official evaluation requires following the provided decrypt-and-map workflow, preserving the fixed-index evaluation semantics.
Who it's for — and tradeoffs
Great fit if you research or develop: retriever architectures, retrieval-augmented generation pipelines, LLM-based web-search agents, or evaluation methodology for compositional retrieval. The benchmark is compact (395 queries per language) and curated to avoid alternative answer paths, making it useful for focused diagnostics and ablation studies.
Look elsewhere if you need: large-scale naturally authored QA collections for training, open-ended generative evaluation, or benchmarks that reflect continuously changing web content—SynthComp uses a fixed, curated index and synthetic compositions that prioritize controlled difficulty over ecological validity.
Where it fits
SynthComp complements manually authored multi-hop benchmarks (e.g., TRuST) and the BrowseComp family by providing rigorously filtered synthetic tasks that minimize keyword leakage and broad-query shortcuts. Use it to stress-test retrieval completeness and agent planning under reproducible, adversarial conditions.
