Provides 324 Russian short-answer web-search tasks with gold supporting documents to evaluate fixed-index retrievers and search agents. Tasks span eight topical categories and five retrieval challenge types (multihop, structured evidence, temporal, entity disambiguation, comparative) and use a Qwen3-Embedding-8B index for evaluation.
Comprehensive benchmark and automated evaluation framework for keyframe-conditioned video generation—decomposes keyframe execution into six metrics and assesses overall video quality with evidence-grounded MLLM judgments and specialized perception models.
Specialized LLM for clinical workflows trained via a human-gated self-evolution loop to improve patient consultation, multimodal clinical reasoning, interactive diagnosis, and EHR tool use. Iteratively refines targeted synthetic and curated data based on benchmark failures to raise specific capabilities without broad regressions.
Analyzes adversarial weaknesses of World-Action Models (WAMs) via BadWAM, a framework that crafts visual perturbations to decouple a model’s imagined future from its executed actions. Introduces two attack modes—action-only (disruptive) and imagination-preserving (stealthy)—and shows large drops in closed-loop task success (e.g., 96.5%→43.1%).
Provides 50 ARC‑AGI‑3 gameplay trajectories (GPT‑5.6 Sol and Claude Opus/Fable) plus a dependency‑free scorer and event logs; includes sanitized session data, snapshots, and utilities to recompute RHAE scores for reproducible agent evaluation and cross-model comparison.
Turns fragile, implicit search progress into explicit, persistent, shared state for multi-agent information seeking — externalizes progress as Frontier Task, Evidence Graph, Coverage Map and Failure Memory, and uses pipeline-parallel scheduling plus a middleware harness to avoid repeated failed searches and improve utilization and throughput.
Policy-adaptive multimodal safety classifier that evaluates text and images against free-form natural-language policies and returns a continuous yes/no safety score. Produces a single-token verdict from a 3B-parameter model, supports multiple languages, and is designed for lightweight real-time moderation.
A vision-language-action foundation model trained on 100k+ hours of real-world robot manipulation trajectories to follow natural-language instructions and adapt to downstream tasks with minimal fine-tuning. Uses a two-stage (pre-/post-) training recipe and a scalable auto-labeling pipeline; shows clear scaling benefits and state-of-the-art sim-to-real transfer on standard benchmarks.
Evaluates whether video models reason according to physical laws by treating generated videos as visible reasoning traces and using a three-stage Perception–Formulation–Deduction protocol. Includes Orchard (400 mechanics videos), chain-of-frames prompting on annotated first frames, and a hybrid MLLM-plus-objective scoring suite for stage-resolved diagnostics.
Quantifies active visual observation in multimodal LLMs with ActiveVision, a 17-task benchmark that forces repeated perception rather than one-shot description. Finds frontier MLLMs fail badly (top model 10.6% vs humans 96.1%) and that model-generated vision code does not close the gap.
A looped-Transformer LLM series using Mixture-of-Experts (20B with 2B active; 6B with 0.6B active) that trades extra pretraining compute for repeated looping. Shows superior compute-efficiency versus matched-compute vanilla baselines and attains gold-medal performance on 2025 IMO and IPhO after a post-training pipeline.
Models long-horizon interactive literary simulation where characters and world co-evolve; introduces an open‑schema framework with a Character Agent and an LLM-based World Model, plus seven trainable tasks and a dataset from 57 books for benchmarking persistent narrative state.