Tag

LEDGER — Long-Context KPI Question Answering & Page Retrieval

Provides page-level relevance judgments and full OCR'd annual-report text for KPI question answering and page retrieval benchmarking — supports retrieval (per-page qrels) and needle‑in‑a‑haystack numeric extraction over long documents, with eval and train configs.

LEDGER Long-Context Multi-KPI

Pairs OCR-extracted annual-report text with ground-truth financial KPI values to benchmark LLM/table-QA and needle-in-a-haystack extraction tasks. Includes Markdown OCR (.mmd), page images for eval, and 31 KPI columns across multiple years—suited for KPI extraction, retrieval, and robustness testing.

TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs

Generates outcome-specific, dialectical rationales with an LLM and derives continuous, calibrated risk scores for irregularly sampled medical time series—mitigating risk polarization. Reports +3.3% average AUPRC and 81% reduction in calibration error across three benchmarks; code released.

Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories

Turns raw datasets into verifiable multimodal news features via a multi-agent newsroom pipeline. Key innovations: (1) an Inspector that links each claim to data/code/external references for re-execution and audit; (2) multimodal asset generation (interactive maps, audio, visuals) tailored to the story.

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

Lets a single LLM simultaneously act as agent and environment to bootstrap co-evolutional training — using state-prediction process rewards (World-In-Agent) and failure-mode retrieval (Agent-In-World) to reshape training data; reports ~4% average benchmark gain.

Russian PII NER Benchmark

Provides a token-level benchmark for Russian PII detection and NER, with 2,841 sentences and 5,614 annotated spans across 21 fine-grained entity types in BIO format. Mixes sanitized production-log examples, synthetic document templates, and hard negatives to evaluate guardrails and anonymization pipelines.

UltraX (openbmb/UltraX-Preview)

Refines large-scale English pretraining corpora by predicting per-instance structured edits (insert, delete, replace) and deterministically applying them to produce cleaner text for LLM training. Provides five ~20B-token refined corpora in parquet with edit metadata and simple loading configs.

Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application

Survey of methods for engineering interactive environments for LLM-based agents, covering environment modeling, symbolic and neural synthesis, evaluation, and agent–environment co-evolution. Identifies evolution paradigms and future directions like Environment-as-a-Service and multi-agent systems.

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

Proposes a router redesign for Mixture-of-Experts (MoE) that aligns each router row with its expert's principal singular direction using Manifold Power Iteration (MPI), improving token–expert affinity. MPI applies a 'power‑then‑retract' step to push router rows toward principal singular vectors while enforcing norm constraints; the paper gives convergence theory and pretraining results on 1B–11B MoE models.

FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Synthesizes shortcut-resistant search tasks to train deep search agents by controlling four shortcut risks across entity selection, evidence-graph construction, question formulation, and adversarial refinement. Produces training trajectories with longer pre-answer search and fewer shortcut patterns; code will be released on GitHub.

EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments

Benchmarks evolving environments as sequences of progressive updates and introduces EvoMem, a patch-based memory that records structured update histories so LLM agents can reason about environment evolution. Demonstrates measurable gains on EvoArena and other benchmarks.

Joyo Kanji Yomi Benchmark

Provides kanji-level evaluation data for Japanese TTS: disambiguated sentence contexts targeting 4,378 kanji-reading pairs (2,136 Jōyō kanji) with 13,095 native-speaker–verified sentences and katakana-marked ground-truth readings for kanji-level error metrics.