Studies looping shared transformer layers in Mixture-of-Experts models under matched budgets and proposes SMELT: loop the middle half twice while matching per-token FLOPs, non-embedding parameters, and KV cache. Shows 6.8–18.0% training-FLOPs savings on the compute-optimal frontier, stronger downstream gains on code and long-context tasks.
Turns sparse per-student records into individualized simulators that both reproduce a student’s responses and update them under tutor guidance using pooled LLM pretraining followed by per-student specialization; releases StudentSimEval and reference simulators across chess, L2 writing, and math.
Compresses conversational histories and long documents into short sequences of continuous soft memory tokens that a frozen decoder can read directly without text reconstruction. Uses a small reader-matched writer that trains only a tiny adapter, achieving 4–16× compression and much faster write/read latencies.
Generates compact keyword sets for both queries and items with LLMs and matches them directly via an inverted index. Uses supervised fine-tuning to align keyword spaces, then alternates GRPO-based reinforcement learning on query- and item-side generators to co-evolve representations and maximize retrieval F1 while staying compatible with keyword-based infrastructure.
Selectively admits dense token-level teacher supervision only after a prompt-level verifier audit, routing prompts that fail the audit to verifier-grounded trajectory supervision instead — reducing harmful updates from confidently wrong teachers and improving teacher GPU utilization.
Introduces Declarative Attention (DA), a zero-shot protocol that has LMs declare which parts of long context to attend to during chain-of-thought, letting the runtime build dynamic attention masks and skip most KV-cache reads. Produces large token savings (up to ~52% on Gemma-4-31B) with modest accuracy loss.
AI Agent Papers2026
Presents two LLM-based search agents (Iris-mini and Iris-pro) trained by alternating supervised fine-tuning and reinforcement learning against live web search. Key features: web-graph-derived multi-hop tasks with entity abstraction, SFT–RL climbing, inference-time context management, and state-of-the-art open-source benchmark results.
Uses lightweight discrete-diffusion adapters to draft blocks of tokens in parallel while keeping the original autoregressive model as the final arbiter, enabling lossless speedups (up to 3×) and compatibility with existing AR weights via a Diffusion Distillation phase.
Zixuan Fu, Bingxiang He +11Affiliation: University of Chinese Academy of Sciences, Affiliation: Tsinghua University +3
Studies on-policy distillation (OPD) at the data-minimal limit by training on a single query, measuring state coverage and alignment dynamics, and showing OPD is often data-overfed but algorithm-starved.
Converts a natural-language function specification into a reusable local neural function by using teacher models to synthesize examples and finetuning a small adapter for a compact interpreter. Achieves higher semantic accuracy (83.6% on FuzzyBench-Hard) at the cost of roughly one minute compile time; produces versionable PAW artifacts for local deployment.
Combines sparse verifier outcomes with dense privileged‑hindsight token scoring to learn an outcome‑calibrated, normalized distribution over complete responses for on‑policy self‑improvement. Key features: sign‑gated guidance (retain/reverse/disable per verifier advantage), profiled trajectory balance with one log‑partition per rollout group, and explicit correction against false‑positive self‑guidance.
Shows that fully quantizing all 496 linear layers—including the recurrent Gated DeltaNet—of a hybrid 27B LLM to 4-bit NVFP4 W4A4 preserves benchmark accuracy while reducing model size to 17.5 GiB and improving prefill speed; includes a calibrated NVFP4 checkpoint.