XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

Enforces output grammars during LLM decoding while reducing CPU validation overhead. Precomputed token masks and persistent stacks support nested formats without checking the full vocabulary at every step.

Visit Website

Introduction

A fast model can still spend time waiting for CPU-side grammar checks. The useful insight is that most token-validity decisions depend only on a local grammar state, so they can be cached before generation rather than recomputed for every token.

Key Findings

Separating cacheable checks from stack-dependent checks preserves recursive grammar support while reducing runtime work.
Context expansion and persistent stacks handle the remaining cases and enable efficient branching and rollback.
The authors report up to 100× faster mask generation than existing methods in their evaluation. Overlapping grammar work with GPU execution makes structured output overhead nearly disappear in tested serving configurations; this is not a universal application speedup.

Who Should Read It

Useful for inference engineers serving tool calls, nested JSON, or domain-specific output languages. It explains where format enforcement belongs in a serving pipeline. Grammar compliance does not establish that an answer is factually correct or that a proposed tool action is appropriate; those need separate checks.

Back

Information

Websitearxiv.org
OrganizationsCarnegie Mellon University, NVIDIA, Shanghai Jiao Tong University, University of California, Berkeley
AuthorsYixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, Tianqi Chen
Published date2024/11/22

More Items

AI Agent Papers2025

Tongyi DeepResearch Technical Report

Baixuan Li, Bo Zhang +54Tongyi Lab, Alibaba Group

Trains long-horizon web researchers by combining agentic mid-training, synthetic trajectories, and reinforcement learning. A 30.5B-parameter model activates 3.3B parameters per token and learns to plan, search, and synthesize evidence.

paper ai-agent LLM RL research+4

Large Language Model Papers2025

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen +6Sea AI Lab, National University of Singapore +1

Explains how base-model behavior and GRPO normalization distort claims about reasoning gains. Introduces Dr. GRPO to reduce length bias, with a 7B training recipe reaching 43.3% on AIME 2024.

paper LLM RL reasoning math

Large Language Model Papers2023

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

Avi Singh, John D. Co-Reyes +39Google DeepMind, Mila

Improves language-model problem solving by generating candidate answers, retaining verified successes, and fine-tuning on them. Repeats this loop with an expectation-maximization interpretation instead of collecting more human-written solutions.

paper llm reasoning ai-train sft+3