AIAny
Icon for item

WorldClaw: Agentic 3D Open-World Generation at Scale

Generates large-scale, explorable 3D open-world scenes from open-ended text prompts, producing editable instance-level assets and a consistent global terrain. Uses agentic planning to convert text into region/terrain/asset specifications and a coarse-to-fine pipeline for terrain construction, mesh reconstruction, and render-based refinement.

Introduction

Most text-to-3D systems either focus on small, photoreal scenes or produce sprawling terrains without editable assets; WorldClaw flips that trade-off by treating open-world generation as a planning and execution problem where agents declare structure first, then synthesize reusable content. The core insight is that agentic, region-aware planning plus a coarse-to-fine synthesis pipeline yields scenes that are both globally coherent and locally detailed while producing explicit assets suitable for editing.

Key Findings
  • Agentic specification: Planning agents translate a text prompt into structured region, terrain, asset, material and spatial-relation specifications — so what? This separates high-level design from low-level synthesis, enabling consistent terrain topology and repeatable placements across runs.
  • Region-aware terrain foundation: The system builds a global height field and semantic layout before asset synthesis — so what? Global terrain consistency prevents disjointed islands and preserves navigability across large maps.
  • Coarse-to-fine synthesis with editable outputs: For detail-demanding regions, WorldClaw generates terrain-conditioned compositions, reconstructs textured, editable meshes, and recovers precise object placements — so what? Results are immediately usable in downstream editing and game engines rather than being ephemeral renders.
  • Render-based refinement: Trajectory-aware render agents refine appearance, object–terrain contacts and micro-details — so what? Final scenes look visually coherent at multiple scales while preserving instance-level editability.
Who it's for and tradeoffs

Great fit if you need large, navigable 3D worlds from natural language where assets must be reusable and scenes editable (game designers, virtual production, simulation labs). Look elsewhere if you only need single-object photoreal synthesis or extremely tight photoreal fidelity per frame; WorldClaw emphasizes scale, coherence, and editability over per-pixel broker-level photorealism. It also presumes a pipeline that can ingest and manage many generated assets, so use is heavier on asset management than quick single-scene demos.

Information

  • Websitearxiv.org
  • AuthorsChunchao Guo, Jinpeng Li, Yang Li, Zilong Huang
  • Published date2026/08/05

More Items

Generates retrieval-centric Chain-of-Thought (RC-CoT) over initially retrieved candidates to improve unified multimodal retrieval via reranking or full-corpus re-retrieval with a dual-mode embedder. Trains an embedder–adviser framework (UniME-R1) using mined hard negatives, supervised learning, and retrieval-oriented reinforcement learning.

Replaces external environment interaction in agentic RL training with 'world rehearsal': the policy alternates between making tool calls and simulating their environment responses, jointly optimizing both roles so the agent internalizes environment dynamics and improves long-horizon tool use and transfer.

Evaluates vision-language model judges on computer-using agent (CUA) trajectories to measure verifier reliability. Provides OSReward-Hard and OSReward-Multi challenge sets, the OS-Shepherd-100K reasoning-annotated corpus, and trained OS-Shepherd reward models that match commercial judges at ~30–60× lower cost.