BlenderWorld reframes 3D generation evaluation by asking not for a single image but for a complete, editable Blender project that meets concrete spec items. That shift forces agents to reason about geometry, materials, lighting, and scene structure—not just visual appearance—and the benchmark automates verification by rendering adaptive views and applying task-specific binary rubrics plus holistic quality ratings.
What Sets It Apart
- Realistic, deliverable-focused tasks: 223 tasks across architecture, products, environments, characters, animation and scientific visualization that require native .blend outputs and optional exports (renders, videos, model files).
- Fine-grained, verifiable scoring: 2,651 task-specific binary criteria combined with eight visual-dimension ratings; final score is 50% rubric compliance and 50% visual quality.
- Harbored, self-contained package: a Harbor-ready dataset (Base64-embedded preview images, task metadata, attachments) with an evaluator and verifier workflow that produces per-task rubric_score, dimension_score and reward outputs.
- Reproducible runtime specification: uses Blender 4.2.10, expects GPU-capable Linux sandboxes, and defines time/GPU/CPU budgets and verifier separation to mirror realistic generation+verification pipelines.
Who It's For and Trade-offs
Great fit if you build or benchmark multimodal visual agents, autonomous Blender pipelines, or render/scene-inspection judges and need reproducible, spec-driven evaluation. It integrates with Harbor and common model endpoints for automated scoring. Look elsewhere if you only need single-image generation metrics or lightweight image-only benchmarks—BlenderWorld requires significant compute, Blender expertise, and an 8-hour per-task generation/verifier budget, so running the full suite is resource-intensive.
