Most scalar metrics calibrated on photos fail on stylized vector content, causing unreliable evaluation and reward-hacking for text-to-SVG systems. RULER reframes the problem: instead of a single opaque score, it elicits a per-instruction, six-item rubric that decomposes semantic, visual, and stylistic goals; a vision–language judge rates each rendered rollout against the rubric and the weighted satisfactions form the RL reward. This simple pivot converts ambiguous visual judgment into verifiable sub-goals and scales without paired SVG ground truth or human preference labels.
Key Findings
- Rubric-based scoring correlates with humans far better than common scalar metrics: Spearman ρ = 0.7929 and Goodman–Kruskal γ = 0.7574, outperforming Aesthetic (ρ≈0.6051) and CLIP (ρ≈0.5518).
- RULER generates a six-item, instance-aware rubric (semantic fidelity, visual quality, rendering style across subitems) from the instruction alone; a judge VLM produces continuous satisfactions s_k ∈ [0,1] that are combined as a weighted average reward.
- Applying RULER with Group Relative Policy Optimization raises rubric scores on MMSVG-Illustration and MMSVG-Icon from 0.432/0.395 to 0.693/0.683, surpassing specialized SVG systems and matching much larger baselines.
Method in brief
RULER prompts a high-capability model to produce a compact rubric of six independently judgeable items grouped into semantic, visual, and rendering-style axes. At training time each rendered SVG is passed to a judge VLM that scores each item per an explicit guide; the agent optimizes the normalized weighted average reward. The rubric is specified at the level of design intentions (readability, composition, style cohesion) to preserve open-endedness and avoid degenerate reconstruction checklists.
Who it's for and tradeoffs
Great fit if you develop or evaluate text-to-SVG generators and need human-aligned, scalable rewards without paired ground truth. RULER helps when visual intent is multi-dimensional (counts, relations, style) and when downstream models can be fine-tuned with RL. Look elsewhere if you require exact path-level reconstruction, have no access to reliable VLM judges, or cannot afford the compute of repeated render-and-evaluate loops. The method’s effectiveness depends on rubric design and judge quality; rubric engineering and VLM selection are active levers for performance.