PaintBench: Deterministic Evaluation of Precise Visual Editing

The Precision Gap in Generative Editing

Current AI models are excellent at making pretty images. However, they are remarkably bad at following exact instructions, such as "move this shape precisely 50 pixels to the right." While generative models excel at open-ended creative tasks, they struggle with the rigid, single-answer constraints required for professional workflows. To bridge this gap, researchers have introduced PAINTBENCH, a benchmark designed to move past "vibes-based" evaluation toward mathematical correctness.

The failure of subjective judging

Evaluating image editing has traditionally relied on a shaky foundation. Most existing benchmarks use human judgment or vision-language models (VLMs; models that can process both text and images) to score outputs. This works for subjective tasks—like "make the sky more blue"—where there are infinite valid answers. But it falls apart when the task is deterministic (having only one correct answer). If you ask a model to recolor a triangle to a specific hex code, a VLM judge might say the result "looks blue." It may fail to notice the color is actually off by several shades.

This lack of precision is a systemic issue. Current models often suffer from "over-editing," where they modify pixels far beyond the intended target region. As shown in, this error is particularly acute for small edit-regions. For the smallest targets, models can change 50 to 1,400 times more pixels than requested. Relying on a learned judge to catch these subtle errors introduces massive bias. It prevents us from knowing if a model is actually improving or just getting better at pleasing the judge.

Deterministic pixel-level verification

The authors propose a departure from learned judges. They use a procedural, deterministic pipeline instead. Instead of curating a static dataset, PAINTBENCH generates problems on the fly using random seeds. This creates an effectively infinite supply of unique test cases. This method makes the benchmark contamination-resistant (preventing models from simply memorizing the test questions).

The mechanism follows a strict logical flow: 1. Procedural Generation: A seed generates an input image, a natural-language instruction, and a mathematically exact ground-truth answer .

1. Atomic Task Mapping: The tasks are categorized into four fundamental primitives: geometric transformations, structural manipulations, color changes, and symbolic reasoning .

2. Color-Space Comparison: The authors use $\Delta E^_{76}$ to measure color accuracy. This is the Euclidean distance (a straight-line mathematical distance) in the CIE $L^a^b^$ color space. It allows them to measure color accuracy based on human perception.
3. mIoU Scoring: The core metric is mean Intersection over Union (mIoU). This calculates the overlap between the model's output and the correct answer. The authors sweep across 11 different color tolerance thresholds ($t=0$ to $t=10$). This prevents the metric from being overly sensitive to one arbitrary threshold. It rewards models that balance edit accuracy with the preservation of unchanged regions .

Industry leaders hit a wall

The results of the benchmark are sobering. The authors evaluated 11 models, including prominent closed-weights generators and open-weights editors. The headline finding is that even the industry leader, NANO-BANANA-2, achieves an mIoU of only 17.1%. This means even the best model gets less than one-fifth of the pixels exactly right across all tasks.

The performance delta across task categories reveals deep structural weaknesses. Geometric transformations (like rotation and scaling) are consistently the hardest. Almost no model exceeds 17.5% mIoU on these tasks. Conversely, "removal" tasks are more tractable. Some models reach 50.6% mIoU on removal [Table 1].

Crucially, the authors demonstrate that these synthetic scores translate to real-world utility. They introduced TINYGRAFIXBENCH, a companion benchmark for editing Matplotlib-rendered data visualizations. They found a strong linear correlation ($R^2 = 0.91$) between PAINTBENCH and TINYGRAFIXBENCH scores . This suggests that mastering fundamental shape and color primitives helps a model succeed in complex domains like automated chart editing.

Diagnosing the breakdown

The paper provides more than just a leaderboard. It offers a diagnostic toolkit. By analyzing metric curves across different color tolerances, the authors identify four distinct failure modes : * Color Imprecision: The edit is structurally correct, but the colors are wrong.

• Execution Omission: The model ignores the instruction entirely.
• Structural Catastrophe: The output bears no resemblance to the input or the target.
• Structural Imprecision: The model hits the right area but misaligns the geometry.

Furthermore, the authors highlight significant brittleness to scene variation. Increasing the object count or introducing striped backgrounds causes substantial performance drops [Table 2]. There is also a clear "color gap." While models maintain decent headline scores on nonstandard color palettes, their ability to achieve exact pixel-match ($\Delta E^*_{76}=0$) plummets compared to standard web colors .

The verdict

If you are looking for a tool to evaluate whether your model can act as a reliable editor for professional design software, the answer is: not yet. The current state of the art is still too prone to over-editing and geometric misalignment.

However, PAINTBENCH is a vital piece of infrastructure. It replaces the "black box" of VLM-based evaluation with a transparent, mathematical ground truth. The code and datasets are available for research and prototyping (GitHub, Hugging Face). If you are developing a model meant for data visualization or precise graphic design, stop relying on perceptual metrics. Start measuring your mIoU against these primitives. The gap is wide, but now we have a ruler to measure it.

Figures from the paper