Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputs

Creating scientific diagrams is notoriously difficult. While text-to-image models excel at photorealistic landscapes, they struggle with rigid, semantic structures. Research requires labeled boxes, directional arrows, and precise spatial relationships. Current AI tools often produce high-quality raster images (static pixel grids). These are essentially "dead" pixels. They cannot be locally revised. A researcher cannot simply click a label to fix a typo or move an arrow.

Existing automated systems generally fall into two camps. Code-generation methods create editable TikZ diagrams but lack visual richness. Agentic pipelines (sequences of AI agents) produce beautiful but uneditable raster images. There was no unified way to move from diverse inputs—like rough sketches or partial layouts—to a professional, structurally editable SVG (Scalable Vector Graphics). This paper introduces CRAFTER and CRAFTEDITOR. This dual-system approach uses a multi-agent "harness" to bridge this gap.

The Problem

The core issue is that scientific figures are composed of discrete semantic components. When a standard generator fails, it produces "localized errors." Examples include garbled text or misaligned connectors.

Solving this requires more than a larger backbone model. If you try to fix these errors using standard iterative prompting, you hit a wall. Successive natural-language commands eventually contradict each other. For example, "enlarge the title" followed by "reduce white space" creates conflict. This causes the generator to lose faithfulness to the original intent. Most current benchmarks only test text-to-image generation. They ignore the reality that researchers often start with a sketch or a specific layout requirement .

How It Works

Instead of building a better generator, the authors build a "harness." This is an orchestration layer that wraps an existing engine (the executor). It manages failures without modifying the engine itself. The architecture, shown in, relies on a shared, evolving "specification" ($S$).

This acts as the system's memory.

The process operates in a structured loop: 1. Intent Reasoning & Planning: An intent reasoner analyzes the input to seed an initial specification. A plan generator ($D$) proposes multiple candidate visual framings in parallel. This helps avoid bad compositional choices. 2. Execution: The image-generation backend ($E$) renders the chosen plan. 3. Verification & Structured Correction: This is the critical step. A directive critic ($V$) does not just give a scalar score (a single number like "7/10"). Instead, it emits a detailed diagnostic of specific defects. A specification refiner ($R$) then converts these diagnostics into "typed edits." These are structured operations like resize element or ban category. These modify the specification $S$ in place. This prevents the prompt degradation seen in free-text systems. 4. Convergence: A judge decides whether to accept the output or loop back for more refinement.

The system extends this logic to CRAFTEDITOR. This tool handles raster-to-vector conversion. It uses an instruction-driven extraction phase to clean a canvas. It then uses an iterative composition phase to assemble extracted assets into an SVG skeleton .

Numbers

The authors report significant gains by moving from standalone generation to this harnessed approach. On the PaperBanana-Bench, CRAFTER (using the Nano Banana 2 backbone) achieved an overall score of 50.34%. This outperformed the strongest agentic baseline (PaperBanana) by 16.61 points. On the more diverse CRAFTBENCH, the lead grew to 22.20 points.

The harness is "executor-agnostic" (works with any underlying model). Replacing the Nano Banana 2 backbone with the more powerful Nano Banana Pro shifted the overall CRAFTBENCH score by only 2.10 points. This suggests the orchestration layer is the primary driver of performance.

Regarding cost, CRAFTER is more expensive than existing agentic frameworks. Using Nano Banana 2, CRAFTER costs approximately \$0.25 per figure. This is double the \$0.11 cost of PaperBanana. CRAFTEDITOR is even heavier. It costs roughly \$0.85 per conversion due to high LLM token usage (the amount of text processed by the model). However, the authors frame this as a worthwhile trade-off. The cost of a few cents is negligible compared to hours of manual labor.

What's Missing

Several gaps remain for practitioners to consider: * Closed-Source Dependency: The pipeline relies heavily on proprietary models. It uses Gemini for the backbone and judges, and Claude for the language agents. This may limit deployment in private or air-gapped environments. * Non-Monotonicity Risks: LLM-driven editing is empirically non-monotonic (it does not always improve with every step). Even with a "best-so-far" reversion mechanism, the system can occasionally regress. * Scale and Latency: The multi-agent, multi-round loop increases wall-clock latency. For a single figure, this is manageable. For massive batch-processing, the API overhead could become a bottleneck.

Should You Prototype This

Yes, if you are building specialized productivity tools for researchers.

The core insight is highly transferable. Structured, typed edits are superior to free-text prompt accumulation for iterative tasks. If you work on tasks requiring high structural fidelity, use the "harness" pattern. This applies to CAD generation or UI synthesis. The code is available at https://github.com/HaozheZhao/Crafter. Don't wait for a "perfect" generator. Build the harness around the one you have.

Figures from the paper