ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

ChartArena: A Unified Bilingual Benchmark for Robust Chart and Diagram Parsing

If you have ever watched a document parsing model fail on a simple flowchart, you know the pain. Most current models excel at extracting numbers from bar charts but collapse when faced with diagrammatic structures. They lack the topological reasoning (the ability to understand spatial relationships and connections) required to map nodes and edges. This paper introduces ChartArena to bridge this gap. It provides a testing ground for both numeric charts and complex diagrams like mind maps.

The Problem

The state of chart parsing is deeply fragmented. Most existing benchmarks focus on a narrow set of numeric charts. They typically only cover bar, line, and pie charts. They almost entirely ignore diagrammatic structures. Flowcharts and mind maps require graph-level understanding rather than simple value extraction.

Furthermore, current datasets rely too heavily on pristine digital renderings. In production, you rarely encounter perfect digital files. You usually deal with blurry, distorted photos of printed reports or whiteboard sketches.

This creates a massive evaluation gap. As shown in, different parsers emit results in incompatible syntactic forms.

Some use Markdown, others use JSON, CSV, or Python code. A model producing a Markdown table cannot be directly compared to one producing JSON. Even if they capture the same data, their formats prevent fair comparison. This makes it impossible to get an honest picture of model capability.

How It Works

The core innovation of ChartArena is a format-agnostic evaluation protocol. This allows for fair comparison regardless of the output syntax. The authors use a two-stage normalization and scoring pipeline, seen in .

1. Format-Agnostic Normalization: The protocol routes every prediction into one of two canonical semantic spaces. For numeric charts, all outputs are mapped into a "normalized triple view." This consists of $(header, entity, value)$ tuples. For diagrammatic charts, various languages like Mermaid or Graphviz are normalized into a "directed graph view." This view uses labeled nodes and directed edges.
2. Structure-Aware Scoring: Models are scored using metrics that prioritize semantic correctness. For numeric charts, the authors use Intersection-over-Union (IoU, a measure of overlap) on the triple sets. They apply Levenshtein distance (a way to measure text similarity) and relative-error thresholds for numbers. For diagrammatic charts, they use the Hungarian algorithm (an optimization method for matching) to assign nodes and edges. This ensures a model is not penalized for formatting quirks. However, it is heavily penalized for breaking the topology (the structural arrangement) of a flowchart.

The benchmark uses a human-agent collaborative pipeline. MLLMs (Multimodal Large Language Models) generate initial drafts. Human annotators then perform multi-stage verification. This ensures high-quality ground truth for noisy, hand-drawn samples.

Numbers

The authors evaluated 26 leading models. These included general-purpose MLLMs, document parsers, and expert chart parsers. Proprietary models lead the group. Gemini 3.1 Pro achieved the highest overall average of 59.2 EN / 73.2 ZH. This score represents the $mAP_{high}$ (mean Average Precision at a high tolerance level).

However, the gap is closing. Strong open-source systems are very competitive. For example, Qwen3.5-35B-A3B delivers a strong 56.0 EN / 67.1 ZH. Several critical failure modes emerged:

• Complexity Gaps: Radar charts are a universal struggle. Even the top performer, GPT-5, only reached 32.0 EN on this category. This suggests estimating angular values is inherently difficult.
• Structural Gaps: Document parsing models handle numeric charts well. But they "fall sharply behind" on diagrammatic structures. On flowcharts, HunyuanOCR scores 39.9 EN. This trails Gemini 3.1 Pro (65.6 EN) by over 25 points.
• Visual Noise: Performance drops as inputs move from digital to hand-drawn. As shown in, models face a tough choice.

They either replace uncertain entries with a placeholder like "–". Or, they hallucinate plausible but incorrect values.

What's Missing

The paper has a few notable omissions. First, the benchmark focuses only on single-page images. Enterprise workflows often involve multi-page documents. Parsers must sometimes aggregate data across different pages. This capability is not tested here.

Second, the authors do not include scatter plots. Scatter plots are notoriously difficult. They require precise point localization and coordinate recovery. This is a different geometric task than reading a bar height. Finally, the paper lacks a deep dive into visual noise. It does not specify which levels of blur or lighting trigger hallucinations.

Should You Prototype This

If you are building a system for real-world, photographed documents, yes. Use this benchmark to vet your models. The findings are clear. Do not rely on standard document-parsing MLLMs for flowcharts or mind maps. They lack necessary topological reasoning.

If your data is hand-drawn or poorly lit, expect a massive performance hit. Build your downstream logic to handle missing values or hallucinations.

The code and dataset are available at https://github.com/pspdada/ChartArena. It is a solid foundation for moving chart parsing to the prod path.

Figures from the paper