Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents

The Exhaustion Gap in Web Agents

Most AI web agents are quite skilled at finding a single, specific answer. They excel when that answer is hidden behind a series of complex constraints. However, they struggle significantly when asked to perform a different kind of task. This task involves listing entire groups of things—such as every airline operating in a specific country—along with all their relevant details. This capacity is known as "breadth." It requires an agent to not only find the right information but to do so exhaustively and accurately across many different sources.

Current benchmarks almost exclusively measure "depth." Depth refers to the ability to navigate a long chain of reasoning to reach one obscure fact. This creates a blind spot in our understanding of agentic capabilities. We do not truly know how well these models can hold a structured set in memory. We also do not know if they can complete a massive, multi-column table without dropping rows or hallucinating values. A new study introduces KO-WIDESEARCH, a Korean-language benchmark designed specifically to expose this gap in breadth-search capabilities.

Beyond the Single-Answer Paradigm

Existing web-agent benchmarks, such as BrowseComp, focus on depth. These tasks typically pin a single answer behind several constraints. This forces the agent to traverse a multi-hop path to recover it. While this tests reasoning, it ignores the structural demands of set enumeration. In a breadth task, the agent must produce a complete table. Every member of a closed set must be accounted for. Furthermore, every attribute cell must be filled correctly.

The authors argue that breadth-search presents a fundamentally different challenge. As shown in, complexity scales in two ways.

Complexity increases by adding more attributes (table width). It also increases by moving from simple lists to 2-D grids. In a 2-D grid, membership itself is a cross-product. An example is every team in every season of a league. The current state of the art fails this test consistently. Even frontier models struggle to maintain the integrity of the entire set. They also struggle to perform granular lookups for every individual row.

An Automated Pipeline for Verifiable Truth

Building a breadth benchmark is notoriously difficult. Certifying that a "gold set" (the ground-truth answer) is both complete and correct is very expensive. This is much harder than verifying a single fact. To solve this, the authors developed an automated synthesize-and-verify pipeline. Instead of relying solely on manual annotation, the process follows a rigorous sequence:

1. Build: A build agent receives a "set-parent" entity (like a specific TV season). It performs an exhaustive web search to construct a gold table.
2. Non-memorizability Gate: The system checks that a closed-book model cannot simply recite the answers from its training data.
3. Completeness Gate: An independent agent re-enumerates the set from the question alone. The gold set is only accepted if the two sets match closely.
4. Cross-source Verification: A separate pass re-looks-up every attribute. This ensures the data is not "source-fragile." Source-fragile means the data relies on a single, potentially unstable webpage.

To handle the nuances of web data, the authors implemented a normalization-aware comparator. This tool ensures that minor formatting differences do not result in a false penalty. For example, it handles a date written as "1948" versus "1948-01-01." It also manages numbers that include a comma.

The Collapse of the Full Row

The results reveal a striking "cascade of failure" across twenty different web agents. Agents are remarkably good at identifying the members of a set. However, they are poor at completing the rows. For example, the strongest model, GPT-5.5, achieved an Item-F1 (membership accuracy) of 92.8. This means it identifies nearly all members. But its Row-F1 (the ability to get every cell in a row correct) fell to 53.7. This indicates it fails to complete more than half of the required details for those members.

The study finds that performance drops sharply as the structural "knobs" of the benchmark are turned. These knobs increase table width or move to 2-D grids. The authors observe that the bottleneck is not the search itself. Instead, the bottleneck is the exhaustive per-cell filling. Interestingly, the data suggests that more effort does not equal better results. The models that performed the highest number of tool calls often had the lowest accuracy. This suggests they "thrash" (repeatedly search without converging on a solution) [Figure 5(c)].

Furthermore, the authors report that difficulty depends on the type of data being extracted. Structured data like names and dates are relatively stable. However, open-ended free-text cells represent the greatest point of failure [Figure 5(a)].

Limits of Specialized Fluency

While the benchmark tests Korean-language navigation, linguistic fluency is not a silver bullet. The authors evaluated several Korean-specialized models, such as A.X-4.0 and Solar-Open-2-preview. They found that these models performed similarly to mid-tier general models.

There are also inherent limitations to the benchmark's coverage. The authors note a significant category skew. The most difficult 2-D grid tasks are heavily weighted toward sports seasons. They make up approximately 67% of the HARD tier. Additionally, because the tasks are grounded in the live web, the benchmark requires periodic re-validation. This accounts for shifting information, such as updated population counts or election results.

Verdict: A Structural Bottleneck

If you are building agents for data aggregation, the verdict is clear. Current models are not ready for autonomous table construction. The research demonstrates that even the most advanced proprietary models solve fewer than 20% of the tables in the benchmark entirely.

The gap between finding a list and completing a table is a structural bottleneck. Improving the "search budget" or spending more on API calls will likely not close this gap. Instead, future development must focus on synthesis. Agents must learn to verify information across disparate sources. They must do this without losing track of the global set structure. Code and the evaluation pipeline are reportedly available; see the paper for the canonical link.

Figures from the paper