K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts

The State Maintenance Gap in Korean Web Agents

Researchers have created a difficult test to see how well AI agents can browse the internet specifically for Korean information. Even the smartest AI models struggle with this. They often get confused when connecting different pieces of information found on Korean websites. While frontier models excel at general reasoning, they hit a performance cliff during agentic tasks (complex, multi-step actions performed by an AI). These tasks require navigating the live web to solve complex queries.

The Question

How capable are web-browsing agents in the specific linguistic and cultural nuances of the Korean web? The authors of K-BROWSECOMP aim to move beyond static benchmarks. Static benchmarks merely test if a model "knows" a fact. Instead, they focus on compositional, agentic evaluation. They want to know if an agent can execute a long-horizon plan. This includes forming search queries, navigating semi-structured local websites, and synthesizing disparate evidence to find a single, stable answer. Crucially, the research asks if failure stems from poor information retrieval (finding the wrong pages) or a failure of reasoning-over-evidence (finding the right pages but processing the data incorrectly).

Why The Old Answer Was Incomplete

Until now, LLM evaluation has largely relied on foundational capabilities. These include instruction following or static knowledge retrieval. While benchmarks like BrowseComp push toward agentic evaluation, they are heavily English-centric. Existing Korean evaluations mostly focus on NLU (Natural Language Understanding) tasks. These measure if a model understands a sentence or a paragraph. They do not test if an agent can act.

This creates a blind spot regarding "AI sovereignty." If a model cannot navigate specific search conventions or local entities, it cannot serve as a reliable agent for Korean users. The field assumed that scaling model size would naturally translate to regional competence. However, Korean agentic benchmarks have been virtually nonexistent. This leaves the community without a way to measure if models actually work in a localized production environment.

What They Did

The researchers constructed K-BROWSECOMP, a 400-problem benchmark. They split this into two parts. The first is a 300-problem K-BROWSECOMP-VERIFIED subset. This was manually crafted and validated by native Korean speakers. The second is a 100-problem SYNTHETIC split designed as a targeted stress test.

To ensure high difficulty, human annotators followed strict guidelines. Questions had to require either multi-hop reasoning (sequentially traversing through websites) or parallel-branching (intersecting multiple independent constraints) .

The authors also developed a trajectory-level failure taxonomy (F0–F8) [Table 1]. This system categorizes exactly where an agent's logic breaks down during a browsing session.

The most interesting methodological move was the creation of the SYNTHETIC split. The authors did not use naive LLM generation. Instead, they used a browsing agent (Claude Code) as a "proposer." This agent was instructed to target specific failure modes identified in the human-verified set. The pipeline included an adversarial filtering step. A candidate question was only accepted if it was searchable but difficult. Furthermore, target models like GPT-5.4-mini and Gemini-3-Flash-Preview had to fail to solve it. This exploits the asymmetry between the ease of verifying an answer and the difficulty of creating the problem.

What They Found

The results reveal a massive performance gap. On the human-verified subset, the strongest frontier models achieved low scores. GPT-5.5 and DeepSeek-V4-Pro reached only 45.67% and 30.00% accuracy, respectively [Table 2]. This represents a substantial drop compared to their performance on the original English-centric BrowseComp. Most strikingly, Korean LLMs released through government-funded programs scored between 0.00% and 10.33% [Table 2]. This shows that even locally specialized models struggle with agentic browsing.

The authors' analysis shows that the bottleneck is rarely the initial search. In fact, for many models, incorrect trials actually utilized more search calls than correct ones [Table 3]. The failure is structural and happens post-retrieval. Through detailed trajectory analysis, the authors identified three dominant patterns of failure :

1. Candidate Capture: The model commits to a plausible entity before all constraints are met. This turns subsequent searches into mere confirmation bias for the wrong answer.
2. Unmerged Evidence Branches: The model retrieves relevant clues but fails to intersect them. It treats clues as separate observations rather than filters on a shared candidate set.
3. Misbound Evidence Chains: The model follows a logical path but assigns an intermediate result to the wrong role. For example, it might assign a university to the wrong political event.

Even the SYNTHETIC stress test showed low success. No model exceeded 26.00% accuracy on this split.

What This Changes

This research shifts the target for agent development. Simply increasing parameter count is not a silver bullet for agentic tasks. The findings imply a need for architectural improvements in how agents manage "state." Specifically, agents must improve how they maintain candidate ledgers, constraint sets, and entity-role bindings across multiple turns.

For practitioners, a model's "intelligence" in a chat interface is a poor proxy for its utility as a web agent. Reliability in the field depends on precise, page-level extractions. Agents must be able to read specific metadata fields from a table. They must also resist the urge to "hallucinate" connections between loosely related search results.

Developers building these systems should implement a structured "scratchpad" or "working memory" module. This module should explicitly track unsatisfied constraints and current candidate sets. Relying on the model's implicit context window to maintain state is clearly insufficient.

Figures from the paper