LLM Agents Can See Code Repositories

Can Agents Navigate via Sight?

Current AI coding assistants mostly read text. Humans use visual cues like folder structures and dependency relationships to orient themselves in large codebases. This study explores whether giving AI agents visual maps of a repository's structure can help them find bugs faster. It aims to use fewer computing resources without losing accuracy. The central tension is whether Multimodal Large Language Models (MLLMs)—models capable of processing both text and images—can provide a functional advantage for software engineering.

Does visual structure aid repository comprehension?

The authors of this paper ask a fundamental question regarding the interface between agents and code. Is text the most effective modality for presenting repository context to modern foundation models? While coding agents have shown remarkable proficiency in resolving issues in large repositories, they currently rely almost entirely on tokenized text. They flatten source code, documentation, and execution feedback into linear sequences for reasoning.

The researchers hypothesize that this text-centric paradigm forces models to infer structural information. This includes things like function dependencies or file hierarchies. These were originally conveyed through non-linear or visual representations. They ask whether adding visual modalities can help agents grasp the codebase more effectively or efficiently. This involves using rendered images of the repository's topology (the structural arrangement of parts). This isn't just about "seeing" code. It is about whether the spatial arrangement of a dependency graph provides a richer information density per unit of prompt than a serialized text description.

The limitations of the text-only paradigm

Until now, the prevailing approach in software engineering (SE) agent research has been to linearize heterogeneous artifacts into sequential inputs. Even when prior work has utilized graph-based abstractions to represent code, the information eventually consumed by the model at inference time is typically converted back into text tokens. The authors argue that this serialization leads to a loss of crucial relational cues.

Existing agents operate via a "grep-then-read" cycle. They search for keywords and then read a file. Often, they realize it is the wrong context and must repeat the process. This process is inherently inefficient. As shown in the case study of the astropy library, a baseline agent relying on shell-based navigation often suffers from "misses." These are repeated accesses to irrelevant files before finally hitting the correct target.

This reliance on trial-and-error textual exploration consumes significant amounts of context window. It also increases the cognitive load on the model to reconstruct the "map" of the repository from fragmented text snippets.

Testing the multimodal hypothesis

To investigate this, the authors developed SeeRepo. This is a framework that augments agents with visual renderings of repository structure. Using AST-based (Abstract Syntax Tree) static analysis, SeeRepo constructs dependency graphs. This method parses code into a tree structure that represents its logical hierarchy. The graphs capture containment, imports, invocations, and inheritance. These graphs are rendered as PNG images using Graphviz. The agent can query these images as a tool.

The researchers conducted a systematic empirical study across four multimodal models: GPT-5-mini, GPT-5.1, Doubao-Seed-2.0-Lite, and Kimi K2.5. They did not just test a "vision-only" mode. They investigated the spectrum of integration. Their most striking move was testing a "vision-only" setting. In this mode, the agent's bash commands returned graph images instead of text. This allowed them to establish a performance boundary. They wanted to see if vision suffices or if it is merely a supplement.

The experimental design also examined several design dimensions. These included visual layout (Graph, Nested, or Tabular) as seen in .

They also studied the optimal stage for tool invocation. They tested whether the agent should "look" at the map during localization (finding the bug), repair (fixing the bug), or validation (testing the fix).

Hybrid modality wins the efficiency game

The findings reveal a nuanced reality. Vision cannot replace text. First, the authors report that a vision-only approach is a failure. It significantly degrades accuracy and causes token costs to explode. For example, Doubao-Seed-2.0-Lite saw a 34.1 point drop in accuracy. Its cost also increased by 268% [Table 1]. The agents essentially became lost. They performed repeated, expensive visual queries to compensate for the lack of readable symbolic text.

However, the "hybrid" approach produced a compelling result. This approach provides visual graphs as a supplement to text. The authors find that integrating SeeRepo reduces input token consumption by up to 26%. It does this while maintaining or even improving issue-resolution accuracy. For GPT-5-mini, Pass@1 accuracy (the percentage of successful resolutions on the first try) improved slightly by 0.4%. Meanwhile, costs dropped by 26% [Table 2].

The most significant takeaway is that visualization acts as a "navigation prior." Instead of the iterative grep cycles seen in the baseline, the multimodal agent can jump directly to the relevant structural neighborhood . Furthermore, the study found that these visual tools are most effective during the fault localization stage. If the agent waits until the repair or validation stage to use the visual tool, the extra structural information becomes "noise." This distracts the model from the precise textual reasoning required for code editing [Table 7].

Implications for the next generation of agents

The implications of this work suggest a shift in how we design Agent-Computer Interfaces (ACIs). If these results generalize, we should treat MLLMs as spatial reasoners. They can navigate complex topologies rather than acting as mere "image captioners" for code.

There are two immediate consequences for practitioners: 1. Modality Budgeting: Developers building coding agents should not attempt to replace text with vision. Instead, they should use vision to compress the structural search space. They should reserve text for the high-precision symbolic work of writing the actual patch. 2. Stage-Aware Tooling: Tool-calling policies should be conditioned on the agent's current task phase. An agent should be encouraged to "look at the map" when it is lost in a directory structure. However, it should be discouraged from doing so once it has settled into the granular logic of a specific function.

I am roughly 80% confident that this efficiency gain is a result of reducing the "exploration tax" inherent in text-only navigation. However, the paper does not show whether this scales to truly massive repositories. In very large codebases, a single Graphviz rendering might become a visual "hairball." This could exceed the model's visual resolution limits. A vital follow-up experiment would be to test this hybrid approach on repositories with much higher cyclomatic complexity (a measure of the number of paths through a program). This would show at what point the visual modality hits a ceiling of utility.

Figures from the paper