Agent Skills Should Go Beyond Text: The Case for Visual Skills

Most AI agents currently learn tasks using only text instructions. This makes them struggle with visual details like where to click or how to count objects. This reliance on language creates a massive blind spot for any agent tasked with navigating a GUI (Graphical User Interface) or inspecting a complex image. A new paper argues that we need to stop treating agent experience as a collection of text prompts. Instead, we should treat it as a multimodal asset. They propose VISUAL SKILL, a paradigm that attaches explicit visual priors—like spatial maps or interactive markers—to textual reasoning.

The Problem

The status quo in agent development relies heavily on textual abstractions. When we build "skills" for an agent, we typically distill past experiences into structured natural language. This includes instructions, reasoning traces, or summarized trajectories. This works for symbolic tasks like SQL querying or API orchestration. In those cases, the logic is inherently linguistic. However, for visual-centric tasks, this creates a "textual bottleneck."

As shown in, traditional text-only skills attempt to compress high-dimensional spatial topology into one-dimensional symbolic sequences.

This fails in two specific ways. First, there is static protocol ambiguity. Text can describe what to do. But it is notoriously bad at encoding fine-grained spatial conventions. Examples include the exact clickable hitbox (the active area for a mouse click) of an icon or the whitespace rhythm of a layout. Second, there is dynamic tracking collapse. Tasks like dense counting or maze solving require a persistent record of which regions have been inspected. Relying on a text-only chain of thought to manage this spatial bookkeeping leads to rapid ambiguity. This causes the double-counting and omissions common in current visual agents.

How It Works

The authors propose moving from text-only instructions to a multimodal entity defined as $S_v = (L, P_v, B)$. This breaks the skill into three distinct layers:

1. Declarative Textual Logic ($L$): This handles the high-level "brain" work. It parses objectives, defines execution steps, and manages error handling. It retains the compositionality and interpretability of standard LLM (Large Language Model) prompting.
2. Reusable Visual Support ($P_v$): This is the core innovation. Instead of just text, the skill includes visual assets tailored to the specific bottleneck. The paper identifies three forms in :
3. Static Priors: External references (like wireframes or templates) that encode stable spatial conventions.

• Dynamic Priors: Executable protocols that allow the agent to write intermediate state—such as numbered anchors or trajectories—directly back onto the task image. This creates a visual working memory.
• Interleaved Visual Skills: A structure that binds specific textual steps to their supporting visual evidence, such as a specific screenshot or a video keyframe.
• Multimodal Binding Protocol ($B$): This is the coordination layer. It dictates exactly when the agent should pull in a visual prior versus when it should rely on text alone.

To make this scalable, they introduce AUTOVISUALSKILL. This is an automated pipeline that analyzes task trajectories. It diagnoses these visual bottlenecks and packages the resulting text and visual assets into a reusable directory .

Numbers

The authors evaluate this by measuring "Textual Degradation Rate" (TDR). This is the performance gap between a visual-skill setup and a text-only setup using the exact same textual rules.

On GUI grounding tasks (using ScreenSpot and GroundUI-18K), the authors report that adding static visual priors consistently outperforms text-only skills. Specifically, for ScreenSpot, the Visual Skill achieved a Point-in-Box Accuracy of 0.930. This was higher than the 0.901 achieved by text-only skills and the 0.873 from no-skill direct prompting [Table 1]. The improvements were most pronounced in Mean IoU (Intersection over Union, a measure of overlap accuracy). This suggests that visual priors help the model calibrate the actual extent and boundaries of UI elements.

The impact on dense perception is even sharper. In object counting tasks on CountBenchQA, the authors found that text-only skills could actually degrade performance compared to direct prompting. This likely happens because extra instructions increase reasoning overhead without a way to track state. However, equipping the agent with a dynamic prior (drawing anchors on the image) boosted exact-match accuracy to 97.12%. It also slashed the Mean Absolute Error (MAE, the average magnitude of errors) to 0.0535. This represents a nearly 60% reduction in error compared to the text-only baseline [Table 2].

What's Missing

While the results are compelling, there are a few gaps that a production engineer should note.

First, the paper acknowledges that visual skills aren't a silver bullet. As seen in, they can introduce new failure modes.

A static prior might "over-specialize." This causes the agent to focus too narrowly on a tiny icon (the glyph) rather than the larger functional container the user actually intended to interact with. There is a tension between structural spatial priors and semantic intent that remains.

Second, the "automatic" part of the AUTOVISUALSKILL pipeline is presented as a proof-of-concept. The paper does not dive deeply into the computational cost of the "diagnostic gate." It also does not detail the complexity of the generation loop. In a real-world production environment, determining whether a task needs a dynamic vs. static prior in real-time could introduce latency.

Finally, the paper focuses heavily on "clean" benchmarks. It remains to be seen how these visual artifacts hold up in high-entropy, unstructured environments. In those settings, the "stable spatial conventions" the priors rely on might not exist.

Should You Prototype This

Yes, if you are building specialized agents for GUI automation or visual inspection.

If your agent's primary failure mode is "hallucinating" clicks or losing track of progress in long-horizon visual tasks, this architecture is the correct move. The ability to externalize state via dynamic anchors (drawing on the screen) is a proven way to bypass the limitations of an LLM's internal context window.

However, do not expect a drop-in replacement for your current prompt library. This requires a shift in how you version and store agent experience. You must move from .txt files to structured directories containing images and manifests. You can find the codebase at https://github.com/Little-Fridge/AutoVisualSkill. Start by prototyping the "dynamic prior" for your most repetitive spatial tasks. The ROI on reducing counting or tracking errors is likely higher than trying to automate the entire skill-authoring pipeline immediately.

Figures from the paper