SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

The Hidden Lifecycle of Agentic Attacks

AI agents are moving from chatbots to autonomous workers. They use "skills"—reusable packages of code, instructions, and reference materials—to perform specialized tasks. Agents are designed to implicitly follow these skills. This creates a massive, unexamined attack surface. If a third-party skill is poisoned, the agent might execute malicious code. It might also leak sensitive data without realizing it has deviated from its mission.

Current research mostly looks at single instances of harm. The industry treats skill-based security as a one-off event. A poisoned file is loaded, a task is run, and we check for breakage. But in production, skills are persistent. They are installed once and reused across many sessions. This paper argues that ignoring the lifecycle of these artifacts misses the most dangerous failure modes.

The Question

The researchers investigated whether current autonomous agents are vulnerable to skill-based attacks that exploit the entire lifecycle of a skill. They wanted to move beyond the assumption that an attack must be immediate. Can an attacker plant a "time bomb" in a skill? This bomb could stay dormant during an initial, benign task. It could then mutate the skill's content to trigger harm during a different session later. Furthermore, they sought to quantify if agent "safety" is actually just the agent failing to interact with poisoned files.

Why The Old Answer Was Incomplete

Until now, literature on skill-based attacks has had a narrow temporal view. Most benchmarks evaluate poisoned skills within a single task execution. They judge success based on whether the agent exhibits unsafe behavior during that specific run. This approach ignores the reality of skill reuse. As shown in the comparison of existing benchmarks in [Table 1], previous studies primarily focused on single-session compromises.

This leaves a blind spot regarding deferred harm. If an attack does not manifest in the first session, existing benchmarks mark it as a failure. This effectively grants the attacker a "pass" for a successful, silent mutation. Additionally, the field has relied on ad-hoc risk lists rather than a systematic taxonomy. This makes it difficult to understand if agents fail due to specific capability exploitations. These include data exfiltration (sending data to an external destination) or system corruption (damaging runtime configurations).

What They Did

The authors developed SkillHarm, a benchmark evaluating two distinct attack scenarios. The first is Fixed-Payload Poisoning (FPP). In FPP, a poisoned skill directly triggers harm during the first invocation. The second is Self-Mutating Poisoning (SMP). In SMP, the attacker uses a "mutation hook"—such as an atexit callback (a function that runs when a program exits)—to silently modify persistent skill files. The harm only materializes when a later, unrelated task reuses that compromised package, as illustrated in .

To populate this benchmark, they built AutoSkillHarm, an automated pipeline. Instead of manual design, they used coding agents driven by natural-language harnesses. The pipeline follows a three-stage process. First, it selects reachable attack targets. Second, it designs contextualized payloads with deterministic evaluators (scripts that check for specific harmful end-states). Third, it applies a quality filter via reviewer agents. This ensures the attacks are executable and plausible. The resulting dataset contains 879 attack samples across 71 skills and 12 risk types.

What They Found

The results suggest that current frontier agents are deeply vulnerable. The authors report Attack Success Rates (ASR) reaching up to 86.3% for FPP and 69.3% for SMP. Even though SMP requires the harder feat of mutating a file and triggering it later, the success rate remains high.

Crucially, the study reveals that many "failed" attacks are not actually successes for safety training. The authors introduced Conditional ASR (cASR). This measures success only when the agent actually engages with the poisoned file. The gap between ASR and cASR shows that vulnerability is much higher than raw ASR suggests. For example, in the SMP scenario, the gap can be as large as 32.1% for certain models. This implies that agent safety often stems from "non-engagement." The agent simply did not bother to read the malicious file. Furthermore, standard defenses, including commercial skill scanners and defensive system prompts, failed to reliably mitigate these threats.

What This Changes

Implicit trust in third-party skill packages is a critical security flaw. We cannot rely on prompt-level refusals or static scanners to protect the system.

First, the distinction between ASR and cASR means that "safety" metrics are currently deceptive. If an agent is "safe" only because it ignores 40% of its context, it is not truly secure. It is merely incomplete. Improving an agent's utility and ability to follow instructions will likely drive ASR upward. This could strip away this accidental layer of protection.

Second, the existence of SMP necessitates a shift toward runtime monitoring. Since attackers can use legitimate execution flows to mutate persistent artifacts, security layers must change. Systems should monitor for unauthorized or suspicious file modifications performed by the agent itself.

The paper does not demonstrate how these attacks behave in multi-tenant environments. Testing if an attacker can poison a skill used across different users is a logical next step.

Figures from the paper