Skill Reuse as Compression in Agentic RL

REUSERL: Boosting LLM Agent Generalization via Skill Reuse as Compression

When training large language model (LLM) agents with reinforcement learning (RL), we often watch them develop incredibly efficient but fundamentally broken behaviors. They learn to solve a specific training task by exploiting a narrow, brittle shortcut. They memorize a sequence of actions that works perfectly in the training simulator. This behavior collapses the moment the environment shifts slightly.

In agentic RL, the goal is to move beyond simple task completion toward generalizable reasoning. Current methods optimize policies through GRPO (Group Relative Policy Optimization, a method that compares groups of samples to calculate advantages) to maximize success rates. However, a critical question remains. How do we ensure the agent learns transferable logic rather than just memorizing a specific path to a reward? This paper proposes that the answer lies in information theory. They suggest that agents generalize better when their successful behaviors are "compressible." This means they can be decomposed into a small set of reusable, abstract skill patterns.

The Problem

The status quo in agentic RL suffers from "reasoning collapse." When we reward an agent solely for task success, the optimization process tends to amplify idiosyncratic, one-off solution patterns. These patterns satisfy the immediate training objective. However, they lack any structural utility for new tasks.

As shown in, there is a massive difference between a "short" trajectory and a "compressible" one.

Standard GRPO optimizes for success alone. This often leads to long, wasteful trajectories filled with repeated or irrelevant steps. A common engineering instinct is to simply penalize the number of steps taken (a "round-length penalty") to force efficiency. However, the authors demonstrate that this is a blunt instrument. As seen in the TextWorld-Cooking case study, a pure length penalty can actually backfire. It causes the agent to perform "cookbook-literal" shortcuts. These lead to irreversible failures, such as burning food because the agent tried to cook an ingredient that was already cooked. The agent becomes "brief" but loses the ability to follow the underlying logic.

How It Works

The authors introduce REUSERL, a framework that treats skill reuse as a data compression problem. It is grounded in the Minimum Description Length (MDL) principle (a way to measure complexity by how much data is needed to describe it). Instead of just looking at how many steps an agent takes, REUSERL looks at how many unique pieces of information are required to describe its successful behavior.

The architecture operates as an EM-style (Expectation-Maximization, an iterative method to find maximum likelihood estimates) alternating optimization loop. It is integrated directly into the GRPO training cycle:

1. Skill Projection ($\phi$): Raw agent actions are mapped to a small, discrete alphabet of "atomic skills" (e.g., EXPLORE, TAKE, TRANSFORM). This abstracts away low-level token noise. It focuses on high-level behavioral primitives.
2. E-step (Dictionary Extraction): Using the current batch of successful trajectories ($B^+$), the system extracts a shared skill dictionary ($C$). They implement this via a greedy Byte Pair Encoding (BPE, a method for merging frequent character sequences) approach. It merges frequently occurring adjacent skill pairs into reusable "phrases" (e.g., CLEAN-THEN-PLACE). This process continues as long as adding a new phrase strictly decreases the total description length of the batch.
3. M-step (Policy Update): The RL objective is augmented with a "segmentation cost" (a penalty based on how many dictionary phrases are needed to cover a sequence). For every successful trajectory, the agent receives a reward shaped by how efficiently that trajectory can be encoded using the newly extracted dictionary.

Mathematically, the agent is penalized by $\text{SegCost}(s_i | C) = \text{seg}(s_i, C)/T$. A trajectory that follows a well-established, reusable subroutine is "cheap" to describe and thus rewarded. Conversely, a trajectory that relies on a weird, one-off sequence of actions is "expensive" and penalized. This forces the policy to converge on behaviors that compose from a compact repertoire of skills.

Numbers

The authors report significant improvements in both in-distribution and out-of-distribution (OOD, data not seen during training) performance across three benchmarks. The most striking result is on the ALFWorld OOD split. REUSERL achieves a success rate (SC@7) of 93.28%. This beats vanilla GRPO (79.85%) and the pure round-length baseline (91.79%).

In the TextWorld-Cooking environment, the delta is even more dramatic. The paper finds that the pure round-length penalty causes the agent to fail via "burned food" 74% of the time. In contrast, REUSERL-SegCost reduces these failures to just 3.3%. It achieves a Pass@1 of 81.73% compared to the 64.03% seen in the length-penalty baseline.

Regarding implementation overhead, the authors note that the greedy BPE dictionary extraction is highly efficient. On synthetic tests, the BPE approach ran 285$\times$ to 2231$\times$ faster than brute-force optimal dictionary searches. This makes it practical to run inline with every GRPO update. The training was conducted on Nvidia RTX Pro 6000 GPUs using the verl-agent framework.

What's Missing

While the results are compelling, there are gaps that a practitioner should consider:

• Manual Schema Design: The skill projection $\phi$ is currently a rule-based mapping. Moving to a new environment requires manual engineering of the atomic skill alphabet. The authors argue this is a fair trade-off for verifiability. However, it prevents the system from being truly "plug-and-play" in novel, unstructured domains.
• Scaling Limits: The experiments were limited to relatively small backbones (1.5B–1.7B parameters). It is unclear if the compression signal remains effective for 70B+ parameter models. These larger models might already possess sufficient internal structural priors.
• The Complexity Assumption: The theoretical guarantee (Theorem 3) relies on the assumption that successful trajectories actually admit a shared, bounded decomposition. In highly chaotic or continuous environments, "skills" might not be clearly defined. In such cases, the segmentation cost may simply degenerate into a standard length penalty.

Should You Prototype This

Yes, if you are building specialized agents for structured environments.

If your deployment target involves tasks with repeatable subroutines—like robotics or warehouse logistics—REUSERL is a high-signal intervention. It solves the specific problem of "shortcut learning" that plagues standard RL. However, if you are working in a highly open-ended domain, the engineering cost of designing the skill alphabet might outweigh the generalization gains.

The implementation is lightweight enough to be added to an existing GRPO loop. The BPE-based extraction is also computationally inexpensive. Start by defining your atomic skills and implementing the rule-based projection. Monitor your "invalid action" and "repeat" rates as early indicators of success. Code is reportedly available under an open-source license; see the paper for the canonical link.