Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

When AI agents attempt complex, long-running tasks, they often lose track of their progress. Their internal summaries become cluttered and incoherent. This problem is common in memory-augmented agents. These systems handle massive information by condensing past interactions into a compact "memory" state. As these summaries are repeatedly rewritten, they accumulate semantic noise (meaningless or distracting data). They also discard critical details. This eventually causes the agent to drift away from the actual task requirements.

Existing approaches typically train these memory policies using outcome-based reinforcement learning. This method only rewards the agent if it succeeds at the very end of a long sequence. This creates a severe "credit assignment" problem. The model knows it failed. However, it does not know which specific summary in the middle caused the error. A new study introduces Metacognitive Memory Policy Optimization (MMPO). This method shifts the focus from mere task success to the clarity of the agent's internal "beliefs."

The breakdown of recursive summarization

Current memory-augmented agents treat interaction history as a stream that must be compressed. This fits the information into a finite context window (the maximum amount of text a model can process at once). This process is like a person taking notes during a long lecture. If the notes become too vague, the listener loses the thread of the argument. The authors argue that this compression induces "belief deviation" .

This occurs when the agent's internal estimate of the task state drifts from reality due to poor-quality summaries.

Standard training methods rely on sparse, terminal rewards. They essentially tell the agent "good job" or "bad job" only after the entire task is finished. The paper finds that this approach fails to localize where memory quality degrades. Because the reward signal is so distant from the intermediate steps, the model lacks granular feedback. It cannot easily suppress the accumulation of noise during the recursive summarization process. Consequently, as the task extends, the agent's reasoning collapses.

Probing the agent's internal certainty

To solve this, the researchers propose a mechanism to preserve "belief clarity." They model the agent's task as a Partially Observable Markov Decision Process (POMDP). This is a mathematical framework for environments where the true state is hidden .

In a summary-based agent, the agent's "belief" is its best guess about the current task state. This belief depends entirely on the quality of the compressed memory.

The core innovation is a metric called Belief Entropy (HBE). This acts as a "metacognitive probe" (a way for the model to monitor its own thinking). Instead of observing the hidden task state directly, the authors ask the model an "anchor question." This question probes the agent's own progress. An example is: "Based on current memory, what is the current task progress and what information is still needed?" The uncertainty in the model's response serves as a proxy for how much information is preserved in the memory.

The MMPO training pipeline operates in three stages : 1.

Trajectory Sampling: The memory policy generates multiple different reasoning paths for a single task. 2. Dense Reward Computation: At every turn, the system calculates the Belief Entropy for the current summary. This provides a "dense" reward. This means the agent gets feedback at every step rather than just at the end. 3. Group-Relative Optimization: Using a technique similar to Group Relative Policy Optimization (GRPO), the system compares different paths at the same time step. It reinforces summaries that lead to both low uncertainty and eventual task success.

Maintaining stability at extreme scales

The authors report that this dense supervision stabilizes long-horizon reasoning. In experiments using the RULER-HotpotQA benchmark, successful trajectories showed a decrease in Belief Entropy as evidence was gathered. Failed trajectories tended to show increasing entropy .

This correlation is strong. The paper reports a Pearson correlation of -0.684 between entropy reduction and task accuracy [Figure 3b].

The performance gains are notable as the context length increases. Many models struggle as the history grows. However, the authors report that MMPO maintains 97.1% performance even at 1.75M-token contexts. Specifically, compared to the RL-MemAgent baseline using a Qwen2.5-7B backbone, MMPO showed its largest gains at the 896K token mark. It improved accuracy by 5.47% at this stage. The study also shows that this works at inference time. Even without retraining, selecting the trajectory with the lowest Belief Entropy among five candidates improves accuracy [Figure 3c].

Limits of the metacognitive proxy

The authors acknowledge several technical trade-offs. First, Belief Entropy is a proxy. It is not a direct measurement of the hidden task state. It assumes that response uncertainty accurately reflects uncertainty about the task. If the anchor question is poorly designed, the reward signal could become misleading. The effectiveness of the method depends heavily on the design of these probes.

Second, there is a computational cost. Calculating Belief Entropy requires an extra forward pass of the model at every turn during training. The authors report this adds approximately 12% wall-clock overhead. While this cost disappears during standard inference, it increases the resources needed for training. Finally, the paper emphasizes that MMPO is a reasoning stabilizer. It does not inherently guarantee factual correctness if the underlying observations are flawed.

The verdict: a new standard for agentic memory

The evidence suggests that MMPO is an effective framework for long-term reasoning. It moves away from "all-or-nothing" terminal rewards. Instead, it uses a "process-oriented" view of memory clarity. This successfully addresses the credit assignment problem in long-horizon tasks. The ability to maintain high performance at the million-token scale is a significant achievement. It brings us closer to truly autonomous, long-running AI assistants.

For practitioners building agents for complex environments, the takeaway is clear. Optimizing for the clarity of the agent's internal state is vital. This is just as important as optimizing for the final answer. If you build memory-augmented systems, the dual-probe approach appears most robust. Tracking both progress and information gaps provides the best way to implement metacognitive oversight.

Figures from the paper