MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery

Task Success is a Poor Proxy for Memory Quality

Current LLM agents are increasingly designed to possess long-term memory. This promises assistants that grow more capable and personalized across multiple sessions. However, we currently lack a reliable way to know if an agent is actually building a durable model of a user. Most existing benchmarks evaluate memory indirectly through downstream behavior. They ask if the agent answered a question correctly or completed a task successfully.

The problem is that high task success does not prove the existence of a functional memory artifact. An agent can solve a problem by retrieving a single relevant snippet. It might rely on the immediate conversation context or even make a lucky guess. None of these methods require a structured understanding of the user. This creates a massive observability gap. We are optimizing for task completion while leaving the actual memory state largely unaudited.

The disconnect between behavior and state

The status quo in agent evaluation relies on behavioral proxies. As shown in, traditional benchmarks focus on whether the agent answered correctly or succeeded at a task.

While these metrics are useful for measuring immediate utility, they are weak diagnostics for memory quality. The authors of MEMPROBE argue that a high task-success score is a deceptive signal. In their experiments, even a memoryless baseline achieved near-perfect task success.

This creates a fundamental decoupling. You can have an agent that is excellent at helping a user navigate a specific workflow. Yet, it might fail to retain any structured information about that user's preferences, skills, or history once the session ends. To bridge this gap, we need to stop looking at what the agent does. We must start looking at what the agent leaves behind.

Auditing the memory artifact

To solve this, the researchers propose treating memory as an auditable, post-interaction artifact. Instead of measuring task success, they measure "recovery." This is the ability to reconstruct a hidden user state from the memory the agent produced. The MEMPROBE pipeline, illustrated in, operates through three distinct phases:

1. Expose: A simulated user interacts with the agent. This user carries a hidden, taxonomy-anchored bank (a structured set of predefined categories) of 31 dimensions. These dimensions cover skills, knowledge, episodic memory (one-off past events), self-models, and preferences. Through "leak-controlled" tasks, the user gradually reveals evidence of their hidden state. Crucially, the tasks never name the target attributes directly.
2. Collect: The agent processes the interaction. It executes its memory policy to decide whether to write, merge, or revise entries in its persistent store.
3. Recover: After the interaction, a separate process attempts to reconstruct the original hidden user state from the resulting memory.

The authors implement a two-mode audit to separate storage from accessibility. They use a dump_all mode, which reads the entire memory store. This checks if the evidence was ever written. They also use a retrieve mode, which uses the system's own top-$k$ search interface. This checks if that evidence is actually reachable. This distinction allows engineers to pinpoint failures. They can see if a problem is on the "write-side" (the agent failed to capture the info) or the "read-side" (the info is there, but retrieval fails).

Moderate recovery and the episodic bottleneck

The empirical results highlight a significant performance ceiling. The paper reports that while task completion rates saturate near 99.9%, the category-balanced reconstruction score ($B$) stays moderate. This score hovers around 0.6. This means the agent only recovers about 60% of the intended user state. Performance drops even further when moving from a full-store dump to a top-$k$ retrieval mode.

The breakdown by category in [Table 3] reveals where the real engineering challenges lie. Assistance preferences are relatively easy to recover. Scores are often above 0.7. This is likely because users explicitly state their preferences during dialogue. However, episodic memory is consistently the hardest category to reconstruct. As demonstrated in the case study in, recovering a specific piece of knowledge requires the agent to bind a discrete event to its specific context and consequence.

Simply logging that "an event happened" is insufficient. The agent must capture the meaning of the event to allow for successful recovery.

Limitations in the synthetic sandbox

While the methodology is rigorous, there are clear boundaries to what these results represent. The benchmark relies heavily on synthetic users and generated tasks. This is necessary for establishing a ground-truth "gold standard." However, it may not capture the chaotic nature of real human interaction. The simulator follows a predictable path toward disclosure. Real users rarely provide such clean signals.

Additionally, the authors note that the scoring mechanism is sensitive to the surface-form choices of the LLM judge. As seen in the "directional slot-label polarity" case study, a system might be penalized unexpectedly. This happens if the descriptor used (e.g., "high tolerance" vs. "low tolerance") is interpreted inconsistently by the evaluator. For practitioners, this means that while MEMPROBE is a powerful diagnostic tool, absolute scores should be viewed as relative indicators. They show trends in improvement rather than absolute measures of intelligence.

Verdict: A necessary pivot for agent design

If you are building agents where user continuity is a core requirement, MEMPROBE is a vital framework. The paper makes it clear: simply increasing the context window is not enough. Adding a RAG (Retrieval-Augmented Generation) layer is also not a substitute for a formal memory policy.

The findings suggest that the next frontier for memory agents is "better consolidation." Agents need write policies that actively distill raw interaction traces. They must turn these traces into compact, high-signal user-state claims. The gap between what is written (dump_all) and what is reachable (retrieve) is the most actionable metric here. If your agent's recovery score collapses in retrieval mode, stop tuning your embedding model. Start looking at how your agent summarizes and organizes its notes.

Code and data are available at https://github.com/sora1998/MemProbe.