Learning User Simulators with Turing Rewards

Most work on user simulation frames the problem as response imitation. We typically train a large language model (LLM) to match a specific ground truth response. This is done by maximizing log probabilities or using a similarity reward. However, this assumes there is only one "correct" thing a person could say. In reality, the set of plausible responses is massive. The same person might respond in many different ways depending on their mood or intent.

This paper enters the field of user simulation. This is the practice of creating digital proxies for human users. These proxies can train AI assistants or study social behaviors at scale. Instead of teaching an AI to copy exactly what a person said, the authors propose training it to act so much like that person that even a judge can't tell the difference. They call this approach Turing-RL.

The failure of response matching

The central tension in current user simulation is the gap between matching and indistinguishability. Existing approaches generally rely on two signals. First, they maximize the log probability of the ground truth. This asks "how likely is this specific sentence?" Second, they use a similarity reward. This asks "does this sentence mean the same thing as the original?"

The authors argue that these signals are fundamentally mismatched with the goal of simulation. A simulator that only learns to replicate a single ground truth response fails to capture human variability. Furthermore, optimizing for content overlap can lead to "robotic" or assistant-like outputs. A model could pass a similarity test but still fail to capture the unique "voice" of a user. A simulator should not just say what the user said. It should produce what the user could have said.

Implementing the Turing reward

The Turing-RL pipeline moves away from static targets toward a discriminative training signal. As illustrated in, the process begins with an SFT (Supervised Fine-Tuning; adjusting a model on a specific dataset) warm-start.

In this stage, the model is trained on ground truth responses. These responses are augmented with Chain-of-Thought (CoT; a technique where the model generates intermediate reasoning steps) traces. These traces explain the underlying intent and style that would lead a human to produce a specific response.

The core innovation is the discriminative Turing reward. This is implemented via the following mechanism:

1. Candidate Generation: For a given user history and context, the policy generates multiple candidate responses.
2. Pairwise Judging: An LLM judge (specifically Qwen3.5-397B-A17B) is presented with a pair. It sees one real human response ($y^\star$) and one model-generated response ($y$).
3. Likert Scoring: The judge assigns a score from 1 to 7. This score indicates which response is more likely to have been written by the human. This judgment is conditioned on the user's history.
4. Policy Optimization: The model is trained using Group Relative Policy Optimization (GRPO; a reinforcement learning algorithm that optimizes a policy by comparing rewards within a group of sampled candidates).

To prevent "reward hacking"—where a model learns to trick the judge with unnatural perfection—the authors cap the reward. They normalize the reward at a maximum of 5 on the 7-point scale .

Performance across domains

The authors evaluate Turing-RL across two distinct environments. These include multi-turn conversational chat (the PRISM dataset) and Reddit forum discussions (the ConvoKit corpus). They report that Turing-RL consistently outperforms both the similarity-based baseline (Sim-RL) and the log-probability baseline (Logprob-RL) on human-likeness metrics.

In the chat domain, the improvement is quite large. The paper reports that Turing-RL achieves a human win rate of 0.57 [Table 1]. This means the model's responses were chosen over the real human's responses in 57% of cases. In contrast, the SFT-initialized and Sim-RL models stayed near the 0.50 chance level. This suggests Turing-RL is significantly harder for humans to distinguish from real users in chat.

Interestingly, Turing-RL does not sacrifice content alignment. It performs comparably to Sim-RL regarding response similarity to the ground truth .

This means the model stays semantically accurate while becoming more human-like.

In the Reddit domain, the task appears much harder. The authors observe that human win rates for all models are closer to chance. They note that Reddit discussions are more difficult for human annotators to judge than chat sessions [Table 10]. Despite this, the LLM-as-a-judge metrics suggest that Turing-RL maintains its edge. It produces responses that feel authentic to the specific user's posting style.

Limitations and risks

There are several caveats to consider. First, the success of the method depends heavily on the quality of the LLM judge. The authors admit that the judge (Qwen3.5-397B-A17B) might have systematic blind spots. If the judge has biases, the simulator may learn to exploit them. This would result in "human-likeness" that is actually just a trick.

Second, the scalability of the approach remains unstudied. The experiments used a Qwen3-8B base model. It is unclear how this gap would behave with frontier-scale models. The results from small models may not generalize to much larger ones.

Finally, there is a significant dual-use concern. A model optimized to be indistinguishable from a specific individual is an effective impersonation engine. Such technology could be used for social engineering or fraud. The authors argue that development must include safeguards. These could include watermarking or improved AI generation detectors.

The verdict

Turing-RL represents a shift in how we think about agentic training. It moves from "match this string" to "pass this test." This allows the model to prioritize the stylistic nuances that make human interaction feel real. The model achieves higher indistinguishability without losing semantic accuracy. This is a vital discovery for anyone building personalized AI.

If you are building a simulator for testing agents or social science research, this is a recommended direction. The code is reportedly available at https://github.com/SusanWYS/turing-rl.git. Just watch out for the "assistant-like" trap. Even with these rewards, LLMs often default to polite, hedged, and verbose prose. This remains a hurdle for true human-likeness.

Figures from the paper