Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

Instead of training an AI to follow just one single reward, this method trains it to handle a range of possible rewards. This prevents the AI from getting stuck in one repetitive way of acting. It helps the model stay creative and flexible.

In classical reinforcement learning (RL), the goal is usually to find a single, deterministic policy (a fixed rule for choosing actions) that maximizes a scalar reward. However, modern applications like fine-tuning large language models (LLMs) face ambiguity. Users might have multiple valid preferences. Our reward models might also be imperfect proxies for truth. When we force a model to commit to one "best" action, we often trigger mode collapse (the loss of behavioral variety).

The Problem

Current methods rely on two crutches to induce diversity: entropy regularization and diversity bonuses. Entropy regularization adds a penalty to the objective. This prevents the policy from becoming deterministic. The downside is a forced trade-off. You gain stochasticity (randomness), but you almost always sacrifice expected reward to get it.

Diversity bonuses reward the model for picking unique actions. As the authors demonstrate in [Figure 2c], these methods often warp the optimization landscape. They can lead to "undesirable ordering." This happens when a policy with lower average reward but higher diversity is ranked higher than a high-reward policy. Essentially, you optimize for the bonus instead of the task. Entropy regularization also forces the model to put mass on everything. This includes actions that are objectively terrible [Figure 2b].

How It Works

The authors propose a framework called Randomized Objectives, Set Actions (ROSA). The core insight is that diversity should be a rational response to reward uncertainty. Instead of a single scalar reward, they treat the reward as a distribution $\rho$ over multiple potential reward functions.

The mechanism uses three primary shifts:

1. Distributional Rewards: The objective assumes the reward $R$ is drawn from a distribution $\rho$. This represents epistemic uncertainty (gaps in knowledge) or ambiguous preferences.
2. Set-Based Aggregation: The objective looks at a multiset of $n$ actions sampled i.i.d. (independently and identically distributed) from the policy. The authors highlight the ROSA+Max criterion. This optimizes the expected maximum reward found within that set across the distribution of rewards.
3. Non-Linear Objective: Using a non-linear function like max over the set of actions avoids the "averaging" trap. Standard policy gradients over a distribution of rewards tend to collapse the policy toward a mediocre average behavior. ROSA encourages the policy to cover all possible optimal actions.

As shown in [Figure 1b], this allows the policy to remain flexible. If one reward function prefers action A and another prefers action B, ROSA+Max incentivizes the policy to assign probability mass to both. It does not settle on a middle-ground action that satisfies neither.

Numbers

The authors validate ROSA across several regimes. In a task with competing, contradictory preferences, vanilla PG and diversity bonuses fail completely. They suffer from reward cancellation, which results in no learning .

In this case, the pass@k curve (a metric for how often at least one sample is correct) is identical to an untrained model. ROSA+Max and ROSA+Softmax successfully balance both preferences.

In LLM length diversity tasks, ROSA achieves the same overall correctness as regular policy gradient. However, it can also sample from all requested length modes, such as short and long responses .

Regarding optimization, the choice of the action set size $n$ is a critical hyperparameter. Any $n \ge 2$ can reach the optimal policy. However, the curvature of the landscape matters. The authors find that a very large $n$ results in a flat landscape. This flatness can slow down optimization. They suggest a sensible range of $2 \le n < 2m$, where $m$ is the number of reward functions.

What's Missing

While the theoretical grounding is strong, there are practical gaps:

• Complexity of Reward Sampling: The method assumes you can sample from a distribution of reward functions $\rho$. Constructing a meaningful, sampleable distribution of reward models is non-trivial in production.
• Scaling to Large Action Spaces: The computational cost scales with the number of sampled actions $n$ and reward functions $m$. For massive vocabularies, the overhead of calculating the max-of-$n$ advantages could impact training throughput.
• Hyperparameter Sensitivity: The framework introduces new moving parts. The interplay between the number of reward samples $m$ and the action samples $n$ is documented. However, the paper does not deeply explore the cost-benefit of increasing $m$ versus $n$ in resource-constrained environments.

Should You Prototype This

Yes, if you are dealing with multi-modal preferences.

Use ROSA if you are fighting mode collapse in LLM alignment. It is also useful if you want to prevent reward model over-optimization. It is valuable when you know your reward model is an imperfect proxy. In these cases, you want the agent to maintain a "portfolio" of high-quality behaviors. If your reward is a stable, unambiguous scalar, the added complexity of sampling reward distributions is likely unnecessary overhead. Start with a small $n$ (e.g., $n=2$ or $n=4$) to minimize the immediate hit to your training loop.

Figures from the paper