SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

Researchers have developed a way to train AI models for complex tasks like deep research and professional writing. This method avoids relying on human-provided answers. One AI creates difficult tasks. Another attempts to solve them using web searches. The models improve through a continuous cycle of challenge and mastery. This approach moves past the limits of supervised learning. It bypasss the need for human-curated data.

The Problem

Current reinforcement learning (RL) methods hit a wall. They struggle to move from verifiable domains—like math or code execution—to open-ended tasks. In math, a solution is either right or wrong. You can automate the reward signal via a compiler. But for a task like "summarize this economic policy," there is no single ground-truth string.

Practitioners usually fall into two traps. They use human-curated prompts. This caps model performance at human levels. Alternatively, they use "frontier-model judges." These are expensive API calls to models like GPT-4 to grade outputs. This creates a heavy dependency on external supervision. Existing self-play frameworks also struggle. They require rule-checkable answers or curated prompts to start. The goal is "data-free" self-play. Here, the model generates its own curriculum and rewards.

How It Works

The SCOPE framework uses a co-evolving dance between three roles. All are initialized from the same base model. These roles are a Challenger, a Solver, and a fixed Judge.

1. The Challenger ($C_t$): This policy generates document-grounded tasks. It looks at a source document. It performs multi-turn retrieval (searching for connected facts) to find information. It then constructs a task that requires that specific info. The Challenger is rewarded for producing tasks at the Solver's "capability frontier." These are tasks that are neither too easy nor impossible.
2. The Solver ($S_t$): This is the primary agent being optimized. It receives the Challenger's task. It must answer using multi-turn retrieval-augmented generation (RAG; generating answers based on retrieved data). It learns to navigate documents and maximize its reward.
3. The Judge ($J$): SCOPE uses a frozen copy of the base model as a self-judge. The Judge reads the source document and the task. It writes natural-language "rubrics" (specific evaluation criteria). These include rules like "identify the specific date of the treaty." The Judge then grades the Solver's response against these rubrics.

To maintain the learning signal, the authors use a "difficulty reward." This peaks when the Solver's success rate is around 0.5. This prevents the Challenger from settling for trivial tasks. They also use a cosine length penalty. This stops the Solver from "reward hacking." This is a failure where the model produces endless, repetitive text to trick the Judge.

Numbers

The authors report significant gains across three model families (Qwen2.5, Qwen3, and OLMo-3). On open-ended benchmarks, SCOPE improves performance by up to +10.4 points on Qwen2.5-7B. This represents a jump from a score of 24.4 to 34.8. Notably, SCOPE matches or exceeds GRPO_data. That baseline used ~9K curated prompts.

Results show monotonic improvement across iterations .

The gains also generalize. The model was trained only on open-ended tasks. Yet, it showed a jump in short-form QA performance of up to +13.8 points. This suggests the training improves underlying retrieval and reasoning.

Ablation studies prove co-evolution is necessary. If the Challenger is kept frozen, the Solver outgrows the tasks. This causes the average rubric score to rise toward triviality .

Also, the bottleneck is rubric generation, not grading. Scaling the grading model from 4B to 32B parameters added only +0.5 points. However, a weak 4B rubric generator dropped performance by 2.8 points .

What's Missing

There are several gaps for a production engineer to note.

First, the computational cost is higher than standard GRPO. The multi-stage pipeline adds significant overhead. It includes task generation, rubric generation, and difficulty filtering. The authors limited their experiments to 7–8B parameter models. It is unclear if these gains hold for 32B or 70B models.

Second, there is a lack of explicit safety filtering. The Challenger generates "challenging" scenarios. This carries a risk of generating biased or inappropriate content from the source corpus. The quality gates filter for relevance and entity clarity. They do not explicitly screen for harmful content.

Finally, the paper does not address how to ensure the model stays aligned with human preferences. Self-improvement is powerful. However, without a tether to human values, it could reinforce idiosyncratic behaviors.

Should You Prototype This

Depends on your data availability.

If you have a massive, unlabeled corpus, SCOPE is a strong candidate. Use it for specialized domains where human labeling is impossible. This includes technical research or niche legal analysis. It replaces expensive frontier-model APIs with a compute-intensive loop.

If you have a limited GPU budget, the complexity might be too high. Managing three roles (Challenger, Solver, Judge) is difficult. However, the framework is mathematically sound. The ablations on reward hacking show the authors addressed practical failure modes. Code is reportedly available; see the paper for the canonical link.

Figures from the paper