QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

QUBRIC: Co-Designing Queries and Rubrics to Unlock Reasoning in RL

When teaching AI using rules (rubrics), if the questions are too vague, the rules become useless. QUBRIC fixes this by rewriting vague questions into specific scenarios and then creating precise rules based on those scenarios. This helps the AI learn much better.

In current reinforcement learning (RL), we have mastered "verifiable rewards." These are tasks like math or coding where a compiler or a ground-truth answer provides a clean signal. Moving RL into open-ended domains is harder. There is no single "correct" answer to check against. To solve this, researchers use rubric-based RL. An LLM judge evaluates responses against structured criteria. The challenge is scaling this to open-ended tasks without the reward signal becoming noise.

The Problem

The status quo assumes training queries (the prompts given to the model) are a fixed distribution. Most researchers focus on optimizing the rubrics. However, this ignores a structural bottleneck. Rubric quality is constrained by query structure.

If a query is too broad—for example, "explain machine learning"—the space of valid responses is too large. Any rubric written for such a query becomes too vague to be useful. Even worse, the authors identify a "fabricated-reference" failure mode. When developers try to "narrow" a query without proper grounding, the rewriter often invents concrete but non-existent references. It might cite a specific nonexistent guideline or glossary. As shown in, this causes the rubric generator to fail.

The rubric ends up testing whether the model refuses to answer the fake reference. It does not test reasoning. Consequently, the model receives uninformative reward signals, and training stalls.

How It Works

QUBRIC breaks this cycle by co-designing the query and the rubric through a three-stage offline pipeline . It treats the prompt as a variable that must be engineered to support evaluability (the ability to be graded easily).

1. Key-Point-Grounded Query Rewriting: To avoid the fabricated-reference trap, the system first extracts "key points" from several high-quality teacher responses. These are atomic pieces of knowledge. It then rewrites the original query into a scenario-based question. This rewrite embeds realistic context, like a professional role. The selected key points emerge as the necessary answer through reasoning, rather than simple recall.
2. Contrastive Rubric Generation: Once the query is grounded, the system generates rubrics. It compares a strong teacher response against a weaker policy response. This "contrastive" approach identifies the exact gap between the two. It turns this delta into "constitutive" criteria. A constitutive rubric contains the specific facts needed for grading within the text. This means the judge does not need external knowledge to decide if a point was made.
3. Learnability Filtering and GRPO: Finally, the system applies a filter. It retains only query-rubric pairs where the initial policy's pass rate falls within a 20–50% difficulty corridor. This ensures the model trains on the most informative examples. These rewards drive optimization via Group Relative Policy Optimization (GRPO). GRPO uses group-relative advantages to stabilize training.

Numbers

The authors report gains that suggest this co-design approach improves actual reasoning. On the ArenaHard benchmark, QUBRIC achieves a +5.5 point gain over the SFT (Supervised Fine-Tuning) baseline. Specifically, it shows a +7.3 point jump in creative-writing scores.

The cross-domain transfer is also notable. Despite being trained only on instruction-following data, the model achieved an average gain of +6.3 points on three held-out benchmarks. These cover legal, moral, and narrative reasoning. The paper highlights a +8.85pp (percentage point) improvement on PLawBench. This is a Chinese legal reasoning benchmark. These gains appear in dimensions requiring structured argumentation.

The authors also examine training stability in .

Original queries and naive rewrites suffer from reward overfitting. This is when the gap between training and validation reward widens significantly. The co-designed QUBRIC approach maintains a much healthier generalization gap of approximately 0.07. This suggests the model learns underlying reasoning logic rather than just memorizing patterns or padding responses with verbosity.

What's Missing

Several areas leave an engineer wanting more:

• Verification of the Validator: The query rewriting stage uses a prompt-based validation step. This checks if the new scenario is consistent and does not leak the answer. The authors admit this is not human-verified. A failure here could potentially poison the training set with low-quality queries.
• The Cost of Scale: The pipeline requires multiple heavy steps. It extracts key points from four teacher models and conducts contrastive analysis for rubrics. It also runs an LLM-as-a-judge. While these steps happen offline, the compute overhead for data synthesis is not fully quantified in the paper.
• Side-Effect Management: In the shopping assistant application, query-level rubrics improved helpfulness. However, they also increased errors like "ID hallucination" (inventing product IDs) and "duplicate recommendations." The authors show that adding global rubrics mitigates this. Still, balancing query-specific optimization with global regularization is a significant operational burden.

Should You Prototype This

Yes, but with caution.

If you want to move RL beyond math and code into complex reasoning, the "fixed query" assumption will likely fail. QUBRIC provides a blueprint for building a curriculum that provides a real signal. However, do not treat this as a "plug-and-play" library. Success depends on the quality of your "teacher" models. It also depends on the precision of your rubric generation prompts.

Start by prototyping the query-rewriting module. Try to turn your existing instruction set into grounded, scenario-based queries. If you can do this without introducing hallucinations, you have solved half the battle. If your reward models are plateauing or your models are "hedging" (using excessive, meaningless verbosity), this co-design approach is likely the cure.