Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

Instead of just asking an AI to give a score, this method gives the AI a "skill kit" containing rules, checklists, and tools. The AI then follows a step-by-step procedure to check evidence before making a final decision. This makes its grading much more accurate and reliable.

In LLM post-training, reward models (RMs) provide critical feedback signals. They steer models via Reinforcement Learning (RL) or Reinforced Fine-Tuning (RFT). Traditionally, we rely on two patterns. Scalar RMs output a single, opaque number. "LLM-as-a-Judge" systems use massive, flat prompts to force a model to reason. Both have a ceiling. Scalar models hide their reasoning. This makes them impossible to debug. Judge models suffer from "context stuffing." They try to cram rubrics and constraints into one prompt. This creates noise and causes the model to lose attention.

The Problem

Current reward modeling struggles to integrate heterogeneous evaluation criteria. Coding tasks require a Python sandbox. Math tasks need a ground-truth verifier. Style enforcement requires a checklist.

Existing approaches concatenate all these resources into one giant prompt. This is brittle. As shown in, the judge must passively absorb all knowledge at once. This leads to high contextual noise. When criteria are complex, the judge often loses the signal. Examples include verifying multi-step tool trajectories or strict JSON formatting. We are essentially asking a single model to hold a disorganized library. We then hope it picks the right book for every question.

How It Works

The authors propose Skill-RM. It shifts the paradigm from passive absorption to active orchestration. It treats reward modeling as the execution of a reusable "Reward-Evaluation Skill." Instead of a monolithic prompt, the system uses a structured artifact. This consists of a procedural specification ($M_{RM}$) and a resource bank ($U_{RM}$).

The workflow operates in three distinct stages:

1. Skill Loading: The agentic judge loads a specific $SKILL.md$ file. This file acts as a blueprint. It defines the scope of the criteria and the protocol for using resources.
2. Agentic Orchestration: The judge performs an action-observation trace (a sequence of steps and responses). It identifies the task and selects relevant resources. It invokes them on demand. This "progressive disclosure" means the model only sees necessary evidence. This significantly reduces noise.
3. Evidence-Grounded Decision: The judge does not just spit out a score. It must generate criterion-level evidence ($e_m$) for each requirement. It documents the specific observation ($q_m$) that justifies its assessment ($s_m$). Finally, a deterministic readout function $A(\tau)$ maps this trace to the required output. This could be a scalar reward or a ranked selection.

Numbers

The authors demonstrate that this orchestration pays off. Using a Qwen3.5-27B backbone, Skill-RM achieves an average score of 86.2. This covers RewardBench2, RM-Bench, and JudgeBench. This outperforms the matched LLM-as-a-Judge baseline of 83.9.

The most significant delta appears in resource organization. In a controlled ablation study, the authors tested the Qwen3.5-27B backbone. They found that simply appending resources to a prompt decreases performance. The average dropped from 83.9 to 81.0. In contrast, using the Skill-RM framework with those same resources raised the average to 86.2. This is a +5.2 point gain over the baseline.

The paper also shows strong results in Best-of-$N$ response selection. On the GSM8K math task, Skill-RM reaches a selection accuracy of 97.8. This is nearly the Oracle upper bound of 97.9. As seen in, gains are pronounced in instruction-following (IFEval) and coding (HumanEval+) tasks.

These areas benefit most from structured verification.

What's Missing

While the results are compelling, some gaps remain for production engineers:

• Inference Overhead: Moving from a single-pass forward pass to an agentic trace increases latency. Every "view_resource" or "python_sandbox" call is an extra step. The paper acknowledges this. However, it does not provide a detailed breakdown of the latency penalty.
• Manual Curation Bottleneck: The framework relies on manually curated skills. There is no mention of an automated pipeline for generating these $SKILL.md$ files. Scaling to thousands of niche tasks would require automated construction.
• Complexity in Deployment: Implementation is not as simple as changing a prompt. You need a tool-calling infrastructure and a sandbox environment. You also need to manage the versioning of skill artifacts.

Should You Prototype This

Yes, but with caveats. If you run high-stakes RL, the quality of the reward signal is vital. The jump from 81.0 to 86.2 in benchmark accuracy is significant. Providing evidence-grounded traces also makes debugging training runs easier.

However, consider your latency requirements. If you need to score thousands of candidates per second, the agentic overhead might be a problem. Start by prototyping the "Skill" structure for your hardest evaluation dimensions. The code is available at https://github.com/Qwen-Applications/Skill-RM.

Figures from the paper