Skills-Coach: Automating the Self-Improvement of AI "Skills"
Large language model agents increasingly depend on modular components called skills to perform specialized tasks such as fetching weather data, editing spreadsheets, or managing cloud services. However, most publicly available skills are created by individual developers to solve narrowly defined problems, resulting in uneven coverage and frequent brittleness when confronted with realistic edge cases. A new framework called Skills-Coach addresses this gap by introducing an automated, closed-loop (a system that continuously collects feedback and uses it to refine itself) system that probes a skill’s limits and rewrites its instructions and code without requiring human intervention or expensive model retraining.
The Problem Space
The authors identify a persistent bottleneck in the current skill ecosystem: while thousands of skills exist across platforms such as ClawHub and Anthropic, their development remains ad hoc and disconnected from systematic stress testing. The central research question therefore becomes: Can an LLM-based agent autonomously discover where a given skill fails, then generate improved versions of that skill through iterative refinement? To answer this, the paper decomposes the challenge into three interconnected sub-questions:
1) How can boundary-probing tasks be generated automatically? Effective skill expansion begins with creating test cases that neither trivialize nor hallucinate difficulty. Boundary-probing tasks (tests deliberately designed to push a skill to its operational limits) must reflect real constraints and expose genuine failure modes.
2) How can skills actually improve themselves? Beyond diagnosis, the agent must translate failure signatures into concrete edits—refining instructions, patching code, adding error handling—while preserving compatibility with the broader skill ecosystem.
3) How can improvements be measured reliably? Any optimization loop requires objective, multi-dimensional feedback to avoid reinforcing local optima or cherry-picked successes.
These questions motivate the design of Skills-Coach as a self-contained pipeline rather than a one-off tuning procedure.
The Skills-Coach Pipeline
Skills-Coach implements a four-module architecture that operates in a closed loop around a target skill:
1) Diverse Task Generation Module: Parses the skill’s metadata (README, Skill.md, configuration files) to extract functional scope, supported commands, I/O formats, constraints, and usage examples. Using this specification, the module synthesizes three tiers of tasks—standard, advanced, and boundary—that collectively span routine operations, complex workflows, and pathological edge cases. The generated tasks are paired with deterministic validation criteria (file existence, keyword presence, regex matches) to ensure objectivity. All samples are partitioned into training and test sets to preserve strict separation and guard against memorization.
2) Lightweight Optimization Module: Serves as the core engine for iterative improvement. Instead of fine-tuning the underlying LLM, the module leverages Training-Free Group Relative Policy Optimization (GRPO), a method that uses the LLM’s own introspective judgments to rank candidate revisions. Each optimization epoch produces multiple instruction and code variants, compares their performance on the training tasks, and selects the top performer as the new baseline. The optimization pathway differs slightly depending on skill type: instruction-only skills receive textual refinements focused on clarity, completeness, and example richness, whereas code-inclusive skills undergo deeper surgical edits—adding caching layers, input validation, error handling, and dependency checks—before command-level refinements and auto-fixing of failures.
3) Comparative Execution Module: Runs both the original and optimized skill versions in isolated, reproducible environments. The module enforces strict execution hygiene: preflight dependency validation, sandboxed temporary workspaces, sequential execution to remove ordering effects, immediate cleanup after each task, and fail-safe logging that preserves partial outputs on crashes. Parallel execution via thread pools maximizes throughput while maintaining isolation guarantees.
4) Traceable Evaluation Module: Quantifies differences between versions across multiple axes—correctness, efficiency, robustness, safety—using both LLM-based scoring and fallback heuristics (keyword matching, structural checks, content statistics). The evaluator applies uniform criteria to both skill versions, aggregates task-level scores into macro metrics (average score, pass rate, standard/advanced task scores), and emits structured reports that justify retention or rejection decisions with transparent evidence.
Two execution modes are supported: virtual mode, which simulates success via keyword presence and deterministic hashes, and real mode, which interacts with live systems and inspects actual outputs, logs, and artifacts. Virtual mode trades realism for speed during iterative refinement, while real mode delivers definitive assessments during final evaluation.
Experimental Validation
To gauge Skills-Coach’s effectiveness, the authors introduce Skill-X, a benchmark composed of 48 diverse skills harvested from Anthropic, ClawHub, and Vercel Labs. The corpus spans two categories: 29 instruction-only skills (documented procedures without executable code) and 19 code-inclusive skills (packages containing scripts or binaries). Skill-X emphasizes real-world applicability by including tasks that demand multi-step orchestration, anomaly detection, and integration with external APIs.
On Skill-X, Skills-Coach delivered substantial gains across all aggregate metrics. The average score rose from 0.37 to 0.84—a jump equivalent to lifting a failing grade to a solid B+. The pass rate climbed from 33.6 percent to 88.0 percent—more than doubling the share of tasks completed successfully. Stratified results confirm broad impact: instruction-only skills improved from 37.9 percent to 91.4 percent pass rate, and code-inclusive skills from 27.0 percent to 82.9 percent. Notably, advanced tasks—designed to probe operational boundaries—benefited disproportionately, jumping from 32.7 percent to 81.6 percent pass rate versus 43.0 percent to 87.4 percent for standard tasks, indicating that the framework uncovers and repairs subtle fragilities rather than merely polishing already-working paths.
Further inspection of individual skills reveals a bimodal pattern. Twenty-three skills exhibited exceptional improvements (≥+0.5 score gain), including four that reached ceiling performance (score = 1.0, pass rate = 100 percent). Conversely, five skills were already optimal and remained unchanged, highlighting diminishing returns once intrinsic quality is high. This suggests a strategic insight: concentrating optimization effort on underperforming skills yields the highest return on investment.
Case Study: Pollyreach
The paper illustrates the pipeline through a detailed case study of Pollyreach, a skill intended to retrieve and summarize news feeds. The Diverse Task Generation Module produced a hierarchy of tests: standard tasks checked basic feed retrieval and formatting, while advanced tasks introduced same-day delivery semantics, JSON schema validation, and parameter-edge cases. During optimization, the Lightweight GRPO loop refined both the instruction narrative and the underlying Python helper, emphasizing clearer error messages and stricter input guards. The Comparative Execution Module then ran original and optimized versions head-to-head, revealing that the optimized skill eliminated silent failures and consistently returned properly formatted artifacts. The Traceable Evaluation Module quantified these gains—raising the advanced task score from 37.5 percent to 100 percent—and produced a structured report that guided the final retention decision.
Limitations and Next Steps
Despite strong headline results, the authors acknowledge important caveats. Skills that already exhibit near-perfect behavior derive little incremental benefit, so optimization budgets should be allocated strategically. The reliance on LLM-based scoring introduces potential biases and latency compared to purely deterministic checks, though the system maintains a heuristic fallback mode when API access is constrained. Virtual mode execution, while computationally cheap, approximates success via proxies rather than direct observation, which may miss subtle runtime pathologies detectable only in real mode. Finally, the Skill-X corpus contains only 48 skills, so broader ecological validity awaits further expansion.
Future directions include scaling the pipeline to larger skill networks, integrating multi-agent collaboration patterns, and tightening the coupling between task generation and real-world telemetry. The release of code and benchmark data invites external researchers to replicate and extend these results.
Explore Further
To learn more about Skills-Coach, visit the project repository and benchmark:
- Code: Available at https://github.com/T1aNS1R/Skills-Coach
- Data: Access the Skill-X benchmark through the project page on ClawHub (https://clawhub.ai/t1ans1r/skills-coach)
The repository provides the implementation and dataset described in the paper. Consult the README and accompanying materials for setup instructions, configuration options, and usage guidelines.