Autodata: An agentic data scientist to create high quality synthetic data

Can an AI Act Like a Data Scientist?

As we push toward the frontier of large language model (LLM) capabilities, we are hitting a wall with human-written training data. To bridge the gap, we increasingly turn to synthetic data. These are data points generated by models themselves to fill reasoning gaps or cover edge cases. But simply asking an AI to "write a math problem" often results in triviality or hallucinations.

The fundamental challenge is one of control. How do we ensure synthetic data is neither too easy nor too hard? It must be "just right" to provide a useful learning signal (the gradient used to update model weights). The researchers at FAIR have proposed a shift in perspective. Instead of treating data generation as a single-shot prompting task, they propose an agentic framework called Autodata. The goal is to enable AI agents to emulate the iterative workflow of a human data scientist to build high-quality training and evaluation sets.

The Search for Discriminative Data

The core question the authors investigate is whether an autonomous agent can systematically optimize the difficulty and quality of synthetic data. Specifically, they ask if an agent can learn to produce examples that are sufficiently challenging to trip up a "weak" model but remain solvable by a "strong" model.

In machine learning, we rely on "discriminative" data. These are examples that clearly separate different levels of capability. If a training example is so easy that even a tiny 4B parameter model gets it right every time, that data point contains almost no signal for Reinforcement Learning (RL). Conversely, if the example is so broken that even a massive 397B parameter model fails, it provides a noisy, unhelpful gradient. The authors aim to find this "Goldilocks zone" of data difficulty through an automated, iterative loop.

Cracks in the Self-Instruct Paradigm

Until now, the industry standard for synthetic data has largely relied on methods like Self-Instruct or CoT (Chain-of-Thought) Self-Instruct. These methods use LLMs to bootstrap new instructions from a small seed set. They often use zero-shot or few-shot prompting.

The authors argue that these "classical" methods lack direct control over data quality and difficulty. Standard CoT Self-Instruct tends to produce questions that are far too easy for the target model. In the computer science (CS) research task setup, the authors note a significant issue. The "CoT Self-Instruct" baseline resulted in a weak solver average of 0.677. It also showed a negligible strong-weak gap of only 0.019 [Table 1]. This gap represents the difference in performance between the two models. Because the gap was so small, the data was essentially useless for pushing the boundaries of the weaker model's reasoning.

The Agentic Self-Instruct Loop

To solve this, the authors implement "Agentic Self-Instruct." This is a multi-agent architecture designed to hunt for that discriminative gap. As illustrated in, the system employs a main orchestrator agent.

This agent directs four specialized subagents: a Challenger to generate the examples, a "Weak" solver, a "Strong" solver, and a Verifier/Judge to evaluate the results.

The investigation follows a rigorous cyclical process. The Challenger proposes a question and a rubric (a set of scoring criteria). The Weak and Strong solvers attempt it. Finally, the Judge assesses the gap between them. If the gap isn't wide enough, the Judge sends feedback to the main agent. The agent then analyzes the failure. It might conclude the question was too generic. The agent then instructs the Challenger to try a different reasoning angle.

This isn't just a single loop. The authors also demonstrate "Meta-Optimization." They treat the agent's entire operational scaffold as code that can be evolved. By using an outer loop to analyze failure trajectories, they can automatically improve the agent's ability to generate high-quality data without manual engineering .

Turning Inference Compute into Training Signal

The results across three distinct domains—computer science, legal reasoning, and mathematical objects—suggest this approach works. In the CS task, the Agentic loop successfully drove the weak solver's score down from 0.677 to 0.458. At the same time, it actually improved the strong solver's score [Table 1]. This confirmed that the agent was moving away from generic summaries. Instead, it moved toward specific, high-reasoning algorithmic questions.

The downstream impact on model training is the most significant finding. When training a Qwen3.5-4B model using GRPO (a reinforcement learning algorithm), the Agentic data outperformed the CoT data across the board. On the harder "Agentic" test set, the performance lift was particularly striking. The Agentic-trained model reached a 0.632 mean@3, compared to just 0.500 for the CoT-trained model [Table 2]. Here, mean@3 refers to the average accuracy across three separate attempts. This shows a massive jump in reliability on difficult tasks.

Interestingly, the authors found that the agentic method doesn't just improve accuracy. It also improves reasoning efficiency. By training on these highly structured, challenging examples, the model's truncation rate dropped significantly. This is the frequency with which a model runs out of tokens before finishing a thought. In certain settings, the rate fell from 23.75% to 4.09% [Table 8].

Implications for the Data Frontier

The implication of this work is a fundamental shift in how we view the relationship between inference and training. Autodata provides a formal way to convert increased inference-time compute into higher-quality model training. This compute is spent by the agents generating the data. If this scales, the bottleneck for AI progress may no longer be the availability of human data. Instead, it may be the amount of compute we are willing to spend on "agentic" data scientists.

One immediate takeaway for practitioners is that "harder" is not always "better." In the legal reasoning experiments, the authors found that standard prompting often produced questions that were too hard. This resulted in zero-score rollouts. Such results kill the RL signal. The goal must be to reshape the reward distribution. It should be "just right" for the target model's current capability.

Researchers looking to prototype this should focus on the "gap" metric. It is not enough to measure if a model is right. One should measure the delta between the smallest and largest models on a synthetic task. If that delta does not widen as the loop iterates, the data generation loop is likely producing high-quality noise.

Figures from the paper