Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

DOMINO: Inductive Data Synthesis via Minimal Sufficient Representation Learning

In the current landscape of Large Language Model (LLM) adaptation, we spend a disproportionate amount of time playing "prompt engineer." We try to force models into specific domains. We treat the model as a deductive engine. We feed it a complex, natural language description of a task. We hope it generates data that follows that rule. But this breaks down when the domain is implicit. Examples include a specific coding style or niche scientific nomenclature. In these cases, the "rules" are easier to show than to explain.

Most existing synthesis methods rely on this deductive paradigm. They require heavy reliance on explicit domain descriptions. This paper proposes an alternative: inductive synthesis. Instead of describing the domain, you provide a handful of reference examples. The model learns the "essence" of the pattern. It then uses this essence to generate many new, diverse examples. These examples follow the same rules without simply copying the original ones.

The Problem

The status quo in data synthesis assumes a human can accurately articulate domain boundaries in text. This is rarely the case in production. If you are adapting a model to a proprietary codebase, writing a prompt is a losing game. You cannot easily verbalize every subtle structural nuance.

When we lack a good prompt, the fallback is often Supervised Fine-Tuning (SFT). SFT is the process of training a model on specific input-output pairs. However, SFT on a small reference set is a trap. It causes the model to memorize superficial, sample-specific noise. This includes variable names or idiosyncratic constraints. This leads to poor generalization on out-of-distribution tasks. The authors demonstrate this failure in Table 1. Naive SFT can actually degrade performance in certain domains. For example, in Live Code Execution, it fails because the model overfits to the narrow distribution of the reference set.

How It Works

The DOMINO framework shifts the burden from prompt engineering to representation learning. The goal is to find a "minimal sufficient representation." This is a latent embedding (a compressed vector representation) that captures essential domain traits. It must discard the noise unique to individual samples.

The architecture operates in two distinct phases:

1. Representation Learning (Training): The authors use prompt tuning to learn a continuous domain embedding $\mathbf{D}$. This consists of $k$ "soft tokens" (learnable continuous vectors that act as a prefix to the LLM). To prevent memorization, they introduce a contrastive disentanglement objective ($L_2$). They simultaneously learn sample-specific soft tokens $\mathbf{S}^{(i)}$ for each reference example. The loss function forces $\mathbf{D}$ to capture shared patterns. Meanwhile, it penalizes $\mathbf{D}$ if it helps reconstruct a different sample $\mathbf{X}^{(j)}$ using $\mathbf{S}^{(i)}$. This pushes the "identity" of the specific sample into $\mathbf{S}$. It leaves the "essence" of the domain in $\mathbf{D}$.

2. Data Synthesis (Inference): Once $\mathbf{D}^$ (the optimized domain representation) is learned, the sample-specific tokens $\mathbf{S}$ are discarded. The LLM is then prompted using only the domain soft tokens $\mathbf{D}^$. Because $\mathbf{D}^*$ is "minimal," it cannot trigger the model's tendency to copy specific details. Instead, it guides the LLM to sample from the broader domain distribution.

This process is illustrated in . It contrasts the joint optimization of tokens during training against the use of only domain tokens during synthesis.

Numbers

The authors report significant gains in domains where defining a prompt is difficult. On the LiveCodeBench benchmark, DOMINO improved Pass@1 accuracy by up to 4.63% over strong, instruction-tuned backbones. Pass@1 measures the probability that at least one generated solution is correct.

For the Live Code Generation domain using an OPENCODER-8B-Instruct backbone, the authors report a Pass@1 of 12.63%. This outperforms the baseline instruction-tuned model. In the Live Code Execution domain, the improvement is even more pronounced. It reaches a Pass@1 of 42.59%. This means the model is significantly more likely to produce executable, correct code.

The efficiency of the representation is also explored via hyperparameters. Increasing the number of domain soft tokens from 10 to 20 improves Pass@1 performance, as seen in .

They also found that performance depends on the amount of reference data. Performance declines as the percentage of reference samples drops from 100% to 20%, as shown in .

What's Missing

While the results are compelling, there are gaps for a practitioner:

• Representative Bias: The framework assumes reference samples are a faithful subset of the target domain. If your initial samples are skewed, DOMINO will learn and amplify that bias. The paper acknowledges this limitation. However, it does not quantify how much bias is tolerable.
• Computational Overhead of Training: The synthesis phase is just a forward pass. However, the training phase requires optimizing soft tokens against the LLM. The authors used 8 NVIDIA A100 80GB GPUs for fine-tuning. The exact resource requirements for the representation learning stage are not explicitly detailed.
• Scaling the Number of Tokens: shows improvements as the soft token count increases. However, the data stops at 20 tokens. It is unclear if there is a point of diminishing returns. We do not know if extremely large $k$ values re-introduce the overfitting problem.

Should You Prototype This

Yes, if you are dealing with "dark" domains. Use this if you have a folder of 100 high-quality logs or code snippets. Use it if you cannot write a formal specification for them. Moving from "here are some examples" to "here is a massive synthetic dataset" is a major win for scalability.

Code is reportedly available; see the paper for the canonical link at https://github.com/tongye98/DOMINO. If you have a small set of gold-standard examples and a decent GPU, this is worth a prototype. Test if the induction holds for your specific data distribution.