Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating

When researchers use Large Language Models (LLMs) to simulate human survey responses—a technique called "silicon sampling"—they encounter a frustrating phenomenon: the AI becomes too predictable. While these models can mimic the average opinion of a demographic, they struggle to replicate the diversity of thought found in real populations. Instead of a wide spectrum of opinions, the models tend to cluster heavily around a central value. This produces results with unrealistically low variance (a measure of how spread out the data points are).

This "mode collapse" makes synthetic data dangerous for serious research. If a model lacks variance, it creates an illusion of consensus. This leads to overconfident estimates and narrow confidence intervals that do not exist in the real world. A new study by Oscar Heath and Rohan Alexander proposes that this isn't because the models don't understand human opinions. Rather, it is because they are fundamentally bad at generating numbers. By switching from numeric prompting to a text-based semantic mapping approach, the authors report they can restore the missing diversity to silicon samples.

The failure of numeric prompting

Current silicon sampling workflows typically ask an LLM to provide a direct numeric score. For example, they might ask for a rating from 0 to 100 on a "feeling thermometer." As shown in, this direct approach often results in highly peaked distributions.

Even when the synthetic mean (the average value) is relatively accurate, the spread of the data is far too narrow compared to real-world datasets like the American National Election Studies (ANES).

The authors argue that this mode collapse stems from a mismatch between the task and the model's training. LLMs are probabilistic engines designed to predict the next text token (a unit of text). They are not calculators or random number generators. When forced to output a discrete integer, they often default to the most statistically "safe" or likely value. This prevents them from sampling the tails of the distribution. This lack of variance is a persistent issue. Newer, larger models have not yet solved it, as evidenced by the continued mode collapse in the raw numeric outputs of the models tested in .

Mapping meaning through Semantic Similarity Rating

To bypass the model's struggle with arithmetic, the authors employ Semantic Similarity Rating (SSR). Instead of asking for a number, the SSR pipeline follows a multi-stage process to convert qualitative language into quantitative data :

1. Textual Elicitation: The LLM is prompted to provide a descriptive, natural language response. It describes its feelings toward a target group instead of providing a digit.
2. Anchor Comparison: The system uses five predefined "anchor points" on the 0–100 scale. Each anchor is paired with a descriptive text string (e.g., "I have a very unfavorable opinion...").
3. Embedding Projection: The text response and the anchor points are converted into high-dimensional vectors using text embeddings (mathematical representations of semantic meaning).
4. Similarity Calculation: The system calculates the cosine similarity—a metric measuring how closely two vectors point in the same direction—between the response and each anchor.
5. Distribution Generation: These similarities are normalized and passed through a temperature-scaled softmax function. This creates a probability distribution, which is finally smoothed into a continuous scale using kernel density estimation (a method to estimate the probability density function of a random variable).

Crucially, the authors introduce a single global temperature parameter, $\mathcal{T}$, to control the variance of this process. This parameter acts like a volume knob for randomness. Adjusting it allows researchers to calibrate how "spread out" the resulting synthetic distribution should be.

Restoring the distribution shape

The efficacy of the SSR method is measured using Kullback-Leibler (KL) divergence. This is a statistical metric that quantifies how much one probability distribution differs from a reference distribution. Lower values indicate a better match. The authors report that moving from numeric prompting to SSR results in a substantial drop in KL divergence across all tested models .

For the Claude-sonnet-5 model, the KL divergence dropped to 0.07 when using SSR on 2016 data. This represents a much closer approximation of the real human distribution.

While the primary goal was to fix the variance, the authors also monitored the "absolute error of the mean." This ensures that fixing the shape doesn't ruin the accuracy of the average response. The results suggest a successful tradeoff. SSR significantly improves the fit of the distribution's shape. Simultaneously, it does so without causing a significant loss in the accuracy of the synthetic means . Furthermore, demonstrates that SSR successfully aligns the standard deviation (a measure of dispersion) of synthetic responses with that of real human responses.

This effectively addresses the mode collapse problem.

Persistent biases in the silicon mind

Despite the success in restoring variance, the paper identifies a critical limitation. SSR does not solve systematic ideological bias. The authors observe that while the shape of the distribution is now realistic, the position of the distribution can still be skewed. For example, the models consistently produce more extreme responses than humans. Synthetic Democrats, for instance, rate Conservatives significantly lower than real Democrats do.

This suggests that the bias toward extreme opinions is not a formatting error. It cannot be fixed simply by changing how we ask for numbers. Instead, it appears to be a fundamental characteristic of how LLMs represent political and psychological concepts. For practitioners, this means that SSR makes silicon sampling more statistically reliable for analyzing variance and uncertainty. However, it does not automatically make the models "truthful" representatives of human nuance.

The verdict

If you are using LLMs to generate synthetic survey data, the verdict is clear: stop asking for numbers. The authors demonstrate that treating LLMs as text generators provides a much more faithful representation of human population variance. Using an embedding-based mapping like SSR captures the necessary diversity. Because the method relies on a single global temperature parameter that generalizes well—as shown by the successful application to 2020 data—it is a low-overhead upgrade for existing pipelines.

Code and data for this research are reportedly available at the Zenodo repository linked in the paper. While SSR fixes the "how much" (variance), researchers must remain cautious about the "what" (inherent model bias). The tendency toward political extremism remains baked into the models themselves.

Figures from the paper