Do LLMs Ask the Right Questions? Evaluating GPT-Generated Surveys as Instruments for Measuring Social Attitudes

Can an algorithm capture the nuance of human conviction? Researchers compared surveys written by AI (GPT-4) against expert-designed surveys on topics like climate change and immigration. They found that while AI surveys capture the main divisions in public opinion, they sometimes lack the fine-grained detail of human-designed ones.

The bottleneck of manual instrument design

Understanding human beliefs relies on carefully designed survey instruments. These are not just random collections of questions. They are theory-informed tools. They encode assumptions about which concepts matter and how they should be measured. Traditionally, this process is slow and resource-intensive. Experts must ground every question in existing literature to ensure "construct coverage" (the degree to which a survey hits all intended thematic targets).

If a survey fails to cover the right dimensions, it creates measurement error. For example, a survey on climate change might ask about personal worry. It might forget to ask about political policy support. This leaves a massive hole in the data. As LLMs become capable of generating fluent text, the temptation to automate this design grows. However, the central question remains. Can an LLM replicate the structural integrity of a human-designed instrument? Or will it produce "noisy" data that misses the subtle textures of human belief?

A structured 3x3 prompting framework

To test this, the authors developed a controlled framework. They moved beyond simple question generation. Instead, they enforced a rigid 3×3 construct space. This architecture ensures every generated survey covers three substantive dimensions: beliefs, perceptions, and behaviors. This totals nine questions. Think of this like a modular building kit. By forcing the model to fill specific slots, the researchers prevented the AI from drifting into irrelevant territory.

The methodology followed four distinct stages. First, the authors used a fixed prompting template with GPT-4. They set the temperature (a parameter controlling randomness) to 0.7. The model acted as an "expert survey methodologist." The prompt demanded neutral, non-leading wording. It also required a uniform 5-point Likert scale (a standard rating scale used to measure attitudes). Second, they constructed human baseline surveys. They assembled items from validated sources like the American National Election Studies (ANES). This matched the length and scope of the AI versions.

Third, the researchers recruited U.S.-based participants. These participants completed both versions of the surveys under randomized ordering. This prevents fatigue or sequence bias. Finally, they applied unsupervised machine learning. Specifically, they used k-means clustering (an algorithm that groups data points into $k$ clusters) to see if responses naturally grouped into meaningful "personas" or ideological blocs.

Measuring the depth of belief

The authors evaluated the quality of these surveys using several mathematical lenses. They looked at "clustering quality" to see if surveys could sort people into distinct groups. They also checked "predictive validity" (how well survey responses forecast downstream behaviors).

The results reveal a striking similarity in high-level structure. In the climate change domain, both the human and GPT surveys captured the same dominant ideological divisions. As seen in, mean responses for both survey types shifted predictably along the persona spectrum.

These shifts went from skeptics to advocates. Furthermore, the authors report that both instruments favored a two-cluster solution ($k=2$). This represents a coarse-grained split in public opinion .

Crucially, the two surveys showed high agreement in how they categorized people. The authors report a transition matrix showing that 86% of respondents in the human baseline and 98% in the GPT survey remained in the same cluster when moving between instruments. However, a tradeoff emerges in resolution. While the GPT surveys are remarkably stable—the authors report a mean ARI (Adjusted Rand Index, a measure of agreement between two clusterings) of 0.93 for climate change—they occasionally exhibit "attenuated separation." This means the AI sometimes merges adjacent personas. It fails to catch the subtle nuances that separate a "moderate" from a "proactive" supporter.

Limits of automated measurement

While the results are promising, the paper highlights several critical boundaries. The study is limited to three social domains: climate change, immigration, and DEI. The authors note that performance may vary for more technical or culturally specific topics. This is a significant caveat for practitioners. An LLM that understands "climate change" might struggle with localized nuances.

Furthermore, the predictive power of the surveys is inconsistent. In the climate change domain, the human baseline outperformed the GPT survey in predicting action likelihood ($R^2 = 0.391$ vs. $0.158$). Note that $R^2$ (a metric measuring how well a model explains variance) indicates the human survey was much better at forecasting actual behavior. This suggests that while the AI is excellent at identifying "who people are" (classification), it is less reliable at predicting "what people will do" (prediction) at the item level. The authors warn that current evidence does not support using LLMs as full replacements for expert-designed instruments. Human oversight is needed to avoid oversimplifying complex social realities.

The verdict: A scaffold, not a substitute

Is the era of the human survey designer over? Not yet. Based on the evidence, the verdict is that LLM-generated surveys are powerful scaffolds rather than autonomous replacements.

They are highly effective for exploratory research. They work well for rapid prototyping and large-scale pilot studies. They are ideal when catching the "broad strokes" of public opinion is the priority. Their ability to rapidly generate structured content is a massive win for throughput. However, for high-stakes social science research, the human expert remains indispensable. This is especially true when mapping the fine-grained edges of human disagreement. Use the LLM to build the foundation. But keep a human in the loop to refine the architecture.

Figures from the paper