Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

Conditional Hypothesis Generation: Steering LLM Discovery with Researcher-Specified Covariates

When using AI to find patterns in text, the model often gets distracted by irrelevant background information. A new paper argues that current LLM-based methods for discovering linguistic patterns tend to select for globally discriminative features. These features may actually be mere confounds (variables that correlate with an outcome but are not the substantive driver).

The field of computational social science aims to understand how language varies across outcomes like political affiliation or instructional quality. The state of the art involves using Sparse Autoencoders (SAEs)—models that decompose text into interpretable, "monosemantic" features—and then using statistical models like LASSO to pick the most predictive ones. An LLM is then prompted to turn those features into natural language hypotheses. However, these methods typically look for global patterns. If you are looking for how Democrats and Republicans differ in speech, a global model might tell you they differ in mentions of "National Parks." This distinction might actually be driven by geography rather than ideology. This paper seeks to solve this by letting researchers specify "covariates" (metadata like time period or topic) to steer discovery toward differences that hold within specific subgroups.

The Problem

Global discrimination is often misleading. A feature might appear highly predictive of an outcome across an entire dataset while actually reflecting a confound. A classic example is the "national park" problem. Mentions of public lands might predict political party because land distribution is geographically skewed. This is not necessarily because the topic is inherently partisan.

Current approaches suffer from two specific statistical failures when researchers look closer. First is stratum imbalance. The specific subgroup a researcher cares about (the "stratum") might be tiny compared to the rest of the corpus. This causes its signal to be drowned out by the majority. Second is sign reversal, a manifestation of Simpson's Paradox. This occurs when a pattern exists within subgroups but flips direction across them. This causes the global signal to cancel itself out. As illustrated in, these two scenarios render standard global LASSO selection ineffective for targeted scientific discovery.

How It Works

The authors build on the SAE-based pipeline where documents are mapped to sparse feature activations. Because the SAE features are fixed before the statistical selection begins, they can inject researcher-specified covariates directly into the regression step. They propose two distinct econometric-inspired paths:

1. Interaction-lasso (for Sign Reversal): Use this if you suspect the relationship between a feature and an outcome might flip across groups. The authors augment the SAE activation matrix with products of the features and the covariates ($Z \odot C_p$). This allows the model to select a feature if it is highly discriminative within a specific stratum. It works even if the global coefficient $\beta$ is zero.
2. Demeaned-Reweighted-Lasso (for Stratum Imbalance): Use this if your target group is too small. First, they perform within-stratum demeaning (subtracting the group mean from both features and outcomes). This isolates the variation within each subgroup. Second, they apply inverse-frequency reweighting. This assigns higher weights to rare strata so they contribute equally to the feature selection process.

These methods are additive. A feature that is globally significant will still be selected alongside these specialized, conditional features.

Numbers

The authors validate these methods using synthetic datasets with known ground truths. In the stratum imbalance scenario, the target signal becomes increasingly rare ($r=0.20$). In this case, the demeaned-reweighted-lasso maintains a surface similarity score of $0.730$. This is a significant improvement over the global LASSO baseline, which drops to $0.393$ [Table 1]. This delta shows the method effectively prevents rare signals from being suppressed.

In the sign reversal scenario, the interaction-lasso is the only method that recovers the signal. The authors report an F1 similarity of $0.576$ for interaction-lasso. This compares to just $0.362$ for the standard LASSO [Table 2]. This demonstrates that modeling interactions is necessary when the direction of a difference changes across strata.

Regarding the LLM component, the authors studied the "prompt budget" (the number of samples shown to the LLM). and show that increasing the number of sampled texts does not consistently improve performance.

This suggests the heavy lifting comes from statistical feature selection. The LLM's role is mainly to interpret the selected features.

What's Missing

There are several gaps that a practitioner should note:

• Covariate Quality: The framework depends on the researcher's ability to provide accurate covariates. Poorly chosen or noisy covariates can steer discovery toward unhelpful strata. The methods cannot discover which covariates matter on their own.
• Continuous Variables: The current implementation focuses on binary or categorical strata. Handling continuous covariates (like age) would require a different residualization strategy. This could introduce new artifacts.
• Complexity vs. Stability: The interaction-lasso method expands the feature space significantly. This increases computational cost. It can also lead to unstable estimates in high dimensions when both SAE features and covariates are sparse.

Should You Prototype This

Yes, if you are doing structured text analysis for social science or policy.

If your work involves moving beyond "what are the most common words in Group A?" to "how does Group A behave specifically in the context of Topic X?", this is a major upgrade. Statistical conditioning beats "LLM-direct" prompting (simply feeding labels into a prompt). Do not rely on the LLM to handle the statistics. Instead, use the stats to select the features, then let the LLM describe them.

Code is reportedly available; see the paper for the canonical link. Since the method relies on existing SAE implementations, the barrier to entry is relatively low. The primary task is implementing the weighted LASSO and interaction terms. If you have a stable SAE pipeline, prototyping this logic is likely worth the effort.

Figures from the paper