When can a posterior predictive check identify the learning rate? Exact degeneracy in Gaussian models and implications for Generalised Bayesian Inerence

The Learning Rate Selector is Blind to Your Data

In Generalized Bayesian inference, we often deal with model misspecification (situations where our assumed model doesn't match reality). To mitigate this, we temper the likelihood using a learning rate $\eta$. Choosing the right $\eta$ is critical. Too high, and the model overfits the noise. Too low, and you lose the signal. Researchers recently proposed using a posterior predictive check (PPC) to pick this rate. This method essentially finds the smallest $\eta$ that does not trigger a statistical alarm.

However, a new analysis reveals a fundamental flaw in this strategy. For the standard Gaussian linear model, the selector does not actually look at your data. Because of a mathematical property called pivotality (where a statistic's distribution does not depend on unknown parameters), the decision is effectively made by a formula before you observe any data. This leads to a "collapse" where the algorithm defaults to the lowest possible learning rate. This results in overly cautious, bloated predictive intervals.

The failure of the PPC selector

The current approach relies on a simple heuristic. Use a log-likelihood PPC $p$-value to find an acceptable $\eta$. The idea is that if the model fits well, the $p$-value should stay above a threshold $\alpha$. If the $p$-value is too small, the model is rejected. We then increase $\eta$ to temper the influence of the misspecified likelihood.

While this worked in complex settings like diachronic sense change, it breaks down in the canonical Gaussian case. The paper demonstrates that the selector is either indifferent to $\eta$ or indifferent to the data. In the first case, it accepts every value in your search grid or none at all. In the second, it ignores the specific values in your dataset. It cannot "learn" anything from the evidence provided. This is a structural degeneracy.

Mechanism of the data-free collapse

The author identifies two ways this mechanism fails, depending on whether you assume the variance ($\sigma^2$) is known or unknown.

1. Known Variance: If $\sigma^2$ is known and you use a flat prior, the paper proves in Theorem 1 that the $p$-value is constant for every $\eta$. As shown in [Figure 1A], the $p$-value remains perfectly flat across the $\eta$ spectrum. The selector becomes binary. It either accepts the entire grid or rejects everything. It provides zero granularity for tuning. Furthermore, if the variance is misspecified, the selector becomes "two-sided non-identifying" [Figure 1B]. It fails to react whether the data is over-dispersed or under-dispersed.

2. Unknown Variance: The situation is worse when $\sigma^2$ is unknown. Using a reference prior, the author derives a closed-form law in Theorem 2. Remarkably, the $p$-value $p_{n,d}(\eta)$ depends only on the sample size ($n$), the dimension ($d$), and the learning rate ($\eta$). It is independent of the realized data and the underlying data-generating process. This is illustrated in .

Monte Carlo $p$-values from different distributions—Gaussian and Student-$t_3$—land precisely on the same deterministic curve.

Because the $p$-value is a fixed function of your experimental setup, the selector usually defaults to the minimum value in your grid. This is visualized in .

The selected $\eta$ collapses to the grid floor across most of the $(n, d)$ plane.

The cost of over-tempering

The practical consequence is a significant drop in model calibration. When the selector defaults to a very low $\eta$, it "over-tempers" the model. This pushes the posterior toward the prior more aggressively than the data warrants.

The paper quantifies this "calibration cost" in .

Because the PPC selector picks a near-constant, low $\eta$, the 90% predictive intervals show massive over-coverage. They hit 0.97–0.99 in practice, which is much higher than the nominal 0.90. More importantly, these intervals are 1.5 to 1.9 times wider than those from more robust methods. This includes held-out selection or simply setting $\eta=1$. You are not just getting a safer model. You are getting a model that is significantly less informative.

Scope and prior sensitivity

This is a "boundary result." The author does not claim all PPC selectors are broken. Instead, the most common implementation in the Gaussian scale-location family is flawed.

The degeneracy is tied to the reference prior and the symmetry of the Gaussian likelihood. If you use more informative priors, such as a proper ridge prior, the $p$-value regains some dependence on $\eta$. However, the paper notes that for typical diffuse priors, the dependence is numerically negligible. The selector still tends to collapse to the grid floor. Unless you have very strong prior beliefs that conflict with the data, the "intelligence" of the PPC selector remains largely illusory.

The verdict

If you work with Gaussian-linear models and use diffuse or reference priors, do not use the log-likelihood PPC to select your learning rate. The math dictates that the tool will ignore your data. It will provide you with unnecessarily wide and poorly calibrated intervals.

The paper suggests two paths forward: * Switch selection methods: Use held-out/ELPPD (Expected Log Predictive Density) selection or generalized posterior calibration. These methods are more likely to utilize the information in your dataset. * Use the pre-screening diagnostic: The author proposes a "cheap, data-free" check. Before fitting a model, compute the theoretical $p$-value curve using the formulas in Theorems 1 and 2. If the minimum $p$-value on your grid is already above your threshold $\alpha$, the PPC selector will be useless. This saves the computational overhead of a failed tuning loop.

Figures from the paper