Legal aid eligibility and court outcomes: a design-based double-machine-learning approach

The Hidden Cost of Legal Inequality

Why does the quality of a person's legal representation change based on their bank account? In many democratic societies, legal aid is a fundamental right. It is designed to ensure that poverty does not dictate judicial outcomes. However, determining exactly how much "better" a private lawyer performs compared to a publicly-funded one is notoriously difficult. This is because the two groups are fundamentally different. People who can afford private lawyers are rarely the same people who qualify for government assistance.

A new study from the University of New South Wales addresses this selection bias head-on. By analyzing administrative data from New South Wales, Australia, the authors find that being denied legal aid and hiring a private lawyer decreases a defendant's probability of incarceration by 8.1 percentage points. This is a substantial shift in the likelihood of going to jail. Yet, there is a striking contradiction. For those who are eventually sentenced to prison, the study finds they actually tend to serve longer terms if they had private representation. This suggests a complex trade-off between preventing jail time altogether and negotiating shorter stays once a sentence is inevitable.

The Selection Bias in Indigent Defense

The core problem in studying legal aid is "selection bias" (a situation where the groups being compared are inherently different). This is the statistical equivalent of trying to compare professional athletes to couch potatoes to see if training works. In the legal system, eligibility for aid is not assigned randomly. Instead, it is determined by a "means test" (a set of rules regarding income and assets). As the paper notes, those who fail the test and hire private lawyers are naturally wealthier.

If a researcher simply compares the court outcomes of legal aid recipients to those who hired private lawyers, the results will be skewed. Wealthier defendants might have better outcomes not just because of their lawyers. They may also have higher stability or different social profiles. This is evident in the raw data. As shown in [Table 1], applicants denied aid have higher incomes, more valuable assets, and higher housing costs. Without a way to account for these "confounders" (variables that influence both the likelihood of getting aid and the eventual court outcome), any conclusion about lawyer quality would be unreliable.

Learning the Latent Assignment Function

To solve this, the author employs a technique called Double Machine Learning (DML). Specifically, the study uses the Interactive Regression Model (IRM). The goal is to reconstruct the "assignment function" (the hidden logic used to decide who gets aid). Because this logic is not a simple formula, the author uses Random Forests (a machine learning algorithm that uses many decision trees) to learn it.

The DML process works in two parallel tracks to achieve "double robustness" (a property ensuring the estimate remains valid even if one of the two models is slightly off). First, the algorithm learns the "propensity score" (the probability that a specific individual will be granted legal aid). Second, it learns a regression model of the actual court outcomes. By combining these, the model can "re-weight" the data. This creates a synthetic comparison where the groups look statistically similar.

The effectiveness of this approach is demonstrated in .

The estimated propensity scores show a clear overlap between the groups. This overlap satisfies the "common support" assumption (the requirement that there are similar subjects in both groups). This means there are enough similar individuals in both the "aid granted" and "aid denied" categories to make a meaningful comparison. Furthermore, the author uses "variable importance" metrics to peek inside the black box. This reveals that income-related variables are much stronger predictors of aid denial than assets, as seen in .

Quantifying the Performance Gap

The results of this modeling reveal a significant disparity in how legal representation impacts freedom. The paper reports that denying legal aid decreases the probability of a defendant being incarcerated by 8.1 percentage points. When the sample is narrowed to exclude self-represented defendants, the effect of hiring a private lawyer becomes even more pronounced. In that group, the probability of incarceration decreases by 9.7 percentage points.

However, the study reveals a nuanced "intensive margin" effect (the impact on the length of a sentence once incarceration is already decided). The authors find that for those sentenced to jail, aid denial is associated with a 5.6-month increase in the length of the incarceration spell. This creates a paradoxical picture. Private lawyers appear better at preventing jail time entirely. However, public lawyers might be more effective at negotiating shorter sentences for those who cannot avoid jail.

This pattern is visualized in .

It shows that while legal aid recipients have a higher unconditional average incarceration length, they actually spend less time in custody once they are behind bars. The paper interprets this as evidence that public lawyers may rely more heavily on plea bargaining (negotiating a guilty plea to avoid trial). This is a time-efficient strategy used to manage high workloads. Such a strategy can lead to more frequent but shorter sentences.

Limitations and Unobserved Confounders

Despite the sophisticated DML approach, the paper identifies several critical boundaries. First, the estimate regarding the length of incarceration is subject to "collider bias" (a bias caused by conditioning on a variable that is influenced by both the cause and the effect). This makes the causal interpretation of sentence length more speculative than the probability of going to jail.

Second, the study cannot definitively prove why the performance gap exists. The authors note that the gap could be driven by "self-selection" (wealthier lawyers choosing private work) or "workload" (public lawyers prioritizing speed due to limited budgets). While the data suggests both are likely, the dataset does not contain enough information to separate them.

Finally, the author performs a sensitivity analysis to address "unobserved confounding" (the fear that some unmeasured factor is driving the results). Using a framework from Chernozhukov et al. (2022), the author tests how strong an unobserved variable would have to be to flip the results. The findings in and suggest that even an "adversarial" unobserved confounder would struggle to overturn the primary finding.

That finding is that denying aid reduces the likelihood of incarceration.

The Verdict: A Resource Trade-off

The evidence suggests that the current legal aid system faces a fundamental quantity-versus-quality trade-off. If a government spreads its budget thinly to ensure broad access, it risks providing representation that relies on rapid plea bargaining. This can lead to more frequent but shorter sentences. If it concentrates resources to improve the quality of each case, it may leave many indigent defendants without any help at all.

For policymakers, this is not just a theoretical debate. It is a measurable tension. The paper demonstrates that the performance gap is real. It is also tied to the systemic pressures of workload and remuneration. Until the funding gap between public and private sectors is addressed, the "equality before the law" promised by the state will remain divided by the ability to pay.

Figures from the paper