A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming

Researchers have been trying to automate feedback for students learning to code for years. While we can easily automate syntax checks or unit tests, providing nuance remains a massive challenge. Providing the kind of insight a human TA (Teaching Assistant) offers—identifying logical flaws without just handing over the answer—is difficult. Current efforts often focus on either rigid, rule-based error messages or broad, unhelpful LLM generations.

This study addresses the gap between these two extremes. By deploying different types of AI-generated feedback in an actual introductory Python course, the authors sought to understand how different modalities affect student learning trajectories. The findings suggest that more "information" isn't always better. Giving students a concrete counterexample (a failing test case) did not necessarily correlate with more effective learning than a well-crafted text hint.

The limits of raw error signals

The status quo in automated programming education relies heavily on execution-based feedback. Students submit code, a test suite runs, and they receive a binary pass/fail. This is excellent for catching syntax errors. However, it leaves students stranded when they hit logical errors. These are bugs where the code runs perfectly fine but produces the wrong result.

Existing research has attempted to bridge this gap using various LLM-based strategies. These include generating Socratic questions or dynamic test cases. Much of this work relies on small-scale user studies or expert annotations of "quality." Such methods do not tell us how a student actually behaves when stuck in a loop of failed submissions. As seen in, immediate improvement following feedback is far from guaranteed.

The mean change in test case pass rate fluctuates near zero across all conditions. Simply injecting an AI signal into the loop does not automatically trigger a successful correction.

Comparing hint modalities via randomization

To move beyond anecdotal evidence, the authors implemented a randomized protocol within a live Python course. They used gpt-4o-mini via the OpenAI API to generate feedback for failing submissions. They chose the smaller model primarily to manage costs for the class size. The experiment compared three distinct groups:

1. No AI Feedback: The control group received only the standard pass/fail verdicts from instructor-authored test cases.
2. Failing Test Case Feedback: The LLM was prompted to generate a specific input-output pair that would cause the student's current code to fail. This serves as a "counterexample" to guide debugging.
3. Natural Language Feedback: The LLM was tasked with providing a concise hint. This hint identifies a likely issue without suggesting specific code changes or providing test cases.

The researchers organized the data into the PROGFEED dataset, which tracks 6,693 submissions. This allows for a longitudinal view. Researchers can watch how a student moves from their first failure to their eventual successful submission.

Natural language associates with convergence

The authors' primary findings center on the relationship between feedback type and "eventual success." Using logistic regression (a statistical method to model the probability of an event), the paper reports that natural language feedback is significantly associated with a higher probability of reaching a correct solution. Specifically, students receiving natural language hints had an odds ratio of approximately 2.33 compared to the no-AI baseline. This means these students were more than twice as likely to eventually reach correctness.

The study also examined the "time to success" using a Cox proportional hazards model (a way to model the time until an event occurs). The authors find that natural language feedback is associated with a 9% increase in the hazard of reaching correctness at any given attempt (HR = 1.09, $p = 0.03$). This suggests that students receiving text hints are associated with faster convergence to a correct solution.

In contrast, the "test case" modality showed much weaker associations. The paper reports that only 66% of AI-generated test cases were actually "valid." A valid test case fails the student's code but passes the reference solution. The remaining 34% were either false positives (failing both) or had no effect. Interestingly, even when the test cases were valid, they were not associated with shorter debugging processes. The authors suggest this may be due to a mismatch in skill. Introductory students often lack the systematic debugging skills required to turn a raw counterexample into a functional fix.

The reliability gap in automated testing

There are significant caveats to consider. First, the study is associative, not causal. Because feedback was only triggered upon a failing submission, the researchers cannot definitively claim that the feedback caused the success. They can only state that the feedback is strongly associated with it.

Second, there is a notable reliability issue with the test case generation. A 34% failure rate in test case validity is a high margin for a system intended to guide a learner. If a student receives a "false positive" test case, they may spend significant time debugging code that is actually correct. This could lead to frustration and attrition.

Finally, the study lacks a human baseline. While we know natural language hints show stronger associations with success than raw test cases, we don't know how they compare to high-quality human instructor feedback. For a practitioner building an automated tutor, the goal is to approach the efficacy of a human TA.

Verdict: Favor hints over test cases

If you are building an automated feedback engine for novices, the data suggests prioritizing natural language hints over raw test case generation. For beginners, the cognitive load (the amount of mental effort required) of interpreting a raw input-output discrepancy may be too high. A text hint acts as a translation layer. It turns a mathematical discrepancy into a conceptual clue.

However, do not treat LLM output as a "set and forget" feature. The 34% invalidity rate in test cases is concerning. Additionally, the authors noted that natural language feedback can sometimes be "vague" (accounting for 5.4% of instances). This means you need a validation layer. Whether that is a second LLM acting as a judge or a formal verification step, you cannot ship raw LLM outputs directly to a student without a quality gate.

The PROGFEED dataset is available at https://github.com/umass-ml4ed/progFeed-dataset-public for those looking to prototype these dynamics themselves.