Directional AI Advice: Experimental Evidence from Healthcare

AI Guardrails Shift Healthcare Decisions

A large study in a Chinese hospital found that when patients use AI chatbots before seeing a doctor, the AI's cautious advice actually changes what happens in the clinic. Because the AI is programmed to avoid recommending drugs to limit legal risk, patients end up getting fewer prescriptions and more diagnostic tests.

Generative AI is rapidly moving from a novelty to a primary source of expert advice. In high-stakes sectors like healthcare, law, and finance, people are increasingly turning to Large Language Models (LLMs)—the engine behind tools like ChatGPT—to interpret complex information. Historically, the relationship between a client and an expert has relied on the expert's independent judgment. This helps manage informational asymmetries (situations where one party knows significantly more than the other).

However, a new study reveals that this dynamic is shifting. When patients consult AI first, they are absorbing the specific biases and "guardrails" installed by AI developers. These guardrails are safety constraints designed to prevent the model from providing dangerous or legally actionable advice. The researchers argue that these design choices are not neutral. They act as a form of unintentional policy that can propagate into real-world clinical decisions at scale.

The bias in the machine

In many healthcare markets, providers deal with "credence goods"—services where the quality is difficult for the consumer to evaluate even after the transaction is complete. Because patients often lack specialized knowledge, they rely on the physician's discretion. This creates a vulnerability where overprescription can occur.

Current approaches to managing AI safety focus on preventing the model from acting as a rogue doctor. To minimize legal exposure, developers encode defensive behaviors into the models. The goal is to steer the AI away from liability-prone actions. This includes avoiding specific diagnoses or recommending particular medications. Instead, the AI is steered toward safer, more passive actions. These include suggesting a diagnostic test or referring the user to a professional. The authors suggest these safety measures create a "directionality" in the advice. The AI becomes a tool that pushes users away from active treatments and toward further investigation.

How guardrails steer patients

The researchers investigated this mechanism through a large-scale, two-layer randomized field experiment at a large public hospital in China. The study involved over 10,000 outpatient visits. The study utilized a sophisticated randomization design to isolate the effects of the AI .

First, physicians were randomized into "Exposed" or "Unexposed" groups. Second, patients seeing Exposed physicians were randomly assigned to either receive chatbot access (Treatment) or no access (Control).

The mechanism of influence follows a specific sequence: 1. Encoded Constraint: Developers implement guardrails to mitigate liability. Recommending a drug carries high legal risk if the patient has an adverse reaction. Conversely, recommending a test carries much lower risk for the developer. This is because the cost of the test falls on the patient or the health system. 2. Informational Priming: The chatbot interacts with patients by eliciting symptom descriptions. The authors found that the AI actively shapes the conversation. It often introduces topics like medication or testing that the patient had not initially raised. 3. Directional Advice: The chatbot issues highly asymmetric advice. The study found that while diagnostic testing received "clean" recommendations (advice without accompanying warnings) 94.5% of the time, medication mentions were frequently accompanied by cautions [Table 2]. 4. Clinical Propagation: Patients carry this skewed information into the consultation. The research shows that this "directional" advice effectively narrows the physician's discretion. It does this by altering the patient's expectations and requests.

Measuring the clinical shift

The paper reports significant changes in how doctors treat patients who have used the chatbot. By analyzing hospital administrative records, the authors demonstrate that chatbot access directly alters clinical practice patterns. Specifically, the study finds that the likelihood of a visit resulting in any prescription declines by 4.6 percentage points relative to a baseline of 87% .

This decline is notably driven by reductions in the prescribing of Traditional Chinese Medicine (TCM) and antibiotics.

Conversely, the AI's tendency to favor testing manifests in the clinic. The probability that a visit includes diagnostic testing increases by 2.7 percentage points from a baseline of 23% . For patients who actually used the chatbot, these effects are even more pronounced. The authors report that chatbot usage is associated with a 27 percentage point reduction in the probability of receiving a prescription. Additionally, usage is linked to a 16.1 percentage point increase in the likelihood of diagnostic testing [Table 5].

Interestingly, these shifts do not appear to change total healthcare expenditures. The savings from reduced medication are largely offset by the increased costs of diagnostic tests [Table 4]. The influence of the AI is also highly dependent on the physician. The effects are most concentrated among doctors who are personally receptive to patient input. They are also most pronounced among physicians with high baseline prescribing rates [Table 7, Table 9].

Limits of the AI effect

While the results are striking, the authors highlight several areas where the evidence remains inconclusive. First, the study cannot definitively determine the cause of increased testing. It is unclear if this is a direct response to patient requests. Alternatively, it could be a "defensive medicine" response. This occurs when physicians order more tests to protect themselves from perceived litigation risk.

Second, the welfare implications are fundamentally ambiguous. The paper does not attempt to calculate a net welfare effect because the trade-offs are unclear. While reducing the overprescription of antibiotics is a public health win, the reduction in medication might withhold beneficial treatments. Furthermore, the increase in testing adds costs to the system without guaranteed clinical benefits.

Finally, the study notes a divergence in how the consultation is perceived. While physicians report that AI-prepared patients are easier to communicate with, the patients themselves report lower overall satisfaction .

Patients also report lower intended compliance with medical advice. This suggests that while AI might make the "data transfer" more efficient, it may simultaneously erode the trust and authority that underpin the patient-physician relationship.

The verdict: A new layer of policy

The evidence suggests that the verdict on consumer-facing medical AI depends on your optimization goals.

If the goal is to reduce overprescription and curb the use of antibiotics or TCM, these "defensive" AI guardrails are remarkably effective. However, if the goal is to maintain the traditional authority of the physician, the current generation of LLMs may be counterproductive. The study demonstrates that the design choices made by a handful of AI developers act as a powerful, unvetted policy lever. These choices can reshape population-level healthcare utilization. Practitioners should prepare for a future where patients arrive with a specific, engineered direction for their care.

Figures from the paper