Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection

Whose hotel does the AI recommend?

Travelers are increasingly turning to large language model (LLM) assistants to plan their trips. They are moving away from traditional search engines. This shift transforms these models into "algorithmic infomediaries"—gatekeepers that decide which specific properties a traveler actually sees. While the industry is already seeing the rise of "generative engine optimization" (GEO) to influence these outcomes, the actual mechanics remain undocumented. Researchers wanted to know: which reputation signals actually move the needle for an AI? Do they weight them the same way humans do?

The mystery of the machine gatekeeper

The fundamental question investigated by Ahmed Baig et al. is a supply-side one. Among the reputation signals a hotel can manage—such as guest ratings, review volume, management response, and eco-certifications—which ones causally change the probability of an LLM recommending the property?

For a hotel manager, the stakes are asymmetrical. If an LLM systematically fails to surface a property, that hotel becomes effectively invisible. Currently, companies attempt to optimize for these models using purely heuristic approaches. They are essentially guessing which levers to pull to win inclusion in a generative response.

Cracks in the human benchmark

Common wisdom in hospitality management is built on decades of electronic word-of-mouth (eWOM) research. This literature establishes a clear hierarchy of signals that move human behavior. Guest rating (valence) is the dominant driver. This is followed by price. Review volume, brand affiliation, and management response play secondary roles.

However, there is a gap in our understanding of whether LLMs reproduce this human ordering. We do not know if the "intelligence" in these assistants picks up on service recovery. Service recovery refers to how a manager handles a bad review. Or is the AI simply latching onto surface-level metadata? The researchers suspected the LLM might be a fundamentally different decision-maker.

Auditing the selection logic

To avoid the pitfalls of simple observation, the authors conducted a randomized, choice-based conjoint algorithm audit. They did not just scrape recommendations. They built a controlled laboratory. As shown in, the study focuses on the "selection" stage of the pipeline.

This is the moment the model chooses one winner from a fixed set of candidates. This isolates the decision rule from the complexities of web retrieval.

The setup was rigorous. The researchers presented twelve different models to the system. This included open-weight systems like Llama-3.2-3B and proprietary APIs like GPT-4o-mini and Claude. Models were shown sets of five synthetic hotel cards. Each card had seven attributes that were independently randomized. By using this randomized approach, the authors could calculate the Average Marginal Component Effect (AMCE). This is the causal change in recommendation probability when a specific attribute is toggled. They ran over 60,000 model calls across various traveler personas. These personas included "budget families" or "eco-couples."

Divergent weights and hidden biases

The results reveal a divergence from human logic. The authors report that while LLMs honor the primacy of guest rating and price, their internal weighting is skewed.

According to the pooled results in, guest rating remains the heavy hitter.

A top rating increases recommendation probability by 31.6 percentage points. Price is equally powerful in the opposite direction. A high price drops selection probability by 30.0 points. However, the "personality" of the AI emerges in the secondary signals. The study finds that LLMs heavily overweight eco-certification (+11.6 pp). This elevates it to the third most important signal. Humans treat it as a minor factor. Most strikingly, the authors report that management response has an effect of only 0.11 pp. This is statistically equivalent to zero.

Furthermore, the audit uncovered a significant "position bias." Even though list position is a content-free artifact, it causally shifts recommendations. The authors convert this into a "price-equivalent" to make it managerially legible. Being in the first slot is worth approximately $11.7 per night [Table 9]. Finally, the study highlights a transparency gap. As shown in, the models' stated reasons correlate poorly with their actual behavior. They tend to over-cite brand affiliation. Meanwhile, they remain silent about the list position or review volume that actually drove their decision.

Implications for the AI economy

These findings suggest that "Generative Engine Optimization" is currently built on guesswork. If these results generalize, the immediate implication is clear. Hotel managers should reconsider spending resources on managing review responses for the sake of AI. The machines appear to ignore them entirely. Instead, investment should pivot toward high ratings and visible eco-certifications.

The research also exposes a structural tension for platforms. Since list position acts as a de facto reputation signal worth real money, platforms face a design challenge. Their UI/UX choices can dictate which hotels get noticed. If a platform wants to be seen as a neutral advisor, they may need to implement position-calibration. This would involve adjusting the order of results to remove this inherent distortion.

Ultimately, this paper serves as a warning. The "rationales" provided by LLMs are often post-hoc justifications. They are not necessarily accurate disclosures of their decision logic. The most critical follow-up experiment would be a longitudinal audit. This would see how these weights drift as models are updated. A hotel's "optimized" profile today might be obsolete by next month's model release.

Figures from the paper