WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts

Does a Pretty Website Actually Work?

Most AI models can make a webpage that looks good. However, they often fail to make it actually work correctly. WebRISE is a new testing system. It checks if a webpage's buttons, filters, and menus actually do what they are supposed to do when clicked.

The field of Multimodal Large Language Models (MLLMs) has moved rapidly. Models now go from understanding images to generating executable web artifacts (files containing HTML, CSS, and JavaScript). While developers can ask an AI to turn a sketch or a video into a website, a fundamental question remains. Is the resulting page truly usable, or is it merely a visual illusion? Current benchmarks often focus on "local evidence." This means they check if a button exists or if the colors look right. They frequently miss the complex, requirement-induced transitions (the changes in a page's state caused by user actions) that determine if a site functions reliably.

Testing the Logic, Not Just the Look

The core problem in evaluating AI-generated web code is the gap between appearance and behavior. A model might generate a perfectly styled "Delete" button. However, clicking it might leave the item list unchanged. A shopping cart might also fail to update the total price after an item is removed.

Evaluating these failures is difficult because web interaction is stateful (the page's behavior depends on previous actions). Think of a website like a vending machine. It is not enough to check if the machine has buttons and a glass front. You must also verify that pressing "B4" actually dispenses the specific snack you selected. This is a state transition. Most current evaluation methods are "event-centric." They check if a single action produces a single response. They fail to catch "state-consistency" errors. These occur when an action works in isolation but breaks a rule elsewhere. Examples include a filter resetting a page unexpectedly or a navigation step wiping out unsaved form data.

The Architecture of Interaction Contracts

To solve this, the authors propose WebRISE. This benchmark treats web generation as a matter of "conformance" (following a specific set of rules). Rather than comparing a generated page to a static reference image, WebRISE uses a framework called an Interaction Contract Graph (ICG).

As illustrated in, traditional evaluation protocols often rely on fragmented inputs.

These include single screenshots or fixed scripts. In contrast, WebRISE derives a formal contract from the task requirements. This contract consists of observable UI states, user-intent transitions, and specific assertions (statements that must be true). The ICG maps out the "legal" moves a user can make and the necessary consequences of those moves.

The evaluation process follows a sophisticated pipeline shown in .

First, the system converts requirements into the ICG. Next, a "contract-guided agent"—an autonomous browser actor—attempts to execute the transitions. Finally, a "dual-channel oracle" (a system that verifies truth) checks the results. This oracle does not just look at the screen. It monitors the Document Object Model (DOM), which is the underlying structural map of the webpage. It ensures that specific elements changed or stayed the same as required. It uses two types of checks. It uses [CHANGE] assertions to catch transient events like a loading spinner appearing. It also uses [AFTER] assertions to ensure the final, stable state of the page is correct.

Breaking the Visual Proxy

By implementing this rigorous testing, the authors reveal a significant disconnect in current AI capabilities. The study finds that visual quality is a poor proxy (a substitute) for functional correctness. For instance, the authors report that the model Qwen3.6-35B-A3B achieved a high visual score of 80.8 on Markdown tasks. Yet, its transition validity—the measure of how well it executes intended actions—was a mere 15.5 [Table 2]. This means the model looked great but failed almost every functional test.

The researchers tested 14 representative models across 442 tasks. They discovered that even the strongest model, GPT-5.5, struggles significantly. GPT-5.5 reaches only 65.6% transition validity and 66.3% requirement coverage. This means that even the leading AI leaves roughly one-third of required interactions broken or unimplemented.

The findings also highlight the importance of the input format. Providing a video of the desired interaction provides the strongest signal for the model. It improves implicit requirement coverage (meeting unspoken rules) by 10.6 percentage points compared to text-only instructions. This suggests that seeing the temporal flow (how things change over time) helps models grasp the "unspoken" rules of web design.

Furthermore, the WebRISE approach proves much more sensitive to bugs than older methods. In a "defect injection" test, researchers intentionally broke specific states in valid pages. The authors report that the ICG-based scoring detected state errors at 2–16 times the rate of traditional checkpoint-style evaluations [Table 4].

Identifying the Bottlenecks

The data allows us to categorize exactly where these digital architects are failing. By grouping failed transitions, the authors identify four main error families: Availability, Execution, State & Logic, and Feedback & Boundary .

The dominant failure mode is "State & Logic." Many models can perform a simple click. However, they fail to manage the ripple effects of that click. As seen in the case studies in and, a model might successfully change a checkbox. But it might fail to update the dependent price total or the user's position in a queue. These are not just simple coding errors. They are failures in reasoning about how different parts of a system must stay synchronized.

Limits of the Sandbox

While WebRISE provides a deep diagnostic tool, it has clear boundaries. The framework is designed to evaluate self-contained HTML artifacts. These are files that run entirely in a browser without needing a complex backend (server-side logic). Consequently, the authors note that the benchmark does not currently cover full production environments. These environments often involve external APIs (software interfaces), persistent databases, or multi-user concurrency.

Additionally, because the evaluation is tied to the specific requirements and assertions defined in the ICG, the system is limited. It is a measure of conformance to a defined set of rules. It is not an exhaustive test of every possible way a human might interact with a website.

Figures from the paper