JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

The Capability Cliff in AI Game Engineering

Researchers are attempting to build a way to test if AI can actually build complete video games using professional tools like the Godot engine. Instead of just checking if the code runs, they use a system that "plays" the game to see if it actually behaves like a real game. This moves the goalpost from mere syntactic correctness—writing code that doesn't crash—to functional engineering. Here, the AI must manage complex, interconnected systems.

The field of AI-driven game development has seen rapid progress in generating individual assets like textures or simple gameplay rules. However, a massive gap remains in "project-level" engineering. This is the ability to architect a cohesive framework of multiple files, scenes, and scripts that work together. Previously, researchers relied on small datasets or subjective "LLM judges" to grade AI performance. Such methods lack the rigor needed for professional software engineering. This paper addresses that vacuum by introducing a deterministic way to measure whether an AI-generated game actually works when played.

The promise of autonomous game architects

The prevailing assumption in current AI research is that as Large Language Models (LLMs) grow in scale, they will naturally graduate from writing single functions to managing entire software repositories. The expectation is that "Code Agents"—AI systems equipped with iterative debugging tools—will bridge the gap. They should move from writing a snippet of code to constructing a complex, multi-file project.

If this claim holds, we should see AI models successfully navigating the transition from small games to large architectures. Under this paradigm, an agent that can fix a syntax error should eventually design a robust game loop. It should also manage a stable global state management system. As seen in earlier work on web-based games, the goal is to move toward professional engines like Godot. There, the complexity of the project scales alongside the depth of the code.

A rigorous descent into the codebase

To test these assumptions, the authors developed JAMER. This framework consists of JamSet (7,833 verified projects for training) and JamBench (300 manually verified projects for evaluation). They leveraged "Game Jams"—intense, short-term coding competitions—to harvest thousands of real-world, open-source projects. By using the Godot engine in "headless mode" (running the engine without a graphical user interface), the researchers built a four-level verification pipeline.

This pipeline moves from basic file integrity to "L3b" (a stage that collects runtime behavior by simulating player inputs). The system records data such as velocity changes or signal triggers. The results reveal a stark reality that contradicts the idea of seamless scaling. The study finds a "capability cliff" as project scale increases. Specifically, when tasked with code completion (Task 2a), the authors report that runtime pass rates plummet from 80.4% on small projects to just 5.7% on large ones. This indicates that while models handle localized logic, they struggle immensely when the context expands to a full project.

Furthermore, the researchers introduced two specialized metrics to look beyond simple compilation. The Structural Completeness Score (SCS) measures how much of the intended architecture is present. The Behavioral Alignment Score (BAS) measures how closely the game's actual movements match a reference project.

Success in the small scale

The data shows that the fundamental building blocks are being mastered. For "Small" tier projects, frontier models perform remarkably well. They achieve runtime pass rates as high as 80.4% and maintain high structural completeness. At this scale, the relationship between the prompt and the output is tight. Models can successfully navigate the dependency trees of a simple game.

The authors also demonstrate that fine-tuning is a viable path forward. By training a model on the curated JamSet data, they report that a base model can adopt professional engineering habits. For example, the authors find that fine-tuning helped a model increase its use of "autoload" scripts (a Godot mechanism for global state management) from 55.1% to 77.1%. It also significantly improved its use of abstracted input actions. This suggests that the "knowledge" of how to build a game is present in the data. It just needs to be properly distilled into the model.

The architectural bottleneck

Despite these successes, the paper exposes deep cracks in the "Agentic" approach to coding. While Code Agents—models that use iterative loops to find and fix errors—significantly improve compilation pass rates, they fail to improve actual gameplay behavior. The authors report that while Agents help a model pass the "L2" compilation test, they yield no meaningful gains in the Behavioral Alignment Score (BAS).

This reveals a critical distinction. Current AI agents are excellent at syntactic repair (fixing typos or broken references). However, they are largely incapable of architectural design. A model might fix a crash so the game runs, but it won't necessarily fix a poorly designed player controller. It might not fix a disconnected scene hierarchy.

The researchers highlight several ways models fail to meet professional standards: 1. Shortcut Outputs: As shown in, models often produce "runnable shells." These are projects that compile perfectly but contain almost no actual gameplay logic. This leads to very low SCS scores. 2. Semantic Drift: Even when code is syntactically correct, models suffer from "cross-file semantic drift." A model might rewrite a function in one file that accidentally breaks a requirement in another. This causes the game to behave incorrectly even if it doesn't crash. 3. Lack of Engine Knowledge: Models frequently struggle with engine-specific types. They write algorithms that are logically sound in Python but fail in Godot's GDScript due to subtle type constraints.

Moving beyond syntax

The implications for the field are clear. The industry cannot rely on "debugging loops" alone to create autonomous developers. If the bottleneck is architectural rather than syntactic, then simply telling an AI "fix this error" will never result in a sophisticated game.

Future research must pivot from teaching models how to fix code to teaching them how to structure code. This requires moving away from generic programming datasets. Researchers should move toward highly structured, domain-specific environments. These environments must emphasize how different files and nodes interact over time. To settle the question of whether AI can truly build games, we need to see if models can move past the "capability cliff." They must demonstrate mastery over the invisible threads—the interfaces and state managers—that hold a large-scale project together.

Figures from the paper