A manifesto on generative games

The Engine That Dreams

World models are turning the game engine from something we write into something we train.

Contents

Part I · The Premise
  1. Games are simulators. What if the simulator were learned?
  2. From game engines to learned engines
  3. What a world model is
  4. A short history: World Models to Genie
Part II · The Machinery
  1. Representation: latents, tokens and frames
  2. Action conditioning and controllability
  3. Memory and consistency: why worlds forget
  4. Real-time inference
  5. Training data
Part III · The Collision
  1. Where learned engines beat hand-built ones
  2. Where they fail
  3. Hybrids: neural rendering over symbolic state
  4. Agents living inside world models
  5. Tooling for designers
Part IV · The Aftermath
  1. Economics: cost per frame vs. cost per asset
  2. Studios, modders and players
  3. IP, data rights and training on gameplay
  4. Safety and the infinite game problem
  5. Predictions: 1, 3 and 5 years

Part I

The Premise

Games are simulators. What if the simulator were learned?

Strip away the art and the story, and every game is a simulator. It holds a state, takes an input from the player, and computes the next state:

st+1=f(st, at)s_{t+1} = f(s_t,\, a_t)

For fifty years, ff has been written by hand. Programmers encode the physics, the rules, the collision checks and the enemy behaviour, one case at a time. The simulator is exact, fast and predictable, and it only knows what someone thought to write down.

A learned world model replaces ff with a function fitted to examples of play. Nobody writes the rules. The model watches enough gameplay that it can predict what happens next. Three questions decide whether this is a curiosity or a new foundation: whether it is creative, whether it remembers, and who stays in the loop.

Creativity

A hand-built engine can only produce what its authors anticipated. Its variety comes from content they made: levels, assets, scripted events. A learned simulator is different, because it generalizes. Asked for a state it has never seen, it still returns something plausible.

That is the promise, and the risk. The same property that lets a model invent a new room also lets it invent a wall you can walk through. In a hand-built game, surprising behaviour is a bug. In a learned one, it is the product, and the designer has to decide which surprises to keep.

Memory

Games depend on persistence. The door you opened stays open, and the item you dropped is still on the floor. A hand-built engine gets this for free, because the state is stored explicitly. A learned model has no such ledger. It conditions only on a finite window of recent frames and actions:

st+1∼pθ( ⋅∣st−k:t, at−k:t)s_{t+1} \sim p_\theta(\,\cdot \mid s_{t-k:t},\, a_{t-k:t})

Anything older than kk steps is gone unless the architecture carries it forward. Errors also compound, because each predicted frame becomes the input to the next. We return to this in Part II, and the interactive chart there shows how quickly small errors grow. For now, the point is that memory is the central open problem, and the main reason a learned simulator cannot yet stand in for an engine.

Humans in the loop

There are two humans in the loop. The player is the obvious one: they supply ata_t and judge whether the world feels right. The designer is the less obvious one. When the rules live in weights instead of code, a designer cannot open a file and change a number. They steer by choosing the training data, by prompting, and by constraining what the model is allowed to do.

That shifts the craft. Designing a game becomes less like writing a program and more like directing a performance: you shape what the system tends to do and review what it does. Whether that is liberating or maddening depends on the tools, which is where Part III picks up.

A learned simulator trades exactness for imagination. The rest of this piece asks what it takes to get the imagination without losing the game.

From game engines to learned engines

Draft this section.

What a world model is

Draft this section.

A world model learns the transition function of an environment:

st+1∼pθ(st+1∣s≤t, a≤t)s_{t+1} \sim p_\theta(s_{t+1} \mid s_{\le t},\, a_{\le t})

In a game, sts_t is what the player sees and ata_t is their input. Training minimizes the negative log-likelihood of observed gameplay:

L(θ)=− E(s,a)∼D∑tlog⁡pθ(st+1∣s≤t,a≤t)\mathcal{L}(\theta) = -\,\mathbb{E}_{(s,a)\sim\mathcal{D}} \sum_t \log p_\theta(s_{t+1}\mid s_{\le t}, a_{\le t})

A short history: World Models to Genie

Draft this section.

201820192020202120222023202420252026Latent dynamicsfor RLNeural gameenginesInteractive worldgenerationMemory &consistencyWorld Models (2018-03): Ha & Schmidhuber: VAE + RNN, agent trained inside its own dreamWorld ModelsPlaNet (2018-11): Planning in a learned latent dynamics modelPlaNetMuZero (2019-11): Learned model used for planning, no rules givenMuZeroDreamer (2019-12): Policy learned purely from latent imaginationDreamerDreamerV2 (2020-10): Discrete latents, Atari at human levelDreamerV2DreamerV3 (2023-01): One configuration across many domainsDreamerV3GameGAN (2020-05): NVIDIA: GAN that learns to emulate Pac-Man from framesGameGANDIAMOND (2024-05): Diffusion world model for Atari agentsDIAMONDGameNGen (2024-08): Diffusion model runs DOOM at 20 FPS on one TPU; paper posted Aug 27GameNGenOasis (2024-10): Decart: playable Minecraft-like model, public release Oct 31OasisMuse (2025-02): Microsoft WHAM-1.6B: trained on 7+ years of Bleeding Edge gameplay; later WHAMM runs Quake II at 10+ FPSMuseMatrix-Game 2.0 (2025-08): Skywork: open-source interactive world model, Aug 12Matrix-Game 2.0Genie (2024-02): DeepMind: playable 2D worlds from a single image, trained without action labelsGenieGenie 2 (2024-12): 3D environments from a prompt imageGenie 2Genie 3 (2025-08): 720p at 24 FPS in real time; consistent for minutes, visual recall about a minute backGenie 3Marble (2025-11): World Labs: persistent, exportable 3D worlds (not real-time); public release Nov 12MarbleProject Genie (2026-01): Genie 3 opened to AI Ultra subscribers, Jan 29Project GenieWorldMem (2025-04): Memory bank of past frames plus poses and timestamps for long-term consistencyWorldMemWorldPlay (2025-12): Real-time generation with long-term geometric consistency via reconstituted context memoryWorldPlay
Milestones in learned world models, 2018 to early 2026. Hover a dot for details. Month-level dates are approximate for papers first posted as preprints.

Part II

The Machinery

Representation: latents, tokens and frames

Draft this section.

Action conditioning and controllability

Draft this section.

Memory and consistency: why worlds forget

Draft this section.

Toy model of autoregressive drift: small per-frame errors compound when the dynamics amplify them.

Real-time inference

Draft this section.

Training data

Draft this section.

Part III

The Collision

Where learned engines beat hand-built ones

Draft this section.

Where they fail

Draft this section.

Hybrids: neural rendering over symbolic state

Draft this section.

Agents living inside world models

Draft this section.

Tooling for designers

Draft this section.

Part IV

The Aftermath

Economics: cost per frame vs. cost per asset

Draft this section.

Studios, modders and players

Draft this section.

IP, data rights and training on gameplay

Draft this section.

Safety and the infinite game problem

Draft this section.

Predictions: 1, 3 and 5 years

Draft this section.