An Interactive Guide to World Models

Anatomy of a world model

Almost every world model has the same parts: something that encodes what the player sees, something that predicts how it changes given an action, some form of memory, and something that turns the prediction back into a frame.

z_t = E(o_t),\quad z_{t+1} \sim p_\theta(z_{t+1} \mid z_{\le t},\, a_{\le t}),\quad \hat o_{t+1} = D(z_{t+1})

Dynamics. The heart of the model: predicts the next latent from the current one, memory and the action. Recurrent, transformer, diffusion or flow.

SSM / RSSM →Transformers →Diffusion & flow →

Click a block. Every approach in this guide changes one or more of these parts.

Family A

How the world is represented

1. Recurrent & State-Space Models

Compress the past into a fixed-size state and update it every step.

The oldest idea in world modelling: keep a hidden state h that summarizes everything seen so far, and update it with each new observation and action. The cost per step is constant no matter how long you play.

Recurrent state-space models (RSSM), used by PlaNet and Dreamer, split the state into a deterministic memory and a stochastic part that captures what cannot be predicted. Structured SSMs such as S4 and Mamba make the recurrence linear, so it can be trained in parallel and remember over very long sequences.

h_t = \bar{A}\,h_{t-1} + \bar{B}\,x_t,\qquad y_t = C\,h_t

A linear state-space layer. The eigenvalues of Ā decide how slowly old inputs fade.

h_t = f_\theta(h_{t-1}, z_{t-1}, a_{t-1}),\qquad z_t \sim p_\theta(z_t \mid h_t)

Dreamer's RSSM: a deterministic memory h plus a stochastic latent z, both driven by the action a.

SSM decay a = 0.950 (half-life 13.5 frames)window size k = 20 frames (cost per frame grows with k)

An SSM remembers everything a little, at constant cost. A window remembers the last k frames perfectly and everything else not at all. Many modern world models combine both.

Strengths

Constant cost per frame, so cheap to run in real time
Natural fit for long play sessions
Fast imagination rollouts for training agents

Limits

Everything must squeeze through one fixed-size state
Memory fades smoothly; exact recall of an old detail is hard
On its own, weak at photoreal frame generation

For games A strong backbone for long sessions: an SSM can carry the gist of the last ten minutes at the same cost per frame as the last second. Often paired with attention over recent frames for sharp short-term detail.

World Models (2018) · VAE + MDN-RNN; agent trained inside its own dreamPlaNet / Dreamer · RSSM latent dynamics, policy learned in imaginationDreamerV3 · One recipe across domains, including Minecraft diamonds from scratchS4, Mamba · Structured SSM backbones for long sequences

2. Self-supervised Representation (JEPA)

Predict what happens next in an embedding space, not in pixels.

Most of a frame is unpredictable detail: leaf textures, sparks, noise. A model trained to predict pixels spends its capacity on that detail and blurs whatever it cannot know. Joint-embedding predictive architectures (JEPA) instead encode both the present and the future, and predict the future's embedding.

The catch is collapse: if the encoder maps everything to the same vector, prediction is trivially perfect. JEPA-style models prevent this with a slowly updated target encoder, stop-gradients or variance regularization.

\mathcal{L} = \big\lVert\, P_\phi\big(E_\theta(x_t),\, a_t\big) - \operatorname{sg}\big(\bar{E}(x_{t+1})\big) \big\rVert_2^2

Predict the next observation's embedding. sg is stop-gradient; Ē is a slow (EMA) copy of the encoder that prevents collapse.

loss 0.069

Predicting pixels: the target includes texture no model can foresee. The best prediction is the trend, yet the loss stays high and most of the gradient chases noise.

Strengths

Capacity goes to what is predictable and useful
Cheap to plan with: compare embeddings, not images
Learns from unlabeled video

Limits

Does not render frames on its own
Collapse must be actively prevented
Hard to inspect what the embedding has kept

For games Less a renderer than a brain: good for agents and NPCs that need to anticipate the world, or as the state space that a separate decoder renders.

I-JEPA, V-JEPA · Self-supervised image and video encodersV-JEPA 2 · Video model whose latent space is used for robot planningDINO-WM · Plans in frozen DINOv2 feature space

3. Latent-Variable & Adversarial Models

Represent uncertainty explicitly, so the model can pick one future instead of averaging all of them.

When the future branches (the player might turn left or right), a model trained with squared error predicts the average of the branches: a blurry ghost, or a path straight into the wall. Latent-variable models add a random code z. Sample z, and you get one coherent future.

VAEs learn z with an encoder and a likelihood bound; GANs skip the likelihood and train a generator against a discriminator that rejects blurry or implausible frames. VQ-VAEs, which make z discrete, became the tokenizers for the token models in Family B.

\mathcal{L}_{\text{ELBO}} = \mathbb{E}_{q_\phi(z\mid x)}\!\left[\log p_\theta(x \mid z)\right] - \beta\, D_{\mathrm{KL}}\!\left(q_\phi(z\mid x)\,\Vert\,p(z)\right)

VAE: reconstruct well, while keeping the latent code close to a simple prior you can sample from.

\min_G \max_D\; \mathbb{E}_{x}\left[\log D(x)\right] + \mathbb{E}_{z}\left[\log\left(1 - D(G(z))\right)\right]

GAN: the generator wins by producing frames the discriminator cannot tell from real ones.

probability the player turns left: 50%

Deterministic model (squared error): predicts the average future. Here that means driving into the wall. In pixel space the same averaging shows up as two faint ghosts.

Latent-variable model: sample z and commit to one branch. Left 0, right 0.

When the future branches, minimizing squared error picks the mean of the branches, which may be impossible. Sampling a latent code picks one plausible branch at a time.

Strengths

Commits to one sharp future instead of a blur
Fast: one forward pass per frame
Discrete codes (VQ) give compact tokens

Limits

VAEs alone tend toward blur
GANs are unstable to train and can drop modes
Largely superseded for pixels by diffusion

For games The fork in the road is everywhere in games: enemy choices, loot drops, the player's own intent. Any world model for games needs some way to sample a branch rather than average them.

World Models (2018) · VAE compresses frames into zGameGAN (2020) · GAN with a memory module that emulates Pac-ManDreamer · Stochastic latents inside the RSSMVQ-VAE / VQGAN · Discrete tokenizers used by token models

Family B

How the next frame is generated

4. Autoregressive & Masked Token Models

Turn frames into tokens and let a transformer write the world like text.

A tokenizer (usually a VQ-VAE) turns each frame into a grid of discrete tokens. A transformer then predicts the next frame's tokens, conditioned on past frames and actions, exactly like a language model predicts words.

Predicting tokens one at a time is faithful but slow: a frame of a few hundred tokens needs a few hundred forward passes. Masked models (MaskGIT) predict all tokens in parallel and keep the confident ones, finishing a frame in a handful of passes.

p(x_{t+1} \mid x_{\le t}, a_{\le t}) = \prod_{i=1}^{N} p_\theta\!\left(x_{t+1}^{(i)} \mid x_{t+1}^{(<i)},\, x_{\le t},\, a_{\le t}\right)

Autoregressive factorization over the N tokens of the next frame.

Autoregressive (raster order) · 0 forward passes

Masked (MaskGIT-style) · 0 forward passes

Both models generate the same 64-token frame, with one forward pass per tick. Autoregressive decoding needs one pass per token; masked decoding fills many tokens per pass on a cosine schedule. Real frames have hundreds of tokens, which is why masked decoding matters for real-time play.

Strengths

Inherits the whole LLM toolkit: scaling, context, sampling
Actions are just more tokens
Discrete sampling handles branching futures naturally

Limits

Sequential decoding is slow
Tokenizer caps visual fidelity
Errors compound over long rollouts

For games Muse (WHAM) treats controller inputs as tokens alongside image tokens, so the same model can predict what a player will do as well as what they will see.

IRIS · VQ autoencoder + GPT world model for AtariGAIA-1 · Token world model for drivingGenie · Spatiotemporal transformer with MaskGIT decodingMuse / WHAM · Image and controller tokens from Bleeding EdgeWHAMM · MaskGIT-style successor; Quake II at 10+ FPS

5. Diffusion & Flow Matching

Start from noise and denoise toward the next frame, conditioned on history and action.

Diffusion models learn to remove noise; generation runs the process backwards from pure noise to a sharp frame. Flow matching learns a velocity field that carries noise to data along (ideally) straight paths, which need fewer steps.

For games, the frame is conditioned on recent frames and the player's action. Two problems dominate: speed (many denoising steps per frame) and drift (each generated frame feeds the next). GameNGen adds noise to context frames during training so the model learns to correct its own errors; Diffusion Forcing gives each frame its own noise level.

\mathcal{L}_{\text{diff}} = \mathbb{E}_{t,\epsilon}\big\lVert \epsilon - \epsilon_\theta(\alpha_t x + \sigma_t \epsilon,\, t,\, c)\big\rVert^2

Diffusion: predict the noise that was added. c is the conditioning: past frames and the action.

\mathcal{L}_{\text{FM}} = \mathbb{E}\big\lVert v_\theta(x_t, t, c) - (x_1 - x_0)\big\rVert^2,\quad x_t = (1-t)\,x_0 + t\,x_1

Flow matching: regress the velocity of a straight path from noise x₀ to data x₁.

time t = 0.00 (pure noise)sampling steps = 4

Blue: true path. Red: Euler sampler with 4 steps. Mean end-point error: 0.000

Straight paths are integrated exactly in one step. Curved paths need many steps, and every step is a full network evaluation, per frame.

Noise (Gaussian) is transported to data (a ring). Paths are shown per sample pair. A trained model learns an averaged velocity field whose paths are only approximately straight, which is why few-step distillation and rectified flows exist.

Strengths

Best visual fidelity today
Samples sharp, diverse futures
Stable to train at scale

Limits

Many steps per frame unless distilled
Drift over long rollouts
Memory limited to the conditioning window

For games The current workhorse of playable neural games. Real-time play depends on cutting steps: distillation, few-step flow models and small context windows.

DIAMOND · Diffusion world model for Atari, also CS:GOGameNGen · DOOM at 20 FPS on one TPUOasis · Diffusion transformer, playable Minecraft-like worldDiffusion Forcing · Per-frame noise levels for stable rolloutsMatrix-Game 2.0 · Open-source real-time interactive world model

6. Latent Action Models

Learn the controls from video that never recorded them.

Most gameplay video online has no controller log. A latent action model infers the action from the change between two consecutive frames, and squeezes it through a tiny discrete codebook. Because the bottleneck is so small, the only thing it can encode is what the player controls.

At play time, the human picks a latent action and the dynamics model renders its effect. This is how Genie learned a playable controller from unlabeled platformer videos.

\tilde a_t = \mathrm{VQ}\big(g_\psi(x_t, x_{t+1})\big),\qquad \hat x_{t+1} = D_\theta(x_{\le t}, \tilde a_t)

Infer a discrete action from the frame change, then predict the next frame from that action.

Strengths

Unlocks internet-scale video as training data
Actions emerge without labels
Same controls transfer across similar games

Limits

Learned actions may not match real buttons
Confuses player actions with other changes
Small codebooks limit expressiveness

For games The answer to the data problem: action-labelled gameplay is scarce, while gameplay video is nearly unlimited.

Genie · Learned a small set of discrete latent actions from unlabeled platformer videoLAPO · Learning policies from observation-only data via latent actions

Family C

Spatial world models

7. Spatial World Models I: 3D Scenes

Store the world as explicit 3D geometry, then render any view of it.

Instead of predicting frames, reconstruct or generate the scene itself: a neural radiance field (NeRF) or a cloud of 3D Gaussians. Any camera view is rendered by blending the primitives along each pixel's ray.

Because the world exists independently of the camera, it persists: turn around and the room behind you is exactly where you left it. The trade-off is that these scenes are static unless something makes them move.

C(\mathbf{r}) = \sum_{i} c_i\, \alpha_i \prod_{j<i} (1 - \alpha_j)

A pixel's color is the front-to-back blend of the primitives its camera ray passes through.

Strengths

Persistence and 3D consistency for free
Renders at high frame rates (Gaussian splatting)
Exports to standard engines and headsets

Limits

Static: no dynamics or interaction by default
Generation is slower than playing
Hard to represent fluids, smoke and deformables

For games A content tool more than a game engine: generate a persistent level, then run it in a conventional engine with conventional physics.

NeRF · Neural radiance fields3D Gaussian Splatting · Explicit Gaussians, real-time renderingMarble (World Labs) · Persistent, exportable 3D worlds from text, images or video

8. Spatial World Models II: 4D Dynamics & Interaction

Make the 3D world move, respond to actions and remember where everything is.

4D models add time: each primitive follows a learned deformation, or is moved by a physics simulator. Interaction adds actions: the scene changes because the player pushed, opened or broke something.

A second line of work keeps video generation but adds spatial memory: frames are stored with their camera poses, and the model retrieves the frames that saw the place you are looking at now. That is what lets a world stay consistent when you walk away and come back.

\mu_i(t) = \mu_i + \Delta_\theta(\mu_i,\, t,\, a_{\le t})

Each primitive's position becomes a function of time and the player's actions.

Frame-window model · world changed 0×

🧍

Spatial memory · world changed 0×

🧍

context window k = 6 steps

Walk around (click the board, then use arrow keys) and come back to where you started. The frame-window model only remembers cells seen in the last k steps; when a forgotten cell comes back into view it is regenerated, and may come back different (red outline). The spatial model stores what it saw by position, so the world stays put.

Strengths

Consistency when revisiting places
Interaction grounded in geometry
Bridges generated worlds and physics

Limits

Expensive to train and run
Physics of contact, fluids and destruction is hard
Young field: few real-time systems

For games This is where memory gets solved. A world model that knows where things are in space can let the player leave a room and find it unchanged.

4D / Dynamic Gaussians · Gaussians that move over timePhysGaussian · Physics simulation applied to Gaussian scenesWorldMem · Memory bank of frames with poses and timestampsWorldPlay · Real-time generation with long-term geometric consistencyNVIDIA Cosmos · World foundation models for robotics and driving

Family D

Using and grounding world models

9. Planning in Imagination

Use the world model as a simulator for an agent to think in.

A world model does not need to render anything to be useful. MuZero learns a model that predicts only what matters for planning (reward, value, policy) and searches over it. Dreamer trains its policy entirely on imagined rollouts.

For games this cuts both ways: the same model that renders the world for a player can train the agents that live in it.

a^\star_{t} = \arg\max_{a_{t:t+H}} \; \mathbb{E}_{\hat s \sim p_\theta}\Big[\textstyle\sum_{k=0}^{H} \gamma^k\, \hat r(\hat s_{t+k}, a_{t+k})\Big]

Pick the action sequence with the best predicted return over a horizon H, inside the model.

Strengths

Huge sample efficiency for agents
Models only what decisions need
Natural route to smarter NPCs

Limits

Agents exploit model errors
Short horizons because of compounding error
Value-only models cannot be played by humans

For games NPCs and testing bots that rehearse inside the world model; DeepMind has used Genie 3 worlds as environments for its SIMA agent.

MuZero · Planning with a learned value-equivalent modelDreamer V1–V3 · Actor-critic trained in imaginationTD-MPC2 · Latent model-predictive control

10. Hybrid: Symbolic State + Neural Rendering

Keep the rules exact, let the network paint the pixels.

Pure neural engines struggle with what games need most: exact rules, inventories, scores and saves. A hybrid keeps an explicit game state, updated by code or by a language model, and uses a neural model only to render or embellish it.

The rules stay debuggable and deterministic; the look of the world becomes generative.

s_{t+1} = f_{\text{code}}(s_t, a_t),\qquad o_t = R_\theta(s_t,\, \text{style})

Exact transition in code; learned renderer from state to pixels.

Strengths

Exact persistence, rules and saves
Designers can edit the logic
Generation where it adds the most value

Limits

Loses some of the open-endedness
Renderer must stay consistent with state
Two systems to build and keep in sync

For games The most likely path to shipping games in the near term. Part III of the essay argues this in detail.

Neural rendering on G-buffers · Learned realism over a conventional engine's outputLLM text-world simulators · Language models as transition functions for text gamesCode world models · LLMs that write the transition function as a program

11. Evaluating World Models

Looking right is not the same as being a good game.

Image metrics (FID, FVD, PSNR, LPIPS) measure how a frame looks. A game needs more: the action must do what the player expects, the world must be the same when you return, and it must stay stable for minutes, not seconds.

Useful tests therefore check action-following, revisit consistency and long-horizon stability, alongside human judgement. GameNGen's human study, where raters were barely better than chance at telling real DOOM from generated, is one example.

\text{consistency} = \frac{1}{|V|}\sum_{v\in V} d\big(o_{\text{first}}(v),\; o_{\text{return}}(v)\big)

One revisit test: compare what a place looked like the first time with when you return to it.

Strengths

Turns 'feels right' into numbers
Exposes drift and forgetting
Lets approaches be compared fairly

Limits

Pixel metrics miss game logic
Human studies are slow and costly
Benchmarks are new and still changing

For games Playtesting does not go away; it gets new instruments.

FID / FVD / LPIPS · Visual quality metricsHuman rater studies · Real vs generated discriminationMIND · Benchmark for memory consistency and action control

Compare approaches

No single family wins on everything. Pick what your game needs and see which families fit, then read the trade-offs above.

What does your game need most?

Family	Real-time speed	Long-term memory	Diverse futures	Visual fidelity	Controllability	Training stability	Fit
Spatial I (3D)	●●●	●●●	●●○	●●●	●○○	●●○	6
Recurrent & SSM	●●●	●●○	●○○	●○○	●●○	●●●	5
JEPA	●●●	●●○	●○○	●○○	●●○	●●○	5
Hybrid symbolic	●●○	●●●	●●○	●●○	●●●	●●○	5
VAE / GAN	●●●	●○○	●●○	●●○	●●○	●○○	4
Spatial II (4D)	●○○	●●●	●●○	●●○	●●○	●○○	4
Token models	●○○	●●○	●●●	●●○	●●●	●●●	3
Diffusion & flow	●●○	●○○	●●●	●●●	●●○	●●●	3

Editorial ratings for each family on its own, not benchmark results. Shipping systems combine several families: for example, a diffusion renderer on a token backbone with spatial memory.