A text classifier that plays Doom from instructions and game state

The first piece was the game loop. ViZDoom normally runs in real time, so a slow policy acts on frames that are already out of date. Here the engine waits: observe, decide, advance one tic, repeat. Every decision sees a fresh state, there is no queue of stale actions, and the game runs exactly as fast as the policy allows. Video can still be replayed at normal game speed, or at the measured wall-clock speed to show how fast the policy really is.

The scenario is defend_the_center: the player stands in a round room while demons and chainsaw marines walk in from the walls. Before any model, a random policy pressing one legal button combination per game second set the baseline.

Before training anything, we measured whether a 149.6M-parameter encoder could keep up with the game. The timing covers the whole decision: building the text, tokenizing, the full encoder forward pass, moving scores back to the CPU and decoding the buttons. Model loading and warm-up are excluded.

Float16 became the default. With hardware video encoding on, the whole game loop completed 175 decisions in 3.72 seconds, 47 decisions per second including the first call.

A charcoal dashboard puts the game on the left and the instrumentation on the right: rolling latency, decisions per second, all seven button scores against a 0.5 marker, the legal combinations and why a button was suppressed. Drawing and encoding run in a background worker that keeps only the newest frame, so the model never waits for video. The first recording, still with the untrained head, made 2,100 decisions in 60 game seconds and took 86.5 seconds of wall time: 30.8 ms per decision and 24.3 decisions per second with the window and video on.

For the trained checkpoint, the game state is described with a small fixed vocabulary. Code reads the nearest visible monster's position from the engine and bins it, along with health and ammo, into text such as:

health=healthy ammo=loaded nearest_enemy=center range=medium

These are perception features, not commands: the text never names a button. A scripted teacher with three behaviours (aggressive, cautious, evasive) labelled 468 examples, two known phrasings per behaviour. Whole state combinations were split between training (62 combinations, 372 examples), validation (8, 48) and test (8, 48), so no tested state was seen in training under any instruction.

Only the head was trained: 595,975 of the model's 149,610,247 parameters. The encoder's outputs were cached during training only; during play, every decision runs the full encoder. Training took 3.6 seconds of wall time, and the selected checkpoint is step 150 of 1,200, chosen on validation.

The checkpoint then played with the dashboard and the video recorder on, deciding every two game tics. From an identical starting state, changing only the instruction moved the ATTACK score from 0.979 to 0.026. In the evade run, ammo stayed at 26 and ATTACK never rose above 0.036: the model's own scores, not a rule in the decoder, kept it from firing.

Two tics per decision need 17.5 decisions per second to keep up with the game's native speed; the model ran at more than twice that, so the films above play faster than real time. They were joined into the 51-second edit at the top of this page.

The model reads a hand-built description of the game, not the screen, and the description comes from the engine's own object positions. It was trained on six phrasings of three behaviours and a finite set of states. It follows instructions it was taught, in situations described in its vocabulary. That is enough to show a text classifier steering a game in real time, and not enough to claim it understands Doom or arbitrary language.