Teaching GLiNER2.5 to play Doom: zero-shot first, then fine-tuned

The ModernBERT policy needed 372 labelled Doom examples to learn its seven button scores. GLiNER2.5 is built to classify text against labels supplied at run time, so it can be handed descriptions of Doom actions and asked which apply, with no Doom training at all. Could it play? And if a little training helped, how little?

Each pipeline scored the same 516 evaluation pairs and played six 20-game-second headless runs: three instructions (attack, evade, cautious) on two seeds, 2,100 live decisions per pipeline.

Classification covers building the text, tokenizing, the complete encoder, moving scores to the CPU and decoding; loading and ten warm-up calls are excluded. The headless loop includes the game, logs and screenshots, and is total decisions over total wall time. All averages stayed above the 17.5 decisions per second that two tics per decision need at native game speed, although one adapted-Base cautious run fell below it.

On the CPU (float32, four threads), Small averaged 36.6 ms and Base 74.9 ms. CPU and Metal scores differed by at most 0.00062 and 0.00223, with identical first decisions, so the weaknesses below are not a float16 artefact.

On an identical healthy state with ammo and an enemy lined up, zero-shot Base's ATTACK score fell from 0.976 for “Attack enemies on sight.” to 0.022 for “Do not shoot. Evade the enemies.” Its evasion runs used no ammo, and it reacted to where enemies were, although it chose right turns inconsistently.

Kills and deaths are not a full measure of skill, and these are short runs. Zero-shot Base scored more cautious kills than ModernBERT, but that does not show reliable conditional behaviour; the probes in chapter 5 test that directly.

Both GLiNER sizes were tried with 32, 128 and 372 training examples, keeping the encoder frozen and adapting only the existing classification layers, 600 steps each with the same settings. The smallest budget reaching the best validation agreement went on to gameplay: 128 for Small and 32 for Base. The original whole-state splits were kept, and the test sets chose nothing.

Head optimisation took 0.75–1.19 seconds per Small budget and 1.79–2.95 seconds per Base budget, plus 8.46 and 13.89 seconds of shared feature extraction. Both adapted heads moved and survived both evasion runs without firing.

“Shoot at enemies, but retreat when health falls below 40” never forbids firing while retreating. To remove that ambiguity, a follow-up tested the explicit known phrasing “Fight, but stop shooting and retreat below 40 health” and one new phrasing, at health 20, 39, 40 and 80, with no model or threshold changed.

The health categories already split at 40, so these probes test the vocabulary, not general numerical reasoning. Neither approach establishes arbitrary instruction following.

Three things limited the first attempt, and none of them was GLiNER itself.

The scripted teacher can label any state, so the second attempt generated them (teacher_data.py). Each sample draws a health value from 1 to 100, half of them near the 20 and 40 boundaries; ammunition from 0 to 50, empty a quarter of the time; and an enemy that is absent or to the left, ahead or right, at a random bearing and distance within its range category. The teacher labels every state for all three behaviours, and each label is paired with one of five or six training phrasings: 4,000 states, 12,000 examples, plus 1,200 for validation.

Training runs through GLiNER's own scoring path, the same calls as inference without inference mode, so the model is trained on exactly the text and labels it scores at play time; a check confirmed the two paths give identical logits. Three epochs, batch 32, AdamW, bfloat16; the checkpoint is chosen on the held-out validation states. To separate the effect of better data from the effect of fine-tuning, the same data also trained only the classification layers with the encoder frozen.

Both fine-tuned models passed all 12 behaviour checks, and on the explicit conditional probes, “Fight, but stop shooting and retreat below 40 health” and the never-trained “Fire only when your health is at least 40. If lower, do not fire and move backward.”, they held fire and backed away at health 20 and 39 and fired at 40 and 80. Speed is unchanged by fine-tuning: the same encoder runs on every decision, at about 54 decisions per second headless. ModernBERT, with its 25–36-token inputs against GLiNER's 227–253, is a little faster.

The two fine-tuned sizes played identical games: both reproduce the teacher's decisions in these states, so the same actions led to the same outcomes. For the showcase, fine-tuned Base played a full 60 seconds per instruction with the dashboard on: 20 kills and 1 death attacking, no shots and 1 death evading, 17 kills and 1 death on the cautious instruction. Over 3,150 live decisions its raw button vector matched the teacher's in 2,686; every disagreement was the direction of the sideways strafe while backing away, which the text cannot always decide (an enemy “directly ahead” may be slightly left or right of the crosshair). Firing, turning and moving forward or back always matched.

This is imitation of a hand-written teacher from a text description of the game, so it shows instruction following within that vocabulary, not general Doom skill. The ModernBERT reference was not retrained on the new data, so the comparison shows what each pipeline reached with its own training, not which encoder is better.