SauerkrautLM-Doom-MultiVec: Real-Time FPS Control

• The paper demonstrates that a 1.3M parameter transformer using ASCII and depth encoding secures 17.8 frags per episode at 31 ms latency in DOOM.
• It employs a dual representation by converting the game screen to a 40×25 ASCII grid and a quantized depth map to preserve both spatial layout and approximate 3D distance.
• Using attention pooling and supervised imitation from human gameplay, the model outperforms larger LLMs, showcasing the benefits of a domain-specific, low-latency design.

SauerkrautLM-Doom-MultiVec is a task-specialized transformer for real-time control in the classic first-person shooter DOOM. It is described as a 1.3 million parameter model that maps a textual 2D ASCII rendering of the game screen and an aligned quantized depth map to a distribution over four discrete actions—shoot, move_forward, turn_left, and turn_right—and is evaluated in the VizDoom defend_the_center scenario (Golchinfar et al., 8 Apr 2026). The model is notable because, within that benchmark, it outperforms much larger LLMs, including Nemotron-120B, Qwen3.5-27B, GPT-4o-mini, and Gemini Flash Lite, while operating at approximately 31 ms per decision on CPU and maintaining native real-time control constraints (Golchinfar et al., 8 Apr 2026). The work positions SauerkrautLM-Doom-MultiVec as evidence that small, domain-specific models trained on task-appropriate data and representations can dominate general-purpose LLMs in latency-sensitive control regimes (Golchinfar et al., 8 Apr 2026).

1. Benchmark setting and operational definition

SauerkrautLM-Doom-MultiVec is evaluated in VizDoom, a research platform derived from Doom and widely used for vision-based reinforcement learning and FPS-agent benchmarking (Wydmuch et al., 2018). The specific scenario is defend_the_center, in which the player stands in a circular arena while enemies spawn around and approach from all directions; the objective is to survive and accumulate frags, with VizDoom returning a reward of +1 for each frag (Golchinfar et al., 8 Apr 2026). Each episode lasts up to 2,100 game tics, corresponding to about 60 seconds of simulated time, and both data collection and evaluation use frame_skip = 4, so the agent acts every four tics, or approximately 114 ms of game time per action (Golchinfar et al., 8 Apr 2026).

The system’s input modality is deliberately compact. A 640×480 RGB frame is converted to grayscale, downscaled to 40×25 through block averaging, and then mapped to one of ten brightness characters, " .:-=+*#%@", producing a 1,024-character serialized grid with newline separators (Golchinfar et al., 8 Apr 2026). In parallel, the per-pixel depth buffer supplied by VizDoom is downscaled to 40×25, normalized to $[0,1]$, and quantized into 16 bins; a 17th “no-depth” bin is reserved for special tokens such as [CLS] and [PAD] (Golchinfar et al., 8 Apr 2026). This yields what the paper describes as a dual representation that preserves 2D spatial layout and approximate 3D distance while remaining textual at the token level (Golchinfar et al., 8 Apr 2026).

The benchmark is explicitly constructed to compare agents receiving equivalent input. All agents, including the tested LLMs, are given ASCII frames and depth maps rather than raw images (Golchinfar et al., 8 Apr 2026). A plausible implication is that the comparison isolates representation handling, spatial parsing, and latency more directly than a multimodal image-input comparison would, although the paper also lists raw-image LLM evaluation as future work rather than reporting such results (Golchinfar et al., 8 Apr 2026).

2. Input encoding, tokenization, and action formulation

The model uses a strict character-level tokenizer rather than a subword tokenizer. The vocabulary contains 75 tokens, including the ten brightness symbols, additional entity markers, lowercase letters, digits, punctuation, six action-specific special tokens used in experiments, and standard special tokens such as [PAD] and [CLS] (Golchinfar et al., 8 Apr 2026). The central design choice is that each character in the 40×25 ASCII grid corresponds to exactly one token, preserving a one-to-one mapping between screen location and sequence position and maintaining the 2D spatial structure through row ordering and newline characters (Golchinfar et al., 8 Apr 2026).

Depth is encoded per character position. Each aligned depth value is normalized, quantized into one of 16 bins, and mapped to a learned depth embedding (Golchinfar et al., 8 Apr 2026). The paper states that the bins provide about 40 cm resolution at typical DOOM engagement ranges, sufficient to distinguish “melee range” from “across the room” (Golchinfar et al., 8 Apr 2026). The resulting token representation therefore fuses symbolic brightness and discretized distance.

The “MultiVec” designation refers to the fact that the encoder retains multiple vectors per frame—one per token—rather than collapsing the scene immediately into a single latent (Golchinfar et al., 8 Apr 2026). For token $i$, the input representation is the sum of character and depth embeddings,

$$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$

with positional information supplied through Rotary Position Embeddings within attention rather than by adding a separate positional vector (Golchinfar et al., 8 Apr 2026). The paper presents this as a scene representation that preserves per-token information until the final pooling stage.

The action space is deliberately narrow: $K=4$, corresponding to shoot, move_forward, turn_left, and turn_right (Golchinfar et al., 8 Apr 2026). These actions match the controls present in the human demonstration data, which did not include strafe keys (Golchinfar et al., 8 Apr 2026). At inference time, the model outputs a four-dimensional probability distribution over these actions, and a composite action strategy allows multiple compatible buttons to be activated simultaneously, approximating human multi-key input (Golchinfar et al., 8 Apr 2026). This is important because the training labels themselves are soft and allow multiple actions to be simultaneously appropriate.

3. Architecture and representation bottleneck

SauerkrautLM-Doom-MultiVec is built around a compact ModernBERT-based encoder with hash embeddings, depth-aware token representations, and an attention-pooling classification head (Golchinfar et al., 8 Apr 2026). The total parameter count is about 1.319M, with approximately 4,480 parameters in the hash embeddings, 2,176 in the depth embeddings, about 1,312,000 in the transformer layers, and 644 in the attention pooling and classifier (Golchinfar et al., 8 Apr 2026).

The backbone is a five-layer ModernBERT variant with hidden size 128 and sequence length of approximately 1,024 tokens (Golchinfar et al., 8 Apr 2026). Its attention pattern alternates between global and local attention: layers 0 and 3 use global attention over the full sequence, while layers 1, 2, and 4 use local attention with a sliding window of 128 tokens (Golchinfar et al., 8 Apr 2026). Rotary position embeddings are used as in ModernBERT (Golchinfar et al., 8 Apr 2026). The architectural rationale given in the paper is that local attention captures spatial neighborhoods in the ASCII grid, whereas global layers propagate information across the entire field of view (Golchinfar et al., 8 Apr 2026).

Instead of a standard $V \times H$ embedding matrix, the model uses a two-stage hash embedding parameterization with vocabulary size $V \approx 75$, hidden dimension $H=128$, and projection dimension $P=16 \ll H$ (Golchinfar et al., 8 Apr 2026). The mapping is written as

$\text{HashEmb}: \{1,\dots,V\} \to \mathbb{R}^H$

and implemented as

$$\mathbf{e}^{\text{char}}_i = \mathrm{LN}\left(\mathbf{E}_2^\top \,\mathbf{E}_1[\text{token}_i]\right),$$

with $i$0 and $i$1 (Golchinfar et al., 8 Apr 2026). The paper notes that the absolute savings are modest for such a small vocabulary—4,480 parameters instead of 16,384—but treats the design as a demonstration of a scalable compression pattern (Golchinfar et al., 8 Apr 2026).

For depth-aware token representations, the depth embedding table is defined as

$i$2

with bin 16 representing “no depth” (Golchinfar et al., 8 Apr 2026). The final input embedding for each token is the sum of character and depth embeddings, allowing the model to distinguish visually similar symbols at different distances (Golchinfar et al., 8 Apr 2026).

The classifier does not use a [CLS] token or mean pooling. Instead, it uses learned attention pooling. If the encoder outputs

$i$3

then the pooling weights are

$i$4

the pooled vector is

$i$5

and the classifier produces logits

$i$6

followed by

$i$7

The paper interprets this as an information bottleneck from a 1024×128 multi-vector scene representation to a single 128-dimensional decision vector (Golchinfar et al., 8 Apr 2026).

A notable part of the architecture’s development is the rejection of more explicit late-interaction mechanisms. Earlier ColBERT-style MaxSim formulations using textual action queries collapsed to a single dominant action, and learned prototype-vector approaches using MaxSim or cross-attention also collapsed, which the authors attribute to overparameterization relative to the four-class, 31k-example setting (Golchinfar et al., 8 Apr 2026). Attention pooling was adopted as the final design because it gave stable training and the best validation and gameplay performance (Golchinfar et al., 8 Apr 2026). This directly counters the assumption that preserving token-level vectors to the very end necessarily improves a small classifier; here, a strong bottleneck was empirically required.

4. Training data, supervision, and optimization

Training uses human gameplay demonstrations collected in the same defend_the_center scenario (Golchinfar et al., 8 Apr 2026). The first author played using native DOOM controls, while VizDoom spectator mode recorded RGB frames, depth buffers, and key presses (Golchinfar et al., 8 Apr 2026). The dataset consists of four recording sessions totaling approximately two hours of gameplay, more than 80 episodes, and 31,645 frames sampled with frame_skip = 4 (Golchinfar et al., 8 Apr 2026). Each frame is paired with a 40×25 ASCII rendering, 16-level quantized depth per token, and action labels restricted to the same four actions used at inference time (Golchinfar et al., 8 Apr 2026).

The supervision signal is not one-hot. Human key presses are converted into four-dimensional soft score vectors $i$8 so that pressed actions receive high scores and non-pressed actions receive a baseline, while allowing multiple actions to be high simultaneously (Golchinfar et al., 8 Apr 2026). This is intended to reflect the fact that multiple actions can be appropriate in a single frame and to preserve “action affinities” such as move_forward and shoot co-occurring (Golchinfar et al., 8 Apr 2026). The paper frames this as a way to capture composite human control more faithfully than hard labels would.

Because labels are soft distributions, the training objective is KL divergence between the model’s softmax output and the soft teacher scores: $i$9 The stated reason for choosing KL over standard cross-entropy is that KL preserves gradient signal even for “correct” predictions when the target assigns non-zero mass to non-maximal actions (Golchinfar et al., 8 Apr 2026).

The final training configuration uses a 90/10 train/validation split, corresponding to 28,480 training frames and 3,165 validation frames, AdamW, an initial learning rate of $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$0, cosine decay with 500 warmup steps, batch size 32, 50 epochs, best-checkpoint selection by validation accuracy, and bfloat16 mixed precision on a single NVIDIA RTX 6000 Ada GPU (Golchinfar et al., 8 Apr 2026). The model reaches 57.7% top-1 validation accuracy on the four-way classification task, compared with a 25% random baseline (Golchinfar et al., 8 Apr 2026). The paper explicitly notes that this apparently modest validation accuracy is compatible with strong gameplay because the task is inherently ambiguous and because near-miss action substitutions can still be behaviorally reasonable (Golchinfar et al., 8 Apr 2026).

A common misconception in this setting is that strong policy performance requires reinforcement learning from environment interaction. In this case, the reported system is trained through supervised imitation learning on approximately 31k human-labeled frames and does not require reinforcement learning during training (Golchinfar et al., 8 Apr 2026). This does not imply that RL is unnecessary for DOOM in general; rather, it shows that under this benchmark, carefully structured imitation learning can be sufficient.

5. Inference policy, evaluation, and comparison with LLMs

At deployment, each frame is converted to 40×25 ASCII and depth bins, tokenized at the character level, encoded by the ModernBERT-Hash model, attention pooled, and classified to produce action probabilities $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$1 (Golchinfar et al., 8 Apr 2026). The inference procedure then applies a composite action rule. Let $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$2 be the top action; initialize the button set from $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$3. If $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$4 and

$$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$5

then shoot is added. If the second-highest probability action $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$6 satisfies $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$7 and is compatible with $$\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),$$8, it is also added (Golchinfar et al., 8 Apr 2026). This can yield outputs such as move_forward + turn_left + shoot (Golchinfar et al., 8 Apr 2026).

The main benchmark compares SauerkrautLM-Doom-MultiVec with GPT-4o-mini, Gemini Flash Lite, Qwen3.5-27B, and Nemotron-120B under the same environment configuration used in training: 640×480 resolution, HUD enabled, frame_skip = 4, and real-time pacing enforced by sleep() so that the environment runs in true real time (Golchinfar et al., 8 Apr 2026). Metrics include number of episodes, average survival, maximum survival, cumulative frags, and wall-clock latency per decision (Golchinfar et al., 8 Apr 2026). The LLMs are given text-only input: a system prompt describing the ASCII and depth encoding and a per-frame user message containing a fenced 40×25 View: block and a 40×25 Depth: block of digits (Golchinfar et al., 8 Apr 2026). They must return a valid action or action pair without explanation (Golchinfar et al., 8 Apr 2026).

Agent	Params	Frags / Latency
SLM-Doom-MV-1.3M	1.3M	178 / 31 ms
GPT-4o-mini	–	0 / 646 ms
Gemini Flash Lite	–	8 / 920 ms
Qwen3.5-27B	27B	2 / 13.3 s
Nemotron-120B	120B	3 / 8.9 s

In the main benchmark, the small model scores 178 frags in 10 episodes, or 17.8 per episode, while all tested LLMs combined achieve 13 frags total (Golchinfar et al., 8 Apr 2026). Its average survival is 388 steps, with a maximum of 525, corresponding to full 60-second survival, and its latency is 31 ms per decision on CPU (Golchinfar et al., 8 Apr 2026). By contrast, GPT-4o-mini records the highest average survival among the LLMs at 104 steps but achieves 0 frags, while the large open-weight models are seconds per frame and therefore severely out of sync with the environment (Golchinfar et al., 8 Apr 2026).

The paper also reports a “fair-mode” benchmark with frame_skip = 20, corresponding to one action every 20 tics, or about 571 ms, to reduce the penalty imposed by LLM latency (Golchinfar et al., 8 Apr 2026). Even there, SauerkrautLM-Doom-MultiVec remains best overall in both survival and frags, scoring 10 frags across 10 episodes with approximately 29 ms latency, while GPT-4o-mini again records 0 frags (Golchinfar et al., 8 Apr 2026). The reported conclusion is therefore not only that the small model is faster, but that it remains behaviorally stronger even when the benchmark is slowed to be more favorable to the slower systems (Golchinfar et al., 8 Apr 2026).

A central misconception addressed by these results is that scale alone should compensate for poor task match. Within this benchmark, Nemotron-120B is 92,300× larger than the 1.3M model yet scores only 3 frags across 5 episodes, and Qwen3.5-27B is approximately 20,800× larger yet scores 2 frags across 3 episodes (Golchinfar et al., 8 Apr 2026). The reported evidence therefore contradicts any simple parameter-count-based expectation of superiority for low-level real-time control.

6. Behavioral interpretation, prior context, and limitations

Qualitatively, SauerkrautLM-Doom-MultiVec is described as the only evaluated agent that reliably engages enemies rather than merely spinning or evading (Golchinfar et al., 8 Apr 2026). It uses combined ASCII and depth cues for target acquisition, rotates toward threats, fires when enemies are near the center and close in depth, and often survives the full 60 seconds while accumulating frags (Golchinfar et al., 8 Apr 2026). Approximately 62% of its actions are composite, such as turn_left+shoot or move_forward+turn_right, producing fluid behavior closer to human multi-key control (Golchinfar et al., 8 Apr 2026).

The LLMs display a different behavioral profile. GPT-4o-mini survives longer than the other LLMs but is heavily skewed toward rotation actions and never secures a frag in the main benchmark (Golchinfar et al., 8 Apr 2026). Gemini Flash Lite achieves the highest frag count among the LLMs, with 8 in the main benchmark, apparently benefiting from lower latency than the large open-weight models (Golchinfar et al., 8 Apr 2026). Qwen3.5-27B and Nemotron-120B are reported to be too slow for effective low-level control, and the paper also argues that their BPE tokenizers disrupt the 2D grid structure of ASCII input, while their prompt-by-prompt operation lacks persistent temporal state across frames (Golchinfar et al., 8 Apr 2026). This suggests that their weakness is not reducible to one factor; latency, tokenizer mismatch, and lack of state all contribute.

The work sits within a broader DOOM and VizDoom tradition in which earlier agents typically relied on deep RL with CNN-based perceptual stacks and often required millions of interaction steps (Wydmuch et al., 2018). The VizDoom competition literature emphasizes the difficulty of navigation, opponent handling, survival, and generalization in FPS settings, and notes that even strong RL bots remained significantly below skilled human performance (Wydmuch et al., 2018). Against that background, SauerkrautLM-Doom-MultiVec differs by using imitation learning from human demonstrations, an ASCII-plus-depth representation instead of raw pixels, and a transformer encoder at a very small parameter scale (Golchinfar et al., 8 Apr 2026). A plausible implication is that the model’s contribution is less about solving general DOOM and more about identifying a narrow but effective design point for fast reactive control.

The paper also records several ablation-like design judgments. ColBERT-style MaxSim and learned action prototypes collapsed; attention pooling produced stable training and best performance; preliminary GPT-4o-mini experiments without depth averaged only about 54 survival steps; and hash embeddings reduced embedding parameters by 73% compared to a standard 128×128 matrix (Golchinfar et al., 8 Apr 2026). These observations support the authors’ argument that attention pooling and depth-fused character-level embeddings are central to the system’s effectiveness (Golchinfar et al., 8 Apr 2026).

The reported limitations are substantial. Training and evaluation are confined to a single scenario, defend_the_center, so generalization to corridors, multi-room maps, outdoor scenes, or different enemy types remains untested (Golchinfar et al., 8 Apr 2026). The model depends on matched game settings such as resolution and HUD presence; changes in those settings alter the ASCII pattern and may degrade performance unless the model is retrained or adapted (Golchinfar et al., 8 Apr 2026). The demonstration corpus comes from one skilled human over approximately two hours, so the policy inherits that player’s style and biases (Golchinfar et al., 8 Apr 2026). The method is frame-wise classification without explicit temporal memory, and while this works in the benchmark, more complex tasks may require recurrent or memory-augmented designs (Golchinfar et al., 8 Apr 2026). Finally, the LLM comparison is constrained by the lower episode counts feasible for the slowest models, which the paper presents as sufficient to show a trend but not as a definitive statistical account (Golchinfar et al., 8 Apr 2026).

Taken together, the evidence presented in the benchmark supports a narrow but clear conclusion: in real-time DOOM control under ASCII-plus-depth input, small task-specific transformers can outperform vastly larger general-purpose LLMs when the representation, supervision, and inference policy are tightly aligned with the control problem (Golchinfar et al., 8 Apr 2026). It does not establish that such models dominate across all embodied tasks or all game settings. It does establish that, for this latency-sensitive FPS benchmark, architectural fit and operational constraints matter more than raw scale.