---
title: 'SauerkrautLM-Doom-MultiVec: Real-Time FPS Control'
url: https://www.emergentmind.com/topics/sauerkrautlm-doom-multivec
type: topic
---

# SauerkrautLM-Doom-MultiVec: Real-Time FPS Control

SauerkrautLM-Doom-MultiVec is a task-specialized transformer for real-time control in the classic first-person shooter DOOM. It is described as a 1.3 million parameter model that maps a textual 2D ASCII rendering of the game screen and an aligned quantized depth map to a distribution over four discrete actions—`shoot`, `move_forward`, `turn_left`, and `turn_right`—and is evaluated in the VizDoom `defend_the_center` scenario [2604.07385]. The model is notable because, within that benchmark, it outperforms much larger large language models, including Nemotron-120B, Qwen3.5-27B, GPT-4o-mini, and Gemini Flash Lite, while operating at approximately 31 ms per decision on CPU and maintaining native real-time control constraints [2604.07385]. The work positions SauerkrautLM-Doom-MultiVec as evidence that small, domain-specific models trained on task-appropriate data and representations can dominate general-purpose LLMs in latency-sensitive control regimes [2604.07385].

## 1. Benchmark setting and operational definition

SauerkrautLM-Doom-MultiVec is evaluated in VizDoom, a research platform derived from Doom and widely used for vision-based reinforcement learning and FPS-agent benchmarking [1809.03470]. The specific scenario is `defend_the_center`, in which the player stands in a circular arena while enemies spawn around and approach from all directions; the objective is to survive and accumulate frags, with VizDoom returning a reward of +1 for each frag [2604.07385]. Each episode lasts up to 2,100 game tics, corresponding to about 60 seconds of simulated time, and both data collection and evaluation use `frame_skip = 4`, so the agent acts every four tics, or approximately 114 ms of game time per action [2604.07385].

The system’s input modality is deliberately compact. A 640×480 RGB frame is converted to grayscale, downscaled to 40×25 through block averaging, and then mapped to one of ten brightness characters, `" .:-=+*#%@"`, producing a 1,024-character serialized grid with newline separators [2604.07385]. In parallel, the per-pixel depth buffer supplied by VizDoom is downscaled to 40×25, normalized to $[0,1]$, and quantized into 16 bins; a 17th “no-depth” bin is reserved for special tokens such as `[CLS]` and `[PAD]` [2604.07385]. This yields what the paper describes as a dual representation that preserves 2D spatial layout and approximate 3D distance while remaining textual at the token level [2604.07385].

The benchmark is explicitly constructed to compare agents receiving equivalent input. All agents, including the tested LLMs, are given ASCII frames and depth maps rather than raw images [2604.07385]. A plausible implication is that the comparison isolates representation handling, spatial parsing, and latency more directly than a multimodal image-input comparison would, although the paper also lists raw-image LLM evaluation as future work rather than reporting such results [2604.07385].

## 2. Input encoding, tokenization, and action formulation

The model uses a strict character-level tokenizer rather than a subword tokenizer. The vocabulary contains 75 tokens, including the ten brightness symbols, additional entity markers, lowercase letters, digits, punctuation, six action-specific special tokens used in experiments, and standard special tokens such as `[PAD]` and `[CLS]` [2604.07385]. The central design choice is that each character in the 40×25 ASCII grid corresponds to exactly one token, preserving a one-to-one mapping between screen location and sequence position and maintaining the 2D spatial structure through row ordering and newline characters [2604.07385].

Depth is encoded per character position. Each aligned depth value is normalized, quantized into one of 16 bins, and mapped to a learned depth embedding [2604.07385]. The paper states that the bins provide about 40 cm resolution at typical DOOM engagement ranges, sufficient to distinguish “melee range” from “across the room” [2604.07385]. The resulting token representation therefore fuses symbolic brightness and discretized distance.

The “MultiVec” designation refers to the fact that the encoder retains multiple vectors per frame—one per token—rather than collapsing the scene immediately into a single latent [2604.07385]. For token $i$, the input representation is the sum of character and depth embeddings,
\[
\mathbf{e}_i = \text{HashEmb}(\text{token}_i) + \text{DepthEmb}(\text{depth\_bin}_i),
\]
with positional information supplied through Rotary Position Embeddings within attention rather than by adding a separate positional vector [2604.07385]. The paper presents this as a scene representation that preserves per-token information until the final pooling stage.

The action space is deliberately narrow: $K=4$, corresponding to `shoot`, `move_forward`, `turn_left`, and `turn_right` [2604.07385]. These actions match the controls present in the human demonstration data, which did not include strafe keys [2604.07385]. At inference time, the model outputs a four-dimensional probability distribution over these actions, and a composite action strategy allows multiple compatible buttons to be activated simultaneously, approximating human multi-key input [2604.07385]. This is important because the training labels themselves are soft and allow multiple actions to be simultaneously appropriate.

## 3. Architecture and representation bottleneck

SauerkrautLM-Doom-MultiVec is built around a compact ModernBERT-based encoder with hash embeddings, depth-aware token representations, and an attention-pooling classification head [2604.07385]. The total parameter count is about 1.319M, with approximately 4,480 parameters in the hash embeddings, 2,176 in the depth embeddings, about 1,312,000 in the transformer layers, and 644 in the attention pooling and classifier [2604.07385].

The backbone is a five-layer ModernBERT variant with hidden size 128 and sequence length of approximately 1,024 tokens [2604.07385]. Its attention pattern alternates between global and local attention: layers 0 and 3 use global attention over the full sequence, while layers 1, 2, and 4 use local attention with a sliding window of 128 tokens [2604.07385]. Rotary position embeddings are used as in ModernBERT [2604.07385]. The architectural rationale given in the paper is that local attention captures spatial neighborhoods in the ASCII grid, whereas global layers propagate information across the entire field of view [2604.07385].

Instead of a standard $V \times H$ embedding matrix, the model uses a two-stage hash embedding parameterization with vocabulary size $V \approx 75$, hidden dimension $H=128$, and projection dimension $P=16 \ll H$ [2604.07385]. The mapping is written as
\[
\text{HashEmb}: \{1,\dots,V\} \to \mathbb{R}^H
\]
and implemented as
\[
\mathbf{e}^{\text{char}}_i = \mathrm{LN}\left(\mathbf{E}_2^\top \,\mathbf{E}_1[\text{token}_i]\right),
\]
with $\mathbf{E}_1 \in \mathbb{R}^{V \times P}$ and $\mathbf{E}_2 \in \mathbb{R}^{P \times H}$ [2604.07385]. The paper notes that the absolute savings are modest for such a small vocabulary—4,480 parameters instead of 16,384—but treats the design as a demonstration of a scalable compression pattern [2604.07385].

For depth-aware token representations, the depth embedding table is defined as
\[
\text{DepthEmb}: \{0,\dots,16\} \to \mathbb{R}^H,
\]
with bin 16 representing “no depth” [2604.07385]. The final input embedding for each token is the sum of character and depth embeddings, allowing the model to distinguish visually similar symbols at different distances [2604.07385].

The classifier does not use a `[CLS]` token or mean pooling. Instead, it uses learned attention pooling. If the encoder outputs
\[
\mathbf{H} = [\mathbf{h}_1, \dots, \mathbf{h}_S] \in \mathbb{R}^{S \times 128},
\]
then the pooling weights are
\[
\alpha_i = \frac{\exp(\mathbf{w}^\top \mathbf{h}_i)}{\sum_j \exp(\mathbf{w}^\top \mathbf{h}_j)},
\]
the pooled vector is
\[
\mathbf{v} = \sum_i \alpha_i \mathbf{h}_i,
\]
and the classifier produces logits
\[
\mathbf{y} = \mathbf{W}_c \mathbf{v} + \mathbf{b}_c,\quad \mathbf{W}_c \in \mathbb{R}^{4 \times 128},
\]
followed by
\[
\mathbf{p} = \mathrm{softmax}(\mathbf{y}) \in \mathbb{R}^4.
\]
The paper interprets this as an information bottleneck from a 1024×128 multi-vector scene representation to a single 128-dimensional decision vector [2604.07385].

A notable part of the architecture’s development is the rejection of more explicit late-interaction mechanisms. Earlier ColBERT-style MaxSim formulations using textual action queries collapsed to a single dominant action, and learned prototype-vector approaches using MaxSim or cross-attention also collapsed, which the authors attribute to overparameterization relative to the four-class, 31k-example setting [2604.07385]. Attention pooling was adopted as the final design because it gave stable training and the best validation and gameplay performance [2604.07385]. This directly counters the assumption that preserving token-level vectors to the very end necessarily improves a small classifier; here, a strong bottleneck was empirically required.

## 4. Training data, supervision, and optimization

Training uses human gameplay demonstrations collected in the same `defend_the_center` scenario [2604.07385]. The first author played using native DOOM controls, while VizDoom spectator mode recorded RGB frames, depth buffers, and key presses [2604.07385]. The dataset consists of four recording sessions totaling approximately two hours of gameplay, more than 80 episodes, and 31,645 frames sampled with `frame_skip = 4` [2604.07385]. Each frame is paired with a 40×25 ASCII rendering, 16-level quantized depth per token, and action labels restricted to the same four actions used at inference time [2604.07385].

The supervision signal is not one-hot. Human key presses are converted into four-dimensional soft score vectors $\mathbf{s}$ so that pressed actions receive high scores and non-pressed actions receive a baseline, while allowing multiple actions to be high simultaneously [2604.07385]. This is intended to reflect the fact that multiple actions can be appropriate in a single frame and to preserve “action affinities” such as `move_forward` and `shoot` co-occurring [2604.07385]. The paper frames this as a way to capture composite human control more faithfully than hard labels would.

Because labels are soft distributions, the training objective is KL divergence between the model’s softmax output and the soft teacher scores:
\[
\mathcal{L} = \mathrm{KL}\left( \mathrm{softmax}(\mathbf{y}) \,\|\, \mathrm{softmax}(\mathbf{s}) \right).
\]
The stated reason for choosing KL over standard cross-entropy is that KL preserves gradient signal even for “correct” predictions when the target assigns non-zero mass to non-maximal actions [2604.07385].

The final training configuration uses a 90/10 train/validation split, corresponding to 28,480 training frames and 3,165 validation frames, AdamW, an initial learning rate of $3 \times 10^{-4}$, cosine decay with 500 warmup steps, batch size 32, 50 epochs, best-checkpoint selection by validation accuracy, and bfloat16 mixed precision on a single NVIDIA RTX 6000 Ada GPU [2604.07385]. The model reaches 57.7% top-1 validation accuracy on the four-way classification task, compared with a 25% random baseline [2604.07385]. The paper explicitly notes that this apparently modest validation accuracy is compatible with strong gameplay because the task is inherently ambiguous and because near-miss action substitutions can still be behaviorally reasonable [2604.07385].

A common misconception in this setting is that strong policy performance requires reinforcement learning from environment interaction. In this case, the reported system is trained through supervised imitation learning on approximately 31k human-labeled frames and does not require reinforcement learning during training [2604.07385]. This does not imply that RL is unnecessary for DOOM in general; rather, it shows that under this benchmark, carefully structured imitation learning can be sufficient.

## 5. Inference policy, evaluation, and comparison with large language models

At deployment, each frame is converted to 40×25 ASCII and depth bins, tokenized at the character level, encoded by the ModernBERT-Hash model, attention pooled, and classified to produce action probabilities $\mathbf{p}$ [2604.07385]. The inference procedure then applies a composite action rule. Let $a_1 = \arg\max(\mathbf{p})$ be the top action; initialize the button set from $a_1$. If $a_1 \neq \text{shoot}$ and
\[
p_{\text{shoot}} > 0.75 \cdot p_{a_1},
\]
then `shoot` is added. If the second-highest probability action $a_2$ satisfies $p_{a_2} > 0.15$ and is compatible with $a_1$, it is also added [2604.07385]. This can yield outputs such as `move_forward + turn_left + shoot` [2604.07385].

The main benchmark compares SauerkrautLM-Doom-MultiVec with GPT-4o-mini, Gemini Flash Lite, Qwen3.5-27B, and Nemotron-120B under the same environment configuration used in training: 640×480 resolution, HUD enabled, `frame_skip = 4`, and real-time pacing enforced by `sleep()` so that the environment runs in true real time [2604.07385]. Metrics include number of episodes, average survival, maximum survival, cumulative frags, and wall-clock latency per decision [2604.07385]. The LLMs are given text-only input: a system prompt describing the ASCII and depth encoding and a per-frame user message containing a fenced 40×25 `View:` block and a 40×25 `Depth:` block of digits [2604.07385]. They must return a valid action or action pair without explanation [2604.07385].

| Agent | Params | Frags / Latency |
|---|---:|---:|
| SLM-Doom-MV-1.3M | 1.3M | 178 / 31 ms |
| GPT-4o-mini | – | 0 / 646 ms |
| Gemini Flash Lite | – | 8 / 920 ms |
| Qwen3.5-27B | 27B | 2 / 13.3 s |
| Nemotron-120B | 120B | 3 / 8.9 s |

In the main benchmark, the small model scores 178 frags in 10 episodes, or 17.8 per episode, while all tested LLMs combined achieve 13 frags total [2604.07385]. Its average survival is 388 steps, with a maximum of 525, corresponding to full 60-second survival, and its latency is 31 ms per decision on CPU [2604.07385]. By contrast, GPT-4o-mini records the highest average survival among the LLMs at 104 steps but achieves 0 frags, while the large open-weight models are seconds per frame and therefore severely out of sync with the environment [2604.07385].

The paper also reports a “fair-mode” benchmark with `frame_skip = 20`, corresponding to one action every 20 tics, or about 571 ms, to reduce the penalty imposed by LLM latency [2604.07385]. Even there, SauerkrautLM-Doom-MultiVec remains best overall in both survival and frags, scoring 10 frags across 10 episodes with approximately 29 ms latency, while GPT-4o-mini again records 0 frags [2604.07385]. The reported conclusion is therefore not only that the small model is faster, but that it remains behaviorally stronger even when the benchmark is slowed to be more favorable to the slower systems [2604.07385].

A central misconception addressed by these results is that scale alone should compensate for poor task match. Within this benchmark, Nemotron-120B is 92,300× larger than the 1.3M model yet scores only 3 frags across 5 episodes, and Qwen3.5-27B is approximately 20,800× larger yet scores 2 frags across 3 episodes [2604.07385]. The reported evidence therefore contradicts any simple parameter-count-based expectation of superiority for low-level real-time control.

## 6. Behavioral interpretation, prior context, and limitations

Qualitatively, SauerkrautLM-Doom-MultiVec is described as the only evaluated agent that reliably engages enemies rather than merely spinning or evading [2604.07385]. It uses combined ASCII and depth cues for target acquisition, rotates toward threats, fires when enemies are near the center and close in depth, and often survives the full 60 seconds while accumulating frags [2604.07385]. Approximately 62% of its actions are composite, such as `turn_left+shoot` or `move_forward+turn_right`, producing fluid behavior closer to human multi-key control [2604.07385].

The LLMs display a different behavioral profile. GPT-4o-mini survives longer than the other LLMs but is heavily skewed toward rotation actions and never secures a frag in the main benchmark [2604.07385]. Gemini Flash Lite achieves the highest frag count among the LLMs, with 8 in the main benchmark, apparently benefiting from lower latency than the large open-weight models [2604.07385]. Qwen3.5-27B and Nemotron-120B are reported to be too slow for effective low-level control, and the paper also argues that their BPE tokenizers disrupt the 2D grid structure of ASCII input, while their prompt-by-prompt operation lacks persistent temporal state across frames [2604.07385]. This suggests that their weakness is not reducible to one factor; latency, tokenizer mismatch, and lack of state all contribute.

The work sits within a broader DOOM and VizDoom tradition in which earlier agents typically relied on deep RL with CNN-based perceptual stacks and often required millions of interaction steps [1809.03470]. The VizDoom competition literature emphasizes the difficulty of navigation, opponent handling, survival, and generalization in FPS settings, and notes that even strong RL bots remained significantly below skilled human performance [1809.03470]. Against that background, SauerkrautLM-Doom-MultiVec differs by using imitation learning from human demonstrations, an ASCII-plus-depth representation instead of raw pixels, and a transformer encoder at a very small parameter scale [2604.07385]. A plausible implication is that the model’s contribution is less about solving general DOOM and more about identifying a narrow but effective design point for fast reactive control.

The paper also records several ablation-like design judgments. ColBERT-style MaxSim and learned action prototypes collapsed; attention pooling produced stable training and best performance; preliminary GPT-4o-mini experiments without depth averaged only about 54 survival steps; and hash embeddings reduced embedding parameters by 73% compared to a standard 128×128 matrix [2604.07385]. These observations support the authors’ argument that attention pooling and depth-fused character-level embeddings are central to the system’s effectiveness [2604.07385].

The reported limitations are substantial. Training and evaluation are confined to a single scenario, `defend_the_center`, so generalization to corridors, multi-room maps, outdoor scenes, or different enemy types remains untested [2604.07385]. The model depends on matched game settings such as resolution and HUD presence; changes in those settings alter the ASCII pattern and may degrade performance unless the model is retrained or adapted [2604.07385]. The demonstration corpus comes from one skilled human over approximately two hours, so the policy inherits that player’s style and biases [2604.07385]. The method is frame-wise classification without explicit temporal memory, and while this works in the benchmark, more complex tasks may require recurrent or memory-augmented designs [2604.07385]. Finally, the LLM comparison is constrained by the lower episode counts feasible for the slowest models, which the paper presents as sufficient to show a trend but not as a definitive statistical account [2604.07385].

Taken together, the evidence presented in the benchmark supports a narrow but clear conclusion: in real-time DOOM control under ASCII-plus-depth input, small task-specific transformers can outperform vastly larger general-purpose LLMs when the representation, supervision, and inference policy are tightly aligned with the control problem [2604.07385]. It does not establish that such models dominate across all embodied tasks or all game settings. It does establish that, for this latency-sensitive FPS benchmark, architectural fit and operational constraints matter more than raw scale.

Source: https://www.emergentmind.com/topics/sauerkrautlm-doom-multivec