Playing DOOM with 1.3M Parameters: Specialized Small Models vs Large Language Models for Real-Time Game Control

Abstract: We present SauerkrautLM-Doom-MultiVec, a 1.3 million parameter model that plays the classic first-person shooter DOOM in real time, outperforming LLMs up to 92,000x its size, including Nemotron-120B, Qwen3.5-27B, and GPT-4o-mini. Our model combines a ModernBERT encoder with hash embeddings, depth-aware token representations, and an attention pooling classification head to select game actions from ASCII frame representations at 31ms per decision. Trained on just 31,000 human gameplay demonstrations, it achieves 178 frags in 10 episodes (17.8 per episode) in the defend_the_center scenario, more than all tested LLMs combined (13 frags total). All agents receive equivalent input: ASCII frames and depth maps. Despite having 92,000x fewer parameters than Nemotron-120B, our model is the only agent that actively engages enemies rather than purely evading them. These results demonstrate that small, task-specific models trained on domain-appropriate data can decisively outperform general-purpose LLMs at real-time control tasks, at a fraction of the inference cost, with deployment capability on consumer hardware.

• The paper demonstrates that a 1.3M parameter model using specialized ASCII and depth encoding dramatically outperforms LLMs on real-time DOOM control.
• It employs a ModernBERT-based encoder enhanced with hash embeddings, character-level tokenization, and attention pooling to preserve spatial coherence.
• The model achieves 17.8 frags per episode with 31ms/frame latency, highlighting its suitability for edge-deployable, real-time applications.

Playing DOOM with 1.3M Parameters: Comparative Analysis of Small Specialized Models and LLMs for Real-Time Game Control

Introduction

This paper introduces SauerkrautLM-Doom-MultiVec, a 1.3M parameter, text-based model trained to play the first-person shooter DOOM in real time. The authors construct a comprehensive empirical comparison, measuring the effectiveness of this highly parameter-efficient model against state-of-the-art multimodal LLMs—Nemotron-120B, Qwen3.5-27B, GPT-4o-mini, and Gemini Flash Lite—on an equivalent closed-loop game control task. Despite its compactness and modest training data (31,000 human-annotated frames), the model vastly outperforms these LLMs, both in frag (enemy defeat) count and in execution latency, underscoring the continued relevance of task-specific, small models for real-time, edge-deployable AI applications.

Model Architecture and Input Pipeline

SauerkrautLM-Doom-MultiVec is built around a ModernBERT-based encoder with several key architectural enhancements:

• Hash Embeddings: A dual-stage embedding process compresses the input vocabulary leveraging hash functions, yielding a 73% reduction in embedding parameters while maintaining representational quality.
• Depth-Aware ASCII Representation: The model fuses ASCII-art-style downscaled video frames (40×25 grid, 10 brightness glyphs) with quantized per-token depth embeddings (16 bins). Each 1,024-token frame carries explicit (quantized) 3D spatial information as summed embeddings.
• Character-level Tokenization: A strictly one-to-one character-token mapping preserves 2D spatial coherence absent in BPE-based LLM tokenizations, a critical property for reasoning over ASCII frames.
• Attention Pooling Head: Per-token output embeddings (1,024×128) are pooled via a learned attention mechanism (rather than mean or [CLS]), focusing the classifier on informative grid regions.

The input pipeline, from VizDoom frame to model embedding, is succinctly captured as follows:

Figure 1: The input transformation pipeline converts a 640×480 RGB DOOM frame and its depth buffer into dual-channel, 40×25 ASCII+depth representations used for token embedding and classification.

Experimental Methodology

All models operate on the same 4-action discrete control set—shoot, move_forward, turn_left, turn_right—and receive equivalently encoded 40×25 ASCII and depth grid inputs. The benchmark scenario, defend_the_center, involves surviving and eliminating enemies that spawn omnidirectionally in a circular DOOM arena.

The small model is trained on 31K high-quality human demonstration frames, with soft action labels derived from simultaneous keypresses, and achieves 57.7% validation top-1 prediction accuracy. LLMs are prompted with the same per-frame ASCII/depth grid and system prompt, but are constrained by their native text tokenization and API call overhead.

Results and Comparative Analysis

Quantitative Performance

• Frag Score: The 1.3M parameter model achieves a cumulative 178 frags in 10 episodes (17.8/ep), compared to 13 combined for all competing LLMs (Nemotron-120B: 3, Qwen3.5-27B: 2, GPT-4o-mini: 0, Gemini Flash Lite: 8). This is a 14× greater frag count than all LLMs combined—even though Nemotron-120B has 92,300× more parameters.
• Survival: The model exhibits the highest survival (388 steps avg, maximum 525/epoch), frequently reaching the episode time limit.
• Latency: SauerkrautLM-Doom-MultiVec executes at 31ms/frame on CPU (meeting DOOM’s 35FPS, sub-100ms human-latency threshold), while LLMs require 0.6–13 seconds/frame—unsuitable for any real-time agentic deployment.
• Composite Actions: 62% of actions are multi-action combinations (e.g., turn_left+shoot), reflecting sophisticated, human-like gameplay. LLMs rarely output composites and predominantly exhibit evasive behavior.

Ablations and Fairness Controls

A fair-mode test—reducing frame-rate to accommodate higher LLM latency—does not close the capability gap. The small model remains dominant in both frags and survival.

Qualitative Observations

The model consistently exhibits complex spatial strategies:

• Acquires targets using positionally-informed rotations.
• Selects composite actions for engagement (e.g., shoot+turn).
• Properly manages close-range threats, showing an understanding of enemy proximity via depth binning.

In contrast, LLMs:

• Are slowed by tokenization and API bottlenecks.
• Exhibit defensive, non-engaging strategy (survive but do not accrue frags).
• Lose spatial coherence due to BPE, resulting in destructive aggregation of ASCII structure.
• Lack memory, processing each frame myopically with no temporal state.

Implications and Theoretical/Practical Outlook

Small Models in High-Fidelity, Real-Time Control

The results decisively demonstrate that for nontrivial, latency-bound interactive scenarios, small, well-architected models with strong task supervision can dramatically outperform large-scale, generalist LLMs even on native text inputs. This holds, even with broad LLM multi-modality and chain-of-thought prompting.

Cost, Latency, and Edge Deployment

• Inference Cost: Small models run indefinitely on commodity CPU hardware, whereas LLMs incur not just computational but direct monetary costs for API calls (~$15–30 for a handful of game episodes).
• Edge/Robotics Applicability: At 1.3M parameters (∼5MB), the model is suitable for embedded hardware, including Raspberry Pi-class devices.
• Data Magnitude: The small model learns effectively from ∼2 hours of gameplay data (31K frames), a minuscule fraction compared to pretraining data for LLMs.

LLM Limitations on Visually Grounded Tasks

The benchmarked LLMs possess substantial pretraining on trillions of text and image tokens but lack explicit mechanisms for:

• Character-level spatial reasoning: ASCII-coherent tokenization is not part of standard LLM training pipelines.
• Fine-grained, real-time action selection: LLM inference is orders of magnitude too slow for closed-loop, high-frequency control; chain-of-thought does not compensate in latency-bound settings.
• Temporal aggregation: LLMs process each frame statelessly, eroding the ability to track enemy trajectories or temporal patterns.

Role of Depth Embedding

Explicit per-token (16-bin) depth embeddings provide crucial 3D structure, supporting effective action disambiguation that brightness-only (ASCII) input or simple visual summarization cannot achieve.

Limitations and Future Directions

While the conclusions concerning domain-specific, small model superiority are robust, several constraints inform future research potential:

• Scenario Generality: The presented model is tuned to a single DOOM scenario with matched settings; richer environments will necessitate broader or more data-diverse training.
• Multi-vector Classification: All experiments with alternative multi-vector aggregation (MaxSim, prototype-based) failed to surpass attention pooling due to poor gradient signal with small data/class counts.
• LLM Benchmarks: Due to API expense, reasoning LLMs are evaluated over fewer episodes, although this does not affect the stark frag disparity.
• Vision Input for LLMs: While only tested on ASCII+depth grids, even multimodal LLMs would likely see only marginal improvement and incur even greater latency/costs if evaluated directly on image input.

Conclusion

The comparative study emphatically supports the thesis that appropriately trained, task-specialized small models remain superior to large multimodal LLMs on real-time action selection and agentic control in latency-critical, visually grounded environments. The design—character-level spatial representation, explicit depth embedding, and attention pooling—provides an efficient, deployable template for similar tasks in robotics and games. The evidence calls for a nuanced view of AI model scaling: “bigger” is not categorically “better”—model, data, and representation must fit the task’s operational realities.

Outlook

Key avenues for further investigation include expanding to multi-scenario, multi-task agents, systematic evaluation of LLMs on raw vision inputs, and architectural strategies for maintaining spatial coherence and temporal state in multilingual agentic LLMs. Additionally, deployment on ultra-low-power edge devices will require ONNX export and quantization experiments. The findings reaffirm an essential AI engineering principle: domain-specific optimization and efficient architecture remain fundamental, even as foundational models dominate the broader discourse.