---
title: 'VISOTHELLO: Multimodal Othello World Model'
url: https://www.emergentmind.com/topics/visothello
type: topic
---

# VISOTHELLO: Multimodal Othello World Model

Searching arXiv for VISOTHELLO and prior Othello model work to support the encyclopedia article.
VISOTHELLO is a VisualBERT-style multimodal Othello world model trained on paired move sequences and board images to predict moves in a grounded, deterministic, rule-based environment. It was introduced to study the symbol grounding debate under unusually clean conditions: in Othello, the latent state is the board configuration, the state transition function is exactly specified by the rules, and the move sequence is a symbolic trace of that world. Against that backdrop, VISOTHELLO asks whether adding visual state information to symbolic move histories makes world-model learning faster, better, and more robust than text-only training alone. The reported results indicate that visual grounding improves sample efficiency, latent board-state encoding, and robustness to semantically irrelevant perturbations, while stopping short of claiming that text-only models are incapable of learning structured world representations [2507.14520].

## 1. Conceptual setting and motivation

The central object of study is not Othello as a competitive game system, but Othello as a laboratory for world modeling. The paper motivates this choice by emphasizing that Othello provides a grounded, deterministic, rule-based world in which the latent state is unambiguous and fully defined by the board configuration. Because state transitions are exactly given by the rules, questions about “understanding” can be operationalized through next-move legality, board-state decoding, and invariance under transformations that preserve game semantics.

VISOTHELLO is positioned against prior Othello-playing language-model work, especially text-only Othello-GPT and BERT-style sequence models. Those earlier systems had already shown that models trained only on move sequences can develop internal representations of board state, which the paper treats as evidence for a rudimentary world model emerging from symbolic data alone. VISOTHELLO therefore does not challenge the existence of text-only world modeling in this domain. Its narrower question is whether explicit perceptual grounding strengthens that capacity.

This framing places the model within the broader symbol grounding debate. One side of that debate holds that world understanding can emerge from text alone; another suggests that grounded learning is more efficient. VISOTHELLO treats Othello as a controlled intermediate case: symbolic traces are available, perceptual state is available, and the correspondence between the two is exact. This suggests that the paper’s contribution is as much methodological as architectural: it turns a philosophical dispute into a measurable multimodal learning problem.

## 2. Representation and model architecture

A game is represented textually as a sequence of move tokens
\[
[m_1, m_2, \dots, m_s],
\]
where each \(m_t\) is one of 64 board positions. The vocabulary therefore contains 64 unique tokens, one per tile on the \(8 \times 8\) board. For each step \(t\), the model also receives a board image \(b_t\) showing the board state after moves \(m_1,\dots,m_{t-1}\), so that \(b_t\) serves as the visual context for predicting the next move \(m_t\).

The visual stream is encoded by an Othello-specific ResNet-18 image encoder,
\[
v_t = \phi(b_t) \in \mathbb{R}^{d_v},
\]
with \(\phi\) denoting the image encoder and \(d_v\) the visual embedding dimension. These visual embeddings are treated as image tokens and concatenated with text tokens, separated by a special \([SEP]\) token. The multimodal backbone is based on VisualBERT, that is, a Transformer operating over a fused token stream of textual and visual embeddings. The paper does not provide a fuller fusion equation beyond this VisualBERT-style concatenation [2507.14520].

The experimental design uses three comparison models chosen to isolate the value of grounding.

| Model | Input | Role |
|---|---|---|
| Othello-GPT | Move sequence | Text-only autoregressive baseline |
| BERT | Move sequence | Text-only masked-modeling control |
| ResNet-18 | Current-board image | Vision-only baseline |
| VISOTHELLO | Move histories + board images | Multimodal model |

This comparison structure matters. BERT is the strongest control for “same language-side objectives, no vision,” while ResNet-18 isolates what can be learned from current-board images without sequential symbolic context. VISOTHELLO is therefore not merely compared to weaker baselines; it is tested against language-only and vision-only controls that are meant to expose the specific contribution of multimodal grounding.

## 3. Training objectives, masking scheme, and data regime

The training paradigm is masked-language-modeling rather than autoregressive generation. The paper argues that an autoregressive multimodal setup would require static image-text inputs to be converted into a pseudo-video token stream, increasing computational cost and complexity, whereas masked-language-modeling enables bidirectional reasoning over static visual-text inputs. The total objective is described as the sum of masked modeling and text-image prediction losses.

The multimodal pipeline proceeds in two stages. First, the ResNet is pretrained to predict the next move from the current board image. Second, the Transformer is trained multimodally with three objectives: text-image prediction, random token masking, and future token masking. In random token masking, following BERT and VisualBERT, move tokens are randomly masked with 80% probability, masking 15% of move tokens while leaving image tokens visible. In future token masking, used with 20% probability, a target step \(t\) is selected and the suffix
\[
m_t, m_{t+1}, \dots, m_s
\]
is masked. To prevent leakage from future board states, the corresponding future visual tokens
\[
v_{t+1}, v_{t+2}, \dots, v_s
\]
are also masked. The third objective, text-image prediction, performs binary classification over matched versus mismatched game/image pairs by replacing the move sequence with a random sequence with probability 50%.

The dataset contains 25,657 real Othello game records collected from the EOTHELLO website. Board images are generated from these games, yielding one image per step and producing 1,559,968 images in total, with an average of about 60.8 images per game. The split is 80% train, 5% validation, and 15% test, corresponding to 20,525 training games, 1,282 validation games, and 3,850 test games. To study sample efficiency, models are also trained on subsets of size 1k, 3k, 5k, 10k, and 20k. Training runs for up to 1000 epochs, validation is performed every 10 epochs, early stopping uses patience 5 validation steps, and checkpoint selection is based on best validation accuracy. Training uses a single NVIDIA A100-40GB GPU. BERT and VISOTHELLO use batch size 128 and learning rate \(10^{-4}\), while ResNet uses batch size 512 and the same learning rate. Models trained from scratch and from pretrained weights are denoted \(-S\) and \(-P\), respectively [2507.14520].

## 4. Predictive performance, sample efficiency, and ablations

The principal evaluation metric is legal move accuracy. Given move history \(m_1,\dots,m_{t-1}\), the prediction \(m_t\) is counted correct if it is legal under Othello’s rules. On this metric, VISOTHELLO is consistently strong, especially in low-data settings and in several larger-data regimes.

At 1k training examples, Othello-GPT reaches 7.34, BERT-S 20.74, ResNet-18-S 6.71, and VISOTHELLO-S 23.64; with pretrained variants, BERT-P remains 20.74, ResNet-18-P 6.71, and VISOTHELLO-P 43.30. At 3k examples, VISOTHELLO-P reaches 91.42, slightly exceeding BERT-S at 90.86, BERT-P at 90.35, and ResNet-18-P at 71.28. At 5k, VISOTHELLO-S reaches 93.87, above BERT-S at 90.81 and ResNet-18-S at 71.27. At 10k, VISOTHELLO-S reaches 93.25, again above BERT-S at 90.79 and ResNet-18-S at 75.58. At 20k, VISOTHELLO-S reaches 94.23, while the best full-data result is VISOTHELLO-P at 94.03, exceeding BERT-P at 92.65 and ResNet-18-P at 92.23 [2507.14520].

These results support the paper’s narrower claim that multimodal grounding improves sample efficiency and often final performance. The effect is especially marked relative to the vision-only baseline in low-data regimes. The pretraining story is more mixed. ResNet benefits substantially from pretraining at small data sizes, but BERT and VISOTHELLO do not show a uniformly positive pretraining effect, which the authors relate to prior suggestions that generic linguistic pretraining may transfer only weakly to structured rule-based worlds such as Othello.

The ablation results sharpen the architecture-level interpretation. Full VISOTHELLO reaches 94.03 legal-move accuracy. Replacing the fine-tuned ResNet image encoder with simpler image representations reduces performance to 92.43 for “Pooling” and 91.80 for “Area.” Using a ResNet-18 encoder without fine-tuning yields 92.04. The most consequential ablation removes future token masking, reducing performance to 62.03. This indicates that the multimodal benefit is not attributable to image input alone: the specific training design, and especially the alignment between future masking and next-move structure, is essential.

## 5. Latent board-state representation and robustness under perturbation

Beyond end-task accuracy, the paper probes whether VISOTHELLO develops stronger internal world representations. Following Nanda et al. (2023), it trains a linear probe on hidden activations to predict each tile’s state as empty, player disc, or opponent disc. The setup is standard linear classification over hidden states \(h\), exemplified by
\[
\hat{y} = W h + b,
\]
with F1 used as the evaluation metric. VISOTHELLO consistently outperforms BERT across training sizes and layers. Even at initialization, corresponding to “0 examples,” VISOTHELLO already contains more board-relevant structure, plausibly because its ResNet encoder has been pretrained on Othello images. After training on 20k examples, VISOTHELLO reaches 77.55 F1 at Layer 18, compared with 62.28 for BERT. The paper interprets this as evidence that multimodal grounding improves latent board-state representation rather than merely improving move prediction [2507.14520].

Robustness is tested through a semantically irrelevant perturbation: 180° rotation of the board at test time. Because Othello’s rules are invariant under this transformation, the perturbation changes spatial encoding without changing game semantics. For image models, the board image is rotated. For text models, move coordinates are deterministically remapped, for example \(D3 \mapsto E6\). All models are trained only on original data and then evaluated on rotated inputs.

The results separate abstraction from superficial cue dependence. BERT remains robust, staying around 90–93% legal move accuracy under rotation. ResNet-18 collapses to around 28–35%, indicating heavy dependence on absolute spatial patterns and weak rotation invariance. VISOTHELLO remains high at about 91–93% after rotation, preserving performance far better than the vision-only model. The paper’s interpretation is that symbolic sequence information stabilizes perception while visual grounding enriches the learned representation. A plausible implication is that multimodal grounding is most effective here when it is constrained by an already structured symbolic trace, rather than when vision is asked to discover the game state in isolation.

## 6. Representation alignment, interpretation, and limitations

The appendix extends the representational analysis with MUSE-style linear mapping between hidden spaces from different models. Given hidden representations \(H_1\) and \(H_2\) from models \(M_1\) and \(M_2\), they are first projected with PCA into a shared dimension \(d=\min(d_1,d_2)\):
\[
H_1' = P_d(H_1), \qquad H_2' = P_d(H_2).
\]
A linear alignment matrix is then learned by minimizing
\[
W^* = \arg\min_{W \in \mathcal{M}(\mathbb{R})} \| H_i' W - H_j' \|,
\]
using iterative Procrustes refinement in the supervised case and adversarial initialization plus Procrustes in the unsupervised case. Evaluation uses average cosine similarity. Alignment generally improves with more data, and Othello-GPT aligns surprisingly strongly with VISOTHELLO: in supervised alignment, Othello-GPT \(\rightarrow\) VISOTHELLO reaches similarity around 82.35 at the largest setting shown, and in unsupervised alignment it reaches 82.46. VISOTHELLO also becomes more alignable to ResNet as data increases, with unsupervised VISOTHELLO \(\rightarrow\) ResNet rising to 57.79 [2507.14520].

This alignment result qualifies a possible misconception about the paper’s thesis. VISOTHELLO does not argue that multimodality produces entirely different latent structure from text-only learning. On the contrary, the strong alignment between Othello-GPT and VISOTHELLO suggests that different training objectives and modalities can converge toward similar Othello-relevant internal organizations. The paper’s conclusion is therefore restrained: text-only models can learn world structure, but visual grounding makes that learning more efficient and more robust in this rule-defined environment.

The limitations are explicit. The work does not compare against autoregressive multimodal LLMs, because the authors regard such a comparison as computationally mismatched and as changing the problem into sequential image-token modeling. It does not benchmark against large-scale text-only LLMs, on the grounds that related questions have been studied elsewhere and language pretraining alone does not appear especially beneficial for Othello. The paper also does not report statistical significance tests, confidence intervals, or many implementation-level architectural dimensions such as full parameter counts. These omissions constrain the precision with which one can generalize the empirical gains.

Taken together, VISOTHELLO occupies a specific position in multimodal learning research. It is neither a claim that perception is necessary for symbolic world modeling nor a claim that text is sufficient in all cases. Rather, it is evidence from a clean synthetic world that grounding symbols in perceptual state can improve sample efficiency, strengthen board-state encoding, and reduce brittleness under semantically irrelevant perturbations. In that sense, Othello functions as a minimal testbed for a broader research hypothesis: when symbols refer to an underlying structured world, access to perception can help neural models build more stable internal representations of that world.

Source: https://www.emergentmind.com/topics/visothello