VisualHints: Multimodal RL Benchmark

• VisualHints is a multimodal RL environment that fuses visual clues and textual observations to solve navigational cooking tasks under partial observability.
• The benchmark employs a POMDP framework with a natural-language action space, challenging agents with compositional decision making and integrated cue processing.
• Key variations in clue informativeness and risk, along with a CNN-LSTM baseline agent, provide actionable insights for advancing multimodal reinforcement learning research.

Searching arXiv for the primary paper and closely related work on “VisualHints” to ground the article in current arXiv records. VisualHints is a multimodal reinforcement learning environment that extends TextWorld’s Cooking Game with visual clues interspersed throughout the environment, so that an agent must use textual observations and visual hints together to predict natural-language action commands for solving the final task of cooking a meal. It was introduced to bridge a gap between benchmarks centered on pure vision, such as Atari and video-based robotic manipulation, and benchmarks centered on pure language, such as text-based games and dialog systems. In VisualHints, the environment remains partially observable, the action space remains natural language, and the visual clue is designed so that text alone is not reliably sufficient (Carta et al., 2020).

1. Conceptual scope and benchmark motivation

VisualHints is built around the claim that many real-world decision problems require simultaneous use of linguistic and visual information, whereas earlier RL environments typically privileged one modality. The environment therefore preserves the text-adventure structure of the cooking domain while introducing a map-like visual clue that must be interpreted jointly with local textual observations.

The underlying task remains the standard cooking objective: the agent must navigate through rooms, gather ingredients, reach the kitchen, prepare a meal, and eat it to win. What changes is the informational regime. Somewhere in the environment, the agent can issue the command examine hint and receive a floor-plan image together with a short textual clue explaining how to interpret the map. The clue may show the kitchen, ingredient rooms, the player location, passages, and potentially a dangerous death room that must be avoided. This converts navigation into a genuinely multimodal inference problem rather than a purely textual one (Carta et al., 2020).

A notable feature of the benchmark is that the visual hint is not treated as decorative side information. The environment is explicitly designed so that both modalities are needed. If passageways are hidden, the agent must rely on text to know which direction is open; if names are masked, it must use numeric importance labels; if both are partially masked, it must integrate modalities more tightly. This design makes VisualHints a benchmark for multimodal state estimation and action selection under partial observability, rather than a benchmark for image classification inside an RL loop.

2. Cooking-world task and formal specification

The task is formalized as a partially observable Markov decision process written as $(S, A, T, \Omega, O, R, \gamma)$, where $S$ is the set of states, $A$ the action space, $T$ the transition function, $\Omega$ the set of observable symbols, $O$ the observation, $R$ the reward, and $\gamma$ the discount factor. In VisualHints, the observation is explicitly multimodal, with textual observation $O_t^{text}$ and visual observation $O_t^{image}$. The policy is defined as

$S$0

and the objective is

$S$1

These definitions make the benchmark a direct instantiation of multimodal decision making in a POMDP (Carta et al., 2020).

The action space is natural language rather than a small fixed control vocabulary. Actions are commands composed from verbs, adjectives, and nouns, which makes the action space large and structured. The benchmark is therefore difficult along at least three axes simultaneously: multimodal observation, partial observability, and compositional action generation.

The environment uses the visual-lingual policy primarily during navigation to reach the kitchen. Once in the kitchen, a text-only policy handles the non-navigation recipe actions, with a “kitchen reached” flag used to switch between navigation and non-navigation behavior. This division isolates the principal role of the visual hint: route planning and safe navigation under incomplete local information.

3. Hint semantics and controllable scenario generation

A major contribution of VisualHints is its systematic variation of clue informativeness and environmental risk. The paper describes seven controllable modes for generating the map and clue: distance of the puzzle, death room, color path, name type, draw passages, draw player, and clue first room. These modes induce a spectrum of benchmark scenarios, from relatively guided navigation to heavily abstracted cases in which the agent must infer legality, identity, and safety from partially masked visual and textual cues (Carta et al., 2020).

The modes have distinct functional roles. Distance of the puzzle controls how much exploration is needed before finding the clue. Death room introduces hazardous states and large negative rewards. Color path can visually mark the correct route. Name type can show literal room names, random numbers, or room-importance labels. Draw passages determines whether open doors are rendered. Draw player determines whether current location is rendered. Clue first room determines whether the clue is guaranteed to appear in the starting room.

This parameterization matters because it allows VisualHints to emulate different multimodal reasoning regimes. Easier cases provide strong visual structure, such as colored paths and visible passages. Harder cases suppress direct semantics and require the agent to use text to resolve visual ambiguity. A plausible implication is that the environment functions not only as a benchmark for policy learning, but also as a controlled testbed for studying when and how each modality contributes to navigation competence.

4. Baseline multimodal agent and auxiliary supervision

The baseline agent combines a text module with a CNN-based visual module and is trained with a model-free Advantage Actor-Critic algorithm. The visual encoder uses a CNN followed by a bidirectional LSTM. The CNN extracts local map features into 512 channels. For an input structured in $S$2, the CNN output is $S$3; in experiments, $S$4. The bidirectional LSTM then encodes spatial context for each sequence of features, producing a fixed-size representation of 128 per input sequence. A second LSTM computes weights over the last hidden layers, and a weighted sum yields the final image embedding. In the fusion stage, the image embedding $S$5 is concatenated with the textual context representation $S$6 (Carta et al., 2020).

The actor and critic share this multimodal encoding. The critic feeds the concatenated representation through an MLP with one hidden layer of size 256 to predict a scalar state value. The actor scores candidate commands using the encoded context and image together with each candidate command encoding, then applies a softmax to obtain a categorical distribution over $S$7 candidate commands.

The visual component is pretrained before RL optimization on 42 auxiliary classification tasks over generated map images. These tasks include easy binary questions such as whether there is a death room or whether a room name appears, and harder relational questions such as how many rooms are blocked by the death room. The CNN-LSTM is trained with an $S$8 loss on 46,080 generated examples and tested on 11,520 examples. Reported F1 is nearly perfect for easy tasks, around or above 0.999, and still about 0.8 for harder tasks. The hardest tasks are those requiring relational reasoning about inaccessible rooms, especially when the map is partially masked. This auxiliary stage indicates that the authors treated map understanding as a prerequisite perceptual problem rather than leaving all visual structure to be learned from sparse RL reward.

5. Empirical behavior, benchmark statistics, and limitations

The benchmark contains 4,440 train, 222 validation, and 514 test games overall. For “Navigation (All)” there are 3,123 train, 156 validation, and 372 test games; for non-navigation there are 1,317 train, 66 validation, and 142 test games. The main evaluation metric is success rate, and the baseline multimodal agent is compared to LeDeepChef, a strong text-only baseline (Carta et al., 2020).

Setting	Multimodal agent	LeDeepChef
6-room navigation	0.74	0.90
9-room navigation	0.30	0.56
12-room navigation	0.13	0.31
Navigation (All)	0.39	0.59
Non-navigation	0.85	0.91
Overall	0.52	0.68

These results show that the baseline multimodal agent does not outperform the text-only baseline. On navigation games it trails LeDeepChef substantially, especially on harder settings, and on non-navigation games it remains somewhat worse though competitive. The paper is explicit that its goal is not to beat the state of the art, but to demonstrate a useful multimodal benchmark and show that visual information can be incorporated into text-game RL.

The paper also reports qualitative analysis of CNN outputs, showing that some convolutional filters respond to map structure, individual rooms, death-room placement, color paths, and player position. This supports the claim that the visual encoder learns semantically meaningful spatial features rather than only low-level pixel regularities.

The limitations are equally explicit. The baseline is intentionally simple. The environment remains centered on a specific cooking domain. The current fusion mechanism is concatenation plus MLPs rather than a richer cross-modal architecture. The results therefore position VisualHints primarily as a benchmark and problem formulation for future multimodal RL, especially in partially observable settings with large natural-language action spaces.

6. Relation to the broader visual-hint literature

VisualHints belongs to a broader research line in which auxiliary cues are introduced to disambiguate search, navigation, perception, or interaction. In classical visual search, short spoken spatial messages such as “On the left” or “At the bottom, on the right” reduced mean selection time from 5674 ms to 1747 ms and reduced errors from 150 to 79 in crowded displays, with strongest benefits for difficult tasks and radial layouts (0708.3575). In human-machine cooperative search, soft highlighting modulated saliency continuously with classifier confidence and outperformed hard highlighting in both synthetic and natural-image tasks, supporting the view that graded cues can communicate uncertainty more effectively than binary marks (Kneusel et al., 2016).

In mobile UI understanding, multimodal icon annotation combines screenshot pixels with view-hierarchy text such as resource-id, content description, and class name; on Rico$S$9, the best detector, CenterNet + VH, achieved F1 0.832 and outperformed both classification baselines and pixel-only detectors (Zang et al., 2021). In mobile testing, NaviDroid uses enriched state-transition graphs and floating-window overlays to show highlighted next operations, yielding 81% median state coverage on 85 open-source apps and improving manual testing outcomes in a user study (Liu et al., 2022). In AR search, visual and audio hints each improved performance, while their combination became clearly best only when paired with gaze-assisted instant post-task feedback (Zhang et al., 2023). In accessibility repair, HintDroid predicts missing hint-text for text inputs from GUI context and improved input accuracy, activity coverage, and state coverage for visually impaired participants (Liu et al., 2024). In hidden mobile interactions, GhostUI uses before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions so that models can infer concealed gestures and post-interaction states (Kweon et al., 27 Jan 2026).

Taken together, these works suggest that “visual hints” are not a single mechanism but a family of auxiliary signals for resolving partial observability. Within that family, VisualHints is distinctive because it embeds the hint inside a sequential decision process with natural-language actions. The environment therefore occupies a specific position: it is neither a static perception benchmark nor a pure language game, but a multimodal RL setting in which the hint must be interpreted, remembered, and operationalized through action selection over time (Carta et al., 2020).