AtlasVA: Visual Skill Memory & Alert Triage

• AtlasVA is a dual-use framework encompassing a visually grounded RL architecture for VLM agents and an astronomical alert-ranking system.
• It employs a three-layer visual skill memory—integrating spatial heatmaps, visual exemplars, and symbolic text skills—to enhance teacher-free learning.
• In astronomy, AtlasVA uses histogram-based classifiers and geometric scoring to rank alerts, significantly reducing manual review workload.

Searching arXiv for papers on AtlasVA and closely related usages to ground the article. I’ll look up the cited arXiv records for “AtlasVA”, “AVA”, and “ATWL” to ensure the article reflects the current literature. AtlasVA most prominently denotes a teacher-free visual skill memory framework for vision-LLM agents that keeps reusable experience visually grounded through spatial heatmaps, visual exemplars, and symbolic text skills, and reuses self-evolving danger and affinity atlases as potential-based shaping rewards for reinforcement learning (Wang et al., 18 May 2026). In separate astronomical usage, “AtlasVA” also refers to the ATLAS Virtual Research Assistant, a machine-learning triage system for the ATLAS transient stream that ranks alerts in a two-dimensional score space defined by “Real” and “Galactic” probabilities (Stevance et al., 11 Jun 2025).

1. Terminological scope

In the available arXiv literature, the designation “AtlasVA” is used for two technically unrelated systems: a visually grounded RL framework for VLM agents, and a production-grade transient-alert triage system in observational astronomy (Wang et al., 18 May 2026, Stevance et al., 11 Jun 2025).

Usage of “AtlasVA”	Domain	Defining characteristics
“AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents”	VLM agents and RL	Three-layer Visual Skill Memory; danger and affinity atlases; PPO/GAE; teacher-free learning
ATLAS Virtual Research Assistant	Time-domain astronomy	Histogram-Based Gradient Boosted Decision Tree Classifiers; “Real” and “Galactic” axes; VRA score; Recall at rank $K$

The first usage is the more explicit and self-contained formalization of “AtlasVA” as a named method. It addresses partially observable environments in which a VLM agent acts on multimodal observations and sparse binary rewards, and it argues that reusable experience should remain visually grounded rather than compressed into text. The second usage is an operational system in which alert ranking, auto-garbage policies, and follow-up triggers are learned from human labels and hand-engineered features.

2. Problem setting and motivation in the VLM-agent framework

In the VLM-agent setting, AtlasVA is formulated in a POMDP

$$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$

where at time $t$ the agent receives a multimodal observation $o_t$ consisting of an RGB image and a language instruction, chooses an action according to $\pi_\theta(a_t \mid o_{\le t})$, transitions under $\mathcal{T}$, and receives sparse binary reward $\mathcal{R} \in \{0,1\}$ on success (Wang et al., 18 May 2026).

The framework is motivated by three criticisms of existing memory-augmented VLM agents. First, most recent systems, including Reflexion, SkillRL, XSkill, and various web or GUI agents, are text-centric: they summarize trajectories, extract skills or rules, and refine or merge skills over time through a strong teacher LLM or VLM, while treating visual observations as transient inputs. Second, this design is described as poorly matched to spatial decision making because converting layout to one-dimensional text discards metric and topological information; rules such as “don’t push box into a corner” do not encode which corners in a particular layout are dangerous. Third, delayed textual feedback is sparse and abstract, and external teachers increase API cost, limit autonomy, and introduce a mismatch with the idea of an autonomous RL agent that learns only from environment interaction.

AtlasVA therefore sets four design goals. It keeps memory visually grounded, preserving spatial structure instead of compressing it into text; provides dense, coordinate-aligned feedback for RL by encoding spatial priors directly in visual form; enables teacher-free learning by deriving memory and shaping signals from trajectory statistics and simple grid heuristics rather than external LLMs; and unifies perception, memory, and optimization by reusing the same visual atlases as prompts to the VLM and as potential functions for reward shaping.

3. Three-layer Visual Skill Memory

The core representational object is a three-layer Visual Skill Memory (VSM). At each time step, the policy receives an augmented observation

$$\tilde{o}_t = \big[ M_{\text{danger}}, M_{\text{affinity}}, \mathcal{E}_{vis}, \mathcal{S}_{text}, o_t \big],$$

where $M_{\text{danger}}$ and $M_{\text{affinity}}$ are RGB heatmaps, $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$0 is a retrieved set of visual exemplars, $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$1 is a set of textual skill rules, and $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$2 is the current RGB observation with instruction (Wang et al., 18 May 2026).

Layer 1 consists of spatial heatmaps over discrete coordinates $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$3. The danger map $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$4 represents the probability or risk of deadlock or failure if the agent ends there. The affinity map $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$5 represents how often successful trajectories visit $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$6, and is described as a value-like statistic related to progress toward goal. In grid worlds such as Sokoban and FrozenLake, the grid is natural; in 3D navigation and manipulation, the environment is projected to 2.5D, with floor $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$7-$$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$8 discretization for navigation and tabletop discretization for manipulation, while $$\mathcal{M} = \langle \mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle,$$9 is handled as task metadata. The final maps are rendered as pure red and pure green heatmaps and are passed to the VLM as separate <image> tokens rather than being overlaid on the main RGB frame.

Layer 2 consists of visual exemplars. The exemplar pool stores RGB screenshots from past trajectories, tagged as positive cases near success and negative cases immediately before irreversible deadlock or failure. Retrieval uses DINOv2 embeddings and cosine similarity:

$t$0

The pool is capped, with examples given as 3 positive and 3 negative exemplars and a maximum of 4 images injected per prompt. DINOv2 is chosen for better patch-level geometry matching, and eviction is FIFO or recency-based.

Layer 3 consists of symbolic text skills. These are static textual heuristics derived from environment specifications rather than generated or refined by external LLMs. They encode general task rules, strategy principles, and mistakes to avoid, and are intended to provide high-level logical scaffolding complementary to the two visually grounded layers.

4. Self-evolving atlases and reward shaping

AtlasVA constructs danger and affinity atlases from two sources: static heuristics derived from environment layout, and trajectory statistics gathered from rollouts (Wang et al., 18 May 2026). The affinity heuristic often comes from BFS distance to goal converted into a monotonic potential, while the danger heuristic comes from structural cues such as corners in Sokoban, holes in FrozenLake, and walls or hazards in other environments.

From training batches, trajectories are partitioned into failed trajectories $t$1 and successful trajectories $t$2. The batch danger map counts failed terminal coordinates:

$t$3

where $t$4 is the terminal coordinate of failed trajectory $t$5. The batch affinity map accumulates normalized visitation frequencies along successful trajectories:

$t$6

Historical statistics are updated with an EMA

$t$7

and final maps are blended with heuristics using

$t$8

where $t$9 increases from $o_t$0 to $o_t$1 over epochs. The reported qualitative evolution is that at step 0 the maps are blank, while by step 200–400 the danger map lights up dead ends and the affinity map traces plausible paths, including sub-goals.

These atlases are also used for shaping rewards. The shaped reward is

$o_t$2

with sparse environment reward $o_t$3. The danger penalty is

$o_t$4

and the affinity gain is defined by the potential

$o_t$5

The affinity term is explicitly described as potential-based, matching the classical form $o_t$6 up to $o_t$7, while the danger term is heuristic and intentionally modifies the policy in favor of safe paths.

The optimization loop uses PPO with GAE on the shaped rewards. The base model is Qwen2.5-VL-3B-Instruct, with actor learning rate $o_t$8, critic learning rate $o_t$9, batch size 128, mini-batch 32, and EMA decay $\pi_\theta(a_t \mid o_{\le t})$0. The observation-space extension is written as

$\pi_\theta(a_t \mid o_{\le t})$1

and policy improvement feeds back into better trajectories, better atlases, and a tighter perception-memory-optimization loop.

5. Experimental results, ablations, and limitations

AtlasVA is evaluated on Sokoban, FrozenLake, 3D embodied navigation using AI2-THOR and EmbodiedBench, and 3D robotic manipulation in PrimitiveSkill on ManiSkill3, where the tasks are Place, Stack, Drawer, Align, and a new Swap task (Wang et al., 18 May 2026). Training uses only training environments, and memory is not exposed to test environments to avoid leakage.

Benchmark or summary	AtlasVA	Comparison reported in the paper
Overall success rate	0.93	GPT-5: 0.69; o3: 0.71; VAGEN: 0.78
Sokoban	0.79	Qwen2.5-VL-3B zero-shot: 0.14; GPT-5: 0.70; VAGEN: 0.61
FrozenLake	0.83	GPT-5: 0.77; VAGEN: 0.71
3D Navigation average	0.86	VAGEN average: 0.79
PrimitiveSkill	1.00 on all five tasks	GPT-5 on Swap: 0.55; Qwen-72B: 0.33

The reported interpretation is that the strongest gains appear on spatially intensive tasks. In Sokoban, the gain is attributed to strong spatial requirements and irreversibility. In PrimitiveSkill, the perfect scores are used to argue for the utility of 2.5D projection and visual priors in complex manipulation. The paper also reports that a 3B AtlasVA agent outperforms proprietary and open-source baselines, including many much larger models.

Learning-curve analysis further emphasizes reward density and modality alignment. In Sokoban, the text-only baseline with only Layer 3 stalls around 0.25 success, whereas AtlasVA climbs to approximately 0.8 success in approximately 140 training steps. In PrimitiveSkill, AtlasVA converges quickly to 1.0 success while the baseline plateaus at approximately 0.6. Ablation studies show large regressions when removing the VSM, removing the heatmap layer, removing the exemplar layer, freezing the atlases to heuristics only, or removing dense visual rewards. The paper’s stated reading is that global spatial guidance, local concrete examples, self-evolution from trajectory statistics, and dense shaping are each important contributors.

The framework has three explicitly stated limitations. First, it depends on 2D or 2.5D discretization, so extremely large or high-dimensional continuous environments may challenge scalability. Second, GridState coordinates and internal simulator state are used during training to build maps and compute shaping rewards; this is privileged information, even though it is not used at test time. Third, current 2.5D projection is presented as effective for tabletop manipulation and relatively simple navigation, but more complex ego-centric 3D robotics with dense occlusions remains future work. The listed future directions are scaling to more complex 3D manipulation with occlusions, integrating more advanced planning with atlas priors, and learning richer structured spatial representations beyond grids.

6. Relation to adjacent visual-agent and workflow formalisms

A plausible conceptual connection links AtlasVA to the Autonomous Visualization Agent paradigm introduced in “AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making” (Liu et al., 2023). AVA defines an agent that understands high-level natural-language goals, perceives visualization outputs, and iteratively refines parameters in a visualization → perception → action loop. Its architecture separates visual perception, action planning, and memory, and it distinguishes heuristic-centric planning from LLM-centric planning. This suggests a shared design motif: visually conditioned iterative control rather than one-shot text generation.

The relation is not identity. AVA is framed around visualization systems such as volume rendering, scatterplots, and dimensionality-reduction plots, and uses GPT-4V for perception and planning. AtlasVA, by contrast, is an RL framework for VLM agents in embodied or grid-based environments, with atlases that are simultaneously prompts and potential functions. Even so, the AVA design space helps situate AtlasVA within a broader class of perception-driven agents that close the loop between visual evidence and action selection.

A second adjacent line is “ATWL: A Formal Language for Representing, Comparing, and Reusing Visual Analytics Workflows” (Andrienko et al., 25 May 2026). ATWL introduces a domain-agnostic, declarative language with eight artifact types—entities, features, arrangements, visualisations, patterns, models, knowledge, and specifications—and eight transform intents—define-unit, characterise, contextualise, visualise, abstract, build-model, generate-knowledge, and assess. This suggests a possible formal substrate for any future “atlas” of reusable workflow knowledge, although ATWL itself is a workflow-representation language rather than a VLM agent architecture.

7. Astronomical usage: the ATLAS Virtual Research Assistant

In time-domain astronomy, AtlasVA denotes the ATLAS Virtual Research Assistant, a production system in the ATLAS survey pipeline that performs preliminary eyeballing on the clean transient data stream (Stevance et al., 11 Jun 2025). It sits after quality cuts, Sherlock context cross-match, and a CNN real-bogus score threshold at rb_pix < 0.2, and it computes two probabilities with Histogram-Based Gradient Boosted Decision Tree Classifiers: $\pi_\theta(a_t \mid o_{\le t})$2 for Real versus Bogus and $\pi_\theta(a_t \mid o_{\le t})$3 for Galactic versus Extragalactic. These outputs define a transient-agnostic two-dimensional score space in which “Good” alerts occupy the region $\pi_\theta(a_t \mid o_{\le t})$4, $\pi_\theta(a_t \mid o_{\le t})$5.

Operational ranking is defined by a geometric score toward the “ideal” coordinate $\pi_\theta(a_t \mid o_{\le t})$6:

$\pi_\theta(a_t \mid o_{\le t})$7

with $\pi_\theta(a_t \mid o_{\le t})$8 in production for ranking. Alerts with VRA score $\pi_\theta(a_t \mid o_{\le t})$9 enter the main extragalactic candidate list, a separate Galactic flag is defined through distance to $\mathcal{T}$0 with $\mathcal{T}$1, and visit-dependent auto-garbage thresholds remove likely bogus alerts from human review. Scores are recomputed up to 15 days after first alert as new light-curve data arrive.

The reported metrics are Recall at rank $\mathcal{T}$2 and AuRaK rather than accuracy or F1 score. Relative to ranking by the upstream CNN RB score alone, the VRA ranking is markedly steeper: the baseline RB-only ranking reaches 95% recall of Good events at approximately 35% of the list and 99% recall at more than 60%, with AuRaK $\mathcal{T}$3; the day-1 VRA ranking reaches 95% recall at the top 15% and 99% at the top 25%, with AuRaK $\mathcal{T}$4; and the day-$\mathcal{T}$5 models reach 95% recall at the top 5% and 99% at the top 20%, with AuRaK $\mathcal{T}$6. In production, the system is reported to have reduced eyeballing workload by 85% with loss of follow-up opportunity $\mathcal{T}$7, while also supporting automatic spectroscopic follow-up through the Lesedi telescope.

The coexistence of these two usages makes “AtlasVA” a rare cross-domain label shared by a visually grounded RL architecture and an astronomical alert-ranking assistant. In both cases, however, the name is attached to systems that replace or reduce text-only or purely manual decision-making with structured, operationally useful representations: visual atlases in one case, and a geometric real-versus-galactic score space in the other.