AnalyzerVLM: Multimodal Model Discovery & Analysis

• AnalyzerVLM is a vision-language model module that analyzes visual, textual, and structured data to propose candidate models and detect anomalies.
• It employs a multi-step control loop combining natural language analysis, code generation, and model proposal to optimize performance.
• Its applications span industrial quality control, video analytics, and logical anomaly detection, providing interpretable, evidence-based outputs.

AnalyzerVLM denotes, in current arXiv usage, both a specific vision-language-model module for automated model discovery and a broader design pattern in which a pretrained VLM serves as an analyzer over mixed visual, textual, and structured context. In "Automated Model Discovery via Multi-modal & Multi-step Pipeline," AnalyzerVLM is the Phase 1 component of a four-phase loop—Model Proposal, Model Fitting, Model Evaluation, and Model Selection—and replaces the human expert by proposing candidate model structures and rough initial parameter guesses (Jung-Mok et al., 30 Sep 2025). Related work uses the same or closely related label for systems in which a VLM performs analysis, explanation, retrieval, or symbolic reasoning over images, plots, and videos rather than only generating a final caption or answer (Jin et al., 3 Jan 2025, Yan et al., 1 May 2025, Piehl et al., 14 May 2026).

1. Definition and conceptual scope

In its narrowest sense, AnalyzerVLM is the proposal module in an agentic discovery pipeline. It receives the current best model $M^*$ or a pool of previous models, the dataset $D$, and a short system prompt $P$, then outputs a small set of candidate model structures such as “LIN × PER” or “PER + SE × LIN” together with rough initial parameter guesses for period, lengthscale, and variance (Jung-Mok et al., 30 Sep 2025). The underlying implementation is an off-the-shelf zero-shot or few-shot VLM such as GPT-4o-mini or Qwen2.5-VL, with no additional learned module and no gradient-based fine-tuning in that work (Jung-Mok et al., 30 Sep 2025).

A broader usage appears in anomaly detection and video analytics. LogicAD is described as showing how a pretrained autoregressive VLM can serve “as an analyzer rather than a generator,” combining VLM-based text extraction, format embedding, and a logic reasoner for anomaly scoring and explanation (Jin et al., 3 Jan 2025). AVAS is described as implementing an Analyzer-VLM for open-ended video analytics by constructing an Event Knowledge Graph and then running an agentic retrieval-generation mechanism over that graph (Yan et al., 1 May 2025). LATERN likewise acts as an AnalyzerVLM by building a validated, image-grounded memory of past events and recursively aggregating textual and visual evidence into coherent anomaly intervals (Piehl et al., 14 May 2026). This suggests that current usage spans both a named module in a discovery pipeline and a broader architectural pattern centered on multimodal analysis.

2. Multi-step control loop and I/O contract

The most explicit formulation of AnalyzerVLM is the multi-step proposal loop in automated model discovery. The overall pipeline alternates four phases each round: Model Proposal by AnalyzerVLM, numerical optimization of kernel parameters, Model Evaluation by EvaluatorVLM using the Visual Information Criterion, and Model Selection by choosing the highest-VIC model (Jung-Mok et al., 30 Sep 2025). Upstream, AnalyzerVLM sees the selected model from the previous round, or top-$k$ models from the pool, together with the context of past analyses and observations; downstream, each proposal is passed to a fitting optimizer and then to EvaluatorVLM (Jung-Mok et al., 30 Sep 2025).

The AnalyzerVLM context $c_t$ may contain a plain-text prompt specifying the task and prior results, a tabular or text-serialized kernel description such as “LIN + SE*(PER + C),” rendered plot images such as matplotlib-generated PNGs, and scalar or array outputs from previously executed code (Jung-Mok et al., 30 Sep 2025). Its action space is partitioned into three subspaces: natural-language analysis $a_t \in \mathcal{L}$, code generation for execution $a_t \in \mathcal{C}$, and model proposal $a_t \in \mathcal{M}$ (Jung-Mok et al., 30 Sep 2025). The analysis action appends a paragraph to the running context; the execute action emits Python code, whose outputs are converted back into tokens or images and appended; the propose action terminates the loop and hands off the candidate models (Jung-Mok et al., 30 Sep 2025).

The process is formalized as a small Markov Decision Process. The state is

$c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$

the next action is sampled from the fixed policy of the pre-trained VLM,

$a_t \sim T(\cdot \mid c_t),$

and terminal action $D$0 yields the final candidate set (Jung-Mok et al., 30 Sep 2025). Algorithm 2 initializes $D$1 and repeats until the token limit is reached or a proposal action is emitted (Jung-Mok et al., 30 Sep 2025). No new loss function is trained for AnalyzerVLM; its behavior is attributed entirely to pre-training, with prompts and few-shot examples used only at inference time (Jung-Mok et al., 30 Sep 2025).

3. External representations and reasoning substrates

A recurring property of AnalyzerVLM-style systems is reliance on explicit intermediate representations beyond the raw VLM forward pass. In the model-discovery setting, the VLM operates over a mixed sequence of text tokens and visual embeddings, and multi-modal attention fuses the current prompt, prior language, and any image patches or plot embeddings; the network “caches its own chain-of-thought within the token stream,” which the paper identifies as the mechanism by which multi-step reasoning emerges (Jung-Mok et al., 30 Sep 2025).

LogicAD uses a markedly different substrate. It begins from a normal reference image $D$2 and query image $D$3, applies GroundingDINO to obtain regions $D$4, and queries a pretrained autoregressive VLM with Guided Chain-of-Thought prompts to obtain multiple textual descriptions per region (Jin et al., 3 Jan 2025). These texts are embedded with text-embedding-3-large, filtered with a Local Outlier Factor detector to reduce hallucinations, reformatted by an LLM into a fixed JSON schema, and embedded again to produce normalized feature vectors $D$5 and $D$6 (Jin et al., 3 Jan 2025). The one-shot anomaly score is the cosine distance

$D$7

A second symbolic layer converts textual descriptions into first-order logic formulas $D$8 and compares them against a normal schema $D$9 augmented with unique-name, functionality, and domain-closure axioms; Prover9 then checks whether $P$0 (Jin et al., 3 Jan 2025).

AVAS uses graph structure as the external substrate. It samples video at 2 FPS, divides it into uniform 3 s chunks, uses Qwen2.5-VL-7B to generate a short textual description per chunk, merges adjacent chunks whose descriptions satisfy $P$1 with $P$2, and then extracts entity-relation triples for each semantic chunk (Yan et al., 1 May 2025). Entity strings are embedded with JinaCLIP and clustered by K-means to create canonical entity nodes, yielding an Event Knowledge Graph $P$3 with event-event temporal relations, entity-entity semantic relations, and entity-event participation relations (Yan et al., 1 May 2025). At query time, tri-view retrieval over events, entities, and frames is fused by weighted Borda counting, and an LLM agent explores the graph with Forward, Backward, Re-Query, and Summarize-and-Answer actions (Yan et al., 1 May 2025).

LATERN uses memory rather than an explicit graph. At time step $P$4, it maintains a sliding buffer of embeddings of the last $P$5 segments, computes normalized center-frame embeddings $P$6, selects up to $P$7 diverse keys by farthest-point sampling, and every $P$8 segments feeds those key frames to a VLM captioner under a strict “What visually happened?” prompt (Piehl et al., 14 May 2026). The summary is then gated using CLIP-based similarity and entropy thresholds, with example thresholds $P$9 and $k$0, before being injected into anomaly scoring (Piehl et al., 14 May 2026).

4. Interpretation and explanation interfaces

AnalyzerVLM-style systems often require explicit evidence interfaces, and two relevant toolkits expose such interfaces for large vision-LLMs. "LVLM-Interpret: An Interpretability Tool for Large Vision-LLMs" presents an interactive application whose conceptual pipeline is Image Input $k$1 Vision Encoder $k$2 Patch Embeddings $k$3 Vision-to-Language Adapter $k$4 LLM $k$5 Answer Tokens, with attention and gradient extraction feeding an explanation engine and UI visualization (Stan et al., 2024). The explanation engine stores cross-modal and self-attention weights $k$6 for each layer and head, computes relevancy maps by transformer relevancy propagation, and computes causal explanations via CLEANN (Stan et al., 2024).

Raw attention visualization in LVLM-Interpret operates on

$k$7

where $k$8 is the number of text tokens and $k$9 is the number of image tokens; attention weights are given by

$c_t$0

Gradient-based relevancy adopts Chefer et al.’s transformer propagation, recursively back-propagating the target-token logit through layers and aggregating scores over image-patch embeddings to produce heatmaps; causal explanation constructs a graph over top-$c_t$1 image and text tokens and identifies a minimal explaining set $c_t$2 of patches whose masking changes the predicted token (Stan et al., 2024). The user interface is Gradio-based and includes chat history, token-level controls, raw-attention viewing, relevancy maps, and causal graphs (Stan et al., 2024).

"VLM-Lens" addresses a different layer of interpretability by extracting intermediate outputs from any layer during the forward pass of open-source VLMs (Sheta et al., 2 Oct 2025). Its core architecture consists of a YAML-configurable entry point, model-specific subclasses of a common BaseModel, a hook manager based on torch.nn.Module.register_forward_hook(), and a SQLite database with schema fields (model_name, arch, image_path, prompt, layer, tensor_dim, tensor_blob) (Sheta et al., 2 Oct 2025). The toolkit supports 16 state-of-the-art base VLMs and over 30 variants, including CLIP, BLIP-2, CogVLM, InternVL, LLaVA, MiniCPM-o, Molmo, Paligemma, Pixtral, and Qwen2-VL (Sheta et al., 2 Oct 2025). Because it stores activations to disk, it can be coupled post hoc to SHAP, LIME, CKA, or TransformerLens-style analyses (Sheta et al., 2 Oct 2025). A plausible implication is that AnalyzerVLM systems operating as analytical agents can benefit from both front-end evidence displays such as LVLM-Interpret and back-end activation extraction such as VLM-Lens.

5. Representative application domains

The AnalyzerVLM pattern appears in model discovery, industrial anomaly detection, logical anomaly detection, open-ended video analytics, and explainable video anomaly detection.

System	Domain	Core output
AnalyzerVLM	Automated model discovery	Candidate kernel structures
LogicAD	Logical anomaly detection	Anomaly score and symbolic explanation
AVAS	Open-ended video analytics	Query answer over ultra-long video
LATERN	Video anomaly detection	Event-level anomaly intervals and explanations
Industrial AnalyzerVLM report	Industrial quality control	Classification/segmentation guidance

In automated model discovery, a full round on the Airline dataset proceeds through analysis and code execution steps: first “visualize data & model,” then analyze residuals, then compute autocorrelation, then compute a Fourier transform, identify a period of approximately $c_t$3 years, and finally propose the kernel “LIN × PER {period:0.35}” (Jung-Mok et al., 30 Sep 2025). Here the AnalyzerVLM role is tightly coupled to scientific model revision rather than direct prediction.

In industrial anomaly detection, the AnalyzerVLM report distinguishes two architectural paradigms: WinCLIP-style sliding-window dense feature extraction and AprilLab-style multi-stage feature alignment with learnable projections (Kakda et al., 19 Jan 2026). For WinCLIP, an input image $c_t$4 is decomposed into overlapping windows, each patch is encoded by frozen CLIP, textual prompts are encoded by the CLIP text encoder, and a dense anomaly heatmap $c_t$5 is computed from cosine similarities between patch and prompt embeddings (Kakda et al., 19 Jan 2026). The report also formalizes compositional prompt ensembles for abnormal and normality prompts and notes a concise-versus-elaborate prompt trade-off: concise prompts yield higher recall but may false-alarm on benign variations, whereas elaborate prompts improve precision on well-defined defect styles but can over-specialize (Kakda et al., 19 Jan 2026).

LogicAD specializes AnalyzerVLM to logical anomaly detection. It reports explanations grounded in minimal contradicting subsets of first-order logic atoms; for the “breakfast-box” example, the minimal explanation is the subset stating that both a nectarine and an apple appear on the left side, violating the normal rule that only one is allowed (Jin et al., 3 Jan 2025). This establishes a version of AnalyzerVLM in which visual analysis is translated into symbolic entailment and human-readable contradiction statements.

AVAS specializes AnalyzerVLM to ultra-long video. It decouples near-real-time index construction from question answering, stores the Event Knowledge Graph in five database tables plus a vector index of raw-frame CLIP embeddings linked back to events, and answers arbitrary natural-language queries by graph retrieval, graph traversal, and final frame checking with a multimodal VLM (Yan et al., 1 May 2025). LATERN specializes it to anomaly detection in temporally evolving video by combining Context-Aware Anomaly Scoring and Recursive Evidence Aggregation, producing event-level explanations such as a single description for a burglary interval rather than inconsistent per-segment descriptions (Piehl et al., 14 May 2026).

6. Empirical behavior, failure modes, and limitations

In automated model discovery, end-to-end performance is reported on seven real time-series. “Ours (GPT-4o-mini)” achieves the lowest average test RMSE, $c_t$6, compared with BoxLM at $c_t$7 and Automatic Statistician at $c_t$8 (Jung-Mok et al., 30 Sep 2025). Ablation shows “– AnalyzerVLM” with test MSE approximately $c_t$9, “– EvaluatorVLM” with test MSE approximately $a_t \in \mathcal{L}$0, and the full pipeline with test MSE approximately $a_t \in \mathcal{L}$1 (Jung-Mok et al., 30 Sep 2025). Restricting AnalyzerVLM to text only increases RMSE by about $a_t \in \mathcal{L}$2, and forcing a single step instead of up to 10 steps increases MSE by about $a_t \in \mathcal{L}$3 on average (Jung-Mok et al., 30 Sep 2025). These results tie AnalyzerVLM performance directly to both multi-modality and multi-step reasoning.

LogicAD reports average AUROC $a_t \in \mathcal{L}$4 and $a_t \in \mathcal{L}$5-max $a_t \in \mathcal{L}$6 on MVTec LOCO AD in the one-shot setting, compared with prior few-shot SOTA AnomalyMoE at AUROC $a_t \in \mathcal{L}$7 and $a_t \in \mathcal{L}$8-max $a_t \in \mathcal{L}$9, and WinCLIP zero-shot at AUROC $a_t \in \mathcal{C}$0 and $a_t \in \mathcal{C}$1-max $a_t \in \mathcal{C}$2 (Jin et al., 3 Jan 2025). On LA-heavy categories of MVTec AD, LogicAD reaches AUROC $a_t \in \mathcal{C}$3 versus WinCLIP at $a_t \in \mathcal{C}$4 (Jin et al., 3 Jan 2025). Its ablation is especially diagnostic: removing Guided CoT reduces AUROC to $a_t \in \mathcal{C}$5, adding ROI only reaches $a_t \in \mathcal{C}$6, adding format embedding reaches $a_t \in \mathcal{C}$7, and adding the logic reasoner yields binary decisions with $a_t \in \mathcal{C}$8 $a_t \in \mathcal{C}$9 (Jin et al., 3 Jan 2025).

For long-video analysis, AVAS achieves $a_t \in \mathcal{M}$0 accuracy on LVBench, $a_t \in \mathcal{M}$1 on VideoMME-Long, and $a_t \in \mathcal{M}$2 on AVAS-100 (Yan et al., 1 May 2025). EKG construction throughput exceeds real-time ingestion: $a_t \in \mathcal{M}$3 FPS on $a_t \in \mathcal{M}$4A100, $a_t \in \mathcal{M}$5 FPS on RTX4090, and $a_t \in \mathcal{M}$6 FPS on RTX3090 for 2 FPS input (Yan et al., 1 May 2025). The paper also notes limitations: deep tree searches beyond depth $a_t \in \mathcal{M}$7 tend to introduce noisy events, the VLM alone may miscount or mislocalize small fast-moving objects, and semantic chunking can over-merge distinct events if textual embeddings are too similar (Yan et al., 1 May 2025).

LATERN reports frame-AUC $a_t \in \mathcal{M}$8 on UCF-Crime, compared with VERA at $a_t \in \mathcal{M}$9; on XD-Violence it reports frame-AUC $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$0 versus $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$1, and AP $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$2 versus $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$3 (Piehl et al., 14 May 2026). On 200 UCF-Crime events, human preference favors LATERN’s event-level explanations in $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$4 of cases, and average explanation length is $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$5 tokens, compared with approximately $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$6–$c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$7 tokens for baselines (Piehl et al., 14 May 2026). The stated limitations are reliance on frozen VLMs, incomplete cue-keyword coverage across domains, and the cost of running large VLMs on each segment for real-time deployment (Piehl et al., 14 May 2026).

Interpretability-focused work provides complementary evidence about failure modes. LVLM-Interpret’s LLaVA case study identifies text-dominant bias, spurious correlations to background text or logos, and ignored regions for small objects such as sunglasses (Stan et al., 2024). Its Table 1 reports grounding accuracy and modality dominance under perturbation: for stable objects with varied phrasing, model accuracy is $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$8 and $c_t=(P,D,M^*,a_1,o_1,\dots,a_{t-1},o_{t-1}),$9 in $a_t \sim T(\cdot \mid c_t),$0 of cases, whereas for color-swapped images with fixed phrasing, model accuracy is $a_t \sim T(\cdot \mid c_t),$1 and $a_t \sim T(\cdot \mid c_t),$2 in $a_t \sim T(\cdot \mid c_t),$3 of cases (Stan et al., 2024). The practical lesson stated there is that raw attention alone can be misleading and should be complemented by causal masks or gradient-based checks (Stan et al., 2024). VLM-Lens adds a tooling limitation of its own: it does not capture attention weights or pre-norm activations by default, has no built-in gradient-based saliency, and can produce large databases, motivating alternative storage backends such as HDF5 or streaming parquet (Sheta et al., 2 Oct 2025).

Across these systems, AnalyzerVLM emerges as a multimodal analysis paradigm built from pretrained VLMs, prompt-time control logic, and explicit intermediate structures such as executed code traces, JSON schemas, logic formulas, event graphs, memory summaries, attention maps, or extracted activations. The common pattern is not a single architecture but an analytical role: ingest heterogeneous evidence, transform it into a structured internal working context, and output proposals, anomaly scores, intervals, or explanations that can be inspected and, in several cases, externally verified (Jung-Mok et al., 30 Sep 2025, Jin et al., 3 Jan 2025, Yan et al., 1 May 2025, Piehl et al., 14 May 2026).