SemaLens: Semantic Assurance for Perception
- SemaLens is a semantic assurance framework that uses vision-language models to bridge the gap between natural language requirements and DNN-based perception outputs.
- It integrates CLIP-like multimodal models to monitor, test, and debug perception systems by translating raw sensor data into human-understandable semantic concepts.
- The framework supports safety-critical applications by employing modules for runtime monitoring, synthetic test generation, and diagnostic explanation to ensure alignment with system requirements.
Searching arXiv for the named concept and associated paper to ground the article in current literature. SemaLens, short for Semantic Analysis of Visual Perception using large Multi-modal models, is a VLM-based semantic assurance layer for DNN-based perception systems in safety-critical settings. Introduced as the perception-side component of the broader “Fighting AI with AI” framework, it uses vision-LLMs—especially CLIP-like multimodal foundation models—as an interpretable semantic layer over perception pipelines so that engineers can reason about, test, explain, and monitor those systems in terms closer to requirements and safety goals than raw pixels or internal activations (Mavridou et al., 25 Nov 2025). Rather than constituting a single algorithm, SemaLens is organized around three main functions—Monitor, Test, and Analyze, Explain, and Debug—with an additional image-generation capability for semantically diverse test synthesis.
1. Definition and problem setting
SemaLens was introduced to address what the paper describes as the semantic gap between high-level requirements and low-level network representations. In the targeted assurance setting, requirements are formulated in natural language and refer to concepts such as pedestrian, traffic cone, metallic, rectangular, rainy, or nighttime, whereas DNN-based perception systems operate on raw pixels and latent features that are difficult to interpret, verify, and trace back to those requirements. The resulting mismatch creates obstacles for traditional verification, runtime monitoring, coverage assessment, and debugging (Mavridou et al., 25 Nov 2025).
The framework is motivated by AI-enabled safety-critical systems, especially aerospace and autonomous driving, where perception is often the principal source of uncertainty. The paper identifies several reasons conventional approaches struggle in this setting: DNNs are opaque and hard to analyze formally; test coverage for high-level semantic features is difficult to define; runtime monitoring is difficult when the system is not explicitly symbolic; and there is a persistent mismatch between requirement language and network internals.
SemaLens addresses this by using a VLM embedding space as a semantic proxy. The central premise is that modern VLMs have learned a large vocabulary of human-understandable visual concepts from large-scale image-text data. This makes it possible to assess perception behavior using the same kind of vocabulary that appears in requirements, operational design documents, and safety cases. A plausible implication is that SemaLens shifts assurance artifacts from pixel-level or class-label-only analyses toward concept-level reasoning.
2. Position within the REACT–SemaLens assurance pipeline
Within the paper’s overall architecture, SemaLens is paired with REACT. REACT operates on the requirements side, using LLMs to bridge informal natural-language requirements and formal specifications. SemaLens operates on the perception side, taking requirements, concepts, and formalized properties and applying them to image sequences, datasets, or model representations (Mavridou et al., 25 Nov 2025).
The paper describes the combined workflow as an end-to-end assurance flow:
- informal requirement
- structured requirement
- formal property
- test cases / monitor
- perception evaluation on images or videos
In this decomposition, REACT defines what the system should do, while SemaLens helps determine whether the perception component is seeing the right things and whether its behavior aligns with those requirements. The relationship is not merely sequential. SemaLens Test and SemaLens Analyze, Explain, and Debug can both use the vocabulary of high-level concepts obtained during requirements elicitation or from operational design documents, while SemaLens Monitor can consume formal temporal properties produced by REACT Formalize.
This division of labor is significant because it ties semantic perception assurance to the same artifacts used for systems engineering. The paper’s broader thesis is that AI can help assure AI systems: LLMs reduce ambiguity and support formalization on the requirements side, while VLMs supply a semantic reference layer on the perception side. SemaLens is therefore best understood not as a perception model, but as an assurance interface between requirement-level intent and perception-level evidence.
3. Monitoring logic and temporal conformance checking
The Monitor component uses VLMs for spatial and temporal reasoning over image sequences or videos. Its purpose is to extract semantic concepts from frames, evaluate predicates derived from requirements, and then determine whether a temporal property is satisfied or violated over the sequence (Mavridou et al., 25 Nov 2025).
The paper presents a concrete workflow. A natural-language requirement is first formalized by REACT into LTLf, then parsed into a Deterministic Finite Automaton (DFA). For each frame, predicates such as on_path and cone_encounter must be evaluated. This is done using a CLIP model: the image is embedded, the caption corresponding to a predicate is embedded, similarity is computed, and if similarity exceeds a threshold the predicate is considered true. The paper gives an explicit example threshold of 0.4. The DFA is then run over the resulting predicate valuations.
The example requirement is:
“[REQ-LIV-002]: Once the rover is navigating a designated path, it shall continue to move and successfully complete the segment by reaching the traffic cone, i.e., the rover must demonstrate non-blocking behavior toward this goal.”
In the example discussed in the paper, the monitor returns True from the third image onward.
The monitoring mechanism supports both offline and online use. Offline, it can search past unlabeled videos for unusual or dangerous sequences, such as accident clips or risk scenarios. Online, it can monitor perception at runtime and flag deviations from requirements. This suggests that SemaLens is intended to serve not only as a post hoc analysis tool but also as a runtime assurance mechanism.
A key technical feature is that the monitor reasons over semantically meaningful predicates rather than raw detector outputs alone. This changes the monitoring unit from low-level perceptual events to concept-grounded temporal behavior, which is closer to requirement language. At the same time, the paper notes that VLMs are still struggling to extract complex spatial relationships from images, so the reliability of predicate evaluation remains an open challenge.
4. Testing, semantic coverage, and image generation
SemaLens includes two related testing-oriented capabilities: Img Generate and Test. Together they support semantically meaningful dataset analysis and synthetic test generation (Mavridou et al., 25 Nov 2025).
The Img Generate module uses text-conditional diffusion models to generate semantically diverse test images and videos. Its stated purposes are to create test inputs consistent with natural-language requirements, generate semantically varied cases under given preconditions, and support perception testing under unusual or hard-to-collect scenarios. The paper explicitly connects this to REACT Generate Test Cases: REACT provides test sequences satisfying temporal specifications, and SemaLens Img Generate turns them into semantically diverse videos. In the paper’s formulation, REACT ensures requirement coverage, while diffusion-based generation broadens semantic coverage.
The Test module defines coverage metrics over image datasets using semantic features extracted by a VLM. An image is said to cover a feature if the similarity between the image embedding and the text embedding of a feature description exceeds a user-chosen threshold. This enables questions such as whether there are enough rainy-night scenes, whether pedestrians, cones, occlusions, or reflective surfaces are sufficiently represented, and which semantic features in the operational design domain are underrepresented.
The paper distinguishes two testing modes:
| Mode | Basis | Purpose |
|---|---|---|
| Black-box testing | Unlabeled dataset analysis | Compute coverage of relevant concepts and identify gaps |
| White-box testing | Mapping model embedding space into CLIP semantic space | Inspect model behavior with respect to high-level concepts |
In white-box mode, SemaLens first maps a perception model’s embedding space into the CLIP semantic space and then measures coverage through that lens. In black-box mode, it analyzes the dataset directly. The significance of this distinction is methodological: black-box testing addresses dataset adequacy, whereas white-box testing addresses the semantic support actually exercised in a specific model.
The paper emphasizes that test adequacy for perception systems should be measured in semantic terms, not only by dataset size or pixel diversity. A plausible implication is that SemaLens reframes dataset quality as coverage over requirement-relevant concepts rather than coverage over classes alone.
5. Analyze, explain, and debug
The Analyze, Explain, and Debug component, abbreviated SemaLens AED, is the framework’s most explicitly diagnostic function. It uses a VLM as a semantic reference model and aligns another vision model’s embeddings with the VLM’s space so that concepts in the VLM space become human-readable proxies for what the target model may be encoding (Mavridou et al., 25 Nov 2025).
The paper describes several capabilities enabled by this alignment:
- Explanation: determine whether the model uses sensible concepts for its decisions.
- Debugging: localize whether errors arise from the vision encoder or from downstream logic.
- Semantic heatmaps: statistically summarize the strength of concepts across behavior.
- Runtime detection: flag adversarial or unsafe inputs when their semantic profile deviates from expected patterns.
The illustrative example is a model classifying an image as a truck. SemaLens can check whether supporting concepts such as metallic and rectangular are also present. If they are absent, the decision may be suspicious or brittle. Likewise, when a model misclassifies an input, the semantic mapping may help determine whether the encoder extracted the wrong concepts or whether the prediction head misused otherwise reasonable concepts.
This capability is important because it changes explainability from a purely saliency-oriented or latent-space-oriented exercise into concept-level auditing. The system does not merely ask which pixels were influential; it asks whether the concepts implicated in a decision are semantically coherent. That makes the debugging signal more legible for engineers working from requirement documents and safety cases rather than from feature maps alone.
The paper also positions AED as relevant to runtime assurance, because deviations in semantic profile can be treated as evidence of adversarial behavior or unsafe perceptual conditions. This suggests a dual role: post hoc diagnosis during development and anomaly detection during operation.
6. Inputs, outputs, assumptions, and limitations
Depending on the module, SemaLens accepts natural-language requirements or concept descriptions, image sequences or videos, unlabeled image datasets, target perception model outputs or embeddings, user-defined semantic concepts and thresholds, and optionally formal temporal properties from REACT. Its outputs include satisfaction or violation of temporal properties, flagged sequences or anomalies, semantic coverage metrics, generated test images or videos, explanations or semantic heatmaps, and debugging cues about likely failure sources (Mavridou et al., 25 Nov 2025).
The paper attributes several assurance tasks to SemaLens: runtime monitoring, offline anomaly detection, semantic robustness testing, coverage analysis, explanation and debugging, alignment of perception behavior with safety requirements, and creation of evidence for safety cases. In certification-oriented settings, this is positioned as supporting evidence rather than as a complete solution.
The paper is explicit about several assumptions and open challenges. First, VLM predicate evaluation relies on thresholds, such as the example value 0.4, which makes the method sensitive to calibration and concept ambiguity. Second, the approach depends on the quality of the chosen concept vocabulary: if the concept set is incomplete or poorly chosen, monitoring and testing may miss important failures. Third, white-box AED assumes that a target vision model can be meaningfully aligned with the VLM embedding space, and the paper states that this alignment is nontrivial and can be imperfect. Fourth, because SemaLens is integrated with REACT, ambiguities in the original requirement can propagate unless REACT’s human-in-the-loop validation resolves them.
The standards context is also carefully delimited. The paper notes that DO-178C remains the baseline for non-AI software, while AI components require emerging guidance such as SAE G-34. SemaLens contributes semantic evidence for assurance, but it does not by itself solve the certification problem.
A common misconception would be to treat SemaLens as a replacement for the perception DNN itself. The paper instead presents it as a semantic reference layer over perception: it does not eliminate the opacity of the target DNN, but supplies a more interpretable interface for monitoring, testing, and explanation. Another likely misconception is that it provides fully solved visual reasoning; the paper explicitly cautions that VLMs are still imperfect at extracting complex spatial relationships from images.
7. Significance
SemaLens is significant because it redefines perception assurance around human-understandable concepts. Instead of treating safety assurance as a direct analysis of raw pixels, latent features, or benchmark labels, it uses CLIP-like semantic embeddings to bridge requirement language and perception behavior (Mavridou et al., 25 Nov 2025).
Within the “Fighting AI with AI” framework, this gives SemaLens a precise role: it is the perception-side mechanism by which concept vocabulary, temporal logic, coverage reasoning, synthetic generation, and diagnostic explanation become operational. Its contribution is therefore architectural as much as algorithmic. It offers a way to connect formal properties, semantic test cases, runtime monitoring, and debugging to the same conceptual vocabulary.
This suggests a broader shift in assurance methodology. If requirements are written semantically and VLMs can operationalize those semantics over images and videos, then monitoring and testing can be expressed in the same language as the safety argument. The paper does not claim that this closes the assurance problem completely. Rather, it presents SemaLens as a promising way to close a long-standing semantic gap that has made DNN perception difficult to verify, interpret, and trace to system-level requirements.
