LangXAI: Unifying XAI and Vision

• LangXAI is a unified framework that fuses saliency-based attributions with GPT-4 Vision’s text generation to produce clear, human-understandable explanations for visual tasks.
• It employs a two-block architecture, first extracting task-specific saliency maps and then synthesizing explanations tailored for classification, detection, and segmentation.
• Preliminary evaluations show high semantic alignment with expert annotations via metrics like BERTScore, enhancing transparency and trust for end-users.

LangXAI is a framework that integrates Explainable Artificial Intelligence (XAI) with advanced vision models to generate textual explanations for visual recognition tasks. It is designed to address an understanding gap that persists when end-users with limited domain knowledge in artificial intelligence and computer vision are asked to interpret conventional XAI outputs such as saliency maps or feature attributions. The framework furnishes text-based explanations for image classification, object detection, and semantic segmentation by combining saliency information, model predictions, and, when available, ground truth labels with a Large Vision Model (LVM), specifically GPT-4 Vision. Preliminary results report enhanced plausibility, with high BERTScore across tasks, supporting a more transparent and reliable AI framework on vision tasks for end-users (Nguyen et al., 2024).

1. Problem formulation and motivation

Traditional XAI methods for vision tasks, including saliency maps and feature attributions, require substantial AI and computer-vision expertise for interpretation. This constrains their usefulness for typical end-users, particularly in settings such as healthcare or banking, where users may need to understand why a model produced a decision without being able to decode pixel-level or attribution-based visualizations directly (Nguyen et al., 2024).

LangXAI is motivated by the goal of bridging this understanding gap through automatically generated, human-readable textual explanations. Rather than replacing model introspection, it translates introspective signals into natural language that explains why a model produced a given output. In this sense, LangXAI is oriented toward end-user transparency: it aims to expose model reasoning in a form that does not depend on expert mediation.

A common misconception is that textual explanation in this setting is equivalent to generic image captioning. LangXAI is more constrained than that. Its explanations are conditioned on model outputs, saliency maps, and structured prompts that ask the LVM to reason about prediction evidence, reliability, and possible confusions, thereby tying generated language to model-specific explanatory context rather than to unrestricted scene description (Nguyen et al., 2024).

2. Two-block architecture and end-to-end workflow

The framework is organized into two principal blocks. The first block performs saliency map extraction with XAI. The second block performs text-based explanation with a Large Vision Model. This decomposition makes explicit the separation between visual attribution and linguistic synthesis (Nguyen et al., 2024).

In the first block, the workflow begins when a user uploads an image and selects a task: classification, segmentation, or detection. A task-specific vision model is then selected. Examples given include Swin Transformer v2 for image classification; DeepLabv3-ResNet50 and ResNet101 for semantic segmentation; and Faster R-CNN and YOLOX for object detection. The user or system also selects an XAI method. The framework supports both gradient-based methods—GradCAM, GradCAM++, SeCAM, HiResCAM, and G-CAME—and perturbation-based methods—AdaSISE, RISE, D-RISE, and D-CLOSE. The selected XAI algorithm is run on the image and model to generate a saliency map that highlights the regions deemed important for the prediction.

In the second block, the framework combines the original image, the saliency map, the model’s prediction, and the ground truth label if available. This bundle is passed, together with a structured prompt, to GPT-4 Vision. The prompt specifies the saliency map focus, the predicted label, and a request to compare prediction and ground truth, reason about potential errors or confusions, and explain the visual evidence. GPT-4 Vision then generates a natural-language explanation intended for non-experts. The web-app interface presents this process in an end-to-end manner and is designed to guide users step-by-step, showing the image, the overlaid saliency, and the generated explanation (Nguyen et al., 2024).

3. Saliency-language fusion and task adaptation

The central mechanism in LangXAI is data fusion between visual saliency and symbolic task context. By combining the saliency map with the model prediction and, when available, the ground truth label, the framework prompts the LVM to ground its explanations in both visual evidence and verbal reasoning. This creates a bridge between what parts of the image the model focuses on and why those regions are relevant to the decision (Nguyen et al., 2024).

The explanation format is adapted to the task. For classification, the generated text describes which image regions support the predicted class. For object detection, it explains why detected bounding boxes correspond to certain objects and how the model’s focus overlaps with the predicted class. For semantic segmentation, it discusses the shape, position, and context of predicted masks and their alignment with the ground truth.

This task adaptation is significant because the explanatory burden differs across vision tasks. Classification explanations are typically class-evidence oriented; detection explanations must relate spatial localization to class identity; segmentation explanations must address region extent, boundary placement, and contextual consistency. LangXAI uses structured prompting to preserve these distinctions while maintaining a unified framework. The automation of this process eliminates the need for human experts to interpret saliency maps on behalf of end-users (Nguyen et al., 2024).

4. Model stack, interface design, and modularity

LangXAI operates as a composite system spanning task models, XAI methods, and an LVM. Its computer-vision layer uses state-of-the-art backbone models for each task, including Swin Transformer v2 for classification, DeepLabv3+ResNet for segmentation, and YOLOX and Faster R-CNN for detection. Its explainability layer spans both gradient-based and perturbation-based XAI algorithms. Its language layer is GPT-4 Vision, described as a multimodal LLM capable of processing images and prompts to synthesize explanations with high human alignment (Nguyen et al., 2024).

This modular arrangement is one of the framework’s defining properties. The paper characterizes LangXAI as scalable and modular, with the capacity to be adapted to various backbones, XAI methods, vision tasks, and future LVMs. That modularity implies that the framework is not tied to a single attribution technique or a single vision backbone; rather, it treats explanation generation as a higher-level orchestration problem over interchangeable components.

The user interface is not incidental to the framework. Figure 1 is described as a user-friendly web-app interface that guides users step-by-step and presents explanations in an end-to-end way. Figure 2 separates the pipeline into white blocks for saliency extraction and gray blocks for LVM-enabled explanation generation. This design reinforces the framework’s stated aim of making technically grounded explanations accessible without requiring users to parse raw attribution artifacts themselves (Nguyen et al., 2024).

5. Evaluation protocol and reported performance

The evaluation reported for LangXAI is based on 5 randomly selected samples per task drawn from ImageNetv2 for classification, TTPLA for segmentation, and MS-COCO for object detection. For each sample, a domain expert produces a reference explanation that serves as ground truth for text evaluation. The generated explanations are then assessed with BLEU, METEOR, ROUGE-L, and BERTScore. The evaluation emphasizes precision so that explanations are accurate rather than merely verbose (Nguyen et al., 2024).

The reported results are as follows:

Task	BLEU	METEOR	ROUGE-L	BERTScore
Classification	0.2971	0.5122	0.5196	0.9341
Semantic Segmentation	0.2552	0.4741	0.4714	0.8594
Object Detection	0.2754	0.4904	0.4911	0.9093

The paper emphasizes BERTScore as the most informative metric because it captures semantic similarity through contextualized embeddings rather than relying only on surface-form overlap. The reported BERTScore values are high across all three tasks, especially for classification at 0.9341, which is interpreted as indicating that generated explanations are semantically very close to expert ground truth. Slightly lower scores for segmentation and detection are attributed to the greater complexity of those explanations (Nguyen et al., 2024).

Because the study uses 5 randomly selected samples per task and explicitly presents the results as preliminary, a plausible implication is that the evaluation establishes proof of concept rather than exhaustive validation. The reported evidence supports plausibility and semantic alignment, but also suggests that broader-scale benchmarking would be necessary to characterize robustness across larger and more heterogeneous datasets.

6. Qualitative behavior, contributions, and significance

The qualitative account of LangXAI states that its explanations capture not only what region in the image was salient but why the model relied on those regions. In detection and segmentation settings, explanations may refer to object positions, possible confusion with background or overlapping objects, and comparison with ground truth to determine whether the model captured the relevant region, missed content, or included noise. This extends the explanatory scope beyond visual highlighting toward explicit verbal reasoning about evidence and possible failure modes (Nguyen et al., 2024).

The paper presents several contributions. LangXAI is described as the first framework to unify XAI-generated saliency with LVM textualization in a pipeline for all major vision tasks. It is also characterized as unified and accessible, since it enables end-users, including non-technical users, to access model reasoning without relying on technical intermediaries. Its explanations are described as human-centric, with an emphasis on transparency and trustworthiness, and its evaluation combines quantitative and qualitative validation.

The broader significance of the framework lies in its reformulation of explainability for vision systems as a multimodal communication problem. Instead of assuming that attribution maps are self-explanatory, LangXAI treats them as intermediate evidence that requires linguistic mediation. This suggests a shift from expert-facing interpretability artifacts toward explanation pipelines that are explicitly targeted at human consumption. In that sense, LangXAI contributes not only a technical workflow but also a particular conception of explainability: saliency is necessary, but not sufficient, for end-user understanding (Nguyen et al., 2024).