Semantic Sensitivity Maps

• Semantic Sensitivity Maps are visual tools that depict how semantic content changes under controlled perturbations, highlighting robustness, uncertainty, and contextual shifts.
• They employ diverse methods such as 2D projections, token- and region-level diagnostics, spatial heatmaps, and gradient-based relevance to capture subtle variations.
• Applications span science mapping, text spatialization, robotics navigation, and kernel dependence analysis, offering insights into model behavior and attribution.

Searching arXiv for the cited papers to ground the article and confirm bibliographic details. Semantic Sensitivity Maps are map-like representations that make semantic variation visible under changes of context, perturbation, uncertainty, or dependence structure. Across the cited literature, the term does not denote a single canonical formalism. Instead, it refers to several related constructions: two-dimensional projections of contextual document embeddings for mapping scientific communities (Chambers et al., 2021), image-grounded diagnostics of invariance to lexical edits and sensitivity to meaning changes (Dumpala et al., 2024), confidence-aware spatial heatmaps for open-vocabulary rover navigation (Klein et al., 15 Jun 2026), conformalized uncertainty maps for semantic reach-avoid planning (Sundarsingh et al., 29 Sep 2025), gradient-based relevance maps for the Hilbert–Schmidt Independence Criterion (Pérez-Suay et al., 2016), and stability analyses of two-dimensional text spatializations under corpus, hyperparameter, and seed perturbations (Atzberger et al., 2024).

1. Definitional range and core semantics

The phrase “Semantic Sensitivity Map” is used in several technically distinct ways. In science mapping, it denotes two-dimensional projections of high-dimensional contextual embeddings that capture how scientific documents and the terms within them position and cluster relative to one another across research communities. In VISLA, it denotes visualizations of where an embedding encoder is invariant to lexical edits that preserve meaning and sensitive to edits that alter meaning. In CrossMaps, it denotes a confidence-aware, language-queryable spatial heatmap reflecting how much the current semantic map would change in response to either new observations or small changes in the language query. In conformalized planning, semantic sensitivity is represented by per-cell prediction sets whose size and composition reflect semantic ambiguity. In HSIC-based analysis, sensitivity maps are gradients of a dependence measure with respect to examples and features. In text spatialization, sensitivity is the stability of a two-dimensional layout under changes in data, dimensionality-reduction hyperparameters, and random initialization (Chambers et al., 2021, Dumpala et al., 2024, Klein et al., 15 Jun 2026, Sundarsingh et al., 29 Sep 2025, Pérez-Suay et al., 2016, Atzberger et al., 2024).

Setting	Map object	Sensitivity notion
Science mapping	2D projection of contextual embeddings	Dispersion, clustering, community separability
VISLA	Token- and region-level diagnostic maps	Invariance to lexical edits; sensitivity to semantic edits
CrossMaps	Confidence-aware semantic heatmap	Perturbation by observations or query embedding
Conformal planning	Per-cell label sets and safety buffers	Semantic ambiguity under user-specified coverage
HSIC/RHSIC	Sample- and feature-level gradient maps	Relevance to a dependence measure
Text spatialization	Stability overlays on 2D layouts	Sensitivity to corpus, hyperparameters, seeds

A common thread is that the map is not merely a representation of semantic content; it is a representation of how semantic content responds to controlled variation. This suggests that “semantic sensitivity” functions as an umbrella notion spanning context dependence, robustness, uncertainty, and attribution, rather than as a single standardized metric.

2. Contextual document maps and text spatializations

In scientific text analysis, the central claim is that sensitivity to context is critical because the same surface text can take on multiple and sometimes contradictory specialized senses across distinct research communities. One workflow concatenates titles and abstracts, lowercases them, replaces numbers with <NUM>, tokenizes with the model’s native tokenizer, extracts top-layer token activations of dimension 768, and pools them into document vectors. The models investigated are bert-base-uncased, scibert-scivocab-uncased, and a static word2vec-PubMed baseline. Pooling is decisive: mean pooling over subword tokens, excluding special tokens, is the default and highest-performing approach; mean pooling over “long” subword tokens only, with token length at least five characters, slightly improves performance; [CLS] pooling degrades performance in the unsupervised setting studied. To mitigate anisotropy, pooled embeddings are demeaned as $E' = E - \mu$, where $\mu$ is the sample mean embedding computed over a large random set with $n=100{,}000$. Discriminability is evaluated on Journal of Neurophysiology versus NeuroImage using Euclidean distance in a PCA-matched subspace of $D=100$, with $P@500$, $P@R$, and $MAP@R$ aggregated across $5{,}000$ queries. On Sample A, scibert-scivocab-uncased attains $P@500=0.91$, $P@R=0.69$, and $\mu$0, versus bert-base-uncased at $\mu$1, $\mu$2, and $\mu$3, and word2vec-PubMed at $\mu$4, $\mu$5, and $\mu$6; Sample B is nearly identical. The same framework introduces derived measurements such as Semantic Breadth,

$\mu$7

Semantic Distance,

$\mu$8

Semantic Likelihood via perplexity,

$\mu$9

and projection along cultural continua,

$n=100{,}000$0

The resulting maps show, for example, that “%” and “percent” co-locate, “human” and “cells” separate, and SciBERT produces distinct density peaks for the two neuroscience journals while general-domain BERT and [CLS] pooling blur them (Chambers et al., 2021).

A related but distinct line of work treats semantic sensitivity as layout stability in two-dimensional text spatialization. A text spatialization maps documents into a latent space using VSM, LSI, NMF, LDA, Doc2Vec, or BERT, then projects them to two dimensions with MDS, SOM, t-SNE, or UMAP. Sensitivity is measured against three perturbation classes: input data changes, dimensionality-reduction hyperparameters, and random seeds. The study computes 38,941 layouts and 42,817 layout-pair measurements across 20 Newsgroups, Lyrics, and Seven Categories, then evaluates ten metrics spanning local structure, global structure, class separation, and alignment diagnostics. Aggregated indices are

$n=100{,}000$1

$n=100{,}000$2

and

$n=100{,}000$3

The reported findings are that embeddings improve stability relative to projecting raw DTMs, t-SNE is consistently strong and more stable than commonly assumed for local structure, UMAP is comparatively strong on global structure, SOM is most sensitive to hyperparameters, and MDS shows high seed sensitivity for local neighborhoods. The same study reports statistically significant stability gains from tf-idf weighting in many combinations and improved LDA stability from a topic-based convex-combination projection

$n=100{,}000$4

These results recast a semantic map as a stability object: the map is evaluated not only by what it shows, but by how much it moves when the pipeline changes (Atzberger et al., 2024).

3. Embedding invariance and semantic alteration in VISLA

VISLA formalizes semantic sensitivity as the separation between lexical invariance and semantic variance. Each sample consists of an image $n=100{,}000$5 and a caption triplet: $n=100{,}000$6, a correct caption; $n=100{,}000$7, a semantically equivalent but lexically altered caption; and $n=100{,}000$8, a hard negative that is lexically close yet semantically opposite. Generic VISLA contains 973 samples and is built upon SUGARCREPE; Spatial VISLA contains 640 samples and is built from VSR. Quality controls include allowed lexical changes such as synonyms, antonyms applied consistently, negations, word reordering, and subject–object swaps; exclusion of added visual details; gender neutrality; and the requirement that positives and negative be distinguishable by text alone.

For an encoder $n=100{,}000$9, lexical invariance is

$D=100$0

semantic sensitivity is

$D=100$1

and the sensitivity margin is

$D=100$2

with cosine similarity as the default similarity function. Retrieval is evaluated in two modes. In text-to-text evaluation, one ranks $D=100$3 last among $D=100$4 using pairwise similarities between text embeddings. In image-to-text evaluation for VLMs, one ranks $D=100$5 last relative to $D=100$6 and $D=100$7 using similarities between the image embedding and caption embeddings. The protocol is strictly off-the-shelf and does not use fine-tuning.

VISLA itself evaluates embedding similarities and does not prescribe localization, but it provides concrete procedures for constructing token- and region-level sensitivity maps. Token-level perturbation defines, for each token $D=100$8 in $D=100$9, an invariance impact

$P@500$0

and a semantic sensitivity impact

$P@500$1

where $P@500$2 and $P@500$3. A targeted edit score,

$P@500$4

separates the effect of semantic alteration from mere token removal. Region-level maps may be obtained by contrasting positive and negative alignments, for example

$P@500$5

or by attention differences

$P@500$6

Empirically, all text encoders struggle to disentangle lexical from semantic variations under VISLA’s strict setup. Spatial semantics are particularly vulnerable to lexical distractors. For ULMs on generic VISLA, $P@500$7-$P@500$8 accuracy is consistently higher than $P@500$9-$P@R$0 accuracy, indicating that models prefer the lexically overlapping positive over the lexically distant paraphrase. For spatial VISLA, the pattern weakens or reverses for multiple ULMs; the reported example is STSB-RoBERTa-large with $P@R$1-$P@R$2 and $P@R$3-$P@R$4. Selected ULM results include Angle-Llama-7b-nli-v2 at 78.93% generic and 52.34% spatial, E5-Mistral-7b-instruct at 78.21% and 52.50%, and Instructor-large at 75.03% and 52.81%. For VLMs, image-to-text performance is consistently higher than text-to-text performance; selected figures include XVLM-ITR-COCO at 61.56% T2T generic and 62.38% I2T generic, and 45.16% T2T spatial and 51.09% I2T spatial. The benchmark further notes that larger models or more data do not fix spatial failures, while multi-objective pretraining generally helps more than contrastive-only training. Limitations include reliance on cosine similarity as a proxy, absence of token- or region-level ground-truth labels, possible prompt artifacts in generated paraphrases, and dataset biases inherited from MS-COCO and VSR (Dumpala et al., 2024).

4. Confidence-aware and conformal spatial maps in robotics

In rover navigation, a Semantic Sensitivity Map is a confidence-aware, language-queryable spatial heatmap. CrossMaps builds such maps from RGB image $P@R$5, depth map $P@R$6, and pose $P@R$7 from SLAM, using multi-scale RGB tiling, CLIP ViT-L/14 visual feature extraction, depth back-projection into 3D, and global alignment. The map representation is split into Short-Term Memory and Long-Term Memory. Each STM cell stores a confidence-weighted semantic embedding, an accumulated confidence, a coherence statistic, and a viewpoint-coverage bitmask; LTM stores persistent semantic landmarks promoted from STM. For an observation $P@R$8 with embedding $P@R$9 and confidence $MAP@R$0, fusion updates are

$MAP@R$1

with coherence

$MAP@R$2

Language querying uses CLIP text embeddings. For cell $MAP@R$3, cosine similarity is

$MAP@R$4

and the heatmap is

$MAP@R$5

The map’s sensitivity to query semantics is made explicit by

$MAP@R$6

with $MAP@R$7 and $MAP@R$8, while sensitivity to coherence is

$MAP@R$9

Low coherence is treated as high sensitivity to new evidence, and the map can overlay $5{,}000$0 to highlight unstable semantics. CrossMaps reports qualitative demonstrations rather than benchmarking: representative heatmaps for “plant” and “hammer” show a noisier STM and a sparser, more reliable LTM; no precision/recall, mAP, localization accuracy, or navigation success rates are reported (Klein et al., 15 Jun 2026).

A different robotics formulation treats semantic sensitivity as uncertainty calibrated by conformal prediction. A semantic map discretizes the environment $5{,}000$1 into grid cells $5{,}000$2 with semantic labels in $5{,}000$3 and a per-cell categorical PMF $5{,}000$4. A conformalized semantic map replaces raw labels or PMFs with prediction sets

$5{,}000$5

where $5{,}000$6 is the empirical $5{,}000$7-quantile of calibration scores

$5{,}000$8

These sets satisfy a distribution-free coverage guarantee under exchangeability:

$5{,}000$9

Planning uses class-dependent safety distances $P@500=0.91$0 through the conservative per-cell buffer

$P@500=0.91$1

enforcing

$P@500=0.91$2

The planner alternates between exploitation, using A*, PRM*, or FMT* under the inflated constraints, and exploration, moving toward uncertainty-reducing cells when no feasible path exists. Under mapping coverage, completeness of the exploitation planner, and exploration that sufficiently reduces uncertainty, the algorithm returns a trajectory satisfying

$P@500=0.91$3

In experiments on 61 test scenarios with a Clearpath Husky in Gazebo, for $P@500=0.91$4 the reported mapping success rates are 93.36%, 90.16%, and 86.89%, and mission success rates are 98.36%, 96.72%, and 95.08%, with average path lengths 74.7 m, 71.7 m, and 69.72 m and exploration fractions 19.0%, 18.54%, and 11.2%. Baselines that ignored or heuristically accumulated uncertainty underperformed substantially. Out-of-distribution tests with randomized tree layouts and lighting reduced mapping success to 80.33% at nominal $P@500=0.91$5, while mission success remained 93.44%, illustrating both robustness and the sensitivity of guarantees to exchangeability violations (Sundarsingh et al., 29 Sep 2025).

5. Dependence-theoretic sensitivity maps via HSIC and RHSIC

In kernel dependence analysis, sensitivity maps are gradients of the Hilbert–Schmidt Independence Criterion with respect to inputs, examples, and features. HSIC is defined as the squared Hilbert–Schmidt norm of the cross-covariance operator between RKHS embeddings,

$P@500=0.91$6

with empirical biased estimator

$P@500=0.91$7

For Gaussian RBF kernels,

$P@500=0.91$8

and analogously for $P@500=0.91$9. Sensitivity with respect to kernel entries is

$P@R=0.69$0

By the chain rule, sensitivity to an input coordinate $P@R=0.69$1 is

$P@R=0.69$2

and for the Gaussian kernel,

$P@R=0.69$3

These gradients are aggregated into sample-wise and feature-wise maps,

$P@R=0.69$4

with analogous expressions for $P@R=0.69$5. The interpretation is direct: a sample-feature entry with large gradient magnitude is influential in the dependence relation.

To address the $P@R=0.69$6 memory and computational burden of exact kernels, randomized HSIC uses Random Fourier Features. For $P@R=0.69$7,

$P@R=0.69$8

with $P@R=0.69$9 and $\mu$00. Stacking features gives $\mu$01 and $\mu$02, and the estimator becomes

$\mu$03

Its gradient with respect to $\mu$04 is

$\mu$05

and input gradients follow by chaining through the random-feature Jacobian

$\mu$06

The exact method has time complexity $\mu$07 and memory $\mu$08, whereas RHSIC with sensitivity maps has time complexity $\mu$09 and memory $\mu$10. Under standard random-feature assumptions, both the estimator and the sensitivity maps converge to their exact HSIC counterparts at rate $\mu$11. The paper positions these maps as tools for feature selection, dependence estimation, diagnosis of influential samples, and exploratory causal inference, while stressing that large sensitivities indicate leverage on HSIC rather than causal effect (Pérez-Suay et al., 2016).

6. Shared methodological themes, limitations, and outlook

Across these literatures, semantic sensitivity is operationalized through different observables: dispersion and clustering of contextual embeddings, nearest-neighbor retrieval, layout stability metrics, lexical invariance and semantic sensitivity margins, coherence and confidence in spatial maps, conformal prediction-set size, and gradients of dependence measures. There is no single scalar “semantic sensitivity” metric shared by all formulations. In the science-mapping setting, the absence of a single scalar metric is explicit, with sensitivity inferred from dispersion, map topology, and retrieval performance. VISLA uses several complementary scores such as $\mu$12, $\mu$13, and $\mu$14. Text spatialization aggregates ten metrics into $\mu$15, $\mu$16, and $\mu$17. CrossMaps combines similarity, coherence, and confidence, and exposes derivatives with respect to the query embedding and coherence. Conformal planning uses the size and composition of $\mu$18 and the induced buffer $\mu$19. HSIC-based work uses gradient magnitudes aggregated over samples and features (Chambers et al., 2021, Dumpala et al., 2024, Atzberger et al., 2024, Klein et al., 15 Jun 2026, Sundarsingh et al., 29 Sep 2025, Pérez-Suay et al., 2016).

Several engineering lessons recur. Domain matching matters: SciBERT outperforms general BERT for biomedical document mapping, and the same work emphasizes that domain-matched training data are more important than generic “state-of-the-art” tools. Pooling and normalization materially affect contextual maps: mean pooling over non-special tokens and demeaning are preferred, while [CLS] pooling degrades unsupervised discriminability. Stability assessment benefits from multiple random samples, fixed seeds, recorded PCA/UMAP fit sets and parameters, and sensitivity analyses over perturbations. At the same time, each formulation has distinct failure modes. Transformer-based science maps inherit social and cultural biases from training data and reveal structure but not causality. VISLA has no token- or region-level ground truth and can reflect prompt artifacts and dataset biases. CrossMaps specifies neither the functional form of its semantic gate nor quantitative performance metrics. Conformalized planning depends on exchangeability between calibration and test scenarios and can degrade under out-of-distribution conditions. Exact HSIC sensitivity maps remain memory-intensive, while large-scale text-spatialization studies are computationally expensive.

Taken together, the literature portrays Semantic Sensitivity Maps as a family of methods for turning semantic behavior into inspectable geometry, uncertainty fields, or attribution structures. The shared purpose is diagnostic rather than purely representational: to reveal where semantic structure is robust, where it is brittle, and which perturbations, observations, or variables are most responsible for change.