Semantic-Aware Reconstruction Error (SARE)

• Semantic-Aware Reconstruction Error (SARE) is a framework that conditions reconstruction error on semantic cues, ensuring that residuals capture meaningful discrepancies rather than raw pixel differences.
• Various formulations—including latent semantic roles, caption-guided reconstructions, and gated semantic fusion—demonstrate its versatility across unsupervised role labeling, image novelty detection, and AI-generated video forensics.
• Empirical studies report significant improvements in detection metrics such as ACC and AUC, highlighting SARE's impact on mitigating attribution mismatches and enhancing forensic accuracy.

Semantic-Aware Reconstruction Error (SARE) denotes a family of reconstruction-based formulations in which the discrepancy between an input and its reconstruction is mediated by semantic information rather than interpreted as a purely raw residual. In the literature considered here, this idea appears in several distinct but related forms: latent semantic roles that condition argument reconstruction in unsupervised semantic role labeling (Titov et al., 2014), saliency-informed interpretations of reconstruction error for novelty localization (Feeney et al., 2021), caption-guided reconstruction differences for diffusion-generated image detection (Kang et al., 13 Aug 2025), and reconstruction-guided semantic fusion for AI-generated video detection (Chen et al., 3 Jun 2026). The unifying theme is that reconstruction error becomes informative only when coupled to a semantic variable, semantic representation, or semantic conditioning signal.

1. Conceptual scope and major formulations

SARE is not a single canonical metric shared unchanged across domains. Rather, the surveyed literature uses semantically conditioned reconstruction in different ways depending on the object being reconstructed and the target inference problem. In one line of work, semantic roles are the latent structure that makes reconstruction of argument fillers possible; in another, captions define the semantic condition under which an image is reconstructed; in a third, patch-level semantic features gate video reconstruction errors before temporal modeling. A related novelty-localization study does not introduce a named SARE method, but it directly motivates semantic weighting by showing that raw reconstruction error is often poorly aligned with the pixels that actually matter for reconstruction loss (Titov et al., 2014).

Paper	Domain	Semantic-aware mechanism
(Titov et al., 2014)	Unsupervised semantic role labeling	Roles condition argument reconstruction
(Feeney et al., 2021)	Image novelty localization	Saliency motivates weighting raw error
(Kang et al., 13 Aug 2025)	Diffusion-generated image detection	Caption-guided reconstruction shift
(Chen et al., 3 Jun 2026)	AI-generated video detection	Gated fusion of error and semantics

Across these formulations, the semantic component serves different technical functions. It can act as a latent variable that carries meaning needed for successful reconstruction, as an external conditioning signal that reshapes the reconstruction itself, or as a context-dependent weight that determines which residuals should influence downstream classification. This suggests that SARE is best understood as a design principle: reconstruction discrepancies are treated as meaningful only insofar as they are aligned with semantics.

2. Saliency, attribution, and the case against raw pixelwise reconstruction error

A central motivation for SARE arises from the analysis of novelty localization in deep autoencoders. That work considers the common assumption that pixels with high reconstruction error indicate unfamiliar or novel regions of an image. Using a convolutional autoencoder for MNIST, trained on digits 0–4 as “normal” and evaluated on digits 5–9 as “novel,” the study defines the reconstruction error map and mean reconstruction loss as

$r(x) = (x - g(f(x)))^2,$

and

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$

Pixel importance is approximated with a gradient-based saliency map following Simonyan et al.:

$$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$

The evaluation compares reconstruction error maps and saliency maps qualitatively and through two quantitative correspondence measures: Top-K agreement and maximum distance to best match (Feeney et al., 2021).

The reported results indicate weak correspondence. For top-5 pixels, 26% of normal images and 19% of novel images have majority agreement between reconstruction error and saliency, while 74% and 81%, respectively, do not. For top-10 pixels, only 19% of normal images and 8% of novel images have majority agreement. Spatial mismatch is also substantial: the mean maximum distance to best match is 8.28 pixels for normal top-5 sets and 9.25 pixels for novel top-5 sets on 28×28 MNIST images; for top-10 sets the corresponding means are 8.14 and 8.90 pixels. The paper further reports that squaring saliency improves numerical similarity to squared reconstruction error, reducing normalized-map MSE from 0.0303 to 0.0071 on normal images and from 0.0305 to 0.0072 on novel images, but this does not resolve the underlying attribution mismatch (Feeney et al., 2021).

The significance for SARE is direct. Reconstruction error highlights output-space discrepancies, whereas saliency measures sensitivity of the overall loss to perturbations of the input. Because encoder-decoder mappings mix information across pixels, a high error at one output location need not imply that the corresponding input pixel is semantically important. The paper therefore undermines the simplistic equation between “large residual” and “meaningful novelty.”

A semantic-aware formulation suggested by these findings is to weight per-pixel error by an importance signal. With per-pixel error $e_i = r_i(x)$ and semantic weights $w_i$, the score can be written as

$\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$

Because reconstruction error is squared, a normalized squared-saliency weighting proposed in the summary is

$w_i = \frac{s_i(x)^2}{\sum_{j=1}^n s_j(x)^2},$

with localization map

$m_i = w_i e_i.$

A plausible implication is that such weighting suppresses spurious boundary-dominated residuals and emphasizes pixels that are truly loss-critical.

3. Semantic roles as latent structure in reconstruction-error minimization

An earlier and structurally different use of semantic-aware reconstruction appears in unsupervised semantic role labeling. The model in (Titov et al., 2014) has two components: an encoding component that predicts semantic roles from rich syntactic and lexical features, and a reconstruction component that predicts argument fillers from the predicate, roles, and the other arguments. The encoder is a feature-rich log-linear model,

$$p(\mathbf{r} \mid x, \mathbf{w}) \propto \exp(\mathbf{w}^T \mathbf{g}(x, \mathbf{r})),$$

and in experiments it is factorized over individual arguments as independent logistic regression classifiers producing posteriors $\mu_{is} = p(r_i = s \mid x, \mathbf{w})$. The feature inventory reuses 14 feature patterns from Johansson and Nugues (2008), including dependency path, syntactic function, and relative position.

The reconstruction component is a tensor factorization model over predicate–role–argument interactions. Let $\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$0 be the embedding of argument lemma $\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$1, $\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$2 a predicate–role-specific projection, and $\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$3 a word-specific bias. The conditional model for reconstructing argument lemma $\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$4 is

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$5

The global score

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$6

makes explicit that role-conditioned pairwise interactions, not isolated argument predictions, define the reconstruction pressure (Titov et al., 2014).

The ideal unlabeled-data objective integrates over latent roles:

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$7

For tractability, the model uses a mean-field approximation over roles and a negative-sampling surrogate:

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$8

The complete joint objective is

$\bar{r}(x) = \frac{1}{n} \sum_{i=1}^n r_i(x).$9

Here semantic awareness is intrinsic rather than post hoc. Roles directly control reconstruction scores through mixtures $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$0, so reconstruction error is already conditioned on semantic structure. Empirically, the model achieves the best reported F1 on English and German in the cited comparison sets: 82.8 on CoNLL-2008 English and 81.3 on SALSA/CoNLL-2009 German, while typically inducing 4–6 roles per predicate (Titov et al., 2014). This formulation shows that SARE can denote an expected negative log-likelihood over semantically structured latent variables, not merely a transformed residual map.

4. Caption-guided SARE for diffusion-generated image detection

The explicit contemporary use of the name SARE appears in diffusion-generated image detection. The formulation in (Kang et al., 13 Aug 2025) starts from the hypothesis that fake images tend to exhibit higher similarity to their captions than real images do, because fake images are generated from text prompts that explicitly specify content, whereas real images contain visual detail that short descriptions often fail to capture. The method therefore reconstructs an image under caption guidance and measures the semantic shift induced by that reconstruction.

Let the input image be $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$1, the generated caption be $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$2, and its text embedding be $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$3. The image is encoded into a latent $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$4 by the VAE encoder of Stable Diffusion v1.4. The forward noising process is

$$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$5

with $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$6 and $$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$7. Reverse denoising uses classifier-free guidance:

$$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$8

followed by the DDIM update

$$s(x) = \Big| \frac{\delta \bar{r}}{\delta x} \Big|.$$9

After decoding the final latent to $e_i = r_i(x)$0, SARE is defined as the absolute pixel-wise difference

$e_i = r_i(x)$1

The main detector extracts image features $e_i = r_i(x)$2 with CLIP: ViT-L/14, semantic features $e_i = r_i(x)$3 with ResNet50 where $e_i = r_i(x)$4, and fuses them with cross-attention,

$e_i = r_i(x)$5

before a fully connected head outputs $e_i = r_i(x)$6. The decision rule is $e_i = r_i(x)$7 (Kang et al., 13 Aug 2025).

The empirical motivation is the failure of raw reconstruction-error assumptions under out-of-distribution generators. The paper notes that methods such as DIRE assume fake images have smaller reconstruction errors than real images because original and reconstructed fakes are drawn from similar generative distributions, but this assumption can fail when the reconstruction model is Stable Diffusion v1.4 and the fake images come from unseen generators such as ADM or BigGAN. SARE shifts the focus from model-specific artifacts to the caption-image relationship.

On GenImage, trained on the Stable Diffusion v1.4 subset and tested across all eight generator-specific subsets, DRCT + SARE with BLIP captions reaches average ACC 93.17% and average AUC 98.15%, compared with DRCT at 88.81% ACC and 95.66% AUC. The OOD gains are especially pronounced on ADM, where AUC rises from 88.47% to 94.87%, and on BigGAN, where ACC rises from 79.76% to 92.05% and AUC from 90.30% to 97.51%. In cross-dataset evaluation, DRCT + SARE reaches 88.11% average ACC on CommunityForensics and 70.53% average ACC with 84.96% average AUC on ForenSynths. The main configuration uses strength $e_i = r_i(x)$8, guidance scale $e_i = r_i(x)$9, $w_i$0, images resized to 512 on the longer side for reconstruction, and 224×224 encoder inputs (Kang et al., 13 Aug 2025).

This formulation is notable because the semantic component is introduced upstream, at reconstruction time. The residual is still an absolute pixel-wise difference, but it becomes semantic-aware because the reconstruction is caption-conditioned. The discriminative signal is therefore not merely “how well the image reconstructs,” but “how much the image changes when forced through a caption-specified semantic bottleneck.”

5. Reconstruction-guided semantic fusion in AI-generated video detection

ReConFuse extends the semantic-aware idea from still images to videos by treating reconstruction error as a temporally organized forensic signal and explicitly aligning it with semantic video features. A pretrained WF-VAE reconstructs the input video $w_i$1,

$w_i$2

The method uses signed, not absolute or squared, reconstruction error. Reconstruction-error maps are resized and patch-embedded to produce error tokens

$w_i$3

while semantic tokens come from the visual encoder of XCLIP,

$w_i$4

After linear projections $w_i$5 and $w_i$6, a reconstruction-guided gate is computed as

$w_i$7

and the fused token is

$w_i$8

The fused sequence is fed to a Mamba-based temporal module, yielding a video representation $w_i$9 concatenated with a global semantic context $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$0:

$\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$1

Training uses binary cross-entropy on the video-level label (Chen et al., 3 Jun 2026).

The paper does not explicitly define SARE, but it provides a consistent formalization. A plausible SARE map, explicitly identified as an inference in the summary, broadcasts the gate-derived semantic weight to the error field:

$\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$2

with $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$3, and corresponding frame- and video-level aggregations

$\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$4

This interpretation treats the gate as a learnable semantics-aware weighting of residual evidence.

The experiments are conducted on GenVideo and GenBuster. On GenVideo one-to-many, ReConFuse improves Accuracy from 79.56% to 85.56% and F1 from 71.75% to 85.03%, with AUROC approximately 92.72%. On GenVideo many-to-many, it achieves average Recall 93.44%, Accuracy 91.93%, F1 91.49%, and AUROC 95.82%. On GenBuster, it reaches average Accuracy 96.66%, F1 91.23%, and AUROC 99.37%. Ablations indicate that error-only input is weak in isolation, with Accuracy approximately 43.65% despite AUROC approximately 80.40%, whereas XCLIP-only semantics yields Accuracy approximately 79.56% and F1 approximately 77.14%; the full model reaches Accuracy approximately 88.33%, F1 approximately 88.18%, and AUROC approximately 95.15%. Signed error performs best, outperforming both $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$5 and $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$6, and Mamba outperforms mean pooling, Transformers, ResNet50, LSTM, and 1D CNN among the tested temporal modules, with the reported Mamba-based configuration at approximately 129.1M parameters (Chen et al., 3 Jun 2026).

The broader implication is that in video forensics, semantics is not only a conditioning variable but also a dynamic selector of which residual structures should propagate through temporal modeling. Error patterns become forensic evidence only after local semantic alignment and sequence-level integration.

6. Limitations, misconceptions, and open directions

A recurrent misconception in reconstruction-based analysis is that raw residual magnitude already identifies the semantically relevant part of an input. The novelty-localization results contradict this directly: reconstruction error maps and saliency maps exhibit low Top-K agreement and substantial spatial mismatch, especially on novel images (Feeney et al., 2021). A second misconception is that reconstruction-based fake detection is inherently generator-agnostic. The image-detection results show that raw reconstruction error assumptions can invert under out-of-distribution generators such as ADM or BigGAN, which is precisely why caption-guided SARE is introduced (Kang et al., 13 Aug 2025).

The surveyed works also show that “semantic-aware” is not a single architectural choice. In (Titov et al., 2014), semantic awareness is realized by latent role distributions that determine reconstruction likelihoods. In (Kang et al., 13 Aug 2025), it arises because captions condition the reconstruction itself, after which the absolute difference image is used as the signal. In (Chen et al., 3 Jun 2026), semantics acts as a learnable gate over signed video residuals before Mamba-based temporal aggregation. The literature therefore supports a plural notion of SARE rather than a uniquely defined metric.

The limitations are similarly domain-specific. The novelty-localization study evaluates only MNIST, one convolutional autoencoder derived from Abati et al. (2019), and only squared error; it does not test alternative attribution methods such as Integrated Gradients, SmoothGrad, or Grad-CAM, and it reports no formal statistical tests (Feeney et al., 2021). The semantic-role model assumes given argument identification, factorizes the encoder over arguments, and lacks explicit modeling of verb senses, which the authors identify as a source of possible improvement (Titov et al., 2014). The diffusion-image detector depends on caption quality and on computationally nontrivial caption-guided image-to-image diffusion; it also does not report extensive robustness testing for compression, resizing, or adversarial perturbations beyond standard augmentations (Kang et al., 13 Aug 2025). ReConFuse does not separately quantify robustness to compression, noise, or occlusion, and the narrative identifies heavy compression, strong noise, motion blur, semantic misalignment, and very short clips as settings in which reconstruction errors may become less informative (Chen et al., 3 Jun 2026).

Open directions follow naturally from these constraints. The novelty-localization analysis explicitly recommends further work on whether saliency maps could replace reconstruction error maps for both detection and localization, and the SARE-oriented summary suggests testing weighting schemes such as $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$7 versus $\mathrm{SARE}(x) = \sum_{i=1}^n w_i e_i.$8, smoothed saliency, broader datasets, and evaluation against ground-truth novelty masks (Feeney et al., 2021). The image-detection work suggests that semantic shift under caption guidance is a transferable cue across generators, but caption dependence remains a structural bottleneck (Kang et al., 13 Aug 2025). The video work indicates that semantically gated residuals benefit from temporal persistence, which suggests further study of alternative temporal backbones and more explicit alignment objectives, although no such auxiliary losses are part of ReConFuse itself (Chen et al., 3 Jun 2026).

Taken together, these works establish SARE as a reconstruction-centered framework in which semantics is not ancillary metadata but part of the mechanism that determines whether an error signal is meaningful. Whether implemented through latent roles, saliency-derived weights, caption-guided reconstructions, or gated multimodal fusion, the defining claim is the same: reconstruction error becomes substantially more useful when it is conditioned on, aligned with, or filtered by semantic structure.