Mirror-CFE: Counterfactual Image Explanations

• Mirror-CFE is a counterfactual explanation method that reflects image features across classifier decision boundaries to flip predictions.
• It operates directly in the classifier’s feature space using geometric reflection and an inverse U-Net-based mapping to produce realistic image transitions.
• The approach ensures counterfactual validity, input resemblance, and classifier faithfulness while revealing the model's internal decision structure.

Searching arXiv for Mirror-CFE and closely related counterfactual-explanation work. Mirror-CFE is a counterfactual explanation method for deep image classification models that generates explanations by operating directly in the classifier’s own feature space rather than in an external latent space. Its central construction treats the pairwise decision boundary between a source class and a target class as a geometric “mirror,” reflects the source feature across that hyperplane, and then learns an inverse mapping back to image space while preserving distance relationships. In this formulation, counterfactual validity, input resemblance, and faithfulness to the trained classifier are addressed jointly, and the method also yields step-wise transitions that visualize how features evolve as classification confidence changes (Chowdhury et al., 20 Sep 2025).

1. Problem setting and conceptual basis

Mirror-CFE is designed for the standard counterfactual question for deep image classifiers: what minimal change to an input image would flip the model’s decision from its original class $s$ to a target class $t$ (Chowdhury et al., 20 Sep 2025). In the formulation used by the method, a good counterfactual explanation should be valid, plausible, proximal, sparse, diverse, and faithful. The last of these is treated as the defining requirement: the explanation should reflect exactly what the classifier has learned, including both its feature representation and its decision boundaries.

The method is motivated by limitations attributed to earlier counterfactual pipelines. Earlier methods are described as typically following one of three patterns: exhaustive search in pixel or latent spaces, reliance on external generative models such as VAEs, GANs, or diffusion models, or region editing by example retrieval. In the Mirror-CFE formulation, such approaches usually ignore the actual decision boundaries and feature embeddings of the trained classifier, which can produce unfaithful explanations, including adversarial noise or unrealistic edits, and provide no direct access to the classifier’s own latent geometry.

The distinctive claim of Mirror-CFE is therefore not merely that it produces realistic edited images, but that it explains the intrinsic feature space and decision boundaries learned by the classifier. This suggests that the method should be understood primarily as a classifier-faithful explanation framework rather than as a generic image-editing pipeline.

2. Geometric formulation in classifier feature space

Mirror-CFE formalizes the classifier as a pretrained feature extractor $F:\mathcal{X}\to\mathcal{Z}$, with $\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$ the image space and $\mathcal{Z}=\mathbb{R}^N$ the feature space, together with a final linear layer $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$ and bias $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$. The classifier probabilities are written as

$$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$

For a source class $s$ and target class $t$, Mirror-CFE defines the pairwise boundary parameters

$t$0

so that the hyperplane $t$1 is the pairwise decision boundary between $t$2 and $t$3 (Chowdhury et al., 20 Sep 2025).

Given a source feature $t$4, the full reflection across this hyperplane is

$t$5

Mirror-CFE further introduces a step factor $t$6, producing intermediate feature points

$t$7

This parameterization has an immediate decision-theoretic interpretation. When $t$8, $t$9 is the orthogonal projection of the source feature onto the mirror; when $F:\mathcal{X}\to\mathcal{Z}$0, it is the full reflection. Any $F:\mathcal{X}\to\mathcal{Z}$1 yields a valid counterfactual feature that flips classification to $F:\mathcal{X}\to\mathcal{Z}$2, whereas $F:\mathcal{X}\to\mathcal{Z}$3 yields a semi-factual that remains in class $F:\mathcal{X}\to\mathcal{Z}$4. The method thereby converts counterfactual generation into a controlled traversal of the classifier’s own decision geometry.

3. Inverse mapping and training objective

The feature-space construction alone does not produce images. Mirror-CFE therefore learns an inverse map $F:\mathcal{X}\to\mathcal{Z}$5, together with a small adversarial discriminator $F:\mathcal{X}\to\mathcal{Z}$6, so that $F:\mathcal{X}\to\mathcal{Z}$7 is a realistic image whose classifier feature is $F:\mathcal{X}\to\mathcal{Z}$8 (Chowdhury et al., 20 Sep 2025). The training objective combines five losses.

The classification loss enforces validity in feature space:

$F:\mathcal{X}\to\mathcal{Z}$9

The adversarial term promotes realism:

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$0

The reconstruction loss imposes cycle consistency for real images:

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$1

The feature-reconstruction loss imposes cycle consistency for counterfactual features:

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$2

The triangulation loss encodes proximity and plausibility. For a source image $\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$3, a target image $\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$4, and a generated image $\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$5, the ratio

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$6

is used to constrain relative image-space distances through

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$7

This is implemented as the one-sided hinge loss

$\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$8

For semi-factuals with $\mathcal{X}\subset\mathbb{R}^{C\times H\times W}$9, Mirror-CFE defines $\mathcal{Z}=\mathbb{R}^N$0 relative to a random same-class sample $\mathcal{Z}=\mathbb{R}^N$1, giving $\mathcal{Z}=\mathbb{R}^N$2. The complete triangulation term is

$\mathcal{Z}=\mathbb{R}^N$3

The generator objective is then

$\mathcal{Z}=\mathbb{R}^N$4

with typical settings $\mathcal{Z}=\mathbb{R}^N$5, $\mathcal{Z}=\mathbb{R}^N$6, and $\mathcal{Z}=\mathbb{R}^N$7 or $\mathcal{Z}=\mathbb{R}^N$8 on some datasets. The overall design couples classifier consistency, visual realism, and geometric faithfulness in a single objective.

4. Architecture and optimization procedure

Mirror-CFE fixes the pretrained classifier $\mathcal{Z}=\mathbb{R}^N$9 and does not fine-tune it during explanation generation, explicitly to guarantee faithfulness (Chowdhury et al., 20 Sep 2025). The decoder $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$0 is implemented as a U-Net with skip-connection controllers (SCC) that modulate the flow of high-frequency features from $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$1 to $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$2 through a small bottleneck, while a discriminator $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$3 evaluates realism.

Training alternates between generator and discriminator updates. The procedure samples real images and latent counterfactual points $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$4, computes $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$5, and $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$6, updates $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$7 with Adam at learning rate $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$8, and then updates $\mathbf{W}\in\mathbb{R}^{N\times|\mathcal{C}|}$9 to maximize $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$0. Reported training schedules are approximately $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$1–$\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$2 epochs on MNIST, Fashion-MNIST, and Blood-MNIST, and approximately $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$3 epochs on CelebA-HQ.

A notable architectural constraint is that the method requires a GAP-based architecture to compute CAM masks in the SCC module. Vision Transformers, which are described as lacking GAP or meaningful CAMs in this setting, would therefore require approximate attentional priors. This limitation is not incidental: it follows from the specific way Mirror-CFE reuses the classifier’s own internal structure to preserve faithfulness.

5. Evaluation protocol and empirical results

Mirror-CFE is evaluated against PGD, REVISE, CEM, ExpGAN, and C3LT on four datasets: MNIST, Fashion-MNIST, Blood-MNIST, and CelebA-HQ (Chowdhury et al., 20 Sep 2025). The reported metrics are Validity, Denoised Validity, Proximity measured by mean $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$4 distance, Sparsity measured by LPIPS, Realism measured by FID, and $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$5Fail, the fraction of test cases for which no counterfactual is found.

The paper reports results for two operating points: the first valid counterfactual at $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$6, and the full reflection at $\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$7.

Dataset	First valid CFE ($\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$8)	Full reflection ($\mathbf{b}\in\mathbb{R}^{|\mathcal{C}|}$9)
MNIST	$$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$0, LPIPS $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$1, FID $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$2, D.Val. $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$3, Val. $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$4, Fail $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$5	$$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$6, LPIPS $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$7, FID $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$8, D.Val. $$\mathbf{p}(\mathbf{z}) \;=\;\sigma\bigl(\mathbf{W}^\top \mathbf{z} + \mathbf{b}\bigr)\;\in\Delta^{|\mathcal{C}|-1}.$$9, Val. $s$0, Fail $s$1
F-MNIST	$s$2, LPIPS $s$3, FID $s$4, D.Val. $s$5, Val. $s$6, Fail $s$7	$s$8, LPIPS $s$9, FID $t$0, D.Val. $t$1, Val. $t$2, Fail $t$3
Blood-MNIST	$t$4, LPIPS $t$5, FID $t$6, D.Val. $t$7, Val. $t$8, Fail $t$9	$t$00, LPIPS $t$01, FID $t$02, D.Val. $t$03, Val. $t$04, Fail $t$05
CelebA-HQ	$t$06, LPIPS $t$07, FID $t$08, D.Val. $t$09, Val. $t$10, Fail $t$11	$t$12, LPIPS $t$13, FID $t$14, D.Val. $t$15, Val. $t$16, Fail $t$17

Across these experiments, Mirror-CFE is reported to achieve the highest validity and denoised validity among the compared methods while maintaining competitive proximity and perceptual similarity, without producing adversarial noise. The distinction between the first valid crossing and the full reflection is also significant: the former isolates the minimal class-flipping perturbation along the mirror direction, whereas the latter shows the endpoint of the geometric reflection procedure.

6. Interpretability, scope, and limitations

Mirror-CFE provides interpretability through step-wise transitions $t$18 generated by varying $t$19 from $t$20 to $t$21 (Chowdhury et al., 20 Sep 2025). These sequences begin at the original image, pass through the decision boundary, and end at the reflected feature, thereby revealing which features evolve as the target-class confidence changes. Reported examples include changes in strokes for MNIST, cell shape for Blood-MNIST, and mouth openness for CelebA-HQ.

The method also supports latent-trajectory analysis by plotting the confidence $t$22 and the average pixel difference $t$23 as functions of $t$24. This allows the transition point where the target confidence crosses $t$25 to be identified explicitly. For multi-class models, the reflection point can be refined with L-BFGS to ensure a true confidence swap between $t$26 and $t$27, and the resulting $t$28 can be visualized with t-SNE to verify that it lies in the target cluster.

The method’s reported advantages are true faithfulness through reuse of the same $t$29, $t$30, and $t$31; animated explanations rather than a single static counterfactual; robustness to adversarial shortcuts, reflected in high denoised validity; and a unified treatment of counterfactual, semi-factual, and factual explanations. Its stated limitations are the dependence on GAP-based architectures for CAM masks and the observation that high-resolution fidelity is governed by skip connections through $t$32, so very high resolution may require richer style priors. Proposed future directions are adaptation to Transformer-based classifiers via learned attention masks, extension of the mirror-based reflection concept to structured-data classifiers, and incorporation of nonlinear “mirrors” through local second-order approximations.

A common misunderstanding would be to treat Mirror-CFE as merely a generator-assisted image-editing method. Its defining property is narrower and more technical: it constructs explanations by reflecting representations in the classifier’s own feature space and by decoding those reflected points back to images under explicit constraints designed to preserve classifier consistency, visual realism, and distance structure. This suggests that its contribution lies as much in explanation geometry as in image synthesis.