Minimal Sufficient Pixel Set (MSPS)

• Minimal Sufficient Pixel Set (MSPS) is defined as the smallest subset of pixels or neural activations that retain sufficient information for target tasks like 6-DoF camera pose estimation.
• MSPS methodology utilizes non-maximum suppression, thresholding based on reliability scores, and delta debugging to prune irrelevant or redundant data.
• Integrating MSPS in vision tasks accelerates computations and improves model interpretability by focusing on the most informative and discriminative evidence.

A Minimal Sufficient Pixel Set (MSPS) is the smallest subset of pixels, image features, or neural activations sufficient to achieve a target downstream task, such as 6-DoF camera pose estimation or faithful model prediction explanations. MSPS methods aim to prune irrelevant, noisy, or redundant visual input, isolating only the most informative evidence as dictated by models and objective sufficiency criteria. Recent progress in both geometric vision and interpretable AI has seen the formalization and empirical validation of MSPS concepts in tasks ranging from camera localization (Altillawi, 2022) to model explanation for deep neural networks (Khadka et al., 22 Feb 2026).

1. Formal Definitions

For geometric vision, the MSPS is concretely defined as the smallest collection of 2D image pixels whose scene coordinate predictions and reliability scores enable accurate pose recovery via traditional Perspective-n-Point (PnP) plus RANSAC. Formally, for an image $I \in \mathbb{R}^{H \times W \times 3}$, network $f$ produces:

• Pixelwise reliability score $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$
• Dense scene-coordinate map $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$

The MSPS is:

$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$

where $s(p) = \tilde{Z}(p)$, and $\tau$ is a threshold tuned for minimal cardinality while ensuring robust 6-DoF pose estimation (Altillawi, 2022).

In explanation, (Khadka et al., 22 Feb 2026) formalizes the MSPS as the minimal set of representation units $S^\star \subset U$ such that keeping only $S^\star$ (zero-masking all others) preserves model prediction $\hat{c}$. MSPS is 1-minimal if removal of any single unit ruins sufficiency:

• Sufficient: $f$0 with $f$1
• 1-Minimal: $f$2

2. Reliability and Discriminability Metrics

In geometric MSPS construction, the reliability score $f$3 is assigned via a reference-guided training loss. Specifically, the network is supervised to concentrate reliability mass on keypoints that coincide with projections of a Structure-from-Motion (SfM) 3D sparse model. The loss used is the cosine similarity between the predicted and reference heatmaps across all $f$4 patches:

$f$5

where $f$6 is binary and indicates keypoint projections (Altillawi, 2022). As such, the learned score tightly correlates with scene parts that are geometrically discriminative.

In explanation tasks, sufficiency is evaluated by directly measuring whether the outcome is preserved when only a subset's activations are retained. Thus, reliability is not externally supervised but inherently validated via prediction preservation (Khadka et al., 22 Feb 2026).

3. Construction Algorithms for MSPS

Geometric Localization

The pipeline at inference consists of: a) Local maxima extraction via 2D non-maximum suppression on reliability map b) Thresholding, keeping only pixels above $f$7 c) Collection of $f$8 tuples for pose computation

The value $f$9 is tuned to yield $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$0–$$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$1 correspondences depending on the scene. This selection obviates further combinatorial search, as the network's training guides it to activate only the most informative pixels (Altillawi, 2022).

Explanation/Saliency

For neural explanations, delta debugging is adapted to minimize the set of required units. The algorithm branches based on the linearity of the classifier head:

• For interacting units (e.g., ViTs, nonlinear heads): recursively partition and test subsets, eliminating unnecessary sets using the DD algorithm, resulting in $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$2 or $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$3 complexity in worst-case
• For non-interacting units (linear heads): units are tested and pruned in a single pass in $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$4

The end result is a uniquely minimal sufficient set $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$5 of final-layer units (Khadka et al., 22 Feb 2026).

4. Integration with Downstream Tasks

Camera Pose Estimation

After selection, the MSPS yields a compact set of 2D–3D correspondences $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$6 which are then supplied to PnP plus RANSAC for camera pose recovery. Downstream speedup is significant: While traditional pipelines may run RANSAC on thousands of matches, MSPS reduction to $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$7–$$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$8 correspondences enables hypothesis set evaluation to execute $$\tilde{Z} = f_{\text{key}}(I) \in [0,1]^{H \times W}$$9 faster ($$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$0 ms for $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$1 vs. $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$2 ms for $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$3), with an $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$4 ms network forward pass (Altillawi, 2022).

Saliency and Explanations

For vision model explanations, $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$5 is mapped back to an image heatmap. Each unit's effect on the output logit is measured by masking the unit, the difference in logit ($$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$6) is normalized to yield weights $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$7, and the final heatmap is constructed. The upsampled, normalized map yields saliency regions deemed minimally sufficient and maximally compact (Khadka et al., 22 Feb 2026).

5. Empirical Performance and Comparative Results

Localization

PixSelect (Altillawi, 2022) demonstrates that MSPS-based localization outperforms prior methods (e.g., DSAC*, PixLoc) at significantly lower point counts without pose priors or reference 3D models at test time. On Cambridge Landmarks, median translation/rotation errors with $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$8 high-confidence pixels are $$\hat{Y} = f_{3D}(I) \in \mathbb{R}^{H \times W \times 3}$$9 m/$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$0 (King’s College), $$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$1 m/$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$2 (Old Hospital), surpassing prior art by up to $$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$3 in translation error. Using lower-confidence pixels of the same count dramatically degrades accuracy, indicating the necessity and efficacy of selecting the “right” pixels.

Explanation and Saliency

DD-CAM (Khadka et al., 22 Feb 2026), defining MSPS as a minimal sufficient set in activation space, outperforms seven leading CAM saliency approaches across faithfulness and localization:

• CNNs (ImageNet): ADCC$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$4 (vs $$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$5), Average Drop$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$6 (vs $$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$7), Coherency$$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$8 (vs $$\operatorname{MSPS}(I) = \{ p_i = (u_i,v_i) \mid s(p_i) \geq \tau,\, p_i \text{ survives NMS} \}$$9)
• ViTs: Average Drop$s(p) = \tilde{Z}(p)$0 (vs $s(p) = \tilde{Z}(p)$1), ADD$s(p) = \tilde{Z}(p)$2 (vs $s(p) = \tilde{Z}(p)$3), Inc$s(p) = \tilde{Z}(p)$4 (vs $s(p) = \tilde{Z}(p)$5)
• ChestX-ray14: IoU$s(p) = \tilde{Z}(p)$6 ($s(p) = \tilde{Z}(p)$7 over best baseline), Precision$s(p) = \tilde{Z}(p)$8 ($s(p) = \tilde{Z}(p)$9), Recall$\tau$0, most compact saliency with Regions$\tau$1

This suggests that MSPS-based saliency produces more faithful and succinct interpretability artifacts than traditional methods.

6. Ablations, Limitations, and Qualitative Analysis

Ablation studies in PixSelect (Altillawi, 2022) show that indiscriminate pixel selection (including low-confidence or ambiguous regions such as sky, trees, or reflective surfaces) leads to poor pose estimation and outliers. Conversely, MSPS maps are concentrated on semantically and geometrically discriminative structures (such as building edges or corners). Results confirm that sufficiency must be paired with minimality for maximal reliability.

In DD-CAM (Khadka et al., 22 Feb 2026), minimality and sufficiency are strictly enforced by set-based masking. Deviations from these constraints either inflate the explanation (lose compactness) or fail to guarantee decision preservation.

A plausible implication is that for both localization and interpretability domains, enforcing minimal sufficiency enhances both statistical efficiency and robustness of downstream tasks.

7. Research Impact and Theoretical Significance

The formalization and empirical validation of MSPS shifts focus from exhaustive processing to efficiency and reliability. By connecting geometric reliability (PixSelect) and explanation minimality (DD-CAM), MSPS offers a unified abstraction for evidence pruning in both physical scene understanding and neural representation analysis.

For geometric localization, this enables significant acceleration of matching and hypothesis testing, demonstrating that full-image or agnostic keypoint methods are suboptimal under practical constraints.

For vision model interpretability, MSPS grounds explanation in necessary and sufficient evidence, encoding both sparsity and invariance properties.

The convergence of these concepts across applications suggests a methodological bridge between high-precision geometric vision and formal, testable interpretability, with minimality and evidence sufficiency as central organizing principles (Altillawi, 2022, Khadka et al., 22 Feb 2026).