Saliency-Based Sampling Strategy (SBSS)

• SBSS is a strategy that selectively directs sampling and computation to informative regions based on saliency cues rather than uniform or random selection.
• It employs techniques like top-K selection, density estimation, and adaptive resampling to focus on task-relevant features in applications such as object tracking, video detection, and data augmentation.
• Empirical studies show that SBSS improves efficiency and accuracy by enhancing resource allocation and parameter usage while addressing trade-offs in regularization and context preservation.

Saliency-Based Sampling Strategy (SBSS) denotes a class of methods that allocate sampling, computation, or training emphasis according to estimated saliency rather than by uniform, exhaustive, or random selection. In the supplied literature, the term is used both explicitly and implicitly across object tracking, saliency detection, video object detection, supervoxel segmentation, black-box interpretability, reinforcement learning explanation, visualization sampling, and data augmentation. The common operational principle is to direct limited representational or computational resources toward regions, parts, states, or samples deemed informative, discriminative, or task-relevant, while reducing effort on uninformative content (Zhou et al., 2021, Hao et al., 8 Aug 2025).

1. Conceptual definition and scope

In the most direct formulation, SBSS replaces holistic or uniform processing with selective sampling based on saliency evidence. In "Saliency-Associated Object Tracking" (Zhou et al., 2021), the strategy is described as sampling only the most promising and trackable parts of the target: fine-grained saliency mining evaluates each exemplar location, the top-$K$ most salient locations are selected, and tracking then focuses on associating these salient parts rather than all spatial parts. The paper explicitly contrasts this with holistic tracking, which operates on the entire target, and naive part-based tracking, which divides the target uniformly into patches and treats all equally (Zhou et al., 2021).

A related formulation appears in joint salient object detection and camouflaged object detection. "Distribution-Specific Learning for Joint Salient and Camouflaged Object Detection" (Hao et al., 8 Aug 2025) uses SBSS to sample the training set of the SOD task so that the SOD training set is both higher in quality and matched in size to the COD set. In this usage, saliency governs dataset subsampling rather than spatial resampling.

Across the corpus, the same principle recurs under different technical realizations. "Learning to Zoom" (Recasens et al., 2018) introduces a saliency-based distortion layer that learns how best to sample from underlying high-resolution data. DECNN learns a density function $\rho$ and redistributes convolutional receptive fields non-uniformly over the domain (Ip et al., 11 Jun 2026). SALISA performs non-uniform saliency-based input sampling for video object detection (Bejnordi et al., 2022). F-PISA uses a gradient-driven image sub-sampling strategy that focuses computation on pixels with large gradient magnitude (Wang et al., 2015). These formulations differ in modality and objective, but they share the same structural move: saliency determines where detail is retained.

This suggests that SBSS is better understood as a general resource-allocation doctrine than as a single algorithm. A plausible implication is that the strategy becomes especially attractive when the domain is large, the budget is constrained, and the informative signal is spatially or temporally sparse.

2. Core mechanisms and mathematical forms

The supplied papers instantiate SBSS through several recurring mechanisms: saliency scoring, ranking and top-$K$ selection, density-induced warping, adaptive perturbation sampling, and importance-driven subset construction.

In SAOT, saliency is computed from a similarity map between exemplar features $\bm F_x$ and search-image features $\bm F_s$. For exemplar location $(u,v)$, the similarity map is

$$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$

Each exemplar pixel receives a saliency score based on intensity, measured by Peak-to-Sidelobe Ratio, concentration, measured as the reciprocal of the main lobe area, and Gaussian spatial regularization: $$s(\bm S_{(u,v)}) = \gamma(\bm S_{(u,v)}) [c(\bm S_{(u,v)})]^\alpha + \lambda g_{\mu_g,\sigma_g}(u,v).$$ The top-$K$ locations are then selected as salient parts $P_x = \{\bm p_x^k\}_{k=1}^K$ (Zhou et al., 2021).

In "Learning to Zoom" (Recasens et al., 2018), saliency is estimated by a saliency network $\rho$0, with

$\rho$1

and the resampling coordinates are computed by kernel-weighted averaging: $\rho$2

$\rho$3

The output image $\rho$4 is produced by bilinear grid sampling from the original image $\rho$5 (Recasens et al., 2018).

In DECNN, the density function $\rho$6 encodes relative regional importance, and the density-equalizing mapping $\rho$7 redistributes sampling so that the probability of sampling $\rho$8 is proportional to $\rho$9: $K$0 The mapping is derived from a diffusion process,

$K$1

so that salient regions are expanded and non-salient regions are compressed (Ip et al., 11 Jun 2026).

In black-box saliency generation, Bayesian optimisation chooses perturbation location and window size adaptively. Each sample is parameterized by $K$2, and the saliency score is

$K$3

A Gaussian Process surrogate with Matérn kernel models the perturbation-response mapping, and Expected Improvement selects the next perturbation to query (Mokuwe et al., 2020).

In the JoNet setting, SBSS scores each SOD image-GT pair using

$K$4

sorts the pairs in descending order of $K$5, and selects the top $K$6 pairs to form $K$7 (Hao et al., 8 Aug 2025).

These formulations show that SBSS does not require a single saliency definition. Depending on the task, saliency may mean match peakiness, density, detector-derived objectness, perturbation impact, Q-value gap, or agreement between prediction and annotation.

3. Representative implementations across domains

The literature exhibits several distinct implementation patterns.

Domain	SBSS realization	Representative paper
Object tracking	Top-$K$8 salient part selection and graph-based association	(Zhou et al., 2021)
Image resampling	Saliency-driven distortion or warping before the task network	(Recasens et al., 2018)
Convolutional sampling	Density-equalizing mapping guided by learned $K$9	(Ip et al., 11 Jun 2026)
Video object detection	TPS-STN-based non-uniform downsampling guided by detector saliency	(Bejnordi et al., 2022)
Supervoxel segmentation	Saliency-guided adaptive seeding with cluster-specific seed resolution	(Gao et al., 2017)
Scatterplot visualization	Perception-weighted max-min sampling using saliency and density	(Moumoulidou et al., 29 Apr 2025)

In supervoxel segmentation, SSV uses VOCUS2 to compute a saliency map, clusters the point cloud by saliency, and assigns seed resolutions $\bm F_x$0 so that highly salient clusters get smaller $\bm F_x$1 and denser seeds, while less salient clusters get larger $\bm F_x$2 and sparser seeds. The seed resolutions follow

$\bm F_x$3

for clusters sorted by increasing saliency (Gao et al., 2017).

In video object detection, SALISA generates saliency maps from detector outputs, labels small-object boxes with saliency $\bm F_x$4, larger boxes with saliency $\bm F_x$5, and background with saliency $\bm F_x$6. A TPS-STN localization network predicts displaced control points, and the resulting grid non-uniformly preserves detail in salient regions while heavily down-sampling background (Bejnordi et al., 2022).

In scatterplot visualization, PAwS defines a perception weight

$\bm F_x$7

where $\bm F_x$8 is saliency weight and $\bm F_x$9 is density weight. The sample is built greedily by maximizing

$\bm F_s$0

thereby coupling coverage with perceptual significance (Moumoulidou et al., 29 Apr 2025).

This diversity of implementations indicates that SBSS can be inserted at different stages of a pipeline: input formation, feature extraction, graph construction, seed placement, training-set curation, or explanatory summarization.

4. Relation to alternative strategies

The papers repeatedly position SBSS against uniform, exhaustive, and randomly selected baselines.

SAOT contrasts saliency-based selection with holistic tracking and naive part-based tracking. The limitation of naive part-based tracking is explicit: not all patches are equally informative, and non-discriminative patches may have adverse effects (Zhou et al., 2021).

"Learning to Zoom" contrasts saliency-based sampling with uniform downsampling, bounding-box cropping, Spatial Transformer Networks, deformable convolutions, and CAM crops. Its saliency sampler predicts a dense, flexible map rather than a global parametric transformation and degrades gracefully to a uniform sampling strategy under uncertainty (Recasens et al., 2018).

The black-box saliency paper contrasts Bayesian optimisation with exhaustive grid-based perturbation. The BO method adaptively chooses region and window size and requires no access to model parameters or gradients, whereas exhaustive search requires orders of magnitude more queries (Mokuwe et al., 2020).

In RL explanation, HIGHLIGHTS and HIGHLIGHTS-DIV sample states according to importance

$\bm F_s$1

rather than by random selection. The paper’s user study reports that summaries including important states significantly outperformed summaries composed of randomly chosen world-states, while saliency-map overlays had mixed effects (Huber et al., 2020).

In salient object detection divergence modeling, random sampling of one annotation per instance during training is used to expose the model to diverse interpretations, in contrast to training only on majority-voted ground truth or all annotations simultaneously (Tian et al., 2021). Although this is not framed as spatial resampling, it is a sampling strategy driven by saliency subjectivity.

A common misconception is that saliency-guided selection is synonymous with attention visualization. The literature shows a broader scope: in some settings saliency determines where input pixels are sampled; in others it determines which states, which exemplars, which training pairs, or which perturbations are queried.

5. Empirical effects, efficiency, and trade-offs

The empirical record in the supplied papers associates SBSS with improved efficiency, better use of model capacity, or stronger robustness under constrained budgets.

In SAOT, the ablation study reports that SAOT achieves the highest AUC and Precision on OTB2015 $\bm F_s$2 and NFS30 $\bm F_s$3, and that replacing all-part association with saliency-mined association yields gains from $\bm F_s$4 AUC on OTB2015 and from $\bm F_s$5 on NFS30 (Zhou et al., 2021).

In DECNN, the reported effect is parameter efficiency: DECNN maintains high accuracy $\bm F_s$6 even with substantial parameter reduction—up to $\bm F_s$7 reduction of parameters—by focusing sample points on the digit area, and it remains robust under complex geometric variations (Ip et al., 11 Jun 2026).

In SALISA, the detection of small objects is a central result. SALISA with an EfficientDet-D1 detector improves the detection of small objects by $\bm F_s$8, and SALISA + EfficientDet-D2 achieves $\bm F_s$9 mAP on UA-DETRAC at $(u,v)$0 GFLOPs, outperforming vanilla EfficientDet-D3 at similar accuracy and substantially higher cost (Bejnordi et al., 2022).

In the JoNet pipeline, reducing the SOD set from $(u,v)$1 to $(u,v)$2 samples nearly halves the required training time in joint training, and joint training with SBSS and SCJoint improves COD performance on COD10K from $(u,v)$3 to $(u,v)$4 relative to joint training without the architectural modification, while the full JoNet configuration reaches $(u,v)$5 (Hao et al., 8 Aug 2025).

In PAwS, the evaluation reports that PAwS consistently outperformed prior art in producing samples with high perceptual similarity, while ApproPAwS achieves up to $(u,v)$6 speed-ups with minimal loss in visual fidelity (Moumoulidou et al., 29 Apr 2025). In F-PISA, runtime is reduced from approximately $(u,v)$7 ms to approximately $(u,v)$8 ms per image while keeping comparable detection accuracy (Wang et al., 2015).

These results do not imply that saliency guidance is universally beneficial. Several papers also expose trade-offs. "Learning to Zoom" notes the need for regularization via a distance kernel and, in early training, blur to encourage focus on detail-rich regions (Recasens et al., 2018). SALISA requires explicit grid supervision for TPS-STN because downstream detection loss alone may not learn to magnify salient regions (Bejnordi et al., 2022). The RL explanation study reports mixed results for local saliency maps, with the main benefit coming from salient state selection rather than overlay visualization (Huber et al., 2020).

6. Extensions, limits, and research directions

The more recent literature expands SBSS beyond 2D image saliency and beyond direct object localization.

In 3D mesh saliency ground-truth acquisition, view cone sampling replaces single-ray sampling with Gaussian-distributed ray bundles inside a foveal cone, and hybrid manifold-Euclidean constrained diffusion prevents topological short-circuits during saliency propagation. The reported internal consistency rises from $(u,v)$9 to $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$0, and the combination of VCS and HCD improves CC from $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$1 to $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$2, lowers KL divergence from $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$3 to $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$4, and raises sAUC from $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$5 to $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$6 (Zheng et al., 6 Jan 2026).

In data augmentation, GuidedMixup uses saliency maps to pair images whose salient regions minimally conflict and to perform pixel-wise mixing: $$\bm S_{((u,v),(p,q))} = \frac{ \bm F_{x,(u,v)} \cdot \bm F_{s,(p,q)} }{ \|\bm F_{x,(u,v)}\| \|\bm F_{s,(p,q)}\| }.$$7 Here, saliency does not choose input regions for a detector or tracker; it structures pair formation and interpolation weights to preserve important features with low computational overhead (Kang et al., 2023).

In saliency evaluation itself, FN-AUC proposes a negative sampling strategy that selects fixation sets farthest from the ground-truth fixation pattern, thereby making the negative set more directional than in S-AUC (Jia et al., 2020). This suggests that SBSS also has a meta-evaluative role: saliency can govern not only model inference and training, but also the sampling process used to assess models.

Taken together, the literature portrays SBSS as a unifying methodology for non-uniform allocation of samples, receptive fields, trajectories, perturbations, or annotations according to task-specific importance estimates. Its success depends on two coupled requirements: the saliency estimator must be informative for the downstream objective, and the sampling rule must preserve enough structure that discarded regions, states, or examples do not remove critical context. A plausible implication is that future work will continue to move from heuristic saliency cues toward jointly learned, task-aware sampling mechanisms that remain interpretable while avoiding the instability, bias, or topology violations documented in earlier approaches (Ip et al., 11 Jun 2026, Zheng et al., 6 Jan 2026).