Trajectory Saliency Detector for Robust Imitation Learning

• The paper demonstrates that isolating precise and agile segments in robot trajectories can achieve similar or superior task performance with 25% less data.
• TSD is a training-free, physics-inspired framework that uses DTW alignment, weighted entropy, and geometric simplification to segment manipulation trajectories.
• It supports dataset compression by retaining critical segments and facilitates dataset expansion by focusing on instructional, high-information motion intervals.

Searching arXiv for papers on trajectory saliency detector and related TSD usages. Trajectory Saliency Detector (TSD) most specifically denotes a training-free, plug-and-play framework for robotic imitation learning that identifies the precise and agile portions of manipulation demonstrations while treating prevalent transitional motion as less relevant to task success (Zhao et al., 22 Jun 2026). In that formulation, trajectory saliency is not a generic visual saliency map but a segmental property of a trajectory: fine-grained manipulation and agile maneuvering are marked as information-dense, whereas longer transitional motions are treated as comparatively redundant. More broadly, adjacent literatures have used closely related ideas—contextual deviation, spatial-temporal accumulation, geometric inconsistency, and target-point emphasis—to define salient trajectories or salient temporal substructures in tracking, video understanding, time-series forecasting, and motion prediction (Maczyta et al., 2020, Zhang et al., 2018, Faisal et al., 2019).

1. Definition and terminological scope

In the 2026 robotics paper, "Trajectory Saliency Detector" is the official name of a method that operates directly on robot trajectories and produces saliency masks that can be used for dataset compression and dataset expansion (Zhao et al., 22 Jun 2026). The method is explicitly framed around efficient imitation learning under data scarcity, and its core claim is that precise and agile segments are critical to task success whereas transitional motions are prevalent but less relevant.

The acronym TSD is not stable across the literature. In other domains it denotes different constructs, even when those constructs are also concerned with temporal structure or trajectory-like data.

arXiv id	Expansion	Relation to the topic
(Zhao et al., 22 Jun 2026)	Trajectory Saliency Detector	Official TSD for efficient imitation learning
(Duong-Trung et al., 2022)	Temporal Saliency Detection	Saliency-aware time-series forecasting, not officially a trajectory saliency detector
(Zhu et al., 28 Jan 2025)	Target-driven Self-Distillation	Motion forecasting under partial observation, where TSD does not mean trajectory saliency detector

A common source of confusion is the assumption that trajectory saliency is always a pixel-level or image-level notion. The record is more heterogeneous. In robotic imitation learning, saliency is attached to task-critical motion segments (Zhao et al., 22 Jun 2026). In recurrent latent-code modeling, a trajectory is salient if it deviates from normal trajectories sharing a common motion pattern related to a given context (Maczyta et al., 2020). In dense video motion analysis, saliency can be induced by a trajectory’s inconsistency with rigid-background epipolar geometry (Faisal et al., 2019). In non-rigid object tracking, temporal accumulation of framewise saliency maps yields a spatial-temporal consistent saliency map that can be interpreted as a trajectory saliency representation (Zhang et al., 2018).

2. Conceptual basis of trajectory saliency

The 2026 TSD formulation begins from an empirical decomposition of manipulation behavior into three motion types: transitional, precise, and agile (Zhao et al., 22 Jun 2026). Transitional motions are long, relatively straight or smooth movements between more consequential phases. Precise motions are fine, careful, slow, and often contact-rich; small deviations can cause failure. Agile motions are high-speed, dynamically constrained, or obstacle-avoiding maneuvers characterized by tight curves, quick reactive corrections, or non-trivial path geometry. The method defines the latter two as trajectory saliency because they are critical to task success.

This definition is segmental rather than global. A trajectory is not assigned a single immutable status as salient or non-salient; instead, salient intervals are extracted from within the trajectory. That differs from the contextual anomaly-style definition in "Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder" (Maczyta et al., 2020), where a trajectory is salient if it deviates from the normal trajectories in a scenario. It also differs from tracking-based formulations in which saliency is accumulated frame by frame into a short-term memory of where an object has been salient over time (Zhang et al., 2018).

The broader implication is that "trajectory saliency" names a family of operational criteria rather than a single mathematical object. In imitation learning, the operative criterion is instructional density: which segments teach the policy the parts of the task that matter most. In contextual deviation models, the criterion is departure from a dominant motion pattern. In geometric motion segmentation, it is violation of a rigid-background multiview model. These formulations are compatible in spirit but distinct in what they measure and in the downstream decisions they support.

3. Physics-inspired construction

The 2026 TSD pipeline is training-free and physically grounded. It first aligns demonstrations using Dynamic Time Warping (DTW), then detects precise segments through weighted spatial entropy and agile segments through geometric simplification plus centripetal acceleration (Zhao et al., 22 Jun 2026). For trajectories $P = (p_1,\dots,p_n)$ and $Q = (q_1,\dots,q_m)$, DTW uses the cumulative cost

$$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$

with a Sakoe–Chiba band constraint $|i-j| \le b$. This produces a reference timeline across demonstrations, which is necessary for cross-demo entropy estimation.

Precise-segment detection is based on the observation that demonstrations converge spatially during fine manipulation. At a reference time $t$, positions within a local window are collected as $X = \{x_1,\dots,x_N\}$, $x_i \in \mathbb{R}^3$. Using a $k$-nearest-neighbor estimator, the local volume is

$V(x) = \frac{\pi^{d/2} R_k(x)^d}{\Gamma(d/2 + 1)},$

and the Shannon entropy is

$$H(t) = \frac{1}{N} \sum_{i=1}^{N} \ln\left( \frac{N \cdot V(x_i)}{k} \right) + \psi(k) - \psi(1),$$

with $Q = (q_1,\dots,q_m)$0. Low entropy indicates that demonstrations cluster tightly. Because low entropy can also occur at task start or end, TSD weights entropy by normalized velocity magnitude and velocity variance, yielding

$Q = (q_1,\dots,q_m)$1

Adaptive low-entropy valley detection is then applied, and valley times are mapped back through the DTW warping path and expanded spatiotemporally to form precise segments.

Agile-segment detection operates within each individual trajectory. The trajectory $Q = (q_1,\dots,q_m)$2 is simplified with Ramer–Douglas–Peucker (RDP), using perpendicular distance to the segment $Q = (q_1,\dots,q_m)$3:

$Q = (q_1,\dots,q_m)$4

Pause points associated with extremely low velocities are filtered out. Local geometric complexity is then quantified by tortuosity,

$Q = (q_1,\dots,q_m)$5

and high-tortuosity segments vote for robust geometric key points. Dynamic intensity is measured by centripetal acceleration,

$Q = (q_1,\dots,q_m)$6

where $Q = (q_1,\dots,q_m)$7 is speed and $Q = (q_1,\dots,q_m)$8 is curvature. Adaptive peak detection is applied to $Q = (q_1,\dots,q_m)$9, and intervals containing both a centripetal-acceleration peak and a geometric key point are labeled as agile segments.

The result is a pair of binary masks per trajectory: one for precise intervals and one for agile intervals. They can overlap. TSD therefore yields segment-level saliency labels without introducing a learned detector or modifying the downstream policy architecture.

4. Integration into efficient imitation learning

TSD is designed as a preprocessing layer placed before a standard imitation-learning pipeline (Zhao et al., 22 Jun 2026). Its inputs are trajectory kinematics, specifically end-effector positions over time; orientation can be used in DTW distance, but orientation is ignored for entropy estimation. Its outputs are per-time-step labels or interval masks indicating whether each segment is precise, agile, or neither. The downstream policy need not share the same input modality. In the reported experiments, the policy is a CNN-based Diffusion Policy using a third-view camera and a wrist camera, while TSD itself operates on kinematic traces.

The method supports two distinct dataset manipulations. In dataset compression, all precise and agile segments are retained, and only enough transitional frames are added to meet a target dataset size. The reported A2/B2 configurations are TSD-condensed datasets: they preserve the same number of trajectories as the corresponding A3/B3 datasets but contain fewer frames because non-salient transitional regions are dropped. In dataset expansion, a small base dataset is first analyzed by TSD, and additional demonstrations are then collected specifically around salient intervals. The details describe this as focusing human collection effort on the critical parts of the task rather than repeatedly recording full trajectories.

This architecture embodies a particular view of imitation-learning data efficiency. Each demonstration is assumed to be heterogeneous in information content. If behavioral cloning treats every time step equally, salient phases can be diluted by the large mass of transitional motion. TSD changes the empirical training distribution rather than the learning rule: the policy, optimizer, and observation stack remain unchanged, but the trajectory timeline is reweighted implicitly through selection.

The approach is most naturally suited to manipulation tasks with clear contact-rich or obstacle-avoiding subphases. The details note a limitation in smoother tasks: if a task contains little precision or agility, the distinction between salient and non-salient segments weakens, and the benefit over random sampling can diminish.

5. Experimental evidence

The 2026 study evaluates TSD in both simulation and real-world manipulation and reports that models trained on TSD-condensed datasets achieve comparable or even superior performance with 25% less data on average (Zhao et al., 22 Jun 2026). Simulation experiments use five Robomimic tasks—Can, Lift, Tool Hang, Transport, and Square—while real-world experiments use Book Fetching, Water Stowing, and Tray Setting. Performance is reported as success rate, and dataset size is reported as the number of frames.

Selected results illustrate the central empirical pattern.

Setting	TSD dataset	Full dataset
Can, B2 vs. B3	13321 frames, 92.0% success	23207 frames, 93.7% success
Transport, B2 vs. B3	79044 frames, 95.3% success	93752 frames, 89.3% success
Book Fetching, B2 vs. B3	14921 frames, 93.3% success	18578 frames, 93.3% success
Tray Setting, B2 vs. B3	21527 frames, 86.6% success	25350 frames, 83.3% success

These numbers show both compression and denoising effects. In Can and Book Fetching, TSD preserves performance with substantially fewer frames. In Transport and Tray Setting, the TSD-selected dataset yields higher success than the full dataset, which suggests that removing less relevant transitional motion can improve supervision quality rather than merely shrink storage.

Ablation studies indicate that both saliency types matter. On Book Fetching, the full TSD configuration B2 achieves 93.3% success with 14921 frames, whereas removing agile segments yields 63.3% success and removing precise segments yields 40.0% success (Zhao et al., 22 Jun 2026). The reported failure modes are consistent with the semantic interpretation of the two segment classes: without agile segments, obstacle avoidance degrades; without precise segments, grasping and fine alignment degrade.

The paper also evaluates numerical consistency of the detector as more demonstrations are added. For agile segments it forms binary masks $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$0, consensus curves $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$1, and the change metric

$$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$2

For precise segments it tracks entropy curves $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$3 and computes

$$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$4

The reported trend is that $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$5 quickly drops and plateaus with about 20 demonstrations, while $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$6 decreases more gradually but still converges. This indicates that the detected saliency structure becomes stable as the dataset grows.

6. Historical development and related formulations

Long before the 2026 paper adopted "Trajectory Saliency Detector" as an official title, related work had already treated saliency as an intrinsic property of trajectories. In "Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder" (Maczyta et al., 2020), trajectories are encoded by a recurrent auto-encoder into latent codes $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$7, and a scenario prototype $$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$8 is defined by the component-wise median. Saliency is then a standardized distance from normality:

$$D(i,j) = d(p_i, q_j) + \min\{ D(i-1,j), D(i,j-1), D(i-1,j-1)\},$$9

with a trajectory marked salient if $|i-j| \le b$0. That formulation is nearly unsupervised, scenario-dependent, and aimed at progressive dynamic saliency in surveillance-style trajectory sets rather than at data selection for imitation learning.

A second line of work grounds trajectory saliency in multiview geometry. "EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency" (Faisal et al., 2019) constructs dense trajectories covering every pixel in a video, estimates background epipolar geometry through trifocal tensors and fundamental matrices, and defines trajectory-level saliency via average epipolar distance. Small epipolar distance indicates consistency with rigid background motion; large distance indicates independently moving foreground. The method then transfers trajectory scores to a dense epipolar-distance image and combines them with optical flow in a UNet-style motion network.

A third line accumulates framewise saliency over time. In "Non-rigid Object Tracking via Deep Multi-scale Spatial-temporal Discriminative Saliency Maps" (Zhang et al., 2018), a tailored fully convolutional network predicts per-frame saliency maps, and temporal aggregation is performed through the Spatial-Temporal Consistent Saliency Map,

$|i-j| \le b$1

The paper explicitly notes that this can be interpreted as a trajectory saliency representation because it encodes where the object has been salient over time.

Related ideas also appear in application-specific fusion systems. "Viewport Prediction for Volumetric Video Streaming by Exploring Video Saliency and Trajectory Information" (Li et al., 2023) combines spatial saliency $|i-j| \le b$2, temporal saliency $|i-j| \le b$3, and trajectory-derived viewport features $|i-j| \le b$4 through attention-based fusion to predict future viewports in point-cloud video. Here, saliency is neither a latent anomaly score nor a manipulation-segment prior, but a content-driven cue combined with smooth head-motion trajectories. In time-series forecasting, "Temporal Saliency Detection Towards Explainable Transformer-based Timeseries Forecasting" (Duong-Trung et al., 2022) uses TSD to denote a U-Net-enhanced transformer for multi-resolution temporal saliency patterns, while "Target-driven Self-Distillation for Partial Observed Trajectories Forecasting" (Zhu et al., 28 Jan 2025) uses TSD to denote a self-distillation method in which target points function as salient waypoints. These works reinforce a general lesson: trajectory saliency is a cross-domain concept, but its operational meaning depends on whether the task is tracking, anomaly-style detection, motion segmentation, forecasting, viewport prediction, or imitation learning.