TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learning

Abstract: For imitation learning in robotic manipulation, high data collection costs result in the scarcity of high quality data. In this paper, we leverage the inherent heterogeneity of trajectories to address this challenge. Based on our observations of manipulation tasks, we categorize motions into transitional, precise, and agile types, defining the latter two as trajectory saliency due to their criticality to task success in contrast to the prevalent but less relevant transitional motions. Therefore, we propose the Trajectory Saliency Detector (TSD), a training-free and plug-and-play framework to identify trajectory saliency. TSD employs two physically-grounded metrics: spatial entropy to capture fine-grained manipulation and centripetal acceleration to detect agile maneuvering. We further leverage TSD to develop a dataset compression method that reduces training costs and a dataset expansion strategy that improves data collection efficiency. Extensive experiments in both simulation and real-world settings demonstrate that models trained on TSD-condensed datasets achieve comparable or even superior performance with 25% less data on average. These results validate the effectiveness of our dataset compression and expansion strategies, thereby confirming the utility of TSD. Consequently, TSD offers a scalable and cost-effective pathway to synthesize information-dense datasets for efficient robot learning. Project page: https://trajectory-saliency-detector.github.io/trajectory-saliency-detector/

• The paper introduces a training-free, physics-inspired framework for identifying task-critical trajectory segments using spatial entropy and centripetal acceleration.
• It demonstrates effective dataset compression and synthetic expansion, reducing data requirements by up to 25% while maintaining or improving task performance.
• Experimental results in simulation and real-world settings validate TSD's numerical consistency, positional robustness, and superior task success rates.

Physics-Inspired Trajectory Saliency Detection for Efficient Imitation Learning

Motivation and Problem Formulation

The principal challenge addressed in "TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learning" (2606.23371) is the high data acquisition and annotation cost inherent in robotic manipulation tasks. Unlike domains like NLP and computer vision—with abundant internet-scale datasets—robotic imitation learning suffers from scarce, expensive, and labor-intensive demonstration collection. Furthermore, current paradigms treat all trajectory data points equally, diluting model capacity by fitting to irrelevant transitional motions and failing to prioritize task-critical phases.

The paper formalizes trajectory heterogeneity by categorizing demonstrations into transitional, precise, and agile types—with the latter two jointly designated as trajectory saliency due to their criticality for task success. The central thesis posits that precise (fine-grained manipulation) and agile (dynamic, obstacle-avoidant maneuvering) segments hold high informational value, warranting resource prioritization for both dataset compression and expansion.

Methodology

Trajectory Saliency Detector (TSD) Framework

TSD is introduced as a training-free, physics-grounded framework for unsupervised salient segment identification. It leverages two metrics:

• Spatial Entropy (Precision): KNN-based entropy calculation captures the spatial density trajectory distribution, where local entropy minima correspond to fine manipulation phases. Velocity-normalized weighting is applied to emphasize low-velocity, high-precision states and adaptive valley detection isolates the relevant segments.
• Centripetal Acceleration (Agility): Combined geometric simplification via Ramer-Douglas-Peucker downsampling and dynamical analysis identifies high-maneuverability operations. Adaptive peak detection in centripetal acceleration signals agile phases corresponding to rapid obstacle avoidance or task-critical adjustments.

Temporal alignment across trajectories is achieved using Dynamic Time Warping, with a hybrid distance metric that respects both translation and rotation, ensuring consistency across demonstrations despite execution variability. The Sakoe-Chiba constraint improves computational efficiency and avoids pathological warping. Precise mapping between reference and original timelines is maintained during saliency propagation.

Dataset Compression and Expansion

TSD facilitates two primary applications:

• Dataset Compression: Redundant transitional segments are pruned, resulting in a condensed, information-dense dataset that preserves task success rates while reducing computational and storage overhead.
• Dataset Expansion: When high-quality demonstration data is limited, TSD guides collection efforts to prioritize critical states, synthetically expanding the dataset without inflating annotation costs.

These mechanisms are validated both in simulation (Robosuite tasks) and real-world settings (single- and dual-arm manipulation scenarios) with pronounced heterogeneity (spatial randomization, obstacle variability).

Experimental Evaluation

Numerical and Positional Consistency

Saliency detection demonstrates rapid convergence and numerical stability:

• Numerical Consistency: Both agility and precision instability metrics ($A_{Cn}, A_{En}$) plateau with modest sample size ($\approx 20$ demonstrations for agility), indicating that TSD reliably extracts task features without noise contamination.
• Positional Consistency: TSD maintains segment localization robustness under strong demonstration randomization, indicating generality across diverse collection regimes.

Simulation and Real-World Results

Strong numerical results are reported:

• In simulation, policies trained on TSD-compressed datasets achieve comparable or superior task success rates, with up to 25% less data required. Notably, Lift and Transport tasks see enhanced efficiency, as TSD-filtered datasets outperform full-dataset counterparts.
• In real-world experiments, TSD-guided expansion yields a 3.3% higher success rate under constrained frame budgets in Tray Setting and Book Fetching tasks, outperforming randomly sampled datasets and demonstrating effective behavioral denoising.

Ablation Studies

Ablations separate the impact of precise/agile segment augmentation. Precise-only augmentation outperforms agile-only augmentation, matching complete dataset performance at the same size and underscoring the importance of fine-grained interaction logic. Conversely, agility omission leads to degraded obstacle avoidance, and precision omission results in failed execution logic.

Practical and Theoretical Implications

The TSD framework operationalizes trajectory heterogeneity, enabling resource-efficient model training and dataset curation for imitation learning. The demonstrated reduction in training cost and data collection labor has direct implications for scalable deployment of robot learning systems, particularly in real-world settings with pronounced scene variability. The plug-and-play, unsupervised nature of TSD also eliminates reliance on auxiliary discriminators, lowering the entry barrier for practical applications.

Theoretically, TSD's methodology links physics-inspired metrics to learning dynamics, providing a principled approach to segment prioritization informed by the Markov property. This foundation could inspire further algorithmic innovations in task abstraction, epistemic uncertainty modeling, and sample-efficient reinforcement/behavioral cloning regimes.

Future Directions

Potential extensions include adaptation to higher-DOF systems (e.g., dexterous manipulation, loco-manipulation tasks), integrating TSD with generative policy learning modalities, and expanding its utility for complex interactive scenarios or cross-domain sim-to-real transfer. Systematic benchmarking vis-à-vis foundation models and large-scale diffusion policies will clarify its scalability and generalization boundaries.

Conclusion

TSD is a physics-inspired, training-free trajectory saliency detector for efficient imitation learning in robot manipulation. The integration of spatial entropy and centripetal acceleration enables unsupervised identification of task-critical phases, facilitating both dataset compression and expansion. Empirical results across simulated and real-world environments validate substantial improvements in data efficiency and model performance. TSD presents an accessible preprocessing solution for robotic imitation learning, with implications for scaling intelligent capabilities and optimizing resource deployment in future robotic systems.