Determine the mechanism by which principal-component processing improves the video graph

Determine whether the performance gain from processing the leading directions of egocentric video embeddings results from removing activity-appearance content or from reweighting the directions in the similarity computation.

Background

The method improves graph-based human activity recognition by removing the leading principal components of pretrained egocentric video embeddings before constructing the participant-level video graph. The leading directions contain both viewpoint information, which helps group clips from the same activity set, and activity-related appearance information, which can cause clips from pauses or visually similar activities to be grouped incorrectly.

The paper reports that subtracting the limb-predictable component reproduces much of the improvement, while truncated whitening—which retains the leading directions but changes their relative weighting—achieves a similar gain. Consequently, the experiments do not establish whether the benefit is caused by removing specific semantic content or merely by correcting anisotropic weighting in the embedding space.

References

Whitening the leading directions without discarding them does as well, however, so we cannot say whether the gain comes from removing this content or from changing how much the leading directions weigh in the similarities.

— What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition  (2610.12926 - Habano, 9 Oct 2026) in Section 6, Discussion, Section 6.1, “What the graph means in the domain”; also stated in Section 7, Conclusion