TCRC: Trajectory-Constrained Residual Condensation

• TCRC is a trajectory-aware residual condensation method that leverages one-to-one temporal linking to maintain token traceability and avoid feature ambiguity.
• It condenses residual tokens using a three-stage process: temporal matching, trajectory pooling, and entropy-regularized spatial condensation.
• The method demonstrates significant improvements in token efficiency and manifold fidelity, benefiting both video LLMs and numerical-analysis applications.

Trajectory-Constrained Residual Condensation (TCRC) is a trajectory-aware residual compression procedure in which residual information is condensed only after admissible correspondences have been established. In the Video LLM framework "GeoTrace: Geometry-Aware Trajectory Token Compression for Video LLMs," TCRC is the second stage of a training-free spatiotemporal token compression pipeline: after Contextual Farthest-Point Anchoring (CFPA) preserves skeleton tokens, TCRC compresses the remaining residual tokens through one-to-one temporal trajectories and constrained near-manifold condensation, producing traceable event tokens with reduced ambiguity (Xie et al., 10 Jul 2026). In a distinct numerical-analysis setting, the same phrase is used to describe a trajectory-aware restriction of residual snapshot spaces to those achievable along the actual online iterative path in ROM preconditioning for parametric radiative transfer equations (Tang et al., 5 Sep 2025). This suggests a common underlying principle: condensation is constrained by trajectory structure rather than performed by unconstrained aggregation.

1. Role within geometry-aware video token compression

In GeoTrace, TCRC operates on the non-skeleton tokens that remain after CFPA. The framework is motivated by a specific limitation of Video LLM inference: efficiency is constrained by the large number of visual tokens, while existing video token compression methods typically rely on frame-wise saliency or heuristic token merging, which can over-focus on locally salient regions and produce ambiguous fused features (Xie et al., 10 Jul 2026). TCRC is introduced to address the two failure modes identified for such baselines: loss of temporal information and production of ambiguous, off-manifold features.

The method is therefore defined not as a generic token merger, but as a residual-token condensation procedure with explicit trajectory constraints. Its stated design objectives are to build one-to-one temporal token correspondences across adjacent frames, condense along near-manifold local groups, and maintain explicit provenance from condensed event tokens back to raw input tokens. Within GeoTrace’s decomposition of video evidence into exact skeleton tokens and traceable residual event tokens, TCRC is the mechanism responsible for the latter category.

A plausible implication is that TCRC should be read as a corrective to two common assumptions in video token pruning: first, that per-frame saliency is sufficient for temporal reasoning; second, that aggressive fusion is acceptable even when it obscures token lineage. GeoTrace’s formulation rejects both assumptions by making trajectory consistency and traceability first-class constraints.

2. Formal mechanism: temporal linking, pooling, and constrained condensation

TCRC is defined per video chunk and takes as inputs the residual tokens after CFPA, the residual token features $\mathbf{x}_{t,i}$ for each frame $t$, and a retained token budget $K_r$ for condensed tokens per frame (Xie et al., 10 Jul 2026). Its first stage is one-to-one temporal trajectory linking across consecutive frame pairs $(t, t+1)$. For residual-token index sets $I_t$ and $I_{t+1}$, the cosine-similarity matrix is

$$S_{ij} = \cos(\mathbf{x}_{t+1,i}, \mathbf{x}_{t,j}).$$

The temporal correspondence is then obtained by optimal one-to-one assignment over bipartite matchings:

$$\mathcal{M}_{t+1} = \arg\max_{\mathcal{M} \in \Pi} \sum_{(i,j)\in \mathcal{M}} S_{ij}.$$

Hungarian matching is used, and a matched pair is accepted only if $S_{ij} \geq \tau$, where $\tau$ is the temporal similarity threshold. The stated effect is that each matched pair forms a link in a trajectory, allowing each token to be part of one trajectory and reducing many-to-one collapse.

The second stage performs temporal pooling over the established trajectories. For a trajectory $t$0 consisting of linked token feature vectors $t$1, the pooled feature is

$t$2

Each trajectory is thereby summarized as an event-level feature, while singletons—tokens not matched temporally—are kept as single-frames. This preserves a distinction between sustained temporal evidence and frame-local residual evidence.

The third stage applies spatial condensation after temporal pooling. For frame $t$3, let $t$4 denote the residual roots after pooling. If $t$5 exceeds the per-frame event-token budget $t$6, TCRC formulates an entropy-regularized optimal transport problem to condense $t$7 tokens into $t$8 output tokens.

Step	Key operation	Mathematical form / algorithm
1. Temporal matching	One-to-one, thresholded token correspondences	Hungarian on similarity matrix
2. Trajectory pooling	Average features over temporal trajectory	Temporal mean
3. Spatial condensation	Entropy-regularized OT (Sinkhorn barycenter)	Solve for $t$9, then update
4. Traceability assignment	Inherit index of largest-mass preimage	Dominant-source index

This staged construction is central to the meaning of TCRC in GeoTrace: residual tokens are not merged directly, but are first organized into temporally admissible trajectories and only then spatially condensed.

3. Near-manifold condensation and traceability

The spatial-condensation stage is posed as an entropy-regularized optimal transport problem. For source tokens $K_r$0 and output event tokens $K_r$1, TCRC solves

$K_r$2

where $K_r$3 is the fractional assignment of source token $K_r$4 to event token $K_r$5, $K_r$6 is the set of transport plans with uniform (or proportional) marginals, and $K_r$7 is the entropy regularization strength (Xie et al., 10 Jul 2026). The condensed features are then updated by barycentric averaging,

$K_r$8

Traceability is enforced by a provenance rule rather than by post hoc attribution. Each output token inherits the global index of the source feature with the largest transport mass:

$K_r$9

The paper characterizes this as explicit provenance: every condensed token remains traceable to an explicit original token. In this sense, TCRC’s “event token” is not only a compressed feature but also a token with a designated source lineage.

Theoretical support is provided in Appendix D via Theorem D.1, which proves that each output event token satisfies a bound on its distance to the true data manifold $(t, t+1)$0, with terms depending on the max path and cluster diameters due to thresholding and condensation, as well as bounds on feature norms and non-locality mass. The qualitative conclusion stated in the source is that TCRC event tokens’ drift from the data manifold is strictly controlled and much smaller than with unconstrained merging (Xie et al., 10 Jul 2026). This is the formal basis for the paper’s near-manifold-condensation claim.

4. Algorithmic realization and empirical behavior in Video LLMs

The appendix-level pseudocode describes TCRC as TCRC(X, mask, G, K_r, tau), where X is chunk features, mask marks residual tokens, G stores global indices, K_r is the per-frame event-token budget, and tau is the similarity threshold (Xie et al., 10 Jul 2026). Temporal matching computes cosine similarity across adjacent frames, applies Hungarian matching, and writes predecessor pointers into pred. A reverse-time accumulation step then aggregates features and counts along accepted chains, after which per-frame root indices are collected and, if necessary, further condensed by a SinkhornBarycenter(Y, B) routine. The barycenter routine is initialized with KMeans++ or a random subset and iteratively alternates between cost computation, regularized optimal transport, and barycenter updates.

Empirically, GeoTrace is evaluated on four Video LLMs across four video understanding benchmarks, with the results described as demonstrating effectiveness and generalization across different model architectures and scenarios (Xie et al., 10 Jul 2026). On LLaVA-OneVision, with only $(t, t+1)$1 visual tokens retained, GeoTrace achieves a $(t, t+1)$2 TFLOPs reduction while preserving $(t, t+1)$3 of the vanilla performance. Since TCRC is the residual-condensation component of GeoTrace, these results establish the performance regime in which TCRC operates.

Ablation evidence is also reported for TCRC-specific design choices. Replacing one-to-one matching with many-to-one matching, or replacing Sinkhorn fusion with KMeans, degrades performance by up to 0.3 points in average score and 0.5% in accuracy. Qualitative figures are described as showing clear, visually coherent trajectories for residual tokens and avoidance of ambiguous merging. Failure cases are noted for abrupt scene changes or ambiguous appearance, but these are described as localized, with skeleton tokens remaining as fallback. The paper’s empirical framing therefore treats TCRC not as an isolated module, but as a robustness-preserving compression mechanism within a dual-path representation consisting of skeleton tokens and event tokens.

5. Distinction from heuristic merging and related misconceptions

The most immediate misconception is to treat TCRC as another many-to-one token merging heuristic. The GeoTrace formulation explicitly rejects that characterization. Its temporal stage is one-to-one rather than many-to-one; its accept/reject mechanism is thresholded by temporal similarity; and its spatial stage is constrained by entropy-regularized transport rather than unrestricted averaging (Xie et al., 10 Jul 2026). The stated purpose of these constraints is to avoid semantically distant fusion and to preserve traceability.

Another misconception is to view TCRC as a generic learned compressor. In GeoTrace, the overall framework is training-free. TCRC does not introduce an additional learned module for residual fusion; instead, it uses matching, pooling, and Sinkhorn-style condensation on existing features. This matters because the claimed gains are attributed to structural constraints on condensation rather than to extra optimization capacity.

The source also places TCRC against several comparator families. Frame-wise or top-$(t, t+1)$4 selection is said to lack temporal and context awareness. Heuristic or attention-based merging is said to lack explicit trajectory traceability and often collapse unrelated residuals, increasing semantic ambiguity and harming token provenance. Cluster-based or graph-based approaches may not ensure close-to-manifold condensation or per-token traceability as in TCRC (Xie et al., 10 Jul 2026). These comparisons define the method’s intended niche: TCRC is a constrained condensation operator whose primary differentiators are temporal one-to-one linkage, manifold-faithful local fusion, and explicit provenance.

6. Cross-domain usage of the term and related trajectory-based condensation ideas

Outside Video LLMs, the phrase “Trajectory-Constrained Residual Condensation” is used in work on synthetic acceleration preconditioners for parametric radiative transfer equations to describe a trajectory-aware restriction of residual snapshot spaces (Tang et al., 5 Sep 2025). There, the core issue is mismatch between offline residual trajectories used to build reduced-order models and the online residual trajectories actually induced by the preconditioner sequence. The proposed remedy is to iteratively construct a sequence of ROMs, where each ROM is trained on trajectories induced by the preconditioner sequence used so far. The details characterize this as TCRC: snapshot selection is not arbitrary but locked to the actual iterative trajectory. Numerical tests on a parametric lattice problem are reported to achieve rapid convergence within only $(t, t+1)$5-$(t, t+1)$6 iterations online, with speedups of up to $(t, t+1)$7 over DSA and $(t, t+1)$8 over classic ROMSAD.

In this numerical-analysis usage, TCRC does not denote token compression but rather condensation of the residual subspace to what is achievable under a constrained iterative history. The commonality with GeoTrace lies in the logic of admissibility: condensation is performed only after the residuals have been filtered through a trajectory constraint, whether those trajectories are visual-token correspondences across time or solver residual paths across iterations. This suggests that TCRC is better understood as a methodological pattern than as a domain-specific implementation.

A related trajectory-aware idea appears in clinical dataset condensation via mode connectivity-based trajectory surrogates (Nganjimi et al., 7 Oct 2025). That work replaces full SGD trajectories with quadratic Bézier curves connecting initial and final model states, yielding smooth, low-loss parametric surrogates that stabilize gradients, accelerate convergence, and eliminate the need for dense trajectory storage. Only the starting, ending, and control points are stored, reducing storage need by up to $(t, t+1)$9. Although the formulation is different, it provides adjacent evidence that condensation methods can benefit from constraining supervision to smoother or more faithful trajectory representations rather than using unconstrained or noisy intermediates.