Temporal Interaction Localization (TIL)

• Temporal Interaction Localization (TIL) is the precise detection of contact and separation timestamps, segmenting interactions into pre-contact, in-contact, and post-contact phases.
• Zero-shot methods like EgoLoc combine 3D hand dynamics with visual language models and closed-loop feedback to reliably pinpoint critical transition frames.
• Empirical evaluations on datasets such as EgoPAT3D-DT and DeskTIL demonstrate TIL's effectiveness using metrics like SR, MAE, MoF, and IoU.

Temporal Interaction Localization (TIL) denotes the fine-grained temporal localization of interaction transitions in video, most commonly the exact timestamps at which a visible hand contacts and later separates from a target object in egocentric human–object interaction. In contrast to coarse temporal action localization, which identifies category-level action segments, TIL targets moment-level transitions that partition a clip into pre-contact, in-contact, and post-contact stages. The acronym is not fully stable across recent literature: in surveillance, “TIL” has also been used for Temporal Intention Localization, where the target is latent suspicious intention rather than observable interaction. The interaction-centered usage, however, is defined by the “when to interact” problem and is exemplified by zero-shot egocentric pipelines such as EgoLoc (Ma et al., 17 Aug 2025, Hu et al., 23 Oct 2025).

1. Definition, scope, and task boundaries

Temporal Interaction Localization addresses the temporal boundary problem for interactions at a finer granularity than standard Temporal Action Localization (TAL). In the egocentric setting, the task is formulated as localizing the exact timestamps when the hand first makes contact with an object and when it later releases that object. This differs from TAL, which typically returns start and end boundaries for semantic action categories such as “open drawer” or “pour water.” TIL therefore does not merely ask how an interaction should be categorized; it asks when the physically decisive transition occurs, often at frame precision and without dependence on verb–noun taxonomies or object masks (Ma et al., 17 Aug 2025).

This finer granularity has immediate representational consequences. A TIL annotation induces three temporal stages: pre-contact, in-contact, and post-contact. In the EgoLoc formulation, contact and separation timestamps are the primary outputs, and the in-contact interval is derived from them. A plausible implication is that TIL can be viewed as a structured boundary localization problem whose semantics are concentrated at transition points rather than over extended action spans.

The term “TIL” is also overloaded. In “SPAN: Continuous Modeling of Suspicion Progression for Temporal Intention Localization,” TIL is explicitly defined as localizing the temporal progression of hidden intentions in untrimmed surveillance video. That work emphasizes that intention refers to latent motives preceding abnormal actions, whereas interaction refers to observable interactions among entities. This distinction is substantive rather than terminological: interaction-centered TIL grounds visible contact dynamics, while intention-centered TIL models hidden motives through continuous suspicion trajectories and concept anchoring (Hu et al., 23 Oct 2025).

2. Mathematical formulation of interaction timing

In the interaction-centered literature, the input is typically an egocentric video with RGB frames and, when available, aligned depth frames. One canonical formulation writes the RGB sequence as

$$V = \{I_t\}, \quad I_t \in \mathbb{R}^{3 \times h \times w}, \quad t = 1, \ldots, N_{\text{obs}},$$

and the depth sequence as

$$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$

The outputs are the contact timestamps

$T_c = \{t_c^i\}$

and the separation timestamps

$T_s = \{t_s^j\},$

with each timestamp lying in the observed temporal range. In the simpler single-interaction formulation, EgoLoc produces the pair $(T_c, T_s)$, first estimating contact and then restricting the search region to estimate separation (Ma et al., 17 Aug 2025, Zhang et al., 4 Jun 2025).

The core kinematic signal is derived from hand motion. A 2D wrist keypoint sequence $H_t^{2D} \in \mathbb{R}^2$ is extracted per frame. With depth and camera intrinsics, this is lifted to camera-frame 3D positions $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$. Inter-frame point cloud registration yields a pose matrix $M_t \in \mathbb{R}^{4 \times 4}$, allowing transformation into a global coordinate frame:

$$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$

Hand speed is then computed as

$$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$

where $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$0 is the fixed frame interval. The velocity signal is smoothed by Savitzky–Golay filtering and cubic spline fitting; local minima of $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$1 correspond to $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$2 and are treated as likely anchors for contact or separation (Ma et al., 17 Aug 2025).

This formulation encodes the empirical assumption that contact and separation occur near low-speed phases of hand motion. In the earlier EgoLoc report, the method selects $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$3 frames with the lowest velocities, samples an anchor frame from this set, and constructs a compact temporal neighborhood for VLM reasoning. In the later formulation, anchor candidates are sampled around velocity local minima with an exponential weight

$$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$4

which explicitly favors low-speed frames. This suggests that TIL, in this line of work, is treated as a hybrid geometric–semantic localization problem: geometry proposes likely transition neighborhoods, and multimodal reasoning resolves the precise frame (Zhang et al., 4 Jun 2025, Ma et al., 17 Aug 2025).

3. Zero-shot egocentric TIL and the EgoLoc pipeline

EgoLoc is a zero-shot, training-free approach designed specifically for TIL in egocentric videos. Its central claim is that high-quality initial guesses around contact and separation can be obtained directly from 3D hand dynamics, after which a vision–LLM can localize the exact frame using compact visual prompts. The later EgoLoc formulation describes five principal stages: 2D wrist detection with HaMeR, 3D lifting with depth and intrinsics, global alignment via point cloud registration, speed computation and local-minimum detection, and VLM-based localization with closed-loop refinement. Boundary-frame hand boxes for attribute identification are obtained with GroundingDINO and expanded by $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$5 pixels to include object context (Ma et al., 17 Aug 2025).

The localized visual prompt is a tiled grid image $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$6 formed from an $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$7 neighborhood centered on the sampled anchor. EgoLoc decomposes reasoning into three subproblems. First, an attribute discriminator classifies the local transition as contact, separation, or neither from the boundary frames of the neighborhood. Second, an attribute-specific localizer predicts the exact frame index within the grid. Third, a checker accepts or rejects the predicted frame with a visual plausibility prompt; if rejected, the system re-runs localization with the previously predicted frame included as a negative in-context example. One feedback round is reported as sufficient and most effective (Ma et al., 17 Aug 2025).

The earlier EgoLoc formulation uses a closely related closed loop, but makes the dynamic criterion explicit. A VLM-based visual discriminator tests whether the predicted frame shows obvious contact rather than separation, and a speed-based discriminator checks whether the predicted timestamp lies within the slowest $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$8 of clip velocities, with $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$9. If either criterion fails, the method constrains re-sampling to later frames or to frames with lower velocity than the rejected prediction. This makes the refinement loop partly semantic and partly kinematic (Zhang et al., 4 Jun 2025).

Several design choices recur across both reports. EgoLoc does not require object masks or action taxonomies; it relies on 3D hand-dynamics-guided sampling, a general-purpose VLM, and compact prompts. Typical settings include $T_c = \{t_c^i\}$0 s for EgoPAT3D-DT and ManiTIL, $T_c = \{t_c^i\}$1 s for DeskTIL, $T_c = \{t_c^i\}$2, and $T_c = \{t_c^i\}$3, with $T_c = \{t_c^i\}$4 reported as most effective in the later study. A plausible implication is that TIL performance depends less on exhaustive temporal search than on concentrating reasoning capacity near physically plausible transition frames (Ma et al., 17 Aug 2025).

4. Benchmarks, metrics, and evaluation protocols

The interaction-centered TIL literature evaluates on egocentric datasets with explicit contact and separation annotations. EgoPAT3D-DT is derived from EgoPAT3D and adds manual contact/separation annotations to finer-grained hand–object interaction sequences. DeskTIL is a new benchmark recorded with a head-mounted RealSense D435i under varying desk clutter levels; it contains 50 videos per clutter level, with average duration approximately 6 s and a single HOI process per video. ManiTIL comprises four long-term tasks at 30 FPS—“tidy up the table,” “pour water,” “use the tools from the drawer,” and “move the bottles in the cabinet”—with 50 videos for each task and multiple contact/separation transitions per video (Ma et al., 17 Aug 2025).

Evaluation is centered on both pointwise timing error and induced contact-segment quality. The later EgoLoc paper defines Mean over Frames (MoF) as

$T_c = \{t_c^i\}$5

and Intersection over Union (IoU) as

$T_c = \{t_c^i\}$6

where $T_c = \{t_c^i\}$7 and $T_c = \{t_c^i\}$8 are estimated and ground-truth in-contact frame sets. Mean Absolute Error (MAE) is the average absolute frame difference between estimated and ground-truth timestamps, and Success Rate with tolerance $T_c = \{t_c^i\}$9 is the fraction of estimates satisfying $T_s = \{t_s^j\},$0. The earlier EgoLoc report uses the same family of metrics, reports SR for $T_s = \{t_s^j\},$1, and averages results across 5 trials to analyze uncertainty (Ma et al., 17 Aug 2025, Zhang et al., 4 Jun 2025).

Dataset	Protocol summary	Notable structure
EgoPAT3D-DT	30 FPS; manual contact/separation annotations	Fine-grained HOI sequences
DeskTIL	RealSense D435i; downsampled to 5 FPS	Single HOI process; clutter levels
ManiTIL	Four long-term tasks at 30 FPS	Multiple HOI processes per video

These benchmarks reflect two complementary evaluation regimes. Short, single-transition clips such as DeskTIL emphasize timestamp precision under clutter. Longer sequences such as ManiTIL emphasize repeated transitions, longer temporal dependencies, and robustness to distractors. This suggests that TIL evaluation is inherently coupled to the temporal density of interactions rather than only to semantic category diversity.

5. Empirical performance and ablation structure

The later EgoLoc study reports strong results across all three benchmarks. On EgoPAT3D-DT, EgoLoc with $T_s = \{t_s^j\},$2 achieves $T_s = \{t_s^j\},$3, $T_s = \{t_s^j\},$4, $T_s = \{t_s^j\},$5, and $T_s = \{t_s^j\},$6, compared with T-PIVOT at $T_s = \{t_s^j\},$7, $T_s = \{t_s^j\},$8, $T_s = \{t_s^j\},$9, and $(T_c, T_s)$0, and GreedyVLM at $(T_c, T_s)$1, $(T_c, T_s)$2, $(T_c, T_s)$3, and $(T_c, T_s)$4. On DeskTIL, EgoLoc achieves $(T_c, T_s)$5, $(T_c, T_s)$6, $(T_c, T_s)$7, and $(T_c, T_s)$8. On ManiTIL, it achieves $(T_c, T_s)$9, $H_t^{2D} \in \mathbb{R}^2$0, $H_t^{2D} \in \mathbb{R}^2$1, and $H_t^{2D} \in \mathbb{R}^2$2 (Ma et al., 17 Aug 2025).

The earlier EgoLoc report, using a different success tolerance and reporting averages over trials, reaches $H_t^{2D} \in \mathbb{R}^2$3, $H_t^{2D} \in \mathbb{R}^2$4, $H_t^{2D} \in \mathbb{R}^2$5, and $H_t^{2D} \in \mathbb{R}^2$6 on EgoPAT3D-DT, and $H_t^{2D} \in \mathbb{R}^2$7, $H_t^{2D} \in \mathbb{R}^2$8, $H_t^{2D} \in \mathbb{R}^2$9, and $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$0 on DeskTIL. Since these numbers are tied to a different protocol, they should not be read as directly interchangeable with the later $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$1 results; rather, they indicate consistency across reporting schemes (Zhang et al., 4 Jun 2025).

Ablation results identify three factors as central. First, self-adaptive sampling matters decisively: on EgoPAT3D-DT, random sampling without SASS yields $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$2, $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$3, $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$4, and $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$5, whereas full 3D SASS reaches $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$6, $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$7, $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$8, and $H_t^{3D,\mathrm{cam}} \in \mathbb{R}^3$9. Second, boundary frames are a better prompt for attribute identification than anchor-only or full-grid inputs. Third, closed-loop feedback reduces both error and uncertainty: on DeskTIL, full feedback improves over no feedback from $M_t \in \mathbb{R}^{4 \times 4}$0, $M_t \in \mathbb{R}^{4 \times 4}$1, $M_t \in \mathbb{R}^{4 \times 4}$2, and $M_t \in \mathbb{R}^{4 \times 4}$3 to $M_t \in \mathbb{R}^{4 \times 4}$4, $M_t \in \mathbb{R}^{4 \times 4}$5, $M_t \in \mathbb{R}^{4 \times 4}$6, and $M_t \in \mathbb{R}^{4 \times 4}$7; on EgoPAT3D-DT, standard deviations over 5 trials also decrease materially with feedback (Ma et al., 17 Aug 2025).

Model choice remains consequential. In the later evaluation on EgoPAT3D-DT, GPT-4o yields $M_t \in \mathbb{R}^{4 \times 4}$8, $M_t \in \mathbb{R}^{4 \times 4}$9, $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$0, and $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$1, outperforming Gemini 2.5 Pro, GPT-4o mini, and Janus-Pro-7B. Runtime is reported as approximately $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$2–$$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$3 the time of T-PIVOT, and the earlier report phrases the same advantage as T-PIVOT requiring approximately $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$4–$$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$5 EgoLoc’s inference time per video (Ma et al., 17 Aug 2025, Zhang et al., 4 Jun 2025).

6. Relation to temporal language localization, applications, and open problems

Although current TIL work is centered on egocentric contact and separation, it sits within a broader family of temporal localization problems. LITA addresses general “When?” questions in video by introducing relative time tokens, SlowFast token pooling, and timestamp-heavy instruction tuning. Its target output is a temporal segment $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$6 rather than contact/separation points, but the paper explicitly notes that these capabilities transfer naturally to TIL because interaction queries are still temporal localization queries over video time. LITA’s time-token discretization,

$$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$7

offers a language-conditioned segment representation that is compatible with interaction-centric prompts such as “When does the person pick up the cup?” (Huang et al., 2024).

An earlier line of work on language-based moment localization, represented by MLLC, contributes a different transferable idea: query-conditioned latent temporal context. Its score for a candidate base moment $$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$8 is

$$H_t^{3D,\mathrm{glob}} = M_t \cdot [H_t^{3D,\mathrm{cam}}; 1].$$9

which enables reasoning about temporal relations such as before, after, then, and while. For interaction localization, this is relevant when the interaction of interest is defined relative to another event, such as release after handover or grasp before opening. The paper’s emphasis on temporal context suggests that TIL can benefit from explicit modeling of interaction phases and surrounding events, not only from local contact evidence (Hendricks et al., 2018).

TIL has already been used as a substrate for downstream systems. EgoLoc provides automatic contact/separation annotations for a hand motion forecaster, MMTwin, in human–robot policy transfer. In simulation with a ViperX 300 in SAPIEN on the task of putting a blue block onto cloth, success rates over 10 trials are reported as EgoLoc $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$0, Threshold $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$1, HOIMask $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$2, and Manual upper bound $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$3. In a real-world cabinet-door task with a Franka Research 3, the corresponding success rates are EgoLoc $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$4, Threshold $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$5, HOIMask $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$6, and Manual upper bound $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$7. The same timestamps can support VR/AR “time-to-contact” overlays and modestly improve egocentric action recognition when injected into RU-LSTM, raising Top-5 accuracy from $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$8 to $$v_t = \frac{\|H_{t+1}^{3D,\mathrm{glob}} - H_t^{3D,\mathrm{glob}}\|}{\delta},$$9 on the RGB branch and from $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$00 to $$D = \{D_t\}, \quad D_t \in \mathbb{R}^{1 \times h \times w}.$$01 on the flow branch (Ma et al., 17 Aug 2025).

The main limitations are also well defined. Current TIL formulations often assume one visible hand, exclude rapid non-impulsive actions such as tapping because of motion blur, and remain sensitive to errors in hand segmentation, depth registration, or ambiguous wrist motion. The later EgoLoc report also notes that it does not explicitly model object semantics and that further feedback rounds did not substantially help while hurting efficiency. Future directions named in the literature include extension to bimanual HOI transitions and rapid actions, stronger multimodal grounding, and preservation of zero-shot generalizability (Ma et al., 17 Aug 2025, Zhang et al., 4 Jun 2025).

The adjacent development of Temporal Intention Localization underscores that “interaction” is only one of several temporal targets now being localized in video. SPAN models continuous suspicion progression in surveillance using a Temporal Point Process-inspired score, multimodal coefficient modulation, and concept-anchored mapping. Because it targets latent motives rather than contact events, it belongs to a distinct problem family. Its inclusion in the recent literature nevertheless clarifies an important taxonomic point: TIL can denote either interaction or intention, and accurate use of the term requires specifying whether the objective is observable transition timing or hidden motivational state estimation (Hu et al., 23 Oct 2025).