ATINet: Angular-Temporal Light-Field Tracker

• ATINet is a light-field object tracking framework that leverages a novel angular-temporal interaction to extract discriminative spatial-angular cues in low-light conditions.
• It employs a two-stream architecture, separating appearance features from geometric representations derived from epipolar-plane structure images and enhanced by Geometry Adaptive Selection.
• The framework achieves state-of-the-art performance in both single and multiple object tracking through cross-frame self-supervised reconstruction and reliable angular feature modeling.

Angular-Temporal Interaction Network (ATINet) is a light-field object tracking framework designed for low-light scenes. It is introduced in “An Angular-Temporal Interaction Network for Light Field Object Tracking in Low-Light Scenes” and is centered on two technical claims: first, that high-quality 4D light field representation with efficient angular feature modeling is crucial for scene perception because it provides discriminative spatial-angular cues for identifying moving targets; second, that recent developments still struggle to deliver reliable angular modeling in the temporal domain, particularly in complex low-light scenes (Wang et al., 29 Jul 2025). ATINet addresses this by combining a novel light field epipolar-plane structure image (ESI) representation with an angular-temporal interaction network that learns angular-aware representations from geometric structural cues and angular-temporal interaction cues, and that can also be optimized in a self-supervised manner to enhance geometric feature interaction across the temporal domain.

1. Problem setting and design objective

ATINet is formulated for light field object tracking in low-light scenes, with single object tracking as the primary task and multiple object tracking as an extension. The underlying premise is that a light field supplies a 4D representation,

$$L(u,v,x,y)\in\mathbb{R}^{U\times V\times W\times H},$$

where $(u,v)$ are angular coordinates and $(x,y)$ are spatial coordinates (Wang et al., 29 Jul 2025).

The method is motivated by a specific limitation of existing work: recent developments still struggle with reliable angular modeling in the temporal domain under difficult illumination. The proposed response is not merely to use multi-view information, but to define a geometric structure within the light field explicitly and then build a tracking architecture around that structure. In this sense, ATINet is not described as a generic video tracker adapted to light fields; it is a light-field-specific tracker whose representation, attention mechanism, and auxiliary objective are all organized around angular-temporal interaction.

A plausible implication is that the method treats low-light robustness less as a problem of raw photometric enhancement and more as a problem of extracting and propagating geometric structural cues that remain discriminative when appearance information becomes unreliable. That interpretation is consistent with the paper’s emphasis on angular gradients, object silhouettes in epipolar planes, and self-supervised reconstruction across time.

2. Epipolar-plane structure image representation

The ESI is the representational core of ATINet. It is derived from epipolar-plane images (EPIs), obtained by fixing one angular and one spatial coordinate. The horizontal and vertical forms are

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$

and

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$

For the horizontal EPI example, the first-order gradient in the angular domain is written as

$$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$

and Taylor expansion with symmetry $(d_l=d_r=d=1)$ gives

$\hat L_u' \simeq [\hat L(u+1)-\hat L(u-1)]/(2d).$

For each EPI pixel, the magnitude of the angular gradient is taken as a geometric-structure point. Projecting all such points back into the $(x,y)$ plane yields two 3D tensors,

$S_H(u,x,y),\qquad S_T(v,x,y).$

The final 2D ESI is

$(u,v)$0

These constructions are explicitly defined in the paper (Wang et al., 29 Jul 2025).

The ESI representation is characterized by three stated properties. By selecting only the largest first-order angular gradients, corresponding to the edges of object silhouettes in the epipolar planes, it discards the bulk of appearance redundancy inherent in full-resolution multi-view imagery, emphasizes purely geometric contour cues, and produces a single-channel image amenable to standard CNN or Transformer backbones even under very low overall light levels. In the logic of the method, ESI therefore functions both as a compression of the original 4D light field and as a structural filtering operation that privileges geometric evidence over redundant appearance content.

This suggests that ESI should be understood less as a conventional image transform than as a task-oriented structural representation: it is tailored to preserve angularly expressed geometry that can support localization and association when raw appearance becomes ambiguous.

3. Network architecture and the angular-temporal stream

ATINet is a two-stream tracker. The first stream is an appearance stream described as a “hybrid CNN.” Its inputs are the template ESI $(u,v)$1 and search ESI $(u,v)$2. A standard CNN backbone, with ResNet given as an example, produces feature maps $(u,v)$3 and $(u,v)$4, and depth-wise cross-correlation of these features yields an appearance correlation map $(u,v)$5 (Wang et al., 29 Jul 2025).

The second stream is the angular-temporal stream, which constitutes the ATINet core. It takes two ESIs from different time steps, $(u,v)$6. The baseline encoder is patchEmbed followed by $(u,v)$7 layers of Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN). If $(u,v)$8 are the patch embeddings, with $(u,v)$9 patches, then layer $(x,y)$0 is defined by

$(x,y)$1

and

$(x,y)$2

The attention term is

$(x,y)$3

with

$(x,y)$4

The paper states that this decomposes into four relation blocks, covering intra-frame and cross-frame interactions.

After encoding, the final $(x,y)$5 representation is split back into two framewise maps, reshaped to 2D to obtain $(x,y)$6 and $(x,y)$7, and then used to form an angular-temporal correlation map $(x,y)$8 through depthwise cross-correlation between template and search outputs. The head concatenates $(x,y)$9 and $$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$0 and applies classification and regression heads for target localization.

Architecturally, the two-stream design separates appearance evidence from angular-temporal geometric evidence, then reunifies them at the localization head. A plausible implication is that this division reduces the burden on any single branch: the CNN branch can preserve standard appearance cues, while the transformer-like branch models geometry-aware temporal relations that are specific to the light-field setting.

4. Geometry Adaptive Selection and self-supervised interaction learning

A central mechanism in ATINet is Geometry Adaptive Selection (GAS), inserted as a plug-in to each MHSA layer. A small MLP with Gumbel-Softmax predicts a 2-class label

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$1

for each embedding, distinguishing geometric embeddings $$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$2 from background embeddings $$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$3. This label is expanded to a 4-column one-hot tensor

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$4

which indicates $$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$5 (Wang et al., 29 Jul 2025).

A relation mask

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$6

is then formed via outer products of the columns of $$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$7 so that background embeddings relate only within their own frame, whereas geometric embeddings relate both within-frame and across-frames. If the original attention-score matrix is

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$8

then the masked attention score is

$$\hat L_H(u,x)=L(u,v_0,x,y_0)\in\mathbb{R}^{U\times W},$$9

followed by re-softmaxing in the usual way.

The baseline self-attention simultaneously models

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$0

for intra-frame relations and

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$1

for inter-frame relations, where

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$2

GAS further forces the network to focus these attention pathways only on embeddings corresponding to geometric structure points.

The angular-temporal stream is additionally trained with a self-supervised angular-temporal loss,

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$3

where

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$4

is the $$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$5 output of the GAS-enhanced encoder, $$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$6 is a random mask with drop rate $$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$7, and $$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$8 is a lightweight decoder consisting of a few layers of MHA and FFN with spatial-attention dropout that encourages cross-frame decoding. The stated purpose of this loss is to force the encoder and GAS to embed strong inter-frame geometric cues so that masked embeddings can be reconstructed only from the other frame’s geometric embeddings.

Conceptually, GAS and the self-supervised loss define the specific meaning of “angular-temporal interaction” in ATINet. It is not only the presence of cross-frame attention, but a masked and reconstruction-constrained interaction in which geometric structure points are granted privileged connectivity across time.

5. Optimization, datasets, and evaluation protocol

For single-object tracking, the total loss is

$$\hat L_T(v,y)=L(u_0,v,x_0,y)\in\mathbb{R}^{V\times H}.$$9

where $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$0 is the self-supervised reconstruction loss, $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$1 is focal loss on a Gaussian heat-map of the ground-truth center, and $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$2 is IoU loss on the predicted bounding box regression. The weights $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$3 are set by cross-validation (Wang et al., 29 Jul 2025).

The paper also introduces a large-scale light-field low-light dataset for object tracking. The capture setup uses a Raytrix R8 light-field camera and RxLive SDK, with raw-ray parsing to 5×5 angular views, 1080×1920 spatial resolution, 25 fps, and real low-light laboratory scenes. The single-object tracking dataset contains 173 light-field video sequences of approximately 150 frames each, with 160 unique objects including glass spheres, toy cars, industrial nuts, and fish, manually annotated with tight bounding boxes in each frame and with ESI computed. Of these, 102 videos, approximately 3,500 samples, are used for training, and 71 videos, approximately 15,380 samples, for testing. The multiple-object tracking dataset contains 26 multi-object light-field sequences with 7–8 objects per video and approximately 150 frames each; following MOT16 protocol, each object gets a unique ID. Thirteen videos, 1,800 samples, are used for training and thirteen videos, 1,500 samples, for testing.

The evaluation metrics are task-specific. For single-object tracking, the metrics are Precision, Normalized Precision, and Success, defined as AUC of overlap. For multiple-object tracking, the metrics are MOTA, IDF1, ID switches (IDS), False Positives (FP), and False Negatives (FN).

Setting	Dataset specification	Metrics
SOT	173 sequences; 102 training videos and 71 testing videos	Precision, Normalized Precision, Success
MOT	26 sequences; 13 training videos and 13 testing videos	MOTA, IDF1, IDS, FP, FN

The dataset construction is significant because the method is evaluated in a native light-field, low-light regime rather than only through adaptation of existing RGB benchmarks. A plausible implication is that ATINet’s reported performance is tied not only to model design but also to the availability of task-aligned supervision and a capture pipeline that preserves angular information.

6. Reported performance and extension to multiple object tracking

The reported experimental results indicate that ATINet achieves state-of-the-art performance in single object tracking. Specifically, the paper reports Success $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$4, Precision $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$5, and Normalized Precision $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$6, beating the best RGB-only baseline by approximately 3–4 points (Wang et al., 29 Jul 2025).

The method is also extended to multiple object tracking under the name AMTrack. This extension is based on the online joint-detection-and-tracking framework TRADES, into which GAS is inserted in the re-ID feature-extraction branch. A dual-stream re-ID layer encodes both appearance, using CNN features, and geometric features, using ATINet embeddings, for each detection. The MOT total loss is

$$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$7

where $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$8 is the same self-supervised loss, $$\hat L_u' = [\alpha\,\hat L(u)+\beta\,\hat L(u+d_l)+\gamma\,\hat L(u-d_r)] + \eta_u,$$9 is the object-detection loss consisting of classification and bounding-box regression, and $(d_l=d_r=d=1)$0 is the cost-volume association loss on the learned re-ID embeddings.

For MOT, AMTrack attains MOTA $(d_l=d_r=d=1)$1, IDF1 $(d_l=d_r=d=1)$2, IDS $(d_l=d_r=d=1)$3, FP $(d_l=d_r=d=1)$4, and FN $(d_l=d_r=d=1)$5, outperforming CenterTrack, Trades, ByteTrack, and Hybrid-SORT. These numbers are presented as evidence that the same angular-temporal modeling principles support not only target localization in SOT but also identity-preserving association in MOT.

A likely misconception would be to treat ATINet as a purely appearance-driven tracker supplemented by light-field input. The reported design and results indicate otherwise: the method’s stated contribution is the combination of a geometry-focused ESI input, multi-frame self-attention with explicit geometric-structure-point filtering through GAS, and a cross-frame self-supervised reconstruction loss. The paper further states that this combination yields a light-field tracking system that robustly localizes and associates targets in very dark environments while adding only modest compute overhead over an off-the-shelf Vision-Transformer backbone.