---
title: Light Field Object Tracking Dataset
url: https://www.emergentmind.com/topics/light-field-object-tracking-dataset
type: topic
---

# Light Field Object Tracking Dataset

A light field object tracking dataset comprises a curated collection of digital plenoptic (light field) images and/or videos, together with ground-truth object annotations, for the development and benchmarking of object tracking algorithms that exploit the angular and spatial richness of light field sensors. Such datasets support fine-grained spatial, angular, and sometimes temporal tracking in scenarios where traditional RGB data is insufficient, notably under complex visual effects (e.g., reflectivity, occlusion, or low-light conditions). Recent datasets have established protocols for 6DoF pose estimation as well as single-object and multi-object tracking benchmarks, incorporating both real and synthetic content, high-precision ground truth, and challenging object/material compositions [2512.13007][2507.21460].

## 1. Sensor Configurations and Light Field Acquisition

Recent datasets employ custom or commercial plenoptic cameras, capturing dense spatial-angular samples for each scene. Key technical configurations include:

- **Custom 4D Plenoptic System** (LiFT-6DoF dataset [2512.13007]):
  - Imaging sensor: global-shutter CMOS, 12 MP (4000×3000 px).
  - Microlens array: 17×17 mm², 0.2 mm pitch, 0.85 mm focal length, placed at sensor focal plane.
  - Objective lens: f = 50 mm, f/2.8.
  - Angular sampling: Uniform 9×9 grid (U = V = 9).
  - Each subview: 444×444 px, ≈0.9 MP.
  - Extrinsic calibration: Each subview indexed by (s, t) has an explicit 4×4 rigid transform $T_{s,t}$, with translation increments of 5 mm and identity rotation.

- **Raytrix R8 System** (large-scale low-light dataset [2507.21460]):
  - Commercial plenoptic R8 camera, with a 5×5 micro-lens array.
  - Per-view resolution: 1080×1920 pixels.
  - Frame rate: 25 Hz.
  - Angular sampling: U = V = 5.

Intrinsic parameters (camera matrices, distortion corrections) are included with each dataset, enabling precise ray-based reconstruction or forward modeling.

## 2. Object Inventory, Scene Design, and Lighting Regimes

Datasets are constructed to probe tracking robustness under diverse object geometry, material response, and environment:

- **LiFT-6DoF dataset** [2512.13007]:
  - **Natural objects**: Box (diffuse), Tea Box (textured), Shiny Box (anodized with specularities), Jug (glossy ceramic, with tilt variation).
  - **Synthetic objects**: Blender-rendered Toy Car (matte and specular variants).
  - Material properties: Diffuse (albedo <10%), Specular (reflectivity >60%, Phong roughness ≈0.1), Textured (PBR-mapped).
  - Lighting: Natural—4 overhead LED panels (500 lux) + 2 spotlights (1,000 lux); Synthetic—HDRI environments.
  - Neutral gray backdrops; distractor clutter in synthetic set.

- **Low-light tracking dataset** [2507.21460]:
  - Object classes: Toy cars, glass marbles, industrial nuts, live fish, generic deforming shapes.
  - Environment: Real indoor scenes captured under low-illumination; motion blur and reduced signal-to-noise are defining characteristics.
  - Tabletop and aquatic setups permit frequent occlusions, high deformation, and distractor similarity.

## 3. Annotation Methodologies and Ground-Truth Quality

Annotation fidelity is central to dataset utility for pose estimation and tracking metrics:

- **LiFT-6DoF** [2512.13007]:
  - Pose annotation: 4×4 SE(3) homogeneous matrices $T_t = [R_t\; t_t; 0\; 1]$, with $R_t \in SO(3)$ and $t_t \in \mathbb{R}^3$. Rotations also recast as Hamiltonian quaternions $(w,x,y,z)$.
  - Coordinate system: Camera-centric, origin at central view’s optical center, z along optical axis.
  - Annotation device: Robotized linear/rotary stage, encoder resolution 0.01°/0.01 mm; resulting accuracy: translation ±0.05 mm, rotation ±0.02°, depth (via RealSense D435i) ±1 mm.
  - Synthetic data inherits ground truth directly from Blender scene state.
  - Disparity maps provided (16-bit PNG) for central view.

- **Low-light dataset** [2507.21460]:
  - 2D bounding boxes: (x, y, w, h), per central image and frame.
  - Unique track IDs (MOT16 protocol); continuity maintained by expert annotation (30 annotators) through manual inspection.
  - Epipolar-plane structure images (ESI) $S(x,y) = \sqrt{(\partial_u L)^2 + (\partial_v L)^2}$ are distributed for geometric cue exploitation.
  - No instance-level segmentation or 3D ground truth supplied.

## 4. Dataset Composition, Structure, and Splits

Datasets are systematically organized to enable reproducible benchmark construction, with clear train/test partitions.

**LiFT-6DoF dataset [2512.13007]**

| Content        | Real (6 seqs)               | Synthetic (3 seqs)        | Per-sequence Frames |
|----------------|----------------------------|---------------------------|---------------------|
| Materials      | Diffuse, glossy, textured  | Matte/shiny car           | 15–20               |
| Directory tree | `real/<object>`, `synthetic/<object>` |                           |                     |
| Data per frame | 9×9 PNG subviews, depth PNG, pose (.txt) |             |                     |
| Train split    | Box, TeaBox                | Car, ShinyCar             |                     |
| Test split     | Jug, ShinyBox, Jug_Tilt, TeaBox_Tilt |           |                     |

Total: ≈108 real + 36 synthetic = ≈144 frames.

**Low-light tracking dataset [2507.21460]**

| Benchmark Task | Sequences | Train/Test Split | Frame Counts              |
|----------------|-----------|------------------|---------------------------|
| MOT            | 52 (26+26)| 26 train / 26 test| ≈3,900                    |
| SOT            | 173       | 102 train / 71 test| ≈3,500 train, 15,380 test |

Both real data and synthetic data are organized as sequences containing raw views, derived features/maps, and annotation files.

## 5. Benchmarking Protocols and Evaluation Metrics

Standardized metrics facilitate comparison across methods and datasets:

- **6DoF pose tracking (LiFT-6DoF [2512.13007]):**
  - ADD (Average Distance of Model Points):
    $$
    \operatorname{ADD} = \frac{1}{|M|} \sum_{x \in M} \| (R x + t) - (\hat{R} x + \hat{t}) \|_2
    $$
  - ADD-S (mean point-to-closest-point for symmetric objects).
  - AUC (area under ADD/[ADD-S] curve, threshold 0.1 m).
  - Absolute rotation error: $\Delta R_{\mathrm{abs}} = \arccos \left( \frac{\operatorname{trace}(R^\top \hat{R}) - 1}{2} \right)$ (in degrees).
  - Absolute translation error: $\Delta t_{\mathrm{abs}} = \| t - \hat{t} \|_2$ (in metres).
  - Relative errors: frame-to-frame change $\Delta T = T_{t-1}^{-1} T_t$.
  - Success rates: % frames with $\Delta t_{\mathrm{abs}} < 0.02$ m and $\Delta R_{\mathrm{abs}} < 5^\circ$.

- **Single/Multi-object tracking (Low-light dataset [2507.21460]):**
  - SOT: Precision (@20 px), Success (IoU curve), Normalized Precision, centered-error and overlap curves as in OTB.
  - MOT: MOTA, IDF1, ID switches, FP, FN, per CLEAR-MOT protocol.

Reported methods demonstrate the efficacy of light field cues. For instance, ATINet achieves SOT Success = 0.64, Precision = 0.79, Norm. Prec. = 0.81; AMTrack MOTA = 87.5%, IDF1 = 85.4% [2507.21460]. In 6DoF pose tracking, dataset baselines compare light-field features and vision foundation model-based splats to state-of-the-art reference model approaches such as FoundationPose, with similar translation and rotation accuracy [2512.13007].

## 6. Unique Features and Comparative Analysis

Light field tracking datasets distinguish themselves from prior benchmarks by:

- Providing dense angular sampling (e.g., 9×9 or 5×5) for spatial-angular method evaluation, unavailable in conventional RGB datasets.
- Enabling robust tracking under:
  - Reflective and visually complex objects (LiFT-6DoF [2512.13007]).
  - Extreme low-light and cluttered conditions, including live, deformable, and highly occluded targets (Low-light dataset [2507.21460]).
- Including precise, robotically measured 6DoF ground truth (sub-millimeter, sub-degree error), as well as large-scale video sequences for temporal modeling.
- Supplying supporting calibration, disparity data, and explicit intrinsic/extrinsic parameters.

Earlier datasets (e.g., HCI-LF, Stanford Lytro Illum, UrbanLF) are limited to static or focal stack images, lacking both dynamic frames and realistic low-light scenarios.

## 7. Applications and Availability

Light field object tracking datasets facilitate research in:

- 3D pose tracking (6DoF) from multi-view inputs, without pretrained object models.
- Robust tracking in robotic manipulation, autonomous driving, and dynamic scene understanding.
- Benchmarking novel representations (e.g., view-dependent Gaussian splats, ESI features), self-supervised temporal modeling (e.g., ATINet), and spatial-angular interaction networks.

Datasets, code, and detailed documentation are publicly released (e.g., https://github.com/nagonch/LiFT-6DoF [2512.13007]), supporting immediate experimental reproducibility with exact camera intrinsics, extrinsics, and file structures defined for direct integration into modern ray-based or epipolar-plane analysis pipelines.

Source: https://www.emergentmind.com/topics/light-field-object-tracking-dataset