TRT-LOS: Transformer for 3D LOS Imaging

• TRT-LOS is a transformer-based method for 3D line-of-sight imaging that integrates dual spatio-temporal attention to reconstruct depth from transient measurements.
• The architecture employs hierarchical local and global encoders to capture fine spatial details and global contextual consistency, ensuring robust performance in low-photon scenarios.
• Experimental validation on synthetic and real-world data shows significant improvements in depth RMSE and detail recovery compared to traditional and CNN-based imaging approaches.

TRT-LOS refers to the transformer-based deep learning architecture for 3D line-of-sight (LOS) imaging reconstruction from time-resolved transient measurements. The term TRT-LOS is introduced as one task-specific instantiation of the generic Time-Resolved Transformer (TRT), purpose-built to address photon-efficient LOS imaging challenges, especially in low quantum efficiency sensor regimes and when measurements are affected by strong ambient noise and sparse photon counts (Li et al., 10 Oct 2025). TRT-LOS integrates dual spatio-temporal attention mechanisms and is benchmarked on both synthetic and real-world datasets, demonstrating significant advances in reconstruction fidelity, resolution, and noise robustness over existing approaches.

1. Architectural Overview

TRT-LOS is built on a hierarchical transformer design tailored for transient spatio-temporal data. Its main components include:

• Shallow Feature Extraction: Initial downsampling and representation of transient measurements (typically large 3D tensors: spatial × spatial × time) using interlaced 3D and dilated convolutions. This module prepares the data for attention-based processing by capturing basic spatial and temporal context.
• Dual Spatio-Temporal Self-Attention Encoders (STSA): Two encoders operate in parallel:
  • Local Encoder: Splits transient features into small spatial patches, then further divides along the temporal axis. Employs window-based spatial multi-head self-attention (Wₛ-MSA) followed by temporal window attention (Wₜ-MSA) and a feed-forward network (FFN).
  • Global Encoder: Downsamples spatial resolution to capture large-scale, nonlocal correlations, applies full spatial (Fₛ-MSA) and temporal (Fₜ-MSA) attention, followed by FFN.
• Spatio-Temporal Cross Attention Decoders (STCA): Two decoders fuse information from local and global branches in the token space:
  • One branch computes deep local features $F_L^*$ using upsampled global features as query and local features as key/value in the cross-attention.
  • The other branch computes deep global features $F_G^*$ by the reverse assignment.
  • The STCA operates by sequentially performing spatial and temporal cross-attention (matrix multiplications in reshaped token space), flanked by $1\times1\times1$ convolutions.
• Deep Feature Fusion & Upsampling: Uses temporal and 3D pixelshuffle operations to upsample features, producing a high-resolution transient reconstruction.
• Soft-argmax Depth Estimation: Final depth map is extracted from predicted histograms using soft-argmax over the temporal dimension.

The separation into local and global attention pathways enables TRT-LOS to capture both fine-grained spatial details (patch-level continuity of depth and intensity) and global contextual consistency (nonlocal scene structure).

2. Technical Mechanisms

Spatio-Temporal Attention Formulas

The key technical operations can be captured by the following LaTeX-expressed formulas:

• Local Encoder:

  $$F_L = \text{FFN}\{ \mathrm{W}_t\text{-MSA} \{ \mathrm{W}_s\text{-MSA} \{ F_S \} \} \}$$

  where $F_S$ is the shallow feature.

• Global Encoder:

  $$F_G = \text{FFN}\{ \mathrm{F}_t\text{-MSA} \{ \mathrm{F}_s\text{-MSA} \{ F_S^{\downarrow} \} \} \}$$

  with $F_S^{\downarrow}$ denoting spatially downsampled features.

• Cross Attention Decoders:

  $$F_L^* = \text{FFN}\left( \mathrm{STCA}[Q = F_G^{\uparrow}, K = F_L, V = F_L] \right)$$

  $$F_G^* = \text{FFN}\left( \mathrm{STCA}[Q = F_L, K = F_G^{\uparrow}, V = F_G^{\uparrow}] \right)$$

  where $Q$, $K$, $V$ (query, key, value) are generated via $1\times1\times1$ convolutions and upsampling ($\uparrow$).

Patch splitting, windowed attention, downsampling, and reshaping into token spaces (e.g., $HW\times D\times C$) ensure efficient self- and cross-attention across massive transient tensors.

Feature Fusion and Upsampling

Final fusion uses temporal and 3D pixelshuffle techniques to upsample the outputs of the decoder branches, yielding dense reconstructions that approximate the ideal transient histogram (each voxel representing photon arrivals over time).

Depth is extracted by soft-argmax over the upsampled temporal dimension, preserving accuracy in photon-limited regimes.

3. Experimental Validation

TRT-LOS is validated on both synthesized and experimental data:

• Synthetic Data: Includes high-resolution (up to $256\times256$ spatial, 1024 time bins) LOS datasets derived from Middlebury2014 and a new synthetic set with varied signal-to-background ratios (SBRs), emulating low photon count and noisy conditions.
• Real-World Measurements: SPAD-based single-photon imaging setups (indoor and long-range outdoor) provide production-grade transient datasets.

Performance Metrics:

• Depth RMSE: TRT-LOS achieves lower root mean squared error in depth estimation than both computational and CNN-based baselines (LM, Shin, Rapp, CAPSI; Lindell et al.; Peng et al.) under all SBR conditions and for both synthetic and real-world scenarios.
• Detail Recovery: Superior reconstruction of fine edges, textures, and depth boundaries (verified via visual, statistical, and scene structure analyses).
• Noise Robustness: Maintains accuracy under sparse photon conditions typical of low quantum efficiency sensors and long-range scenes.

Efficiency: The attention mechanisms allow better handling of the high-dimensional transient data by making multi-scale context integration more tractable.

4. Comparison with Traditional LOS Imaging

TRT-LOS departs from classic physical, iterative, or CNN methodologies by directly modeling the spatio-temporal correlations in transient measurements using transformer mechanisms:

• Handcrafted Models: Physics-based iterative methods may fail under high noise and sparse measurements, lacking global context for consistent reconstruction.
• CNN-based Approaches: While computationally efficient, CNNs are limited in capturing nonlocal dependencies, especially for 3D transient data.
• Attention-based Approach: Dual attention in TRT-LOS allows fusion of patchwise (local) and scene-level (global) features, explicitly addressing noise, sparsity, and contextual ambiguity.

This indicates a generational shift toward transformer-style architectures in photon-efficient imaging modalities.

5. Broader Applications and Implications

TRT-LOS is suited to several scientific and technological domains:

• Remote Sensing and LiDAR: Recovery from sparse, noisy photon event histograms for high-resolution 3D mapping.
• Autonomous Navigation: Robust depth estimation in adverse environments with low illumination or strong background light.
• Security and Scientific Imaging: Enables accurate 3D reconstruction in challenging photon-limited regimes (e.g., space exploration, biomedical imaging).

A plausible implication is that attention-based architectures like TRT-LOS will become foundational for future advances in 3D imaging from time-resolved data, especially in applications demanding photon efficiency and resilience to ambient noise.

6. Impact and Future Directions

By establishing a transformer-based approach for LOS imaging (and its NLOS sibling, TRT-NLOS), TRT-LOS highlights:

• The feasibility of learning-based methods for photon-limited, high-dimensional transient reconstruction tasks.
• The technical advantage of integrating multi-scale spatio-temporal attention for robustness and accuracy.
• The prospect for further optimization (e.g., larger scales, deeper architectures, integration of physical priors).

Future research may explore training more complex variants, better integration with hardware, and extension to broader time-resolved imaging applications.

Summary Table

Component	Function	Methodology
Feature Extraction	Downsample & preprocess transient data	Interlaced & dilated 3D convolutions
STSA Encoder	Capture local/global correlations	Patch/window-based self-attention (local/global)
STCA Decoder	Fuse local & global features	Dual cross-attention in token space
Feature Fusion	Upsample & reconstruct histogram	Pixelshuffle, soft-argmax
Experimental Eval	Assess accuracy and robustness	Synthetic/real data, RMSE, visual analysis

TRT-LOS exemplifies state-of-the-art transient imaging reconstruction by leveraging dual spatio-temporal transformer attention mechanisms, achieving superior accuracy and resilience in photon-efficient LOS imaging scenarios (Li et al., 10 Oct 2025).