LATTE: LiDAR Annotation System

• LATTE is an open-source modular system that accelerates LiDAR point cloud annotation for autonomous driving through sensor fusion and one-click interface tools.
• It integrates sensor fusion pre-labeling, geometric box estimation, and tracking-assisted propagation to streamline the labeling process and reduce redundancy.
• Field evaluations on KITTI-like data show a 6.2× speed-up and improved IoU, precision, and recall, proving its efficacy for large-scale, high-quality 3D dataset generation.

to=arxiv_search.search 体育彩票天天json content='{"query":"(Wang et al., 2019) OR title:LATTE", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}' / to=arxiv_search.search 尚度json content='{"query":"all:LiDAR point cloud annotation sensor fusion tracking one-click annotation", "max_results": 5, "sort_by": "relevance", "sort_order": "descending"}' / LATTE is an open-source, modular system for accelerating and improving LiDAR point cloud annotation in autonomous driving, especially for L4–L5 applications where large, high-quality 3D datasets are needed to train deep models. It addresses three specific impediments to LiDAR labeling: sparsity and low resolution in point clouds, the operational complexity of drawing 3D boxes, and redundancy across synchronized sequences. Its pipeline combines sensor fusion pre-labeling, one-click annotation, and tracking-assisted propagation, yielding both faster annotation and better label quality on KITTI-like data (Wang et al., 2019).

1. Problem setting and design objective

LATTE is motivated by the mismatch between the data requirements of LiDAR-based detection and the labor intensity of producing labels. The system description identifies three root causes. First, LiDAR point clouds are sparse and low resolution. A 64-line LiDAR’s vertical angular resolution is approximately $0.41^\circ$, so at $50$ m the spatial sampling is only approximately $0.36$ m; distant objects such as poles or cyclists therefore appear as very sparse collections of points. Second, 3D box annotation is intrinsically higher-dimensional than 2D image annotation: even top-view boxes still require yaw, two scales, and a translation, while full 3D boxes additionally require center and height parameters. Third, autonomous-vehicle datasets are collected as synchronized sequences, so consecutive frames are highly correlated and a naively manual workflow repeats nearly identical annotation work across time (Wang et al., 2019).

Within this setting, LATTE is designed as a human-in-the-loop system rather than a fully automatic annotator. Image-derived labels are treated as pre-labels, box propagation is treated as prediction, and the annotator remains responsible for validation, adjustment, and category confirmation. This framing is central to the system’s quality-control logic: automation reduces operator burden, but final authority remains with the annotator.

2. System architecture and sensor-fusion pre-labeling

Architecturally, LATTE begins from a calibrated camera–LiDAR pair. LiDAR points are projected into image coordinates, image-based masks are predicted, and those masks are lifted back into 3D to highlight candidate points for annotation. In the simplified linear model used in the system description, a LiDAR point $\mathbf{p}_i=(x_i,y_i,z_i)$ maps to an image pixel $\mathbf{q}_i=(u_i,v_i)$ through

$\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$

while the more general perspective form is

$$\lambda \begin{bmatrix} u\ v\ 1 \end{bmatrix} = K[\,R\mid t\,] \begin{bmatrix} X\ Y\ Z\ 1 \end{bmatrix}.$$

LATTE uses Mask R-CNN to generate per-pixel semantic masks $\mathcal{M}$, and transfers labels to LiDAR points by

$l_i=\mathcal{M}(u_i,v_i).$

The highlighted 3D points then guide the annotator’s attention, and points inside a selected box can be projected back to the image to produce a cropped patch for visual category confirmation (Wang et al., 2019).

This sensor-fusion stage is explicitly not regarded as ground truth generation. The system description emphasizes detector error, projection error, calibration error, synchronization error, and occlusion as practical failure sources. Cropped image confirmation and human adjustment are therefore part of the core workflow, not ancillary tooling. A common misconception is that LATTE replaces 3D annotation with 2D detector transfer; in fact, the transferred masks only bootstrap the annotator, while the one-click clustering and box fitting stages operate in 3D space and compensate for incomplete or misaligned image masks.

3. One-click annotation and geometric box estimation

LATTE’s one-click interface reduces the operator burden to a single click on a target point, after which the system performs ground removal, local clustering from the clicked seed, and top-view oriented rectangle fitting. The ground is modeled as piecewise planar, with each plane satisfying

$\mathbf{n}^{T}\mathbf{p}=d.$

An initial normal is estimated from low-$50$0 points using covariance and SVD, then iteratively refined by thresholding points close to the plane: $50$1 Removing ground points reduces clutter before object extraction (Wang et al., 2019).

Local clustering is then performed by a density-based expansion similar to DBSCAN. Starting from the clicked seed $50$2, LATTE searches neighbors within radius $50$3, expands via breadth-first search, and forms the object cluster from the visited points. To keep latency low on dense frames, the implementation prunes or downsamples with size priors; the description notes that box areas empirically concentrate near approximately $50$4, typical of cars.

For box fitting, LATTE represents the cluster in ground-plane coordinates and searches headings $50$5. With orthogonal unit vectors

$50$6

the cluster is projected onto the two axes, and a search-based loss over distances to candidate rectangle edges is minimized to obtain $50$7. The resulting rectangle is parameterized by center $50$8, dimensions $50$9, and yaw $0.36$0. LATTE can then infer a 3D box or convert the box to point-wise labels for semantic segmentation by assigning all points inside the box to the object class, relying on the observation that road objects rarely overlap in 3D (Wang et al., 2019).

The significance of this design is operational rather than purely algorithmic. Instead of asking an annotator to manipulate all box degrees of freedom directly, LATTE converts the annotator’s role into seed selection and occasional fine-tuning. This shifts most geometric estimation into deterministic preprocessing and thereby reduces both per-instance time and inter-annotator inconsistency.

4. Tracking-assisted sequence annotation

To address sequence redundancy, LATTE integrates tracking so that labels from one frame can be propagated forward and then validated or corrected. The tracker models the object center in the ground plane with a Kalman filter whose state is

$0.36$1

using a constant-acceleration transition model with time step $0.36$2. Prediction and covariance update follow the standard linear form

$0.36$3

and the annotator’s corrected box center serves as the measurement

$0.36$4

The update then uses the usual Kalman gain $0.36$5, observation matrix $0.36$6, and observation covariance $0.36$7 set by LiDAR resolution (Wang et al., 2019).

The workflow distinguishes rigid and non-rigid objects. For rigid objects such as cars, shape is maintained while yaw may be re-estimated. For non-rigid objects such as pedestrians, LATTE re-runs the one-click estimation pipeline at each step, using the predicted center as guidance and the annotator’s adjustment as the measurement update. The system description notes that more elaborate multi-target association could be added, for example with the Hungarian algorithm, but the practical implementation relies on human confirmation for each proposed continuation.

This tracking design clarifies another misconception: LATTE does not treat tracking as autonomous label propagation. Rather, it uses tracking to propose likely continuations and to reduce duplicated work while preserving an explicit correction loop. The intended gain is therefore in annotation throughput and consistency, not in replacing supervision.

5. Evaluation, annotation quality, and deployment practice

LATTE was evaluated on 30 KITTI sequences, each with 5 frames, comprising 1,116 instances and annotated by nine volunteers. Ground truth was curated by an expert, and human–human agreement on IoU averaged $0.36$8 with standard deviation $0.36$9. Using a baseline in which annotators manually drew top-view boxes, the full LATTE pipeline achieved a $\mathbf{p}_i=(x_i,y_i,z_i)$0 speed-up from $\mathbf{p}_i=(x_i,y_i,z_i)$1 s to $\mathbf{p}_i=(x_i,y_i,z_i)$2 s per instance, while operations decreased from $\mathbf{p}_i=(x_i,y_i,z_i)$3 to $\mathbf{p}_i=(x_i,y_i,z_i)$4. Label quality also improved: box IoU increased from $\mathbf{p}_i=(x_i,y_i,z_i)$5 to $\mathbf{p}_i=(x_i,y_i,z_i)$6, precision from $\mathbf{p}_i=(x_i,y_i,z_i)$7 to $\mathbf{p}_i=(x_i,y_i,z_i)$8, and recall from $\mathbf{p}_i=(x_i,y_i,z_i)$9 to $\mathbf{q}_i=(u_i,v_i)$0. The paper defines

$\mathbf{q}_i=(u_i,v_i)$1

with a true-positive threshold of IoU $\mathbf{q}_i=(u_i,v_i)$2 for instance-level precision and recall (Wang et al., 2019).

Ablation results indicate that each major component contributed independently. Sensor fusion yielded $\mathbf{q}_i=(u_i,v_i)$3 s per instance with IoU $\mathbf{q}_i=(u_i,v_i)$4, Precision $\mathbf{q}_i=(u_i,v_i)$5, and Recall $\mathbf{q}_i=(u_i,v_i)$6. One-click annotation yielded $\mathbf{q}_i=(u_i,v_i)$7 s with IoU $\mathbf{q}_i=(u_i,v_i)$8, Precision $\mathbf{q}_i=(u_i,v_i)$9, and Recall $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$0. Tracking yielded $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$1 s with IoU $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$2, Precision $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$3, and Recall $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$4. The full pipeline therefore outperformed any individual component alone.

Deployment guidance in the system description is correspondingly pragmatic. Calibration between camera and LiDAR should be well maintained; timestamps should be synchronized; Mask R-CNN is run on images and its masks projected to point clouds; first-frame annotation relies on sensor fusion and one-click fitting; subsequent frames rely more heavily on tracking proposals. Parameter tuning is likewise operational: $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$5 should match LiDAR point spacing, the ground threshold should capture the road plane while excluding low obstacles, cluster pruning should reflect realistic size priors, and $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$6 and $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$7 should be tuned in physical units while residuals are monitored for drift.

6. Limitations, comparisons, and naming ambiguity

LATTE’s limitations are explicitly tied to the same three modules that make it effective. Dense traffic and severe occlusion can cause image masks to miss object parts, clustering to merge adjacent objects, and tracking associations to become ambiguous. Poor calibration or time synchronization increases projection and back-projection error. Far-range sparsity can leave too few points for reliable fitting. Abrupt motion or non-linear trajectories can cause the constant-acceleration tracker to drift. The recommended mitigations are narrower $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$8, stricter pruning, recalibration, accurate timestamps, shorter $\mathbf{q}_i = P\mathbf{p}_i + \mathbf{t},$9, retuning $$\lambda \begin{bmatrix} u\ v\ 1 \end{bmatrix} = K[\,R\mid t\,] \begin{bmatrix} X\ Y\ Z\ 1 \end{bmatrix}.$$0, and liberal human correction; annotators may also skip very low-visibility instances, with the system description noting a visibility rule such as “$$\lambda \begin{bmatrix} u\ v\ 1 \end{bmatrix} = K[\,R\mid t\,] \begin{bmatrix} X\ Y\ Z\ 1 \end{bmatrix}.$$1 two edges visible” for labeling (Wang et al., 2019).

In comparative terms, LATTE is distinguished from primarily image-focused annotation tools such as VIA, PolygonRNN, VATIC, and Scalabel by three characteristics stated in the paper: camera–LiDAR sensor fusion to bootstrap pre-labels, one-click 3D annotation for rapid box estimation, and tracking-based propagation across frames. It is also contrasted with prior LiDAR fusion pipelines such as Apolloscape and work by Piewak et al., which either focused on static background or were not open-sourced. LATTE instead targets moving objects and is released as an open, modular system.

The name itself is not unique within the arXiv literature. Unrelated systems later reused “Latte” or “LATTE” for collaborative test-time adaptation of vision-LLMs in federated learning (Bao et al., 29 Jul 2025), a layerwise timestep-expert multimodal generator (Shen et al., 8 Jun 2025), a quantum decoding architecture (Zhang et al., 4 Sep 2025), and a cross-framework evaluation package for latent-based generative models (Watcharasupat et al., 2021). This suggests that “LATTE” functions as a recurring acronym across multiple fields rather than as a single continuing research lineage. Within autonomous-driving annotation, however, the term specifically denotes the LiDAR annotation tool defined by sensor fusion, one-click annotation, and tracking-assisted sequence labeling (Wang et al., 2019).