Papers
Topics
Authors
Recent
Search
2000 character limit reached

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Published 4 Aug 2026 in cs.CV and cs.AI | (2608.07579v1)

Abstract: The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Summary

  • The paper demonstrates that a calibrated geometry-first pipeline achieves 13.04 HOTA versus 0.12 for estimated-depth pseudo-LiDAR, showing that cross-view metric consistency matters more than dense monocular depth in warehouse tracking.
  • The paper projects 2D detection footpoints onto a shared world ground plane, fuses observations across synchronized cameras, and uses motion-based tracking plus offline stitching to improve association from 14.53 to 16.71 AssA.
  • The paper identifies detection and Sim2Real generalization as the main geometry-route bottlenecks, with DetA at 10.79 and real-world recall losses, while inconsistent reconstructed floors drive pseudo-LiDAR localization down to 9.23 LocA.

The paper presents a controlled comparison of two RGB-only approaches to multi-camera 3D tracking under the synthetic-to-real conditions of AI City Challenge 2026 Track 1 (2608.07579). Its central claim is that cross-view metric consistency is more important than the per-image accuracy or detail of monocular depth. The evidence is deliberately operational: a calibrated, geometry-first pipeline obtains 13.04 3D HOTA, whereas an estimated-depth pseudo-LiDAR system obtains only 0.12 HOTA. The approximately two-order-of-magnitude gap is attributed primarily to the failure of independently estimated monocular depths to agree when transformed into a common world coordinate system.

Task formulation and experimental premise

The task requires synchronized multi-camera detection and tracking of seven object categories in large indoor warehouses. Predictions consist of world-coordinate 3D bounding boxes, object identities, frame indices, dimensions, and yaw. Training data includes synthetic RGB, depth, calibration, maps, and annotations, but inference is RGB-only. The hidden test set introduces real-world imagery and additional visual stressors, making the problem both a multi-camera association task and a Sim2Real generalization problem.

Evaluation uses 3D HOTA, decomposed into detection accuracy, association accuracy, and localization accuracy. This decomposition is important to the paper’s analysis because it distinguishes failures caused by missing or spurious scene-level detections from failures in identity maintenance or geometric localization. The authors’ best system has relatively stable localization, with LocA of 51.58, while DetA remains only 10.79. Thus, the headline HOTA score is constrained primarily by the quality of the generated world-frame detections rather than by the tracker’s ability to associate already-correct observations.

The comparison is designed around a specific contrast. The geometry-first route imposes a shared calibrated ground plane across cameras, thereby privileging cross-view agreement while discarding dense depth. The pseudo-LiDAR route retains dense per-pixel depth structure but estimates each camera’s geometry independently. The paper therefore tests whether explicit geometric consistency or learned monocular metric detail is the more consequential property for this setting.

Geometry-first pipeline

The primary system uses YOLO11x at a $1280$ input resolution for per-camera 2D detection. Each detection is represented by a bounding box and class label, with a low confidence threshold used during inference to preserve recall. Subsequent filtering is performed during fusion, tracking, and submission post-processing.

The central geometric operation is a homography-based projection of the detection footpoint into the ground/world plane. Specifically, the bottom-center of the 2D box is projected using camera calibration, with class-level priors supplying the object’s vertical coordinate and 3D dimensions. The selected projection configuration uses an inverted homography, a footpoint parameter of 1.00, and the world xyxy plane. The resulting median validation lift error is approximately 0.633 m, providing a comparatively reliable metric basis despite the absence of inference-time depth.

Figure 1

Figure 1: Comparison between the geometry-first RGB route and the estimated-depth pseudo-LiDAR route.

After lifting, detections from synchronized cameras are fused in world space on a per-frame and per-class basis. Spatially proximate observations are merged to reduce duplicate detections from overlapping views. This design makes the multi-camera constraint explicit: all observations are expressed in the same calibrated coordinate system before identity association.

Tracking is performed independently for each class. A fused observation is assigned to the nearest active track when the track’s motion-predicted position lies within a class-dependent distance gate. Short gaps are bridged by constant-velocity extrapolation, or coasting, for a fixed number of frames. Tracks and observations that violate class priors or fail the relevant filtering criteria are removed.

Figure 2

Figure 2: World-coordinate fusion, gated association, motion-based coasting, and track termination.

The authors add offline tracklet stitching to repair fragmentation caused by gaps longer than the online coasting window. Same-class tracklets are linked when their temporal gap and spatial residual relative to velocity extrapolation satisfy predefined thresholds. One-to-one matching and union-find chaining produce scene-level identities. Because this operation changes only identity labels and not box geometry, its effect isolates association quality. It increases AssA from 14.53 to 16.71 and HOTA from 12.49 to 13.04 while leaving DetA and LocA essentially unchanged. The implication is direct: identity fragmentation is a measurable but secondary error source, and offline relinking is effective precisely because it does not disturb the already more reliable geometric component.

Estimated-depth pseudo-LiDAR

The alternative system follows the standard pseudo-LiDAR formulation: monocular depth is estimated independently for each camera, back-projected using calibration, transformed into the world frame, fused into a scene point cloud, and processed by V-DETR. The authors evaluate D4RT and Metric3D v2, including fine-tuning on estimated-depth point clouds to address the train–test depth-domain mismatch.

The results reject the assumption that improved monocular depth can simply substitute for provided depth in a multi-camera 3D detector. D4RT requires an approximately 4.3×4.3\times scale correction to approach metric scale. Metric3D v2 is metric by construction, but its reconstructed floors remain inconsistent across cameras. Scale correction therefore addresses only a global scale error; it does not enforce agreement between independently estimated camera geometries.

The paper quantifies this failure using floor coherence, an annotation-free diagnostic based on the known planar warehouse floor. With provided depth, 39% of points lie within ±0.3\pm 0.3 m of the floor and none lie below it. Scale-corrected D4RT places only 20% of points in that band, with 20% below the floor. Metric3D v2 reconstructs essentially no coherent floor. Since objects rest on the floor, a non-planar or floating floor implies systematically displaced object geometry after fusion. The diagnostic consequently measures a necessary property of usable scene reconstruction, not merely a superficial correlation with performance.

The resulting pseudo-LiDAR system obtains HOTA of 0.12, DetA of 0.05, AssA of 0.26, and LocA of 9.23. In contrast, the geometry-first system reaches HOTA 13.04, DetA 10.79, AssA 16.71, and LocA 51.58.

Route HOTA DetA AssA LocA
Geometry-first lifting 13.04 10.79 16.71 51.58
Estimated-depth pseudo-LiDAR 0.12 0.05 0.26 9.23

The particularly large LocA degradation establishes that the pseudo-LiDAR failure is not primarily a detector or association problem. V-DETR receives a geometrically incoherent point cloud, so its predicted boxes are poorly localized before tracking can contribute meaningfully.

Figure 3

Figure 3: World-coordinate predictions and ground truth, showing moderate alignment for matched predictions but substantial over-detection and lift-induced outliers.

Domain-adaptation fine-tuning does not repair the pseudo-LiDAR route within the available computational budget. This result limits the paper’s claim: it demonstrates that the tested monocular-depth models, calibration procedure, fusion strategy, and V-DETR configuration are inadequate in combination, rather than proving that all learned depth approaches are intrinsically unsuitable. Nevertheless, the floor-coherence measurements provide a plausible mechanistic explanation for the observed collapse.

Ablation evidence and bottleneck localization

The ablations support a clear partition of failure modes. Detector-side interventions—including low-confidence thresholding, SAHI, RT-DETR ensembling, test-time augmentation, and domain-randomized YOLO26 training—do not improve the baseline. SAHI is the most informative negative result: it more than doubles the number of detections but lowers DetA from 10.79 to 7.27 and HOTA from 13.04 to 11.49. The added recall is overwhelmed by false positives and erroneous lifted boxes. The implication is that increasing detection quantity without improving precision and cross-view validity is counterproductive in this pipeline.

The detector’s validation statistics support this interpretation. Across the evaluated classes, overall precision is 0.907 but recall is only 0.603. Recall varies sharply, from 0.806 for NovaCarter and 0.723 for Person to only 0.239 for PalletTruck. The detector therefore exhibits a precision–recall imbalance that directly limits DetA. The substantially higher training-split recall indicates a domain-generalization failure rather than an inability to represent the object categories.

Class or aggregate Precision Recall mAP50 mAP50–95
All evaluated classes 0.907 0.603 0.655 0.486
Person 0.817 0.723 0.753 0.577
Forklift 0.812 0.601 0.646 0.438
NovaCarter 0.989 0.806 0.840 0.709
Transporter 0.968 0.647 0.729 0.521
PalletTruck 0.950 0.239 0.309 0.185

Geometry-side interventions are more damaging. A learned bounding-box-to-3D MLP reduces HOTA to 0.94 and LocA to 39.0, indicating that a crop- or box-only regressor lacks sufficient information for reliable metric localization. Replacing calibrated lifting with monocular-depth pseudo-LiDAR is even more severe. These results support the paper’s stronger design claim: the calibrated geometric lift is not merely a simple baseline component; under the tested Sim2Real regime, it is the most trustworthy source of shared metric structure.

Association-side results are mixed. Appearance-based Re-ID lowers HOTA to 10.51, presumably because low-resolution, self-similar, domain-shifted crops yield unreliable embeddings. Offline stitching is the sole successful intervention, raising HOTA by 0.55 points. This pattern implies that association is not the dominant bottleneck, although a specific form of association error—fragmentation after extended detection gaps—can still be corrected profitably.

Qualitative failure analysis

The qualitative overlays reveal two complementary manifestations of the same 2D-to-3D weakness. In synthetic scenes, projected world-frame boxes frequently over-detect and place their ground-plane bases incorrectly. In real scenes, 2D detections of people may be visually correct but fail to propagate into stable world-frame tracks, causing under-coverage. Thus, the lift and tracking stages are not uniformly defective: the system can maintain trajectories when valid world-frame observations exist, but the production of those observations is unreliable across domains.

Figure 4

Figure 4: Synthetic-scene over-detection and homography misplacement contrasted with real-scene under-coverage despite valid 2D detections.

A validation-frame analysis further exposes the DetA ceiling. The final submission produces 120 predictions for 67 ground-truth objects, including 74 false positives and 46 true positives. Matched predictions align reasonably with ground truth, consistent with LocA being the strongest component, but the large number of false positives prevents detection accuracy from improving. Some errors originate in incorrect lifting, duplicate multi-camera observations, or prolonged coasting outside valid object locations.

The Sim2Real gap is especially important. Real scenes exhibit markedly lower recall than synthetic scenes, and therefore contribute disproportionately to missed detections. The paper’s claim that detection quality is the primary geometry-route bottleneck is supported by three convergent observations: substantial class-wise recall variation, simultaneous false-positive and false-negative errors, and a pronounced synthetic-to-real degradation. A more granular analysis by object distance, occlusion, and camera would be useful, but the paper explicitly notes that such a breakdown is deferred.

Limitations and open questions

The reported leaderboard scores are obtained from the public evaluation server, which ranks submissions on approximately half of the test set; full-test performance may differ. The study also does not provide complete per-scene HOTA or recall breakdowns, limiting the ability to quantify whether the pseudo-LiDAR collapse is uniform or concentrated in particular cameras, object classes, or domains.

Several comparisons are controlled at the systems level rather than through exhaustive optimization of every component. The pseudo-LiDAR route uses V-DETR trained on provided-depth clouds, with estimated-depth fine-tuning attempted only within a stated compute budget. Consequently, the negative result leaves open whether stronger depth adaptation, temporal depth consistency, joint multi-camera depth estimation, explicit floor-plane constraints, or a detector trained from the outset on the induced pseudo-LiDAR distribution could materially change the outcome. Likewise, the geometry-first route uses class-level dimensions and heuristic yaw, so its localization ceiling may reflect these simplifications even though LocA is comparatively strong.

The floor-coherence metric is useful because it is annotation-free and directly probes global consistency, but it is only a proxy. It does not fully characterize object-level reprojection error, dynamic-scene reconstruction, or errors in camera calibration. The paper also does not establish whether the same ordering persists in environments without a dominant visible planar floor. These limitations qualify the broad formulation “geometry beats estimated depth”: the demonstrated conclusion is specific to calibrated warehouse multi-camera tracking under the tested RGB-only Sim2Real conditions and implementation budget.

Conclusion

The paper provides a technically focused comparison between explicit calibrated geometry and estimated-depth pseudo-LiDAR for RGB-only multi-camera 3D tracking. Its strongest empirical result is the disparity between 13.04 HOTA for geometry-first lifting and 0.12 HOTA for estimated-depth pseudo-LiDAR. The analysis attributes this gap to cross-view metric inconsistency in monocular depth, while identifying Sim2Real detection quality as the principal limitation of the calibrated route. Offline tracklet stitching improves association, but detector-side additions and learned geometric substitutes do not. Within the evaluated setting, preserving a shared calibrated world frame is more valuable than introducing dense but mutually inconsistent monocular depth.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. What is this paper about?

This paper studies how to find and follow objects in 3D using several cameras in a warehouse.

The objects might be people, forklifts, robots, and pallet trucks. The cameras see the warehouse from different angles, and the computer must figure out:

  • What objects are present?
  • Where are they in the real warehouse?
  • Which object in one frame is the same object seen earlier?
  • Which views from different cameras show the same object?

A major challenge is that the computer can use depth information while learning, but it cannot use depth when making final predictions. It must rely only on ordinary color images, called RGB images.

The paper compares two ways to create 3D information from RGB images:

  1. Geometry-first method: Uses camera positions and warehouse maps to place objects in the correct world location.
  2. Estimated-depth method: Tries to guess how far away each pixel is and turns those guesses into a 3D point cloud.

The main conclusion is that reliable shared geometry is much more useful than separately estimated depth from each camera.

2. What questions did the researchers ask?

The researchers focused on one main question:

Is it more important for all cameras to agree about the 3D location of objects, or is it more important for each camera to estimate depth accurately on its own?

They also wanted to discover:

  • Which parts of their tracking system work well?
  • What causes the biggest mistakes?
  • Can better object detectors improve the results?
  • Can appearance recognition help the system know when two detections belong to the same object?
  • Can estimated depth replace real depth?
  • Does training on synthetic images transfer well to real warehouse videos?

This last issue is called Sim2Real, short for “simulation to reality.” It means a computer learns from artificial, computer-generated scenes but must later work on real scenes.

3. How did they conduct the research?

The researchers built and tested two complete systems.

Geometry-first system

The first system worked like this:

  1. Detect objects in each camera image. They used a model called YOLO11x. It draws rectangles around objects and labels them, such as “person” or “forklift.”
  2. Convert image locations into warehouse locations. The system looked at the bottom-center of each rectangle, roughly where the object touches the floor. Using information about each camera’s position and angle, it projected that point onto the warehouse floor.

This process is called homography lifting. A simple analogy is placing a transparent grid over a photograph and using known landmarks to match the photograph to a real floor plan.

  1. Estimate 3D box sizes. The researchers used typical sizes for each object class. For example, a forklift was assigned an approximate length, width, and height.
  2. Combine views from different cameras. If two cameras appeared to see the same forklift at nearly the same world location, the system merged the detections into one observation.
  3. Track objects over time. The system connected observations from frame to frame. If an object disappeared briefly, it predicted where the object would probably move.
  4. Join broken tracks afterward. Sometimes the system gave the same object two different identities because it temporarily missed it. An offline process called tracklet stitching joined these broken pieces together.

Estimated-depth pseudo-LiDAR system

The second system followed a different plan:

  1. A depth-estimation model, such as D4RT or Metric3D v2, guessed how far away every part of the image was.
  2. These guesses were converted into a 3D collection of points, similar to data from a LiDAR laser scanner. This is called pseudo-LiDAR.
  3. The points from all cameras were combined.
  4. A 3D detector called V-DETR searched this point cloud for objects.

The researchers also tried fine-tuning the system so that it would better handle the difference between true depth and estimated depth.

How was success measured?

They used a score called 3D HOTA. This score combines three abilities:

  • Detection accuracy (DetA): Did the system find the right objects?
  • Association accuracy (AssA): Did it keep the correct identity for each object?
  • Localization accuracy (LocA): Did it place objects in the correct 3D locations?

A higher score means better overall tracking.

4. What were the main results?

The geometry-first method performed much better

The best geometry-first system achieved:

Method HOTA DetA AssA LocA
Geometry-first method 13.04 10.79 16.71 51.58
Estimated-depth pseudo-LiDAR 0.12 0.05 0.26 9.23

The difference was enormous. The geometry-first method scored about 100 times higher in HOTA than the estimated-depth method.

Why did estimated depth fail?

Each camera estimated depth separately. Even when the depth predictions looked reasonable in individual images, they did not agree when combined.

For example:

  • One camera might think the floor is at one height.
  • Another camera might place the floor slightly higher.
  • A third camera might place some floor points below the actual floor.

When the computer combined these inconsistent predictions, the 3D point cloud became warped and messy. Objects were then placed in the wrong locations.

The researchers found that correcting the overall depth scale was not enough. The different cameras still disagreed with one another.

This shows an important lesson:

A depth estimate can look acceptable in one image but still be unsuitable for combining many camera views.

The main weakness was object detection

The geometry-first system’s localization was fairly stable, but it did not always find the correct objects.

The detector had:

  • Overall precision of about 91%
  • Overall recall of about 60%

Precision asks, “When the computer says it found an object, how often is it correct?” Recall asks, “How many of the real objects did it find?”

The system often missed small, distant, hidden, or unusual objects. Pallet trucks were especially difficult, with recall of only about 24% on validation data.

Real warehouse scenes were also harder than synthetic scenes because of differences in lighting, camera quality, object appearance, and backgrounds.

Most attempted improvements did not help

The researchers tested many possible improvements:

  • Using another detector
  • Combining two detectors
  • Looking at image sections separately with SAHI
  • Using test-time image changes
  • Adding appearance-based re-identification
  • Replacing camera geometry with a learned model
  • Training with more visual variation
  • Using a lower detection threshold

Most of these changes made the final score worse. Adding more possible detections often added too many false alarms.

The one improvement that helped was offline tracklet stitching. It joined pieces of the same object’s track after the object had temporarily disappeared.

It improved:

  • Association score from 14.53 to 16.71
  • Overall HOTA from 12.49 to 13.04

However, it did not improve the actual object locations or detection quality. It only corrected object identities.

5. Why are these findings important?

The paper shows that, in this particular warehouse tracking problem, agreement between cameras is more important than detailed depth guesses from individual cameras.

The geometry-first method works because every camera uses the same calibrated warehouse coordinate system. This is like giving every camera the same map and asking them to place objects on that map.

The estimated-depth method fails because each camera creates its own slightly different version of reality. When those versions are combined, the result no longer lines up.

The research also identifies two different bottlenecks:

  • For the geometry-first system, the biggest problem is finding objects reliably.
  • For the pseudo-LiDAR system, the biggest problem is placing objects consistently in 3D.

In other words, the geometry-first system usually knows where its detected objects are, but it misses some objects. The depth-based system tries to create richer 3D information, but its 3D world is too inconsistent to be useful.

Simple conclusion and possible impact

The researchers recommend that future systems should:

  1. Keep using trusted camera calibration and explicit geometry.
  2. Focus on improving object detection, especially in real warehouse scenes.
  3. Avoid adding large numbers of uncertain detections.
  4. Be cautious about replacing camera geometry with monocular depth estimates.
  5. Use methods such as tracklet stitching to repair broken identities.

This could help warehouses build better systems for monitoring people, forklifts, and robots. More accurate tracking could improve safety, prevent collisions, and help manage warehouse operations.

The paper’s main message is simple:

When several cameras must work together, it is better for them to agree on a shared map than to make separate depth guesses that do not line up.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper leaves the following issues unresolved:

  • Limited evaluation coverage: The reported leaderboard scores are based on approximately half of the hidden test set, so the conclusions may not hold on the full test distribution.
  • Missing per-scene analysis: Per-scene HOTA, DetA, AssA, and LocA breakdowns are deferred, leaving it unclear whether the gains and failures are consistent across warehouses, camera layouts, and domain conditions.
  • Incomplete class coverage: The detector table reports only five classes, although the task contains seven; performance for FourierGR1T2 and AgilityDigit is not quantified.
  • Insufficient statistical validation: The study reports single aggregate results without confidence intervals, repeated evaluation under different training seeds, or significance testing for the relatively small gains from offline stitching.
  • Unclear causal isolation of geometry versus depth: The two routes differ in more than the source of geometry: they use different representations, detectors, training procedures, and post-processing. The comparison therefore does not fully isolate cross-view consistency from other pipeline differences.
  • Unmatched computational budgets: Domain-adaptation fine-tuning for pseudo-LiDAR is reported to fail “within our compute budget,” but the budget, training schedule, number of iterations, and resources are not specified or matched against the geometry-first pipeline.
  • Limited pseudo-LiDAR exploration: Only D4RT and Metric3D v2 are evaluated. It remains unknown whether newer, warehouse-specialized, multi-view, video-based, or explicitly scale-consistent depth models could substantially reduce the observed gap.
  • No multi-view depth-consistency correction: The pseudo-LiDAR route does not appear to apply joint optimization, camera-wise scale alignment, plane fitting, bundle adjustment, temporal filtering, or cross-view self-supervision. Whether such corrections can recover useful geometry remains open.
  • Unresolved role of the 3D detector: The conclusion attributes pseudo-LiDAR failure primarily to geometry, but the V-DETR detector may also be poorly calibrated for noisy estimated-depth clouds. Comparisons using multiple 3D detectors or detector-free geometric localization are needed.
  • Floor-coherence metric is not fully validated: The proposed floor-coherence diagnostic is evaluated on limited scenes and is treated as predictive of tracking performance, but its correlation with HOTA, LocA, and object-level 3D error is not quantitatively established across the dataset.
  • Strong assumptions about scene geometry: Homography lifting assumes a reliable, shared ground plane. The robustness of the method to ramps, uneven floors, cluttered surfaces, calibration errors, or objects not resting on the assumed plane is not investigated.
  • Uncertainty in camera calibration sensitivity: The paper does not measure how errors in intrinsics, extrinsics, synchronization, or homography estimation affect lift accuracy, fusion, and tracking performance.
  • Weak 3D box modeling: Object height, width, length, and vertical position are assigned using class-level priors, while yaw is estimated heuristically. The effect of instance-specific dimensions, orientation estimation, and articulated or irregular object shapes remains unexplored.
  • No detailed error decomposition for lifting: The reported median lift error of approximately $0.633$ m does not distinguish errors caused by inaccurate footpoints, calibration, occlusion, truncation, camera viewpoint, object class, or distance from the camera.
  • Unexplored detection improvements for the real domain: The paper identifies real-domain detection recall as the main bottleneck but does not evaluate targeted approaches such as real-image adaptation, synthetic-to-real style transfer, pseudo-labeling, class-balanced training, hard-negative mining, or domain-specific calibration.
  • Limited analysis of object scale and occlusion: The paper suggests that small, distant, and occluded objects drive errors, but it does not provide performance stratified by distance, image scale, visibility, occlusion level, or camera viewpoint.
  • Unclear generality of the detector conclusion: The claim that adding detections is harmful is based on specific thresholding, filtering, and fusion settings. Adaptive confidence calibration, class-specific thresholds, uncertainty-aware suppression, or recall-oriented track-level confirmation could produce different outcomes.
  • Restricted association modeling: Tracking uses nearest-neighbor world-coordinate matching with simple motion extrapolation. The potential benefits of probabilistic data association, learned motion models, global assignment, occlusion reasoning, or multi-camera temporal association are not tested.
  • Offline stitching limits practical deployment: The only successful association intervention uses future track evidence, making the best system offline. The paper does not determine whether comparable gains can be achieved under strict online or low-latency constraints.
  • Re-ID evaluation is underdeveloped: The failed Re-ID variant is described as “light,” but its embedding model, training data, crop quality, temporal aggregation, and fusion with geometric cues are not detailed. More robust cross-domain or trajectory-level appearance methods remain untested.
  • Temporal information is underused: Both detection and depth estimation are largely treated frame by frame. Temporal feature aggregation, video object detection, multi-frame depth consistency, and temporal ground-plane estimation could address missed detections and unstable geometry.
  • No calibration of class-specific fusion and tracking parameters: Spatial radii, motion gates, coasting duration, track-length filters, and stitching thresholds appear heuristic. Their sensitivity and transferability across scenes and classes are not reported.
  • Potential evaluation mismatch is not examined: The relationship between the alignment diagnostic, floor coherence, 3D IoU, LocA, and official HOTA is not fully characterized, leaving uncertainty about which intermediate metrics best predict leaderboard performance.
  • Limited comparison with stronger RGB-only baselines: The study does not compare against modern multi-view RGB methods that directly learn feature-level cross-camera geometry, occupancy, BEV representations, or calibrated multi-view transformers.
  • No investigation of hybrid geometry-depth systems: The paper compares explicit homography lifting with estimated-depth pseudo-LiDAR, but does not test hybrid approaches that use homography as a geometric constraint while using depth for object height, occlusion handling, foreground segmentation, or uncertainty estimation.
  • Unresolved transfer beyond the challenge dataset: The findings are demonstrated primarily for AI City Challenge 2026 warehouses. Whether “geometry beats estimated depth” holds in outdoor environments, non-planar industrial spaces, different camera calibrations, or other Sim2Real multi-camera benchmarks remains unknown.
  • Reproducibility details are incomplete: The paper refers to a complete reproducible pipeline, but key implementation details—including training splits, hyperparameters, augmentation policies, threshold values, model checkpoints, and exact post-processing settings—are not provided in the text.
  • No analysis of annotation and calibration quality: The study assumes that synthetic annotations, validation labels, camera metadata, and ground-plane definitions are reliable, but does not assess their noise or its effect on the reported lift and depth-consistency measurements.
  • Open question about the ultimate performance ceiling: Even with perfect 2D detection, the paper does not quantify the maximum achievable HOTA for the homography-plus-prior representation, so it remains unclear whether the proposed geometry formulation can support substantial improvement beyond the reported $13.04$ HOTA.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The paper’s strongest near-term contribution is a deployable RGB-only, calibration-based multi-camera tracking workflow for structured indoor environments. These applications rely on synchronized cameras, known camera calibration, a sufficiently visible ground plane, and object categories with reasonably stable physical dimensions.

  • Warehouse and logistics monitoring — industry
    • Deploy the geometry-first pipeline to track people, forklifts, pallet trucks, transporters, and mobile robots across multiple warehouse cameras.
    • A practical workflow would use YOLO11x or a comparable 2D detector, project detections into a shared world coordinate system using homographies, fuse duplicate observations, and maintain identities with world-coordinate motion tracking.
    • Potential products include warehouse dashboards showing object locations, movement paths, dwell time, congestion, and zone occupancy.
    • Dependencies: accurate camera calibration, synchronized video streams, a largely planar floor, stable camera placement, and detector performance on the target warehouse domain. The paper’s low recall for PalletTruck-like objects and real-scene imagery indicates that deployment should include class- and domain-specific validation.
  • Forklift–person proximity and safety alerts — industrial safety
    • World-coordinate tracks can support geofencing and proximity rules, such as alerting when a person enters a forklift operating zone or when a vehicle travels through a restricted area.
    • The system can generate real-time or near-real-time alerts from estimated positions rather than relying solely on image-space distances, which vary with camera perspective.
    • Dependencies: sufficiently accurate localization, conservative safety thresholds, camera coverage of relevant zones, and independent safety validation. With the reported localization and detection limitations, the system should initially be treated as an advisory or monitoring tool rather than a sole safety interlock.
  • Warehouse traffic and process analytics — logistics operations
    • Aggregate world-coordinate tracks to identify bottlenecks, recurring congestion, inefficient routes, excessive idle time, and underused storage or transit areas.
    • A possible workflow is: export tracked trajectories, map them to warehouse zones, compute dwell and transit statistics, and integrate the results with warehouse-management or fleet-management software.
    • Dependencies: consistent object identities, sufficient temporal coverage, and filtering of false positives and track fragments. Offline tracklet stitching is particularly useful for retrospective analytics because it improves identity continuity without changing object locations.
  • Multi-camera robot and fleet supervision — robotics
    • Mobile robots and automated vehicles can be monitored across camera blind spots using a shared world-frame representation.
    • The system could provide a lightweight external perception layer for fleet-management software, including robot occupancy maps, route compliance, and detection of stalled or misplaced vehicles.
    • Dependencies: known robot classes and size priors, adequate camera coverage, and reliable localization near occlusions. The method is more appropriate for structured, mostly planar facilities than for environments with ramps, shelves that obscure the ground plane, or frequent camera movement.
  • Retrospective incident investigation — industry and insurance
    • Store world-coordinate tracks and stitched identities for post-event reconstruction of collisions, near misses, unauthorized access, or inventory-handling incidents.
    • Compared with reviewing individual camera feeds, a common coordinate system can make it easier to reconstruct the sequence of movements across views.
    • Dependencies: synchronized timestamps, secure and tamper-evident video storage, calibrated cameras, and clear policies governing retention and access to worker-tracking data.
  • A reproducible baseline for multi-camera 3D perception — academia
    • Researchers can use the released pipeline as a reference implementation for RGB-only Sim2Real tracking, including detection, homography lifting, fusion, tracking, and offline stitching.
    • It provides a controlled baseline against which to evaluate new detectors, calibration methods, association algorithms, or domain-adaptation techniques.
    • Dependencies: access to compatible annotations and calibration metadata, reproducible preprocessing, and evaluation on scenes beyond the paper’s public leaderboard subset. The reported scores may not represent full hidden-test performance.
  • Annotation-free geometry quality monitoring — software and MLOps
    • The proposed floor-coherence metric can be integrated into data pipelines to detect whether reconstructed multi-camera geometry is globally consistent before running a 3D detector.
    • For example, a deployment pipeline could reject or flag cameras whose projected floor points are dispersed, below the floor, or inconsistent with the shared elevation.
    • This can reduce debugging time by distinguishing scale errors from cross-view alignment failures.
    • Dependencies: a known or estimable floor plane, sufficient floor visibility, valid calibration, and thresholds adapted to camera height, image resolution, and scene geometry.
  • Camera calibration and maintenance diagnostics — facilities and infrastructure
    • Repeated floor-coherence checks can identify camera movement, incorrect calibration files, synchronization problems, or changes in camera mounting.
    • A maintenance tool could periodically compare projected floor distributions across cameras and issue a recalibration ticket when coherence degrades.
    • Dependencies: stationary cameras and a floor that remains visible. Occlusion, reflective surfaces, uneven floors, or large changes in camera pose can produce false alarms.
  • Offline identity repair in video analytics — software
    • The tracklet-stitching method can be added to existing tracking systems when real-time identity continuity is imperfect but retrospective processing is acceptable.
    • It is particularly suitable for workforce-flow analysis, security review, and operational reporting, where future observations can be used to reconnect short track fragments.
    • Dependencies: a bounded gap between fragments, reliable motion extrapolation, class-aware matching, and acceptance of offline rather than strictly causal results. It should not be used for applications requiring immediate identity decisions.
  • Operational guidance for system design — policy and procurement
    • Organizations procuring multi-camera perception systems can use the paper’s findings to prioritize:
    • calibrated cameras and shared world geometry;
    • real-domain detection data;
    • per-class recall testing;
    • precision–recall analysis rather than simply increasing detection volume; and
    • validation of cross-view consistency before adopting monocular-depth-based 3D solutions.
    • The results caution against assuming that an apparently detailed pseudo-LiDAR point cloud will outperform a simpler geometric lift.
    • Dependencies: procurement teams must verify the findings in their own environments, since the experiments focus on warehouse scenes and a specific challenge configuration.

Long-Term Applications

The longer-term opportunities involve improving robustness beyond the paper’s validated setting, extending the method to less structured environments, or developing depth and geometry models that preserve cross-camera metric consistency.

  • High-reliability autonomous warehouse navigation — robotics
    • A stronger version of the geometry-first system could provide external multi-camera perception for autonomous forklifts, delivery robots, and humanoid or transport robots.
    • World-frame tracks could feed collision prediction, shared occupancy maps, route planning, and cooperative multi-robot scheduling.
    • Dependencies: substantially higher detection recall and lower false-positive rates than those reported; low-latency processing; robust handling of occlusion; uncertainty estimates; and formal safety certification. The current system is not sufficient as the sole perception source for autonomous control.
  • Cross-camera metric depth reconstruction — computer vision and robotics
    • The paper’s negative result suggests that future monocular-depth systems should be trained or optimized for cross-view consistency, not only per-image depth accuracy.
    • Possible research directions include jointly estimating depth across synchronized cameras, enforcing a common floor-plane constraint, optimizing inter-camera reprojection error, and learning scene-level scale and pose corrections.
    • A future product could produce a metrically coherent fused point cloud from RGB cameras for 3D detection and mapping.
    • Dependencies: synchronized multi-camera training data, reliable calibration, computational resources, and methods capable of handling dynamic objects without distorting the shared scene geometry.
  • Hybrid geometry–learning perception systems — industry and academia
    • Learned models could improve detection confidence, object dimensions, yaw, or uncertainty while retaining homography-based world positioning as a hard geometric constraint.
    • For example, a detector could predict whether a footpoint is reliable, estimate class-specific height variation, or select among multiple geometric hypotheses without replacing calibrated lifting.
    • Dependencies: sufficiently representative real-world data and careful constraints preventing the learned component from introducing cross-view inconsistency. The paper’s failed learned MLP lift indicates that unconstrained end-to-end replacement of calibration is risky.
  • Adaptive Sim2Real detection for warehouses — AI and software
    • Since detection quality, particularly on real scenes and small or occluded classes, is the main bottleneck, future systems could use continual adaptation based on real warehouse imagery.
    • Potential workflows include active learning from false positives and false negatives, human review of uncertain detections, synthetic data generation targeted at difficult classes, and class-specific detector retraining.
    • The system could prioritize PalletTruck, Forklift, and distant or partially occluded objects rather than applying generic augmentation.
    • Dependencies: access to real operational data, privacy-preserving annotation processes, stable class definitions, prevention of model drift, and evaluation across lighting, layouts, and camera configurations.
  • Multi-level or nonplanar facility tracking — construction, manufacturing, and ports
    • Extending the approach from a single homography and ground plane to piecewise planar maps, elevation-aware maps, ramps, mezzanines, and three-dimensional facility models could support factories, distribution centers, parking structures, and port terminals.
    • A possible implementation would associate detections with a digital map containing multiple surfaces and use surface-aware projection rather than assuming all objects rest on one plane.
    • Dependencies: detailed facility maps, reliable elevation estimation, visibility reasoning, and handling of objects on shelves or platforms. The current method’s assumptions about a common ground plane would otherwise produce systematic localization errors.
  • Digital twins and facility-scale simulation — industry and urban systems
    • Persistent world-coordinate tracks could populate a digital twin of a warehouse or factory, enabling simulation of traffic flows, staffing, robot deployment, and safety interventions.
    • The same representation could be used to compare observed behavior with simulated schedules or to evaluate layout changes before physical implementation.
    • Dependencies: long-term calibration stability, consistent identity management, integration with operational databases, and methods for quantifying uncertainty and correcting accumulated tracking errors.
  • Privacy-preserving workplace analytics — policy and industry
    • Because the pipeline can operate on object detections and world coordinates, future systems could discard or restrict access to raw video after extracting operational statistics.
    • This could support occupancy, safety, and traffic monitoring while reducing the amount of identifiable imagery retained.
    • Dependencies: robust anonymization, governance over worker identification, legal compliance, explicit retention policies, and safeguards against using tracking outputs for unjustified individual surveillance. World-coordinate IDs can still constitute sensitive personal data when they correspond to workers.
  • Standards for cross-view consistency in 3D perception — policy and research
    • The floor-coherence diagnostic could motivate benchmark protocols that report not only detection and tracking accuracy but also calibration quality, inter-camera consistency, below-ground mass, plane-fit residuals, and reprojection stability.
    • Such standards would help organizations compare RGB-only 3D systems without relying exclusively on end-task scores.
    • Dependencies: agreement on thresholds and test scenes, extension beyond warehouse floors, and validation that the proposed proxy correlates with performance across datasets and camera arrangements.
  • Failure-aware deployment and automated fallback — safety-critical software
    • A production system could monitor detector recall proxies, floor coherence, camera health, and track confidence, then switch to degraded modes when geometry or detection quality becomes unreliable.
    • Examples include notifying operators, restricting robot speed, disabling automated route changes, or falling back to single-camera 2D alerts.
    • Dependencies: calibrated uncertainty estimates, validated safety policies, low false-alarm rates, and integration with facility-control systems. The paper demonstrates useful diagnostics but does not yet establish operational thresholds for safety decisions.
  • Generalization to transport hubs and public spaces — transportation and security
    • The shared-world tracking concept could eventually be applied to airports, train stations, parking facilities, and large public venues for crowd-flow analysis, service optimization, and incident response.
    • Dependencies: substantially more varied camera viewpoints, nonplanar environments, privacy protections, crowd-specific association methods, and rigorous testing under severe occlusion. The warehouse-specific class priors and floor assumptions would need to be replaced with broader scene models.
View Paper Prompt  View All Prompts 

Glossary

  • 3D HOTA: A higher-order metric for multi-object tracking that jointly evaluates detection, association, and localization in three dimensions. “The official metric is 3D HOTA”
  • Association accuracy (AssA): A tracking metric measuring how consistently predicted identities correspond to ground-truth identities over time. “We report the leaderboard components HOTA, DetA, AssA, and LocA.”
  • Back-projection: The process of converting image pixels and depth estimates into three-dimensional points using camera calibration. “Per-camera depth is predicted from RGB with a monocular model---D4RT~\cite{zhang2025d4rt} or Metric3D~v2~\cite{yin2023metric3d}---back-projected using calibration”
  • BEV (bird’s-eye view): A top-down representation of a scene in world or ground-plane coordinates. “Figure~\ref{fig:bev} makes this concrete on a validation frame”
  • Camera calibration: Parameters describing a camera’s geometry and its relationship to a shared spatial coordinate system. “Each 2D detection is lifted into the world coordinate system using the camera calibration”
  • Class-level priors: Predefined assumptions about typical object properties, such as dimensions, for each object category. “The vertical coordinate and 3D box dimensions are assigned using class-level priors”
  • Coasting: Maintaining a track during frames in which an object is temporarily undetected by extrapolating its previous motion. “We include optional coasting to bridge short gaps”
  • Cross-view geometric consistency: Agreement between three-dimensional geometry reconstructed independently from multiple camera viewpoints. “Our hypothesis is that cross-view geometric consistency, not monocular depth accuracy, is the dominant factor”
  • DetA (detection accuracy): The HOTA component measuring the quality of detected objects, including missed detections and false positives. “This recall weakness directly limits DetA and final HOTA.”
  • Domain adaptation: Techniques for reducing performance differences between a model’s training domain and its deployment domain. “we additionally fine-tune it on estimated-depth clouds to reduce the train/test depth-domain gap”
  • Domain randomization: Training with randomized simulated appearances or conditions to improve transfer from synthetic data to real environments. “YOLO26 trained on a domain-randomized~\cite{tobin2017domainrand} stress-augmented set”
  • Domain shift: A difference between the data distribution used for training and that encountered during evaluation or deployment. “We attribute this to domain shift, small or distant objects, real camera artifacts”
  • Estimated-depth pseudo-LiDAR: A method that turns monocular depth predictions into a point cloud so that LiDAR-oriented 3D detectors can process RGB imagery. “The second is estimated-depth pseudo-LiDAR”
  • False positive: A predicted object that does not correspond to a real ground-truth object. “false positives (74) exceeding true positives (46)”
  • Floor coherence: A measure of how closely reconstructed scene points cluster around the known ground plane. “As a proxy we measure floor coherence on a validation scene”
  • Footpoint projection: Projection of an object’s bottom-center image location onto the ground or world plane. “using the camera calibration and a homography-based footpoint projection”
  • Frame fusion: Combining detections from multiple synchronized cameras for the same time step. “detections are first fused per frame and class in world space”
  • Ground-plane lifting: Mapping two-dimensional image detections onto a shared horizontal plane in three-dimensional space. “homography ground-plane lifting with class-size priors”
  • HOTA: A tracking metric that balances detection, identity association, and localization performance. “stitching raises AssA from 14.53 to 16.71 (and HOTA from 12.49 to 13.04)”
  • Homography: A projective transformation that maps points between two planes, such as an image plane and a ground plane. “Each 2D detection is lifted into the world coordinate system using the camera calibration and a homography-based footpoint projection.”
  • Identity association: The process of determining which detections belong to the same physical object across frames or camera views. “The tracker associates fused detections to active tracks”
  • Inter-view reprojection error: The discrepancy produced when a reconstructed point is projected between camera views and fails to align with its corresponding observation. “Plane-fit residuals or inter-view reprojection error are alternatives”
  • LocA (localization accuracy): The HOTA component measuring the spatial accuracy of predicted object locations or bounding boxes. “The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA).”
  • Metric depth: Depth expressed in a physically meaningful scale, such as meters, rather than an arbitrary relative scale. “Metric3D~v2 is metric by construction”
  • Metric grounding: The property of a reconstruction being tied to a consistent real-world spatial scale and coordinate system. “a learned substitute that lacks metric grounding and cross-view agreement”
  • Monocular depth estimation: Estimation of scene depth from a single camera image. “Monocular depth estimation has advanced rapidly”
  • Multi-camera fusion: Integration of observations from several cameras into a common spatial representation. “multi-camera fusion into a scene point cloud”
  • Offline tracklet stitching: Post-processing that joins fragmented trajectory segments using information from the complete sequence, including future observations. “We add an offline stitching stage that links same-class fragments”
  • Pseudo-LiDAR: A representation that converts image-derived depth into a point cloud resembling LiDAR data for three-dimensional detection. “Pseudo-LiDAR~\cite{wang2019pseudolidar} back-projects estimated depth into a point cloud”
  • Re-identification (Re-ID): Recognition of an object as the same identity after it disappears or is observed from another view. “Appearance Re-ID~\cite{ye2021reid} from object crops did not improve association”
  • Scene-level detection: Detection performed in a shared representation of the entire environment rather than independently in individual image views. “the availability and quality of scene-level detections”
  • Scale correction: Rescaling estimated depth so that it matches a physically meaningful metric scale. “D4RT required a near-constant ∼\sim4.3×\times correction to reach metric scale”
  • Sim2Real: Transfer of a model or system from synthetic simulation data to real-world data. “The challenge setting is Sim2Real.”
  • Sliced inference: Running an object detector on overlapping image tiles to improve detection of small or distant objects. “Sliced inference (SAHI), which tiles each frame to recover small and distant objects”
  • Synthetic-to-real gap: The performance discrepancy caused by differences between simulated training imagery and real-world evaluation imagery. “The test set mixes synthetic scenes (Warehouses~23--25) and real-world scenes (Warehouses~26--27).”
  • Test-time augmentation (TTA): Applying transformations to inputs during inference and combining the resulting predictions to improve robustness. “YOLO11x TTA”
  • Track fragmentation: The splitting of one object’s continuous trajectory into multiple track segments because of missed detections or association failures. “The online tracker fragments identities whenever an object is missed”
  • Tracklet: A short sequence of detections believed to belong to the same object. “offline tracklet stitching”
  • Transformer detector: An object detector based on transformer architectures that model relationships among image features using attention mechanisms. “compare against transformer detectors”
  • Union-find: A data structure for efficiently maintaining and merging groups of connected elements. “chained via union-find”
  • V-DETR: A transformer-based three-dimensional object detector designed for point-cloud data. “A V-DETR 3D detector~\cite{shen2024vdetr} trained on the provided-depth clouds”
  • World-coordinate tracking: Tracking objects using positions represented in a shared physical coordinate system rather than image coordinates. “We include optional coasting to bridge short gaps---when a track has no detection in a frame, its position is extrapolated”
  • World frame: A common coordinate system used to express the positions of objects observed by multiple cameras. “The projected point defines the object position in the world plane.”
  • Yaw: An object’s rotation angle around the vertical axis. “Yaw is estimated heuristically.”
  • YOLO: A family of real-time neural-network object detectors that predict object locations and classes directly from images. “We trained an Ultralytics YOLO11x detector at 1280 input resolution.”
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.