3D Pseudo Box Generation

• 3D pseudo box generation is the process of automatically constructing surrogate 3D bounding boxes when dense human annotations are scarce or noisy.
• It employs varied methodologies including direct teacher–student self-training, monocular depth-ambiguity techniques, and multimodal proposal fusion to generate reliable object proposals.
• Empirical results show that refined pseudo boxes significantly boost detection AP and enable effective cross-dataset and open-vocabulary adaptation in 3D object detection.

Searching arXiv for recent and foundational papers on 3D pseudo box generation and closely related pseudo-annotation methods. 3D pseudo box generation denotes the automatic construction of surrogate 3D bounding boxes, or of intermediate pseudo annotations that can be converted into boxes, when dense human 3D annotation is unavailable, scarce, expensive, or noisy. In the cited literature, the problem appears in LiDAR self-training for 3D object detection, unsupervised and open-vocabulary 3D detection, cross-dataset unsupervised domain adaptation, and monocular depth-ambiguity mitigation. Closely related box-supervised 3D instance segmentation methods generate pseudo masks or pseudo point labels rather than pseudo boxes, but remain relevant because those masks can later be converted into axis-aligned or fitted 3D boxes (Caine et al., 2021, Ji et al., 28 Aug 2025, Zhang et al., 2024, Huang et al., 2022, Liu et al., 12 Aug 2025, Ngo et al., 2023).

1. Scope and problem definition

Within the literature, “3D pseudo box generation” is not a single algorithmic template. Direct methods output pseudo 3D boxes for detector training. Indirect methods start from 3D boxes and generate pseudo masks, pseudo instance labels, or pseudo point labels; these are not themselves pseudo-box generators, but they supply object support estimates that can later be converted into boxes. A third line treats boxes as structured abstractions for scene layout or generative shape scaffolds rather than as detector labels (Ngo et al., 2023, Liu et al., 2022, Lu et al., 2024, Yoo et al., 14 Oct 2025, Deng et al., 22 May 2025, Li et al., 2020, Koo et al., 24 Feb 2026).

Formulation	Direct output	Representative papers
Detection self-training, UDA, open-vocabulary, monocular ambiguity mitigation	Pseudo 3D boxes	(Caine et al., 2021, Zhang et al., 2024, Ji et al., 28 Aug 2025, Huang et al., 2022, Liu et al., 12 Aug 2025)
Box-supervised 3D instance or semantic segmentation	Pseudo masks or pseudo point labels	(Ngo et al., 2023, Liu et al., 2022, Lu et al., 2024, Yoo et al., 14 Oct 2025, Deng et al., 22 May 2025)
Structured scene or generative abstraction	Scene cuboids or part boxes	(Li et al., 2020, Koo et al., 24 Feb 2026)

A recurring misconception is that every method using boxes under weak supervision is a pseudo-box method. GaPro generates pseudo instance masks and uncertainty from axis-aligned 3D boxes, not pseudo boxes (Ngo et al., 2023). Box2Seg generates pseudo point-wise semantic labels from 3D boxes and subcloud-level class tags, not detection boxes (Liu et al., 2022). BSNet and BEEP3D likewise target overlap disambiguation for box-supervised 3D instance segmentation through pseudo masks rather than direct box regression (Lu et al., 2024, Yoo et al., 14 Oct 2025). Sketchy-3DIS begins from inaccurate “sketchy bounding boxes” and converts them into pseudo point-level instance supervision (Deng et al., 22 May 2025). By contrast, BoxSplitGen explicitly generates oriented 3D part bounding boxes with varying granularity, and Box Program Induction reconstructs a scene-level box-like layout from a single image under a strong box prior (Koo et al., 24 Feb 2026, Li et al., 2020).

2. Canonical generation pipelines

A canonical direct pipeline is LiDAR teacher–student self-training. In “Pseudo-labeling for Scalable 3D Object Detection,” pseudo-label training is explicitly described as three stages: “Train a teacher detector on human-labeled 3D boxes,” “Run the teacher on unlabeled point clouds to generate pseudo 3D boxes,” and “Train a student detector on the union of labeled and pseudo-labeled data” (Caine et al., 2021). The teacher is selected by validation performance and can be a standard PointPillars model, a wider PointPillars model, or a multi-frame PointPillars teacher using concatenated point clouds from current and previous frames transformed into the current frame coordinate system. The student is often a smaller, cheaper model, so the pipeline also functions as label-space distillation.

A distinct formulation appears in monocular 3D detection. OBMO does not mine unlabeled scenes; instead, it augments each ground-truth object with several frustum-consistent pseudo 3D boxes by shifting the original 3D center along the viewing frustum while preserving dimensions and yaw (Huang et al., 2022). For an object parameterized as $(\text{class}, X, Y, Z, H, W, L, \text{yaw})$, the default depth offsets are

$\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$

with

$$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$

The resulting pseudo box keeps $(H,W,L,\text{yaw})$ unchanged and is scored either by projected 2D IoU or by a linear displacement-based score. This converts hard one-to-one depth supervision into ambiguity-aware multi-target supervision.

Multimodal unsupervised and open-vocabulary methods shift the emphasis from detector self-training to proposal construction. DFU3D first performs bi-directional data-level fusion: real LiDAR points are projected to image space to inherit class labels and instance IDs, while segmented foreground pixels are back-projected with estimated depth to create dense pseudo 3D points (Ji et al., 28 Aug 2025). HQ-OV3D starts from VLM 2D detections and SAM masks, projects LiDAR points into each camera, clusters the points that fall inside each foreground mask, cross-validates clusters by projected IoU against the 2D detection, and then refines proposals with a diffusion-style denoiser trained on annotated base classes (Liu et al., 12 Aug 2025). PERE occupies another position in the design space: it is an iterative self-training framework for cross-dataset 3D UDA in which a two-stage voxel-based detector generates target-domain pseudo boxes, then refines how those boxes are accepted or edited during training (Zhang et al., 2024).

3. Refinement, denoising, and confidence control

The simplest refinement strategy is confidence thresholding. In the LiDAR teacher–student pipeline, pseudo-labeled boxes are retained only if their classification score exceeds a threshold selected on a validation set; “0.5 works well for most models,” while some “multi-frame pedestrian teachers” benefit from “0.3” because they are “poorly calibrated and systematically under-confident” (Caine et al., 2021). The same paper is explicit that it does not use teacher ensembling, box voting, temporal aggregation of pseudo boxes over trajectories, tracking-based smoothing, uncertainty-based rejection, or a separate distillation loss. It also reports that “soft labels” were worse than hard labels, “ambiguous score ranges with zero loss” gave no gain, and multiple relabeling rounds produced only small gains.

DFU3D introduces a much richer refinement stack before box fitting. After fusing real LiDAR foreground points $R$ and image-derived pseudo points $V$, it applies local radius filtering:

$$F = \left \{ v_j \mid E(r_i,v_j) \leq \lambda \cdot E(r_i, o) \right \} \cup R,$$

with $\lambda = 0.01$, followed by global statistical filtering to remove segmentation-induced outliers (Ji et al., 28 Aug 2025). It then refines pseudo boxes through a data-level fusion based dynamic self-evolution strategy. Updates are triggered when loss stabilization satisfies

$|t_{e} - t_{e-1}| \leq \psi pe^{-p},$

with $\psi = 0.1$, and box replacement uses a 3D IoU threshold $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$0. The reported ablations show that filtering is not peripheral: “fusion without filtering” gives 9.6 mAP, while “fusion + local filtering + global filtering” gives 24.5 mAP.

PERE turns pseudo-label refinement into explicit intervention on point clouds rather than reweighting noisy boxes. For a pseudo box $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$1 with predicted 3D IoU confidence score $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$2, thresholds $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$3 define low-confidence, unreliable, and high-confidence regimes (Zhang et al., 2024). Unreliable boxes are never used directly. Instead, Complementary Augmentation chooses between PointRemove and BoxReplace with

$\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$4

High-confidence boxes are cached in a database, pseudo-label updates occur every $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$5 epochs, and additional proposal generation plus cross-domain RoI feature alignment improve the proposals and RoI features that later produce refreshed pseudo boxes.

HQ-OV3D adds a learned box denoiser after heuristic proposal generation. The ACA Denoiser treats the initial proposal as a noisy version of a latent clean box and performs DDIM-style refinement conditioned on local BEV features, box embeddings, timestep embeddings, and a super-category embedding derived from geometric similarity (Liu et al., 12 Aug 2025). Residuals are predicted for center, size, and yaw, and an IoU-aware confidence branch trained with Varifocal Loss is fused with the VLM semantic score; the best ablation uses IoU confidence weight 0.6 and VLM confidence weight 0.4.

4. Empirical behavior and scalability

The strongest recurring empirical finding is that pseudo-box quality tracks teacher or proposal quality. In LiDAR self-training, “better teacher AP leads to better student AP,” and pseudo-label-trained student models “can outperform supervised models trained on 3-10 times the amount of labeled examples” (Caine et al., 2021). With 10% labeled data, Vehicles improve from 49.1 AP to 58.9 AP on Waymo and from 26.1 AP to 37.2 AP on Kirkland when pseudo labels are generated by a 4× width, 4-frame PointPillars teacher for a 1× width, 1-frame PointPillars student. For Pedestrians in the same setting, the corresponding change is 53.4 / 14.5 to 64.6 / 27.1. In the large-scale setting with about 75k unlabeled run segments, the paper reports that explicit ratio control becomes necessary because unlabeled data can overwhelm labeled data; “generally 1:5 labeled:pseudo-labeled ratio worked best,” except for one pedestrian setup where “1:1 worked best.”

In unsupervised 3D detection, pseudo-box quality is validated mostly through downstream detector performance and ablations. DFU3D reports 28.4 mAP on the nuScenes validation benchmark for the detector trained by its pseudo boxes, and the bi-directional fusion ablation shows “Real points only” at 8.5, “Pseudo points only” at 7.5, and “Real + pseudo fusion” at 9.6, before filtering is added (Ji et al., 28 Aug 2025). The dynamic self-evolution study further shows 26.2 for “pseudo-point densification, fixed-1” and 28.4 for “pseudo-point densification, dynamic-1.”

In monocular 3D detection, OBMO reports consistent improvements across detectors and datasets, with KITTI validation gains summarized as $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$6 mAP in BEV and $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$7 mAP in 3D (Huang et al., 2022). On Waymo validation with GUPNet, overall LEVEL 1 3D mAP/mAPH at IoU=0.5 improves from 17.52/17.37 to 20.71/20.53. The paper attributes the effect to more stable depth learning under ambiguity-aware supervision.

For open-vocabulary pseudo labels, HQ-OV3D directly evaluates pseudo-box quality by training a downstream TransFusion detector with the generated pseudo labels. On novel classes, the reported mAP rises from 33.65 for Find n’ Propagate to 41.02 for HQ-OV3D (GLIP), and the paper summarizes this as a 7.37% improvement in mAP on novel classes (Liu et al., 12 Aug 2025). PERE similarly presents strong indirect evidence that pseudo-box refinement, rather than simple thresholding, is the main performance driver: on NuScenes $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$8 KITTI with PV-RCNN, Car $\Delta_z=\{-8\%, -4\%, +4\%, +8\%\},$9 rises from 57.79 for baseline self-training to 68.34 with the full PERE configuration (Zhang et al., 2024).

5. Box-to-mask intermediates and alternative box abstractions

Several influential methods are central to the topic precisely because they do not directly generate pseudo boxes. GaPro formulates box-supervised 3D point cloud instance segmentation as a two-step pipeline: generate pseudo instance masks and uncertainty from axis-aligned 3D boxes using Gaussian Processes, then train a standard 3D instance segmentation network on those pseudo labels (Ngo et al., 2023). The method partitions points into determined and undetermined regions, resolves overlap by pairwise GP classification, and produces a pseudo mask $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$0 together with mean and variance maps. It later derives axis-aligned 3D boxes from predicted masks by taking coordinate-wise minima and maxima, yielding Box AP$$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$1 67.0 for GaPro + ISBNet on ScanNetV2 validation.

Box2Seg is also not a pseudo-box method in the detection sense. It uses 3D boxes plus subcloud-level class tags to generate pseudo point-wise semantic labels through Attention-based Self-Training and Point Class Activation Mapping (Liu et al., 2022). Its relevance lies in inside-box foreground extraction, overlap disambiguation with a confidence threshold of 0.8 for overlap points, and confidence-based refinement that keeps the top 20% highest-confidence background pseudo labels. These mechanisms can tighten support regions before any downstream box fitting, but the paper does not perform box regression or box fitting itself.

BSNet and BEEP3D treat overlap ambiguity as the principal obstacle in box-supervised 3D instance segmentation. BSNet decomposes a two-object overlap into $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$2, $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$3, and $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$4, pretrains on simulated overlaps, then uses a confidence-filtered Mean Teacher to generate pseudo masks in the overlap region with threshold $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$5; overlap pseudo-label mAcc increases from 52.5 for the base neural pseudo-labeler to 59.6 with Mean Teacher plus simulated sample generation (Lu et al., 2024). BEEP3D uses an EMA teacher as an online pseudo-labeler, instance center-based query refinement, and two consistency losses; on ScanNetV2 validation it reports 57.3 AP, and the overlap pseudo-mask mean accuracy rises from 73.4 to 79.9 when instance center refinement is used (Yoo et al., 14 Oct 2025).

Sketchy-3DIS addresses inaccurate boxes directly. It perturbs ground-truth boxes with scaling, translation, and rotation to create “sketchy bounding boxes,” then learns an adaptive box-to-point pseudo labeler and a coarse-to-fine instance segmentator (Deng et al., 22 May 2025). Its core pseudo supervision is not a new box set but compact pseudo instance labels produced by filtering background points inside a noisy box and resolving overlap points with a point-to-box assignment model. Beyond detection and segmentation, BoxSplitGen treats boxes as generative part abstractions: each box is an oriented 15D part box $$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$6, and the model learns a coarse-to-fine splitting process

$$Z^{(k)} = Z + \Delta_z^{(k)} Z,\qquad \frac{X^{(k)}}{Z^{(k)}} = \frac{X}{Z},\qquad \frac{Y^{(k)}}{Z^{(k)}} = \frac{Y}{Z}.$$7

to generate box sets of varying granularity (Koo et al., 24 Feb 2026). Box Program Induction goes in yet another direction by recovering a scene-level box-like layout from a single image under an inner-view or outer-view box prior, using vanishing points, wireframes, and orthogonality constraints rather than detector-style object boxes (Li et al., 2020).

6. Limitations, misconceptions, and research trajectory

One consistent limitation is that pseudo-box performance is highly sensitive to calibration, geometry quality, and the relative amount of unlabeled data. LiDAR self-training explicitly notes that some teachers are under-confident and require lower thresholds, that pseudo-label gains shrink when unlabeled data is small relative to labeled data, and that too much unlabeled data can overwhelm labeled data (Caine et al., 2021). OBMO still struggles with “occluded objects” and “truncated objects,” because these cases weaken the image evidence needed to define a reasonable frustum-consistent depth range (Huang et al., 2022). HQ-OV3D states that 2D VLM detection remains a bottleneck; replacing VLM detections with GT 2D boxes yields up to 80% performance improvement, indicating that high-quality 3D pseudo boxes can still be limited by the 2D stage (Liu et al., 12 Aug 2025).

Another recurrent issue is structural simplification. PERE assumes access to a sufficient database of high-confidence target-domain boxes for BoxReplace and is designed around two-stage IoU-aware detectors (Zhang et al., 2024). BSNet is organized around pairwise overlaps with two foreground queries, and GaPro resolves multi-box overlap by keeping only the pair with the largest overlap in the minority of cases involving three or four boxes (Lu et al., 2024, Ngo et al., 2023). Sketchy-3DIS reports that performance degrades severely when sketchy boxes become immensely inaccurate (Deng et al., 22 May 2025). BoxSplitGen depends on SMART hierarchies and trains separate box-splitting models per class, while Box Program Induction assumes an image that is either an inner view or an outer view of a box in 3D (Koo et al., 24 Feb 2026, Li et al., 2020).

A final misconception is that progress necessarily requires increasingly elaborate pseudo-label denoisers. The LiDAR self-training results show the opposite for one important regime: the best pseudo-box pipeline is “intentionally simple,” relying mainly on teacher quality and score filtering, while soft labels, ambiguous score bands, and repeated relabeling did not materially improve performance (Caine et al., 2021). At the same time, multimodal and open-vocabulary settings demonstrate that when the initial geometry is weak, stronger refinement can matter substantially: DFU3D derives large gains from filtering and dense self-evolution, PERE from complementary augmentation, and HQ-OV3D from diffusion-style denoising (Ji et al., 28 Aug 2025, Zhang et al., 2024, Liu et al., 12 Aug 2025). This suggests that the field has bifurcated into two robust trajectories: direct pseudo-box generation from detector outputs, multimodal proposals, or denoisers, and box-to-mask intermediate pipelines in which ambiguous ownership is resolved first and boxes are fitted afterward.