Pseudo-Boxes: Surrogate Representations

• Pseudo-boxes are surrogate representations used in computer vision and related fields to provide approximate localization when exact annotations or unique geometries are unavailable.
• In vision, they serve as teacher-generated 2D/3D bounding boxes refined through multi-round corrections, enhancing weakly or unsupervised detection performance.
• Extensions include point-induced pseudo-boxes for tracking, non-unique realizations in combinatorial geometry, and formal constructs in non-signalling theory and cryptography.

Searching arXiv for recent and relevant papers on “pseudo boxes” and closely related usages across detection, tracking, 3D perception, geometry, and non-signalling boxes. Pseudo-boxes are box-like surrogate representations used when exact box annotations, exact geometry, or unique box identity are unavailable, unreliable, or intentionally relaxed. In contemporary arXiv literature, the expression appears most prominently in computer vision, where pseudo-boxes are teacher-generated 2D or 3D bounding boxes on unlabeled data and function as supervisory signals in semi-supervised, weakly supervised, unsupervised, and domain-adaptive detection. Related work in tracking replaces a direct rectangular representation by a learned point set that induces a pseudo bounding box. In combinatorial geometry, an analogous phenomenon appears when one planar shape folds into multiple non-isomorphic boxes. Outside geometric perception, “box” also denotes non-signalling conditional distributions and cryptographic S-boxes, producing a separate but terminologically adjacent lineage (He et al., 2023, Zhang et al., 2024, Zeng et al., 2023, Qian et al., 1 Jun 2025, Joshi et al., 2011, Dumas et al., 2014).

1. Scope and principal meanings

Across the cited literature, pseudo-boxes are not a single standardized object but a family of box-like substitutes. In 2D and 3D perception, they are predicted bounding boxes used as pseudo supervision; in tracking, they are boxes recovered from learned sample points; in combinatorial folding, they are planar structures compatible with multiple box realizations; and in non-signalling theory, a “box” is a probability distribution rather than a geometric region.

Research area	Box-like object	Representative role
SSOD / weak supervision	Predicted 2D bounding box	Supervision on unlabeled or weakly labeled images
Unsupervised / UDA 3D detection	Pseudo 3D bounding box	Self-training target under noisy LiDAR geometry
Single-object tracking	Point-induced pseudo bounding box	Internal object representation
Combinatorial geometry	Common unfolding of multiple boxes	Non-unique box realization
Non-signalling theory	Conditional probability “box”	Resource-theoretic object
Cryptography	S-box from pseudo-random or pseudo-extension design	Nonlinear substitution component

The dominant computer-vision meaning is especially precise. PCL distinguishes pseudo-labels, meaning the full teacher-predicted supervisory signal on unlabeled data, from pseudo boxes, meaning specifically the localization component of those pseudo-labels (He et al., 2023). This distinction matters because classification noise and localization noise behave differently. The perception literature therefore treats pseudo-boxes not merely as weak labels, but as geometric hypotheses whose errors can corrupt assignment, regression targets, and later rounds of self-training.

2. Pseudo-boxes in 2D detection

In semi-supervised object detection, the canonical setting uses a labeled set $D_s=\{x_i^s,y_i^s\}_{i=1}^{N_s}$, an unlabeled set $D_u=\{x_i^u\}_{i=1}^{N_u}$, and a mean-teacher pipeline in which the teacher produces pseudo-labels for weakly augmented unlabeled images and the student is trained on both labeled and pseudo-labeled data. PCL writes the overall loss as

$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$

and argues that localization noise arises both in pseudo-box generation and in later learning, where noisy boxes alter proposal assignment and regression targets (He et al., 2023).

The core empirical objection to naive pseudo-box use is that class confidence is an imperfect proxy for localization quality. PseCo reports that pseudo-box precision is about 81% at IoU = 0.3, but only 31% at IoU = 0.9, and explicitly states that classification confidence and localization precision are not strongly correlated (Li et al., 2022). This makes confidence thresholding alone insufficient: a pseudo box can have the right category and a high score while still being spatially coarse.

Several methods therefore modify either the pseudo boxes themselves or the way they are consumed. PCL introduces pseudo-label correction through multi-round refining and multi-vote weighting. Refinement iteratively feeds initial pseudo boxes $B^0$ back into the teacher RCNN head,

$B^r, S^r = \mathcal{M}(F, B^{r-1}),$

and then locally perturbs each refined box with jittered boxes that are rescored and fused by

$$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$

PCL further proposes an unsupervised regression weight

$$w_i^{reg} = sigmoid\!\left(\frac{1}{V_{IoU}^{\lambda}}\right),$$

which is negatively correlated with teacher-student IoU. On COCO-standard, it reports 24.36, 33.28, and 36.47 mAP under 1%, 5%, and 10% labeled splits, improving on SoftTeacher by 3.90, 2.54, and 2.43 mAP (He et al., 2023).

PseCo addresses noisy pseudo boxes differently. Its Prediction-guided Label Assignment (PLA) scores proposals by

$q = s^{\alpha} \times u^{1-\alpha},$

combining classification score and localization quality, and its Positive-proposal Consistency Voting (PCV) estimates pseudo-box reliability by

$\sigma^{j} = \frac{\sum_{i=1}^{N}u_{i}^{j}}{N},$

then uses $\sigma^j$ as an instance-wise weight in unlabeled regression. This explicitly treats coarse pseudo boxes as potentially useful rather than automatically discardable, and on COCO it reports gains over Soft Teacher of 2.0, 1.8, and 2.0 points under 1%, 5%, and 10% labeling ratios (Li et al., 2022).

“Mind the Gap” pushes the same idea into a dedicated refinement module. It trains separate category and box polishing networks on labeled images using synthesized noisy pseudo-label/ground-truth pairs. For pseudo bounding boxes, the network extracts RoI features from the original pseudo box, four shifted boxes, and two enlarged boxes, concatenates the resulting 7 RoI features, and predicts the deviation between the pseudo box and the true box. The paper further disentangles polished pseudo categories and polished pseudo boxes for separate classification and regression, allowing more unlabeled objects to enter training (Zhang et al., 2022).

A different branch of work avoids sparse pseudo-boxes altogether. Dense Teacher argues that pseudo-box pipelines are a lossy intermediate representation because they require score thresholding, NMS, box filtering, label assignment, and hyper-parameter tuning. It replaces sparse pseudo-boxes with Dense Pseudo-Labels (DPLs), namely the teacher’s post-sigmoid dense outputs,

$D_u=\{x_i^u\}_{i=1}^{N_u}$0

followed by top-$D_u=\{x_i^u\}_{i=1}^{N_u}$1 region selection based on

$D_u=\{x_i^u\}_{i=1}^{N_u}$2

This reframes pseudo-boxes as one possible interface, rather than the inevitable one, for unlabeled supervision (Zhou et al., 2022).

Weak supervision yields yet another interpretation. “Semi-Weakly Supervised Object Detection by Sampling Pseudo Ground-Truth Boxes” generates pseudo GT boxes online from selective-search proposals rather than from a teacher network. Proposal probabilities are maintained by

$D_u=\{x_i^u\}_{i=1}^{N_u}$3

and $D_u=\{x_i^u\}_{i=1}^{N_u}$4 proposals per class per image are sampled as pseudo GT, with IoU-based score propagation

$D_u=\{x_i^u\}_{i=1}^{N_u}$5

This replaces offline pseudo-label generation by an online sampled latent-variable mechanism (Meethal et al., 2022).

3. Pseudo 3D boxes in unsupervised and domain-adaptive detection

In unsupervised 3D detection, pseudo-boxes are typically initial 3D bounding boxes obtained from clustering or object discovery, then reused in self-training. UL3D emphasizes that such pseudo 3D boxes are not uniformly wrong: errors can be coordinate-specific in location / position $D_u=\{x_i^u\}_{i=1}^{N_u}$6, shape / size, and orientation $D_u=\{x_i^u\}_{i=1}^{N_u}$7. It therefore estimates uncertainty at the box-coordinate level by comparing a primary and an auxiliary detector,

$D_u=\{x_i^u\}_{i=1}^{N_u}$8

with $D_u=\{x_i^u\}_{i=1}^{N_u}$9, and rectifies the dense-head loss by

$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$0

This yields coordinate-wise partial trust rather than whole-box acceptance or rejection. The paper reports gains of +6.9% AP$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$1 and +2.5% AP$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$2 on nuScenes, and +4.1% AP$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$3 and +2.0% AP$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$4 on Lyft (Zhang et al., 2024).

A complementary strategy improves pseudo-boxes before training the detector. “Enhancing Pseudo-Boxes via Data-Level LiDAR-Camera Fusion for Unsupervised 3D Object Detection” argues that label-level fusion is limited because it does not repair sparse LiDAR geometry. DFU3D instead uses SEEM for instance segmentation and DepthAnything for dense depth estimation, performs bi-directional LiDAR-camera fusion, and then filters the resulting pseudo points locally and globally. Its local filtering keeps pseudo points satisfying

$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$5

with $$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$6, while global filtering retains points

$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$7

Dynamic self-evolution is triggered when

$$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$8

with $$\mathcal{L}= \frac{1}{N_s} \sum_{i=1}^{N_s} \mathcal{L}_s(x^s, y^s)+ \frac{1}{N_u} \sum_{i=1}^{N_u} \alpha \mathcal{L}_u(x^u, y^u),$$9. On nuScenes validation, the method reports 28.4 mAP in class-aware detection (Ji et al., 28 Aug 2025).

Cross-dataset 3D UDA introduces a further complication: pseudo boxes can be unreliable because LiDAR beam-count mismatch changes instance point populations. PERE formalizes pseudo-box confidence intervals with $B^0$0, discards low-confidence boxes, keeps high-confidence boxes, and treats middle-confidence boxes as unreliable. Its Complementary Augmentation chooses between removing all points inside an unreliable box or replacing that region by a transformed high-confidence box of the same class, using

$B^0$1

The same framework also adds interpolation and extrapolation proposals and RoI feature alignment losses to reduce beam-count-induced pseudo-label degradation (Zhang et al., 2024).

Taken together, these 3D methods show three distinct pseudo-box doctrines: coordinate-wise uncertainty weighting, data-level pseudo-box enhancement before detector training, and pseudo-label refinery that edits or replaces ambiguous pseudo-box regions. The shared assumption is that pseudo 3D boxes are necessary but structurally noisy, and that robust learning requires modeling where that noise resides rather than only how confident the detector appears to be.

4. Point-induced pseudo-boxes in single-object tracking

RTrack introduces a different notion of pseudo-box. Instead of treating the object as a directly regressed rectangle, it represents the target by a learned point set

$B^0$2

with default $B^0$3, and derives a pseudo bounding box from those points. The refinement stage predicts offsets

$B^0$4

and the final pseudo-box is recovered by a conversion $B^0$5, with the main implementation using a min-max operation over both axes of the TrackPoints (Zeng et al., 2023).

This design is motivated by the claim that a genuine axis-aligned box is a poor internal representation for tracking deformable or rotating targets because it encloses cluttered background and does not explicitly capture arbitrary appearance. RTrack therefore preserves benchmark compatibility—its output is still a box—but moves the representation burden to a deformable point set that can highlight informative local areas before collapsing back to a rectangle.

The optimization strategy is equally important. RTrack uses a one-to-many leading assignment strategy: the init stage is center-based and one-to-one, while the refine stage selects top-$B^0$6 candidate positives by IoU and sets a dynamic threshold as the sum of the mean and variance of the candidate samples. This relaxes the extreme sparsity of one-positive-sample training in SOT. The final losses include weighted focal classification, stage-weighted GIoU detection losses with $B^0$7 and $B^0$8, and a mutual-affinity term

$B^0$9

Empirically, the paper reports 74.6% mAO for RTrack-256 and 76.4% mAO for RTrack-384 on GOT-10k, with 2.1M and 3.6M training pairs respectively, and states that training cost is reduced to 10% of previous SOTA trackers’ training costs (Zeng et al., 2023). In this literature, a pseudo-box is therefore not a noisy label but an induced box recovered from a more flexible latent representation.

5. Non-unique box realizations in combinatorial geometry

A combinatorial analogue of pseudo-box behavior appears in the study of common box unfoldings. “Unfolding Boxes with Local Constraints” examines polyominoes that can be folded into multiple non-isomorphic rectangular boxes. The canonical example is area 22, where one polyomino folds into both $B^r, S^r = \mathcal{M}(F, B^{r-1}),$0 and $B^r, S^r = \mathcal{M}(F, B^{r-1}),$1. The paper defines $B^r, S^r = \mathcal{M}(F, B^{r-1}),$2 as the smallest area allowing a common unfolding of $B^r, S^r = \mathcal{M}(F, B^{r-1}),$3 boxes and proves

$B^r, S^r = \mathcal{M}(F, B^{r-1}),$4

thereby ruling out 46, 54, and 58 as the smallest areas allowing a common unfolding of three boxes and refuting a conjecture of Xu et al. (2017) (Qian et al., 1 Jun 2025).

The technical contribution is a SAT encoding that replaces global connectivity and acyclicity constraints by local constraints on cut-edge structures, square orientations, and directed edge patterns. The paper begins from four necessary properties of valid cut-edge graphs—connectivity, touching all 8 corners, acyclicity, and a $B^r, S^r = \mathcal{M}(F, B^{r-1}),$5 local necessity condition—and then enforces local orientation and sink-directed constraints with a finite set of forbidden induced subgraphs. This makes large-scale search feasible: the abstract reports that previous approaches could find common unfoldings of two boxes only up to area 88, whereas the new method scales beyond 150, and that enumeration scales from area 30 to 60 (Qian et al., 1 Jun 2025).

Here the pseudo-box phenomenon is not noisy supervision but non-unique inverse realization: a single planar object is compatible with several latent boxes. The common unfolding behaves like a box net but does not determine a unique box identity.

A useful geometric contrast is provided by the literature on ordinary axis-parallel boxes. “Empty axis-parallel boxes” studies true Cartesian products of intervals in $B^r, S^r = \mathcal{M}(F, B^{r-1}),$6, proving

$B^r, S^r = \mathcal{M}(F, B^{r-1}),$7

and constructing point sets whose largest empty axis-parallel box has volume at most

$B^r, S^r = \mathcal{M}(F, B^{r-1}),$8

This paper concerns exact boxes rather than surrogate ones, but it provides the canonical baseline geometry against which more generalized box-like objects can be contrasted (Bukh et al., 2020).

6. Non-geometric box formalisms and cryptographic name overlaps

In non-signalling resource theory, a box is a family of conditional probability distributions rather than a geometric region. The basic object is

$B^r, S^r = \mathcal{M}(F, B^{r-1}),$9

or more generally $$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$0, subject to non-signalling constraints. The 2011 no-broadcasting result studies locality-preserving operations $$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$1 and defines anti-Robustness by

$$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$2

Its main theorem states that any nonlocal $$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$3 box cannot be broadcast by linear operations that map local boxes to local boxes and fully non-signalling boxes to fully non-signalling boxes (Joshi et al., 2011). In this setting, “box” is already a formal resource-theoretic noun; the term has no relation to bounding boxes, but it preserves the broader idea of a boxed abstract object manipulated under structural constraints.

Cryptography introduces a different terminological branch through S-boxes. “On the Design of Chaos-Based S-boxes” argues that many chaos-based constructions overemphasize average coordinate nonlinearity value (ACNV) and underreport true vectorial nonlinearity relevant to linear cryptanalysis. Starting from pseudo-random bijective S-boxes, its heuristics repeatedly reach ACNV = 114.0, and a combined method reaches ACNV = 114.5, outperforming previously published chaotic-function-based S-boxes on that comparison metric (Dimitrov, 2020). This is a “pseudo-random box” usage rather than a pseudo bounding box usage.

“Generating S-Boxes from Semi-fields Pseudo-extensions” is closer in vocabulary to the present topic because it constructs S-Boxes from a pseudo-extension of a semifield. It defines pseudo-irreducibility for

$$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$4

by the condition

$$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$5

and then a pseudo-inverse map

$$b_c = \sum_{i=1}^{n_j} \hat{B}_j^i \times \frac{S_j^i}{\sum_{i=1}^{n_j} S_j^i}.$$6

The paper reports 12,781 non equivalent S-Boxes and 2,684 APN functions (Dumas et al., 2014). These cryptographic objects are substitution boxes, not geometric boxes, but they show how “pseudo-” and “box” combine in yet another formal lineage.

Across these non-geometric literatures, the shared element is lexical rather than representational. A non-signalling box is a probability table; an S-box is a vectorial Boolean function; a pseudo-extension-based S-Box is a cryptographic design constructed from semifield algebra. The connection to perception pseudo-boxes lies mainly in the reuse of the word “box,” not in a common mathematical ontology.

Pseudo-boxes therefore form a cross-disciplinary but non-unified concept. In 2D and 3D perception they are chiefly surrogate bounding boxes used under annotation scarcity and geometric noise; in tracking they are boxes induced from flexible point sets; in combinatorial geometry they arise as non-unique box realizations; and in non-signalling theory and cryptography they reflect entirely different formal uses of “box.” The strongest common thread is methodological: a pseudo-box is rarely the final object of interest, but rather an intermediate, substitute, or multiply realizable structure that makes learning, inference, or formal analysis possible when a unique, exact, or fully observed box is unavailable.