HiddenObject: Challenges in Invisible Object Detection

• HiddenObject is a family of vision tasks where objects become hidden through occlusion, truncation, adversarial patches, or unknown labeling, requiring reasoning beyond simple appearance matching.
• Techniques to localize hidden objects leverage temporal memory, geometric priors, multimodal fusion, and hierarchical semantics to infer invisible targets.
• Benchmarks like SceneFunRI, LA-CATER, and methods such as OPNet, HAFT, and multimodal frameworks illustrate both the challenges and evolving solutions in hidden-object detection.

In the cited literature, HiddenObject denotes not a single benchmark but a family of vision problems in which an object is not straightforwardly observable. Depending on the formulation, the object may be entirely occluded in a single image, contained or carried in a video sequence, only partially visible because of occlusion or truncation, unknown to the detector’s label space yet present in cluttered background, concealed by adverse sensing conditions, or intentionally suppressed from a detector by an adversarial patch. What unifies these formulations is that localization cannot be reduced to ordinary visible-appearance matching: successful systems must instead rely on temporal memory, affordance priors, geometric structure, multimodal fusion, hierarchical semantics, or explicit search over uncertainty (Chen et al., 14 May 2026, Shamsian et al., 2020, Liang et al., 2023, Zhao et al., 2020, Song et al., 28 Aug 2025).

1. Problem space and core definitions

A recurrent distinction across the literature is between physical invisibility and semantic unknownness. In SceneFunRI, the target is a functional object that is entirely occluded in the image and must be inferred from a task instruction such as “unplug the lamp on the side table” (Chen et al., 14 May 2026). In the semantic-abstraction line of work, hidden objects are those that cannot be directly identified by a 2D VLM because they are at least partially occluded, often inside or behind receptacles such as cabinets or drawers (Pais et al., 22 Dec 2025). In object-permanence work, the invisible target may be merely occluded, fully contained, or carried by a moving container, which changes the latent state transition structure of the problem (Shamsian et al., 2020).

A different regime appears in open-world detection. There, “hidden” refers to unknown objects hidden in the background: unlabeled foreground instances that closed-set detectors tend to suppress as background during training and post-processing (Liang et al., 2023). A related but older formulation addresses partial visibility due to occlusion or image-boundary truncation by treating the visible mask as a latent variable inside a sliding-window detector (Ott et al., 2013). Still another regime is adversarial: HiddenObject can mean suppressing detections of target objects in images/videos by placing small, localized adversarial patches, so that real objects disappear after thresholding and NMS (Zhao et al., 2020).

These formulations show that HiddenObject is best understood as a family of tasks indexed by three variables: the source of invisibility, the supervision available, and the representation over which inference is performed. A common misconception is to treat all such cases as variants of ordinary object detection. The surveyed papers instead formalize hidden-object reasoning as task-conditioned search over invisible regions (Chen et al., 14 May 2026), memory-based belief propagation through occlusion (Tokmakov et al., 2022), latent visibility inference (Ott et al., 2013), or unknown-object discovery under open-world semantics (Liang et al., 2023).

2. Task-driven localization of invisible objects in static scenes

SceneFunRI makes hidden-object localization an explicit vision-language benchmark by deriving 2D Reasoning-the-Invisible instances from SceneFun3D, which was introduced by Delitzas et al. for 3D affordance-centric scene understanding. Given an image $I$, a task instruction $T$, and optional scene metadata $S$, the target is to predict either a bounding box $b=[x_1,y_1,x_2,y_2]$ or a heatmap $H(x,y)\in[0,1]$ for a fully occluded functional object. The benchmark’s primary output is a bounding box, and its probabilistic factorization is written as $p(b\mid I,T)\propto p(b\mid I)\cdot p(b\mid T,K)$, where $p(b\mid I)$ captures visual plausibility and $p(b\mid T,K)$ captures functional priors such as “power outlets near floor level, behind furniture” (Chen et al., 14 May 2026).

The construction pipeline begins with 230 scenes and computes two visibility ratios for each frame: projection visibility $V$ and mesh-based visibility $V_o$. Frames are kept when $T$0 and $T$1, meaning that the functional object lies within the camera frustum but is fully occluded in the rendered image. Ground-truth boxes are obtained by projecting the 3D point-cloud extent of the hidden object into the 2D image plane. This produced 1,232 candidates, which were then filtered by a human quality-control pass to 855 high-quality instances. The task emphasizes functional targets—sockets, switches, faucets, handles—rather than appearance-defined objects, so the benchmark isolates task intent, commonsense placement priors, and spatial grounding rather than category recognition alone (Chen et al., 14 May 2026).

SceneFunRI evaluates predictions with IoU, a containment-based criterion CAcc@$T$2, and a normalized corner-distance metric Dist. Because the target is invisible, the benchmark treats containment and compactness as more actionable than exact contour agreement. The strongest baseline, Gemini 3 Flash, reaches only CAcc@75 = 15.20, CAcc@50 = 17.19, mIoU = 0.74, and Dist = 28.65. The best open-source family result is Qwen 3.5 122B with CAcc@75 = 12.63, CAcc@50 = 16.14, mIoU = 0.41, and Dist = 39.76, while human performance is much higher at CAcc@75 = 66.43, mIoU = 3.40, and Dist = 13.39 (Chen et al., 14 May 2026).

Prompting analysis in SceneFunRI distinguishes Strong Instruction Prompting, Reasoning-based Prompting, and Spatial Process of Elimination. Explicitly telling the model that the target is hidden consistently helps: for Gemini 3 Flash, the “Hidden” prompt raises CAcc@75 to 25.03 and CAcc@50 to 31.23, albeit with mIoU slightly reduced to 0.69 and Dist worsened to 31.91. Chain-of-Thought and CoT with Self-Reflection mainly help larger Qwen models; for Qwen 3.5 122B, CoT yields CAcc@75 = 15.79, CAcc@50 = 20.23, and Dist = 37.41. SPoE operationalizes iterative elimination by repeatedly splitting the image, asking the model which half is less plausible, and narrowing the region; for Gemini 3 Flash, Dist decreases steadily with iteration count and drops below the single-shot baseline after about four iterations, although stopping criteria remain unstable (Chen et al., 14 May 2026).

A complementary 3D variant appears in the extension of Semantic Abstraction to hidden objects. There, CLIP relevancy maps are treated as “abstract object” representations, back-projected into 3D using camera intrinsics and extrinsics, fused into occupancy or point-cloud beliefs, and combined with historical placement priors learned by fitting a Gaussian Mixture Model with EM. The system defines first-try success as a predicted location within 0.3 m of ground truth and evaluates 100,000 randomized trials. With asymmetric priors such as 0.7/0.2/0.1 over three clusters, the learned GMM materially improves first-try success over random search; with uniform priors, it offers no advantage, which the paper presents as expected behavior. Multi-view capture also helps disambiguate false positives, while transparent objects remain difficult (Pais et al., 22 Dec 2025).

3. Object permanence and hidden-object tracking in video

In video, HiddenObject becomes a problem of maintaining a latent state when the target disappears. “Learning Object Permanence from Video” decomposes the problem into four scenarios—visible, occluded, contained, and carried—and treats the fourth as the most difficult because the target is invisible and moves only through its container’s motion (Shamsian et al., 2020). On LA-CATER, which augments CATER with per-frame bounding boxes and visibility labels, the target is the golden small sphere (“snitch”), and the train/dev/test split is 9,300/3,327/1,371 videos. Frame-type fractions are approximately visible 63%, occluded 3%, contained 29%, and carried 4.5% (Shamsian et al., 2020).

The resulting OPNet couples a perception module based on Faster R-CNN with two recurrent reasoning modules. The first LSTM answers “who to track?” by assigning attention weights over the detected objects, so that attention can shift from the target itself to a container when containment occurs. The second LSTM answers “where is it?” by regressing the target box from the attended proxy and temporal state. On the LA-CATER test split, OPNet obtains mean IoU 88.89 on visible frames, 78.83 on occluded frames, 76.79 on contained frames, 56.04 on carried frames, and 81.94 overall. Under perfect perception, carried IoU rises to 76.42, indicating that the carried case is dominated by perception and proxy-selection errors rather than box regression alone (Shamsian et al., 2020).

HAFT addresses short-term full occlusion in tracking by hallucinating feature embeddings rather than bounding boxes or pixels. A ResNet-18 backbone initialized from DiMP produces dense feature maps, a ConvGRU predicts the next-step feature embedding, and a lightweight discriminator regularizes this hallucination adversarially in feature space. At test time, HAFT fuses real features $T$3 and hallucinated features $T$4 via

$T$5

with ablation selecting $T$6. This improves robustness under occlusion: on OTB100, the baseline DiMP-18 AUC of 0.660 rises to 0.679 with GAN + L2 hallucination; on VOT2018 HAFT reports Accuracy 0.587, Robustness 0.155, and EAO 0.432 versus 0.402 for DiMP-18; it also reports top-ranked or near-top-ranked results on LaSOT, TrackingNet, and UAV123 (Zhang, 2020).

A more explicitly self-supervised account of object permanence appears in RAM, which builds a space-time memory graph over ConvGRU memories and learns a Markov random walk across those memories without invisible-location annotations. If $T$7 is a one-hot walker initialized at the first visible center, then

$T$8

where each $T$9 is a transition matrix computed from similarities between spatial memory nodes at consecutive times. The training loss supervises the walker only when the object is visible, including after reappearance, so hidden-state propagation is learned indirectly through temporal coherence. RAM reports mIoU 91.7 on visible, 79.3 on occluded, 82.2 on contained, and 63.3 on carried in static LA-CATER, and 90.0/62.5/55.1/51.8 respectively on LA-CATER-Moving. It also improves Track-AP over CenterTrack and PermaTrack on Parallel Domain and KITTI, reaching 72.0 on PD and 75.7 on KITTI validation (Tokmakov et al., 2022).

Taken together, these works separate two mechanisms that are often conflated. Occlusion-only tracking can be handled by motion-consistent feature forecasting or memory propagation, whereas containment and carried motion require proxy tracking of the visible container. This distinction is explicit in OPNet (Shamsian et al., 2020), implicit in HAFT’s feature hallucination (Zhang, 2020), and distributed through RAM’s memory transitions (Tokmakov et al., 2022).

4. Partial visibility and visibility-aware detection

Before the recent wave of transformers and VLMs, partial visibility was already formalized as a latent structured variable. “Detection of Partially Visible Objects” models an object window not as wholly present or absent but as a set of blocks with binary visibility labels $S$0. For a window $S$1 and visibility configuration $S$2, the score of mixture component $S$3 is

$S$4

augmented by an Ising-style contiguity prior to prefer spatially coherent visible regions. Inference over $S$5 is solved exactly by graph cuts after a fast upper-bound filter, and the final detector retains not just a bounding box but a block-level visibility mask indicating which parts are visible and which are occluded or truncated (Ott et al., 2013).

This formulation matters because standard IoU-based NMS treats hidden extent and visible extent as equivalent. The visibility-aware alternative defines overlap from the Jaccard similarity of visible blocks rather than the full box and then solves a globally optimal selection problem with branch-and-bound. On PASCAL VOC 2010, adding latent visibility improves mean AP from 12.2% to 12.9%, with car increasing from 22.3% to 23.5% and horse from 13.4% to 14.9%. The full visibility-aware NMS variant yields mean AP 12.3%, slightly below the greedy-NMS version but with more principled handling of overlapping occluded instances and richer visible-part output (Ott et al., 2013).

A common misconception is that occlusion-aware detection is just ordinary detection with more tolerant thresholds. The latent-visibility work shows instead that the detector’s internal representation must change: missing evidence should be compensated by learned unary terms, contiguous visibility should be explicitly modeled, and post-processing should reason over visible support rather than hallucinated full extents (Ott et al., 2013).

5. Unknown objects in open-world detection

Open-world object detection treats hidden objects as unlabeled foreground instances embedded in clutter and therefore at risk of being learned as background. UnSniffer addresses this by introducing a Generalized Object Confidence head supervised only on proposals intersecting known objects, thereby avoiding direct background supervision that could suppress unknowns. It complements this with Negative Energy Suppression, which pushes the lowest negative-energy proposals toward non-object behavior, and replaces hand-designed NMS with recursive normalized cuts over a graph whose nodes are proposals and whose edge weights are IoU overlaps (Liang et al., 2023).

On the UOD-Benchmark introduced in the same work, UnSniffer reports on COCO-OOD: U-AP = 0.454, U-F1 = 0.479, U-PRE = 0.433, and U-REC = 0.535, while known mAP on VOC falls only slightly from 0.483 for Faster R-CNN to 0.464. On COCO-Mix it reports U-AP = 0.150, U-F1 = 0.287, U-PRE = 0.222, U-REC = 0.409, and A-OSE = 398. Ablation shows that GOC mainly drives discovery, NES reduces false positives, and graph-based box determination improves precision by selecting a single box per object cluster (Liang et al., 2023).

SS-OWFormer moves the open-world setting into a semi-supervised regime. It uses a Deformable DETR-style detector with a novelty head, an objectness branch, feature alignment between original and photometrically augmented images, and pseudo-labeling for unknowns via query-modulated encoder maps pooled over predicted boxes. On COCO Task 1, SS-OWFormer reports U-Recall 12.26 and mAP 59.85 with 100% labels, 11.13/55.51 with 50% labels, 11.98/46.87 with 25% labels, and 11.92/32.81 with 10% labels; in each of these settings it outperforms OW-DETR at the same label fraction in unknown recall. The same framework is also evaluated on PASCAL, Objects365, and DOTA, where it adds oriented-box support and uses 250 queries for remote sensing (Mullappilly et al., 2024).

O1O argues that geometric pseudo-labels improve unknown recall but also inject confusion into known-class classification. Its remedy is to group known classes into superclasses and compute an odd-one-out score from superclass-conditioned class probabilities:

$S$6

The detector is trained with geometric pseudo-boxes derived from a normals-based RPN, while unknown classification remains indirect. On M-OWOD Task 1, O1O reports U-Recall 49.3 versus 28.3 for PROB†, with known mAP 65.1 versus 66.4; on S-OWOD Task 1 it reports U-Recall 49.8 and mAP 72.6. The paper attributes the trade-off improvement to better partitioning of feature space via superclasses (Yavuz et al., 2024).

Hyp-OW addresses the same basic regime through hierarchical hyperbolic structure. Queries are mapped into a Poincaré-ball embedding, class embeddings are regularized toward superclass centroids via a SuperClass Regularizer, and unmatched proposals are relabeled as unknown if they fall within an adaptive hyperbolic distance threshold determined by matched known proposals. Across OW-DETR, OWOD, and a Hierarchical split, the paper reports unknown-recall gains up to about 6 points over prior state of the art, with particularly large reductions in A-OSE on the Hierarchical split—for example, 7,420 versus 14,962 for PROB in Task 1 (Doan et al., 2023).

Across these OWOD papers, “unknown” is not synonymous with “invisible.” The object may be visually present yet semantically absent from the label set. This distinction underlies the design choice in UnSniffer to avoid background supervision for objectness (Liang et al., 2023), in SS-OWFormer to learn from unlabeled data without dense oracle input (Mullappilly et al., 2024), and in O1O and Hyp-OW to structure the representation space so that unknowns are treated as out-of-partition foreground rather than as residual background (Yavuz et al., 2024, Doan et al., 2023).

6. Adversarial concealment and multimodal recovery

A very different HiddenObject problem is to make a visible object disappear from a detector. “Object Hider” studies localized adversarial patches as a practical alternative to imperceptible full-image perturbations. Its objective sums the confidence scores of all detections above threshold $S$7 across a set of target detectors and updates patch parameters by FGSM:

$S$8

with

$S$9

Two placement strategies are proposed. The heatmap-based method generalizes Grad-CAM to detection by aggregating box-level heatmaps over multiple layers and boxes, while the consensus-based method first probes per-model sensitivity with an $b=[x_1,y_1,x_2,y_2]$0-regularized iterative attack and then votes on patch locations across detectors before final FGSM optimization (Zhao et al., 2020).

On the AIC Phase IV CIKM-2020 challenge, the basic heatmap and consensus variants obtain FinalScore 1390 and 1507 respectively; with grid sparsification these rise to 1936 and 2400; with both grid sparsification and multi-scale patch ensembles over $b=[x_1,y_1,x_2,y_2]$1 they reach 2713 and 3071. The consensus-based method consistently outperforms the heatmap-based method, and the system placed in the top 7 among 1701 teams. The paper also notes important limitations: the perturbations are visually obvious, EOT is not used, robustness to detector retraining is unstudied, and fixed patch sizes can be suboptimal (Zhao et al., 2020).

The converse problem—recovering hidden objects rather than suppressing them—appears in the multimodal “HiddenObject” framework for RGB, thermal, and depth sensing. This system uses a dual-stream modality-agnostic encoder built on VMamba, a Multimodal Feature Fusion module based on selective state-space scans, and a channel-wise Mamba decoder. Rather than outputting bounding boxes, it predicts dense segmentation or saliency-style masks for hidden or partially concealed targets under occlusion, camouflage, low illumination, and adverse weather (Song et al., 28 Aug 2025).

Its reported results span both RGB-T and RGB-D segmentation settings. On MFNet, the model reports mAcc 73.8 and mIoU 61.5; on PST900, mAcc 93.3 and mIoU 88.5; on NYU Depth V2, mIoU 56.8; on SUN RGB-D, mIoU 52.1. On ACOD-12K, which targets concealed crops, it reports $b=[x_1,y_1,x_2,y_2]$2, $b=[x_1,y_1,x_2,y_2]$3, and $b=[x_1,y_1,x_2,y_2]$4, which the paper describes as state-of-the-art across all reported metrics. The main claim is not that one modality dominates, but that thermal is critical under low light, depth improves boundaries and concealed-crop delineation, and the selective Mamba fusion can down-weight noisy channels while remaining flexible to missing modalities (Song et al., 28 Aug 2025).

Taken together, the adversarial and multimodal lines expose two complementary facts. First, modern detectors can be made to “ignore” real objects through localized perturbations that directly target confidence thresholds (Zhao et al., 2020). Second, the same notion of hiddenness can be mitigated by adding sensing modalities that recover complementary evidence under camouflage or illumination failure (Song et al., 28 Aug 2025). A plausible implication is that future HiddenObject systems will increasingly combine uncertainty-aware reasoning, explicit priors, and cross-modal robustness rather than relying on unimodal appearance alone.