Reliable evidence grounding for hidden-object reasoning

Establish reliable use of task-relevant visual evidence by vision-language models when identifying objects that are completely occluded, particularly when the objects must be recovered from physical projections such as shadows and reflections, in order to address the stated open challenge in current vision-language models.

Background

The paper introduces OccludeBench to test whether vision-LLMs can recover fully occluded objects from the evidence remaining in an image. It distinguishes direct cases, where the answer depends on a shadow or reflection, from inferential cases, where broader scene context is sufficient.

The experiments show a substantial performance gap between direct and inferential cases for current models. OccludeTrace localizes the main failure to extracting the relevant object from a projection while that projection is embedded in the full image. Although the inpaint-then-confront intervention reduces the gap for models with image-generation capabilities, the broader capability of grounding predictions reliably in task-relevant visual evidence remains unresolved.