ROOMELSA: 3D Indoor Object Retrieval

• ROOMELSA is a challenge setting for indoor 3D object retrieval that uses masked 2D scene images combined with natural language descriptions.
• It addresses issues such as distorted viewpoints, textureless masked regions, and ambiguous language, which complicate both visual and semantic matching.
• The SAMURAI method leverages CLIP-based semantic matching and shape-aware retrieval with ensemble voting to achieve robust performance under noisy multimodal conditions.

Searching arXiv for papers mentioning ROOMELSA and related challenge context. ROOMELSA is a challenge setting for indoor 3D object retrieval in which a system must identify a target 3D object from a candidate object pool using only a masked 2D scene image and a natural-language description. In the formulation described by "SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification" (Vo et al., 26 Jun 2025), ROOMELSA limits access to full 3D scene context, so retrieval must proceed under partial visual evidence and linguistic ambiguity rather than direct scene-level geometric access. The task is defined over complex indoor environments and is complicated by distorted viewpoints, textureless masked regions, ambiguous language prompts, and noisy segmentation masks.

1. Task definition and formal setting

ROOMELSA is framed as a retrieval problem over indoor 3D objects. For each query scene, the input consists of three elements: a masked RGB image of a scene in which the target object region is marked using a fixed color; a natural language query describing that masked object; and a set of candidate 3D objects represented by object assets including an RGB image, texture, and .obj model. The required output is a ranked top-10 list of the most likely matching candidate objects (Vo et al., 26 Jun 2025).

This formulation makes ROOMELSA a multimodal identification problem rather than a pure vision or pure language benchmark. The masked image provides localized but incomplete appearance evidence, while the text query contributes semantic description that may or may not be sufficiently discriminative. The ranked-output requirement further places the task within retrieval rather than single-label classification.

2. Information constraints and failure modes

A defining property of ROOMELSA is that access to full 3D scene context is restricted. The challenge therefore complicates reasoning about object appearance, geometry, and semantics. In the problem description, neither language alone nor image appearance alone is consistently reliable: masked regions may be noisy, incomplete, distorted by viewpoint, or nearly textureless, while text prompts may be ambiguous or underspecified (Vo et al., 26 Jun 2025).

Several concrete failure modes are explicitly identified. Distorted viewpoints can alter the apparent geometry of the target object. Textureless masked regions reduce the usefulness of appearance-based matching. Ambiguous language prompts weaken semantic specificity. Noisy segmentation masks further corrupt the already limited visual signal. This suggests that robust ROOMELSA systems must tolerate both modal uncertainty and cross-modal inconsistency rather than assuming a clean correspondence between text, image crop, and object asset.

3. Query representation and retrieval target

The query in ROOMELSA is intrinsically hybrid. The masked RGB scene image encodes local visual evidence about the target object as it appears in the scene, but only within the region identified by the fixed-color mask. The natural-language query supplies a separate textual description of that same masked object. The candidate set, by contrast, consists of 3D object assets with an RGB image, texture, and .obj model (Vo et al., 26 Jun 2025).

This asymmetry is methodologically important. The query side is partial, viewpoint-dependent, and potentially noisy; the candidate side includes richer object-level assets. A plausible implication is that retrieval systems are encouraged to exploit features that remain stable across this asymmetry, particularly semantic cues from language and geometry-sensitive cues derived from object shape. The ranked top-10 output also indicates that ROOMELSA evaluates retrieval quality over a short candidate list rather than requiring a single irrevocable prediction.

4. Methodological responses: the SAMURAI approach

The principal method described for ROOMELSA in the provided literature is SAMURAI, a training-free multimodal retrieval system designed specifically for this challenge. SAMURAI combines CLIP-based semantic matching with shape-aware retrieval from binary silhouettes, together with an ensemble-style majority voting scheme that stabilizes predictions across different query types (Vo et al., 26 Jun 2025).

Its design is explicitly motivated by the unreliability of single-modality evidence in ROOMELSA. CLIP-based semantic matching addresses the language-conditioned identification component, while shape-guided re-ranking derived from binary silhouettes of masked regions introduces a geometry-sensitive signal that is less dependent on texture. A dedicated preprocessing pipeline improves mask quality by extracting the largest connected component and removing background noise. The reported result is competitive performance on the ROOMELSA private test set, and the paper interprets this as evidence that combining shape priors with language understanding is important for robust open-world 3D object retrieval (Vo et al., 26 Jun 2025).

From a systems perspective, SAMURAI operationalizes a broader ROOMELSA principle: when appearance evidence is degraded, retrieval benefits from coupling semantic alignment with explicit shape reasoning. The majority voting component further indicates that prediction stability is itself a design target under noisy multimodal conditions.

5. Research significance within multimodal 3D retrieval

ROOMELSA occupies a technically specific niche at the intersection of language grounding, masked-image reasoning, and 3D object retrieval. The challenge is not simply to localize an object in 2D, nor merely to classify a visible instance, nor to reconstruct scene geometry. Instead, it requires recovering the identity of a 3D object from partial scene evidence and an accompanying linguistic description under limited contextual access (Vo et al., 26 Jun 2025).

This setup foregrounds several research themes. First, it tests whether semantic matching alone is sufficient when descriptions are ambiguous or underspecified; the ROOMELSA evidence indicates that it is not consistently reliable. Second, it tests whether local image appearance alone is sufficient when masked regions are incomplete, distorted, or nearly textureless; again, the answer is negative. Third, it motivates hybrid retrieval architectures that fuse language understanding with shape priors. In that sense, ROOMELSA can be read as a constrained evaluation regime for multimodal robustness: systems must succeed when both modalities are informative but imperfect.

A plausible implication is that ROOMELSA is especially relevant for retrieval settings in which the target object is only partially observed and the query must bridge heterogeneous representations, namely 2D masked imagery, free-form language, and 3D asset collections. The emphasis on candidate ranking rather than direct reconstruction reinforces its orientation toward practical retrieval pipelines.

6. Terminological disambiguation

ROOMELSA should be distinguished from several unrelated terms that appear lexically similar in contemporary arXiv literature. In the ELSA paper, "ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over Resource-Constrained Edge Networks" (Yang et al., 20 Jan 2026), the authors explicitly state that the paper introduces ELSA only and that there is no occurrence of the term “ROOMELSA,” no module or acronym expansion with that name, and no explicit naming evidence that links ELSA to ROOMELSA. The most faithful conclusion from that source is therefore that ELSA and ROOMELSA are unrelated.

ROOMELSA is also unrelated to ROME, the head-avatar system introduced in "Realistic One-shot Mesh-based Head Avatars" (Khakhulin et al., 2022). ROME addresses creating a realistic, rigged, mesh-based human head avatar from a single photograph, with a FLAME-compatible rigged mesh backbone, learned non-face geometry deformation, neural texture, and deferred neural rendering. Its domain is one-shot personalized head-avatar creation and cross-person reenactment, not indoor 3D object retrieval.

The available evidence therefore supports a narrow and precise usage: ROOMELSA denotes the indoor 3D object retrieval challenge context described in SAMURAI, whereas ELSA and ROME refer to distinct systems in hierarchical federated LLM fine-tuning and mesh-based head avatars, respectively.