What to Distinguish and How? Opportunities and Challenges of Augmenting Multiple, Cluttered Objects in Complex Scenes for People with Low Vision

Abstract: People with low vision (PLV) struggle to perceive complex scenes like busy kitchens and crowded streets, which contain many objects, visual clutter, and dynamic elements. Prior AR systems for low vision either enhance low-level visual features or augment task-relevant objects for single tasks in simple settings, leaving multi-object augmentation in complex scenes underexplored. Informed by a formative study characterizing important objects and their perceived importance for PLV, we built SceneGlance, a wearable AR system that recognizes important objects and visually distinguishes them by importance level. Through a controlled lab study with 12 PLV in a mock-up kitchen scene and a free-form think-aloud study with 13 PLV navigating an outdoor route, we found that AR distinction on object importance shifted PLV's attention toward objects of higher importance, and supported perception strategies such as building mental snapshots from the augmentation distribution and hierarchical scanning by importance. However, this attention shift came with a tradeoff, as augmenting many objects reduced overall scene recall. The studies also surfaced challenges posed by AR augmentations in complex scenes, such as adjacent augmentations blending or interfering with each other, yielding design implications for more practical AR vision enhancement systems in the complex real world.

• The paper presents SceneGlance, an AR system that distinguishes multiple objects by importance to guide visual attention for people with low vision.
• It employs fine-tuned RTMDet-Ins-l models and precise 2D–3D mapping to achieve robust, real-time recognition in complex indoor and outdoor scenes.
• Experimental results reveal increased detection of primary objects (79.2%) but reduced overall scene recall, highlighting a trade-off in augmentation design.

Augmenting and Distinguishing Cluttered Objects in Complex Scenes for Low Vision: Technical Insights and Design Implications

Research Motivation and Problem Space

Perceiving and interpreting complex visual scenes comprising numerous, cluttered, or visually ambiguous objects remains a fundamental barrier for people with low vision (PLV), impacting daily activities such as cooking or street navigation. Conventional AR systems predominantly enhance low-level scene features indiscriminately, exacerbating visual clutter and cognitive overload when deployed in such complex environments. Prior solutions typically focus on augmenting single, task-relevant objects, leaving the challenge of simultaneously distinguishing multiple objects of graded importance in real-world scenes underexplored. The paper presents the SceneGlance system, addressing the dual question: “What to distinguish and how?” when augmenting multiple, cluttered objects in dynamic, everyday contexts for PLV (2607.10902).

Figure 1: Overview of the research method, covering a formative study with PLV, the SceneGlance system design, and its evaluation in both kitchen and outdoor street scenarios.

Formative Study: Object Importance Taxonomy and AR Distinction Design

A formative home-based study with PLV established a technical taxonomy of object importance grounded in three categories: safety-related (hazards/indicators), visually challenging (low-contrast, transparent, cluttered, small, ambiguous shape), and frequently used objects. Importance level was modulated by risk severity and visual difficulty. The study identified AR augmentation preferences, favoring static outlines and solid overlays for less visual interference, reserving dynamic effects for high-risk objects, and advocated flexible distinction methods—by form, color, or additional visual information—to visually encode object importance.

Figure 2: Visualization of the three AR distinction methods in SceneGlance: by form, color, and additional information.

System Architecture and Object Recognition Pipeline

SceneGlance utilizes a head-mounted HoloLens platform, integrating streaming RGB-depth sensing and real-time backend inference. Object recognition is performed by fine-tuned RTMDet-Ins-l segmentation models on custom Kitchen-Importance and Outdoor-Importance datasets, targeting primary and secondary-important objects. The HoloLens renders augmentations after precise 2D–3D mapping using camera intrinsics and environmental mesh. Latency is minimized via pose reprojection for static objects (11 ms perceptible delay; 176 ms pipeline latency, 28.56 FPS).

Figure 3: End-to-end pipeline for real-time recognition, localization, and augmentation with measured latencies at each stage.

Recognition accuracy benchmarks demonstrate robust performance (Kitchen mAP = 0.412; Outdoor mAP = 0.324) with substantial improvement over baseline MS-COCO models (Kitchen mAP = 0.168; Outdoor mAP = 0.144). Fine-tuned models sustain segmentation and classification fidelity under partial occlusion and real-world clutter.

Figure 4: Examples of model inference outputs in complex kitchen and street scenes, highlighting robustness under occlusion.

Controlled Evaluation: Perception Strategies and Attention Modulation in Kitchen Task

A quantitative study with PLV in a densely populated kitchen setup (30–33 objects; primary, secondary, and non-important) assessed the cognitive effects of distinction-based augmentation compared to reality and uniform AR conditions.

Figure 5: Kitchen task setup (A) and AR-augmented participant view (B), with primary-important objects highlighted in solid overlays and secondary-important in outlines.

SceneGlance induces a marked attention shift: the proportion of primary-important objects noticed first increased from 20.8% in the natural setting to 79.2% with distinction-augmented AR. Distinction-based augmentation facilitated the formation of spatial “snapshots,” hierarchical scanning, and anchor-object referencing, aiding cognitive organization and task focus. Contradictorily, augmenting numerous objects reduced overall scene recall, revealing an inherent trade-off between focused attention and holistic memory.

Multi-Object Augmentation Challenges: Occlusion, Merging, and Clutter

Detailed error analysis in kitchen trials reveals that static outlines and overlays amplify ambiguity for partially occluded and visually similar objects, while adjacent augmentations can visually merge, producing misleading composite shapes. Icon label augmentations, while providing categorical clarity, introduce spatial misalignment and new occlusions, increasing cognitive load and interaction error risk.

Figure 6: Representative perception challenges—occlusion, visual similarity, merging overlays/outlines, and icon misalignment/occlusion.

Outdoor Navigation Study: Ground Surfaces, Dynamic Objects, and Distinction Preferences

Free-form outdoor evaluation exposes unique demands for continuous surface augmentation (sidewalks, crosswalks) and dynamic object tracking (pedestrians, vehicles). Augmentation of walkable areas supports safety but introduces outline/overlay conflicts and amplifies segmentation errors. For dynamic objects, importance should be context-dependent (path-crossing probability). Recognition errors impact safety asymmetrically—false negatives are critical, false positives less so.

Figure 7: The urban navigation route with intersections and railway crossings.

Figure 8: Outdoor perception challenges—outline intersections, ambiguous icon placement, and imprecise surface boundaries.

Distinction design preferences shift in outdoor contexts: outdoor lighting degrades color differentiation, favoring form or informational augmentations. Adaptive distinction via outline thickness or icon usage is proposed.

Design Implications and Future Directions

The system demonstrates that AR distinction reliably guides PLV attention and facilitates cognitive scene organization in complex environments. However, augmentation-induced clutter, merging, and spatial misalignment can undermine holistic perception and introduce new accessibility barriers. Importance must be modeled as context- and trajectory-dependent, and distinction granularity should adapt dynamically to user focus and environmental conditions.

Practical implications include integration of scene graph generation for spatial relation modeling, amodal segmentation for occlusion management, and trajectory prediction for dynamically prioritizing moving objects. Adaptive aggregation and group augmentation should be considered to balance cognitive load. Theoretical implications suggest that AR systems for PLV must go beyond object-level importance, contextualizing augmentation within user tasks, spatial relations, and environmental dynamics.

Conclusion

This work provides a technical framework and empirical evidence for the augmentation and distinction of multiple objects in complex scenes for low vision. SceneGlance showcases how graded AR distinction can effectively direct user attention but introduces a trade-off with overall recall and scene complexity. The findings inform the design of future adaptive, context-aware AR vision support systems capable of scaling to large, dynamic, real-world environments, emphasizing the need for rigorous augmentation strategies that minimize clutter, manage occlusion, and adapt to user and environmental context.