MemoryDiorama: Generating Dynamic 3D Diorama from Everyday Photos for Memory Recall

Abstract: We present MemoryDiorama, a prototype system that introduces augmented memory cues, a concept that extends captured personal media with AI-generated contextual information to enhance autobiographical memory recall. MemoryDiorama transforms everyday photos into dynamic 3D dioramas in mixed reality by integrating LLM-based scene analysis with 3D object generation, animation, and spatial composition. The system extracts geographic information, object attributes, lighting conditions, and atmospheric elements from the photos. It then animates these elements with generative components such as object animations, human motion, geographical effects, and particle effects to provide richer cues for memory recall. We evaluated MemoryDiorama in a within-subject user study with 18 participants, comparing three conditions: Photo-Only, Static Diorama, and MemoryDiorama. Compared with both Photo-Only and Static Diorama, MemoryDiorama elicited more internal and in-cue details during recall. It also increased perceptual details and visual vividness ratings, suggesting richer recollective experience.

• The paper introduces augmented memory cues with a three-phase pipeline that transforms everyday photos into dynamic 3D dioramas.
• The paper employs multimodal LLMs and generative models to extract and reconstruct spatial, sensory, and contextual elements, achieving 100% scene analysis and over 90% element generation success.
• The paper demonstrates statistically significant improvements in autobiographical recall and visual vividness compared to static methods, validated through user studies with p-values < 0.05 to < 0.01.

MemoryDiorama: Generative Augmented Memory Cues for 3D Autobiographical Recall

Conceptual Framework: Augmented Memory Cues

The paper develops the conceptual foundation for "augmented memory cues," a novel class of memory triggers that leverage generative AI to enrich personally captured media with inferred contextual information. This twofold grounding—autobiographical connection and cue diversity—is articulated through both theoretical analysis and formative user study. The system ensures cues remain tightly anchored to the user's lived experience while introducing diverse, complementary modalities such as spatial expansion, sensory augmentation (smell, touch, ambient sound), and dynamic 3D scene reconstruction. This theoretical groundwork positions augmented memory cues distinctively—providing richer recall pathways without severing experiential authenticity.

Figure 1: Conceptual placement of augmented memory cues across autobiographical connection and cue diversity.

Representative implementations include multimodal augmentations, as systematized visually:

Figure 2: Examples of augmented memory cues, including sensory (smell, tactile, ambient sound) and visual-spatial (expanded views, 3D) extensions.

Formative Study: Cue Patterns in Autobiographical Recall

A design elicitation study with participants empirically identifies five recurring cue patterns: object-related, human-related, geographical, lighting-related, and particle-related cues. Crucially, recall was not limited to static visuals; dynamic, spatial, and non-visual contextual dimensions repeatedly surfaced. These cue typologies directly inform the MemoryDiorama augmentation layers and drive the focus on reconstructing rich, multi-dimensional scene elements in the system.

Figure 3: Recurring cue patterns elicited from the formative study.

MemoryDiorama System Architecture

MemoryDiorama operationalizes the augmented cue concept via a three-phase pipeline: (1) photo analysis, (2) element generation, and (3) placement and route generation. Scene analysis employs multimodal LLMs (Gemini 3) to extract spatial, semantic, and environmental features from photo collections. Segment Anything Model 3 (SAM 3D) and generative image models (Nano Banana 2) synthesize 3D assets and textures, while spatial placement and animation paths are calculated via LLM-assisted annotated mapping and OpenCV pixel-to-3D conversion.

Figure 4: MemoryDiorama system pipeline transforming photo collections into 3D dioramas.

Layered Cue Components

Each augmentation layer directly reflects empirical cue typologies:

• Object Layer: 3D generation and animation of salient objects (vehicles, animals), driven by semantic extraction and physical scaling.

  Figure 5: Object layer visualizations for air balloon, train/car, and boat.

• Human Layer: Spatial deployment and behavioral animation of human figures, leveraging density estimations and route-based movement.

  Figure 6: Human layer examples: dancing, pedestrians walking, ski jumping.

• Particle Layer: Environmental particle effects, with dynamic intensity modulation based on atmospheric cues.

  Figure 7: Particle layer: snow, fog, cherry blossom.

• Lighting Layer: Scene illumination via adjustable sunlight, streetlights, and custom emissive effects.

  Figure 8: Lighting layer: sunset lighting, streetlight, illuminated objects.

• Geographical Layer: Terrain and hydrological augmentation, including snow-covered ground and animated rivers/oceans.

  Figure 9: Geographical layer: snow-covered terrain, animated river, animated ocean.

Technical Feasibility

End-to-end processing demonstrates robust performance, with a mean pipeline runtime of 15.70 minutes per diorama. Scene analysis achieves 100% success; element generation and spatial placement exhibit minor failure modes (merged objects, mis-annotation), but exceed 90% and 75% success rates, respectively. The pipeline integrates geospatial plugins (Cesium for Unity) and state-of-the-art generative model APIs for scalable deployment.

User Study: Quantitative Impact on Memory Recall

A within-subject study (n=18) compares MemoryDiorama with Photo-Only and Static Diorama conditions, matched for event salience and temporal proximity. Results show that MemoryDiorama elicits statistically significant increases in:

• Internal details: $p < .05$ versus both baselines.
• In-cue details: $p < .01$ versus both baselines.
• Perceptual details: $p < .05$.
• Visual vividness ratings: $p < .05$.

  Figure 10: User study results: MemoryDiorama outperforms Photo-Only and Static Diorama in internal, in-cue, and perceptual detail counts; increases visual vividness and enjoyment ratings.

The total proportion of internal details is not selectively enhanced (no significant difference), indicating broad recall expansion (not specificity). Enjoyment is significantly higher ($p < .01$) for MemoryDiorama, while NASA-TLX workload remains comparable across conditions.

Practical and Theoretical Implications

The findings substantiate the claim that AI-augmented cues, as implemented in MemoryDiorama, support deeper autobiographical recall and perceptual re-experiencing. Practically, this system opens avenues for mixed-reality reminiscence, immersive memory archiving, and retroactive data enrichment. Theoretical implications include nuanced expansion of encoding specificity principle (contextual congruence via generative augmentation), and the potential for cross-modal cue design beyond vision (e.g., olfactory, haptic augmentations).

A salient ethical consideration—false memory induction through generative augmentation—remains unresolved in this study. Past research indicates susceptibility to misinformation via manipulated media, warranting rigorous assessment of MemoryDiorama's risk profile in future studies.

Future Developments

Key directions include:

• Multimodal expansion: Incorporation of olfactory and tactile augmentations, based on modality-specific retrieval effects.
• Accuracy validation: Systematic assessment of false memory prevalence and mitigation strategies.
• Component isolation: Experimental breakdown of individual cue layer contribution.
• Real-world deployment: Contextual adaptation for everyday reminiscence, beyond MR laboratory setups.

Conclusion

MemoryDiorama operationalizes "augmented memory cues" with generative, layered 3D dioramas rooted in personal photo collections. Quantitative evaluation demonstrates significant enhancement of sensory-rich autobiographical recall and recollective vividness, validating the system's theoretical claims. The broader implications encompass mixed-reality memory systems, generative cue theory, and ethical inquiry into memory authenticity. MemoryDiorama establishes, within rigorous technical and empirical bounds, a viable framework for AI-augmented memory support (2604.06773).