Looking Into the Past: Eye Movements Characterize Elements of Autobiographical Recall in Interviews with Holocaust Survivors

Abstract: Eye movement and memory retrieval are deeply and bidirectionally intertwined, however existing literature is generally confined to controlled lab settings. We investigate the relationship between eye gaze and memory recall in free-form autobiographical recall, which comprises both autonoetic consciousness -- the ability to mentally place oneself in the past or future -- and various affective states. Using a large video corpus of semi-naturalistic interviews with Holocaust survivors (N = 806), we examine eye movements with respect to episodic, semantic, affective, and temporal dimensions of traumatic and highly emotional autobiographical recall. We observe gaze patterns vary significantly across certain temporal contexts, most prominently in vertical eye movements. We additionally train intra-subject sequence models to predict temporal context of sentences from segments of gaze features, and find that eye movements entirely preceding sentence onset are sufficient for prediction. Our results corroborate prior findings in literature linking eye movements to memory in controlled and semi-structured settings, reinforcing the role of eye gaze in retrieving and constructing memories, especially in highly emotional and remote memory recall.

• The paper demonstrates that vertical gaze aversion patterns predict the temporal context of autobiographical recall using GAMMs and deep temporal models.
• A combination of eye-tracking, automated transcript annotation, and speech emotion recognition enabled precise measurement of affective and temporal gaze dynamics.
• Robust statistical tests and ablation studies confirm that both vertical and horizontal gaze metrics are essential for accurate contextual prediction in traumatic recollections.

Eye Movement Signatures of Autobiographical Memory Recall in Holocaust Survivor Interviews

Introduction

The paper "Looking Into the Past: Eye Movements Characterize Elements of Autobiographical Recall in Interviews with Holocaust Survivors" (2604.22016) explores the temporal and affective dimensions of eye movements associated with autobiographical memory recall. Leveraging a large-scale, ecologically valid dataset of Holocaust survivor interviews, the research analyzes gaze dynamics aligned with internal (episodic) and external (semantic) recall across different life periods—pre-war, wartime, liberation, post-war, and present-day. In contrast to tightly controlled laboratory studies, this work addresses the interplay of gaze behavior and memory retrieval in semi-naturalistic, emotionally intense contexts.

Dataset and Methodology

Interviews from the USC Shoah Foundation Visual History Archive provide both dense, word-aligned transcript annotation and continuous video at 30 FPS. The distilled corpus consists of over 239,000 sentences classified by recall type and temporal context. Autobiographical recall annotations are produced automatically by prompting a state-of-the-art LLM, GPT-OSS-120B, fine-tuned to match Autobiographical Interview protocol guidelines. Emotional valence, arousal, and dominance (VAD) are estimated from audio using established SER frameworks.

Eye gaze is extracted with OpenFace 2.0 and transformed to local coordinates relative to each interviewee; features include horizontal and vertical position, gaze angle, and relative offset from inferred interviewer position. High-level features are robustly normalized, and all gaze windows are segmented relative to sentence onsets.

Affective and Temporal Gaze Dynamics

The paper demonstrates that both affective state (VAD) and gaze behavior co-vary around the onset of memory recall utterances. Notably, significant changes in valence and dominance precede sentence onsets, especially when discussing emotionally remote periods such as pre-war childhood. Low overall valence and dominance metrics across contexts reflect the traumatic nature of the narratives. These affect trajectories often align temporally with upswings in vertical gaze aversion.

Figure 1: Valence, arousal, and dominance estimated from audio recordings and aligned with transcripts.

Temporal Structure of Eye Movement Patterns

Generalized Additive Mixed Models (GAMMs) reveal clearly structured gaze dynamics across temporal contexts. In all categories, vertical gaze direction increases—indicating gaze is directed upward or away from the interviewer—peaking about one second post sentence onset. This vertical aversion is most pronounced when recalling pre-war experiences.

Figure 2: Gaze dynamics (GAMM) across temporal contexts around sentence onset.

Vertical gaze variability further differentiates temporal contexts. For pre-war recall, the upward deviation is marked and persistent, while wartime, liberation, post-war, and present-day displays are comparatively attenuated. Horizontal gaze components, in contrast, show less temporal specificity or statistical significance in the context of recall type.

Figure 3: Vertical gaze direction in local coordinates.

Statistical Associations and Predictive Modeling

Statistical analysis using Welch’s t-tests confirms that vertical gaze features (mean and standard deviation) differ significantly for sentences annotated as internal (episodic) versus external (semantic). These distinctions are robust after controlling for interview progression and individual baseline behaviors.

A critical finding is that deep temporal models (TCNN and S4 architectures) trained solely on gaze sequences can predict the temporal context (e.g., pre-war, wartime, post-war) of upcoming utterances with AUCs well above chance. This predictive capability holds even for gaze windows entirely preceding the participant’s speech onset, underscoring that eye movements anticipate or prepare for periods of heightened internal recall.

Figure 4: CNN performance across context window offsets.

An ablation experiment shows that both vertical and horizontal gaze features are jointly necessary for optimal temporal context prediction, despite univariate statistical tests highlighting the vertical component. When controlling for the confound that consecutive sentences often belong to the same context, model performance drops, but remains significantly above chance, highlighting the genuine encoding of temporal context in gaze patterns.

Figure 5: Model prediction results by prior context. Performance drops when temporal context changes, but remains above chance. '-1s offset' refers to the eye gaze window starting one second prior to sentence onset.

Theoretical and Practical Implications

The research substantiates a strong, temporally ordered coupling between gaze aversion—especially in the vertical axis—and autobiographical memory retrieval. This extends laboratory findings (e.g., upward gaze with internal attention [servais2022gaze, martarelli2017time]) to a noisy, emotionally complex, and far longer timescale. It demonstrates that eye movements provide nonverbal markers for mental time travel, not only reflecting affective load but also the episodic/semantic distinction in natural recall.

From a practical perspective, these findings enable automatic prediction of memory context and affect from eye movement data in interview or clinical settings, even with low-resolution recordings and coarse annotation. The results have implications for the development of multimodal cognitive assessment tools, memory aids, or trauma-informed interviewing systems.

Theoretically, the study prompts further examination of the neural and computational mechanisms by which gaze behavior is coordinated with internal narration, attentional shifts, and affective processing, particularly under high emotional load and self-referential cognitive activity. Future research avenues include integrating text, prosody, facial expression, and gaze for richer sentiment analysis or reconstructing lived experience chronology from multimodal behavioral signatures.

Conclusion

This work provides a large-scale, ecologically valid demonstration that eye movement dynamics—especially vertical gaze aversion—encode the temporal and affective structure of autobiographical memory recall in Holocaust survivor interviews. Statistical and neural sequence models show that gaze patterns anticipate the type and epoch of forthcoming recall, contributing to our understanding of the embodied markers of memory and supporting the use of automated analysis pipelines for cognitive and affective state inference in real-world settings.