- The paper introduces LiFT, a longitudinal instruction fine-tuning framework that combines temporal prompts, curriculum learning, temporal–label conditioning, and multi-objective LoRA training to improve in-context learning.
- LiFT consistently outperforms base-model in-context learning across four model families, five datasets, and 0-, 1-, and 3-shot settings, including gains such as OLMo-7B on LRS from .410 to .578 and LLaMA-8B on Reddit from .324 to .521.
- Mechanistic analyses show that LiFT makes models attend more strongly to recent, temporally ordered history and rely causally on it, while the largest benefits appear in imbalanced change-detection tasks and held-out domains.
Motivation and problem setting
Longitudinal NLP tasks—stance switch detection, moments-of-change identification in mental health timelines, persuasion outcome prediction—require reasoning over temporally ordered sequences of observations rather than independent instances. The paper's central observation is that LLMs perform poorly on such tasks under standard in-context learning (ICL): predictive signals are distributed across multiple observations, histories are sequentially interdependent, change events are rare, and simply concatenating historical posts into a prompt forces the model to infer temporal structure implicitly. Prior work has shown that improving prompting strategies alone does not close this gap.
The paper introduces LiFT (Longitudinal Instruction Fine-Tuning), a framework that unifies diverse longitudinal tasks under a shared instruction schema and fine-tunes models to use temporally structured history more effectively. The core claim is that instruction fine-tuning (IFT) designed around temporal structure improves ICL itself: after LiFT training, models extract more value from in-context demonstrations than their base counterparts under identical prompts.
Framework design
LiFT consists of two modules. The IFT Builder converts temporal datasets into longitudinal instruction prompts with rolling historical contexts: examples are grouped by a sequence key (timelineID, topic, or userID), sorted chronologically, and the oldest history is truncated iteratively to satisfy a fixed token budget. History lines are prefixed with relative time indices (t-3, t-2, t-1). Prompts follow a structured schema with <instruction>, <few-shot>, <query>, <output>, <hist>, and <curr> control tokens added to the tokenizer vocabulary.
The IFT Trainer applies LoRA-based parameter-efficient adaptation (base weights frozen) with three components:
- Curriculum learning: a three-stage curriculum of increasing temporal difficulty and context length—AnnoMI (~142 tokens/timeline, k=1), LRS (~216 tokens, k=2), TalkLife (~1.0K tokens, k=3)—with LoRA rank increasing across stages (r=4→8→16), previous adapter weights loaded via rank-resize initialization with earlier rank slices frozen.
- Temporal–label conditioning: an embedding module encoding absolute position, relative position-to-end, global label id, and absolute timestep id, projected through an MLP and injected additively into input representations.
- Multi-objective loss: next-token cross-entropy on prompt tokens excluding history spans, focal LM loss on output tokens only, focal classification loss over a shared global label head, and an auxiliary history-only classification loss computed from pooled
<HIST> hidden states, encouraging extraction of predictive signal from history.
Experimental setup
Evaluation covers five datasets: AnnoMI (motivational interviewing client-talk classification), LRS (stance switch detection on Twitter rumour threads), TalkLife (moments of change in mental health timelines), and two held-out OOD sets—Reddit mood-change detection and Change My View (view-change prediction from reply threads). Backbones span four scales and architectures: OLMo-1B/7B, LLaMA-3.1-8B-Instruct, and Qwen2.5-14B-Instruct. LiFT-trained models are compared against base-model ICL under identical demonstrations, formatting, and decoding, at 0-, 1-, and 3-shot settings, using macro-F1.
Main results
LiFT consistently outperforms base-model ICL across all datasets, shot settings, and model sizes. Representative macro-F1 figures:
| Model |
Dataset |
Base (0/1/3-shot) |
LiFT (0/1/3-shot) |
| OLMo-7B |
LRS |
.401/.381/.410 |
.551/.567/.578 |
| OLMo-7B |
TalkLife |
.056/.050/.059 |
.104/.121/.123 |
| LLaMA-8B |
Reddit |
.283/.267/.324 |
.454/.498/.521 |
| Qwen-14B |
CMV |
.512/.524/.538 |
.549/.562/.577 |
Two patterns deserve emphasis. First, base models frequently fail to benefit monotonically from additional demonstrations—for example, OLMo-1B drops from .250 to .211 on AnnoMI moving from 0- to 1-shot—whereas LiFT models improve reliably with shots (e.g., OLMo-7B on Reddit rises .283→.301→.333). This indicates the fine-tuning specifically repairs few-shot contextualization, not merely task accuracy. Second, gains are largest on imbalanced datasets with sparse minority change events: TalkLife is the clearest case, where OLMo-1B's near-collapsed base performance (.044) roughly doubles under LiFT (.101), and per-class results show the IS (switch) class moving from 0.000 F1 in the base model to non-trivial values. Improvements also transfer to held-out Reddit and CMV, which were never seen during training; the largest OOD gain is LLaMA-8B on Reddit (+.171 at 0-shot).
A context-source analysis on CMV shows that restricting demonstrations to the author's prior same-topic interactions yields the strongest performance (Qwen-14B 3-shot: .585 vs. .577 author-all vs. .537 conversation-level), suggesting topic-aligned historical signals provide a useful inductive bias for persuasion prediction.
Ablations
Ablations on all four backbones compare full LiFT against standard SFT on the same data and against variants removing temporal conditioning, curriculum learning, or the history-aware auxiliary loss. On OLMo-7B, removing temporal conditioning causes the largest degradation (Reddit 1-shot: .301→.239; LRS 3-shot: .578→.407), and removing curriculum learning produces comparable drops. SFT provides only modest improvements over the base model and fails to capture longitudinal structure. The history-aware loss shows mixed effects across settings—usually harmful to remove but occasionally neutral or slightly beneficial when removed—which the authors acknowledge as evidence of complex interactions among objectives rather than an uniformly necessary component.
Mechanistic analysis
Three analyses on OLMo-7B (TalkLife) address whether gains arise from better use of temporally ordered history versus generic capacity improvements:
- Probing: layer-wise linear probes show higher label decodability in
hist_mean representations under LiFT, particularly in early-to-mid layers, with reduced decodability at current-token positions—task-relevant information shifts toward history rather than strengthening local classification.
- Attention analysis: LiFT increases normalized attention to recent history bins (small relative-time indices) and shifts prediction-position attention away from instruction scaffolding toward
hist and curr regions.
- Activation patching: shuffling history posts while preserving the
<HIST> span and patching corrupted activations into clean runs harms LiFT substantially more than the base model, peaking around layers 6 and 12—causal evidence that LiFT predictions depend on temporally structured history content.
Together these results support the paper's interpretive claim: LiFT makes historical context more informative, selectively attended, and causally necessary, rather than merely boosting local classification ability.
Limitations and open questions
The mechanistic analyses establish correlation and partial causal dependence on history but do not fully explain all internal mechanisms behind the observed gains. Evaluation is confined to controlled experimental settings with relatively small test sets (e.g., 9 AnnoMI test topics, 100 TalkLife test timelines); no real-world deployment is assessed. The mixed ablation behavior of the history-aware loss leaves its contribution unresolved. Open questions include how the framework scales to longer temporal horizons beyond the ~1K-token context windows used here, whether the curriculum ordering generalizes to other task families, and what evaluation protocols would be required before use in sensitive domains—the authors explicitly caution against clinical application without professional oversight.
Conclusion
LiFT demonstrates that instruction fine-tuning structured around temporal difficulty, few-shot demonstration injection, and explicit temporal–label conditioning improves ICL for longitudinal modelling across four open-weight backbones and five datasets, with the strongest gains on minority change events and out-of-distribution data. Probing, attention, and activation-patching analyses indicate the mechanism is increased reliance on temporally structured history. The framework offers a reproducible recipe for adapting LLMs to trajectory-level prediction, while leaving open the questions of longer horizons, broader curricula, and deployment-grade validation.
