Historical Trajectory Retriever (HTR)

• Historical Trajectory Retriever (HTR) is a framework that integrates deep neural networks, transformer models, and deformable convolutions to retrieve and model diverse trajectory data.
• It leverages advanced techniques like CRNNs, transfer learning, and ensemble strategies to overcome annotation scarcity, variability, and degradation in data quality.
• HTR applications span historical document digitization, autonomous navigation, and agent behavior analysis, achieving significant reductions in error rates and improved recall metrics.

The Historical Trajectory Retriever (HTR) encompasses a range of architectures and computational paradigms for retrieving, modeling, or decoding trajectory data in diverse domains, notably historical handwritten text recognition, path prediction, route optimization, document digitization, and agent behavior retrieval. The terminology HTR has evolved and diversified, covering both string-valued "trajectory" (as in written text or agent actions) and spatial or semantic trajectories (such as motion paths in autonomous driving or POI recommendation). Modern HTR solutions employ deep neural networks, transformer-based architectures, retrieval-augmented LLM pipelines, and specialized matching algorithms. Approaches consistently emphasize robustness to annotation scarcity and trajectory variability—both foundational in historical and behavioral data modeling.

1. Foundational Models in Historical Handwriting Recognition

Pioneering HTR work focused on overcoming the manual annotation bottleneck in historical document transcription (Chammas et al., 2018). In the context of offline handwriting recognition, an HTR system is typically instantiated as a deep convolutional recurrent neural network (CRNN) comprising 13 convolutional layers (VGG16-inspired, 3×3 filters, ReLU, batch normalization), followed by three bidirectional LSTM layers (256 units with peephole connections), and trained end-to-end with Connectionist Temporal Classification (CTC) loss. Feature vectors of length 1024 are extracted via a sliding window over normalized text-line images.

Incremental training paradigms leverage a small seed of labeled lines (typically 10%) to bootstrap recognition, apply automatic segmentation (baseline and contour detection), and align recognized outputs with paragraph-level ground-truth using the Levenshtein distance (edit distance ≤ 50% line length). A match criterion thus admits automatically segmented "pseudo-labeled" lines, which are then incorporated into retraining—yielding reductions in label error rates (LER) from 9.2% to 7.4%. Data augmentation techniques address variability in writing scale, employing Jenks natural breaks for vertical scale categorization and synthetic scaling to bolster minority classes; this multiscale augmentation further reduces LER by 12%. Model-based normalization (image scaling with factors spanning [0.7, 1.3], voting via ROVER) yields an additional relative improvement of 14% in word error rate (WER).

HTR approaches grounded in these methods demonstrate that robust document transcription can be achieved with limited granular annotation, by iteratively expanding training sets, simulating writing variability, and normalizing scale during inference.

2. Transfer Learning, Augmentation, and Error Mitigation in HTR

Transfer learning is pivotal for historical text recognition under data scarcity (Aradillas et al., 2020). Typical strategies involve pretraining a CRNN (CNN+BLSTM+CTC) on large annotated datasets (IAM, ICFHR18-G), with fine-tuning on small target corpora. Empirical analysis suggests that fixing only the earliest CNN layer and fine-tuning all subsequent layers yields optimal performance—attributable to the generality of low-level features and the domain specificity required for higher-level representations.

Data augmentation, essential for mitigating overfitting on scarce data, usually employs affine transformations and random warp grid distortions. When merged with transfer learning, the recommended scheme is DA-TL (augmentation on source only), since DA-TL-DA (target set augmentation) can induce overfitting and marginally increase CER. Purging annotation errors is facilitated by Corrupted Labels Purging (CLP), which partitions the target set, fine-tunes excluding each subset, computes CER, and discards samples exceeding an error threshold (ε, e.g., 50%). CLP enhances performance by systematically filtering mislabeled samples and is substantiated by pronounced CER reductions on error-prone datasets.

These layered approaches—transfer learning, restrictive augmentation, cross-validation error purging—collectively reduce CER by up to 6% in challenging historical contexts.

3. Transformer-Based HTR and Ensemble Strategies

Recent advances exploit transformer-based architectures, notably TrOCR and vision-LLMs for historical text sequence modeling (Ströbel et al., 2022, Meoded, 15 Aug 2025). TrOCR integrates a vision transformer (BEiT or ViT, Conv2d patch embedding) with an XLM-RoBERTa decoder for multilingual transcription. Fine-tuning is feasible even on Latin text for which the pretrained model has no prior exposure; transfer is enabled by architectural invariance to alphabet and generalizable image representations.

Preprocessing steps (segmentation via PAGE-XML, binarization, background normalization, dimension alignment to IAM standards) are crucial in adapting historical input domains to pretrained pipelines. Advanced augmentation (random rotation, elastic distortion, perspective/shear transformations) simulates manuscript degradation and handwriting variability.

Model ensembles (top-5 beam voting across multiple augmentation variants) yield substantial error reductions—on the Gwalther dataset, CER improves from 1.86 (best augmented single model) to 1.60 (ensemble), representing a 50% decrease over baseline TrOCR and 42% over prior SOTA. This demonstrates the effectiveness of domain-specific augmentation and ensemble approaches in historical digitization.

4. Deformable Convolutional Networks in Trajectory and Text Retrieval

Addressing geometric variability in historical handwriting and trajectories, deformable convolutional networks dynamically adapt kernel sampling grids by learning content-dependent offsets (Cascianelli et al., 2022). This mechanism is formalized as

$$(I \circ k)(p) = \sum_{d \in \mathcal{N}} k(d) \cdot I(p + d + \delta(d))$$

where $\delta(d)$ encodes spatial offsets and bilinear interpolation $B(s, p + d + \delta(d))$ averages neighboring points.

DefConv CRNN and DefConv 1D-LSTM architectures systematically outperform standard convolution-based HTR on both modern and historical datasets; on IAM, CER decreases from 7.8 to 6.8, on ICFHR14 from 3.9 to 3.6. These deformable variants focus receptive fields on handwritten strokes and localize irregular formats, boosting robustness to page degradation and layout variability—properties essential for effective historical trajectory retrieval.

5. History-Aware Retrieval and Trajectory Modeling in Non-Text Domains

HTR paradigms have expanded to non-textual domains, encompassing motion prediction, object tracking, and agent trajectory modeling. In multiple object tracking, history-aware transformations re-project appearance features based on past trajectory statistics using Fisher Linear Discriminant (FLD) optimization (Gao et al., 16 Mar 2025):

$$J(W) = \mathrm{tr} \left\{ (W^\top S_W W)^{-1} (W^\top S_B W) \right\}$$

Here, historical trajectory features serve as conditional cues, enhancing discrimination among similar targets within the same video. Temporal-shifted centroids and fusion of transformed/original spaces via weighted cosine similarity further refine association accuracy. This training-free projection yields robust improvements and zero-shot generalization across MOT benchmarks.

Universal multimodal trajectory retrieval for GUI agents (Zhang et al., 27 Jun 2025) models state–action sequences from deterministic MDPs,

$$\tau = (s_1, a_1, s_2, a_2, ..., s_n, a_n),\qquad s_{i+1} = \mathcal{T}(s_i, a_i)$$

paired with unified vision-language retrieval (GAE-Retriever). Token selection and GradCache mechanisms enable scalable contrastive training across massive high-resolution trajectories, achieving Recall@1 boosts of up to 12.9 points over strong VLM baselines.

In POI recommendation, the HTR is formalized as a TF-IDF-based retrieval of semantically similar historical user trajectories (Li et al., 21 Sep 2025),

$$\operatorname{sim}(v_q, v_i) = \frac{v_q \cdot v_i}{\| v_q \| \| v_i \|}$$

$$I^* = \operatorname*{argmax}_{I: |I| = k} \sum_{i \in I} \operatorname{sim}(v_q, v_i)$$

which supplies context to LLMs, followed by spatial reranking (DWDTW) and agentic rectification.

6. Performance Metrics and Evaluation

Performance measurement in HTR encompasses task-specific metrics:

• For handwriting recognition: Character Error Rate (CER), Word Error Rate (WER), and Label Error Rate (LER) (Chammas et al., 2018, Aradillas et al., 2020, Meoded, 15 Aug 2025).
• In trajectory modeling and retrieval: Recall@K for semantic and subtrajectory matches (Zhang et al., 27 Jun 2025).
• In object tracking: HOTA, IDF1, association accuracy (Gao et al., 16 Mar 2025).
• In route optimization and POI recommendation: hit ratio (HR), NDCG, mean absolute error of predicted travel times (Siampou et al., 2 Nov 2024, Li et al., 21 Sep 2025).

Empirical results demonstrate that transfer learning, augmentation, deformable architectures, and retrieval-based paradigms consistently reduce error rates or improve recall across diverse datasets.

7. Implications, Applications, and Future Directions

HTR systems have direct implications for archival digitization, autonomous agent planning, information retrieval, navigation, and behavioral modeling. Bootstrapping approaches enable robust performance with limited manual annotation, while augmentation and normalization account for historical variability. Recent transformer-based and vision-language ensemble strategies drive SOTA in manuscript transcription.

Trajectory retrieval frameworks support agent-centric planning, demonstration-based context learning, and real-world navigation—leveraging example-based retrieval in LLMs, historical route continuity, and multimodal embeddings. The emergence of history-aware feature transformations, scalable retrieval pipelines, and collaborative platforms (e.g., Transkribus, HTR-United) contributes to data sharing, methodology innovation, and accelerated progress in historical and behavioral data analysis.

A plausible implication is that continued integration of custom augmentation, scalable retrieval, and context-aware learning modules will further advance both transcription accuracy and generalization in heterogeneous trajectory domains. Adaptations to new architectures (e.g., transformer-based deformable blocks, adaptive retrieval mechanisms) and joint modeling of appearance, motion, and semantic trajectories remain promising avenues for further research and deployment.