Align-then-Slide: Two-Stage Evaluation

• Align-then-Slide is a two-stage framework that first establishes semantic, temporal, or representational correspondence between data elements and then refines outputs through sliding-window evaluation or visual self-verification.
• It addresses challenges in document-level machine translation, slide generation, video-to-slide alignment, and cross-modal fine-tuning by leveraging dynamic programming and iterative refinement techniques.
• Empirical results indicate improved correlation with human judgments and enhanced processing efficiency, demonstrating the value of intermediate alignment as a robust evaluation and optimization signal.

Across the available literature, Align-then-Slide denotes a two-stage pattern in which an initial alignment step establishes semantic, temporal, or representational correspondence, and a subsequent slide step performs sliding-window evaluation, visual realization, or downstream refinement on top of that alignment. The expression is the formal name of a complete evaluation framework for ultra-long document-level machine translation, while closely related decompositions are used to organize missing-slide generation, lecture video–to–slide alignment, and cross-modal fine-tuning; a further conceptual analogue appears in Direct Feedback Alignment as an alignment phase followed by a memorisation phase (Guo et al., 4 Sep 2025, Xu et al., 21 Feb 2025, Anderer et al., 2024, Shen et al., 2023, Refinetti et al., 2020).

1. Terminological scope and recurring structure

The literature uses the phrase in more than one sense. In document-level machine translation, Align-then-Slide is a named framework with two explicit stages: an Align stage that rebuilds sentence-level correspondence between source and whole-document translation, and an $n$-Chunk Sliding Evaluate stage that scores aligned chunks under 1-, 2-, 3-, and 4-chunk windows (Guo et al., 4 Sep 2025). In missing-slide generation, the same decomposition is used more literally: the system first aligns and generates the semantic content of the missing slide in context, then generates and iteratively refines its visual layout through a textual-to-visual self-verification loop (Xu et al., 21 Feb 2025). In MaViLS, the phrase refers to a multimodal sequence-alignment problem that links lecture video frames to PDF slides via dynamic programming over speech, OCR, and visual similarity (Anderer et al., 2024). In ORCA, the corresponding formulation is align-then-refine: first align a target modality to the pretraining modality’s feature distribution, then fine-tune the pretrained model on the aligned representation (Shen et al., 2023).

Context	Align stage	Second stage
Ultra-long doc-MT	Infer sentence correspondences and rebuild target length	1–4 chunk sliding evaluation
Missing-slide generation	Retrieve sections/figures and generate contextual content	Generate and refine layout visually
Video-to-slide alignment	Build multimodal frame–slide similarity and DP path	Use aligned sequence for downstream interaction
Cross-modal fine-tuning	Learn target embeddings aligned to source modality	Fine-tune pretrained model

This suggests that the common abstraction is not a single algorithm but a family of pipelines organized around an intermediate representation: once correspondence is made explicit, the downstream stage can be optimized with stronger structural constraints than a one-shot system typically provides.

2. Document-level machine translation framework

In the machine-translation setting, the framework addresses a specific evaluation failure mode: LLMs can output fluent whole-document translations whose sentence boundaries do not match the source, so standard sentence-level metrics can mis-score whole-sentence omissions, many-to-one mappings, one-to-many mappings, variable target sentence counts, and reordering across long contexts (Guo et al., 4 Sep 2025). The Align stage therefore begins by segmenting source and target into sentences,

$$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$

constructing a similarity matrix

$score[i][j]=EVAL(src_i,tgt_j),$

and solving a monotonic dynamic-programming problem,

$$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$

Backtracking yields a path that maps each target sentence to a nondecreasing source index. The target is then rebuilt to have exactly the same length as the source: omitted source sentences receive "", and one-to-many mappings are concatenated into a single reconstructed target unit. Many-to-one mappings are deliberately not merged on the source side, because doing so would change source segmentation across systems and impair fair comparison.

The Slide stage evaluates the aligned document at multiple granularities. For each $k \in \{1,2,3,4\}$, the method slides a stride-1 window over the aligned sequences, scores each chunk with a backbone metric such as COMET20, COMET22, or COMETKiwi, averages scores within each chunk size to obtain $M_k$, and then averages over chunk sizes: $M_{\text{ASD}}=\frac{1}{4}\sum_{k=1}^{4} M_k.$ The 1-chunk component remains highly sensitive to omissions, while 2-, 3-, and 4-chunk windows soften the penalty for many-to-one mappings by allowing neighboring source sentences to be evaluated jointly against a shared translation.

Empirically, the framework substantially improves correlation with human judgments. On WMT 2020 Chinese→English, sentence-level COMET20 achieved Pearson $0.679$ and Kendall $0.524$ against MQM ranking, whereas ASD20 achieved Pearson $0.929$ and Kendall $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$0; ASDKiwi reached Pearson $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$1 and Kendall $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$2 (Guo et al., 4 Sep 2025). On a curated real-world test set derived from CommonCrawl, ASD20 again aligned closely with human rankings, with Pearson $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$3 on ZH→EN. The same scores were then used to create preference data for CPO and scalar rewards for GRPO, and both procedures yielded translations preferred over a vanilla SFT baseline. In this domain, Align-then-Slide is therefore both an evaluation framework and an optimization signal.

3. Missing-slide generation and textual-to-visual self-verification

In presentation generation, the formulation arises from a realistic task: given a paper $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$4, a partially existing slide deck $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$5, a missing slide $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$6, the missing slide’s topic $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$7, and the immediately preceding and following slides, generate the missing slide so that its content is coherent with the paper and neighbors and its layout is visually good and consistent with the deck (Xu et al., 21 Feb 2025). Each slide is represented as a set of elements $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$8, separating content $$S=\{src_1,\dots,src_m\}, \qquad T=\{tgt_1,\dots,tgt_n\},$$9 from layout $score[i][j]=EVAL(src_i,tgt_j),$0. The paper states that this mirrors a typical academic workflow: first decide what the slide should say, then decide how it should be laid out.

The alignment stage is a retrieval-augmented content generator. The paper is segmented at section level, embeddings are computed with Salesforce’s SFR-Embedding-Mistral, and top-$score[i][j]=EVAL(src_i,tgt_j),$1 sections are selected by cosine similarity between the topic and each section. Relevant figures and tables are extracted with PDFFigures 2.0, using references in or near the retrieved sections. The content generator LLM then receives retrieved text, recommended figures, the target topic, and neighboring slide content, and performs three tasks: generate concise slide text, select figures or tables, and ensure deck-level coherence by avoiding redundancy and maintaining logical flow.

The slide stage begins with JSON layout generation conditioned on the generated content and neighboring slide layouts. The central novelty is an iterative textual-to-visual self-verification loop. A draft JSON layout is rendered to a slide image with colored bounding boxes and element IDs; a multimodal Reviewer evaluates visual criteria including object overlapping, text overflow, image quality or distortion, element alignment and spacing, text formatting consistency, and overall visual balance; a Refiner then edits the JSON to apply the recommendations; and the process iterates until quality is acceptable. The paper argues that JSON-only self-verification is weak because layout defects are easier to judge visually than from raw coordinates.

The experiments use ACL 2024 In-Person Poster Session 1 papers and slide PDFs, with paper parsing by GROBID, slide conversion to JSON with element-level information, and figure extraction by PDFFigures 2.0. Content quality is measured with ROUGE-1, ROUGE-2, and ROUGE-L, while layout quality is measured by an LLM-as-Judge on alignment, logical flow, coherence and text-visual consistency, visual appeal, and readability. For GPT-4o, ROUGE-L F1 improved from 18.31 in the baseline to 21.97 in Proposed Method (5), and removing neighboring slides reduced GPT-4o ROUGE-1 F1 from 28.39 to 26.13. In layout evaluation, the baseline scored 2.0 on alignment, 3.0 on logic, 3.3 on coherence, 2.0 on visual appeal, and 2.5 on readability; the textual-to-visual self-verification method scored 3.0, 3.8, 3.4, 2.8, and 3.0, respectively. By contrast, JSON-based refinement slightly improved coherence but worsened logic, visual appeal, and readability. In this usage, Align-then-Slide denotes a separation between semantic planning and visual realization, with an explicit multimodal verification loop in the second stage.

4. Video-to-slide alignment as multimodal dynamic programming

MaViLS formulates Align-then-Slide as a frame-level sequence-alignment problem between lecture videos and corresponding slide decks (Anderer et al., 2024). For sampled video frames $score[i][j]=EVAL(src_i,tgt_j),$2 and slides $score[i][j]=EVAL(src_i,tgt_j),$3, it defines a multimodal similarity score $score[i][j]=EVAL(src_i,tgt_j),$4 and uses dynamic programming to find the best cumulative score $score[i][j]=EVAL(src_i,tgt_j),$5, with penalties for slide jumps and an optional linearity prior. The method does not impose hard monotonicity; backward transitions are allowed but penalized more strongly than forward transitions. This design is intended to model realistic teaching behavior, including revisiting previous slides and skipping forward multiple slides.

The benchmark contains 20 lectures, more than 22 hours of video content, and 12,830 distinct video segments with audio transcript segments and aligned slide labels. Most lectures come from MIT OpenCourseWare, with two from the University of Tübingen and one from DeepMind. Ground truth is defined at the sentence level: faster-whisper produces timestamped transcripts, and human raters manually map each sentence to a slide index or to $score[i][j]=EVAL(src_i,tgt_j),$6 when no slide is visible or the slide is ambiguous. The dataset intentionally spans varying video quality, audio quality, camera perspectives, slide density, volatility, and no-slide/slide ratios.

The algorithm builds three similarity matrices. The OCR/text matrix uses Tesseract OCR and sentence-transformers/distiluse-base-multilingual-cased; the audio matrix uses faster-whisper transcripts encoded by the same sentence transformer; and the visual matrix uses MBZUAI/swiftformer-xs. These are combined by mean, max, or weighted sum,

$score[i][j]=EVAL(src_i,tgt_j),$7

and the resulting matrix is decoded by the dynamic program. A moderate jump penalty is beneficial: $score[i][j]=EVAL(src_i,tgt_j),$8 yields the best average F1 of 0.82, whereas $score[i][j]=EVAL(src_i,tgt_j),$9 yields 0.76; $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$0 performs best overall.

The main empirical outcome is that multimodal alignment is both more accurate and more efficient than a SIFT baseline. Mean, max, and weighted multimodal combinations all achieve average F1 0.82; unimodal text reaches 0.76, image 0.64, audio 0.53, and SIFT 0.56. Runtime is approximately 3.5 minutes per lecture for multimodal mean or max combination, 4.6 minutes for weighted combination, and 39.5 minutes for SIFT on a CPU, making MaViLS about 11 times faster than SIFT. The paper further reports that OCR contributes the most to high matching accuracy, followed by image features, while audio transcripts remain useful when OCR data is lacking. The aligned output supports interactive navigation, search, accessibility enhancements, and pedagogical conversational agents.

5. Related align-then-refine formulations in cross-modal transfer

A closely related formulation appears in ORCA, a general cross-modal fine-tuning framework described as an align-then-refine workflow (Shen et al., 2023). The setting assumes a source domain $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$1 in which a transformer was pretrained and a target domain $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$2 with a different input space, label space, and distribution. ORCA decomposes the target model as

$$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$3

where $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$4 is a target embedder mapping arbitrary modality inputs into the sequence space expected by the pretrained transformer $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$5, and $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$6 is a target head. The first stage performs dimensionality alignment and distribution alignment: only the embedder is trained, while the pretrained transformer body remains frozen. The second stage performs full fine-tuning of $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$7 on the target task.

For distribution alignment, the paper compares Euclidean distance, MMD, and Optimal Transport Dataset Distance (OTDD), and reports that OTDD works best. OTDD aligns not only embedded features but also label-conditional geometry by measuring Wasserstein distances between class-conditional distributions in the embedded space. For dense prediction tasks with continuous labels, pseudo-labels are obtained by k-means clustering before computing OTDD. In practice, ORCA uses proxy source datasets rather than original pretraining corpora—CIFAR-10 for Swin and CoNLL-2003 for RoBERTa—and the embedder-learning phase adds on average about 11% of fine-tuning time across NAS-Bench-360 tasks.

The experimental scope is broad: ORCA reports state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforms a wide range of hand-designed, AutoML, general-purpose, and task-specific methods, and demonstrates particular utility in data-limited regimes. On the NAS-Bench-360 tasks, ORCA achieves the lowest error on 7/10 tasks and remains in the top three on all tasks. On the LIFT tabular benchmark, ORCA reaches average accuracy 83.80%, compared with 78.21% for XGBoost and 79.63% for LIFT with GPT-3. The paper’s framing is not terminologically identical to Align-then-Slide, but the structural homology is direct: alignment creates a transferable intermediate representation, and refinement exploits that representation through supervised adaptation.

6. Theoretical analogue, misconceptions, and limitations

A theoretical analogue appears in the analysis of Direct Feedback Alignment, where learning is described as proceeding in two phases: an alignment phase, in which forward weights adapt so that the approximate gradients used by DFA align with true backpropagation gradients, followed by a memorisation phase, in which the network fits the data (Refinetti et al., 2020). In shallow networks, the paper shows analytically that second-layer weights initially grow along the direction of the fixed random feedback vector, producing a plateau structure in the loss and a subsequent specialization phase. In deep linear networks, the analysis introduces alignment matrices whose conditioning determines whether weak weight alignment can become strong enough to support gradient alignment. The same work uses this mechanism to explain why DFA can train state-of-the-art models such as Transformers yet notoriously fails on convolutional networks: convolutional weight sharing imposes structural constraints that prevent generic alignment to random dense feedback matrices.

One recurring misconception is that Align-then-Slide always concerns literal presentation slides. The literature does not support that restriction. In document-level MT, “slide” refers to $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$8-Chunk Sliding Evaluate rather than graphical slides (Guo et al., 4 Sep 2025). In MaViLS, the core contribution is temporal alignment between lecture video frames and PDF slides rather than generation (Anderer et al., 2024). In slide generation, by contrast, the phrase is literal and tied to the decomposition between semantic content planning and visual layout synthesis (Xu et al., 21 Feb 2025).

Across domains, the main limitations cluster around the quality and tractability of the alignment stage. In document-level MT, the similarity matrix and dynamic-programming alignment require $$dp[i][j]=\max_{0\le i'\le i} dp[i'][j-1] + score[i][j].$$9 time and memory, and the monotonic alignment assumption can degrade under extreme reordering (Guo et al., 4 Sep 2025). In slide generation, poor initial layouts, limited design sophistication, and restricted dataset diversity remain open problems (Xu et al., 21 Feb 2025). In MaViLS, high volatility, long no-slide segments, variable recording quality, and discrepancies between PDF slides and the slides actually shown in video continue to challenge alignment (Anderer et al., 2024). In ORCA, extreme domain gaps, structural mismatches between modality and embedder, and pretrained-model selection remain practical constraints (Shen et al., 2023). This suggests that the effectiveness of Align-then-Slide pipelines depends less on the nominal two-stage decomposition than on whether the intermediate alignment is stable, information-preserving, and computationally tractable.