Segment Transformer: Design & Applications

• Segment Transformer is a design paradigm that organizes transformer computations around explicit segment units to enhance structural fidelity and reduce computational complexity.
• It applies hierarchical and segment-aware attention mechanisms, achieving impressive metrics such as up to 53.63 mIoU in semantic segmentation tasks.
• The approach enables efficient processing of long sequences in text, audio, and bio-data by aligning model computation with natural data segmentation.

Searching arXiv for recent and foundational papers on “Segment Transformer” to ground the article in published work. “Segment Transformer” is not a single canonical architecture but a family of Transformer-based designs in which segmentation, segments, or segment-aware structure become the primary inductive bias. Across the literature, the term has been used in several distinct senses: as a pure Transformer for semantic segmentation in vision, as a hierarchical transformer over textual or audio segments, as a segment-aware positional encoding scheme for language modeling, and as a segment-level sequence model in speech, retrieval, time series, and biosequence analysis. What unifies these uses is a shift away from undifferentiated token streams toward explicit intermediate units—image patches grouped into semantic masks, sentences grouped into topical spans, music divided into four-bar regions, proteins partitioned into sequence segments, or long documents decomposed into retrieval segments—so that Transformer computation can capture structure at a more appropriate granularity (Strudel et al., 2021, Lo et al., 2021, Bai et al., 2020, Kim et al., 10 Sep 2025, Zhang et al., 26 Jul 2025, Chen et al., 2022).

1. Scope and meanings of the term

The phrase “Segment Transformer” has been used in at least four major technical senses. In semantic segmentation, it denotes models that adapt Transformer encoders and decoders to dense prediction, either directly over image patches as in “Segmenter” (Strudel et al., 2021) or through hierarchical multi-scale designs such as SSformer (Shi et al., 2022), Transformer Scale Gate (Shi et al., 2022), Superpixel Transformers (Zhu et al., 2023), and SegMaFormer (Nguyen et al., 23 Mar 2026). In text and document processing, it denotes architectures that either predict segment boundaries explicitly, as in Transformer$^2$ for supervised text segmentation (Lo et al., 2021), or encode long documents via interacting segments for retrieval, as in SeDR (Chen et al., 2022), or choose query-relevant segments for training, as in BeST (Kim et al., 2021). In audio and speech, it denotes segment-level modeling of long sequences, including s-Transformer for long-form text-to-speech (Wang et al., 2020), Segment Transformer for AI-generated music detection (Kim et al., 10 Sep 2025), and Fusion Segment Transformer for full-audio detection via structure-aware fusion (Kim et al., 20 Jan 2026). In other domains, the term appears in line segment detection with LETR (Xu et al., 2021), segment-selective generation for hashtags (Mao et al., 2021), segment attention for time series forecasting in PSformer (Wang et al., 2024), and segment-level protein modeling for enzyme temperature stability (Zhang et al., 26 Jul 2025).

These usages are related but not interchangeable. Some papers treat “segment” as a latent semantic or structural unit, such as contiguous text spans or musically meaningful bars (Lo et al., 2021, Kim et al., 10 Sep 2025). Others define segments operationally as fixed-length chunks, passage windows, or position-aligned patches (Mao et al., 2021, Kim et al., 2021, Wang et al., 2024). In vision, “segment transformer” may refer either to semantic segmentation by Transformers (Strudel et al., 2021, Shi et al., 2022) or, in a more literal sense, to Transformers that predict geometric line segments directly (Xu et al., 2021). This suggests that the most precise encyclopedic reading of the term is not a single architecture but a design pattern: Transformer computation organized around explicit segment structure.

2. Segment as the primary computational unit

A defining property of segment-transformer models is that they move the principal representational unit away from the raw atomic token. In “Segmenter,” an image is split into non-overlapping patches, encoded by a Vision Transformer, and decoded into per-class patch masks that are then reshaped and upsampled to dense predictions (Strudel et al., 2021). In Superpixel Transformers, the model first decomposes the image into a smaller set of learned soft superpixels, applies multi-head self-attention in superpixel space, and projects class predictions back to pixels through learned associations (Zhu et al., 2023). In Transformer$^2$, a document is represented as a sequence of sentence embeddings produced by pre-trained transformers, and an upper Transformer reasons over sentence-level units to predict segment boundaries and topic labels jointly (Lo et al., 2021). In Segment Transformer for music, the model uses beat-aware four-bar segmentation and learns inter-segment relationships at the track level (Kim et al., 10 Sep 2025), while Fusion Segment Transformer extends this with dual streams over content embeddings and self-similarity structure (Kim et al., 20 Jan 2026).

This design usually serves one of two purposes. The first is structural fidelity: the segment unit corresponds more closely to the phenomenon of interest than a flat token stream does. Segmenting music into four-bar downbeat-aligned regions is intended to align the model with phrase-level musical structure (Kim et al., 10 Sep 2025), and Segment Transformer for enzyme stability is motivated by the claim that different contiguous sequence regions contribute unequally to thermal behavior (Zhang et al., 26 Jul 2025). The second is computational reduction: a sequence of segments is shorter than a sequence of raw tokens or pixels, making global context modeling more tractable. Superpixel Transformers reduce dense pixel space to a much smaller segment space before applying global self-attention (Zhu et al., 2023), while s-Transformer for speech processes aligned chunks with cached memory rather than one extremely long text-to-spectrogram sequence (Wang et al., 2020).

A recurring implication is that the Transformer is not discarded but relocated. Instead of attending over the entire raw sequence at full resolution, the model first constructs segment-level representations, then applies Transformer reasoning where it is most informative or affordable. This suggests that segment transformers are often best understood as hierarchical Transformers even when that label is not used explicitly.

3. Architectural patterns

Several recurrent architectural templates appear across the literature.

The first is the hierarchical two-stage encoder, in which a lower module computes local or unit-level embeddings and an upper Transformer models relationships among those units. Transformer$^2$ exemplifies this pattern: bottom-level sentence encoders based on fixed pre-trained transformers yield sentence vectors, and a five-layer upper Transformer with 24 self-attention heads and feed-forward size 1024 predicts binary segmentation labels and topic labels over a document truncated or padded to 150 sentences (Lo et al., 2021). The original and fusion music Segment Transformer papers use the same broad pattern: a short-segment feature extractor first encodes local musical content, and a second-stage Transformer reasons over the sequence of segment embeddings for full-audio classification (Kim et al., 10 Sep 2025, Kim et al., 20 Jan 2026). Segment Transformer for enzyme stability also follows a multi-stage hierarchy: ESM-2 residue embeddings are converted into segment-level features, processed by Dual Grouped Segment Attention, then pooled for scalar regression (Zhang et al., 26 Jul 2025).

The second pattern is segment-local attention plus global recurrence or interaction. s-Transformer processes speech in aligned segments and reuses cached hidden states as memory in both encoder and decoder self-attention, while keeping encoder-decoder attention local to the current segment pair (Wang et al., 2020). SeDR encodes long documents as segments, but injects cross-segment interaction via other segments’ [CLS] tokens at every Transformer layer so that each segment representation is document-aware and segment-sensitive (Chen et al., 2022). SRformer replaces full decoder cross-attention with segmented local attention plus recurrent attention over the complementary segments, using Recurrent Accumulate-and-Fire neurons to summarize the nonlocal part (Long et al., 2023).

The third pattern is segment-aware positional or structural encoding rather than explicit segment-level tokenization. Segatron defines each token position as a tuple $\mathbf{I}=\{t_i,s_i,p_i\}$ of token-in-sentence, sentence-in-paragraph, and paragraph-in-document indices, replacing the flat positional index in Transformer-XL and BERT-style models with structured position encodings (Bai et al., 2020). Transformer Scale Gate uses encoder self-attention and decoder cross-attention maps to infer which scale is most useful for each image patch in semantic segmentation, effectively performing per-patch scale selection rather than unconditional multi-scale fusion (Shi et al., 2022). These models make the Transformer segment-aware without making segments the only tokens.

The fourth pattern is direct set prediction of segments as output objects. LETR treats line segment detection as a direct set prediction problem with 1000 learned line entities, a coarse-to-fine two-scale encoder-decoder, Hungarian matching, and direct endpoint distance loss (Xu et al., 2021). Here the “segment transformer” phrase is literal: the model predicts a set of line segments directly rather than recovering them from edges or junctions.

4. Representative domain instantiations

4.1 Semantic segmentation in vision

“Segmenter” is among the clearest uses of the term in vision. It encodes image patches with a ViT and decodes them either with a point-wise linear decoder or a mask transformer using learned class embeddings (Strudel et al., 2021). The model is trained with standard pixel-wise cross-entropy and shows that segmentation can be cast as dense prediction from Transformer patch embeddings without convolutional encoders or decoders. On ADE20K validation, performance improved with larger models and smaller patch sizes; for example, Seg-B/32, Seg-B/16, and Seg-B/8 achieved 43.07, 48.06, and 49.54 mIoU respectively, while Seg-L-Mask/16 reached 53.63 multi-scale mIoU (Strudel et al., 2021).

Subsequent work diversified this template. SSformer retains a Swin Transformer encoder but replaces heavy decoders with a lightweight all-MLP multi-level fusion head, yielding 47.71 mIoU on ADE20K with 87.5M parameters and 91.01G FLOPs, versus 50.31 mIoU and 297.2G FLOPs for a heavier Swin-T setup in the reported comparison (Shi et al., 2022). Transformer Scale Gate inserts a lightweight plug-in scale-selection module into hierarchical vision-transformer segmenters and reports consistent gains on Pascal Context and ADE20K, such as 50.2 to 54.5 mIoU for Swin-Tiny on Pascal Context and 44.4 to 47.5 on ADE20K (Shi et al., 2022). Superpixel Transformers compress dense images into learned superpixel tokens, perform global self-attention there, and project predictions back to pixels, achieving 80.4 mIoU on Cityscapes with 29M parameters and 15.3 FPS using a ResNet-50 backbone (Zhu et al., 2023). SegMaFormer adapts the SegFormer-style philosophy to 3D volumetric segmentation by using Mamba in early high-resolution stages and Transformer attention only later, reaching 83.79 average Dice on BraTS and 91.11 on ACDC with 2.02M parameters (Nguyen et al., 23 Mar 2026).

This body of work shows two distinct conceptions of “segment transformer” in vision: one in which the task is semantic segmentation and the Transformer predicts segment labels over image regions (Strudel et al., 2021), and another in which explicit intermediate image segments—superpixels, scales, or multiresolution features—mediate the Transformer computation (Shi et al., 2022, Zhu et al., 2023).

4.2 Text segmentation and long-document modeling

Transformer$^2$ is a direct segment transformer for text segmentation. It models a document as a sequence of sentences $$\boldsymbol{S}=\langle \boldsymbol{s}_1,\ldots,\boldsymbol{s}_I\rangle$$, builds sentence embeddings from [CLS] outputs of pre-trained transformers, concatenates single-sentence and pairwise-sentence embeddings, and uses an upper Transformer to predict per-sentence binary segmentation labels and topic labels with the multitask objective $L=L_{\text{seg}}+L_{\text{topic}}$ (Lo et al., 2021). The model is trained on WikiSection and evaluated with $P_k$, using topic supervision to encourage within-segment topic coherence (Lo et al., 2021). It shows that a segment transformer for text need not operate on full documents end-to-end; instead, a transformer over sentence embeddings can capture discourse transitions efficiently.

Long-document retrieval provides a different variant. BeST does not build a new segment interaction architecture, but treats segment selection as a latent supervision problem: instead of always training on the first segment, it iteratively selects the query-related segment $i_{q,d}=\arg\max_i f_\theta(q,d_i)$ for each document and trains a BERT ranker on these selected segments (Kim et al., 2021). SeDR goes further by learning segment representations jointly: documents are split into segments of 512 tokens, each segment is wrapped with [CLS]/[SEP], and a Segment-Interaction Transformer lets tokens in one segment attend to local tokens plus [CLS] summaries from other segments (Chen et al., 2022). A document is then represented by a variable number of segment vectors, and relevance is computed by max-pooling over query-segment similarities. On MS MARCO and TREC-DL, SeDR improved over independent MaxP-like baselines while keeping latency at 2.7 ms and parameter count at 125M in the reported comparison (Chen et al., 2022).

These models correct a common misconception: segment-based processing of long documents is not merely a workaround for input limits. In the strongest versions, segment structure becomes an explicit modeling choice that shapes supervision, cross-segment interaction, and the granularity of relevance or segmentation decisions (Lo et al., 2021, Chen et al., 2022).

4.3 Audio and speech

In speech synthesis, s-Transformer addresses long-form TTS by splitting an utterance into aligned text and mel-spectrogram segments, reusing cached hidden states as memory for both encoder and decoder self-attention, and keeping encoder-decoder attention local to the current segment (Wang et al., 2020). It also introduces a modified relative positional self-attention bias to generalize to sequence lengths possibly unseen in training (Wang et al., 2020). On extra-long sentences, it improved MOS from 3.79 to 3.99 over the baseline, with equal 4.29 MOS on short sentences and similar 4.20 versus 4.22 on long sentences (Wang et al., 2020). Here the segment transformer idea is tightly linked to recurrence and local alignment.

In music detection, the original Segment Transformer and Fusion Segment Transformer operationalize music structure through beat-aware segmentation. The earlier model divides songs into four-bar downbeat-aligned segments, extracts segment embeddings using stage-1 short-audio encoders such as Wav2vec 2.0, Music2vec, MERT, or FXencoder, and feeds the resulting sequence to a dual-pathway transformer that models both segment content and a self-similarity matrix over segments (Kim et al., 10 Sep 2025). On SONICS full-audio detection, the reported MERT-based Segment Transformer achieved ACC 0.9992, F1 0.9992, and AUC 0.9999 (Kim et al., 10 Sep 2025). Fusion Segment Transformer retains the two-stage idea but replaces simple late concatenation with bi-directional cross-attention between the content stream $X_{EMB}$ and the self-similarity stream $^2$0, followed by a gated fusion layer: $^2$1 It reports ACC 0.9867, F1 0.9868, and AUC 0.9995 on AIME with MERT, slightly improving over the previous Segment Transformer (Kim et al., 20 Jan 2026).

A notable point across these papers is that segment selection is musically grounded rather than arbitrary. In Fusion Segment Transformer, replacing four-bar downbeat-aligned segmentation with fixed-length windows reduced validation accuracy on AIME from 0.9867 to 0.966 (Kim et al., 20 Jan 2026). This suggests that in some domains, the value of a segment transformer depends critically on whether the segmentation unit corresponds to natural structure.

4.4 Time series and biological sequences

PSformer defines a time-series segment as the concatenation of same-position patches from all variables, transforming $^2$2 into $^2$3, then applying Spatial-Temporal Segment Attention and aggressive parameter sharing within the encoder (Wang et al., 2024). Attention is computed over the segment-derived representation rather than ordinary time tokens, and the final forecast is produced by reshaping back to $^2$4 and projecting to horizon $^2$5 via $^2$6 (Wang et al., 2024). The model is accurate and unusually parameter-efficient; for ETTh1 at horizon 96, the full model has 52,416 parameters, with only 3,168 in the encoder according to the appendix figures summarized in the data (Wang et al., 2024).

Segment Transformer for enzyme temperature stability likewise replaces residue-level processing with segment-level modeling. It starts from ESM-2 residue embeddings, generates multi-scale segment features by downsampling and fixed-length segmentation, applies Dual Grouped Segment Attention to capture short- and long-range relations among segments, and pools them with attention for scalar regression (Zhang et al., 26 Jul 2025). Trained with a weighted RMSE objective to address label imbalance, it reaches RMSE 24.03, MAE 18.09, and Pearson and Spearman correlations of 0.33 on the curated enzyme dataset (Zhang et al., 26 Jul 2025). Its engineering case study on cutinase further reports a selected mutation A78E yielding a 1.64-fold increase in relative activity after heat treatment and a 3.9-fold increase in half-life at 60°C (Zhang et al., 26 Jul 2025). This suggests that segment transformers can be useful not only for prediction but for interpretable region prioritization in scientific design tasks.

5. Core mechanisms

Several mechanisms recur across segment transformer designs.

Segment construction is the first. Segments may be contiguous fixed-length blocks (Mao et al., 2021), sentence sequences with padding (Lo et al., 2021), same-position multivariate patches (Wang et al., 2024), four-bar downbeat-aligned music units (Kim et al., 10 Sep 2025), or learned superpixels (Zhu et al., 2023). The choice is usually domain-specific and often decisive for performance. When segment boundaries correspond to meaningful structure, gains tend to be larger; when segments are merely convenient windows, models often need additional mechanisms to compensate.

Inter-segment interaction is the second. Some models use a document-level upper Transformer over segment embeddings (Lo et al., 2021), some exchange information through [CLS]-mediated bridges (Chen et al., 2022), and some approximate missing cross-segment context with recurrent accumulation (Wang et al., 2020, Long et al., 2023). Fusion Segment Transformer performs explicit cross-attention between content and structural views of the segment sequence (Kim et al., 20 Jan 2026). This suggests that segment transformers usually work best when segmentation is paired with some nontrivial mechanism for recombining information across segments.

Auxiliary structure signals form the third mechanism. Transformer$^2$7 adds topic-label prediction as auxiliary supervision (Lo et al., 2021). Segatron injects paragraph, sentence, and token coordinates into the attention machinery (Bai et al., 2020). Transformer Scale Gate derives per-patch scale gates from self-attention and cross-attention cues (Shi et al., 2022). Segment Transformer for music augments content tokens with self-similarity structure (Kim et al., 10 Sep 2025), and the fusion variant strengthens this relation through bidirectional attention (Kim et al., 20 Jan 2026). Such signals often replace or complement explicit coherence losses.

Efficiency-oriented compression is the fourth. Segmenter works on image patches instead of pixels (Strudel et al., 2021). Superpixel Transformers reason globally in a space that is often $^2$8 times smaller than the input resolution (Zhu et al., 2023). s-Transformer limits attention to segment length plus memory rather than full utterance length (Wang et al., 2020). SeDR holds the complexity of split-and-pool while enriching segments with document context (Chen et al., 2022). SegMaFormer uses Mamba in early stages specifically because sequence length is largest there (Nguyen et al., 23 Mar 2026). A plausible implication is that “segment transformer” is as much an efficiency strategy as a representational one.

6. Evaluation regimes and empirical tendencies

Across domains, segment transformers are usually evaluated against either flat Transformers or local baselines that ignore segment structure. The empirical pattern is consistent though nuanced.

In semantic segmentation, Transformer-based segmenters outperform or match strong CNN baselines when sufficient pretraining and appropriate decoder design are available, but the best efficiency-accuracy tradeoff often comes from hierarchical or segment-compressed variants rather than flat global attention (Strudel et al., 2021, Shi et al., 2022, Zhu et al., 2023). Superpixel Transformers are especially notable for shifting global reasoning away from dense pixel space while maintaining competitive mIoU (Zhu et al., 2023).

In text segmentation and retrieval, segment-aware models usually beat heuristically selected or independently encoded segments. Transformer$^2$9 reports that both single and pairwise pre-trained knowledge contribute in most cases, though the gain from single-sentence embeddings is dataset-dependent and sometimes negative (Lo et al., 2021). BeST shows that a standard BERT ranker trained on query-selected segments can match or nearly match more sophisticated long-input architectures (Kim et al., 2021). SeDR shows that context-aware segment embeddings outperform independent MaxP-like baselines with similar latency (Chen et al., 2022).

In audio and speech, segment transformers tend to matter most on long-form tasks. s-Transformer matches baseline TTS quality on short utterances but improves markedly on extra-long ones (Wang et al., 2020). Segment Transformer and Fusion Segment Transformer target full-audio detection rather than short clips specifically because long-form structure is assumed to differentiate human and AI-generated music more clearly (Kim et al., 10 Sep 2025, Kim et al., 20 Jan 2026).

In biology and time series, segment-centric formulations appear most helpful when local motifs and medium-range structure matter more than isolated positions. Segment Transformer for enzyme stability explicitly argues that segment-level representations improve predictive performance relative to amino-acid-level features (Zhang et al., 26 Jul 2025), and PSformer’s ablations support the importance of its segment attention beyond simple patching (Wang et al., 2024).

A recurring caveat is that segment transformers do not always dominate flat models universally. Transformer$^2$0 notes that combining single and pairwise sentence embeddings is not always beneficial (Lo et al., 2021). Fusion Segment Transformer reports only modest statistical margin over its predecessor with $^2$1 (Kim et al., 20 Jan 2026). SegT does not outperform the strongest baselines on every unseen polyp dataset (Chen et al., 2023). This suggests that segment structure is helpful when it matches the data-generating process, but not automatically sufficient.

7. Misconceptions and conceptual distinctions

One common misconception is that any Transformer that processes chunks is a segment transformer. The literature draws finer distinctions. BeST is better described as a query-driven segment supervision method than as a new segment-transformer architecture (Kim et al., 2021). Attend-and-Select for hashtag generation is a Transformer with a segment-selection bottleneck, not a model in which segments replace tokens everywhere (Mao et al., 2021). DocSegTr is a transformer-based instance segmentation model for documents, but its relation to “segment transformer” is through segmentation as output, not through segment-native computation in the same sense as Transformer$^2$2 or Segment Transformer for music (Biswas et al., 2022).

A second misconception is that segment transformers are necessarily hierarchical in the classic two-level sense. Segatron instead changes positional encoding so that a flat token Transformer becomes segment-aware (Bai et al., 2020). Transformer Scale Gate is a plug-in scale-selection module, not a standalone segment transformer (Shi et al., 2022). Thus, segment awareness may enter through tokenization, through attention routing, through positional coordinates, or through decoder design.

A third misconception is that segment transformers always reduce complexity. Often they do, but not always in a simple asymptotic way. PSformer’s attention scales over a segment-derived axis $^2$3, which can be advantageous or not depending on variable count (Wang et al., 2024). SeDR increases index size relative to single-vector retrieval even while improving long-document modeling (Chen et al., 2022). Fusion Segment Transformer adds a second stream and gated fusion beyond the original Segment Transformer (Kim et al., 20 Jan 2026). In some cases, segmentation trades one form of complexity for another.

8. Limitations and open directions

Several limitations recur across the literature. Boundary quality and small-structure sensitivity remain difficult in segmentation. Segmenter benefits from smaller patch sizes but pays a severe computational cost (Strudel et al., 2021). SegMaFormer is relatively weaker on small organs and sharp anatomical boundaries (Nguyen et al., 23 Mar 2026). SegT introduces explicit edge priors partly because a transformer encoder alone is insufficient for ambiguous medical boundaries (Chen et al., 2023).

Dependence on segmentation quality is another issue. Segment Transformers for music depend on beat/downbeat tracking (Kim et al., 10 Sep 2025, Kim et al., 20 Jan 2026). s-Transformer relies on alignment-aware chunking in TTS (Wang et al., 2020). Segment-aware LLMs such as Segatron depend on paragraph and sentence segmentation during preprocessing (Bai et al., 2020). This suggests that segment transformers are often only as good as the segmentation pipeline or structural annotations that feed them.

Coverage versus granularity is a third tension. Segment-level modeling can improve robustness and efficiency, but may blunt sensitivity to fine local effects. Segment Transformer for enzyme stability explicitly notes reduced sensitivity to single-residue mutation effects as a tradeoff of segment-level modeling (Zhang et al., 26 Jul 2025). Superpixel Transformers show that too few or too many superpixels both hurt performance (Zhu et al., 2023). Transformer$^2$4 finds that richer bottom-level embeddings do not always help (Lo et al., 2021). Determining the “right” segment granularity remains a central unresolved problem.

Universal versus domain-specific segment definitions form a fourth limitation. Some domains have natural segments, such as bars in music or sentences in text (Lo et al., 2021, Kim et al., 10 Sep 2025). Others require learned soft assignments, as with superpixels (Zhu et al., 2023), or heuristic fixed blocks, as in hashtag generation (Mao et al., 2021). A plausible implication is that future segment transformers may need adaptive or task-conditioned segmentation rather than fixed boundaries.

9. Synthesis

Across arXiv, “Segment Transformer” denotes a broad research program rather than a single model family. The common principle is to make segment structure explicit in Transformer computation: by reasoning over sentence embeddings for text segmentation (Lo et al., 2021), injecting paragraph-sentence-token coordinates into positional encoding (Bai et al., 2020), predicting semantic masks from patch tokens in vision (Strudel et al., 2021), building global context over learned superpixels (Zhu et al., 2023), modeling long speech through segment recurrence (Wang et al., 2020), analyzing music via inter-segment structure (Kim et al., 10 Sep 2025), or aggregating sequence regions for protein thermal prediction (Zhang et al., 26 Jul 2025). In many cases, the segment serves simultaneously as a better inductive bias and a more economical computational unit.

The most general lesson of this literature is that Transformer performance often improves when the model is not forced to discover all structure from a flat token stream. Segment transformers impose an intermediate level of organization—region, span, passage, patch group, phrase, or bar—and then let self-attention operate where that organization is most informative. This suggests that the enduring significance of the segment-transformer idea lies not in any one implementation, but in a methodological shift: Transformer architectures become substantially more effective when segmentation is treated as part of representation design rather than merely as preprocessing.