Boundary-Centric Pretraining

• Boundary-centric pretraining is a design principle that organizes representation learning by emphasizing essential discontinuities such as compositional, lexical, and spatial boundaries.
• It employs targeted objectives—from auxiliary segmentation tasks in text to boundary-forced masking in vision—to steer models toward more predictive transitions.
• Empirical results across multiple domains indicate that emphasizing boundary interfaces improves compositional transfer, multilingual alignment, and dense perception.

Boundary-centric pretraining is a family of pretraining designs in which a boundary is treated as a first-class organizing variable for representation learning. The relevant boundary may be a partition in the pretraining distribution, such as whether an entity ever appears in a compositional context; an initialization boundary, such as whether multilingual alignment is established before large-scale language modeling; a latent segmentation boundary, such as word or phrase spans in Chinese or multilingual sequence labeling; a document boundary across which contextual dependence can be made predictive; a temporal event boundary in video; or a spatial and geometric discontinuity in images, medical scans, and CAD boundary representations. Taken together, these works suggest that pretraining often fails not because information is absent, but because the model was not trained to use that information across the relevant boundary (Karmim et al., 8 Jun 2026, Li et al., 2024, Zhang et al., 2024, Fu et al., 6 Jul 2026).

1. Scope and organizing principle

The literature grouped under boundary-centric pretraining does not define a single loss family. Rather, it repeatedly reassigns epistemic importance to boundaries that generic objectives underuse. In some cases the boundary is distributional: a model memorizes atomic facts but fails to compose them outside the region of entities exposed to compositional contexts. In others it is structural: Chinese PLMs are asked to encode latent word boundaries, decoder features for segmentation are encouraged to align cluster transitions with image edges, or a vision transformer is forced to allocate masked modeling capacity to boundary-bearing tokens rather than randomly chosen patches. In still others it is topological or sequential: semantically related documents are packed across document boundaries, temporal transitions are synthesized as pretraining targets, or native CAD faces and edges are preserved as the substrate of multimodal alignment (Karmim et al., 8 Jun 2026, Shi et al., 2023, Xu et al., 2020, Peng et al., 2022, Deng et al., 18 Dec 2025).

Domain	Operative boundary	Representative pretraining move
LLM factual reasoning	Exposure to compositional contexts	Partition entities into exposed and held-out populations
Multilingual LM	Early alignment boundary	Align translated words before main LM pretraining
Chinese PLMs	Latent word/span boundaries	Add boundary-aware auxiliary objectives
Long-context LM	Document boundaries	Sort related documents adjacently
Video	Temporal action boundaries	Synthesize and classify boundary types
Vision and medical imaging	Spatial discontinuities	Boundary-forced masking or edge-aligned dense clustering
CAD-LLMs	Faces, edges, and topology	Encode native Brep entities before language alignment

A plausible implication is that “boundary-centric pretraining” is best understood as a design principle about where learning pressure is applied. The common intervention is not merely to add information, but to decide which transitions, interfaces, or partitions should become predictive during pretraining.

2. Exposure and initialization as pretraining boundaries

A particularly explicit formulation appears in work on implicit multi-hop composition in LLMs. The central setup separates individuals into a compositionally exposed population $\mathcal{P}_{\text{comp}}$ and a held-out population $\mathcal{P}_{\text{held}}$, with the guarantee that no member of $\mathcal{P}_{\text{held}}$ appears as a head, bridge, or target entity in any compositional sequence during pretraining. The task is to answer composed queries such as $f_2(f_1(x))$ in a single forward pass when the model already retrieves both constituent facts separately. On LoRA fine-tuning with 1-hop questions only, 1-hop accuracy is $1.00$ on $\mathcal{P}_{\text{comp}}$ and $0.97$ on $\mathcal{P}_{\text{held}}$, but 2-hop and 3-hop are both $0.01$. Under 1+2-hop fine-tuning, $\mathcal{P}_{\text{comp}}$ reaches 2-hop accuracy $\mathcal{P}_{\text{held}}$0, whereas $\mathcal{P}_{\text{held}}$1 remains at $\mathcal{P}_{\text{held}}$2; GPT-2 Small, Medium, and Large all stay at $\mathcal{P}_{\text{held}}$3 2-hop on $\mathcal{P}_{\text{held}}$4 despite 1-hop $\mathcal{P}_{\text{held}}$5. Even conditioning on both constituent 1-hop subquestions being answered correctly, the conditional composition success rate on $\mathcal{P}_{\text{held}}$6 is $\mathcal{P}_{\text{held}}$7. Data-centric augmentation changes performance only inside the exposed region: implicit 2-hop NL reaches $\mathcal{P}_{\text{held}}$8 on $\mathcal{P}_{\text{held}}$9, implicit RDF $\mathcal{P}_{\text{held}}$0, implicit+explicit NL-RDF $\mathcal{P}_{\text{held}}$1, while 2-hop accuracy on $\mathcal{P}_{\text{held}}$2 remains at chance across conditions. The paper therefore identifies a strict exposure boundary: compositional training transfers to unseen questions about exposed individuals, but not to individuals absent from compositional pretraining contexts (Karmim et al., 8 Jun 2026).

A second boundary appears earlier in the training trajectory. PreAlign argues that multilingual correspondence is too weak if it is left to emerge spontaneously during standard language-model pretraining. Its intervention is to establish multilingual alignment before the main LM phase using a layer-wise contrastive objective,

$\mathcal{P}_{\text{held}}$3

with word representations $\mathcal{P}_{\text{held}}$4, and then preserve that geometry by input-only code-switching during pretraining. In the synthetic English–English-Clone setting with 10B English tokens and 0.1B English-Clone tokens, Joint Training gives English-Clone LM perplexity $\mathcal{P}_{\text{held}}$5, XNLI zero-shot accuracy $\mathcal{P}_{\text{held}}$6, and CLKA $\mathcal{P}_{\text{held}}$7, whereas PreAlign reaches $\mathcal{P}_{\text{held}}$8, $\mathcal{P}_{\text{held}}$9, and $f_2(f_1(x))$0. The ablation shows that code-switching alone improves to $f_2(f_1(x))$1, alignment initialization alone to $f_2(f_1(x))$2, and the combination to $f_2(f_1(x))$3. Input-only code-switching also reduces mixed-script contamination from $f_2(f_1(x))$4 under vanilla code-switching to $f_2(f_1(x))$5. This makes the initialization boundary itself part of the pretraining hypothesis: early geometry is not a neutral starting condition but a determinant of later knowledge sharing (Li et al., 2024).

These two lines of work treat different objects—entity-specific composition and multilingual transfer—but converge on the same structural claim. Failure can persist despite strong local competence, and scale alone need not remove it. What matters is whether pretraining crosses the right boundary at the right stage.

3. Lexical and span boundaries in text models

Chinese boundary-aware pretraining makes latent lexical segmentation an explicit pretraining target rather than a downstream afterthought. BABERT derives unsupervised statistical boundary information from raw corpora by computing PMI, Left Entropy, and Right Entropy for $f_2(f_1(x))$6-grams up to length $f_2(f_1(x))$7, constructing a character-level boundary-aware representation $f_2(f_1(x))$8, and supervising shallow BERT states through

$f_2(f_1(x))$9

The model keeps the BERT architecture unchanged at inference time. On ten Chinese sequence-labeling benchmarks, BABERT improves the average F1 from $1.00$0 for BERT to $1.00$1, with consistent gains on all ten datasets. In few-shot Onto4 NER, BERT scores $1.00$2 with 10 examples, whereas BABERT reaches $1.00$3. The results support the claim that unsupervised boundary statistics can substitute for a substantial amount of manually curated lexical knowledge while remaining complementary to lexicon-enhanced fine-tuning (Jiang et al., 2022).

Semi-BABERT extends the same boundary-centric logic by adding supervised, lexicon-derived span supervision to BABERT’s unsupervised statistical signals. Its full objective,

$1.00$4

adds a span-based supervised boundary recognition task trained with Positive-Unlabeled learning over 2–4 character $1.00$5-grams. The lexicon is built from the OwnThink Knowledge Graph, filtered to 30 million words; corpus quality is also filtered with Qwen-7B, removing the bottom $1.00$6 of scored sentences and retaining $1.00$7 of a 3B-token corpus. Semi-BABERT-base reaches average F1 $1.00$8 versus $1.00$9 for BABERT and $\mathcal{P}_{\text{comp}}$0 for BERT, and Semi-BABERT-lite reaches $\mathcal{P}_{\text{comp}}$1 versus $\mathcal{P}_{\text{comp}}$2 for BERT-lite. In low-resource Onto4 with 10 examples, Semi-BABERT-base scores $\mathcal{P}_{\text{comp}}$3 versus $\mathcal{P}_{\text{comp}}$4 for BABERT. The paper also introduces the Boundary Information Metric,

$\mathcal{P}_{\text{comp}}$5

which gives $\mathcal{P}_{\text{comp}}$6 for Semi-BABERT-base, $\mathcal{P}_{\text{comp}}$7 for BABERT, and $\mathcal{P}_{\text{comp}}$8 for BERT on CTB6, indicating that the pretrained representations themselves become more boundary-aware (Zhang et al., 2024).

A multilingual variant of the same idea appears in CalibreNet, where the pretraining target is not wordhood but span correction. The calibration architecture receives an initial answer and full context, then refines start and end positions. Its dedicated pretraining task, Phrase Boundary Recovery, corrupts Wikipedia anchor-text spans by add, delete, and shift operations and trains the calibration module to recover the original boundaries with an MRC-style objective. The motivation is quantitative: on XGLUE-NER with XLM-R$\mathcal{P}_{\text{comp}}$9, boundary errors account for $0.97$0 of errors in Spanish and $0.97$1 in German. Full CalibreNet reaches average F1 $0.97$2 on XGLUE-NER, while removing both PBR objectives drops performance to $0.97$3; on MLQA it improves from $0.97$4 to $0.97$5 in F1/EM. Here the boundary-centric pretraining signal is explicitly about repairing span extent, not merely encoding semantics around the span (Liang et al., 2020).

Across these text models, the recurring pattern is that boundary information is injected before task-specific supervision, either as statistical lexical structure, semi-supervised span recognition, or boundary recovery from corrupted spans. This suggests that sequence labeling benefits when a PLM is pretrained to represent unit boundaries directly rather than only infer them indirectly from generic contextual prediction.

4. Boundaries across documents and time

In-context Pretraining reinterprets document boundaries as wasted supervision under random packing. Standard causal LM pretraining concatenates short documents to fill context windows, but prior documents provide no signal for predicting the next document if ordering is random. ICPT changes only the ordering policy: documents are embedded with Contriever, top-$0.97$6 neighbors are retrieved by FAISS over 235,266,464 documents, a sparse document graph is constructed, and a greedy graph traversal approximates a maximum traveling-salesman ordering so that adjacent documents are semantically related without repetition. The next-token loss is unchanged, but tokens immediately after a document boundary are now often conditioned on useful prior documents. Empirically, average in-context learning accuracy rises from $0.97$7 under Standard to $0.97$8 under ICLM, reading comprehension from $0.97$9 to $\mathcal{P}_{\text{held}}$0, NQ-Swap from $\mathcal{P}_{\text{held}}$1 to $\mathcal{P}_{\text{held}}$2, and MemoTrap from $\mathcal{P}_{\text{held}}$3 to $\mathcal{P}_{\text{held}}$4. An ablation on ordering quality gives perplexity $\mathcal{P}_{\text{held}}$5 for Random, $\mathcal{P}_{\text{held}}$6 for Clustering, and $\mathcal{P}_{\text{held}}$7 for Links; omitting semantic deduplication degrades perplexity from $\mathcal{P}_{\text{held}}$8 to $\mathcal{P}_{\text{held}}$9. The operative claim is boundary-aware rather than boundary-labeled: cross-document transitions become predictive because the data stream is reordered (Shi et al., 2023).

Temporal boundaries are made explicit in video pretraining. Boundary-sensitive Pre-training synthesizes temporal boundaries from trimmed action clips and uses boundary-type classification as a pretext task for temporal localization. The rebuttal document does not provide the full method section, but it does establish several concrete properties: the model uses equal ratio for each boundary type in training; the Diff-class boundary includes an action transition period controlled by $0.01$0, set empirically to $0.01$1; and the goal is to transfer to temporal action localization and video grounding rather than clip classification alone. Relative to a vanilla encoder pretrained on Kinetics-400 in a fully supervised manner, BSP improves ActivityNet 1.3 from $0.01$2 to $0.01$3, THUMOS14 from $0.01$4 to $0.01$5, and HACS 1.1 from $0.01$6 to $0.01$7. In this case the boundary is literally the onset, offset, or transition zone that downstream localization must recover, and pretraining is redesigned to make that transition salient (Xu et al., 2020).

These methods occupy opposite ends of explicitness. ICPT does not introduce a boundary-specific label, whereas BSP does. Yet both alter pretraining by asking what should happen immediately across a sequential boundary and by making that transition informative rather than incidental.

5. Spatial, anatomical, and geometric boundary modeling

The most direct visual formulation appears in masked boundary modeling for dense spatial perception. LingBot-Vision starts from the claim that DINO-style self-distillation, CLIP-style language supervision, and standard masked image modeling favor semantic invariance and do not explicitly preserve dense spatial structure. Its solution is to predict a sub-pixel categorical boundary field, identify boundary-bearing tokens online, and force them into the student mask via $0.01$8. Boundary tokens then receive dual supervision from semantic masked-token distillation and a geometric boundary loss,

$0.01$9

On ViT-L/16, the DINO+iBOT baseline gives $\mathcal{P}_{\text{comp}}$0 k-NN and NYUv2 linear depth $\mathcal{P}_{\text{comp}}$1 RMSE. Adding only the categorical boundary target improves depth to $\mathcal{P}_{\text{comp}}$2 RMSE, adding dual supervision reaches $\mathcal{P}_{\text{comp}}$3 k-NN and $\mathcal{P}_{\text{comp}}$4 RMSE, and the final RoPE recipe reaches $\mathcal{P}_{\text{comp}}$5 k-NN and $\mathcal{P}_{\text{comp}}$6 RMSE. With ViT-g, LingBot-Vision obtains $\mathcal{P}_{\text{comp}}$7 RMSE on NYUv2, compared with $\mathcal{P}_{\text{comp}}$8 for DINOv3-7B and $\mathcal{P}_{\text{comp}}$9 for V-JEPA 2.1 ViT-G. The paper therefore treats boundaries not as a downstream contour target but as the organizing signal of scalable self-supervision for dense perception (Fu et al., 6 Jul 2026).

In medical imaging, boundary-centric pretraining appears as edge-aligned dense clustering. Boundary-aware Information Maximization uses a U-Net-like encoder-decoder and replaces local dense contrastive learning with an over-segmentation pretext task. Dense decoder features are projected to $\mathcal{P}_{\text{held}}$00 clusters, the joint distribution across transformed views is regularized by a modified mutual-information objective, and the entropy map of the predicted cluster distribution is aligned with a Sobel-filter edge map through a local normalized cross-correlation loss. The total pretraining objective is

$\mathcal{P}_{\text{held}}$01

with $\mathcal{P}_{\text{held}}$02. On ACDC-RV, MI alone gives mean $\mathcal{P}_{\text{held}}$03, whereas MI+CC reaches $\mathcal{P}_{\text{held}}$04; on ACDC-Myo, $\mathcal{P}_{\text{held}}$05 rises to $\mathcal{P}_{\text{held}}$06; on PROMISE12, $\mathcal{P}_{\text{held}}$07 rises to $\mathcal{P}_{\text{held}}$08. CC alone performs poorly, indicating that edge alignment is a regularizer on meaningful clusters rather than a standalone pretext task. The boundary cue here is not a human-annotated contour but the coincidence between high-entropy cluster transitions and image gradients (Peng et al., 2022).

Boundary-centric pretraining in CAD adopts a still more literal meaning. BrepLLM operates directly on raw CAD boundary representations rather than point clouds, voxels, images, or tokenized CAD programs. Faces are adaptively sampled in UV space into 10D attributes, boundary edges are adaptively sampled into 8D attributes, and both are constrained to sampling counts in $\mathcal{P}_{\text{held}}$09. A hierarchical BrepEncoder extracts $\mathcal{P}_{\text{held}}$10 from face geometry, $\mathcal{P}_{\text{held}}$11 from edge-conditioned neighborhood aggregation, and $\mathcal{P}_{\text{held}}$12 from topology, concatenating them into 128-dimensional node tokens and a global graph token. Pretraining aligns the global Brep token with a frozen CLIP text encoder using the symmetric loss

$\mathcal{P}_{\text{held}}$13

On downstream Brep2Text tasks, BrepLLM reaches $\mathcal{P}_{\text{held}}$14 by SimCSE and $\mathcal{P}_{\text{held}}$15 average generative classification accuracy. The ablations most relevant to pretraining show adaptive UV sampling gains of $\mathcal{P}_{\text{held}}$16, $\mathcal{P}_{\text{held}}$17, and $\mathcal{P}_{\text{held}}$18 across stages, and hierarchical BrepEncoder gains of $\mathcal{P}_{\text{held}}$19, $\mathcal{P}_{\text{held}}$20, and $\mathcal{P}_{\text{held}}$21. In this literature, “boundary-centric” denotes learning directly from faces, edges, and their topology as the native representation of the object (Deng et al., 18 Dec 2025).

These vision, medical, and CAD systems differ in objective and substrate, but they share a strong commonality: the model is trained not to smooth over discontinuities, but to concentrate representational capacity where discontinuities occur.

6. Interpretive consequences, misconceptions, and open problems

Several misconceptions are directly challenged by this body of work. One is that strong local retrieval should automatically yield strong composition. The multi-hop study shows the opposite: $\mathcal{P}_{\text{held}}$22 1-hop accuracy on held-out entities can coexist with chance 2-hop composition, and larger GPT-2 variants do not change that pattern outside the exposure region (Karmim et al., 8 Jun 2026). A second is that multilingual transfer will emerge “for free” if enough data or scale is applied. PreAlign shows that spontaneous alignment exists but remains weaker than alignment established prior to LM pretraining; most of the synthetic CLKA gain comes from aligned initialization rather than code-switching alone (Li et al., 2024). A third is that boundary awareness is merely a byproduct of model size. Semi-BABERT-lite exceeds larger BABERT in BIM, and BERT and BERT-lite have the same BIM despite different scales (Zhang et al., 2024).

A broader lesson is that boundary-centric interventions often change where transfer occurs rather than simply improving average performance. In multi-hop factual QA, augmentation transfers across unseen questions but not across unexposed individuals. In ICPT, ordering related documents helps tasks that depend on using prior context, such as in-context learning, reading comprehension, retrieval augmentation, and faithfulness to previous context. In LingBot-Vision, the gains are strongest on dense spatial tasks, and the paper notes that depth gains are stronger in boundary-rich settings like NYUv2 than in driving scenes like KITTI. This suggests that boundary-centric pretraining is typically task-aligned and locality-sensitive rather than uniformly beneficial across all axes (Karmim et al., 8 Jun 2026, Shi et al., 2023, Fu et al., 6 Jul 2026).

The literature also leaves several open questions. The multi-hop work leaves open whether long-context natural corpora might substitute for explicit compositional supervision and whether retrieval-augmented or architectural methods can overcome the exposure boundary without relying on parametric implicit composition (Karmim et al., 8 Jun 2026). PreAlign is limited to models up to 1.3B parameters and depends on GPT-4-derived translation tables, raising questions about scale, lexical noise, and coverage in richer multilingual settings (Li et al., 2024). Semi-BABERT and BABERT remain tied to Chinese lexical statistics and lexicon heuristics, while BIM still requires segmented text for evaluation (Zhang et al., 2024, Jiang et al., 2022). LingBot-Vision adds considerable complexity through dense field prediction and a-contrario validation, and some images yield too few validated segments for boundary loss (Fu et al., 6 Jul 2026). BrepLLM aligns only the global token during pretraining and truncates node-token sequences to $\mathcal{P}_{\text{held}}$23, leaving fine-grained node-level alignment and larger CAD assemblies as open problems (Deng et al., 18 Dec 2025).

Taken together, these papers support a precise interpretation. Boundary-centric pretraining is not simply pretraining with extra side information. It is pretraining that assigns algorithmic importance to the interfaces at which generic objectives are known to underperform: compositional interfaces between facts, alignment interfaces between languages, latent lexical boundaries, transitions across documents and events, and discontinuities in spatial or topological structure. The empirical record across these domains suggests that pretraining succeeds when those interfaces are made predictive, and fails in systematic ways when they are not.