CrossTL: Transfer Learning in Heterogeneous Domains

• CrossTL is a concept representing a family of methods that transfer structure across heterogeneous spaces by leveraging shared intermediate representations.
• It includes implementations such as XTFormer for cross-table pretraining, cross-modality detection using pseudo-images, and cross-lingual distillation for multilingual tasks.
• Empirical results demonstrate significant gains, with XTFormer outperforming traditional models like XGBoost on 72% of evaluated tasks and enhancing domain adaptation.

CrossTL is a recurrent label for methods that transfer structure across heterogeneous spaces rather than within a single shared modality, schema, or language. In the papers considered here, it appears most directly as cross-table learning / cross-table pretraining for heterogeneous tabular prediction, but related work uses the same label or explicitly describes CrossTL-style behavior for cross-modality object detection, cross-domain recommendation, multilingual supervised fine-tuning, cross-lingual text classification, cross-semantic regression, cross-table generation, and universal source-to-source code translation. A common pattern is the replacement of pairwise bespoke mappings with a shared transferable space: a meta-function space for tables, a pseudo-image or latent representation for multimodal and multilingual learning, an all-pairs coefficient fusion space for regression, or a unified intermediate representation for code (Chen et al., 2024, Niketan et al., 28 Aug 2025).

1. Term and conceptual range

The label appears in several distinct but structurally related forms. In each case, the central problem is heterogeneity: features differ across tables, modalities differ across sensors, schemas differ across business domains, languages differ across multilingual tasks, or source languages differ across programming ecosystems. The methods therefore introduce an intermediate object that can be shared, aligned, or calibrated rather than requiring direct pairwise compatibility.

Use of CrossTL	Shared mechanism	Representative paper
Heterogeneous tabular prediction	Meta-function space with CaLinear	(Chen et al., 2024)
2D-to-3D object detection	DHS pseudo-images reused by 2D detectors	(Shen et al., 2022)
Cross-domain CTR prediction	SCN/IFN/REN with sample-wise transfer weights	(Zhang et al., 2023)
Multilingual transfer	Parallel-data distillation or latent FFN fusion	(Xu et al., 2017, Ye et al., 1 Jun 2025)
High-dimensional regression	All-pairs weighted fusion penalty	(Jiang et al., 25 Dec 2025)
Programming-language translation	CrossGL unified intermediate representation	(Niketan et al., 28 Aug 2025)

This range is not merely terminological. It indicates that CrossTL is less a single algorithm than a family of architectures for transferring knowledge when direct alignment is weak, partial, or expensive to engineer.

2. Cross-table pretraining and the XTFormer meta-function space

The most explicit machine-learning meaning of CrossTL in this corpus is cross-table learning / cross-table pretraining for heterogeneous tabular data. The concrete model is XTFormer, a cross-table pretrained Transformer designed to learn a universal function space rather than a single fixed predictor. Its backbone is a 4-layer FT-Transformer-style architecture in which the feedforward linear layers are replaced with CaLinear, a mixture of basis linear functions: $$f(x; \mathbf{c}, \Phi)=\sum_{m=1}^{M} c_m \phi_m(x), \qquad \mathbf{c}=M_{cal}(v)=\text{Softmax}(\text{MLP}(v)).$$ At the layer level,

$$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$

and the two-layer CaLinear feedforward block is

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$

With $M=4$ basis functions and $L=4$ transformer blocks, the paper gives an intuitive function-space size of

$M^{2L}=4^8=65{,}536.$

The key claim is that downstream transfer should be framed as coordinate positioning in this meta-function space rather than copying a fixed model across datasets (Chen et al., 2024).

Pretraining uses OpenTabs V1 source datasets: 294 binary classification datasets and 366 regression datasets. Downstream evaluation covers 190 downstream tasks from 38 datasets, with 20 binary classification datasets and 18 regression datasets, each tested under T-full, T-200, T-100, T-50, and T-20 regimes. Adaptation proceeds in two stages. During task calibration, the shared body is frozen, while the dataset-specific embedding layer, output layer, learnable context vector(s) $v$, and normalization layers are adapted. After calibration, a short refinement stage unfreezes all parameters for 5 epochs. The empirical result reported in the paper is unusually strong for deep tabular learning: among 190 downstream tasks, XTFormer beats XGBoost and CatBoost on 137 tasks (72%), beats FT-Transformer on 144 tasks (76%), and beats XTab on 162 tasks (85%). Its average rankings are 3.84 on T-full, 2.84 on T-200, 2.50 on T-100, 3.16 on T-50, and 2.08 on T-20, with lower rank being better (Chen et al., 2024).

This formulation is notable because it treats heterogeneity as a property of the target function rather than merely the input schema. The reusable object is not a shared feature vocabulary, but a calibrated combination of basis functions. A plausible implication is that CrossTL, in this tabular sense, is closer to transferable function-space geometry than to conventional representation transfer.

3. Cross-modality and cross-domain transfer architectures

In vision, CrossTL appears as a deliberately minimal cross-modality transfer learning recipe. simCrossTrans studies transfer from 2D RGB pretraining to a 3D-sensor-only detection pipeline by converting point clouds into 3-channel pseudo-images so that an almost unchanged 2D detector can be reused. The pipeline is explicitly summarized in three steps: (1) pretrain a 2D vision system on RGB images; (2) convert point clouds into pseudo-images; (3) fine-tune the pretrained model on those pseudo-images. The encoding is DHS, where D is depth from the sensor, H is height along the sensor-up direction, and S is signed angle. The paper gives

$D_{i,k} = X_{i,k+1} - X_{i,k}, \qquad S_{ik} =sgn(\mathbf{D_{i,k}\cdot \mathbf{D_{i,k-1})*A_{ik},$

followed by per-channel normalization to $[0,1]$. The detection framework is Mask R-CNN in mmdetection, with ImageNet pretraining, then COCO pretraining, then fine-tuning on SUN RGB-D pseudo-images. On SUNRGBD16, simCrossTrans yields absolute AP50 gains of 13.2% for the ResNet-50 variant and 16.1% for the Swin-T variant; ResNet-50 improves from 29.8 AP50 to 43.0 AP50, Swin-T from 36.6 AP50 to 52.7 AP50, and the ViT-based system is reported as 9.7% better than the ConvNet one. Against Frustum VoxNet, the Swin-T version improves by +15.4% mAP50, and the depth-only system is said to be only about 1% behind the previous RGB-based 2D detection SOTA. The runtime trade-off is explicit: 70 ms / 14.3 FPS for the ResNet-50 version versus 105 ms / 9.5 FPS for the Swin-T version on a Titan X GPU (Shen et al., 2022).

In recommendation, the analogous issue is not sensor mismatch but domain shift and negative transfer across business domains with different CTR distributions and feature schemas. CCTL addresses this with three coupled modules: Symmetric Companion Network (SCN), Information Flow Network (IFN), and Representation Enhancement Network (REN). SCN compares a mixed tower trained on source and target data with a pure tower trained only on target data, defining the source impact as

$r = Loss_{gain} = Loss_{pure} - Loss_{tgt}.$

IFN then estimates a per-source-sample transfer weight

$$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$0

and updates the selector by REINFORCE using accumulated reward $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$1 with $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$2. REN adds the auxiliary objective

$$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$3

thereby maximizing similarity between sequence embeddings while minimizing similarity between ID embeddings. The framework is evaluated on Amazon, Taobao, and a large Meituan industrial dataset. Reported offline results include AUC 0.7001 / LogLoss 0.5288 on Meituan, AUC 0.7790 / LogLoss 0.4230 on Amazon, and AUC 0.8521 / LogLoss 0.3578 on Taobao. In Meituan deployment, the paper reports 4.37% CTR and 5.43% GMV lift, while preserving lightweight online inference by exporting only the synchronized pure tower (Zhang et al., 2023).

These two systems illustrate two different CrossTL strategies. simCrossTrans reduces heterogeneity by converting the target modality into a representation that looks like the source modality. CCTL keeps domains separate but learns a transfer gate that determines which source samples should be allowed to influence the target model.

4. Cross-lingual and cross-semantic formulations

A classical cross-lingual formulation is model distillation over parallel documents. In Cross-lingual Distillation for Text Classification, a source-language classifier is first trained on labeled source data and then used to generate soft probabilistic predictions on the source side of a parallel corpus. The target-language classifier is trained to match those soft labels on the target side: $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$4 An adversarial feature adaptation module with a gradient reversal layer reduces distribution mismatch between labeled source data, parallel data, and target test data. On Amazon review classification, the paper reports average accuracy 79.33 for CLD-KCNN and 81.08 for CLDFA-KCNN; on English-Chinese Yelp hotel reviews, CLDFA-KCNN reaches 41.82%, compared with 41.04% for ADAN and 40.96% for CLD-KCNN (Xu et al., 2017).

A later and more internalist multilingual formulation appears in CC-Tuning, which shifts transfer from the data level to the latent FFN activation level during joint multilingual supervised fine-tuning. For each non-English sample $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$5, the model also processes a parallel English input $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$6, extracts English FFN activations

$$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$7

and uses a trainable Decision Maker

$$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$8

together with Gumbel-Softmax to select a beneficial English layer. The selected activation is then fused into the non-English forward pass: $$\mathbf{z}'=\sum_{m=1}^{M} c_m\,Linear_m(\mathbf{z}),$$9 The training objective remains standard SFT,

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$0

with no auxiliary loss. At inference time, a Transform Matrix $$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$1 is learned from 1,000 parallel pairs by least squares so that $$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$2, enabling monolingual inference. On six benchmarks covering 22 unique languages, CC-Tuning improves LLaMA-3.1-8B from 31.88 to 38.42 on XNLI, from 65.23 to 70.60 on XStoryCloze, from 14.64 to 15.94 on MKQA, from 60.42 to 61.85 on XQuAD, and from 12.27 to 12.88 on XLSum. Training time is about 1.12–1.16× vanilla SFT and inference time about 1.09–1.10×, while the Decision Maker adds only 0.0016% of LLaMA-3.1-8B parameters and 0.0013% of Qwen2.5-7B parameters (Ye et al., 1 Jun 2025).

A mathematically distinct but conceptually related formulation is Cross-Semantic Transfer Learning (CSTL) for high-dimensional linear regression, also described as CrossTL in the paper. Here the problem is that source and target domains may not share semantically aligned features. CSTL therefore compares every target coefficient with every source coefficient through

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$3

The all-pairs weighted fusion penalty is combined with SCAD-derivative-based weights and optimized by ADMM. The theory establishes that, under the stated conditions, CSTL attains the oracle estimator with overwhelming probability. Empirically, the paper reports that CSTL outperforms Lasso, TransLasso, and TransGLM in simulations and on the Communities and Crime Unnormalized dataset, where the source is New Jersey (211 samples) and the target Washington (40 samples) (Jiang et al., 25 Dec 2025).

Taken together, these formulations show an evolution from transfer through soft labels on parallel text, to transfer through direct latent activation coupling, to transfer through coefficient-value similarity without feature alignment. The shared methodological theme is that transfer is defined by semantic or functional correspondence rather than literal input identity.

5. From generative tabular models to universal code translation

CrossTL also extends beyond discriminative prediction. CTSyn is a cross-table generative foundation model for tabular data generation. It combines an aggregator that maps heterogeneous rows into a unified latent vector, a conditional latent diffusion model operating in that latent space, and type-specific decoders for categorical and numerical cells. A row is represented as

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$4

embedded into a unified sequence $$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$5, and aggregated into

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$6

The aggregator is trained with a contrastive objective and a magnitude-aware loss,

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$7

while the latent diffusion model uses the simplified denoising loss

$$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$8

The model supports Conditional generation (Cond Gen) and Conditional column augmentation (Cond Aug), the latter decoding both the observed fine-tune columns and additional columns that appear in the holdout test set. In experiments on healthcare datasets such as NPHA, Obesity, Diabetes, Liver Patients, and Sick, and on cross-domain datasets including Abalone, Churn Modeling, and Shoppers, the paper reports that CTSyn outperforms existing synthesizers on fidelity, utility, and diversity, with Cond Aug achieving the best average rank in both healthcare and cross-domain settings (Lin et al., 2024).

A formally different use of the term appears in CrossTL: A Universal Programming Language Translator with Unified Intermediate Representation. Here CrossTL is a source-to-source translator whose central object is CrossGL, a unified intermediate representation intended to reduce the scaling problem of pairwise translators from $$\mathbf{z}' = CaLinear(ReLU(CaLinear(\mathbf{z}))).$$9 to $M=4$0. CrossGL includes a unified type system, expression trees, control-flow graphs, function definitions, data structures such as structs, memory operations and buffer patterns, and an attribute system for preserving source-language details. The architecture is a three-stage pipeline: parse source language, convert AST to CrossGL, and generate target language code. The system supports translation among CUDA, HIP, Metal, DirectX HLSL, OpenGL GLSL, Vulkan SPIR-V, Rust, and Mojo, with Slang support in active development. The implementation section states 8 fully implemented target languages, and the evaluation reports a 100% success rate for the listed test programs across the supported backends, including SimpleShader.cgl, ComplexShader.cgl, MatrixMultiplication.cgl, ParticleSimulation.cgl, and UniversalPBRShader.cgl (Niketan et al., 28 Aug 2025).

This suggests a broader abstraction of CrossTL: a hub-and-spoke design in which transfer or translation is mediated by a common latent or intermediate layer. In machine learning that layer is usually learned; in programming-language translation it is explicitly specified as an IR.

6. Assumptions, limitations, and research significance

Despite their diversity, these systems share strong assumptions. XTFormer explicitly acknowledges that the meta-function space may not perfectly contain every target function, that refinement is still needed, and that too many calibration steps can hurt when data is extremely limited (Chen et al., 2024). simCrossTrans assumes that source and target modalities observe the same underlying world/object categories, is demonstrated specifically on indoor SUN RGB-D 2D detection, uses supervised pretraining only, and remains dependent on converting point clouds into a 2D raster-like format rather than operating as a native point-cloud model (Shen et al., 2022). Cross-lingual Distillation for Text Classification depends on a parallel corpus and is sensitive to distribution mismatch, which is why it adds adversarial feature adaptation (Xu et al., 2017). CC-Tuning requires parallel English inputs during training, learns its inference-time transform from sampled parallel data, was tested only on LLaMA-3.1-8B and Qwen2.5-7B, and notes that if the training mix becomes mostly English, the mechanism effectively becomes an English-to-English connection and the benefit drops (Ye et al., 1 Jun 2025). CSTL incurs the computational burden of an all-pairs fusion term over $M=4$1 coefficient differences and its theory relies on accurate initialization and minimum-signal conditions (Jiang et al., 25 Dec 2025). CrossTL as a programming-language translator acknowledges that advanced language-specific features are difficult to represent universally and that some semantic mappings require backend adaptation (Niketan et al., 28 Aug 2025).

The significance of CrossTL, across these usages, lies in a consistent shift away from rigid one-to-one correspondence assumptions. In tabular learning, transfer is performed by calibrating a point in a reusable function space. In multimodal detection, point clouds are encoded so that 2D pretrained detectors remain usable. In recommendation, source information is filtered sample by sample rather than indiscriminately shared. In multilingual models, representations are connected internally rather than only through data augmentation. In regression, transfer occurs through coefficient-value similarity rather than feature-name alignment. In programming-language translation, a universal IR replaces an exploding set of pairwise translators. The collective result is not a single doctrine but a recognizable research program: construct an intermediate space rich enough to preserve task-relevant semantics, and then make cross-domain reuse a problem of alignment, calibration, or decoding rather than direct structural identity.