Orion-Bix: Tabular ICL with Biaxial Attention

• The paper introduces Orion-Bix—a foundation model employing biaxial attention and meta-learned in-context reasoning to overcome tabular learning challenges.
• It replaces a flat row-wise Transformer with a specialized row encoder that alternates multiple attention modes to handle heterogeneous features and large label spaces.
• It uses episodic meta-training on synthetically generated, diverse tables to achieve strong few-shot performance and competitive results in structured domains.

Searching arXiv for the specified term to ground the article in the cited paper. Orion-Bix is a tabular foundation model for in-context learning that combines biaxial attention with meta-learned in-context reasoning for few-shot tabular learning. It is introduced as a response to persistent difficulties in general-purpose tabular modeling, including mixed numeric and categorical fields, weak feature structure, limited labeled data, and large label spaces. The model replaces a single flat row-wise Transformer with a row encoder that alternates standard, grouped, hierarchical, and relational attention, couples this encoder to a label-aware ICL head, and is meta-trained on synthetically generated, structurally diverse tables with causal priors. The reported implementation is delivered as a scikit-learn compatible foundation model and is presented as an extension of prior tabular foundation-model work, especially TabICL (Bouadi et al., 28 Nov 2025).

1. Definition and problem setting

Orion-Bix is positioned within tabular machine learning, where gradient-boosted trees remain strong baselines and where direct transfer of LLM-style in-context learning is difficult. The motivating claim is that tabular data are not merely “sequences of columns”: real tables often contain heterogeneous features, semantic feature groups, multi-scale dependencies, and large label spaces. On that basis, Orion-Bix is designed to encode row structure more explicitly than earlier tabular transformers and to align training more closely with few-shot evaluation (Bouadi et al., 28 Nov 2025).

The model addresses five difficulties emphasized in its description: mixed data types, weak or ambiguous structure, small-data or few-shot regimes, large label spaces, and efficient reasoning over large tables. The proposed remedy has three principal parts: a biaxial row encoder using multiple attention patterns, a meta-learning episodic training regime built from synthetic tables, and a label-aware ICL head with hierarchical routing for large class spaces (Bouadi et al., 28 Nov 2025).

A common misconception would be to treat Orion-Bix as a conventional supervised tabular classifier with a standard fixed-head predictor. The reported design instead frames it as a foundation model for tabular in-context learning: support examples and query examples are explicitly separated, support labels are injected into the row representations, and the attention mask is structured so that query rows attend to support rows but not to other query rows (Bouadi et al., 28 Nov 2025).

2. System architecture

The model retains two components from TabICL: a column-wise SetTransformer embedder and a label-aware in-context learner. Its main architectural change is the replacement of the original row encoder with a bi-axial attention module. The input batch is represented as

$X \in \mathbb{R}^{B \times n \times m},$

where $B$ is the number of tasks or tables in a batch, $n$ the number of rows, and $m$ the number of features. The implementation also uses $N_{\text{CLS}}$ reserved class-token slots and a per-task feature-count tensor $d \in \mathbb{N}^B$ for variable-width tables (Bouadi et al., 28 Nov 2025).

The reported pipeline has five stages: column-wise SetTransformer embedding, biaxial row encoding, multi-CLS summarization, label-aware ICL over rows, and decoder or hierarchical classification routing (Bouadi et al., 28 Nov 2025). This decomposition is important because Orion-Bix does not impose a single homogeneous attention mechanism across the entire table. Instead, column processing and row processing are separated, with the row encoder operating on feature sequences obtained after column-wise embedding.

The column encoder treats each column as a set of row values. The model reserves $N_{\text{CLS}}$ feature positions at the start of the feature dimension, padded with a special skip value so that they can later become learnable summary tokens. For each cell, it applies a skippable linear projection:

$$\tilde{x}_{b,n,m} = \begin{cases} \text{skip\_value}, & \text{if } x_{b,n,m} = \text{skip\_value} \ w_h x_{b,n,m} + b_h, & \text{otherwise} \end{cases}$$

which is intended to ensure that missing or padded entries do not contaminate representations. SetTransformer-style attention blocks are then applied over rows for each column, with support for linear attention using configurable feature maps such as ELU, identity, and hedgehog, and for induced self-attention using inducing points to reduce complexity (Bouadi et al., 28 Nov 2025).

Instead of outputting a single embedding per column, the column encoder predicts feature-specific affine parameters,

$$W_{b,n,m} \in \mathbb{R}^{D}, \qquad b_{b,n,m} \in \mathbb{R}^{D},$$

and computes

$e_{b,n,m} = x_{b,n,m} \cdot W_{b,n,m} + b_{b,n,m}.$

This yields

$B$0

with $B$1. The stated intuition is that this allows distribution-aware feature transformations, normalization behavior, and column-type-sensitive embeddings (Bouadi et al., 28 Nov 2025).

3. Bi-axial attention and row summarization

The central architectural idea is the bi-axial row encoder. Orion-Bix argues that TabICL’s original row encoder, described as a single flat Transformer stack over all features, is too coarse for structured tables because it does not explicitly model local feature groups, forces one homogeneous attention pattern to handle both short- and long-range dependencies, and compresses a row into a single bottleneck vector too early. The reported motivation is especially strong for domains such as medical and finance, where features often cluster naturally and interactions occur at multiple scales (Bouadi et al., 28 Nov 2025).

After column embedding, the tensor

$B$2

is reshaped so that each row becomes a sequence,

$B$3

A single BiAxialAttentionBlock then applies four attention modes in sequence (Bouadi et al., 28 Nov 2025).

First, standard attention performs full self-attention across all features:

$B$4

Second, grouped attention partitions features into $B$5 groups of size approximately $B$6 and applies attention within each group:

$B$7

Third, hierarchical attention splits the feature sequence into two large partitions,

$B$8

and models coarse cross-partition interactions via

$B$9

$n$0

Fourth, relational attention applies another full self-attention pass:

$n$1

The stated purpose of this ordering is to combine generic pairwise dependency modeling, local group refinement, coarse-scale interaction, and a final global integration pass (Bouadi et al., 28 Nov 2025).

Row summarization is performed through multiple CLS tokens rather than a single pooled vector. Let

$n$2

These tokens are broadcast per row and used as queries in a dedicated CLS attention module:

$n$3

producing

$n$4

After stacking $n$5 such blocks, the CLS tokens are flattened,

$n$6

and reshaped to

$n$7

The paper’s explicit rationale is to avoid a single-representation bottleneck; this suggests a design preference for preserving multiple row-level summary channels rather than forcing early compression (Bouadi et al., 28 Nov 2025).

4. In-context learning head and large-label-space routing

After row encoding, Orion-Bix performs label-aware in-context classification over support and query rows. For support rows, labels are embedded and added to the row representation:

$n$8

followed by

$n$9

The label embedding is implemented via a one-hot plus linear layer, denoted $m$0 (Bouadi et al., 28 Nov 2025).

The row-wise Transformer uses a split attention mask encoding the few-shot protocol. For row index $m$1 and key index $m$2,

$m$3

Accordingly, support rows attend only to support rows, query rows attend to all support rows, and query rows do not attend to other query rows (Bouadi et al., 28 Nov 2025). This is a defining feature of the model’s ICL interpretation, since the attention structure itself enforces support/query asymmetry.

The output encoder state is

$m$4

and a small MLP maps each row to logits,

$m$5

If a task has more than $m$6 classes, Orion-Bix uses a hierarchical classification tree rather than a flat classifier. Labels are grouped recursively into at most $m$7 super-classes per level; internal nodes predict coarse group assignments; leaf nodes predict final classes; and test-time probabilities are obtained by traversing the tree and combining probabilities via the chain rule (Bouadi et al., 28 Nov 2025).

This routing mechanism is presented as a solution to scalability in large label spaces. A plausible implication is that Orion-Bix is intended not only for canonical low-class-count benchmarks but also for tasks where the final layer width would otherwise be a limiting factor. The source description, however, restricts the concrete claim to the recursive grouping and coarse-to-fine prediction procedure (Bouadi et al., 28 Nov 2025).

5. Meta-training regime and practical interface

A major contribution of Orion-Bix is its training regime. The model is meta-trained on synthetically generated, structurally diverse tables with a broad prior over feature counts, label spaces, sequence lengths, distributions, and table structures, often based on causal structure. The training process has four reported steps: generate synthetic tables from a configurable prior, convert tables into episodes with support and query splits, optionally use kNN-based support selection to choose informative and diverse support rows, and train in a meta-learning regime over many episodes (Bouadi et al., 28 Nov 2025).

The motivation is a contrast with TabICL’s reported synthetic training schedule, which treats each synthetic table as a single supervised task and therefore only indirectly resembles few-shot evaluation. Two issues are identified: implicit few-shot signal and uncontrolled support selection. Orion-Bix addresses these by explicit episode construction and optional support-set design (Bouadi et al., 28 Nov 2025). This suggests that alignment between pre-training and test-time evaluation is treated as a first-order design principle.

The implementation includes a MetaLearningDataset and MetaLearningTrainer using micro-batching, gradient accumulation, mixed precision, and optional distributed training to handle large episode counts (Bouadi et al., 28 Nov 2025). Those components situate Orion-Bix within scalable foundation-model training practice rather than within a small bespoke benchmark-only setup.

For inference and deployment, the model is wrapped in a scikit-learn–compatible classifier. The wrapper can detect numerical versus categorical features, impute missing numerical values, normalize via several schemes, clip outliers, and shuffle features via several strategies. It also constructs multiple transformed versions of the same dataset, including different normalization methods and different feature permutations, evaluates each view separately, and averages the resulting predictions (Bouadi et al., 28 Nov 2025). This ensemble-of-transformed-views procedure is described as improving robustness to feature order and stability on skewed data.

6. Empirical results and comparative performance

The reported evaluation groups datasets from TALENT and OpenML-CC18 into domain-specific suites, including Medical, Finance, and Energy, with Medical and Finance highlighted most prominently. The reported metrics are Accuracy (ACC), class-weighted F1, and mean rank across datasets within a domain. Few-shot curves are evaluated at support sizes

$m$8

and support quality is studied at $m$9 by comparing uniform or random sampling with diverse kNN-based support selection (Bouadi et al., 28 Nov 2025).

The baselines include XGBoost, RandomForest, CatBoost, and LightGBM among tree-based methods, and TabICL, TabPFN, TabDPT, Mitra, and ContextTab among tabular foundation models or in-context learners (Bouadi et al., 28 Nov 2025). The comparative framing is therefore broad: Orion-Bix is evaluated against both classical tabular learners and recent foundation-model architectures.

The domain-specific results explicitly reported for Medical and Finance are as follows.

Domain	Model	Reported results
Medical	Orion-Bix	rank 4.10, ACC 0.7893, F1 0.7759
Medical	TabICL	rank 5.54, ACC 0.7819, F1 0.7696
Medical	TabPFN	rank 5.04, ACC 0.7984, F1 0.7857
Finance	Orion-Bix	rank 5.39, ACC 0.8206, F1 0.8125
Finance	TabICL	rank 6.60, ACC 0.8125, F1 0.7942
Finance	CatBoost	ACC 0.8117, F1 0.8015
Finance	LightGBM	ACC 0.8095, F1 0.7974

In Medical, Orion-Bix is reported to have the best mean rank, while TabPFN slightly exceeds it on raw ACC and F1. In Finance, Orion-Bix is reported as best on all three tabulated metrics, improving over TabICL and remaining competitive with or better than the listed gradient-boosting baselines (Bouadi et al., 28 Nov 2025). This distinction matters because the source does not claim universal dominance across all metrics; rather, it presents stronger consistency in some domains and stronger top-line performance in others.

The few-shot results show a specific regime dependence. Orion-Bix is described as strongest at very low support sizes, $N_{\text{CLS}}$0, outperforming all compared baselines by roughly 2–4 points in that regime; performance rises with support size for all methods; the gap narrows at larger support sizes; and on OpenML-CC18, TabPFN can slightly overtake Orion-Bix at $N_{\text{CLS}}$1 (Bouadi et al., 28 Nov 2025). The support-quality analysis at $N_{\text{CLS}}$2 finds that uniform sampling slightly outperforms kNN-based selection by about 3–5 points, while Orion-Bix remains better than TabICL under both support-selection strategies (Bouadi et al., 28 Nov 2025).

7. Significance, limitations, and interpretation

The principal significance of Orion-Bix lies in its attempt to make tabular in-context learning more structure-aware. Its stated contributions are biaxial row encoding, multi-CLS summarization, a label-aware ICL head, hierarchical routing for large label spaces, episodic meta-learning on synthetic diverse tables, and a practical scikit-learn-style interface (Bouadi et al., 28 Nov 2025). Taken together, these elements define Orion-Bix less as a minor variant of a generic Transformer and more as a specialized tabular ICL architecture.

The strongest empirical claim concerns few-shot behavior and structured domains. The source argues that biaxial attention with episodic meta-training enables robust, few-shot-ready tabular learning and reports gains over TabICL together with competitive performance against tree-based methods and other tabular foundation models (Bouadi et al., 28 Nov 2025). A plausible implication is that Orion-Bix is particularly targeted at low-label, heterogeneous, semantically structured table regimes rather than at all tabular workloads uniformly.

The paper also presents several caveats. The reported benchmark suite is limited to the domain groupings and public benchmarks discussed; broader evaluation would strengthen the claims. The reported results are mainly classification results, and no regression results are described. Some benefits appear strongest in few-shot and structured domains, while gains may be smaller when support sets are large. TabPFN remains competitive or better on some metrics or datasets, especially in certain few-shot settings. The synthetic-data prior and routing strategy add implementation complexity relative to simpler tabular learners (Bouadi et al., 28 Nov 2025).

An objective reading therefore places Orion-Bix within an active line of work on tabular foundation models rather than as a settled endpoint. It is defined by a particular synthesis: structure-aware feature-axis attention, explicit support/query reasoning, and synthetic episodic meta-training. The reported results support the view that these design choices improve performance in the low-shot regime and on structured domains, while also indicating that alternative tabular foundation models, especially TabPFN, remain strong comparators on some benchmarks (Bouadi et al., 28 Nov 2025).