Slice-Depth Transformer Overview

• Slice-Depth Transformer is a class of architectures that introduce a dedicated slice axis to separately process local (in-slice) and global (inter-slice) features.
• They combine efficient 2D CNN or ViT-based feature extraction with transformer blocks focused on long-range interactions along the slice dimension.
• These models are applied in medical imaging, LiDAR, and language tasks, achieving improved segmentation, localization, and computational efficiency.

“Slice-Depth Transformer” (Editor’s term) denotes a family of transformer architectures that introduce an explicit slice axis into the model’s computational structure. In the most common usage, the slice axis is the ordered depth stack of a volumetric scan, so the transformer models inter-slice dependencies without necessarily treating the input as a full 3D token grid (Chen et al., 2023). Closely related formulations treat 3D scans as sequences of 2D slice embeddings (Müller-Franzes et al., 2024), angular LiDAR sectors as slice tokens (Ibrahim et al., 2023), contiguous embedding subvectors as slices routed to experts (Vejendla, 5 Oct 2025), or recurrently reused transformer depth as a sliced depth dimension (Shen et al., 2021, He, 5 May 2026). This suggests that the term names a design pattern rather than a single canonical architecture.

1. Conceptual scope

Across the literature represented here, the unifying idea is not a specific backbone but a specific axis of factorization. Rather than applying attention only over patch tokens in a 2D image or over a dense 3D token lattice, these models isolate a structured “slice” dimension and give it dedicated transformer machinery. In medical imaging, the slice dimension is the stack of axial, sagittal, or cine slices; in LiDAR, it is a partition of a $360^\circ$ scan into angular sectors; in conditional-computation transformers, it can be the embedding dimension itself; and in recurrent-depth models it becomes the iteration axis of a shared block (Hung et al., 2022, Li et al., 2022, Jin et al., 2024).

A concise way to view the family is as a separation of local and global structure. Local within-slice structure is often handled by 2D CNNs, point encoders, or per-slice ViTs, while the transformer is reserved for long-range interactions along the slice axis or for a second, factored notion of depth (Chen et al., 2023, Müller-Franzes et al., 2024). This division is especially common in anisotropic data, where through-plane resolution is much weaker than in-plane resolution, and in small-data regimes, where reusing strong 2D backbones is preferable to training full 3D models from scratch (Hung et al., 2022, Müller-Franzes et al., 2024).

Domain	Slice unit	Transformer role
Volumetric CT/MRI/CMR	Ordered 2D slices	Inter-slice attention, refinement, or diagnosis (Chen et al., 2023, Müller-Franzes et al., 2024)
LiDAR localization	36 overlapping angular sectors	Global attention over axial slices (Ibrahim et al., 2023)
MoE, recurrent depth, sorting	Embedding slices, shared depth loops, or feature channels	Routing, recursive reuse, or slice-wise permutation (Vejendla, 5 Oct 2025, Shen et al., 2021, Yuan et al., 2023)

2. Canonical formulation on stacked 2D slices

The clearest instantiations arise in volumetric medical imaging. In "Enhancing Cardiac MRI Segmentation via Classifier-Guided Two-Stage Network and All-Slice Information Fusion Transformer" (Chen et al., 2023), a cardiac cine MRI volume at a single time point is represented as a stack of $S$ 2D slices, and the transformer is inserted at the UNet bottleneck in both a segmentation stage and a refinement stage. The design does not tokenize the full volume in ViT or UNETR style. Instead, all slices are concatenated along the batch dimension before the 2D CNN encoder, the bottleneck features are regrouped volume-wise, and two consecutive transformer blocks perform inter-slice attention and intra-slice attention respectively. The paper is explicit that positional encoding is removed “to facilitate the variable input slice length,” since the number of slices varies across subjects (Chen et al., 2023). That design makes the inter-slice block a lean transformer over the slice or depth axis, while the intra-slice block restores long-range in-plane context.

CAT-Net applies the same principle at multiple encoder scales for prostate zonal segmentation in anisotropic MRI (Hung et al., 2022). Its Cross-Slice Attention Transformer modules are inserted after encoder stages in nnU-Net or nnU-Net++ style backbones. Convolutions remain purely 2D; slice interaction is delegated to a transformer that computes an attention matrix $A \in \mathbb{R}^{l \times l}$ between slices, with sinusoidal positional encodings along the slice axis. Queries and keys are formed from spatially pooled slice features for efficiency, whereas values retain full spatial resolution. The resulting representation is then returned to the skip pathway, so depth modeling is multi-scale rather than confined to a single bottleneck (Hung et al., 2022).

SATr adapts slice attention to universal lesion detection in CT by customizing $q$, $k$, and $v$ for a key slice and its neighbors (Li et al., 2022). Patch embeddings are produced both per slice and jointly across slices. The Slice Attention Transformer then uses upper and lower slices to form queries and keys, while values are derived from the sum of the key-slice embedding and an all-slice embedding. This arrangement is meant to strengthen inter-slice reasoning rather than rediscovering features already available in the key slice through the CNN backbone. In effect, SATr is a hybrid slice-depth module that can be plugged into multi-slice detection backbones without redesigning the entire detector (Li et al., 2022).

The Medical Slice Transformer takes a more explicitly sequential formulation (Müller-Franzes et al., 2024). Each standardized $224 \times 224 \times 32$ MRI or CT volume is decomposed into 32 slices; each slice is encoded by DINOv2 into a $d$-dimensional vector; and the sequence $[z_1,\dots,z_{32}]$, augmented with a CLS token, is processed by a single transformer encoder layer with 12 or 16 heads. In ablation, the transformer without positional embeddings gives the highest AUCs, while linear aggregation and average pooling are weaker (Müller-Franzes et al., 2024). This is the purest version of the pattern: a 3D volume is converted into a 1D depth sequence, and self-attention supplies the through-plane context.

Diff3Dformer keeps the same slice-sequence logic but changes the token type (Jin et al., 2024). Instead of raw slices or CNN bottleneck maps, each CT slice is encoded by a diffusion autoencoder into a 512-dimensional latent vector, slices are clustered with spherical K-means into $K=64$ prototypes, and a six-layer Clustering Transformer Encoder performs clustering attention with complexity $S$0 rather than $S$1. In this formulation, the slice-depth transformer is not only sequential but also prototype-based: repetitive slices are compressed into cluster representatives before global reasoning (Jin et al., 2024).

3. Representation, supervision, and conditioning

A recurrent feature of slice-depth models is that supervision is often slice-aware rather than purely volume-level. In the two-stage CMR system of (Chen et al., 2023), the refinement stage includes an MLP classifier branch at the bottleneck that predicts per-slice presence or absence of RV, LV, and MYO. The refinement loss is

$S$2

with $S$3, and the segmentation term is gated so that class-specific segmentation loss is applied only when the classifier predicts the anatomy is present on that slice. During inference, a class is shown in the final segmentation for a slice only if its classifier probability exceeds $S$4 (Chen et al., 2023). This couples inter-slice context to an explicit slice-level prior and is intended to suppress spurious basal or apical fragments.

Diff3Dformer combines slice-wise self-supervised representation learning with patient-level classification (Jin et al., 2024). The diffusion autoencoder is trained by

$S$5

so each 2D slice is encoded into a latent sufficient for DDIM-style denoising. After clustering, the final patient-level risk score is

$S$6

where $S$7 is a global cluster attention weight, $S$8 is the average slice risk score within cluster $S$9, and $A \in \mathbb{R}^{l \times l}$0 is the ratio of slices assigned to that cluster for the patient (Jin et al., 2024). The model thereby makes the slice distribution itself part of the classifier.

Slice-based supervision also appears in self-supervised LiDAR localization (Ibrahim et al., 2023). A $A \in \mathbb{R}^{l \times l}$1 scan is partitioned into $A \in \mathbb{R}^{l \times l}$2 overlapping angular slices, grouped into four quadrants of nine slices, and the pretext task is to predict which of the $A \in \mathbb{R}^{l \times l}$3 quadrant permutations was applied. The downstream localization model reuses the pretrained backbone, applies a learned feature-filtering mask, pools across slices, and regresses translation and rotation with a combined loss

$A \in \mathbb{R}^{l \times l}$4

The slice reordering task forces the model to learn long-range angular relationships before any pose labels are used (Ibrahim et al., 2023).

A plausible implication across these systems is that the slice axis is not merely an organizational convenience. It often becomes the unit at which priors, pseudo-labels, or halting decisions are defined, and this in turn differentiates slice-depth transformers from plain 2.5D stacking.

4. Alternative meanings of “slice” and “depth”

Outside volumetric imaging, the same structural idea reappears with different axes. SliceMoE replaces token-level routing with routing of contiguous slices of a token’s hidden vector (Vejendla, 5 Oct 2025). A $A \in \mathbb{R}^{l \times l}$5-dimensional embedding is partitioned into $A \in \mathbb{R}^{l \times l}$6 non-overlapping slices,

$A \in \mathbb{R}^{l \times l}$7

and a shared router sends each slice independently to top-$A \in \mathbb{R}^{l \times l}$8 experts. A slice-level capacity loss based on the squared coefficient of variation of expert slice counts is added to improve balance, together with cross-slice dropout and fused batched GEMM kernels (Vejendla, 5 Oct 2025). Here, “depth” no longer means physical scan depth; it becomes a factorization of the embedding space into routed subspaces.

SReT and RD-ViT reinterpret depth itself as a recurrent dimension (Shen et al., 2021, He, 5 May 2026). SReT reuses transformer weights recursively across depth and reduces the extra computation through group self-attention on slices of the token sequence across recursive layers. RD-ViT extends recurrent depth to dense prediction by replacing a deep stack of unique transformer blocks with a shared block looped $A \in \mathbb{R}^{l \times l}$9 times. Its recurrence is

$q$0

with $q$1 parameterized so that $q$2, and augmented with Adaptive Computation Time, depth-wise LoRA, and optional MoE FFNs (He, 5 May 2026). In these models, a slice-depth transformer is a transformer whose effective depth is generated by recurrence rather than by a stack of distinct layers.

Sliceformer moves the notion of slicing to the feature channel axis (Yuan et al., 2023). It replaces QKV attention with SliceSort,

$q$3

where $q$4 and each feature dimension is sorted independently across tokens. The resulting implicit attention maps are permutation matrices, hence sparse, full-rank, and doubly stochastic (Yuan et al., 2023). This is not a slice-sequence model, but it shows that slice-based transformer design can also mean channel-wise factorization of the attention operation itself.

The theoretical graph literature offers a complementary interpretation (Yehudai et al., 3 Mar 2025). There, a “slice-depth” viewpoint corresponds to making transformers very shallow and compensating by increasing width. For directed 2-cycle detection with adjacency-row tokenization, the paper proves

$q$5

with residual connections, while also showing that linear width permits constant-depth constructions for powers of the adjacency matrix and other fixed-length graph computations (Yehudai et al., 3 Mar 2025). This suggests that a second family of slice-depth transformers trades sequential layer depth for wider per-token state.

5. Applications, empirical behavior, and explainability

In cardiac MRI segmentation, the two-stage network with the all-slice information fusion transformer reports better Dice score than previous CNN-based and transformer-based models, produces segmentation shapes resembling human annotations, avoids holes or fragments, and achieves the highest RV Dice across base, mid, and apex according to the paper’s ablation summary (Chen et al., 2023). MOSformer, another inter-slice fusion architecture, reports 85.63%, 92.19%, and 85.43% DSC on Synapse, ACDC, and AMOS, respectively, and attributes the gain to dual encoders and an inter-slice fusion transformer at multi-scale feature maps (Huang et al., 2024).

For universal lesion detection, SATr improves average sensitivity across several baselines on DeepLesion (Li et al., 2022). Examples include 3DCE, which rises from 73.36 to 77.61 average sensitivity, and A3D, which rises from 86.54 to 87.92. The gains are larger for models with weaker pre-existing multi-slice fusion and remain visible even when only 25% or 50% of the training data are used (Li et al., 2022). CAT-Net shows a similar pattern for prostate zonal segmentation: on the internal dataset, CAT-nnU-Net reaches TZ Dice 90.4 and PZ Dice 86.1, and the reported region-wise analysis emphasizes improved consistency at apex, mid-gland, and base slices (Hung et al., 2022).

The Medical Slice Transformer demonstrates the diagnostic and explanatory side of the pattern (Müller-Franzes et al., 2024). Against a 3D ResNet, it reaches AUC $q$6 versus $q$7 on breast MRI ($q$8), $q$9 versus $k$0 on chest CT ($k$1), and $k$2 versus $k$3 on knee MRI ($k$4). Its attention-based saliency maps are judged by a radiologist to have slice correctness 136/150 and lesion correctness 57/150, compared with 37/150 and 0/150 for Grad-CAM on the 3D ResNet (Müller-Franzes et al., 2024). In this case, slice-level attention is directly interpretable as evidence allocation over depth.

Diff3Dformer targets small 3D CT datasets and reports that the combination of diffusion slice latents and clustering attention improves both AUC and robustness (Jin et al., 2024). In ablation, the full diffusion + clustering ViT configuration reaches AUC 0.91 on CC-CCII and 0.79 on the fibrotic lung disease task, outperforming both contrastive features and a plain ViT without clustering attention (Jin et al., 2024). Slice3D extends slice reasoning beyond diagnosis: it predicts multi-slice images from a single RGB view and then integrates them into a 3D model using a coordinate-based transformer for signed distance prediction, with all reported results produced by networks trained on a single Nvidia A40 GPU and inference time less than 20 seconds (Wang et al., 2023).

The pattern is not confined to medical imaging. In outdoor LiDAR localization, the Slice Transformer pretrained by quadrant permutation achieves mean translation error 5.23 m and mean rotation error 0.59° on Perth-WA, compared with 25.0 m and 0.75° for PointLoc; it also reports average translation RMSE 0.030 m on Apollo-SouthBay and transfers to object classification with 92.4% on ModelNet40 and 84.5% on ScanObjectNN (Ibrahim et al., 2023). In language and translation, SliceMoE reports up to 1.7x faster inference than dense baselines, 12 to 18 percent lower perplexity than parameter-matched token-MoE, and better expert balance, while also reaching WikiText-103 perplexity 25.4 and WMT En–De BLEU 29.8 in the detailed results (Vejendla, 5 Oct 2025). These results indicate that slice-oriented factorization can improve either spatial coherence, conditional computation, or interpretability depending on the domain.

6. Design tradeoffs, limitations, and open directions

A consistent tradeoff is between full 3D modeling and axial factorization. The all-slice fusion transformer explicitly avoids full 3D ViT tokenization and removes positional encodings to accommodate variable input slice length, which is useful when slice counts vary from 4 to 18 or 6 to 18 across datasets (Chen et al., 2023). CAT-Net likewise keeps all convolutions 2D because prostate MRI is anisotropic, and through-plane attention is cheaper than full 3D attention (Hung et al., 2022). Diff3Dformer reduces slice-sequence redundancy through clustering attention, replacing $k$5 interactions with $k$6 interactions over cluster prototypes (Jin et al., 2024). These choices suggest that slice-depth transformers are often best understood as structured efficiency mechanisms as much as representation-learning mechanisms.

The limitations are equally domain-specific. MST standardizes all scans to $k$7 because real clinical CT and MRI often exceed $k$8 pixels and hundreds of slices, and the paper notes that downsampling and cropping may discard fine detail and long-range context (Müller-Franzes et al., 2024). CAT-Net is parameter heavy, with CAT-nnU-Net reported at $k$9 parameters and CAT-nnU-Net++ at $v$0, even though its convolutions remain 2D (Hung et al., 2022). SliceMoE depends on efficient fused batched GEMM kernels and is sensitive to the number of slices $v$1, top-$v$2, capacity-loss weight $v$3, and cross-slice dropout rate (Vejendla, 5 Oct 2025). Sliceformer is focused on discriminative tasks and inherits the nondifferentiability of sorting, while the authors report only 64.77% top-1 on full ImageNet for their variant (Yuan et al., 2023).

Theoretical results also bound how far slice-depth factorization can go. The graph analysis of (Yehudai et al., 3 Mar 2025) shows that shallow wide transformers can solve many tasks once width becomes linear or super-linear, but Eulerian cycle verification cannot be solved with $v$4 unless depth is $v$5, assuming the cited conjecture. This implies that some global computations cannot be collapsed into a tiny number of depth slices without near-quadratic width (Yehudai et al., 3 Mar 2025).

The most immediate open direction is extension along additional ordered axes. The cardiac MRI work explicitly notes applicability to other multi-slice CMR such as LGE and suggests extension to temporal or spatiotemporal correlations by attention along time, or jointly along time and slice axes (Chen et al., 2023). RD-ViT already shows that recurrent-depth formulations can support both 2D and 3D inputs, adaptive per-patch halting, MoE specialization, and depth extrapolation at inference without degradation (He, 5 May 2026). A plausible implication is that future slice-depth transformers will increasingly combine multiple axes—slice, time, channel, and recurrent depth—within a single factored attention system rather than choosing only one.