Multidimensional Task Learning: A Unified Tensor Framework for Computer Vision Tasks

Abstract: This paper introduces Multidimensional Task Learning (MTL), a unified mathematical framework based on Generalized Einstein MLPs (GE-MLPs) that operate directly on tensors via the Einstein product. We argue that current computer vision task formulations are inherently constrained by matrix-based thinking: standard architectures rely on matrix-valued weights and vectorvalued biases, requiring structural flattening that restricts the space of naturally expressible tasks. GE-MLPs lift this constraint by operating with tensor-valued parameters, enabling explicit control over which dimensions are preserved or contracted without information loss. Through rigorous mathematical derivations, we demonstrate that classification, segmentation, and detection are special cases of MTL, differing only in their dimensional configuration within a formally defined task space. We further prove that this task space is strictly larger than what matrix-based formulations can natively express, enabling principled task configurations such as spatiotemporal or cross modal predictions that require destructive flattening under conventional approaches. This work provides a mathematical foundation for understanding, comparing, and designing computer vision tasks through the lens of tensor algebra.

• The paper introduces Multidimensional Task Learning, representing computer vision tasks as tensor configurations that specify contracted and preserved dimensions rather than separate architecture families.
• The framework uses Generalized Einstein MLPs and tensor-valued gradients to recover classification, dense prediction, segmentation, and YOLO-style detection through shared computations, with structure preservation ranging from 0.33 to 1.0.
• The paper proposes temporal, volumetric, and spatiotemporal task configurations, but provides no experiments, leaving practical efficiency, learnability, and comparisons with convolutional architectures open.

Overview and motivation

"Multidimensional Task Learning: A Unified Tensor Framework for Computer Vision Tasks" (Elichi and Jbilou) proposes a formulation-level unification of computer vision tasks built on tensor algebra. The authors observe that classification, semantic segmentation, and object detection operate on the same tensor-structured data but are conventionally treated as architecturally distinct problems, each with dedicated networks (ResNet-style classifiers, FCNs, YOLO), losses, and training procedures. They attribute this fragmentation to what they call matrix-based thinking: standard layers manipulate matrix-valued weights and vector-valued biases, forcing intermediate flattening of tensor data and implicitly restricting which tasks can be expressed without structural information loss.

The paper's response is Multidimensional Task Learning (MTL), a framework in which a task is identified with a dimensional configuration rather than an architecture family. The computational substrate is the Generalized Einstein MLP (GE-MLP), a multilayer perceptron whose weights and biases are tensors combined with inputs via the Einstein product (2602.23217). The central claim is that classification, segmentation, and detection are special cases of MTL differing only in which dimensions of the data are contracted versus preserved.

Generalized Einstein MLPs

A GE-MLP layer takes an input tensor $$\mathcal{X}^{(\ell)} \in \mathbb{R}^{I_1 \times \cdots \times I_N \times J_1 \times \cdots \times J_M}$$, where the $I$-modes are contracting dimensions (e.g., channels or features) and the $J$-modes are preserved dimensions (e.g., spatial positions or batch). The forward pass is

$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$

with weight tensor $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$ mapping contracted modes to new output modes $K_1,\dots,K_P$, and bias tensor $\mathcal{B}^{(\ell)}$ aligned to both output and preserved modes. The number of output modes $P$ is explicitly user-selected, which is precisely the mechanism by which task structure is controlled. The authors pair this architecture with Generalized Einstein Gradient Descent (GEGD), giving elementwise gradient and update rules for the tensor-valued parameters that mirror standard backpropagation for MLPs.

The complexity analysis is straightforward but important for the paper's claim of parity with specialized architectures. A layer costs $$O\!\left(\prod_i I_i \cdot \prod_m J_m \cdot \prod_p K_p\right)$$ in compute and FLOPs ($2\times$ that product), with memory dominated by weight plus bias tensors. The paper asserts that GE-MLPs achieve computational complexity identical to specialized architectures while eliminating flattening; this claim is argued analytically from contraction counts rather than demonstrated empirically, since the paper contains no experiments.

Formalizing tasks as tuples

The MTL formalism defines a task configuration as a tuple $I$0: the number of output contracting dimensions, the number of preserved dimensions, the loss function, and an output interpretation function (argmax, thresholding, etc.). A multidimensional task is then a map $I$1 producing outputs in $I$2, so each preserved position receives predictions over the $I$3-modes. The framework also introduces a structure preservation index

$I$4

where $I$5 indicates complete contraction and $I$6 complete preservation. This index is the paper's main quantitative device for comparing tasks: it collapses the architectural differences among tasks into a single scalar describing how much input structure survives processing.

Unification theorems

The theoretical core consists of recovery theorems establishing that canonical tasks instantiate specific MTL configurations.

Classification: with $I$7, $I$8 classes and only the batch dimension preserved, the GE-MLP forward pass reduces exactly to flatten-then-linear-softmax classification, with $I$9 when measured against $J$0.

Dense classification and segmentation: with $J$1 preserving $J$2 and contracting only channels, the model performs per-position softmax classification with shared bias, mathematically equivalent to a fully convolutional network with $J$3. Notably, the segmentation theorem states that semantic segmentation is identical to dense classification at this level of abstraction—the distinction is purpose (spatial coherence priors) rather than structure. This is one of the paper's bolder claims; it deliberately abstracts away the spatial-coherence modeling (e.g., atrous convolutions, CRF postprocessing) that distinguishes practical segmentation systems (2602.23217).

Detection: YOLO-style grid detection corresponds to $J$4 with $J$5 for box coordinates, objectness, and class probabilities over grid $J$6, trained with the standard weighted multi-task loss combining MSE, binary cross-entropy, and categorical cross-entropy, followed by thresholding and non-maximum suppression. The proof reproduces YOLO's per-cell multi-modal prediction exactly.

The culminating unification theorem asserts that all these tasks share the same computational mechanism (GE-MLPs via Einstein product) and differ only in $J$7, i.e., in $J$8. Tasks thus become points in a formally defined space $J$9:

Task	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$0	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$1	Output modes $$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$2	Preserved $$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$3	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$4
Classification	1	1	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$5	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$6	0.33
Dense classification	1	3	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$7	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$8	1.0
Segmentation	1	3	$$\mathcal{Y}^{(\ell)} = f\left(\mathcal{W}^{(\ell)} \ast_N \mathcal{X}^{(\ell-1)} + \mathcal{B}^{(\ell)}\right)$$9	$$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$0	1.0
Detection	3	3	$$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$1	$$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$2	1.0

Novel task configurations

Because $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$3 ranges over all natural numbers $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$4 and $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$5, the paper argues it strictly contains the space of tasks naturally expressible by matrix-based layers. Conventional architectures require destructive flattening whenever multiple structural dimensions must be simultaneously maintained, collapsing structural information irrecoverably; GE-MLPs avoid this by design. The discussion enumerates concrete configurations outside common practice, including temporal classification $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$6, spatiotemporal hierarchical prediction $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$7, 3D volume segmentation $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$8, and 4D spatiotemporal detection $$\mathcal{W}^{(\ell)} \in \mathbb{R}^{K_1 \times \cdots \times K_P \times I_1 \times \cdots \times I_N}$$9. The implication stated here is significant: within this framework, designing a new task reduces to selecting a tuple, rather than engineering a new architecture, loss, and pipeline from scratch. However, these novel configurations are proposed as mathematical possibilities; the paper does not train models on any of them, so their practical learnability and utility remain open.

Limitations and open questions

Several caveats bear directly on the results. First, the equivalence proofs operate at the level of dense linear layers: the recovery of classification, segmentation, and detection uses GE-MLPs without convolutional weight sharing, so "exactly recovers" means functional equivalence to a flattened linear head applied per position, not equivalence to full modern architectures (ResNets, U-Nets, YOLO backbones). Second, the claim that GE-MLP complexity matches specialized architectures is analytical; no benchmarks, parameter counts under realistic dimension choices, or comparisons to convolutional parameter sharing are provided. Third, GEGD is described elementwise for plain gradient descent—extensions to stochastic mini-batch settings, adaptive optimizers, and deep-network backward-pass propagation through multiple GE-MLP layers are not derived. Fourth, the strict-superset claim about $K_1,\dots,K_P$0 relative to matrix-expressible tasks is asserted rather than proven with a formal expressivity argument. Finally, whether the enumerated novel tuples correspond to well-posed learning problems with available supervision is left entirely open.

Conclusion

The paper contributes a clean algebraic reframing in which computer vision tasks are dimensional configurations of a single tensor-native computation, quantified by a structure preservation index and organized in an explicit task space. Its recovery theorems for classification, dense classification/segmentation, and grid-based detection are correct at the level of generality stated, and the framework offers a principled vocabulary for proposing structured prediction problems—particularly spatiotemporal and multi-modal ones—that sit awkwardly in matrix-based pipelines. The main gap between formalism and practice is empirical: demonstrating that GE-MLPs trained end-to-end can match specialized architectures, and validating the proposed unexplored task configurations, remain the immediate open questions raised by this work.