MiniTensor: A Lightweight, High-Performance Tensor Operations Library

Abstract: We present MiniTensor, an open source tensor operations library that focuses on minimalism, correctness, and performance. MiniTensor exposes a familiar PyTorch-like Python API while it executes performance critical code in a Rust engine. The core supports dense $n$ dimensional tensors, broadcasting, reductions, matrix multiplication, reverse mode automatic differentiation, a compact set of neural network layers, and standard optimizers. In this paper, we describe the design of MiniTensor's architecture, including its efficient memory management, dynamic computation graph for gradients, and integration with Python via PyO3. We also compare the install footprint with PyTorch and TensorFlow to demonstrate that MiniTensor achieves a package size of only a few megabytes, several orders of magnitude smaller than mainstream frameworks, while preserving the essentials needed for research and development on CPUs. The repository can be found at https://github.com/neuralsorcerer/minitensor

• The paper introduces MiniTensor, a lightweight tensor operations library that combines a Rust execution engine with a PyTorch-like Python API, delivering a deep learning stack in a 2.6 MB binary distribution.
• The library achieves competitive CPU performance with smaller footprint making it suited for environments where resource efficiency is crucial.
• MiniTensor's design prioritizes minimalism, correctness, and performance, omitting GPU support but providing critical tools to a range of users needing an efficient, auditable deep learning solution.

Overview

MiniTensor, presented by Soumyadip Sarkar (2602.00125), is an open-source tensor operations library that combines a PyTorch-like Python API with a Rust execution engine. The library's stated design goals are minimalism, correctness, and performance, and its central contribution is demonstrating that a functional deep learning stack—dense $n$-dimensional tensors, broadcasting, reductions, matrix multiplication, reverse-mode automatic differentiation, a compact set of neural network layers, and standard optimizers—can be delivered in a binary distribution of only a few megabytes. This stands in sharp contrast to mainstream frameworks whose wheels routinely exceed hundreds of megabytes.

Motivation and Positioning

The paper's motivation rests on the observation that production frameworks such as PyTorch and TensorFlow are extremely complex and heavyweight, consisting of millions of lines of code and large binary footprints that complicate understanding, maintainability, auditing, and deployment in constrained environments. MiniTensor targets the middle ground between these systems and ultra-minimal pure-Python projects such as micrograd and tinygrad. Those minimalist Python libraries trade performance for simplicity: due to interpreter overhead and the GIL, their execution can be orders of magnitude slower than optimized C++ or Rust implementations. By implementing the core in Rust while preserving an eager, dynamic-graph API inspired by PyTorch, MiniTensor aims to retain the accessibility of minimal frameworks without their performance penalty.

The paper is candid about where MiniTensor does not compete: it explicitly concedes that in terms of optimizations and performance, both PyTorch and TensorFlow currently outperform it. The claim to "high performance" is therefore scoped to competitive constant factors on CPU workloads rather than parity with production frameworks.

Architecture

MiniTensor uses a three-layer design: a Python API, a PyO3 bindings layer, and a Rust execution engine.

Tensors and primitives. Tensors are stored as typed contiguous row-major buffers with lightweight shape/stride metadata. Matrix multiplication computes $Y = XW^\top$, elementwise operations apply pointwise maps, and reductions implement linear functionals such as sum and mean. Broadcasting follows NumPy and PyTorch semantics, expanding singleton dimensions virtually rather than materializing them—for example, adding a bias vector $\mathbf{b} \in \mathbb{R}^d$ across a batch dimension of $x \in \mathbb{R}^{b \times d}$ without copying.

Reverse-mode autodiff. During the forward pass, whenever a tensor requires gradients, the engine records a computation graph $\mathcal{G}$ whose nodes store parent references and local pullbacks mapping output cotangents to input cotangents. Gradients propagate through vector-Jacobian products $\bar{x} = \bar{y}\, J_f(x)$, composed via the chain rule for sequences of primitives. For scalar losses this yields all parameter gradients at time complexity proportional to a small constant multiple of the forward cost. The paper gives standard pullbacks for addition, Hadamard product, and matrix multiplication ($\bar{X} \mathrel{+}= \bar{Y} W$, $\bar{W} \mathrel{+}= \bar{Y}^\top X$). Gradient buffer allocation is deferred until the backward pass needs them.

Layers, losses, optimizers. Despite its size, the library provides dense layers, 2D convolution with stride and zero padding, ReLU/Sigmoid/Tanh/GELU activations, batch normalization with learnable scale and shift, dropout, cross-entropy and MSE losses, and SGD with momentum (including weight decay), Adam with debiased moment estimates, and RMSprop. These are mathematically standard formulations, so the contribution here is completeness within a small footprint rather than algorithmic novelty.

Bindings and engine. PyO3 bindings convert between Python objects and Rust buffers with zero-copy where possible, including views over compatible NumPy arrays, enabling interoperability with the scientific Python ecosystem. The Rust engine relies on ahead-of-time compilation, LLVM auto-vectorization of inner loops, portable SIMD abstractions dispatching to x86 or Arm vector instructions, and chunk-level parallelism for multi-core scaling.

Footprint Results

The strongest quantitative claim in the paper concerns package size, measured from official PyPI wheels:

Package (platform)	Version	Wheel size
MiniTensor (Linux x86_64)	0.1.1	2.6 MB
TensorFlow (Linux x86_64)	2.20.0	620.7 MB
PyTorch (Linux x86_64)	2.8.0	887.9 MB

At 2.6 MB against 620.7 MB for TensorFlow and 887.9 MB for PyTorch, MiniTensor's wheel is roughly two orders of magnitude smaller than either mainstream framework. The paper attributes this directly to design choices: a minimal kernel surface, reliance on Rust's standard library, and omission of bundled GPU backends from the default wheel. The practical implication is reduced download time, disk usage, and attack/audit surface, which matters for embedded deployment, education, and environments where full-scale frameworks are impractical. A caveat worth noting is that this comparison partly reflects scope: the large GPU backends excluded from MiniTensor's wheel are precisely what enables the distributed and accelerator capabilities the paper concedes it lacks.

Correctness Validation

Correctness is addressed through unit tests covering tensor arithmetic, broadcasting, autograd rules, and layer gradients, complemented by finite-difference checks on random inputs using central differences as a reference for edge cases and broadcasting semantics. End-to-end examples train small models and confirm consistent loss descent. Finite differences are acknowledged to be slow, so they serve as a validation reference rather than a runtime mechanism. Notably, the paper reports no systematic benchmark suite comparing gradient accuracy or training convergence against PyTorch on identical workloads; correctness evidence is qualitative and repository-based.

Comparison with Mainstream Frameworks

The comparison section is appropriately restrained. PyTorch and TensorFlow offer broad operator coverage, mature GPU/TPU backends, and distributed training stacks; MiniTensor explicitly does not attempt to replicate that scope. Its value proposition is a compact CPU core with an imperative autograd API mirroring common research workflows. The paper states plainly that users requiring very large models, extensive operator sets, or multi-device training should choose PyTorch or TensorFlow, positioning MiniTensor for users who prioritize small binaries, auditability, or teaching.

Limitations

The paper identifies several concrete limitations. MiniTensor supports only dense tensors of 32-bit floats and a practical subset of neural network primitives. GPU backends are marked as a roadmap item rather than an implemented feature, and advanced linear algebra and additional datatypes are left to future contributions. Optimizer loops execute at parameter granularity in Python, which may introduce overhead for massive models; the proposed mitigation—migrating these loops into batched Rust kernels—is described but not implemented or evaluated. More broadly, the paper offers no quantitative latency or throughput benchmarks against PyTorch or TensorFlow on CPU tasks, despite claiming "competitive constant factors," leaving that claim unsupported by reported measurements.

Conclusion

MiniTensor demonstrates that a Rust-backed engine behind PyO3 bindings can deliver reverse-mode automatic differentiation, standard neural network layers, and common optimizers through a familiar eager Python API in a 2.6 MB wheel—roughly two orders of magnitude smaller than PyTorch or TensorFlow distributions. The library's contribution is architectural and practical rather than algorithmic: it shows the essential functionality of a deep learning framework fits within a small, auditable codebase suitable for CPU research, education, and embedding. The open questions the paper leaves are whether its CPU constant factors can be rigorously benchmarked against mainstream frameworks, and whether GPU backend support can be added without eroding the footprint advantage that constitutes its primary result.