Efficient hardware realization of tensorized layers

Develop kernel-level implementations that make tensorized neural-network layers execute efficiently on modern GPU hardware, thereby translating parameter and FLOP reductions into actual speedups relative to dense general matrix multiplication.

Background

Tensorized layers replace large dense matrix multiplications with chains of smaller tensor contractions. Although this can reduce parameter counts and theoretical arithmetic, modern GPUs are optimized primarily for large, regular GEMM operations, while sequential or irregular contractions may suffer from launch overhead, poor utilization, and costly intermediate materialization.

The paper identifies the resulting compression-realization gap as a central systems problem. Fused kernels, structure-aware kernels, and optimized Triton or CUDA implementations are suggested as possible approaches, with evaluation based on measured end-to-end speedup rather than theoretical compression alone.