WebGPU-based deployment of Tiny-QMoE beyond terminal-only usage

Develop a WebGPU-based deployment that enables Tiny-QMoE—a quantization and dictionary-based compression framework for LLaMA 3.2 models—to run outside of a terminal environment, thereby making the system publicly accessible beyond terminal-only execution.

Background

Tiny-QMoE introduces an approach to run quantized and compressed LLaMA 3.2 models on memory-constrained devices, prioritizing hardware-agnostic CPU execution and avoiding CUDA dependencies to broaden accessibility.

Despite demonstrating strong compression rates and maintaining model performance, the current implementation remains terminal-bound. The authors explicitly note their inability to provide a WebGPU-based deployment, which would allow running the system in browser environments and make the work more publicly accessible beyond terminal usage.