--- title: 'SeedLM Overview: Compression & Multimodal Modeling' url: https://www.emergentmind.com/topics/seedlm type: topic --- # SeedLM Overview: Compression & Multimodal Modeling SeedLM designates two distinct paradigms related to large language models (LLMs): (1) a post-training weight compression technique that encodes model weights as seeds for pseudo-random generators, and (2) a vision-language approach in which image data are discretized into causal semantic tokens, allowing unified text-image modeling under a Transformer architecture. The shared principle across both is the use of discrete seeds (code indices or pseudo-random generator initializations), either for weight reconstruction or multimodal content representation, enabling efficient, scalable, and modality-agnostic language model deployment [2410.10714, 2310.01218]. ## 1. Weight Compression via Seeds and Pseudo-Random Generators The SeedLM compression algorithm enables encoding large language model weights with minimal accuracy loss using only a tiny seed and quantized coefficients per block. For each block of weights, the process is as follows [2410.10714]: - Partition each weight matrix $W$ into blocks $w \in \mathbb{R}^C$. - For each block, select a seed $s \in \{1, ..., 2^K-1\}$ for a $K$-bit Linear Feedback Shift Register (LFSR). - LFSR, initialized with $s$, produces a deterministic integer matrix $V(s) \in \mathbb{N}^{C \times P}$, which is normalized to $U(s)$ so that entries are in $[-1,1]$. - Reconstruct each weight block by $\hat{w} = U(s)t$ where $t \in \mathbb{R}^P$ is a small quantized coefficient vector. - Store only $s$ (the seed), a shared exponent $e$ (4 bits), and $P$ 4-bit two’s-complement coefficients $q_1, ..., q_P$ for each block. Block selection, seed search, and coefficient quantization are performed offline. For $K=16$, exhaustive search is practical since there are only $2^{16}-1$ possible seeds per block. Each block’s optimal $(\hat{s}, \hat{t}^*)$ minimizes $\,||w - U(s)t||_2^2\,$ using Moore–Penrose pseudoinverse and quantization. The method achieves $3$–$4$ bits per weight—e.g., for $M=4$ bits, use $C=8$, $P=3$, $K=16$; for $M=3$, $C=12$, $P=4$, $K=16$. Importantly, this strategy is data-free: no calibration or activation statistics are required, in contrast to techniques such as AWQ, GPTQ, or OmniQuant [2410.10714]. ## 2. On-Chip Inference and Memory-Bound Acceleration During inference, SeedLM reconstructs each weight block on-the-fly via the lightweight LFSR (requiring just $K$ flip-flops and XOR gates), streaming out $P$ basis vectors, scaling by $q_j \cdot 2^e$, and summing. A 4-bit compressed model fits four times as many weights in a DRAM burst (128 vs. 32 per 64 B), dramatically reducing high-latency weight fetches from external memory. Idle DSP cycles, typically unavailable due to memory bottlenecks in 16-bit matmuls, are now utilized for basis generation and accumulation, adding minimal overhead. On hardware, such as FPGA (AMD Virtex7), SeedLM’s 4-bit model achieves nearly $4\times$ speedup in measured matrix–matrix multiplication throughput compared to an FP16 baseline (e.g., $2048 \times 2048$ matmul: $136,559$ cycles (FP16) vs. $34,331$ cycles (SeedLM 4-bit); $\approx 4\times$) with resource usage well below capacity constraints [2410.10714]. ## 3. Empirical Evaluation and Comparative Performance SeedLM is evaluated across Llama 2 (7B, 13B, 70B) and Llama 3 (8B, 70B) on zero-shot tasks (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, BoolQ) and perplexity benchmarks (WikiText-2, $2048$ seq-length): - 4-bit SeedLM retains $97$–$99\%$ of FP16 accuracy (e.g., Llama 3 70B FP16: $79.51$, SeedLM: $78.06$ average). - Competing 4-bit approaches (AWQ, OmniQuant, QuIP#) lose $4$–$10$ points or cannot run (out-of-memory) on large models. - 3-bit SeedLM outperforms or matches calibration-based 3-bit techniques (e.g., Llama 2 70B: SeedLM $73.83$, AWQ $73.91$, OmniQuant $59.72$). - Perplexity: Llama 3 70B FP16 ($2.9$), SeedLM 4-bit ($3.8$), AWQ ($4.7$), OmniQuant ($\infty$, OOM). - Resource cost for on-chip LFSR and conversion logic is modest compared to overall hardware utilization [2410.10714]. These results confirm that SeedLM enables drastic model compression without accuracy degradation and with no need for calibration data. ## 4. Multimodal Seed Tokenization and the Vision-Language Paradigm The SEED tokenizer, introduced in the context of SEED-LLaMA, discretizes images into a sequence of 1D causal, high-level semantic tokens ("SEED tokens"), which can be incorporated into LLMs' token streams identically to text. The tokenizer is VQ-based and optimized both for semantic alignment (contrastive InfoNCE loss with paired text) and accurate image reconstruction. 1D causal dependency is enforced using a Causal Q-Former, ensuring compatibility with left-to-right autoregressive modeling. The final distribution factorizes as $$ P(z_1, z_2, \dots, z_N) = \prod_{i=1}^N P(z_i \mid z_{