CoSiNE: Cosine-Based ML Paradigm

• CoSiNE is a family of cosine-based formulations in machine learning that leverage angular similarity to decouple magnitude as a nuisance variable.
• It applies cosine normalization across various architectures, improving tasks like hyperspectral classification, sparse autoencoders, and LLM serving.
• Empirical results demonstrate significant gains in latency, throughput, and energy efficiency, validating its use in diverse ML and hardware systems.

CoSiNE denotes a family of cosine-based formulations in contemporary machine learning and numerical analysis rather than a single universally fixed algorithm. In the literature surveyed here, the label appears both as a named system—most explicitly CoSine, “Collaborative Speculative Inference for Efficient LLM Inference Serving”—and as a shorthand for cosine-normalized scoring, cosine-based routing, or cosine-defined geometry in attention, sparse coding, graph construction, and optimization. The common thread is the replacement of magnitude-sensitive Euclidean or dot-product scoring with angular or cosine-based structure, often to improve robustness, specialization, or computational efficiency (Gao et al., 13 Mar 2025, Ahmad et al., 2 Apr 2026, Naihin et al., 13 Jun 2026).

1. Terminological scope and unifying idea

A recurring CoSiNE principle is that similarity should often depend on direction rather than raw magnitude. In standard attention and neural pre-activations, the dot product

$$\mathbf{q}^\top \mathbf{k}=\|\mathbf{q}\|_2\|\mathbf{k}\|_2\cos\theta$$

mixes magnitude and orientation. Several works in the present corpus argue that this is suboptimal when magnitude is a nuisance variable, such as illumination in hyperspectral signatures, norm inflation under BatchTopK sparse autoencoders, or activation-scale variation in generic neural layers (Ahmad et al., 2 Apr 2026, Naihin et al., 13 Jun 2026, Luo et al., 2017).

This geometric reframing has multiple concrete realizations. In hyperspectral attention, queries and keys are projected onto a unit hypersphere and scored by squared cosine similarity,

$$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$

with $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$ and $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$ (Ahmad et al., 2 Apr 2026). In cosine-scored sparse autoencoders, the encoder replaces inner-product scoring with

$$s_i(x)=\exp(a\log\|x_c\|+b)\,\cos(x_c,w_i)+b_{\mathrm{enc},i},$$

so that norm dependence is learned rather than hard-coded (Naihin et al., 13 Jun 2026). In cosine normalization for neural networks, the pre-activation itself is replaced by cosine similarity or centered cosine similarity,

$$\frac{\vec{w}^\top \vec{x}}{\|\vec{w}\|\,\|\vec{x}\|}, \qquad \frac{(\vec{w}-\mu_w)^\top(\vec{x}-\mu_x)}{\|\vec{w}-\mu_w\|\,\|\vec{x}-\mu_x\|},$$

thereby bounding neuron inputs in $[-1,1]$ (Luo et al., 2017).

Theoretical work sharpens this view. When embeddings are constrained to the unit sphere $\mathbb{S}^{d-1}$, cosine distance becomes exactly half the squared Euclidean distance,

$$d_C(\mathbf{x},\mathbf{y})=\frac{1}{2}d_E^2(\mathbf{x},\mathbf{y}),$$

and the diagonal gauge ambiguity identified for unconstrained matrix factorization embeddings vanishes (Bouhsine, 23 Feb 2026). This suggests that many CoSiNE formulations are best understood as geometry alignment: they make the scoring rule match the invariances of the underlying representation.

2. Cosine geometry as an inductive bias

In the works surveyed here, cosine geometry is not treated as a cosmetic normalization step but as an inductive bias with domain-specific meaning. Hyperspectral image classification provides the clearest physical motivation: practitioners often use angular similarity because illumination, atmospheric effects, and sensor gain alter spectral magnitude, while the direction or shape of the spectrum is more indicative of material identity. Cosine-normalized attention therefore measures angular overlap on the unit hypersphere rather than Euclidean inner products in $\mathbb{R}^D$, and controlled comparisons within a fixed spatial–spectral Transformer show that cosine-based scoring consistently ranks among the best variants, with squared cosine often best or near-best (Ahmad et al., 2 Apr 2026).

The same logic appears in mechanistic-interpretability SAEs. Standard SAE scores scale with both directional alignment and input norm, so under BatchTopK high-norm tokens inflate all pre-activations simultaneously and claim dictionary slots regardless of content alignment. CoSiNE corrects this by making feature selection primarily directional, with a learned power-law norm factor $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$0 rather than a fixed linear dependence. Training is free to recover inner-product scoring with $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$1, but in the reported experiments it never does; no feature chooses more than half-magnitude dependence, and shallow layers converge close to pure cosine (Naihin et al., 13 Jun 2026).

A related but more general argument appears in “In Defense of Cosine Similarity.” The paper separates two claims often conflated in practice: cosine similarity on arbitrary learned embeddings can be arbitrary under gauge freedom, but cosine geometry on normalized embeddings is well-defined. The stated conclusion is that “the ‘problem’ with cosine similarity is not cosine similarity, it is the failure to normalize” (Bouhsine, 23 Feb 2026). This directly contextualizes why CoSiNE-like mechanisms frequently normalize first and score second.

3. Architectural realizations

Cosine-based design appears in multiple neural architectures with different computational roles.

In Transformers, cosine scoring can modify only the attention score computation while keeping multi-head parallelism, softmax, value aggregation, residuals, and MLPs unchanged. In the hyperspectral Transformer studied in (Ahmad et al., 2 Apr 2026), the implementation inserts row-wise $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$2 normalization for $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$3 and $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$4 and replaces $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$5 with $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$6. In “Cottention,” cosine attention is pushed further: softmax is replaced by row-wise cosine similarity,

$$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$7

and associativity is exploited to obtain native linear memory complexity with respect to sequence length, plus an RNN reformulation with finite hidden state for causal inference (Mongaras et al., 2024).

In residual networks, cosine can define dynamic routing rather than attention. CosineGate computes the Cosine Incompatibility Ratio,

$$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$8

between identity and residual features, and uses it as a self-supervised skip signal for per-sample, per-block gating. Low CIR implies redundancy and favors skipping; high CIR implies novelty and favors executing the residual branch. Gumbel-Softmax relaxation makes the gates differentiable during training, and hard thresholding at inference yields deterministic per-input computation graphs (Thota, 21 Dec 2025).

In graph neural networks for speech emotion recognition, cosine defines graph topology. CoGCN constructs a graph whose nodes are frame-level acoustic features and whose adjacency is thresholded cosine similarity: $$\mathbf{A}_h^{\text{CoSiNE}}=\left(\tilde{\mathbf{Q}}_h\tilde{\mathbf{K}}_h^\top\right)^{\odot 2},$$9 A GCN with skip connections then aggregates primarily among acoustically similar frames, which is reported to improve robustness to vacuum regions and noise (Kim et al., 2022).

In generic feed-forward and convolutional networks, cosine normalization replaces the raw dot product at every neuron. Because the resulting pre-activation is bounded in $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$0, the formulation targets large activation variance, sensitivity to input scaling, and internal covariate shift without relying on batch statistics or extra hidden-layer scale parameters (Luo et al., 2017).

4. Systems and hardware instantiations

The name CoSine is used most directly as a system identifier in large-language-model serving. “Collaborative Speculative Inference for Efficient LLM Inference Serving” decouples drafting and verification across different machines, routes requests to specialized drafters based on expertise, fuses drafter outputs through a confidence-based token fusion mechanism, and dynamically orchestrates drafting and verification in a pipelined manner with batch scheduling and adaptive speculation control. Under equivalent resource costs, the reported gains reach up to a 23.2% decrease in latency and a 32.5% increase in throughput relative to baseline speculative approaches (Gao et al., 13 Mar 2025).

A separate hardware realization appears in COSIME, a FeFET-based associative-memory engine for in-memory cosine similarity search. COSIME reformulates cosine search in a current-mode analog pipeline: FeFET 1FeFET1R cells realize binary dot products in memory, translinear analog circuitry computes a quantity proportional to $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$1, and winner-take-all circuitry selects the maximum-similarity row. At the array level, the reported improvements are 333X in latency and 90.5X in energy relative to an approximate cosine-search associative memory; for an HDC problem, the reported average gains relative to a GPU implementation are 47.1X speedup and 98.5X energy-efficiency improvement (Liu et al., 2022).

These two systems represent different ends of the CoSiNE spectrum. CoSine is a distributed serving architecture using cosine-derived confidence and collaboration at the system level; COSIME is a device-level realization of cosine similarity search in non-volatile memory. The shared feature is that cosine is treated as a primitive to be exploited architecturally rather than merely evaluated numerically.

5. Empirical record across application domains

The empirical literature attached to the CoSiNE label is heterogeneous but unusually broad. It spans hyperspectral remote sensing, LLM serving, sparse representation learning, speech emotion recognition, image super-resolution, classification hardware, and classical neural-network normalization.

Domain	Cosine-based formulation	Reported outcome
Hyperspectral image classification	Cosine-normalized attention with squared cosine scores	OA up to $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$2 on Salinas; cosine variants match or exceed recent Transformer and Mamba baselines (Ahmad et al., 2 Apr 2026)
Sparse autoencoders	Learned norm-scaled cosine encoder score	Top-1 sparse probing improves from 0.667 to 0.800 and 0.815 at matched FVE $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$3 (Naihin et al., 13 Jun 2026)
Speech emotion recognition	Cosine similarity-based graph construction in CoGCN	WA 62.64 and UA 63.67 with 56K parameters; competitive with CA-GRU using roughly 1/30 parameters (Kim et al., 2022)
Generic neural networks	Cosine normalization / centered cosine similarity	Better performance than batch, weight, and layer normalization on MNIST, 20NEWS GROUP, CIFAR-10/100, and SVHN (Luo et al., 2017)
Image super-resolution	Heterogeneous CNN with cosine annealing schedule	Competitive with state-of-the-art; e.g. Set14 $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$4: 34.12 / 0.9216 (Tian et al., 23 Jan 2026)
KNN on covariance-structured data	Variance-adjusted cosine similarity	Modified formula shows 100% test accuracy on the Wisconsin Breast Cancer Dataset (Sahoo et al., 4 Feb 2025)
LLM serving	Collaborative speculative inference with confidence-based fusion	Up to 23.2% lower latency and 32.5% higher throughput (Gao et al., 13 Mar 2025)

This empirical pattern suggests that cosine-based formulations are especially attractive when one or more of the following hold: magnitude is a nuisance factor, representations are already normalized or effectively spherical, similarity search dominates runtime, or resource allocation benefits from angular confidence signals. That conclusion is interpretive, but it is consistent with the domains in which gains are reported.

6. Mathematical generalizations

Beyond machine learning systems, the surveyed literature develops more formal notions of cosine structure.

In derivative-free optimization, the cosine measure quantifies the richness of direction sets. The 2024 extension “The cosine measure relative to a subspace” defines

$$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$5

for a nontrivial subspace $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$6 and a possibly infinite set of directions $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$7. A central equivalence is

$$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$8

and the paper proposes a deterministic algorithm to compute $$\tilde{\mathbf{Q}}_h=\mathbf{Q}_h/\|\mathbf{Q}_h\|_2$$9 for finite $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$0 (Audet et al., 2024). This extends CoSiNE-style cosine reasoning from similarity scoring to the geometry of positive spanning sets and subspace search.

In harmonic analysis, cosine appears as a manifestation of Banach-algebra spectra. “Cosine manifestations of the Gelfand transform” introduces the cosine convolution

$$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$1

establishes a bijection between $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$2 and a cosine class $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$3, and shows that the associated Gelfand transform reduces, in canonical cases, to the continuous cosine transform, discrete-time cosine transform, cosine Fourier coefficients on $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$4, and DCT-type transforms on $$\tilde{\mathbf{K}}_h=\mathbf{K}_h/\|\mathbf{K}_h\|_2$$5 (Krukowski, 2021).

Theoretical work on embedding geometry and covariance structure adds two further qualifications. First, normalized embeddings remove gauge ambiguity and make cosine and Euclidean neighbor rankings identical on the unit sphere (Bouhsine, 23 Feb 2026). Second, when data reside in a random-variable space with nontrivial variance and covariance, raw cosine may be misleading; the proposed variance-adjusted cosine instead computes cosine after a whitening transform induced by the covariance matrix, effectively producing a Mahalanobis-aware angular similarity (Sahoo et al., 4 Feb 2025).

7. Limits, ambiguities, and open questions

A persistent misconception is that CoSiNE denotes a single method. In the present literature it does not. It can refer to a speculative inference system, a cosine-normalized attention score, a cosine-scored SAE encoder, a cosine-normalized neuron, a subspace-relative richness measure, or a hardware cosine-search engine. The commonality is geometric, not architectural.

Another misconception is that cosine is universally preferable to magnitude-sensitive scoring. Multiple papers explicitly qualify this. The hyperspectral attention paper states that the method is tailored to hyperspectral data and notes that in domains where magnitude carries crucial semantic information, pure cosine similarity might discard useful cues; it also notes that squared cosine loses sign information (Ahmad et al., 2 Apr 2026). The cosine-scored SAE paper reports that the advantage is not universal across tasks or depths, that sentiment is a reversal case, and that LayerNorm models can shrink or reverse the advantage because LayerNorm preserves a different kind of magnitude signal (Naihin et al., 13 Jun 2026). The variance-adjusted cosine paper likewise argues that traditional cosine is valid only in Euclidean space, whereas original data may reside in a random-variable space with variance and correlation that must be modeled explicitly (Sahoo et al., 4 Feb 2025).

System-level CoSiNE formulations have their own constraints. CoSine assumes a multi-node, heterogeneous serving environment with enough concurrency for pipeline balancing to matter, while COSIME currently targets binary vectors and analog in-memory realization (Gao et al., 13 Mar 2025, Liu et al., 2022). Architectural formulations also leave open scale questions: hyperspectral cosine attention is evaluated on three benchmarks with a lightweight backbone rather than very large-scale models; cosine-scored SAEs raise open questions about interaction with alternative sparsity selectors and behavior beyond the examined models; and linear cosine attention still faces tradeoffs against softmax on some benchmarks (Ahmad et al., 2 Apr 2026, Naihin et al., 13 Jun 2026, Mongaras et al., 2024).

What persists across these caveats is a narrower, defensible statement: CoSiNE methods are most coherent when the task geometry, data invariances, or system bottlenecks genuinely favor angular structure. Where that condition holds, cosine-based formulations can function as a principled inductive bias, a scalable computational primitive, or a mathematically natural transform. Where it does not, the literature itself recommends caution rather than universal adoption.