SAP: Syntactic Attention Pruning Overview

• SAP is a statistical framework that partitions candidate elements into syntactic categories (e.g., tokens, genes) to inform attention modulation and hypothesis testing.
• It quantifies green-enrichment and red-depletion using one-sided tests and combines signals with methods like weighted Fisher’s for robust detection.
• Practically, SAP underpins applications in LLM watermarking, spatial omics, and intersection analysis, enhancing sensitivity and computational efficiency.

Syntactic Attention Pruning (SAP) is a methodological class that exploits syntactic or semantic partitions within a structured candidate set—such as tokens, objects, or neighborhoods—to modulate or discard attention, draws, or associations, with explicit statistical controls for enrichment and depletion. SAP underpins diverse tasks, including watermark detection in generative models, spatial omics neighborhood testing, and discrete object intersection analysis. By quantifying “green-enrichment” (overrepresentation in a syntactic class) and “red-depletion” (underrepresentation), SAP provides hypothesis tests to determine whether a structured sample departs significantly from a null model of random draws or assignments.

1. Conceptual Foundation

Syntactic Attention Pruning refers to a statistical paradigm in which candidate elements (e.g., tokens, genes, or spatial points) are partitioned into disjoint syntactic or semantic categories. The method executes or constrains attention, sampling, or analysis by referencing these categories, and applies univariate or composite statistical tests on the realized counts of “highlighted” (green-analogous) versus “excluded” (red-analogous) elements. Syntactic structuring can be fixed arbitrarily (e.g., vocabulary splits), data-driven (e.g., label-based neighborhoods), or derived from higher-order decompositions (e.g., graph, sequence, or hypergeometric intersections).

The essential SAP workflow:

• Partition the candidate space into green/yellow/red (or analogous) syntactic sets.
• At each instance, track membership and record observed counts.
• Under the null hypothesis (random, unmodulated draws), derive expectations and variances.
• Compute one-sided enrichment (upper-tail: green-enrichment) and depletion (lower-tail: red-depletion) statistics.
• Aggregate significance for hypothesis testing and interpretation.

2. Statistical Formulation in Key Domains

Syntactic Attention Pruning has been instantiated with precise statistical recipes in multiple domains.

A. Triple-Set Watermark Detection for LLMs

In HATS watermarking, SAP is realized through per-token partitioning into green/yellow/red via a pseudorandom keyed function. Only green and yellow tokens are permitted at each decoding step; red tokens are explicitly pruned. At detection, “green-enrichment” and “red-depletion” statistics for the decoded sequence are evaluated:

• Green-enrichment $z$-score:

$$z_G = \frac{\hat{p}_G - \gamma_g}{\sqrt{\gamma_g(1-\gamma_g)/L}}$$

• Red-depletion $z$-score:

$$z_R = \frac{\gamma_r - \hat{p}_R}{\sqrt{\gamma_r(1-\gamma_r)/L}}$$

• $p$-values are assessed as:

$p_G = 1-\Phi(z_G),\quad p_R = 1-\Phi(z_R)$

• Aggregation via weighted Fisher’s method:

$$S_\lambda = -2[\lambda\ln p_G + (1-\lambda)\ln p_R]$$

The method controls false-positive rate (FPR) by thresholding $S_\lambda$ against the $\chi_4^2$ distribution, with sliding window corrections and Poisson–Binomial generalization for non-iid steps (Hu et al., 22 Dec 2025).

B. Spatial Omics Neighborhood Enrichment

SAP is operationalized as the neighborhood enrichment test:

• The adjacency matrix $W$ and label vectors define neighborhood relations.
• The expected green–red neighbor-pair count under null (with-replacement draw):

$$\mu = n_{\text{green}}\,E[y],\quad E[y]=\frac{1}{N}\sum_{i=1}^N y_i$$

• Variance:

$$\sigma^2 = n_{\text{green}}\left[ \frac{1}{N}\sum_{i=1}^N y_i^2 - (E[y])^2 \right]$$

• $z$-score:

$z_{g,r} = \frac{o_{g,r} - \mu}{\sigma}$

• $p$-values for both one-sided (green/upper, red/lower) and two-sided tests are derived using the normal CDF (Andersson et al., 23 Jun 2025).

C. Hypergeometric Intersection Analysis

When sampling without replacement from $N$ urns, intersection statistics test enrichment/depletion:

• PMF:

$$P(X=v \mid a_1, \ldots, a_N, n) = \text{[explicit nested-sum formula]}$$

• Enrichment $P$-value (upper-tail): $P(X \geq x_{\text{obs}})$
• Depletion $P$-value (lower-tail): $P(X \leq x_{\text{obs}})$ Implementation is available in the R package ‘hint’ (Kalinka, 2013).

3. Theoretical Properties and Test Power

SAP methodology is grounded in distributional theory for sums of Bernoulli, Poisson–Binomial, or intersection counts, justified by the Central Limit Theorem (CLT) or exact enumeration (hypergeometric/hint).

Key theoretical points:

• For large $L$ (text length, neighborhood size), test power increases as $1/\sqrt{L}$.
• Variance and mean under the null model can be adjusted for non-uniform or correlated draw probabilities (e.g., Poisson–Binomial, spatial autocorrelation).
• Fisher’s method for combining complementary signals (green-enrichment and red-depletion) improves detection sensitivity by leveraging joint tail probabilities, effectively increasing the degrees of freedom (as in $\chi^2_4$ in HATS) (Hu et al., 22 Dec 2025).

4. Exemplary Applications

Domain	Syntactic Set Structure	Pruning Mechanism / Test
LLM watermarking (HATS)	Green/Yellow/Red tokens	Generation bias and sampling ban
Spatial omics	Point labels/clusters	Analytical z-score for neighbor
Intersection analysis	Category/urn membership	Hypergeometric/Bernoulli sums

A. LLM Watermarking

HATS deploys SAP to modulate the output space, achieving empirical TPR ≃ 62% at FPR ≃ 0.5% for L∼250, γ_g≈0.2, γ_r≈0.02, outperforming two-set schemes. The red-depletion test doubles Fisher’s degrees of freedom, strengthening statistical tail behavior (Hu et al., 22 Dec 2025).

B. Spatial Omics

SAP accelerates enrichment/depletion tests over brute-force Monte Carlo (10–70× faster for large N), with high fidelity (Pearson r > 0.95 vs MC, N_MC=128). Analytical $z$-score is robust for moderate n_g, but variance inflation and CLT breakdown can arise for rare/extreme label counts (Andersson et al., 23 Jun 2025).

C. Intersection Testing

SAP underpins discrete set-feature overlap analysis, e.g., gene lists or colocalization in imaging. Enrichment/depletion $P$-values provide stronger conclusions for clustering or association than raw intersection sizes (Kalinka, 2013).

5. Boundary Conditions and Interpretative Caveats

SAP relies on explicit null models (with/without replacement, spatial independence, pseudorandom partitioning), and test calibration can degrade in various cases:

• For rare labels or extreme sample sizes, normal approximation is inaccurate; exact enumeration is necessary when feasible (Andersson et al., 23 Jun 2025, Kalinka, 2013).
• In LLM watermarking, top-k/nucleus sampling or special-token masking induces non-uniform nulls; Poisson–Binomial corrections are recommended (Hu et al., 22 Dec 2025).
• For spatial omics, real tissue may violate the assumption of random label permutations; SAP assesses deviation from spatial randomness, not tissue-matched nulls (Andersson et al., 23 Jun 2025).
• Intersection tests generalize to duplicate/asymmetric object sampling but require additional combinatorial handling (Kalinka, 2013).

6. Impact and Generalization

Syntactic Attention Pruning delivers robust, interpretable statistical control wherever syntactical (or categorical) structure can be partitioned and sampled. Its high statistical power for both enrichment and depletion, combined with scalable analytical implementations, provides a framework extendable to:

• Language generation auditing and watermarking (Hu et al., 22 Dec 2025)
• Spatial association testing in high-throughput imaging and omics (Andersson et al., 23 Jun 2025)
• Gene list and trait overlap assessment, image colocalization, discrete multisets (Kalinka, 2013)

A plausible implication is that SAP can be generalized across modalities and sampling paradigms wherever the syntactic class structure is well posed and the null model fully characterized. Continued work is required to extend SAP for rare-event regimes, dependent structures, and hybrid sampling protocols.