SLASH: Structural Attention Sharpening

• SLASH is a plug-and-play technique that sharpens attention maps by amplifying latent structural signals in serialized graphs.
• It reduces the attention sink effect by reallocating mass from initial tokens to topology-aware tokens, improving graph and molecular predictions.
• Empirical evaluations on GraphInstruct and MoleculeNet show that SLASH boosts accuracy in general LLMs without requiring parameter updates.

SLASH, short for StructuraL Attention SHarpening, is a training-free, plug-and-play attention redistribution technique for LLMs processing serialized graphs. It is designed to amplify latent structural signals that emerge inside intermediate attention maps when a graph $G=(V,E)$ is serialized into edge tuples and paired with a task query, so that the model computes $P(Y \mid S(G), Q)$. The central claim is that LLMs spontaneously reconstruct graph topology internally, but that this capability is diluted by the attention sink, a pre-trained bias toward allocating excessive attention to initial tokens; SLASH reduces that sink budget and redistributes the recovered mass toward non-sink tokens at inference time, improving graph reasoning and molecular prediction without parameter updates (Liu et al., 11 May 2026).

1. Problem setting and mechanistic motivation

The method addresses graph reasoning in LLMs under a serialized input format rather than via external graph encoders or task-specific fine-tuning. In the formulation studied, a graph is converted into edge tuples $(u,v)$ and fed to the model together with a task query $Q$, yielding predictions from $P(Y \mid S(G), Q)$ (Liu et al., 11 May 2026). To expose local structure in the token sequence, the paper introduces Source-Node Aggregation in serialization, where edges with the same source node are placed contiguously, creating spatial blocks in the sequence that correspond to local graph neighborhoods. An example given is that $(0,1)(2,3)(0,3)$ becomes $(0,1)(0,3)(2,3)$ (Liu et al., 11 May 2026).

The mechanistic observation motivating SLASH is that intermediate attention maps exhibit a distinct sawtooth pattern aligned with the graph structure implied by serialization. This alignment is formalized through a token-level adjacency matrix $M_{gt}$ defined by

$M_{gt}[i,j]=1$

if tokens $x_i$ and $P(Y \mid S(G), Q)$0 with $P(Y \mid S(G), Q)$1 belong to edge descriptions from the same source node (Liu et al., 11 May 2026). The attention maps structurally align with this matrix, indicating that the model is internally reconstructing graph neighborhoods even without special graph training (Liu et al., 11 May 2026).

The obstacle is the attention sink. In a Transformer, each attention row is Softmax-normalized so that

$P(Y \mid S(G), Q)$2

The model often allocates disproportionately high attention to initial tokens. The paper characterizes this as beneficial for semantic processing in natural language, but in conflict with the local aggregation needed for graph structure (Liu et al., 11 May 2026). SLASH is therefore framed as a way to sharpen an already existing internal capability rather than to implant a new one.

2. Representation bottleneck and formal model

The paper analyzes a causal decoder head through the update

$P(Y \mid S(G), Q)$3

Guided by the sawtooth pattern aligned with $P(Y \mid S(G), Q)$4, the update is decomposed into sink, structural, and residual components:

$P(Y \mid S(G), Q)$5

where

$P(Y \mid S(G), Q)$6

Empirically, the residual term carries a small fraction of attention mass, motivating the simplified approximation

$P(Y \mid S(G), Q)$7

(Liu et al., 11 May 2026).

This induces a zero-sum competition between sink attention and topology-aware aggregation. Writing $P(Y \mid S(G), Q)$8, the simplified model becomes

$P(Y \mid S(G), Q)$9

where

$(u,v)$0

The paper interprets $(u,v)$1 as a locally normalized, GAT-like aggregation term, while the sink term sustains the anisotropic bias characteristic of LLMs (Liu et al., 11 May 2026).

The resulting representation bottleneck is formalized in two ways. At the node level, the theorem labeled Geometric Contraction states that, assuming $(u,v)$2,

$(u,v)$3

As $(u,v)$4 grows, structurally different nodes become geometrically indistinguishable (Liu et al., 11 May 2026). At the graph level, with Dirichlet energy

$(u,v)$5

the proposition Dirichlet Energy Decay gives

$(u,v)$6

so the structural component is quadratically attenuated, acting as a low-pass filter on topology (Liu et al., 11 May 2026).

The conceptual conflict is explicit in the paper: semantic anisotropy in LLMs supports language processing, whereas graph reasoning requires topology-aware local aggregation analogous to MPNNs or GATs. SLASH targets that conflict directly (Liu et al., 11 May 2026).

3. Attention sharpening rule and inference-time intervention

SLASH operates by scaling down the sink entry in each attention row and redistributing the freed budget proportionally across non-sink tokens. With control factor $(u,v)$7, for a row $(u,v)$8 with sink weight $(u,v)$9 and non-sink weights $Q$0 for $Q$1, the sharpening rule is

$Q$2

$Q$3

This preserves normalization:

$Q$4

(Liu et al., 11 May 2026).

The paper gives two amplification guarantees. If $Q$5 denotes the sink weight, then the geometric expansion factor is

$Q$6

and the spectral amplification becomes

$Q$7

In the paper’s phrasing, this reverses contraction and acts as a controllable high-pass amplifier of topology (Liu et al., 11 May 2026).

The hyperparameter $Q$8 tunes the trade-off between structural amplification and semantic stability. The limiting regimes are explicit: $Q$9 maximizes amplification but risks representation collapse, while $P(Y \mid S(G), Q)$0 leaves the model unchanged (Liu et al., 11 May 2026). Calibration is performed per model-task pair on a small held-out set by scanning values in $P(Y \mid S(G), Q)$1 and selecting the best one (Liu et al., 11 May 2026).

SLASH is injected at inference time by intercepting attention matrices in topology-aware layers or heads and applying the sharpening rule. The paper reports that layer-level intervention, in which all heads in layers containing topology-aware heads are modified, is more stable and effective than head-level intervention, presumably because a Transformer layer acts as a coordinated unit (Liu et al., 11 May 2026).

4. Identification of topology-aware layers and implementation pipeline

The method requires an offline stage to identify the relevant attention heads and layers. The procedure has four steps (Liu et al., 11 May 2026).

First, entropy-based activity filtering computes a matrix-based entropy

$P(Y \mid S(G), Q)$2

where $P(Y \mid S(G), Q)$3 are singular values. High entropy indicates active, complex attention (Liu et al., 11 May 2026).

Second, structural feature extraction isolates the region covering edge tokens, applies top-$P(Y \mid S(G), Q)$4 binarization with $P(Y \mid S(G), Q)$5 equal to the number of nonzero entries in $P(Y \mid S(G), Q)$6, and then applies morphological closing—binary dilation followed by erosion with a $P(Y \mid S(G), Q)$7 square—to obtain a smoothed map $P(Y \mid S(G), Q)$8 (Liu et al., 11 May 2026).

Third, the method computes an attention concentration score over regions $P(Y \mid S(G), Q)$9 and $(0,1)(2,3)(0,3)$0 aligned to $(0,1)(2,3)(0,3)$1:

$(0,1)(2,3)(0,3)$2

$(0,1)(2,3)(0,3)$3

$(0,1)(2,3)(0,3)$4

This combines attention alignment inside the structural region with sparsity outside it (Liu et al., 11 May 2026).

Fourth, automatic thresholding uses Otsu’s method to set thresholds $(0,1)(2,3)(0,3)$5 and $(0,1)(2,3)(0,3)$6, selecting

$(0,1)(2,3)(0,3)$7

SLASH then targets all heads in these layers during inference (Liu et al., 11 May 2026).

The practical reproduction recipe is also explicit. Graphs are serialized; Source-Node Aggregation may be applied offline; entropy and concentration are computed to identify layers; $(0,1)(2,3)(0,3)$8 is calibrated on a small held-out set; and, online, the attention matrices in selected layers are intercepted and sharpened row-wise (Liu et al., 11 May 2026). The method adds no parameters, and the online arithmetic is described as negligible, but it requires materializing full attention maps and is therefore not compatible with FlashAttention kernels (Liu et al., 11 May 2026).

5. Empirical behavior on graph and molecular tasks

The empirical evaluation covers GraphInstruct for graph computational tasks and MoleculeNet for molecular property prediction, where SMILES are converted into explicit graph descriptions with atoms as nodes $(0,1)(2,3)(0,3)$9 and bonds as edges $(0,1)(0,3)(2,3)$0 (Liu et al., 11 May 2026). The model families include general-purpose LLMs such as the Llama-3.x series and Qwen3, alongside fine-tuned baselines such as GraphWiz variants and MolecularGPT (Liu et al., 11 May 2026).

On GraphInstruct, the reported average accuracies are:

Model	Baseline	With SLASH
Llama-3.1-8B	0.282	0.352
Qwen3-8B	0.174	0.367
Qwen3-4B	0.151	0.195

These gains are described as large and consistent for general LLMs, whereas fine-tuned GraphWiz models show marginal changes, for example GraphWiz-LLaMA2-7B averaging 0.642 versus 0.639, which the paper interprets as evidence that fine-tuning already optimizes structural attention (Liu et al., 11 May 2026).

On MoleculeNet, improvements are again concentrated in general-purpose models. For Qwen3-4B, the reported changes include BACE $(0,1)(0,3)(2,3)$1, ClinTox $(0,1)(0,3)(2,3)$2, and HIV $(0,1)(0,3)(2,3)$3. Llama-3.1-8B improves on BBBP from $(0,1)(0,3)(2,3)$4 to $(0,1)(0,3)(2,3)$5 and shows slight average gains, while MolecularGPT changes minimally, such as BBBP remaining at $(0,1)(0,3)(2,3)$6 and Tox21 moving from $(0,1)(0,3)(2,3)$7 to $(0,1)(0,3)(2,3)$8 (Liu et al., 11 May 2026).

The ablation studies support the method’s intended operating regime. General LLMs show model-specific $(0,1)(0,3)(2,3)$9 optima, while fine-tuned LLMs are comparatively stable. Layer-level intervention consistently outperforms head-level intervention. Source-Node Aggregation helps both vanilla and SLASH settings, but SLASH remains effective without it, which the authors take as confirmation of robustness to edge permutations (Liu et al., 11 May 2026). The combination of entropy and concentration is reported as the best criterion for identifying topology-aware heads; using only one criterion either selects active but structurally unfocused heads or inactive ones (Liu et al., 11 May 2026).

The paper also reports gains on extended tasks such as Cora node classification, with small gains in Link Prediction and Graph QA, which is consistent with SLASH’s structural focus (Liu et al., 11 May 2026).

6. Qualitative analysis, limits, and relation to broader attention studies

Qualitative visualizations show attention maps in Llama-3 and Qwen3 with sawtooth patterns aligned with $M_{gt}$0 in intermediate layers, and SLASH visibly increases the mass allocated to neighborhood blocks while reducing the sink budget (Liu et al., 11 May 2026). In a connectivity case study, the vanilla model hallucinates a path and answers “Yes,” while SLASH sharpens structure and answers “No” (Liu et al., 11 May 2026). This suggests that the intervention primarily affects structural faithfulness rather than generic confidence.

The paper is explicit about limitations. Performance is task dependent: if topology is not the main driver, sharpening can degrade results. A concrete example is Graph-SST2 for Llama-3.1-8B, which drops from $M_{gt}$1 to $M_{gt}$2, though Qwen3-8B improves on the same task (Liu et al., 11 May 2026). The method requires full attention maps and is incompatible with FlashAttention-style kernels unless those kernels are modified (Liu et al., 11 May 2026). A small per-model and per-task calibration for $M_{gt}$3 is required, and very small $M_{gt}$4 can destabilize representations (Liu et al., 11 May 2026).

The memory overhead is modest relative to eager full attention but substantial relative to FlashAttention. For a 6,942-token example, the reported peak memory is about $M_{gt}$5 GB with eager full attention versus about $M_{gt}$6 GB with FlashAttn2 on 4090 GPUs (Liu et al., 11 May 2026). This limitation is implementation-level rather than conceptual, but it materially constrains long-context deployment.

The broader implication is that LLMs are not structurally blind; the paper argues that intermediate attention layers act as implicit causal GATs performing topology-aware local aggregation, but that this signal is throttled by the sink (Liu et al., 11 May 2026). Related mechanistic work on Slash-Dominant Heads attributes slash patterns in attention to the interaction of RoPE with nearly rank-one queries and keys, showing that medium- and high-frequency RoPE components can induce offset-specific concentration along sub-diagonals (Cheng et al., 13 Jan 2026). That line of analysis concerns the emergence of slash patterns in general attention geometry, whereas SLASH the method uses an explicit redistribution rule to amplify topology-aware structure in serialized graph reasoning.

Future extensions proposed in the paper include applying the sharpening principle to trees, heterogeneous graphs, and tabular relational schemas; integrating sink-aware regularization or gated attention during training; and designing dynamic or data-dependent redistribution schemes that remove the need for offline calibration (Liu et al., 11 May 2026). The open theoretical question is how to reconcile semantic anisotropy with topology aggregation without sacrificing either (Liu et al., 11 May 2026).