---
title: RANA Attention Loss for Few-Shot KG Completion
url: https://www.emergentmind.com/topics/rana-attention-loss
type: topic
---

# RANA Attention Loss for Few-Shot KG Completion

RANA Attention Loss denotes the attention-based multi-negative objective introduced in the Relation-Aware Network with Attention-Based Loss (RANA) framework for few-shot knowledge graph completion (FKGC), where the task is to predict unseen facts of a relation from only a handful of support triples. The mechanism was proposed to address two problems identified in prior FKGC methods: a zero-loss problem caused by randomly sampled negatives that are often too easy, and the inadequacy of assigning one fixed embedding to an entity across all relation contexts. In RANA, the loss is not an auxiliary term; it is the core training objective used together with a dynamic relation-aware entity encoder and a MAML-style meta-learning procedure [2306.09519]. A similarly named 2025 RaNA framework for Transformer compression is distinct and explicitly does not define a dedicated attention loss [2503.18216].

## 1. Problem setting and functional role

RANA is designed for FKGC, in which a model must infer missing entities for a relation that has only a few reference entity pairs. The framework addresses the observation that earlier approaches typically select one negative sample for each reference entity pair and optimize a margin-based ranking loss. According to the RANA formulation, this practice is wasteful because the sampled negative is often far from the positive sample, falls outside the margin, and produces zero loss. The same work also argues that an entity should not have one fixed embedding across all relation contexts, since the representation needed for a relation such as **WorkIn** need not match the representation needed for **DieIn** [2306.09519].

Operationally, RANA has three stages. It first builds context-dependent entity representations using local graph neighborhoods and the target few-shot relation. It then performs negative sampling more carefully by pruning irrelevant candidates and selecting multiple negatives instead of only one. Finally, it assigns attention weights to those negatives so that negatives more similar to the positive sample influence learning more strongly. This weighted loss is used inside a MAML-like optimization loop so that the model can adapt from a support set $\mathcal{S}$ to a query set $\mathcal{Q}$ for a new few-shot relation.

## 2. Formal definition of the loss

For a positive entity pair $(h_i,t_i)$, RANA defines a translational distance
$$
d_{(h_i,t_i)} = || \mathbf{h_i+R-t_i} ||_{L2},
$$
where $\mathbf{R}$ is the learned relation representation. A score is then defined as
$$
s_{(h_i,t_i)} = \gamma - d_{(h_i,t_i)},
$$
with $\gamma$ acting as a margin-like hyperparameter. Using multiple negative triples $(h_i,t_{ij}^-)$, the attention-based loss is
$$
\mathcal{L} = -\sum_{i=1}^I \log\, \sigma(s_{(h_i,t_i)}) -\sum_{i=1}^I\sum_{j=1}^J \beta_{ij} \log\, \sigma(-s_{(h_i,t^-_{ij})}),
$$
where $I$ is the number of positive entity pairs, $J$ is the number of negative samples per positive, $\sigma(\cdot)$ is sigmoid, and $\beta_{ij}$ is the attention weight for the $j$-th negative of the $i$-th positive pair [2306.09519].

The defining property of the loss is that negative samples are not treated uniformly. Each negative contribution is scaled by $\beta_{ij}$, so a negative sample contributes in proportion to its relevance to the positive sample. This differs from a standard ranking formulation in which a single negative pair may dominate the learning signal or, if it is too easy, produce no gradient at all.

## 3. Negative selection and attention weighting

RANA uses a two-step negative sampling strategy. Before sampling, it prunes tail candidates that are too dissimilar to the true tail entity. Given the true tail embedding $\mathbf{t}$ and a candidate tail embedding $\mathbf{t^-}$, similarity is computed as
$$
f(\mathbf{t,t^-}) = \mathbf{t^{-T}t}.
$$
If
$$
f(\mathbf{t,t^-}) < \tau,
$$
where $\tau$ is a threshold, the candidate $t^-$ is removed. This pruning step keeps the remaining negative pool closer to the positive sample and increases the likelihood that sampled negatives are informative rather than trivial [2306.09519].

After pruning, RANA samples multiple negatives $t_{ij}^-$ for each positive triple rather than a single negative tail entity. Attention is then computed between the positive pair and each negative pair. The positive sample is represented as
$$
\mathbf{p_i = h_i \oplus t_i},
$$
and the $j$-th negative sample as
$$
\mathbf{n_{ij} = h_i \oplus t_{ij}^-}.
$$
Their similarity is given by a scaled dot product,
$$
f(\mathbf{p_i,n_{ij}}) = \frac{\mathbf{n_{ij}^T p_i}}{\sqrt{|p|}},
$$
where $|p|$ is the dimensionality of $\mathbf{p_i}$. The attention weights are then
$$
\beta_{ij}=\frac{\exp\, f(\mathbf{p_i,n_{ij}})}{\sum_{j=1}^J \exp\, f(\mathbf{p_i,n_{ij}})}.
$$
Under this construction, a negative triple that is more similar to the positive triple receives a larger $\beta_{ij}$ and therefore a larger coefficient in the loss. Easier or irrelevant negatives receive small weights and have little effect. The paper’s interpretation is that negatives more similar to positive samples contribute more strongly to learning the boundary.

## 4. Relation-aware encoder and context dependence

The attention-based loss is paired with a dynamic relation-aware entity encoder whose purpose is to produce relation-specific entity embeddings. The few-shot relation embedding is initialized from a support pair $(h,t)$ by a TransE-style difference,
$$
\mathbf{r = t - h}.
$$
For an entity $h$, the background neighborhood is denoted $\mathcal{N}_h=\{(r_i,c_i)\}$, where each neighbor consists of a relation $r_i$ and neighboring entity $c_i$. To measure how relevant a neighboring relation $r_i$ is to the target few-shot relation $r$, RANA applies an MLP:
$$
m(r,r_i)=\mathbf{W_2[tanh(W_1[r \oplus r_i])]}.
$$
Neighbor attention is then computed as
$$
\alpha_i = \frac{\exp(m(r,r_i))}{\sum_{r_i \in \mathcal{N}_h} \exp(m(r,r_i))}.
$$
Neighbors whose relation types are semantically close to the target relation receive higher attention [2306.09519].

The relation-aware neighborhood aggregation is
$$
\mathbf{A_{r_i,c_i} = \sum_{(r_i, c_i) \in \mathcal{N}_h} \alpha_i \mathbf{W_3[r_i \oplus c_i]},
$$
with trainable $\mathbf{W_3}$. The final context-dependent entity embedding is
$$
\mathbf{h'} = \sigma \mathbf{(W_4(h + A_{r_i,c_i}))},
$$
where $\mathbf{W_4}$ is trainable and $\sigma$ is sigmoid. This construction allows the same entity to produce different embeddings under different target relations. For each support pair $(h_i,t_i)$, the relation representation is
$$
\mathbf{R_{(h_i,t_i)} = FC^{\sigma}_{\mathbf{W_5}\mathbf{[h'_i \oplus t'_i]}},
$$
and the support-set relation embedding is the average
$$
\mathbf{R^s} = \frac{\sum_{i=1}^I\mathbf{R_{(h_i,t_i)}}}{I}.
$$

## 5. Meta-learning objective and comparison with margin-based training

RANA embeds the encoder and attention-based loss in a MAML-like optimization loop. The support set $\mathcal{S}$ of a task is used to construct a support relation representation $\mathbf{R^s}$. Support loss is computed and used to update task-specific parameters, yielding $\mathbf{R^q}$. The query loss is then computed on $\mathcal{Q}$ using $\mathbf{R^q}$, and the global parameters $\mathbf{\Theta}$ are meta-updated by minimizing query losses across tasks. The meta-training objective is written as
$$
\mathcal{L} = arg\min_\mathbf{\Theta} \sum \mathcal{L}_{q}.
$$
In this setup, the attention-based loss is used both for support adaptation and for query optimization [2306.09519].

The framework is explicitly positioned against conventional margin-based ranking loss. Earlier FKGC methods usually sample one negative triple and enforce that the positive triple score should be better than the negative by a margin. When the sampled negative is too easy, the loss becomes zero. RANA instead uses the log-sigmoid objective with multiple weighted negatives, which avoids the brittle behavior of a hinge or margin loss in the single-negative case. The authors further note that this differs from RotatE’s self-adversarial negative sampling: RotatE weights negatives by a probability distribution over negatives, whereas RANA weights them by similarity to the positive sample, making the weighting explicitly positive-aware.

## 6. Empirical results and ablations

On the benchmark datasets NELL-One and Wiki-One, RANA reports the best 5-shot KGC results among the evaluated methods. On NELL-One, the reported scores are MRR **0.361**, Hits@10 **0.573**, Hits@5 **0.475**, and Hits@1 **0.253**. On Wiki-One, the reported scores are MRR **0.379**, Hits@10 **0.480**, Hits@5 **0.437**, and Hits@1 **0.329**. The paper reports improvements over the runner-up of **+4.9% MRR, +10.2% Hits@10, +8.2% Hits@5, +2.8% Hits@1** on NELL-One, and **+2.2% MRR, +2.3% Hits@10, +4.3% Hits@5, +3.1% Hits@1** on Wiki-One [2306.09519].

The ablation study isolates the contribution of the attention-based loss and its associated negative sampling scheme. When only one negative sample is used, NELL-One MRR drops from **0.372** to **0.294**, and Wiki-One MRR drops from **0.387** to **0.349**. Removing negative attention yields **0.293** on NELL-One and **0.298** on Wiki-One. Removing both candidate pruning and negative attention yields **0.257** on NELL-One and **0.286** on Wiki-One. Replacing Eq. (13) with RotatE self-adversarial negative sampling yields **0.268** on NELL-One and **0.310** on Wiki-One. These results indicate that both multiple negative sampling and attention weighting contribute materially to performance, and that the positive-aware weighting strategy is better suited to FKGC than the generic adversarial alternative tested in the paper.

## 7. Terminological ambiguity and the distinct RaNA adapter literature

A common source of confusion is the later appearance of the acronym **RaNA** in the Transformer efficiency paper “Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters” [2503.18216]. In that work, **RaNA** stands for **Rank and Neuron Allocator** and refers to an adaptive rank-allocation framework for compressing linear layers at inference time. Its attention-related mechanism operates on the linear QKV projections and is governed by a reconstruction objective,
$$
\argmin_{A,B,r} \mathbb{E}_x\big(\|Wx - A(r(x)\odot Bx)\|_F^2\big),
$$
together with a budgeted sparsity mask objective.

That 2025 work explicitly states that it does **not** define an explicit special “attention loss” term. The closest related objective is the reconstruction loss on the QKV projection outputs, plus an optional binary cross-entropy loss for an auxiliary MLP-Sigmoid masker. Accordingly, the term **RANA Attention Loss** properly refers to the attention-based multi-negative FKGC loss introduced in the 2023 knowledge-graph paper, not to the RaNA adapter objective for Transformer compression. This distinction is necessary because the two works share a near-identical acronym while addressing different model classes, different optimization criteria, and different meanings of “attention.”

Source: https://www.emergentmind.com/topics/rana-attention-loss