ApartFF: Dual Roles in Retrieval & LLM Serving

• The paper on asymmetric feature fusion demonstrates that leveraging complementary gallery-side features improves retrieval accuracy without increasing query latency, as evidenced by significant mAP gains on large-scale datasets.
• ApartFF in image retrieval uses a dynamic mixer and a lightweight query model to fuse diverse global and local descriptors into a compact embedding for efficient offline storage and fast retrieval.
• In LLM serving, ApartFF separates FFN from Attention to optimize throughput by independently scaling compute-intensive operations and memory-bound tasks, achieving near-optimal theoretical ratios.

Searching arXiv for the cited papers to ground the article. arXiv search query: (Wu et al., 2024) ApartFF is used in two distinct arXiv contexts. In image retrieval, it denotes Asymmetric Feature Fusion (AFF), a paradigm for asymmetric retrieval systems in which multiple complementary gallery-side features are fused into a single compact embedding while the query side remains a single lightweight model (Wu et al., 2024). In LLM serving, the term is used to denote keeping FFN apart from Attention within Attention–FFN disaggregation (AFD), specifically the $rA$–$1F$ topology where $r$ Attention instances feed one shared FFN instance during decoding (Song et al., 29 Jan 2026). The shared label reflects a common engineering pattern—moving complexity away from the constrained path—but the two usages address different tasks, resource models, and optimization criteria.

1. Terminological scope

In the image-retrieval literature, AFF is sometimes referred to as “ApartFF.” The method is defined for asymmetric retrieval systems, where the query side is often resource-constrained and the gallery side usually runs on server or cloud infrastructure with abundant compute and storage. The central claim is that complementarity among different features should be exploited only on the gallery side, because gallery features can be precomputed offline and fused into one compact embedding without increasing query-time cost (Wu et al., 2024).

In the LLM-serving literature, “ApartFF” is used in a different sense: “keeping FFN apart from Attention.” There it corresponds exactly to the $rA$–$1F$ AFD topology, in which state-heavy, KV-cache-dominated Attention computation is separated from stateless, compute-intensive FFN computation and connected by per-step communication. The objective is not representation learning but throughput-optimal provisioning of Attention and FFN resources during decoding (Song et al., 29 Jan 2026).

This suggests that “ApartFF” is best treated as a disambiguation term rather than a single standardized acronym.

2. ApartFF as Asymmetric Feature Fusion in image retrieval

AFF is proposed for the dilemma faced by asymmetric retrieval systems: a lightweight query model is desirable for low memory, latency, and power cost, but its limited capacity makes it difficult to match the accuracy of symmetric systems that use a large model on both query and gallery sides. Existing compatibility-learning approaches improve alignment between query and gallery embeddings, yet the paper states that accuracy remains notably lower in large-scale scenarios because the small query model cannot align perfectly to the powerful gallery model (Wu et al., 2024).

The key idea is to leverage feature diversity at the gallery side. Each gallery image is embedded into multiple complementary features, including global descriptors such as DELG, Token, DOLG, and CVNet, and local descriptors such as HOW and DELG local. Global descriptors are described as holistic semantic descriptors, whereas local descriptors capture fine-grained correspondences. AFF explicitly does not use location or scale metadata from local features; it uses only descriptors.

Formally, for a gallery image $I_g$, the method assumes $K$ global feature models and $M$ local feature models:

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$

$$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$

All features are projected to a common dimension $1F$0 through learned linear layers:

$1F$1

The projected features are stacked into a single sequence

$1F$2

where $1F$3.

The significance of this construction is architectural rather than merely ensemble-based. The gallery stores one fused vector per image at retrieval time, so the method aims to retain the discriminative benefits of multi-feature extraction without converting retrieval into a multi-vector search problem.

3. Dynamic mixer, query model, and joint optimization

AFF has three components: gallery-side multi-feature extraction, a dynamic mixer, and a lightweight query model. The dynamic mixer is a lightweight transformer-based fusion module $1F$4. A learnable fusion token $1F$5 is prepended to the stacked feature sequence:

$1F$6

The sequence is processed for $1F$7 layers by

$1F$8

$1F$9

with

$r$0

where $r$1, $r$2, $r$3, and $r$4. The final fusion token is used as the aggregated gallery embedding,

$r$5

On the query side, only a single lightweight CNN is deployed, such as MobileNetV2, ShuffleNetV2, EfficientNet-lite, or GhostNet, with GeM pooling and whitening. It produces an $r$6-normalized embedding

$r$7

Retrieval uses cosine similarity,

$r$8

Training couples the mixer and the lightweight query model through ArcFace classification and a shared momentum-updated classifier. Gallery feature extractors are frozen. The mixer classifier has weight matrix $r$9, while the query model uses a moving-averaged copy $rA$0:

$rA$1

with $rA$2. Only $rA$3 receives gradients; $rA$4 is not updated by back-propagation. The paper states that this decoupling stabilizes training, preserves mixer discriminativeness, and guides the lightweight query model with a slowly changing classifier (Wu et al., 2024).

The training pipeline uses GLDv2 with 1.58M images and 81,311 classes. In each minibatch, precomputed gallery features are fetched, the mixer produces $rA$5, the query network produces $rA$6, ArcFace losses are applied to both branches, and the query classifier is updated via EMA. In ablations, accuracy improves monotonically with $rA$7 and is best for $rA$8; $rA$9 is reported as strong across datasets.

4. Empirical behavior, efficiency, and deployment of AFF

The reported experiments use GLDv2-Test, ROxford, and RParis, including large-scale evaluation with $1F$0 distractors. Metrics are mAP@100 for GLDv2-Test and mAP for Medium and Hard settings on ROxford and RParis. Training details include randomly resized images, crop $1F$1, color jitter, SGD with weight decay $1F$2, initial learning rate $1F$3 with linear decay, 20 epochs, batch size 128 on $1F$4, $1F$5 transformer layers, $1F$6, ArcFace margin $1F$7, and scale $1F$8 (Wu et al., 2024).

Against asymmetric baselines such as Contextual Similarity Distillation (CSD), the method is reported to improve accuracy while keeping query-side cost unchanged. Examples with a MobileNetV2 query model and fused mixer gallery include ROxf+1M Hard mAP $1F$9 versus $I_g$0, RPar+1M Hard $I_g$1 versus $I_g$2, ROxf Medium $I_g$3 versus approximately $I_g$4, RPar Medium $I_g$5 versus approximately $I_g$6, and GLDv2-Test private/public $I_g$7 versus approximately $I_g$8. Symmetric fusion, where both gallery and query use the mixer, is also reported to outperform concatenation ensembles and large single backbones.

The efficiency claim is central. Query-side overhead does not increase beyond a single lightweight model. Typical feature extraction latency is reported as approximately $I_g$9 ms, versus $K$0 ms for CSD, with identical embedding dimension $K$1 and ANN search cost. Gallery-side feature extraction and fusion are performed offline. At retrieval time, the system stores only one $K$2-dimensional vector per image, enabling standard ANN indexing such as PQ and IVF without special modifications. Memory per image is stated to be the same as for single-feature embeddings of dimension $K$3, for example 512 or 2048.

Ablations are used to argue that the gains arise from complementarity and from the mixer design rather than from naive aggregation. Each added global feature improves performance, the transformer mixer with a fusion token outperforms concatenation+MLP and attention-based baselines such as LAFF and OrthF, and the mixer is robust to weak or noisy gallery features. For example, adding weak global features degrades concatenation-based ensembles on ROxf+1M from $K$4 mAP to $K$5, whereas the AFF mixer is described as almost unaffected. Injecting Gaussian white noise into gallery features lowers concatenation to $K$6 mAP, while AFF suffers only a slight drop.

5. ApartFF as Attention–FFN disaggregation in LLM serving

In the second usage, ApartFF corresponds to AFD in an $K$7–$K$8 topology. Attention computation is stateful and memory-bound because it reads KV caches whose effective size grows with sequence length; FFN computation is stateless and compute-intensive; and the two are linked by per-step communication. The decoding step for one Transformer layer is decomposed into Attention, Attention-to-FFN communication, FFN, and FFN-to-Attention communication. Microbatch pipelining is used to overlap these phases (Song et al., 29 Jan 2026).

For one Attention instance with microbatch size $K$9, the total token load is

$M$0

where $M$1 is the prefill length and $M$2 is the decode index. The paper models decode length $M$3 as geometric with termination probability $M$4,

$M$5

With continuous batching and large-$M$6 horizon averaging, the expected token load per Attention instance becomes

$M$7

where $M$8 is the mean prefill length.

Latency is modeled linearly:

$M$9

The bundle cycle time is

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$0

and throughput per instance is

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$1

The rationale is resource separation. AFD enables independent scaling of memory-side Attention capacity and compute-side FFN capacity. The paper’s problem is that mis-sizing the Attention/FFN ratio causes blocking at step boundaries and idle time on one side of the pipeline.

6. Closed-form sizing law and simulator validation for the $$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$2–$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$3 topology

The paper derives a closed-form rule for the optimal Attention/FFN ratio. Let

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$4

The key candidate ratios are

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$5

The theorem gives

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$6

with optimal throughput

$$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$7

The theoretical model assumes linear latency laws, layer-wise homogeneity, geometric decode lengths, continuous batching, and synchronized steps across Attention instances. The paper states that the theoretical optimal A/F ratio matches the simulation-optimal within 10% relative error across workloads. The simulator is discrete-event and cycle-by-cycle, using a six-state FSM per batch and two batches in flight to overlap computation and communication. Latency coefficients are calibrated on DeepSeek-V3 traces on Huawei Ascend 910C NPUs, with $$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$8 cycles/token, $$g_i = \phi_i(I_g) \in \mathbb{R}^{D_i}, \qquad i = 1,\ldots,K,$$9 cycles, $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$0 cycles/request, $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$1 cycles, $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$2 cycles/token, and $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$3 cycles (Song et al., 29 Jan 2026).

A worked example uses $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$4, $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$5, and $$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$6. Then

$$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$7

$$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$8

This yields

$$\ell_i = \psi_i(I_g) \in \mathbb{R}^{n_i \times d_i}, \qquad i = 1,\ldots,M.$$9

so $1F$00, consistent with the reported theoretical optimum $1F$01. The per-instance throughput rises from approximately $1F$02 tokens/cycle/instance at $1F$03 to approximately $1F$04 at $1F$05. The simulation also reports idle-ratio crossover near $1F$06–$1F$07, with FFN idle ratio exceeding 60% at small $1F$08 such as $1F$09, and Attention idle ratio exceeding 60% at large $1F$10 such as $1F$11.

7. Limitations, misconceptions, and future directions

A common misconception is that ApartFF denotes one method. The supplied literature instead assigns the label to two unrelated technical constructs. One concerns asymmetric representation learning for image retrieval; the other concerns disaggregated serving for LLM decoding. The overlap lies in asymmetry of deployment rather than in algorithmic content.

For AFF in image retrieval, the stated limitations are dependence on feature diversity, sensitivity to domain shift in gallery-side feature models, and the offline compute cost of very long feature sequences for the mixer. Practical deployments may subsample local features or cap sequence length. Future directions listed in the paper include adaptive feature selection, cross-domain adaptation, and compression-aware fusion with quantization-aware training for PQ/IVF indexing (Wu et al., 2024).

For ApartFF as AFD, the stated limitations include network contention, batching variability, stragglers from continuous batching and geometric decode lengths, layer heterogeneity, and imperfect load balancing across Attention instances. The paper notes growing discrepancy between theory and simulation at large $1F$12, about 15% at $1F$13. Suggested extensions include $1F$14–$1F$15 topologies and heterogeneous-device settings with device-specific $1F$16 and $1F$17 parameters (Song et al., 29 Jan 2026).

Taken together, the two usages of ApartFF exemplify a shared systems principle: move state, compute, or representational richness to the side of the pipeline where offline processing, aggregation, or specialized provisioning is feasible, while preserving a simpler latency-critical path. In image retrieval, that path is the lightweight query encoder; in LLM serving, it is the balanced decode step across disaggregated Attention and FFN resources.