TwoTower: Dual-Network Architectures in AI

• TwoTower is a dual-network architecture that separates context/user and item/product encoding to enable efficient retrieval and diffusion tasks.
• It underpins large-scale systems by precomputing embeddings and employing inner product scoring, facilitating fast approximate nearest-neighbor search.
• Extensions like CS3 and HIT add cross-tower synchronization and lightweight interactions, enhancing representation while balancing latency and quality.

Searching arXiv for papers on TwoTower across language modeling and recommendation systems. TwoTower denotes a class of architectures organized around two cooperating networks with distinct representational roles. In large-scale retrieval and recommendation, the canonical form separates context or user encoding from item or product encoding and scores relevance through an inner product, which supports pre-computation, caching, and approximate nearest-neighbour retrieval at scale (Osowska-Kurczab et al., 19 Jul 2025, Wang et al., 21 Apr 2026, Yang et al., 26 May 2025). In diffusion language modeling, "Nemotron-TwoTower" uses the term for a frozen autoregressive context tower and a trainable diffusion denoiser tower, thereby decoupling causal context representation from iterative masked-token refinement (Reda et al., 25 Jun 2026). Across these uses, the recurring principle is role specialization under a shared inference objective.

1. Core architectural pattern

The most direct formulation of a two-tower retriever is given by two separate embedding functions, $f_{\text{user}}:\mathcal{U}\to\mathbb{R}^d$ and $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$, with relevance scored as $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$ (Wang et al., 21 Apr 2026). Allegro’s deployed system instantiates the same pattern with an Item (Product) Tower and a Context (Query/User) Tower, both built around a shared Product Encoder, and uses normalized embeddings so that the dot product is equivalent to cosine similarity (Osowska-Kurczab et al., 19 Jul 2025). HIT adopts the same basic decomposition for pre-ranking in display advertising: a user tower and an ad tower produce $L_2$-normalized outputs, and the vanilla matching score is $\hat y=h_u^\top h_a$ (Yang et al., 26 May 2025).

Nemotron-TwoTower preserves the two-role structure but changes the semantics of the towers. One copy of Nemotron-3-Nano-30B-A3B remains frozen as an AR context tower that causally processes clean tokens and emits per-layer key/value caches together with Mamba-2 boundary states. The second copy is fine-tuned as a diffusion denoiser tower that refines masked blocks in parallel via bidirectional within-block attention and cross-attention to the corresponding layers of the context tower (Reda et al., 25 Jun 2026).

Setting	Tower roles	Coupling mechanism
Retrieval and recommendation	Context/User tower; Item/Product tower	Inner product or cosine similarity; ANN retrieval
Nemotron-TwoTower	Frozen AR context tower; trainable diffusion denoiser tower	Per-layer attention to context KV, cross-attention, seeded Mamba states

This comparison suggests that "TwoTower" is not a single algorithm but a structural template. What remains stable is the decomposition into two complementary computational paths; what changes is the object being represented, the point at which the towers interact, and the latency-quality trade-off being optimized.

2. Canonical two-tower retrieval in e-commerce

Allegro’s system provides a concrete large-scale realization of two-tower retrieval for recommendation across similarity search, complementary recommendation, and inspirational discovery (Osowska-Kurczab et al., 19 Jul 2025). Product features are represented as $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$, and the embeddings are defined as

$$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$

and $$f_{\text{context}}(u)=\mathrm{Norm}(\mathrm{MLP}([e_1(u),e_2(u),\dots,e_F(u)]))\in\mathbb{R}^D$$. The feature embeddings $e_k(\cdot)$ are low-dimensional trainable embeddings; typical embedding sizes are 32–64 per categorical feature, the MLP has 2–3 fully connected layers with hidden size approximately 512 and ReLU activations, and the deployed embedding dimension is $D=256$ (Osowska-Kurczab et al., 19 Jul 2025).

The relevance score is

$f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$0

which is cosine similarity because both vectors are $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$1-normalized (Osowska-Kurczab et al., 19 Jul 2025). For similarity search, training uses sampled softmax with a negative set combining uniform random negatives from the catalogue and hard negatives sampled according to item popularity under "mixed negative sampling" (Osowska-Kurczab et al., 19 Jul 2025). For complementary retrieval, the context tower also receives a target complementary category $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$2 and optimizes $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$3, so that the query representation encodes the intended complementary category (Osowska-Kurczab et al., 19 Jul 2025).

Serving is organized around ANN indexing. The system uses Faiss with IVF–PQ or HNSW for sub-millisecond neighbourhood search over approximately $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$4M product vectors, supports a daily offline pipeline that re-embeds all products and rebuilds the hierarchical index, and serves 20k QPS across desktop and mobile at 40 ms p99 CPU latency for full retrieval and post-processing (Osowska-Kurczab et al., 19 Jul 2025). Daily index refresh also addresses catalogue churn: new items embed immediately from content, without an ID-table cold-start (Osowska-Kurczab et al., 19 Jul 2025).

The same backbone is adapted to three tasks by changing inputs or serving logic rather than replacing the architecture. Similarity-TT uses a single query product and retrieves top-$f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$5 nearest neighbours; Complementary-TT generates one query embedding per complementary category and interleaves candidate groups; Inspirational-TT builds a hierarchical ANN index via k-means clustering, aggregates the last 100 viewed products by category, skips the closest clusters to avoid substitutes, and interleaves results across categories to maximize diversity (Osowska-Kurczab et al., 19 Jul 2025). Over two years of A/B testing, this unified system produced statistically significant gains in engagement and GMV-related metrics across desktop and mobile placements, including CTR increases for similarity and complementary placements and CTA/CVR improvements for inspirational layouts (Osowska-Kurczab et al., 19 Jul 2025).

3. Limits of isolated towers and the CS3 framework

The standard two-tower retriever is efficient because item embeddings can be precomputed, user embeddings cached, and retrieval implemented through ANN search such as FAISS (Wang et al., 21 Apr 2026). The same paper identifies three structural weaknesses of the isolated design. First, representation capacity is constrained because each tower is typically a lightweight MLP with no awareness of the other tower or downstream signals. Second, embedding-space alignment can degrade because the towers do not exchange information before the dot product. Third, cross-feature interactions are absent within the retriever, so richer interactions must be deferred to later ranking stages (Wang et al., 21 Apr 2026).

CS3, or Capability Synergy, is proposed as an online-compatible framework that strengthens two-tower retrieval while preserving real-time constraints (Wang et al., 21 Apr 2026). Its first component, Cycle-Adaptive Structure (CAS), replaces each fully connected layer with a "pre-forward → denoise → cycle-forward" block:

$f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$6

$f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$7

$f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$8

By sharing $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$9 between pre-forward and cycle-forward, CAS adds negligible extra parameters; in the reported system, CAS is integrated directly into the main cross-entropy prediction loss rather than via a separate CAS objective (Wang et al., 21 Apr 2026).

The second component, Cross-Tower Synchronization (CTS), injects into each tower a cached cross vector summarizing the partner tower’s recent positive representations. For positive feedback, the vectors are updated by EMA:

$$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$0

The paper emphasizes that CTS is "loss-free": no explicit alignment loss is introduced; the injected vectors and the main prediction loss are sufficient (Wang et al., 21 Apr 2026). The third component, Cascade-Model Sharing (CMS), reuses a downstream ranker’s intermediate representation $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$1 through EMA-cached vectors $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$2 and $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$3, again without an additional distillation loss (Wang et al., 21 Apr 2026).

Empirically, CS3 reports consistent gains over strong baselines on TaobaoAd, KuaiRand, and RecSys2017. For DSSM, the base model yields test AUC / LogLoss of $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$4 on TaobaoAd, $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$5 on KuaiRand, and $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$6 on RecSys2017, whereas DSSM + CS3 reaches $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$7, $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$8, and $$s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$$9, with $L_2$0 over base (Wang et al., 21 Apr 2026). In a large-scale advertising deployment with 400M DAUs and 10% traffic for 7+ days, the full CS3 system produced +8.356% revenue and +0.468% DAC in Scenario A, while QPS changed by -0.589%; aggregated gains were also reported for Scenarios B and C (Wang et al., 21 Apr 2026). Latency remained within ms-level constraints, with CMS fetches contributing p99 latency below 5 ms and CAS causing less than a 1% QPS drop in the user tower (Wang et al., 21 Apr 2026).

A common misconception is that a two-tower retriever must remain interaction-free until the final dot product. CS3 demonstrates a narrower claim: the base architecture is interaction-light, but lightweight mutual awareness, self-revision, and downstream feature sharing can be added without abandoning the two-tower serving regime.

4. Interaction-enhanced two-tower pre-ranking

HIT, the Hierarchical Interaction-Enhanced Two-Tower model, addresses two limitations explicitly attributed to vanilla two-tower pre-ranking: the absence of cross-tower information exchange before the final score and the inability of a single vector plus a plain inner product to capture multi-faceted user interests and multi-dimensional ad attributes (Yang et al., 26 May 2025). Its design preserves the two-tower decomposition while introducing coarse-grained and fine-grained interaction modules.

The coarse-grained component is a dual-generator framework. Two simple MLPs, $L_2$1 and $L_2$2, each with two hidden layers, consume only static features and produce mimic vectors $L_2$3 (Yang et al., 26 May 2025). Generator 1 is trained on positive samples to mimic the holistic opposite-tower representation for target items, while generator 2 is trained on negative samples to mimic non-target representations. The user-side generation loss is

$L_2$4

with an analogous ad-side loss, and the total objective is

$L_2$5

where $L_2$6 and $L_2$7 is cosine distance (Yang et al., 26 May 2025). Stop-gradient is applied on generator inputs and multi-head outputs during generation-loss computation (Yang et al., 26 May 2025).

The fine-grained component is the multi-head representer. For each head $L_2$8,

$L_2$9

and the final score is

$\hat y=h_u^\top h_a$0

Because all ad-head vectors can be precomputed and cached offline, online inference retains the characteristic two-tower efficiency: only the user tower and a small number of head-level dot products are computed online (Yang et al., 26 May 2025).

The reported implementation uses embedding dimension $\hat y=h_u^\top h_a$1 for all sparse features, a three-layer MLP backbone with widths $\hat y=h_u^\top h_a$2, generator widths $\hat y=h_u^\top h_a$3, $\hat y=h_u^\top h_a$4 heads with output dimension $\hat y=h_u^\top h_a$5, batch size $\hat y=h_u^\top h_a$6, Adam, and Tesla T4 GPUs (Yang et al., 26 May 2025). Offline, HIT achieves the best AUC on Alibaba, MovieLens-1M, and Amazon-Electronics; on Alibaba, the paper reports AUC $\hat y=h_u^\top h_a$7 for HIT versus $\hat y=h_u^\top h_a$8 for DSSM and $\hat y=h_u^\top h_a$9 for IntTower, with $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$0 for all gains (Yang et al., 26 May 2025). Online, on Tencent’s advertising platform with more than 3.6B training samples, 1B users, and 10M ads, HIT improved GMV by 1.66% and ROI by 1.55% versus MVKE, while inference latency and success rate at 35K QPS were nearly identical to vanilla two-tower (Yang et al., 26 May 2025).

HIT is important because it sharpens a recurrent tension in two-tower design. Efficiency derives from decomposition and pre-caching, but retrieval quality often requires some form of interaction modeling. HIT’s result is not that the decomposition should be discarded; rather, interaction can be pushed into lightweight generator and head-projection structures while preserving the serving mechanics that make two-tower systems practical.

5. TwoTower in diffusion language modeling

Nemotron-TwoTower extends the name beyond recommendation and retrieval into sequence generation (Reda et al., 25 Jun 2026). Starting from Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model trained on approximately 2.1T tokens, the method instantiates two identical 52-layer copies, each composed of 23 Mamba-2 layers, 6 self-attention layers, and 23 MoE layers (Reda et al., 25 Jun 2026). One copy remains frozen as the AR context tower; the other is trained as the diffusion denoiser tower.

The model factorizes the sequence block-autoregressively,

$$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$1

where each block density is modeled by a masked-diffusion process conditioned on prefix context (Reda et al., 25 Jun 2026). In the forward process, each token is replaced by [MASK] independently with probability $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$2, using the linear schedule $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$3:

$$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$4

The reverse process parameterizes

$$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$5

for masked positions through the trainable denoiser tower (Reda et al., 25 Jun 2026).

The architectural coupling between towers is explicit and multi-scale. At denoiser layer $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$6, self-attention is bidirectional within the current noisy block but causal with respect to past clean blocks by attending to

$$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$7

Each denoiser layer also cross-attends to the corresponding context-tower layer, and the denoiser’s Mamba-2 layers are seeded with the context tower’s Mamba states so that the convolutional SSM state carries forward across blocks (Reda et al., 25 Jun 2026). Time conditioning is injected through adaLN-single (Reda et al., 25 Jun 2026).

Training minimizes the average negative log-likelihood over masked positions,

$$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$8

omitting the theoretical time-weight $$x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$$9 for stability (Reda et al., 25 Jun 2026). Only the denoiser tower is trained, using BF16 precision, AdamW with warmup–stable–decay, peak learning rate $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$0, final learning rate $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$1, block size $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$2, and up to $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$3 denoising steps. The denoiser adds only 1.5M adaLN parameters, while the context tower head remains frozen (Reda et al., 25 Jun 2026).

Inference alternates between advancing the frozen context tower over newly committed blocks and running denoising steps on the next block. The paper describes a confidence-unmasking sampler that initializes a block as all [MASK], computes $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$4 as the fraction of masked positions, evaluates the denoiser, and commits positions whose maximum predicted probability exceeds threshold $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$5 (Reda et al., 25 Jun 2026). With $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$6, $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$7, and $$f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$$8 chosen so most blocks finish, Nemotron-TwoTower retains 98.7% of the baseline AR model’s aggregate benchmark quality while achieving a 2.42X wall-clock speedup on H100 GPUs (Reda et al., 25 Jun 2026).

This use of TwoTower departs from the retrieval setting in mechanics but not in design logic. The context tower preserves the pretrained causal next-token machinery; the denoiser tower specializes in masked-token restoration. The decoupling is therefore between context representation and iterative refinement rather than between query and item representation.

6. Comparative interpretation, trade-offs, and recurring themes

Across the recommendation papers, the primary operational advantage of two-tower design is efficient large-scale retrieval. Item or ad representations can be precomputed and cached; retrieval can be executed with ANN infrastructure such as Faiss; and online compute is concentrated on the context or user side (Osowska-Kurczab et al., 19 Jul 2025, Wang et al., 21 Apr 2026, Yang et al., 26 May 2025). In the language-modeling setting, the efficiency gain appears in a different form: multiple masked tokens can be committed per denoising step, and Nemotron-TwoTower reports that "the first step alone often unpacks half a block," contributing to higher throughput (Reda et al., 25 Jun 2026).

The same papers also make clear that decomposition creates systematic weaknesses. Allegro’s design emphasizes flexibility and maintenance reduction but still requires task-specific serving logic for similarity, complementary, and inspirational placements (Osowska-Kurczab et al., 19 Jul 2025). CS3 explicitly identifies limited representation capacity, embedding-space misalignment, and missing cross-feature interactions in isolated towers (Wang et al., 21 Apr 2026). HIT identifies coarse similarity metrics and missing cross-domain interaction as central liabilities of vanilla two-tower pre-ranking (Yang et al., 26 May 2025). Nemotron-TwoTower introduces a different trade-off: approximately 1.3% quality loss and a larger fixed model footprint because two 30B networks are instantiated, even though only one is trained (Reda et al., 25 Jun 2026).

A common misconception is that TwoTower refers only to recommender systems. Nemotron-TwoTower shows that the term can also designate a block-wise autoregressive diffusion model with a frozen context tower and a trainable denoiser tower (Reda et al., 25 Jun 2026). Another common misconception is that two-tower architectures are defined by complete independence of the towers until a final score is computed. That statement is accurate for the strict vanilla retriever, but it does not hold universally: CS3 injects cached cross vectors and cascade vectors, HIT adds mimic-vector conditioning and head-wise interaction, and Nemotron-TwoTower uses per-layer cross-attention and state seeding (Wang et al., 21 Apr 2026, Yang et al., 26 May 2025, Reda et al., 25 Jun 2026).

This suggests that TwoTower is best understood as a constrained decomposition principle rather than a prohibition on interaction. The decomposition isolates the computations that are most reusable, precomputable, or stable; the remaining design space concerns how much information may cross between the towers without destroying the latency or throughput advantages that motivated the architecture. The open problems named in the surveyed work follow directly from that tension: further weight sharing, more adaptive noise schedules, or hierarchical block sizes in Nemotron-TwoTower; continued online-compatible alignment and cross-stage consistency in CS3; and lightweight interaction modules that preserve near-vanilla serving cost in HIT and large-scale e-commerce retrieval (Reda et al., 25 Jun 2026, Wang et al., 21 Apr 2026, Yang et al., 26 May 2025).