--- title: 'TwoTower: Dual-Network Architectures in AI' url: https://www.emergentmind.com/topics/twotower type: topic --- # TwoTower: Dual-Network Architectures in AI Searching arXiv for recent papers on TwoTower across language modeling and recommendation systems. TwoTower denotes a class of architectures organized around two cooperating networks with distinct representational roles. In large-scale retrieval and recommendation, the canonical form separates context or user encoding from item or product encoding and scores relevance through an inner product, which supports pre-computation, caching, and approximate nearest-neighbour retrieval at scale [2508.03702] [2604.19269] [2505.19849]. In diffusion language modeling, "Nemotron-TwoTower" uses the term for a frozen autoregressive context tower and a trainable diffusion denoiser tower, thereby decoupling causal context representation from iterative masked-token refinement [2606.26493]. Across these uses, the recurring principle is role specialization under a shared inference objective. ## 1. Core architectural pattern The most direct formulation of a two-tower retriever is given by two separate embedding functions, $f_{\text{user}}:\mathcal{U}\to\mathbb{R}^d$ and $f_{\text{item}}:\mathcal{V}\to\mathbb{R}^d$, with relevance scored as $s(u,v)=\langle f_{\text{user}}(u), f_{\text{item}}(v)\rangle$ [2604.19269]. Allegro’s deployed system instantiates the same pattern with an Item (Product) Tower and a Context (Query/User) Tower, both built around a shared Product Encoder, and uses normalized embeddings so that the dot product is equivalent to cosine similarity [2508.03702]. HIT adopts the same basic decomposition for pre-ranking in display advertising: a user tower and an ad tower produce $L_2$-normalized outputs, and the vanilla matching score is $\hat y=h_u^\top h_a$ [2505.19849]. Nemotron-TwoTower preserves the two-role structure but changes the semantics of the towers. One copy of Nemotron-3-Nano-30B-A3B remains frozen as an AR context tower that causally processes clean tokens and emits per-layer key/value caches together with Mamba-2 boundary states. The second copy is fine-tuned as a diffusion denoiser tower that refines masked blocks in parallel via bidirectional within-block attention and cross-attention to the corresponding layers of the context tower [2606.26493]. | Setting | Tower roles | Coupling mechanism | |---|---|---| | Retrieval and recommendation | Context/User tower; Item/Product tower | Inner product or cosine similarity; ANN retrieval | | Nemotron-TwoTower | Frozen AR context tower; trainable diffusion denoiser tower | Per-layer attention to context KV, cross-attention, seeded Mamba states | This comparison suggests that "TwoTower" is not a single algorithm but a structural template. What remains stable is the decomposition into two complementary computational paths; what changes is the object being represented, the point at which the towers interact, and the latency-quality trade-off being optimized. ## 2. Canonical two-tower retrieval in e-commerce Allegro’s system provides a concrete large-scale realization of two-tower retrieval for recommendation across similarity search, complementary recommendation, and inspirational discovery [2508.03702]. Product features are represented as $x_i=(\mathrm{title},\mathrm{price},\mathrm{category},\dots)$, and the embeddings are defined as $f_{\text{item}}(x_i)=\mathrm{Norm}(\mathrm{MLP}([e_1(x_i),e_2(x_i),\dots,e_F(x_i)]))\in\mathbb{R}^D$ and $f_{\text{context}}(u)=\mathrm{Norm}(\mathrm{MLP}([e_1(u),e_2(u),\dots,e_F(u)]))\in\mathbb{R}^D$. The feature embeddings $e_k(\cdot)$ are low-dimensional trainable embeddings; typical embedding sizes are 32–64 per categorical feature, the MLP has 2–3 fully connected layers with hidden size approximately 512 and ReLU activations, and the deployed embedding dimension is $D=256$ [2508.03702]. The relevance score is $$ s(u,i)=f_{\text{context}}(u)^\top f_{\text{item}}(x_i)\in[-1,1], $$ which is cosine similarity because both vectors are $\ell_2$-normalized [2508.03702]. For similarity search, training uses sampled softmax with a negative set combining uniform random negatives from the catalogue and hard negatives sampled according to item popularity under "mixed negative sampling" [2508.03702]. For complementary retrieval, the context tower also receives a target complementary category $c^*$ and optimizes $\mathcal{L}_{\mathrm{comp}}=\mathcal{L}_{\mathrm{sim}}+\lambda\|\mathrm{MLP}_{\mathrm{rec}}(f_{\text{context}}(x_q))-e_{\mathrm{cat}}(c^*)\|_2^2$, so that the query representation encodes the intended complementary category [2508.03702]. Serving is organized around ANN indexing. The system uses Faiss with IVF–PQ or HNSW for sub-millisecond neighbourhood search over approximately $200$M product vectors, supports a daily offline pipeline that re-embeds all products and rebuilds the hierarchical index, and serves 20k QPS across desktop and mobile at 40 ms p99 CPU latency for full retrieval and post-processing [2508.03702]. Daily index refresh also addresses catalogue churn: new items embed immediately from content, without an ID-table cold-start [2508.03702]. The same backbone is adapted to three tasks by changing inputs or serving logic rather than replacing the architecture. Similarity-TT uses a single query product and retrieves top-$k$ nearest neighbours; Complementary-TT generates one query embedding per complementary category and interleaves candidate groups; Inspirational-TT builds a hierarchical ANN index via k-means clustering, aggregates the last 100 viewed products by category, skips the closest clusters to avoid substitutes, and interleaves results across categories to maximize diversity [2508.03702]. Over two years of A/B testing, this unified system produced statistically significant gains in engagement and GMV-related metrics across desktop and mobile placements, including CTR increases for similarity and complementary placements and CTA/CVR improvements for inspirational layouts [2508.03702]. ## 3. Limits of isolated towers and the CS3 framework The standard two-tower retriever is efficient because item embeddings can be precomputed, user embeddings cached, and retrieval implemented through ANN search such as FAISS [2604.19269]. The same paper identifies three structural weaknesses of the isolated design. First, representation capacity is constrained because each tower is typically a lightweight MLP with no awareness of the other tower or downstream signals. Second, embedding-space alignment can degrade because the towers do not exchange information before the dot product. Third, cross-feature interactions are absent within the retriever, so richer interactions must be deferred to later ranking stages [2604.19269]. CS3, or Capability Synergy, is proposed as an online-compatible framework that strengthens two-tower retrieval while preserving real-time constraints [2604.19269]. Its first component, Cycle-Adaptive Structure (CAS), replaces each fully connected layer with a "pre-forward → denoise → cycle-forward" block: $$ z_i=f_{\theta_i}(x_i)=\sigma(W_i x_i+b_i), $$ $$ e_i=g_{\phi_i}(z_i)=\mathrm{sigmoid}(W_i' z_i+b_i'), \qquad \tilde x_i=x_i\circ(2\cdot e_i), $$ $$ z_i'=f_{\theta_i}(\tilde x_i)=\sigma(W_i \tilde x_i+b_i). $$ By sharing $\theta_i$ between pre-forward and cycle-forward, CAS adds negligible extra parameters; in the reported system, CAS is integrated directly into the main cross-entropy prediction loss rather than via a separate CAS objective [2604.19269]. The second component, Cross-Tower Synchronization (CTS), injects into each tower a cached cross vector summarizing the partner tower’s recent positive representations. For positive feedback, the vectors are updated by EMA: $$ c_u^t=\alpha c_u^{t-1}+(1-\alpha)v^t,\qquad c_v^t=\alpha c_v^{t-1}+(1-\alpha)u^t. $$ The paper emphasizes that CTS is "loss-free": no explicit alignment loss is introduced; the injected vectors and the main prediction loss are sufficient [2604.19269]. The third component, Cascade-Model Sharing (CMS), reuses a downstream ranker’s intermediate representation $h_{uv}^t$ through EMA-cached vectors $s_u^t$ and $s_v^t$, again without an additional distillation loss [2604.19269]. Empirically, CS3 reports consistent gains over strong baselines on TaobaoAd, KuaiRand, and RecSys2017. For DSSM, the base model yields test AUC / LogLoss of $0.6194 / 0.2289$ on TaobaoAd, $0.6646 / 0.6763$ on KuaiRand, and $0.6855 / 0.6707$ on RecSys2017, whereas DSSM + CS3 reaches $0.6855 / 0.2198$, $0.7484 / 0.5731$, and $0.8380 / 0.5308$, with $p<0.05$ over base [2604.19269]. In a large-scale advertising deployment with 400M DAUs and 10% traffic for 7+ days, the full CS3 system produced +8.356% revenue and +0.468% DAC in Scenario A, while QPS changed by -0.589%; aggregated gains were also reported for Scenarios B and C [2604.19269]. Latency remained within ms-level constraints, with CMS fetches contributing p99 latency below 5 ms and CAS causing less than a 1% QPS drop in the user tower [2604.19269]. A common misconception is that a two-tower retriever must remain interaction-free until the final dot product. CS3 demonstrates a narrower claim: the base architecture is interaction-light, but lightweight mutual awareness, self-revision, and downstream feature sharing can be added without abandoning the two-tower serving regime. ## 4. Interaction-enhanced two-tower pre-ranking HIT, the Hierarchical Interaction-Enhanced Two-Tower model, addresses two limitations explicitly attributed to vanilla two-tower pre-ranking: the absence of cross-tower information exchange before the final score and the inability of a single vector plus a plain inner product to capture multi-faceted user interests and multi-dimensional ad attributes [2505.19849]. Its design preserves the two-tower decomposition while introducing coarse-grained and fine-grained interaction modules. The coarse-grained component is a dual-generator framework. Two simple MLPs, $g_1(\cdot)$ and $g_2(\cdot)$, each with two hidden layers, consume only static features and produce mimic vectors $r_k^m\in\mathbb{R}^p$ [2505.19849]. Generator 1 is trained on positive samples to mimic the holistic opposite-tower representation for target items, while generator 2 is trained on negative samples to mimic non-target representations. The user-side generation loss is $$ \mathcal{L}_{gu} = -\frac{1}{N}\sum_{i=1}^N \bigl[ y_i\,\mathrm{Dist}(r_{u,i},r_{1,i}^m) + (1-y_i)\,\mathrm{Dist}(r_{u,i},r_{2,i}^m) \bigr], $$ with an analogous ad-side loss, and the total objective is $$ \mathcal{L}=\mathcal{L}_{CE}+\alpha(\mathcal{L}_{gu}+\mathcal{L}_{ga}), $$ where $\alpha=10^{-3}$ and $\mathrm{Dist}$ is cosine distance [2505.19849]. Stop-gradient is applied on generator inputs and multi-head outputs during generation-loss computation [2505.19849]. The fine-grained component is the multi-head representer. For each head $j=1,\dots,J$, $$ r_{u,(j)}=W_{u,(j)}h_u^i+b_{u,(j)},\qquad r_{a,(j)}=W_{a,(j)}h_a^i+b_{a,(j)}, $$ and the final score is $$ \hat y=\sum_{j_u=1}^J \max_{j_a\in\{1,\dots,J\}} \bigl(r_{u,(j_u)}^\top r_{a,(j_a)}\bigr). $$ Because all ad-head vectors can be precomputed and cached offline, online inference retains the characteristic two-tower efficiency: only the user tower and a small number of head-level dot products are computed online [2505.19849]. The reported implementation uses embedding dimension $d=32$ for all sparse features, a three-layer MLP backbone with widths $[300,300,32]$, generator widths $[64,32]$, $J=2$ heads with output dimension $z=16$, batch size $256$, Adam, and Tesla T4 GPUs [2505.19849]. Offline, HIT achieves the best AUC on Alibaba, MovieLens-1M, and Amazon-Electronics; on Alibaba, the paper reports AUC $0.7226$ for HIT versus $0.6579$ for DSSM and $0.6827$ for IntTower, with $p<0.01$ for all gains [2505.19849]. Online, on Tencent’s advertising platform with more than 3.6B training samples, 1B users, and 10M ads, HIT improved GMV by 1.66% and ROI by 1.55% versus MVKE, while inference latency and success rate at 35K QPS were nearly identical to vanilla two-tower [2505.19849]. HIT is important because it sharpens a recurrent tension in two-tower design. Efficiency derives from decomposition and pre-caching, but retrieval quality often requires some form of interaction modeling. HIT’s result is not that the decomposition should be discarded; rather, interaction can be pushed into lightweight generator and head-projection structures while preserving the serving mechanics that make two-tower systems practical. ## 5. TwoTower in diffusion language modeling Nemotron-TwoTower extends the name beyond recommendation and retrieval into sequence generation [2606.26493]. Starting from Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model trained on approximately 2.1T tokens, the method instantiates two identical 52-layer copies, each composed of 23 Mamba-2 layers, 6 self-attention layers, and 23 MoE layers [2606.26493]. One copy remains frozen as the AR context tower; the other is trained as the diffusion denoiser tower. The model factorizes the sequence block-autoregressively, $$ \log p_\theta(\mathbf{x}) = \sum_{b=1}^B \log p_\theta(\mathbf{x}_b\mid \mathbf{x}_{