---
title: 'SilverTorch: Unified GPU-based DLRM Serving'
url: https://www.emergentmind.com/topics/silvertorch
type: topic
---

# SilverTorch: Unified GPU-based DLRM Serving

SilverTorch is a unified, model-based system designed for serving large-scale deep learning recommendation models (DLRM) entirely on GPUs, replacing traditional multi-service pipelines dependent on CPU-based approximate nearest neighbor (ANN) and filtering systems. SilverTorch encodes the entire retrieval and ranking stack as a single tensorized model graph, compiled for GPU execution, thereby enabling significant efficiency, scalability, and modeling improvements in production recommendation systems [2511.14881].

## 1. Model-Based Recommendation Serving Architecture

SilverTorch formalizes a model-based serving paradigm in which all retrieval, filtering, multi-task score aggregation, and early-stage ranking operations are expressed as tensor operators within a single PyTorch-derived computation graph. This approach eliminates the need for separately managed microservices for indexing, ANN search, and filter logic. The system consolidates the following model components:

- User Tower: Computes user embedding.
- Bloom-index Layer: Performs feature-level filtering using a GPU-optimized Bloom filter.
- Tensor-native Int8 ANN Layer: Executes approximate kNN search via quantized inner products.
- OverArch Scoring Layer: Captures complex user-item interactions and supports multi-task outputs.
- Value Model: Aggregates multi-task outputs into single scores through in-model logic.
- Embedding Cache: Stores precomputed item embeddings for efficient early-stage ranking (ESR).

End-to-end, the pipeline involves offline training and quantization of multi-tower models, construction of IVF+Int8 ANN and Bloom-filter indexes as GPU-resident tensors, TorchScript model artifact baking, and a serving-time “single forward pass” executing all retrieval, filtering, scoring, and aggregation within one model invocation [2511.14881].

## 2. GPU Bloom Index Algorithm for Feature Filtering

SilverTorch utilizes a GPU-optimized Bloom index for pre-retrieval feature filtering. Each item $i$ is assigned an $M$-bit signature $VB_i$. Features are hashed via $k$ independent hash functions; the query’s features yield an analogous query vector $QB$. An item passes the filter if

$$
(QB \wedge VB_i) = QB
$$

False-positive probability for $n$ distinct query features is

$$
P_{fp} = (1 - e^{-kn/m})^k
$$

Typical deployment parameters are $m=1024$ bits and $k=5$, yielding $P_{fp} \approx 0.07\%$. Item signatures are laid out in transposed $u64$-aligned memory for warp-efficient bitwise computation. Insertions are $O(N_{items}\cdot k)$, and queries are $O((M/64)\cdot(N_{items}/64))$ bit-operations. The Bloom mask’s compact representation enables contiguous loading and masking across large candidate sets in a fully parallelized fashion [2511.14881].

## 3. Fused Int8 ANN Search: Indexing and Query Pipeline

Index construction leverages KMeans++ partitioning into $C$ centroids in $\mathbb{R}^D$, quantizing all item and centroid embeddings to int8 values:

$$
x_i^{(8)} = \left\lfloor 128 \cdot \frac{x_i - x_{min}}{x_{max} - x_{min}} \right\rceil \in [-128,127]
$$

At query time, user embeddings are similarly quantized; a fused batched matmul computes $B \times C$ int8 inner products via the dp4a instruction:

$$
S_{b,c} = \sum_{d=1}^D q^{(8)}_{b,d} \times c^{(8)}_{c,d}
$$

The top $P$ centroids are probed; cluster items’ int8 embeddings are streamed for further dp4a-based scoring with the quantized user embedding. No intermediate gather buffer is required, maximizing tensor throughput. The global top-$K_0$ heap is maintained in registers/shared memory. Overall complexity per query is $O(P\cdot|\text{cluster}| \cdot D / 32)$ dp4a ops. Int8 quantization halves the ANN memory footprint compared to float32 [2511.14881].

## 4. Co-Design of Filtering and ANN Indexes

In SilverTorch, Bloom and ANN indexes are co-designed as GPU tensors, permitting direct application of the Bloom bitmask within the ANN matmul/gather operation. This eliminates the need to instantiate a full per-item boolean mask:

- Memory requirement for masking reduces from $N_{items}$ bytes (bool8) to $N_{items}/8$ bytes (bit-packed).
- The effective scanned-item count after Bloom filtering is proportional to the filter’s selectivity $\alpha$: if $\alpha \approx 0.1$, the number of items scored by ANN is reduced by approximately 90%, saving dp4a operations and bandwidth.
- Zeroing out scores for non-matching items is accomplished in-place by direct bitmask application during the fused computation [2511.14881].

## 5. Multi-Task OverArch Layer and Value Model Aggregation

Post-ANN, SilverTorch retrieves cached float16 item embeddings $E \in \mathbb{R}^{K_0 \times D}$, concatenates user, item, and cross features, and applies the OverArch block—either a shallow MLP or Mixture-of-Logits (MoL)—to produce $T$ logits for $T$ distinct tasks. The Value Model aggregates these via a weighted sum or specified logic:

$$
s_i = \sum_{t=1}^T \alpha_t f_\theta^{(t)}(u,i)
$$

A plausible implication is that arbitrary business logic or external JSON-driven rules can be encoded in the Value Model. In training, a multi-task objective is used:

$$
\mathcal{L} = \sum_{t=1}^T \alpha_t \mathcal{L}_t(f_\theta^{(t)}(u,i), y_t)
$$

OverArch and Value Model are trained offline within the main two-tower setup. After aggregation, the top-$K_1$ items are passed to downstream ESR scoring, again leveraging embedding caching [2511.14881].

## 6. Embedding Caching for Early-Stage Ranking Acceleration

For Early-Stage Ranking (ESR), precomputing and caching all item embeddings $E_i$ during publish time, and storing them as a contiguous $(N_{items} \times D)$ GPU tensor, enables basket gathers at inference:

$$
E_{ESR} = \mathsf{embedding\_cache}[\text{top-}K_0\ \text{IDs}]
$$

This obviates per-item recomputation and GPU–GPU fetch overhead, resulting in up to $10\times$ fewer GPU cycles and $10\times$ higher query-per-second (QPS) rates in ESR. The throughput advantage scales linearly with $K_0$ [2511.14881].

## 7. Empirical Evaluation and Deployment Outcomes

Evaluation on two datasets (10M and 80M items, $D=128$, A100-40GB GPU; 5,000 request replay) demonstrates substantial improvements:

| Configuration               | Throughput (QPS) | p99 Latency (ms) | Cost/1k req | Recall@200 (E-task) |
|-----------------------------|------------------|------------------|-------------|---------------------|
| CPU-ANN + filter            | 51               | ~160             | $0.158      | 0.291               |
| GPU-ANN + filter (Faiss)    | 340              | ~40              | $0.0272     | -                   |
| ST-Retrieval                | 1,210            | ~14              | $0.0077     | 0.291               |
| ST-Retrieval+OverArch       | 771              | ~28              | $0.012      | 0.331               |

SilverTorch achieves up to $23.7\times$ greater throughput and $11.4\times$ lower p99 latency than CPU baselines; cost-efficiency improvement is $13.35\times$. OverArch+ValueModel delivers recall gains (“E-task” Recall@200 increases from 0.291 to 0.331, or $+14.6\%$). SilverTorch is deployed across hundreds of retrieval and ESR models in production, recommending content to billions of daily active users [2511.14881].

Source: https://www.emergentmind.com/topics/silvertorch