---
title: Request-Only Optimizations (ROO)
url: https://www.emergentmind.com/topics/request-only-optimizations-roo
type: topic
---

# Request-Only Optimizations (ROO)

Request-Only Optimizations (ROO) denotes a family of techniques that make the request itself the primary unit of optimization. In the most explicit usage, ROO in recommendation systems treats a user request, rather than an individual impression, as the unit of training data, data processing, and model computation [2508.05640]. Closely related uses appear in batched LLM serving with request-owned mutable state, in adaptive request insertion for waterfall recommender systems, in read-only fast paths for Byzantine state machine replication, in durable read-only transactions on persistent hardware transactional memory, and in caching algorithms that operate from request-derived noisy estimates rather than exact request streams [2605.28053] [2206.12296] [2107.11144] [2410.16110] [2309.02055]. This suggests a broader systems pattern: computational and storage redundancy can be removed only if the semantics attached to each request—ownership, versioning, consistency, resource budget, or locality—remain explicit.

## 1. Defining abstractions and design space

The technical meaning of ROO is context dependent. In recommendation training, a request is written as \(R=(u,\{i_1,i_2,\dots,i_k\})\), where \(u\) denotes user features and each \(i_j\) denotes item features and feedback. In read-write test-time training, each token decode step is exposed as an event
\[
e_i=(\mathrm{owner}_i,\mathrm{version}_i,\mathrm{effect}_i),
\]
with \(\mathrm{effect}_i\in\{\mathsf{READ},\mathsf{WRITE}\}\). In PBFT read-only optimization, clients issue a `READ(r,ts)` directly to replicas and accept a value after collecting a quorum of matching replies. In DUMBO, read-only transactions perform no updates but must still observe a crash-consistent view. In NFPL-based caching, the online policy acts on noisy request estimates \(\hat r_t\) rather than exact request vectors [2508.05640] [2605.28053] [2107.11144] [2410.16110] [2309.02055].

Across these settings, the recurring abstractions are request identity, request-local state, request-local observations, and request-local resource accounting. The optimization target is not merely lower latency or lower storage, but lower redundancy under semantic constraints. A plausible implication is that ROO is best understood not as a single algorithmic template but as a design doctrine: represent what is private to a request only once, batch or shortcut only compatible request phases, and preserve the invariants that determine correctness.

## 2. Request-level data and model co-design in recommendation systems

In the recommendation setting, ROO is a training and modeling paradigm that treats a user request as the unit of data. The contrast is with impression-level training, where each item in a request becomes a separate example and duplicates the entire user feature vector. If \(B_{RO}\) is the number of requests in a mini-batch and \(B_{NRO}\) is the total number of impressions, typically \(B_{NRO}\approx k\cdot B_{RO}\). Under impression-level storage,
\[
S_{\text{impr}}=B_{NRO}(S_{RO}+S_{NRO}),
\]
whereas request-only storage is
\[
S_{\text{ROO}}=B_{RO}S_{RO}+B_{NRO}S_{NRO}
=\bigl(B_{NRO}/k\bigr)S_{RO}+B_{NRO}S_{NRO}.
\]
The resulting storage reduction ratio is
\[
\frac{S_{\text{impr}}}{S_{\text{ROO}}}
=
\frac{k(S_{RO}+S_{NRO})}{S_{RO}+kS_{NRO}}
\xrightarrow[k\gg1]{} k.
\]
The same deduplication appears in preprocessing and embedding fan-out. If \(C_{RO}\) is the cost of user-side processing and \(C_{NRO}\) is the cost of item-side processing, then impression-level cost per mini-batch is \(C_{\text{impr}}=B_{NRO}C_{RO}+B_{NRO}C_{NRO}\), while request-only cost is \(C_{\text{ROO}}=B_{RO}C_{RO}+B_{NRO}C_{NRO}\). The theoretical speedup on RO-side work is therefore \(B_{NRO}/B_{RO}=k\) [2508.05640].

The data pipeline buffers events by request, accumulates all impressions for that request, and emits a single training sample with separate RO and NRO fields. Downstream preprocessing flattens these into two variable-length tensors: a \([B_{RO}\times d_{RO}]\) dense tensor plus a KeyedJaggedTensor for the RO side, and a \([B_{NRO}\times d_{NRO}]\) dense tensor plus a KeyedJaggedTensor for the NRO side. This data representation is paired with ROO-friendly architectures. In two-tower and early-stage ranking models, the user tower depends only on RO features and is computed once per request, then shared across the \(k\) impressions. UserArch applies a Linear Compress Embedding layer on the RO side:
\[
f(X)=g(X)W+b,
\]
with \(X\in\mathbb R^{B_{RO}\times d_{in}\times n_{in}}\), \(W\in\mathbb R^{n_{in}\times n_{out}}\), and amortized cost \(O(B_{RO}\,n_{in}\,n_{out}\,d_{in})\). For autoregressive or generative recommenders, impression-level training re-encodes the user sequence for each target item at cost
\[
C_{\text{imp}}\approx m(n^2d+nd^2),
\]
whereas ROO encodes the concatenated sequence once at cost
\[
C_{\text{ROO}}\approx (n+m)^2d+(n+m)d^2.
\]
For \(n=1000\), \(m=10\), \(d=256\), this is a \(\approx 9.8\times\) saving, and the production implementation is a Hierarchical Sequential Transduction Unit (HSTU).

Across three major Meta recommendation products, the reported effects span storage, throughput, and quality:

| Area | Efficiency result | Quality/result highlight |
|---|---:|---|
| Training sample volume | +43%–150% | prior down-sampling eliminated |
| Retrieval | +48% to +570% throughput | HSTU (ROO) at 0.99× relative FLOPs/example; Recall@100 +3.87% and +2.41%; Topline +4.77% |
| ESR and LSR | ESR: +125% to +266%; LSR: +32% to +100% | NE reductions and statistically significant topline lifts |

The retrieval comparison on Product B is especially notable. Relative to a baseline at \(1.00\times\) FLOPs/example, HSTU trained at impression level uses \(6.8\times\) FLOPs/example and yields Recall@100 gains of \(+3.53\%\) and \(+2.35\%\), whereas HSTU under ROO uses \(0.99\times\) FLOPs/example with Recall@100 gains of \(+3.87\%\) and \(+2.41\%\), plus \(+4.77\%\) topline. In ranking, reported Normalized Entropy deltas range from \(-0.31\%\) to \(-0.86\%\), with corresponding lifts in consumption and engagement. The central significance is that request-level deduplication is not confined to storage; it changes the feasible model class by making more expensive RO-side architectures trainable at production scale.

## 3. Adaptive request insertion in waterfall recommender systems

A different request-centric line of work concerns when to issue an additional paging request inside a waterfall recommender feed. The formal problem defines a set of decision cases \(X=\{X_1,\dots,X_N\}\), local context features \(c_i\), user \(u_i\), behavior history \(b_i\), and binary action \(z_i\in\{0,1\}\), where \(z_i=1\) means inserting a request. The edge-side policy \(F\) maximizes cumulative feedback subject to a global budget \(\theta\):
\[
\text{Maximize } \sum_{i=1}^N S\bigl(F(c_i,u_i,b_i),X_i\bigr)
\]
subject to
\[
\sum_{i=1}^N R\bigl(F(c_i,u_i,b_i),X_i\bigr)\le \theta.
\]
With approximately constant request cost \(\lambda\), the problem reduces to a 0–1 knapsack:
\[
\text{Maximize}_{z_i\in\{0,1\}} \sum_i z_i\cdot CATE_P(X_i)
\quad
\text{subject to }
\sum_i z_i\cdot \lambda\le \theta.
\]
The optimization target is therefore the conditional uplift in purchase rate generated by an inserted request [2206.12296].

AdaRequest decomposes this problem into user-intention modeling, causal uplift estimation, and budgeted decision making. The CUBE module fuses exposed items \(F_{exp}\), session clicks \(F_{sclk}\), long-term clicks \(F_{clk}\), candidate items \(F_{cands}\), and coarse user/context features through GRUs, self-attention, and scaled dot-product attention. For example,
\[
F_{cands\_emb}=\mathrm{mean}\bigl(\mathrm{SelfAttn}(\mathrm{GRU}(F_{cands}))\bigr),
\]
and the session-click matching weights are
\[
\alpha_i=
\frac{\exp(\langle F_{cands\_emb},\hat F_{sclk\_item}^i\rangle)}
{\sum_j \exp(\langle F_{cands\_emb},\hat F_{sclk\_item}^j\rangle)},
\qquad
M_{sclk}=\sum_i \alpha_i F_{sclk\_embs}^i.
\]
The final fused representation is
\[
F_{fusion}=F_{cands\_emb}\Vert M_{exp}\Vert M_{sclk}\Vert M_{clk}\Vert F_{user}\Vert F_{context}.
\]

CREST then estimates uplift under the Conditional Independence Assumption,
\[
\{Y(0),Y(1)\}\perp z\mid X,
\qquad
CATE_P(X)=E[Y(1)\mid X]-E[Y(0)\mid X].
\]
Two heads share the CUBE backbone:
\[
P_{ctrl}=\sigma(Logits_{ctrl}),\qquad
P_{trt}=\sigma(Logits_{ctrl}+V_{uplift}),
\]
so that
\[
\widehat{CATE_P}=P_{trt}-P_{ctrl}.
\]
The DRP decision layer ranks cases by \(\widehat{CATE_P}(X_i)\) and selects the top \(M\%\) per time window, using the previous period’s threshold as a real-time proxy.

The reported experiments use \(1.76\times 10^8\) request-invocation records, \(1.01\times 10^7\) distinct users, and \(2.98\times 10^7\) items from Taobao’s “Guess You Like” waterfall feed. Offline, AdaRequest achieves Qini AUUC \(=1.8288\) versus OneModel \(=1.4767\) and TwoModel \(=1.4007\), Qini@50% \(=4.3450\) versus \(3.8859\), and AUC \(=0.8145\) versus \(0.7980\). Online, PR@10 improves by \(+2.289\%\) versus PoolR \(+1.177\%\), and A/B tests report a \(+3\%\) lift in GMV under a \(+5\%\) to \(21\%\) QPS allowance. Unlike request-level data deduplication, this variant of request optimization concerns the timing of extra requests under a resource cap; the shared idea is that requests are treated as optimizable interventions rather than passive transport events.

## 4. Request-owned mutable state in batched LLM serving

RW-TTT addresses a serving problem created by test-time training. In TTT, each request carries mutable model-side state—fast weights, low-rank parameter deltas, or streaming learner state. Standard batched LLM serving assumes shared static weights; serial execution is correct but slow, while naive batching can corrupt per-request state. RW-TTT formulates this as read-write TTT serving. Each decode event is tagged by owner, version, and effect, and two events are batch-compatible iff they share the same effect, run the same backend operator shape, and their versions match the runtime’s committed versions:
\[
v_i=V(o_i),\qquad v_j=V(o_j).
\]
Grouping is performed by a key \(\kappa=(\mathrm{effect},\mathrm{backend\_type},\mathrm{operator\_shape})\), so that only compatible phases are dispatched together [2605.28053].

The crucial invariant lies in the commit path. After a WRITE batch, the backend returns updated state slices \(\Delta S\) aligned with an owner map \(\mu\). For each batch slot \(b\),
\[
V\bigl(\mu(b)\bigr)\gets V\bigl(\mu(b)\bigr)+1,\qquad
s_{\mu(b)}^{V(\mu(b))}\gets \Delta S_b.
\]
Each request’s new version is therefore published only to its owner. The throughput model defines
\[
T_{\mathrm{total}}^{\mathrm{RW\text{-}TTT}}=\sum_{i=1}^N T_i,
\]
while sequential serving is
\[
T_{\mathrm{total}}^{\mathrm{seq}}=\min_i T_i,
\]
with speed-up
\[
S=\frac{T_{\mathrm{total}}^{\mathrm{RW\text{-}TTT}}}{T_{\mathrm{total}}^{\mathrm{seq}}}.
\]

On one A100-equivalent GPU with eight In-Place-TTT streams, the reported aggregate throughput is \(274.61\) tok/s, compared with \(29.51\) tok/s for sequential TTT and \(79.78\) tok/s for \(3\times\) batch-1 replicas under an equal \(\sim 33\) GiB memory budget. The speed-ups are \(9.31\times\) over sequential and \(3.44\times\) over replicas, with peak memory \(\sim 33.5\) GiB matching the replica baseline. The method preserves behavior on RULER and passes owner/version checks. The reported trade-offs are phase skew, state movement overhead that grows with the number of owners, and smaller effective batch sizes on WRITE-heavy workloads. The broader importance is that mutable per-request adaptation can coexist with accelerator batching only if ownership and versioning are elevated to explicit runtime contracts.

## 5. Read-only requests in Byzantine state machine replication

PBFT’s read-only optimization is a classical fast path for pure reads. In the normal path, an ordinary request proceeds through `REQUEST`, `PRE-PREPARE`, `PREPARE`, `COMMIT`, and then replica replies; the client accepts after collecting \(f+1\) matching replies. Under the read-only optimization, the client broadcasts `READ(r,ts)` to all \(n=3f+1\) replicas, each replica executes the read immediately and returns a signed reply, and the client accepts after collecting a quorum \(q>2n/3\), namely \(2f+1\) matching replies. The optimization reduces latency from five communication steps to two, and linearizability follows from quorum intersection: any read quorum of size \(2f+1\) intersects the update quorum so that at least one correct replica has seen the latest committed update [2107.11144].

The same work shows that this optimization can violate liveness. In the isolating-leader attack, a Byzantine primary sends `PRE-PREPARE(o)` only to \(2f\) replicas, omitting \(f\) honest replicas. The primary and \(2f\) others complete `PREPARE/COMMIT`, execute the update, and reply, but the client obtains at most \(2f\) matching replies and never reaches the \(2f+1\) required by the read-only fast path. Linearizability is preserved, but a correct client can remain blocked indefinitely.

Two repairs are given. Decision broadcasting causes any replica that commits \((c,v)\) with a valid \(2f+1\)-signature COMMIT certificate to broadcast `FWD-DECISION(c,v,\Gamma)` to all replicas; any replica that receives a valid proof decides, executes, and re-broadcasts. Decision forwarding makes this repair on demand: an isolated replica that collects \(f+1\) ACCEPTs but lacks the decision sends `REQ-DECISION(c)` to \(2f\) peers, and any decided peer returns `FWD-DECISION(c,v,\Gamma)`. Both preserve safety by quorum-intersection arguments, and both restore liveness by ensuring that isolated replicas can eventually obtain a valid decision proof.

The BFT-SMaRt prototype quantifies the trade-off. In LAN, ROO read latency stays around \(2\) ms up to saturation, while ordered reads jump above \(20\) ms. In WAN, median ROO reads are \(115\)–\(255\) ms, versus \(350\)–\(550\) ms for ordered reads; on average, ROO is \(2.16\times\) faster at the median and \(2.54\times\) faster at the \(90\)th percentile. Decision broadcasting adds an extra all-to-all broadcast and reduces throughput by about \(10\%\)–\(15\%\), with latency up by about \(10\%\). Decision forwarding matches baseline performance in the fault-free case, is about \(5\%\) faster than broadcasting in LAN under attack, and is worse in WAN under attack because it incurs an additional `REQ-DECISION` round-trip. A common misconception corrected by this work is that read-only fast paths are automatically benign; the result shows that bypassing ordering can preserve linearizability yet still compromise liveness.

## 6. Durable read-only transactions on persistent hardware transactional memory

DUMBO studies read-only transactions in persistent hardware transactions (PHTs), where a read-only transaction performs no writes but must still observe a crash-consistent view of persistent memory. Two bottlenecks dominate prior designs such as SPHT and Pisces. First, HTM imposes read-capacity bounds: if the read set exceeds the transactional cache, the transaction aborts and may fall back to a single-global-lock path. With \(P_{\rm cap}(n)\) denoting the probability of capacity abort on \(n\) threads,
\[
T_{\rm RO}=T_{\rm HTM}(n)\bigl(1-P_{\rm cap}(n)\bigr)+T_{\rm SGL}P_{\rm cap}(n).
\]
Second, durable opacity forces a post-commit durability wait. If \(O(R)\) is the set of observed or concurrent updaters,
\[
T_{\rm wait}(R)=\max_{U\in O(R)}\bigl(\mathrm{Dur}(U)-\mathrm{CommitTime}(R)\bigr)^+.
\]
As thread count rises, these effects can collapse throughput [2410.16110].

DUMBO exploits IBM POWER9 support for suspending and resuming both load and store tracking in HTM. The system maintains a copy-on-write shadow of the persistent heap in DRAM, per-thread redo logs in PM, and a global circular array of durability markers. Threads export
\[
\mathit{state}[t]\in\{\mathrm{inactive},\mathrm{active}(s),\mathrm{nondurable}(s)\}
\]
and a logical durability timestamp \(\mathit{durTS}[t]\). The key operational idea is that suspend/resume allows a transaction to expand its read footprint without capacity aborts and to perform out-of-path synchronization without aborting. IsolationWait scans currently active transactions and enforces the invariant that concurrent update transactions do not read one another’s writes; DurWait prunes the wait set to earlier-committed nondurable transactions.

This changes the asymptotic read-only cost model. For SPHT,
\[
T_{\rm RO}^{\rm SPHT}=\tau_{\rm exec}+\tau_{\rm wait}^{\rm SPHT}\approx \tau_{\rm exec}+O(N)\tau_{\rm poll},
\]
whereas for DUMBO,
\[
T_{\rm RO}^{\rm DUMBO}=\tau_{\rm exec}+O(1)\tau_{\rm poll}\approx \tau_{\rm exec}.
\]
The experimental evaluation uses an IBM POWER9 system with 2 sockets, \(16\) cores each, SMT \(2\), for \(64\) virtual cores, with \(8\) GB DRAM and HTM flushes emulating a \(310\) ns PM latency. On TPC-C StockLevel, where read sets are large, SPHT and HTM throughput goes to zero due to capacity aborts and fallback, while DUMBO and Pisces scale to \(64\) threads, with DUMBO at \(1.0\times\) relative throughput versus Pisces \(0.72\times\). On OrderStatus, DUMBO matches HTM up to \(32\) threads and then continues scaling. On update-only Payment, DUMBO reports an \(11\%\)–\(17\%\) speedup over SPHT; on Delivery, DUMBO-SI yields \(2\times\)–\(4\times\) higher throughput than SPHT/HTM; and in read-dominated mixed workloads, overall throughput is \(2\times\)–\(3\times\) higher. The reported headline result is up to \(4.0\times\) over the state of the art.

## 7. Request-only observations and insertion policies in caching

Caching work uses request-only information in two distinct ways. NFPL assumes that the cache may not know the exact request sequence and instead observes only noisy request estimates. Time is slotted, each slot collects a batch of \(B\) requests, the request vector is \(r_t\in\mathbb N^N\), and the cache decision \(x_t\in\{0,1\}^N\) marks files missing from the cache. The slot cost is
\[
\langle r_t,x_t\rangle.
\]
Against the best static hindsight cache \(x^*\), expected regret is
\[
R_T(A)=\sup_{r_1,\dots,r_T}E\!\left[\sum_{t=1}^T\langle r_t,x_t\rangle-\langle r_{1:T},x^*\rangle\right].
\]
NFPL chooses
\[
x_t=M(\hat r_{1:t-1}+\gamma_t),
\]
where \(\hat r_t\) is an unbiased noisy estimate and \(\gamma_t\sim\mathrm{Uniform}([0,\eta]^N)\). Under unbiasedness and uniform boundedness assumptions, the final guarantee is
\[
R_T(\mathrm{NFPL})\le 2\sqrt{\hat R\,\hat A\,D\,T}=O(\sqrt T).
\]
Special cases include NFPL-Fix and NFPL-Var, with bounds \(R_T\le 2\sqrt2\,B\sqrt{CT}\) and \(R_T\le 2\sqrt2\,(B/f)\sqrt{CT}\), respectively [2309.02055].

Empirically, NFPL is evaluated on Zipf-distributed requests, a real Akamai CDN trace, and an adversarial round-robin trace. On stationary Zipf and Akamai traces, LFU approaches OPT quickly and FPL/NFPL converge more slowly due to noise; on adversarial round-robin, only FPL and NFPL attain near-OPT performance, while LRU and LFU fail. A notable reported observation is that lower sampling can help on round-robin because the added randomness outperforms a static bias. Practical guidance is to set \(\eta\approx \sqrt{\hat R\hat A T/D}\), update every \(B\) requests, and use approximate counters only when they preserve unbiasedness or controlled bias.

A second caching line studies “cache on \(M^{\rm th}\) request” insertion under elastic storage and bandwidth costs. If storage incurs cost \(c_s\) per unit time and misses incur bandwidth cost \(c_b\), with \(R=c_b/c_s\), the offline-optimal cost for a request sequence with inter-request gaps \(a_i\) is
\[
C_{\rm opt}^{\rm offline}=c_b+\sum_{i=2}^N \min\{c_s a_i,c_b\}.
\]
Several online policies are analyzed: Always-on \(1^{\rm th}\), Always-on \(M^{\rm th}\), Single-window on \(M^{\rm th}\), and Dual-window on \(2^{\rm d}\). With \(T=R\), Always-on \(1^{\rm th}\) has competitive ratio at most \(2\); Always-on \(M^{\rm th}\) and Single-window on \(M^{\rm th}\) have competitive ratio at most \(M+1\); and Dual-window on \(2^{\rm d}\) with \(W=T=R\) has competitive ratio at most \(3\). Across exponential, Erlang, deterministic, and Pareto inter-request distributions, the single-parameter dual-window policy \(W=T=R\), \(M=2\), is reported as a robust compromise. Under exponential arrivals, the peak ratio is approximately \(1.588\), and on a campus-network YouTube trace the window-2nd policy yields about \(20\%\)–\(50\%\) total-cost savings over “cache on 1st” [1812.07264].

Taken together, these caching results show two complementary meanings of request-only optimization. One treats request information as incomplete and still seeks no-regret behavior under noise; the other uses request counts themselves as admission signals and optimizes insertion thresholds against elastic costs. In both cases, direct access to full future demand is unavailable, so the request sequence becomes the operative object for online control rather than a fully known workload trace.

Source: https://www.emergentmind.com/topics/request-only-optimizations-roo