---
title: 'Onyx-ORAM: Optimized ORAM for SSD-Backed ANN'
url: https://www.emergentmind.com/topics/onyx-oram
type: topic
---

# Onyx-ORAM: Optimized ORAM for SSD-Backed ANN

Searching arXiv for the primary Onyx paper and closely related ORAM references.
Onyx-ORAM is the ORAM component of **Onyx**, a TEE-based private approximate nearest neighbor (ANN) search system for **disk-hosted** vector indices. It is designed to hide the **disk access pattern** generated by ANN search when the index resides on external SSDs outside the TEE trust boundary. In the Onyx architecture, the system is divided into two co-designed layers: **Onyx-ANNS**, which is a bandwidth-efficient ANN search engine, and **Onyx-ORAM**, which is an access-count-efficient ORAM client over untrusted SSD storage. The defining design claim is an inversion of prior oblivious ANN co-design practice: rather than minimizing **access count in ANN** and **bandwidth in ORAM**, Onyx minimizes **bandwidth** in the ANN layer and **access count** in the ORAM layer, because ANN search can exploit approximation for bandwidth efficiency, whereas ORAM is argued not to face an analogous fundamental lower bound on **I/O request count** [2604.20401].

## 1. Position within ORAM and private storage research

Onyx-ORAM belongs to the line of work on oblivious storage that seeks to hide access patterns to outsourced or untrusted storage, but it is specialized to a TEE setting in which the CPU package and enclave memory are trusted while the host and external SSDs are not. In this model, the host can observe the sequence of SSD I/Os issued by the enclave, and the paper treats that observation channel as first-order leakage because access-pattern attacks can recover queries or infer returned nearest neighbors [2604.20401].

This specialization distinguishes Onyx-ORAM from general single-server online ORAM analyses. In the standard online ORAM model, bandwidth overhead is subject to an inherent logarithmic lower bound. “Stronger Lower Bounds for Online ORAM” proves that any statistically secure online ORAM with internal memory of size \(m\) must perform \(\Omega(n\log n)\) probes in expectation on input sequences of length \(n\), yielding \(\Omega(\log n)\) bandwidth overhead, and that this remains true even when the adversary does **not** know the boundaries between low-level accesses corresponding to logical requests [1903.03385]. The same asymptotic lower bound is obtained for **strong computational security**, while only an \(\omega(1)\) lower bound is proved for **weak computational security** [1903.03385].

That lower-bound context is directly relevant to Onyx-ORAM’s design emphasis. The Onyx paper does not claim sublogarithmic ORAM bandwidth overhead in the standard online sense. Instead, it argues that the relevant optimization axis for SSD-backed ANN inside a TEE is **access count**, because SSD latency and IOPS make the number of I/O operations a dominant cost, and because ORAM has no comparable fundamental lower bound on **access count**. This suggests that Onyx-ORAM should be read not as a refutation of classical ORAM lower bounds, but as a workload- and medium-specific rebalancing of costs under the constraints imposed by disk-hosted ANN search [2604.20401].

## 2. Design objective and architectural role

The Onyx paper frames the central inefficiency of prior oblivious ANN systems as a cross-layer mismatch. DiskANN-style systems are effective in ordinary SSD settings because random 4 KB reads are cheap enough that **access count** dominates, whereas ORAM amplifies bandwidth by \(O(\log N)\), making every extra byte costly. At the same time, many tree-based ORAMs incur many individual I/O requests per logical access, which is particularly harmful on SSDs because latency and throughput are limited by both raw bandwidth and **IOPS** [2604.20401].

Onyx-ORAM addresses this by serving as an **access-count-efficient** ORAM beneath an ANN layer that has already been made bandwidth-efficient. The paper states that the state-of-the-art ORAM-ANN design minimizes access count at the ANN level and bandwidth at the ORAM level, with each optimization worsening the other resource, leaving the combined system overutilizing both bandwidth and access count. Onyx reverses that allocation of responsibilities: ANN search minimizes bandwidth, while ORAM minimizes access count [2604.20401].

Architecturally, Onyx uses **two separate ORAM instances**. A **traversal ORAM** stores small blocks containing adjacency lists \(N\) plus pruning hints \(H\), while a **refinement ORAM** stores full-precision vectors \(F\). The main text states that `Setup(I)` initializes both ORAMs over the ANN index and loads in-memory index state. Expensive per-block ORAM initialization can be avoided using prior **bulk-loading** techniques such as BULKOR and EnigMap. The paper focuses primarily on search, but it also supports `Insert(p, x_p)` and `Delete(p)`, and the appendix discussion says that the decoupled Onyx-ANNS layout extends naturally to FreshDiskANN-like dynamic indices. Updated neighborhoods can be written back piggybacked on ORAM eviction, since any accessed ORAM block must be rewritten anyway [2604.20401].

A plausible implication is that Onyx-ORAM is not intended as a fully general ORAM abstraction independent of workload shape. Its design is tuned to the specific access-size distribution induced by Onyx-ANNS, especially the prevalence of **small traversal blocks** of roughly **256–512 B**.

## 3. Core construction: RingORAM-derived shallow-tree ORAM

Onyx-ORAM is a modified **RingORAM**, specifically the **non-recursive variant of RingORAM** with the **position map stored in the TEE**, and without the XOR trick because the SSD has no compute capability [2604.20401]. Its main innovation is a **“locality-aware shallow tree design”** comprising three stacked changes:

1. **Store bucket metadata locally and increase bucket size**.
2. **Fetch entire buckets during evictions**.
3. **Replace the binary tree by a wide \(d\)-ary shallow tree** [2604.20401].

The baseline RingORAM organization is a tree of buckets in which each logical block is assigned a random leaf according to \(PosMap\) and must reside either on the root-to-leaf path of that leaf or in the client stash. Each bucket stores up to \(Z\) real blocks and is padded to \(Z+S\) slots with dummies. A regular RingORAM access looks up the assigned leaf, reads metadata from every bucket on that path to identify the correct slot, reads either the real block or a dummy at each level, places the block in the stash, and remaps it to a fresh random leaf. Every \(A\) accesses, RingORAM performs deterministic eviction along a fixed path. Ignoring rare early reshuffles, the paper gives the multipliers
\[
AccessCount \approx \left(2 + \frac{Z + 1}{A}\right)\cdot L
\]
and
\[
Bandwidth \approx \left(1 + \frac{2Z+S}{A}\right)\cdot L
\]
where \(L\) is tree depth [2604.20401].

Onyx-ORAM’s first optimization, **Local Bucket Metadata (LBM)**, stores bucket metadata inside the TEE rather than on disk. The metadata stored per bucket includes a version counter \(ver\), a permutation array \(ptrs[0..Z+S-1]\), validity bits \(valid[0..Z+S-1]\), a dummy counter \(dummy\), and an access counter \(count\). This allows the client to determine exactly which slot to read without issuing a metadata read to disk, reducing regular access to **one block read per tree level** rather than metadata plus block. The paper says this changes the access-count expression to
\[
AccessCount \approx \left(1 + \frac{Z+1}{A}\right)\cdot L.
\]
It also enables larger \(Z\), since metadata cost no longer scales as disk I/O [2604.20401].

The second optimization, **Full Bucket Reads during Eviction (FBR)**, trades bandwidth for fewer I/O requests. Instead of RingORAM’s bandwidth-conscious approach of reading only \(Z\) blocks per bucket during eviction, Onyx-ORAM reads the **entire bucket**—all \(Z+S\) slots—in one sequential I/O. This reduces amortized per-level eviction access cost from roughly \((Z+1)/A\) toward \(2/A\), giving
\[
AccessCount \approx \left(1 + \frac{2}{A}\right)\cdot L \approx 1\cdot L
\]
when \(A\) is sufficiently large [2604.20401].

The third optimization, **Shallow \(d\)-ary Tree (ST)**, is the main structural change. Replacing the binary tree by a \(d\)-ary tree reduces depth from
\[
L=\log_2 N
\]
to
\[
L=\log_d N.
\]
For \(d=8\), the paper states that this yields a \(3\times\) reduction in access count. Wider trees require more frequent eviction, and the paper proves that in \(d\)-ary RingORAM the maximum eviction period satisfies
\[
A \le \frac{2Z}{d-1},
\]
compared with \(A \le 2Z\) in the binary case. However, Onyx argues that smaller \(A\) also lets \(S\) shrink proportionally in large-bucket RingORAM, reducing dummy padding. For \(d=8\), the net effect is reported as a \(3\times\) reduction in access count with **essentially no bandwidth penalty** relative to the optimized binary version [2604.20401].

The paper summarizes the resulting I/O multipliers as follows:

| Design | AccessCount | Bandwidth |
|---|---:|---:|
| RingORAM | \(\approx \frac{5}{2}\log(N)\) | \(\approx 3\log(N)\) |
| PathORAM \((\ell=2)\) | \(\approx \log(N)\) | \(\approx 12\log(N)\) |
| PathORAM \((\ell=3)\) | \(\approx \frac{2}{3}\log(N)\) | \(\approx 18.5\log(N)\) |
| Onyx-ORAM | \(\approx \frac{1}{3}\log(N)\) | \(\approx 4\log(N)\) |

This table situates Onyx-ORAM between bandwidth-efficient RingORAM and locality-packed PathORAM, with markedly lower access count than either while remaining much closer to RingORAM on bandwidth [2604.20401].

## 4. Protocol state and access procedure

Onyx-ORAM maintains client-side state consisting of a **position map** \(PosMap\), a **stash** \(Stash\), **per-bucket metadata** \(Meta\), and a **global access counter** \(G\). The server stores an encrypted \(d\)-ary tree of depth \(L\), with each bucket containing \(Z+S\) encrypted slots [2604.20401].

For each block address \(a\),
\[
PosMap[a] = (leaf,\; lvl,\; slot),
\]
where \(leaf\) is the assigned leaf and \((lvl, slot)\) identifies the bucket level and logical slot currently holding the block, or \(\bot\) if the block is in the stash. The cached \((lvl, slot)\) is critical because it lets `ReadPath` directly fetch the correct slot at the matching level and dummies elsewhere [2604.20401].

For bucket \(b\), \(Meta[b]\) stores:
- \(ver\): a monotonic version counter,
- \(ptrs[0..Z+S-1]\): a pseudorandom permutation from logical slots to physical disk offsets,
- \(valid[0..Z+S-1]\): validity bits,
- \(dummy\): which dummy slot to consume next,
- \(count\): number of accesses since the last write, used to trigger early reshuffle at \(S\) [2604.20401].

Each slot is authenticated-encrypted using AES-GCM with associated data
\[
aad = (b \,\|\, k \,\|\, Meta[b].ver),
\]
binding ciphertexts to bucket \(b\), physical slot offset \(k\), and version [2604.20401].

The access protocol `Access(a, op, data')` proceeds as follows. It reads the current position \((leaf_a, lvl_a, slot_a)\) from \(PosMap[a]\), samples a fresh random leaf
\[
leaf' \gets \text{UniformRandom}(0, d^L - 1),
\]
and immediately remaps the block by setting
\[
PosMap[a] \gets (leaf', \bot, \bot).
\]
It then calls `ReadPath(leaf_a, lvl_a, slot_a)`. If the block is not found on disk, it is removed from the stash. For writes, `data` is replaced by `data'`. The updated block is inserted into the stash with its new leaf assignment, the global counter is incremented, `EvictPath()` is called when \(G \bmod A = 0\), `EarlyReshuffle(leaf_a)` is invoked, and the block is returned for reads [2604.20401].

`ReadPath(leaf_a, lvl_a, slot_a)` touches exactly one slot per level. At the known matching level \(i=lvl_a\), it reads the known logical slot \(j=slot_a\); at all other levels it sets \(j=\bot\), meaning “read the next unread dummy.” The logical slot is translated to a physical offset by `GetOffset`, which consumes dummy slots when needed, increments the access count, maps logical slot \(j\) through \(Meta[b].ptrs\), marks the slot invalid after use, and returns the physical offset [2604.20401].

`EvictPath()` computes a deterministic eviction leaf
\[
\ell_e \gets ReverseDigits_{d,L}(G/A \bmod d^L),
\]
reads the **entire bucket** at every level on path \(P(\ell_e)\), merges readable real blocks into the stash, and then traverses levels bottom-up calling `WriteBucket` [2604.20401]. `WriteBucket` selects up to \(Z\) stash blocks assignable to the bucket, removes them from the stash, samples a fresh random permutation
\[
Meta[b].ptrs \gets RandomPerm(0, Z+S-1),
\]
increments the version counter, fills the first \(Z'\) logical slots with real blocks and the rest with dummies, encrypts every physical slot, resets metadata, updates \((lvl,slot)\) entries in \(PosMap\), and writes the whole bucket back as a single I/O [2604.20401].

`EarlyReshuffle(leaf)` checks every bucket on the accessed path and, when
\[
Meta[P(leaf,i)].count = S,
\]
reads the whole bucket, rewrites it, and resets metadata. This ensures that no bucket is read too many times without reshuffling [2604.20401].

## 5. Security model, formal guarantee, and analytical bounds

The Onyx paper formalizes security through a **disk-access privacy** game for a disk-resident ANN system running in a TEE and accessing storage via ORAM. The goal is to hide the physical disk access pattern so that an adversary observing physical disk addresses and ciphertexts cannot distinguish between two equal-size datasets and two matched-type operation sequences. The paper states:

> A disk-resident ANN system provides disk-access privacy if no non-uniform PPT adversary can win the security game with probability non-negligibly higher than random guessing [2604.20401].

What remains public is explicit. Public leakage includes ORAM and ANN public parameters, the operation type (`Search`, `Insert`, `Delete`), and the number and granularity of **logical** accesses per operation, because Onyx fixes these by public parameters such as \(L\), \(L'\), and block sizes. Thus the system hides *which* blocks are accessed, but not the public fixed count or type schedule of accesses [2604.20401].

The formal theorem is:

\[
\textbf{Theorem 1. } \text{Onyx is disk-access private when instantiated with a secure authenticated encryption scheme.}
\]

The proof sketch combines authenticated-encryption security, the fact that Onyx-ANNS makes a **fixed number of logical accesses** depending only on public parameters, ORAM’s obliviousness guarantee for each logical access, and the claim that Onyx-ORAM’s modifications preserve RingORAM’s invariants and security guarantees. The hybrid proof argues that the physical disk address traces are identically distributed because `ReadPath` depends on random leaf assignments and hidden local metadata, `EvictPath` follows deterministic public reverse-lexicographic order, and `EarlyReshuffle` depends only on public bucket access counts [2604.20401].

The main analytical theorem generalizes RingORAM stash bounds to \(d\)-ary trees. For a \(d\)-ary Onyx-ORAM with \(N\) blocks, bucket capacity \(Z\), eviction period \(A\), and depth
\[
L = \left\lceil \log_d \frac{2N}{A(d-1)} \right\rceil,
\]
let
\[
a = \frac{A(d-1)}{2}.
\]
If
\[
q = Z \ln(Z/a) + a - Z - 1 - \ln d > 0,
\]
then
\[
\Pr[\mathit{stash} > R] \;\leq\; \frac{(a/Z)^{R}}{1 - e^{-q}}.
\]
The paper notes that setting \(d=2\) recovers the original RingORAM result. A key consequence is again
\[
A \le \frac{2Z}{d-1},
\]
so wider trees require proportionally more frequent eviction [2604.20401].

This theorem should be distinguished from the lower bounds for general online ORAM. The lower-bound work shows that statistically secure and strongly computationally secure online ORAMs incur \(\Omega(\log n)\) bandwidth overhead and that obscuring the boundaries between logical requests does not evade that asymptotic cost [1903.03385]. Onyx-ORAM does not dispute that claim. Instead, it modifies the tradeoff between **bandwidth consumption** and **I/O request count** within a RingORAM-derived construction under a TEE threat model and an SSD workload.

## 6. Performance characteristics, evaluation, and relation to prior ORAM directions

The paper’s effective overhead summary for the chosen \(d=8\) regime is
\[
AccessCount \approx \frac{1}{3}\log(N)
\]
and
\[
Bandwidth \approx 4\log(N),
\]
as reported in Table 1 [2604.20401]. The appendix further reports measured per-block memory footprints of approximately \(\sim 8.7\) bytes/block for Onyx-ORAM with \(d=8\), \(\sim 8.6\) bytes/block for RingORAM, \(\sim 7\) bytes/block for PathORAM, and \(\sim 17.6\) bytes/block for binary RingORAM with local bucket metadata. This indicates that the \(d\)-ary shallow tree is not only an access-count device but also a way to make local metadata economically feasible in TEE memory [2604.20401].

The ORAM microbenchmarks were run on AMD EPYC 9554 with **AMD SEV-SNP** and Micron 7450 MAX SSDs. The primary ORAM evaluation used a **1-SSD** configuration with 1 vCPU, 4 GB RAM, 1 SSD slice, 85K IOPS, 330 MB/s read bandwidth, 175 MB/s write bandwidth, and 187.5 GB storage. The evaluation used **20M blocks**, matching WIKI-20M scale. Parameters included early reshuffle rate \(0.1\%\), \(Z \in [32,128]\) for RingORAM, \(Z \in [128,1024]\) for Onyx-ORAM, and \(Z=4\) for PathORAM with packed-level parameter \(\ell\) tuned by block size [2604.20401].

At small block sizes of **256–512 B**, Onyx-ORAM sustains **over 10K logical accesses/s**. At **512 B** on 1-SSD, it achieves up to **\(5.1\times\)** higher throughput and **\(2.7\times\)** lower latency than RingORAM, and up to **\(3.4\times\)** higher throughput and **\(2.0\times\)** lower latency than locality-optimized PathORAM. At 4-SSD, the advantage remains substantial: **\(3.9\times\)** higher throughput and **\(2.7\times\)** lower latency than RingORAM, and **\(2.9\times\)** higher throughput and **\(2.0\times\)** lower latency than PathORAM [2604.20401].

The ablation study validates each component of the design: LBM removes metadata I/O, FBR collapses many eviction reads into one sequential read, and ST with varying arities further improves performance, with **\(d=8\)** giving the best tradeoff analytically and experimentally [2604.20401].

The end-to-end evaluation attributes a substantial portion of full-system gains to Onyx-ORAM. On small-vector datasets, **Onyx-ORAM + DiskANN** comes within roughly **10–20%** of full Onyx, indicating that ORAM is the main bottleneck there. Replacing Onyx-ORAM with RingORAM or PathORAM while keeping Onyx-ANNS causes slowdowns of **\(4.9\times\)–\(5.0\times\)** for Ring + Onyx-ANNS and **\(2.3\times\)–\(2.4\times\)** for Path + Onyx-ANNS [2604.20401].

Storage amplification for ORAM alone is reported as about **\(1.7\times\)–\(1.8\times\)** plaintext, and end-to-end Onyx storage over plaintext index as **\(2.0\times\)–\(2.7\times\)**. The corresponding baselines are **\(2.1\times\)–\(3.8\times\)** for PathORAM + DiskANN and **\(7.5\times\)–\(7.7\times\)** for RingORAM [2604.20401].

These results place Onyx-ORAM in contrast with other practical ORAM directions. **Succinct Oblivious RAM** focuses on minimizing server storage overhead, achieving server space \(n+o(n)\) while retaining worst-case \(O(\log^2 n)\) bandwidth blowup and small client space in the single-server setting [1804.08285]. That work is a benchmark on the **space-efficiency** axis rather than the SSD access-count axis. By contrast, **vORAM** and the HIRB-based oblivious map construction target practical outsourced key/value maps, variable-size blocks, secure deletion, and bounded history leakage, with practical emphasis on map-layer efficiency and post-compromise guarantees rather than disk-resident ANN search [1505.07391]. Onyx-ORAM is therefore best understood as an ORAM specialized to SSD-backed ANN inside a TEE, rather than a general replacement for succinct ORAMs or ORAM-backed oblivious maps.

## 7. Scope, limitations, and interpretive cautions

Several limitations are explicit in the paper. First, Onyx-ORAM increases **client-side metadata** by storing bucket metadata locally. The \(d\)-ary tree mitigates this, but it remains a deliberate memory tradeoff and one reason memory use can exceed that of PathORAM-based baselines [2604.20401].

Second, **full-bucket eviction** intentionally spends extra bandwidth to reduce access count. This is favorable in the target SSD regime, especially after dummy-slot shrinkage from the shallow tree, but the tradeoff could be less favorable where bandwidth is the sole bottleneck [2604.20401]. Third, the design’s strongest advantage occurs at **256–512 B** block sizes; at very large block sizes, Onyx-ORAM becomes bandwidth-bound and converges toward RingORAM performance [2604.20401]. This dependence underscores that the ORAM is co-designed with Onyx-ANNS rather than workload-agnostic.

Fourth, performance depends on practical tuning of \(d\), \(Z\), \(S\), \(A\), and the early reshuffle rate. The paper reports that \(d=8\) is best; \(d=16\) reduces access count further but harms bandwidth [2604.20401]. Fifth, the security model excludes TEE side channels such as cache attacks, page-fault leakage, timing side channels, transient execution, and a compromised TEE. The paper is specifically about **disk access pattern leakage** [2604.20401].

Sixth, the evaluation mainly focuses on search. Insertions and deletions are supported, and the dynamic-index appendix indicates likely applicability to FreshDiskANN-like structures, but update-heavy workloads are less fully characterized [2604.20401]. Seventh, the scheme does not hide everything: the operation type and the public fixed search budget remain visible because logical access count and granularity are fixed by public parameters [2604.20401].

A common misconception would be to treat Onyx-ORAM as an argument against established ORAM lower bounds. The evidence does not support that interpretation. The lower-bound result for online ORAM states that statistically secure and strongly computationally secure online ORAMs with small client memory incur \(\Omega(\log n)\) bandwidth overhead even when adversaries cannot identify request boundaries, so boundary obfuscation is not a route to asymptotically better online ORAM [1903.03385]. Onyx-ORAM instead exploits a different degree of freedom: it reorganizes the internal structure of a RingORAM-derived design to reduce **I/O request count** while accepting a carefully managed bandwidth tradeoff, and it does so in a setting where access count is the dominant practical cost [2604.20401].

This suggests a more precise characterization. Onyx-ORAM is a **RingORAM-derived, shallow \(d\)-ary, access-count-optimized ORAM** for hiding SSD access patterns in TEE-based disk-resident ANN search. Its main contribution is not a new asymptotic lower-bound escape hatch, nor a general-purpose ORAM definition, but a concrete protocol and evaluation showing that in SSD-backed ANN workloads, **access count** can be a more valuable ORAM optimization target than the bandwidth-centric metrics that dominate much of the classical ORAM literature [2604.20401].

Source: https://www.emergentmind.com/topics/onyx-oram