---
title: 'memshare: Shared Memory for R'
url: https://www.emergentmind.com/topics/memshare-e95b7fb1-c6b0-4473-bcf4-7d8846944ce1
type: topic
---

# memshare: Shared Memory for R

memshare is an R package for shared memory multicore computation that allocates buffers in C++ shared memory and exposes them to R through ALTREP views. Its design targets large-object, process-level parallelism in settings such as PSOCK clusters, where ordinary parallel R sessions would otherwise duplicate memory. The package is introduced together with a semantic comparison to SharedObject, a column-wise apply benchmark, and a downstream genomics use case based on feature selection by mutual information with densities estimated via Pareto Density Estimation (PDE) [2509.08632].

## 1. Architectural model

memshare directly allocates buffers in **C++ shared memory**, using `MapViewOfFile` on Windows and `shm` plus `mmap` on Unix, and exposes these buffers to R as **ALTREP** objects [2509.08632]. ALTREP, available since R 3.5.0, permits custom back-ends for R vectors and matrices while remaining transparent to most of R’s API. In memshare, this mechanism is used to present shared-memory-backed objects as ordinary R vectors or matrices.

The package organizes shared memory around two core abstractions: **pages** and **views**. Pages are owned objects, typically created by a master session, whereas views are ALTREP R objects that reference the shared buffer. Multiple R processes can acquire views to the same page and access or update the same physical data without serialization or deserialization overhead. This page/view distinction is central to the package’s semantics, because ownership and access are explicitly separated rather than being folded into a single opaque shared object [2509.08632].

The implementation also introduces **namespaces**, described as string labels that isolate contexts so that different sessions avoid accidental interference. This namespace mechanism is part of the memory lifecycle and safety model rather than a convenience feature alone. It provides a way to separate shared-memory regions used by different workflows or independent parallel jobs.

A plausible implication is that memshare treats shared memory not as an implementation detail hidden behind parallel R workers, but as a first-class storage substrate whose lifetime and visibility are explicitly managed. That orientation distinguishes it from approaches that expose sharing primarily through copy-on-write or subset semantics.

## 2. Programming interface and execution workflow

memshare is designed for **parallel, process-level computation**, especially with R’s **PSOCK clusters** [2509.08632]. In the intended workflow, a master process registers or exports large R objects to shared memory. Worker processes then retrieve views by namespace and memory-page name and operate on those views without duplicating the underlying data.

At the R level, the package provides wrapper functions such as **`memApply()`** and **`memLapply()`**, which mimic `parallel::parApply` and `parallel::parLapply` while automating object registration, buffer management, and cleanup. This gives the package a high-level interface while retaining explicit shared-memory semantics underneath. The interface is therefore intended to reduce the operational burden of creating and releasing shared objects in a multicore workflow.

Memory release is tied to view lifetime: shared memory is released only after all views are gone. This prevents invalid references to deallocated buffers. For observability and debugging, the package includes diagnostic functions **`pageList()`** and **`viewList()`**, which track memory and page usage [2509.08632].

The paper also states an important operational constraint: because **R is single-threaded at the C level**, written changes ideally occur only in controlled scenarios, and memshare is best used with **read-only worker patterns**. A common misconception is to read “shared memory” here as a general-purpose concurrent shared-state mechanism for arbitrary mutation across workers. The implementation details suggest a narrower and safer interpretation: shared physical storage for multicore analytics, with the strongest fit in read-dominant pipelines.

## 3. Semantics, safety, and comparison with SharedObject

The paper compares memshare with **SharedObject (Bioconductor)** along the dimensions of ALTREP usage, API structure, performance, memory behavior, safety, and robustness [2509.08632]. Both use ALTREP-backed representations, but the semantics differ materially.

memshare uses **ALTREP views directly on C++ shared-memory mapped buffers**. Its API emphasizes explicit, high-level functions and a clear separation between “pages” owned by the master and “views” used by workers. SharedObject is described as ALTREP-backed with **copy-on-write** and **partial sharing semantics**, and as offering a lower-level API with finer-grained control, including `copyOnWrite` and `sharedSubset`, but with less clear semantics. The paper further notes that SharedObject’s copy-on-write and ambiguous subset sharing can lead to surprising side effects, as well as possible memory leaks.

A compact summary of the comparison given in the paper is as follows:

| Aspect | memshare | SharedObject |
|---|---|---|
| ALTREP usage | Views directly on C++ shared memory mapped buffers | ALTREP-backed, copy-on-write and partial sharing |
| API style | `memApply`, `memLapply`, explicit pages/views, explicit cleanup | Lower-level API, fine-grained control |
| Safety and semantics | Namespaces, explicit registration/release, diagnostics | Ambiguous subset sharing, possible memory leaks |

The paper also reports that memshare can **handle extremely large objects** and remained stable in the reported experiments, whereas SharedObject is described as failing at extreme scales; in the appendix case at \(100{,}000 \times 100{,}000\), SharedObject triggers an **RStudio crash** [2509.08632]. This robustness claim is tied to the benchmarked workload rather than presented as an unrestricted guarantee for all workloads.

## 4. Empirical performance and scalability

The main benchmark is a **column-wise apply** task, such as computing a per-column standard deviation, across square matrices ranging from \(10^1 \times 10^1\) to \(10^5 \times 10^5\) on a **PSOCK cluster**, using an **18-core Xeon W** system with **256GB RAM** [2509.08632]. The benchmark focuses on elapsed time and resident memory usage under multicore execution.

The reported results characterize memshare as **90–170% faster** than SharedObject in the benchmark. For \(10^2 \leq n \leq 10^4\), the table states that memshare’s memory usage is **132%–153% less** than SharedObject. At the largest tested size, \(10^5\), memshare completes in **108s** with **76GB RSS**, whereas SharedObject crashes with memory exhaustion or leak. The paper further states that memshare’s resident set size across all processes aligns closely with the **baseline (serial)** result, which is presented as evidence of **no per-worker duplication** [2509.08632].

An extract of the reported median results is:

| Configuration | Time / memory result | `mem.after.call` |
|---|---|---|
| SharedObject, \(10{,}000 \times 10{,}000\) | 16.6 sec, 6909.9 MB diff | 41,330 MB |
| memshare, \(10{,}000 \times 10{,}000\) | 1.39 sec, 928 MB diff | 22,821 MB |
| memshare, \(100{,}000 \times 100{,}000\) | 108 sec, 76,312 MB diff | 114,774 MB |

These measurements support the package’s central claim: the use of shared-memory-backed ALTREP views can keep total resident memory close to one physical copy of the data while still enabling process-level parallelism. This is significant specifically for R workflows that are otherwise limited by the replication behavior of standard PSOCK-based parallelization.

## 5. RNA-seq feature selection by mutual information and PDE

The paper’s downstream application is **feature selection by mutual information** on an **RNA-seq** dataset with \(N = 10{,}446\) samples and \(d = 19{,}637\) gene expressions after **TPM normalization**, with a total memory footprint of **more than 47 GB RAM** [2509.08632]. The analytical goal is to compute the mutual information \(I(X;Y)\) between each gene’s continuous expression value \(X\) and a discrete disease class \(Y\).

The reported formula is

$$
I(X;Y) = \sum_{y} p(y) \, KL(p(x|y) \| p(x)).
$$

Here, \(p(y)\) is the probability of class \(y\), \(p(x)\) is the global density for feature \(X\), \(p(x|y)\) is the class-conditional density for \(X\) given \(Y=y\), and \(KL(\cdot\|\cdot)\) is Kullback-Leibler divergence [2509.08632].

For density estimation, the workflow uses **Pareto Density Estimation (PDE)**, described as a **non-parametric univariate density estimator**. Its bandwidth, the **Pareto radius**, is chosen by maximizing information content according to the **Pareto rule (80–20)**, and for each density the bandwidth is approximately the **18% quantile distance**. Because the workflow requires fitting a PDE per gene and per class, it is both RAM- and CPU-intensive.

memshare facilitates this analysis by sharing the entire \(N \times d\) gene-expression matrix and associated vectors so that all worker processes access the same physical data through ALTREP views. The use case is explicitly described as **one core per column**, with all workers reading the same shared data and no redundant RAM use. The paper contrasts this with ordinary PSOCK parallelization, where \(n\_\text{copies} = n\_\text{workers}\); with 10 workers, the workload would require **470 GB+ RAM** and would likely be infeasible. With memshare, the analysis reportedly completed using **~47 GB RAM in two hours** on a shared-memory machine [2509.08632].

The reported mutual-information values across the **19,637 genes** ranged from **0 to 0.54**. PDE and histogram plots revealed **bimodality**, and deviations from normality were confirmed. The paper states that this enables **threshold/Mixture-model-based feature selection** for downstream machine-learning tasks. The significance of the example lies less in novelty of the estimator than in the fact that a workload already fitting into server RAM becomes tractable in multicore R without multiplying memory consumption by the number of worker processes.

## 6. Terminological scope and related uses of the name

The label **“Memshare”** is not unique in the research literature. Besides the R package discussed above, it has also been used for a **dynamic multi-tenant memory key-value cache** and for **memory efficient inference for large reasoning models through KV cache reuse** [1610.08129] [2507.21433]. These systems are unrelated in implementation and domain.

| Name | Domain | Key idea |
|---|---|---|
| memshare | R multicore computation | C++ shared memory exposed through ALTREP views |
| Memshare | Multi-tenant web cache | Dynamic private/shared memory allocation in a log-structured cache |
| MemShare | LRM inference | Reusable KV cache blocks and zero-copy cache reuse |

The 2016 **Memshare** system is a memcached-compatible, multi-tenant DRAM cache with an arbiter, cleaner, and log-structured allocator. It guarantees **private memory** while dynamically allocating **shared memory** to optimize overall hit rate, and on a week-long commercial trace it increased combined hit rate from **84.7%** to **90.8%** and reduced misses by **39.7%** without affecting throughput or latency [1610.08129].

The 2025 **MemShare** system for large reasoning models partitions KV cache into fixed-size blocks and uses a **two-stage hierarchical collaborative filtering** procedure to identify reusable blocks, followed by **zero copy cache reuse**. On MATH-500 with DeepSeek-R1-32B it reports up to **84.79% improvement in throughput** while maintaining better accuracy than existing KV cache management methods [2507.21433].

A broader literature on shared-memory design addresses adjacent but distinct settings, including **multi-GPU systems with truly shared memory** [2008.02300] and **secure inter-VM shared memory** [1909.10377]. This suggests a recurring systems pattern across domains: replacing replication or remote access with explicit shared-memory mechanisms, while treating safety, lifecycle management, coherency, or isolation as first-order design constraints. For the R package memshare, that pattern appears in the combination of ALTREP-backed views, namespace isolation, explicit cleanup, and read-oriented multicore execution [2509.08632].

Source: https://www.emergentmind.com/topics/memshare-e95b7fb1-c6b0-4473-bcf4-7d8846944ce1