memshare: Shared Memory for R

• memshare is an R package for shared-memory multicore computation that allocates C++ buffers and exposes them to R as ALTREP views.
• It employs a page/view architecture with namespaces, allowing multiple processes to access and update the same physical data without duplication.
• Empirical benchmarks demonstrate 90–170% speed improvements and significant memory savings over alternatives like SharedObject in large-scale genomics tasks.

memshare is an R package for shared memory multicore computation that allocates buffers in C++ shared memory and exposes them to R through ALTREP views. Its design targets large-object, process-level parallelism in settings such as PSOCK clusters, where ordinary parallel R sessions would otherwise duplicate memory. The package is introduced together with a semantic comparison to SharedObject, a column-wise apply benchmark, and a downstream genomics use case based on feature selection by mutual information with densities estimated via Pareto Density Estimation (PDE) (Thrun et al., 10 Sep 2025).

1. Architectural model

memshare directly allocates buffers in C++ shared memory, using MapViewOfFile on Windows and shm plus mmap on Unix, and exposes these buffers to R as ALTREP objects (Thrun et al., 10 Sep 2025). ALTREP, available since R 3.5.0, permits custom back-ends for R vectors and matrices while remaining transparent to most of R’s API. In memshare, this mechanism is used to present shared-memory-backed objects as ordinary R vectors or matrices.

The package organizes shared memory around two core abstractions: pages and views. Pages are owned objects, typically created by a master session, whereas views are ALTREP R objects that reference the shared buffer. Multiple R processes can acquire views to the same page and access or update the same physical data without serialization or deserialization overhead. This page/view distinction is central to the package’s semantics, because ownership and access are explicitly separated rather than being folded into a single opaque shared object (Thrun et al., 10 Sep 2025).

The implementation also introduces namespaces, described as string labels that isolate contexts so that different sessions avoid accidental interference. This namespace mechanism is part of the memory lifecycle and safety model rather than a convenience feature alone. It provides a way to separate shared-memory regions used by different workflows or independent parallel jobs.

A plausible implication is that memshare treats shared memory not as an implementation detail hidden behind parallel R workers, but as a first-class storage substrate whose lifetime and visibility are explicitly managed. That orientation distinguishes it from approaches that expose sharing primarily through copy-on-write or subset semantics.

2. Programming interface and execution workflow

memshare is designed for parallel, process-level computation, especially with R’s PSOCK clusters (Thrun et al., 10 Sep 2025). In the intended workflow, a master process registers or exports large R objects to shared memory. Worker processes then retrieve views by namespace and memory-page name and operate on those views without duplicating the underlying data.

At the R level, the package provides wrapper functions such as memApply() and memLapply(), which mimic parallel::parApply and parallel::parLapply while automating object registration, buffer management, and cleanup. This gives the package a high-level interface while retaining explicit shared-memory semantics underneath. The interface is therefore intended to reduce the operational burden of creating and releasing shared objects in a multicore workflow.

Memory release is tied to view lifetime: shared memory is released only after all views are gone. This prevents invalid references to deallocated buffers. For observability and debugging, the package includes diagnostic functions pageList() and viewList(), which track memory and page usage (Thrun et al., 10 Sep 2025).

The paper also states an important operational constraint: because R is single-threaded at the C level, written changes ideally occur only in controlled scenarios, and memshare is best used with read-only worker patterns. A common misconception is to read “shared memory” here as a general-purpose concurrent shared-state mechanism for arbitrary mutation across workers. The implementation details suggest a narrower and safer interpretation: shared physical storage for multicore analytics, with the strongest fit in read-dominant pipelines.

3. Semantics, safety, and comparison with SharedObject

The paper compares memshare with SharedObject (Bioconductor) along the dimensions of ALTREP usage, API structure, performance, memory behavior, safety, and robustness (Thrun et al., 10 Sep 2025). Both use ALTREP-backed representations, but the semantics differ materially.

memshare uses ALTREP views directly on C++ shared-memory mapped buffers. Its API emphasizes explicit, high-level functions and a clear separation between “pages” owned by the master and “views” used by workers. SharedObject is described as ALTREP-backed with copy-on-write and partial sharing semantics, and as offering a lower-level API with finer-grained control, including copyOnWrite and sharedSubset, but with less clear semantics. The paper further notes that SharedObject’s copy-on-write and ambiguous subset sharing can lead to surprising side effects, as well as possible memory leaks.

A compact summary of the comparison given in the paper is as follows:

Aspect	memshare	SharedObject
ALTREP usage	Views directly on C++ shared memory mapped buffers	ALTREP-backed, copy-on-write and partial sharing
API style	memApply, memLapply, explicit pages/views, explicit cleanup	Lower-level API, fine-grained control
Safety and semantics	Namespaces, explicit registration/release, diagnostics	Ambiguous subset sharing, possible memory leaks

The paper also reports that memshare can handle extremely large objects and remained stable in the reported experiments, whereas SharedObject is described as failing at extreme scales; in the appendix case at $100{,}000 \times 100{,}000$, SharedObject triggers an RStudio crash (Thrun et al., 10 Sep 2025). This robustness claim is tied to the benchmarked workload rather than presented as an unrestricted guarantee for all workloads.

4. Empirical performance and scalability

The main benchmark is a column-wise apply task, such as computing a per-column standard deviation, across square matrices ranging from $10^1 \times 10^1$ to $10^5 \times 10^5$ on a PSOCK cluster, using an 18-core Xeon W system with 256GB RAM (Thrun et al., 10 Sep 2025). The benchmark focuses on elapsed time and resident memory usage under multicore execution.

The reported results characterize memshare as 90–170% faster than SharedObject in the benchmark. For $10^2 \leq n \leq 10^4$, the table states that memshare’s memory usage is 132%–153% less than SharedObject. At the largest tested size, $10^5$, memshare completes in 108s with 76GB RSS, whereas SharedObject crashes with memory exhaustion or leak. The paper further states that memshare’s resident set size across all processes aligns closely with the baseline (serial) result, which is presented as evidence of no per-worker duplication (Thrun et al., 10 Sep 2025).

An extract of the reported median results is:

Configuration	Time / memory result	mem.after.call
SharedObject, $10{,}000 \times 10{,}000$	16.6 sec, 6909.9 MB diff	41,330 MB
memshare, $10{,}000 \times 10{,}000$	1.39 sec, 928 MB diff	22,821 MB
memshare, $100{,}000 \times 100{,}000$	108 sec, 76,312 MB diff	114,774 MB

These measurements support the package’s central claim: the use of shared-memory-backed ALTREP views can keep total resident memory close to one physical copy of the data while still enabling process-level parallelism. This is significant specifically for R workflows that are otherwise limited by the replication behavior of standard PSOCK-based parallelization.

5. RNA-seq feature selection by mutual information and PDE

The paper’s downstream application is feature selection by mutual information on an RNA-seq dataset with $N = 10{,}446$ samples and $d = 19{,}637$ gene expressions after TPM normalization, with a total memory footprint of more than 47 GB RAM (Thrun et al., 10 Sep 2025). The analytical goal is to compute the mutual information $10^1 \times 10^1$0 between each gene’s continuous expression value $10^1 \times 10^1$1 and a discrete disease class $10^1 \times 10^1$2.

The reported formula is

$10^1 \times 10^1$3

Here, $10^1 \times 10^1$4 is the probability of class $10^1 \times 10^1$5, $10^1 \times 10^1$6 is the global density for feature $10^1 \times 10^1$7, $10^1 \times 10^1$8 is the class-conditional density for $10^1 \times 10^1$9 given $10^5 \times 10^5$0, and $10^5 \times 10^5$1 is Kullback-Leibler divergence (Thrun et al., 10 Sep 2025).

For density estimation, the workflow uses Pareto Density Estimation (PDE), described as a non-parametric univariate density estimator. Its bandwidth, the Pareto radius, is chosen by maximizing information content according to the Pareto rule (80–20), and for each density the bandwidth is approximately the 18% quantile distance. Because the workflow requires fitting a PDE per gene and per class, it is both RAM- and CPU-intensive.

memshare facilitates this analysis by sharing the entire $10^5 \times 10^5$2 gene-expression matrix and associated vectors so that all worker processes access the same physical data through ALTREP views. The use case is explicitly described as one core per column, with all workers reading the same shared data and no redundant RAM use. The paper contrasts this with ordinary PSOCK parallelization, where $10^5 \times 10^5$3; with 10 workers, the workload would require 470 GB+ RAM and would likely be infeasible. With memshare, the analysis reportedly completed using ~47 GB RAM in two hours on a shared-memory machine (Thrun et al., 10 Sep 2025).

The reported mutual-information values across the 19,637 genes ranged from 0 to 0.54. PDE and histogram plots revealed bimodality, and deviations from normality were confirmed. The paper states that this enables threshold/Mixture-model-based feature selection for downstream machine-learning tasks. The significance of the example lies less in novelty of the estimator than in the fact that a workload already fitting into server RAM becomes tractable in multicore R without multiplying memory consumption by the number of worker processes.

6. Terminological scope and related uses of the name

The label “Memshare” is not unique in the research literature. Besides the R package discussed above, it has also been used for a dynamic multi-tenant memory key-value cache and for memory efficient inference for large reasoning models through KV cache reuse (Cidon et al., 2016, Chen et al., 29 Jul 2025). These systems are unrelated in implementation and domain.

Name	Domain	Key idea
memshare	R multicore computation	C++ shared memory exposed through ALTREP views
Memshare	Multi-tenant web cache	Dynamic private/shared memory allocation in a log-structured cache
MemShare	LRM inference	Reusable KV cache blocks and zero-copy cache reuse

The 2016 Memshare system is a memcached-compatible, multi-tenant DRAM cache with an arbiter, cleaner, and log-structured allocator. It guarantees private memory while dynamically allocating shared memory to optimize overall hit rate, and on a week-long commercial trace it increased combined hit rate from 84.7% to 90.8% and reduced misses by 39.7% without affecting throughput or latency (Cidon et al., 2016).

The 2025 MemShare system for large reasoning models partitions KV cache into fixed-size blocks and uses a two-stage hierarchical collaborative filtering procedure to identify reusable blocks, followed by zero copy cache reuse. On MATH-500 with DeepSeek-R1-32B it reports up to 84.79% improvement in throughput while maintaining better accuracy than existing KV cache management methods (Chen et al., 29 Jul 2025).

A broader literature on shared-memory design addresses adjacent but distinct settings, including multi-GPU systems with truly shared memory (Mojumder et al., 2020) and secure inter-VM shared memory (Sreenivasamurthy et al., 2019). This suggests a recurring systems pattern across domains: replacing replication or remote access with explicit shared-memory mechanisms, while treating safety, lifecycle management, coherency, or isolation as first-order design constraints. For the R package memshare, that pattern appears in the combination of ALTREP-backed views, namespace isolation, explicit cleanup, and read-oriented multicore execution (Thrun et al., 10 Sep 2025).