Protected User-Level Libraries
- Protected user-level libraries are defined as user-space components that shield internal states and resource accesses through explicit API mediation and isolation.
- They employ architectural patterns like sandboxing, token-based permissions, and trusted execution to enforce refined, least-privilege access controls.
- These libraries mitigate process-level authority issues by transforming conventional libraries into distinct security principals with clear accountability boundaries.
Protected user-level libraries are user-space libraries or library-like service components whose internal state, privileged interfaces, or resource accesses are protected from their callers or from peer applications while remaining callable through familiar APIs. In the cited literature, the term spans several concrete designs: Android “proactive libraries” that augment reactive APIs with runtime enforcers (Riganelli et al., 2019), in-app mandatory-access-control libraries for Android components (Wu et al., 2018), token-based library permissions in Rust (Gehring et al., 13 Jun 2025), compiler- and hardware-enforced intra-process partitions for C/C++ libraries (Agadakos et al., 2022), fine-grain sandboxing of third-party browser libraries (Narayan et al., 2020), protected CUDA services and trusted library adapters inside one GPU context (Yang et al., 4 Jun 2026), protected kernel-bypass services shared among mutually untrusting applications (Beadle et al., 2 Sep 2025), enclave-based cryptographic libraries (Mofrad et al., 2017), and x86 ring-based “PrivUser” compartments for in-process secret handling (Lee et al., 2018). Across these systems, the common objective is to replace ambient authority at the library boundary with explicit mediation, isolation, or capability transfer.
1. Problem setting and conceptual scope
The underlying problem is that conventional operating systems and language runtimes typically assign authority at the process or application level, while modern software composes functionality through libraries. In Rust, this appears as the mismatch between process-level permissions and the desire to manage access to system resources “per library”; PermRust states that OS permissions “can at most manage access on the process-level,” so any third-party library inside the process inherits the app’s authority unless the language introduces a finer-grained permission system (Gehring et al., 13 Jun 2025). In JavaScript, Mir makes the same point in dynamic-language terms: libraries often execute with “significantly more privilege than needed,” including ambient access to globals, imports, process.env, filesystem, and network facilities (Vasilakis et al., 2020).
On Android, the issue is not only confidentiality but also protocol correctness and system stability. “Proactive libraries” are defined as reactive libraries augmented with proactive modules, or software enforcers, that can suppress, insert, or modify API calls so that resource usage remains correct even when apps misuse cameras, media players, wakelocks, or location services (Riganelli et al., 2019). SCLib generalizes this library-boundary view to inter-component communication by performing in-app mandatory access control on behalf of app components without firmware modification or repackaging (Wu et al., 2018).
A related privacy motivation arises when libraries become cross-application principals. The intra-library collusion study shows that a single third-party library embedded in multiple Android apps can exploit the union of those apps’ permissions at the server side; on 57.6% of devices in its dataset, at least one library could gain extra permissions in this way, and the risk increased over time (Taylor et al., 2017). This suggests that “protected user-level libraries” are not only a systems abstraction for isolation, but also a way to recognize libraries as security principals with their own authority, state, and accountability boundaries.
2. Architectural patterns
One major pattern is library-boundary mediation. AgileOS “virtualizes CUDA at the library boundary”: applications link against client-side CUDA Runtime, Driver, and selected library shims, while a trusted worker owns the real CUDA context, manages virtualized CUDA object tables, validates pointers and handles, loads transformed PTX, and invokes trusted adapters for libraries such as cuFFT and future PyTorch support (Yang et al., 4 Jun 2026). Mir intercepts require/import, constructs a wrapped context, rewrites the module into a closure bound to that context, and wraps exported interfaces so that every cross-module access is mediated by an RWXI permission policy (Vasilakis et al., 2020).
A second pattern is intra-process compartmentalization. Polytope assigns code and data to partitions through C++11 attributes and pragmas, embeds policy metadata into LLVM IR, and enforces source-level read/write partition privileges with Intel MPK and a run-time support library (Agadakos et al., 2022). LOTRx86 creates an intermediate x86 privilege layer called PrivUser by combining ring 2 compatibility mode, ring 1 gate mode, segmentation, call gates, and a privcall ABI, thereby placing sensitive routines and data in a process-local supervisor-only subspace inaccessible to ring 3 code (Lee et al., 2018).
A third pattern is sandboxed library execution with typed interfaces. RLBox isolates browser libraries such as libjpeg, libpng, libtheora, libvpx, libvorbis, zlib, and production libGraphite using lightweight backends based on SFI, process isolation, or WebAssembly, while expressing trust boundaries in the C++ type system via tainted<T>, sandbox_invoke, sandbox_callback, and sandbox_malloc (Narayan et al., 2020). CUP instruments all user-space code, including libc and other system libraries, using a hybrid metadata scheme that preserves the ABI while attaching object-aware metadata to pointers, thereby removing user-level libraries from the trusted computing base (Burow et al., 2017).
A fourth pattern is trusted execution or protected service delegation. The SGX-based cryptographic library keeps keys, IVs, and cryptographic state inside an enclave and exposes only high-level operations via ECALLs; plaintext and ciphertext remain outside the enclave, while key material never leaves enclave memory (Mofrad et al., 2017). Safe sharing of fast kernel-bypass I/O pushes the pattern further: protected libraries run in the caller’s thread context but inside MPK-protected regions, with a trusted daemon handling asynchronous events and a bounded-time execution discipline preserving OS reclaimability (Beadle et al., 2 Sep 2025).
3. Protection mechanisms and formal models
At the protocol level, Android proactive libraries model enforcement as edit automata. The enforcer observes method-entry, method-exit, and lifecycle events, and can allow them unchanged, suppress them, or inject new calls; the camera policy “An activity that is paused while having the control of the camera must first release the camera” is encoded as an automaton that inserts Camera.release() on after#Activity.onPause and re-acquires the camera on resume (Riganelli et al., 2019). SCLib uses a different but related model: in-app mandatory policies operate at component entry points and predicate decisions on caller identity, exported status, permissions, actions, and SQL-like parameters for providers (Wu et al., 2018).
At the permission-model level, PermRust adapts the Access Matrix Model to functions and I/O wrapper functions, and defines “permission respecting” and “privilege escalation free” programs over an Abstract Function Tree. Its non-escalation condition is stated as
and the enforcement mechanism is a zero-cost token discipline in Rust’s type system: I/O-relevant functions require unforgeable token arguments such as &ReadPerm, and only code with access to token constructors can create those capabilities (Gehring et al., 13 Jun 2025). Mir applies a parallel idea to JavaScript with object-path permissions of the form f : [r w x i], where R, W, X, and I regulate reads, writes, execution, and imports at module boundaries (Vasilakis et al., 2020).
At the memory-isolation level, AgileOS separates a single CUDA context into protected MMIO, protected module memory, and a user allocation region; host-side checks accept a device pointer only when it lies inside a live user allocation, formalized as
Because kernels can otherwise compute arbitrary addresses inside a context, AgileOS also injects PTX-level guards before ld.global and st.global so that accesses into protected and I/O ranges trap or follow a chosen violation policy (Yang et al., 4 Jun 2026). CUP addresses the same general problem for C/C++ user space with capability IDs and base/end metadata, providing complete spatial and probabilistic temporal memory safety for heap, stack, and globals while preserving the ABI (Burow et al., 2017).
At the compartment and call-gate level, Polytope inserts privilege switches around direct and indirect calls, uses partition-aware allocators, and tracks address-taken functions so that indirect call targets receive only their declared privileges (Agadakos et al., 2022). RLBox enforces static information-flow constraints by requiring that all library-originated data remain tainted until explicitly validated or copied-and-verified, and prevents pointer leakage, unchecked callbacks, and double-fetch patterns across the sandbox boundary (Narayan et al., 2020). LOTRx86 uses call gates, lret, and controlled entry dispatch to ensure that ring-3 code can invoke only enumerated PrivUser routines and cannot directly access PrivUser pages or arbitrarily redirect control into protected code (Lee et al., 2018).
4. Representative implementations across domains
The topic is best understood as a family of domain-specific designs rather than a single mechanism.
| System | Protected asset | Main mechanism |
|---|---|---|
| AgileOS | CUDA services, MMIO mappings, library state | CUDA virtualization, pointer validation, PTX memory guard |
| RLBox | Third-party browser libraries | Tainted types plus SFI, process, or Wasm sandboxing |
| PermRust | Library-level I/O authority | Token-based permissions in Rust’s type system |
| LOTRx86 | In-process secrets and routines | PrivUser, call gates, segmentation, privcall |
| HodorDDS | Shared kernel-bypass NIC service | MPK-protected library, bounded-time calls, permanent buffers |
On mobile systems, protected libraries often enforce protocol or policy rather than raw memory isolation. Proactive libraries augment Android APIs such as android.hardware.Camera or android.media.MediaPlayer with enforcers that can repair resource protocols at runtime (Riganelli et al., 2019). SCLib, also in Android, places security logic inside a deployable app library and protects exported components by checking caller identity and component attributes before allowing, denying, or alerting on ICC requests (Wu et al., 2018). The intra-library-collusion literature adds a broader privacy interpretation: libraries embedded across many apps can accumulate authority beyond any single host, motivating per-library identities and permissions (Taylor et al., 2017).
In systems and browser software, the focus shifts to memory safety and exploit containment. RLBox treats image, audio, video, font, and compression libraries as untrusted and isolates them with low-cost sandboxes while retaining typed, high-throughput interfaces (Narayan et al., 2020). CUP instruments not only applications but also libc and other libraries, thereby removing those libraries from the trusted computing base and detecting spatial and temporal violations inside them (Burow et al., 2017). Polytope instead starts from source-level privilege-separation policy and uses MPK to turn partitions into protected components inside one C++ process (Agadakos et al., 2022).
In accelerators and user-level I/O, the same architectural idea appears in service form. AgileOS protects GPU-resident services, MMIO apertures, and trusted library state inside a single CUDA context, hiding raw addresses and handles from untrusted kernels (Yang et al., 4 Jun 2026). HodorDDS demonstrates a protected user-level DDS implementation that safely shares a kernel-bypass NIC among mutually untrusting applications by combining MPK isolation, bounded library calls, futex-based suspension outside the library, a trusted daemon, and permanent buffers (Beadle et al., 2 Sep 2025).
In cryptography, the emphasis is on nondisclosure of keys and resistance to low-level exploitation. The SGX-based cryptographic library keeps AES keys, HMAC keys, IVs, and cryptographic state entirely inside the enclave and exposes only streaming operations on 4 KB buffers (Mofrad et al., 2017). LOTRx86 offers a different, non-enclave route for the same goal by placing private-key operations in PrivUser and using privcall to sign or load keys without ever exposing them to ring-3 memory (Lee et al., 2018). The secure-compilation perspective complements both: protected cryptographic libraries should combine constant-time coding with compiler flags and secure-compilation techniques against side-channel and code-reuse attacks (Tsoupidi et al., 2024).
5. Deployment, performance, and quantitative outcomes
The overhead profile of protected libraries varies sharply with mechanism and domain. Android proactive libraries were evaluated on 15 apps with 27 reported misuses across 12 resources; the average overhead was about 2%, the worst case was about 6%, and with 60 enforcers the overhead rose to about 12% (Riganelli et al., 2019). SCLib, evaluated on ten high-profile open-source apps, protected 35 risky components with less than 0.3% stub code and nearly no slowdown to normal intra-app communications; the worst-case performance overhead to stop attacks was about 5% (Wu et al., 2018). Mir reported 99.33% automatically inferred required permissions on 50 popular JavaScript libraries and about 1.93% runtime overhead, showing that a per-library RWXI boundary can be both fine-grained and automatable (Vasilakis et al., 2020).
Browser and C/C++ isolation systems show higher but still practical costs. RLBox imposed about 3% page-load overhead with SFI and about 13% with process isolation across representative websites, with transient memory overheads of about 25% and 18% respectively, while maintaining no visible degradation in media playback and enabling production deployment of Wasm-sandboxed libGraphite (Narayan et al., 2020). Polytope reported a worst-case 10.1% latency overhead in its Nginx plus OpenSSL case study and an average 11.1% overhead on SQLite speedtest1, with higher costs when workloads make many cross-partition calls (Agadakos et al., 2022). LOTRx86 measured privcall overhead in the syscall range and, in its Nginx proof of concept, observed about 35% overhead for 5 KB responses on Intel, about 40% on AMD, and 6.25% overhead for 500 KB responses on Intel (Lee et al., 2018).
Memory-safety instrumentation tends to be more expensive but also broader in coverage. CUP incurred about 158% average overhead on SPEC CPU2006 with musl, but reduced false negatives by 10x to 0.1% on the Juliet suite and produced no false positives while instrumenting all user-space code, including system libraries (Burow et al., 2017). HodorDDS, by contrast, uses protected libraries to accelerate rather than slow the I/O path: relative to commercial FastDDS, it achieved approximately 50% lower latency and up to 7x throughput, with lower CPU utilization, and the authors state that this is the first successful sharing of a kernel-bypass NIC among mutually untrusting applications (Beadle et al., 2 Sep 2025). AgileOS is explicitly an initial design and prototype; it does not yet report performance benchmarks, and detailed evaluation is deferred to future work (Yang et al., 4 Jun 2026).
The deployment burden also differs. RLBox required only modest caller-side refactoring and small validators but depends on a supported sandbox backend (Narayan et al., 2020). PermRust is described as a zero-cost abstraction but requires tokenized wrappers and Cargo permission features (Gehring et al., 13 Jun 2025). SGX and LOTRx86 both demand trusted low-level substrates—enclaves in one case, x86 rings and a kernel module in the other (Mofrad et al., 2017, Lee et al., 2018). Safe sharing of kernel-bypass I/O additionally requires OS support for non-preemption during protected-library calls, permanent buffers, and fast process identity (Beadle et al., 2 Sep 2025).
6. Limitations, controversies, and open directions
The surveyed systems are united as much by their trust assumptions as by their protections. AgileOS explicitly trusts the host OS and NVIDIA driver, does not defend against a compromised OS kernel or GPU driver, and does not yet provide full performance isolation or complete CUDA and library coverage (Yang et al., 4 Jun 2026). The SGX cryptographic library excludes physical side channels, passive address-translation attacks, and current key sealing from its security goals, and its prototype is session-only (Mofrad et al., 2017). LOTRx86 assumes a trusted kernel and does not attempt to protect against kernel compromise, side channels, or bugs inside PrivUser code (Lee et al., 2018). RLBox excludes side channels and relies on correct validators at the boundary; a flawed validator can still reintroduce unsafe trust (Narayan et al., 2020).
Coverage remains uneven across languages and runtimes. PermRust assumes safe Rust, a trusted compiler and macro system, and token-protected standard-library interfaces; unsafe code and Cargo feature unification remain structural weaknesses (Gehring et al., 13 Jun 2025). Mir does not defend against malicious libraries, native addons, denial of service, or side channels, and its privilege-reduction metric is conservative because it does not account for transitive permissions (Vasilakis et al., 2020). CUP requires instrumentation of all relevant code, including difficult assembly-heavy libraries, and still imposes substantial overhead (Burow et al., 2017). Polytope inherits MPK’s finite key space and currently does not provide per-partition syscall restriction (Agadakos et al., 2022). Proactive Android enforcement based on Xposed depends on rooted-device-style hooking and only handles API-visible protocol misuse (Riganelli et al., 2019).
A persistent conceptual controversy is whether the protected unit should be the library instance, the library provider, the application, or the service. The intra-library-collusion results argue that app-centric permission systems are fundamentally misaligned when the same third-party code appears across many apps and aggregates permissions server-side (Taylor et al., 2017). The kernel-bypass literature reaches a related conclusion from the systems side: a protected library should be treated as an OS service with bounded-time execution, reclaimability, permanent buffers, and explicit client identity, rather than as ordinary linked code (Beadle et al., 2 Sep 2025). These results suggest a broader direction in which libraries become first-class security principals with their own permissions, call gates, trusted state, and resource-management contracts.
Future work follows directly from those constraints. AgileOS identifies incomplete CUDA and cuFFT coverage, future PyTorch integration, and PTX-guard optimization as open problems (Yang et al., 4 Jun 2026). The SGX design points toward sealing, remote attestation, and persistent key management (Mofrad et al., 2017). PermRust proposes finer-grained permission attachment and Cargo support beyond the current feature mechanism (Gehring et al., 13 Jun 2025). Secure-compilation work on cryptographic libraries points to stronger integration of constant-time preservation, diversification, and code-reuse hardening in mainstream build pipelines (Tsoupidi et al., 2024). Taken together, these systems indicate that protected user-level libraries are evolving from isolated mechanisms into a general architecture for least-privilege software composition above the kernel yet below the application.