rush: Scalable Asynchronous Distributed Computing via Shared State in R

Abstract: Many algorithms in statistics and machine learning can be parallelized in an asynchronous manner where workers need to communicate through shared state rather than execute independent tasks dispatched by a central controller. Especially in modern hyperparameter optimization and parallel black-box optimization with expensive objectives, this decentralized approach has become widespread, and several Python frameworks adopt it (e.g., Optuna, DeepHyper, and Hyperopt). However, all popular R packages for parallel computing follow a centralized controller-worker architecture that does not support this pattern. We present rush, an R package that provides a shared-state coordination layer for asynchronously parallelized iterative algorithms. rush uses a Redis database as a shared key-value store: workers read and write task data through the database and independently execute their own loops. The package provides a high-level API for managing tasks and their lifecycle, featuring sub-millisecond per-task overhead, robust error handling with automatic detection of lost workers, and efficient caching. rush optionally integrates with the mlr3 ecosystem, powering asynchronous optimization in the bbotk and mlr3tuning packages. We demonstrate the practical utility of rush by implementing asynchronous decentralized Bayesian optimization (ADBO) and benchmarking it on hyperparameter optimization of LightGBM across four datasets using 448 workers.

• The paper introduces rush, a decentralized, asynchronous computing framework for R that leverages Redis to overcome centralized bottlenecks in distributed tasks.
• It details the implementation of asynchronous decentralized Bayesian optimization, achieving over 300× speedup and nearly optimal CPU utilization compared to traditional approaches.
• Practical benchmarks demonstrate rush’s robust error handling and efficient resource management, enabling large-scale hyperparameter and black-box optimization workflows.

rush: Scalable Asynchronous Distributed Computing via Shared State in R (2606.21430)

Motivation and Context

The paper addresses a significant deficit in the R parallel computing ecosystem: the lack of decentralized, asynchronous distributed frameworks that enable worker-driven, shared-state coordination, particularly for iterative algorithms such as hyperparameter optimization (HPO) and black-box optimization. Existing R packages such as parallel, future, mirai, batchtools, and crew uniformly employ a centralized controller-worker architecture, limiting expressiveness and scalability. In contrast, Python frameworks (e.g., Optuna, DeepHyper, Hyperopt) facilitate decentralized execution via shared databases, enabling worker autonomy. The paper introduces rush, an R package leveraging Redis as a robust, low-latency key-value store to support shared-state coordination for scalable, asynchronous distributed algorithms.

Architecture and API Design

rush implements a decentralized topology where multiple worker processes communicate exclusively through a shared Redis database. Workers execute independent loops, reading/writing task state via the database without dependence on a central controller. The API provides primitives for creating, monitoring, and terminating networks (manager class), launching workers locally or remotely (via mirai daemons, processx, or standalone scripts), and granular operations for task lifecycle and shared state access.

Tasks are encapsulated as key-indexed Redis hashes containing input, output, extra metadata, and computational state (queued, running, finished, failed). rush supports both queue-based (centralized task creation; workers consume and mark as running) and fully autonomous loop patterns. The API integrates robust error handling: workers detect task-level failures and signal state transitions, while lost workers are detected automatically or via heartbeats with Redis key expiration.

Key architectural features include sub-millisecond per-task overhead, efficient local caching for result fetch, seamless logging integration (lgr), and transparent deployment across heterogeneous OS/hardware (local, SSH, HPC clusters via SLURM/SGE).

Asynchronous Decentralized Optimization

The paper demonstrates rush's capabilities through the implementation and benchmarking of asynchronous decentralized Bayesian optimization (ADBO). Unlike synchronous batch or asynchronous centralized parallelism, ADBO enables each worker to independently run a surrogate fitting and acquisition optimization loop, reading the global archive from the shared database, proposing new configurations, imputing current running tasks, and updating results asynchronously. Diversity among proposals is induced by local stochastic acquisition functions (e.g., variable $\lambda$ for confidence bounds).

This paradigm is enabled by rush's shared-state model, allowing rapid, conflict-free coordination at scale, eliminating central bottlenecks and maximizing resource utilization—especially crucial for workloads with heterogeneous evaluation durations.

Performance Benchmarks

Comprehensive microbenchmarking evaluates rush's core operations. Task push and finish operations demonstrate sub-millisecond latency for typical HPO workloads (up to tens of scalar hyperparameters), with cache-optimized fetch scaling linearly in new results rather than archive size. For large archives ($>$10^4 tasks), caching reduces fetch times by two orders of magnitude over uncached operations (e.g., 488 ms vs. 5 ms for 10,000 tasks with 10 parameters).

The paper benchmarks CL (synchronous batch via constant liar), ACBO (asynchronous centralized), and ADBO (asynchronous decentralized via rush) across 448 workers on four OpenML datasets (credit-g, KDDCup09_appetency, adult, airlines) using LightGBM HPO. ADBO achieves 94–100% effective CPU utilization versus single-digit utilization for CL/ACBO, with short-running tasks (credit-g) seeing a $>$300$\times$ improvement in completed evaluations (15,451 vs. $\sim$400). Centralized approaches suffer from controller and synchronization bottlenecks, especially as surrogate fitting becomes computationally expensive.

ADBO maximizes throughput, scales without bottlenecks, and adheres more precisely to wall-clock budgets versus batch approaches, which overshoot due to delayed batch terminations.

Practical and Theoretical Implications

rush fundamentally expands the class of algorithms and computational patterns expressible in R, enabling fully decentralized iterative workflows previously exclusive to Python. The infrastructure supports large-scale HPO/black-box optimization with near-ideal CPU utilization—critical as both dataset sizes and model complexity increase. Decentralized optimization is essential for scaling to cluster/cloud environments, heterogeneous evaluation times, and forms the backbone for emerging distributed machine learning algorithms.

The results demonstrate that infrastructure bottlenecks, rather than algorithmic inefficiency, dominate scalability in centralized architectures; eliminating them yields multiplicative gains in resource utilization. As computational budgets grow, the implications for high-throughput optimization, automated machine learning, and distributed statistical computation are substantial. Potential future work includes adaptive caching, support for complex task dependency graphs, and broader interoperability with cross-language frameworks, further reinforcing rush as a foundational building block.

Conclusion

rush fills a critical gap in the R parallel computing landscape by providing scalable, asynchronous, decentralized computing via shared-state coordination using Redis. The architecture enables autonomous workers, robust error-handling, and superior throughput for iterative statistical and machine learning algorithms, ensuring near-maximal CPU utilization in distributed environments. The practical benchmarks validate its efficacy, especially for massive HPO workloads. The scalability and flexibility of rush will catalyze further research and development of distributed optimization methods in R, paralleling advancements in other language ecosystems.