---
title: 'FirecREST v2: Scalable HPC REST API'
url: https://www.emergentmind.com/topics/firecrest-v2
type: topic
---

# FirecREST v2: Scalable HPC REST API

FirecREST v2 is an open-source RESTful API for programmatic access to high-performance computing (HPC) resources, representing a comprehensive, ground-up redesign of its predecessor to deliver a 100× performance improvement. The new architecture emphasizes asynchronous, stateless operation, high throughput, and robust security. It incorporates lessons learned from bottlenecks identified in proxy-based APIs handling I/O-intensive HPC tasks and provides validated performance metrics and design practices for scalable, secure HPC access [2512.11634].

## 1. Layered Architecture and Component Overview

FirecREST v2 adopts a fully asynchronous, lightweight proxy model organized in four stateless, sequential layers. The request pipeline progresses as follows:

- **Client**
- **Authentication Layer** (OIDC/JWT offline validation)
- **Authorization Layer** (JWT claims or external OpenFGA checks)
- **Health Checker** (cached subsystem status)
- **Forwarding Layer** (modular clients, SSH/HTTP, connection pooling)
- **HPC Subsystems** (job scheduler, filesystem, object store)

This architectural flow is depicted below:

\[
\begin{tikzpicture}[node distance=1.5cm, every node/.style={draw,rectangle,align=center}]
  \node (C)  {Client};
  \node (A) [right=of C] {Authentication};
  \node (Z) [right=of A] {Authorization};
  \node (H) [right=of Z] {Health Checker};
  \node (F) [right=of H] {Forwarding};
  \node (S) [right=of F] {HPC Subsystems};
  \draw[->] (C) -- (A) -- (Z) -- (H) -- (F) -- (S);
\end{tikzpicture}
\]

Key innovations in v2 include an asynchronous HTTP server stack (Uvicorn + FastAPI + asyncio), offline JWT signature validation to eliminate network dependency for token verification, a pluggable authorization layer supporting claims and OpenFGA models, a Health Checker for subsystem liveness caching, and a modular Forwarding Layer with service-specific client abstractions and AsyncSSH-based user connection pools.

## 2. Performance Benchmarks and Optimization Strategies

Throughput ($T$) and latency ($L$) are formalized as

\[
T = \frac{N_{\mathrm{requests}}}{t_{\mathrm{total}}} \quad ; \quad L = t_{\mathrm{response}} - t_{\mathrm{request}}
\]

v1 exhibited critical bottlenecks due to Gunicorn’s multi-threaded model, online JWT introspection, per-request SSH connections, and real-time subsystem health checks, each creating synchronous I/O and thread pool saturation.

v2 replaces threads with asyncio-based concurrency, employs offline JWT validation, introduces SSH connection pooling (AsyncSSH), and implements asynchronous, cached health checks—altogether removing major sources of blocking and scaling limits.

Benchmarks involved (A) Postman-driven stress tests simulating core HPC operations on a Cray EX cluster and (B) AiiDA-integrated tests with up to 1000 concurrent 1 KB file downloads using Python httpx’s AsyncClient. Representative download results:

| N    | FirecREST v1        | FirecREST v2        | FirecREST v2 (SSH pool)   |
|------|---------------------|---------------------|---------------------------|
| 1    | 1.5 ± 0.1 s         | 0.8 ± 0.02 s        | 0.4 ± 0.3 s               |
| 10   | 13.7 ± 1.1 s        | 2.7 ± 0.05 s        | 0.5 ± 0.2 s               |
| 100  | 129.6 ± 1.5 s       | 19.5 ± 3.4 s        | 1.5 ± 0.7 s               |
| 1000 | >1000 s             | 176.3 ± 4.5 s       | 15.5 ± 8.9 s              |

For large $N$, SSH pooling enables up to two orders of magnitude improvement (∼100×), as real-world performance shifts from thread and I/O bound to event-driven, parallelizable workloads. Variability in results (standard deviation) is primarily cluster-resource dependent; allocating dedicated testing resources reduces confidence intervals.

## 3. Security and Horizontal Scalability Mechanisms

FirecREST v2 balances aggressive throughput goals with stringent access controls and systemic robustness:

- **Authentication** employs OIDC-compliant JWTs, validated offline using pre-fetched public keys.
- **Authorization** is dual-mode, using embedded JWT claims for direct role/cluster mapping or querying external OpenFGA stores for RBAC/ABAC enforcement.
- **Encryption** ensures all HTTP exchanges use TLS, while SSH operations utilize AsyncSSH.
- **Scalability** is driven by strict statelessness, allowing any pod instance to serve any request without session affinity, and Uvicorn-based server pods are horizontally deployable behind Kubernetes or hardware load balancers.
- **Health Checker** logic forwards requests only to subsystems marked healthy, reducing overload propagation risk.
- **Asynchronous I/O** ensures minimal CPU overhead, even at tens of thousands of parallel connections.

## 4. Design Principles and Implementation Lessons

Major technical conclusions from the v2 redesign include:

1. **Async > Threads for I/O-bound proxies:** In the context of Python, asyncio/Uvicorn outperforms Gunicorn/threading for proxy layers manipulating high-volume I/O, mainly due to the GIL and context-switch overhead of threads.
2. **Offline token validation:** Local signature checks remove per-request latency imposed by external introspection, saving hundreds of microseconds per call at scale.
3. **Resource pooling:** Maintaining per-user SSH pools bypasses hard concurrency limits imposed by backend daemons.
4. **Health-cache:** Asynchronous, proactive subsystem checks enable the proxy to make routing decisions instantaneously, flattening tail latencies and preventing wasted cycles.
5. **Modularity:** Encapsulating HPC backends (e.g., Slurm, S3) behind client abstractions assures maintainability and extension for heterogeneous environments.

For API designers targeting HPC, best practices are codified as: employ stateless async architectures, pool expensive or stateful connections, minimize remote handshakes, externalize trust and policies, and standardize on OIDC, JWT, ASGI, and REST protocols.

## 5. Validation Results and Ongoing Development

Extensive peer validation involving CUG reviewers and CSCS integration partners substantiated both the 100× speedups and operational stability under sustained load. Confirmed gains are attributed not only to event-driven I/O and pooling but also to the clean separation of concerns and trust minimization at each architectural boundary.

Current and prospective enhancements include supporting two-phase upload/download operations for large data transfers while pursuing statelessness, implementing resumable transfers and push notifications (WebSockets or server-sent events), expanding exported health metrics (including fine-grained latency and error histograms) with Prometheus endpoints, and evaluating gRPC or HTTP/2 for additional throughput increases.

## 6. Synthesis and Impact

FirecREST v2 demonstrably advances secure, scalable RESTful programmatic access to HPC, achieving up to 100× speedup over its predecessor through a strict asynchronous, modular, and stateless design [2512.11634]. Its documented testing regimen, reproducible performance metrics, and modular middle-layer abstractions serve as a reference model for high-throughput, security-conscious resource proxies in large-scale computing facilities. Future development will continue to emphasize stateless, asynchronous operation, compatibility with emerging data-transfer protocols, and comprehensive observability.

Source: https://www.emergentmind.com/topics/firecrest-v2