PQC-LEO: An Evaluation Framework for Post-Quantum Cryptographic Algorithms

Abstract: Advances in quantum computing threaten digital communication security by undermining the foundations of current public-key cryptography through Shor's quantum algorithm. This has driven the development of Post-Quantum Cryptography (PQC), a new set of algorithms resistant to quantum attacks. While NIST has standardised several PQC schemes, challenges remain in their adoption. This paper introduces the PQC-LEO framework, a benchmarking suite designed to automate the evaluation of PQC computational and networking performance across x86 and ARM architectures. A proof-of-concept evaluation was conducted to demonstrate the framework's capabilities and highlight its application in supporting ongoing research on the adoption of PQC algorithms. The results show that there is a greater performance reduction in implementing PQC methods with higher security on ARM architectures than on the x86 architecture.

• The paper introduces an automated framework that measures CPU cycles, execution time, memory use, and TLS 1.3 performance for classical, PQC, and hybrid algorithms across x86 and ARM devices.
• The proof-of-concept finds that ML-KEM and ML-DSA often match or outperform classical options, while higher-security configurations impose a disproportionately larger performance penalty on ARM systems, including a 34% handshake reduction.
• The framework enables reproducible testing across physically networked machines and supports emerging algorithms, but future studies must examine energy use, network conditions, broader hardware, and performance variability.

The transition to post-quantum cryptography (PQC) requires empirical performance data across heterogeneous hardware, yet existing benchmarking efforts suffer from limited automation, reliance on deprecated libraries, and a lack of testing over physically networked devices. "PQC-LEO: An Evaluation Framework for Post-Quantum Cryptographic Algorithms" (2603.06149) addresses these gaps by presenting an automated benchmarking suite that evaluates PQC computational and networking performance on x86 and ARM architectures, integrating both NIST-standardised schemes and candidates from the ongoing Additional Signatures competition. This essay summarises the framework's design, its proof-of-concept evaluation, and the principal findings.

Motivation and related work

Shor's algorithm breaks the integer factorisation and discrete logarithm assumptions underlying RSA and ECC in polynomial time, while Grover's algorithm imposes only a quadratic speedup against symmetric primitives—a threat manageable through parameter sizing. NIST's standardisation process has produced FIPS 203 (ML-KEM), FIPS 204 (ML-DSA), and FIPS 205 (SLH-DSA), with Falcon and HQC undergoing finalisation; the Additional Signatures competition advanced 14 of 40 initial candidates to round two. Adoption challenges remain acute for resource-constrained IoT environments, where larger keys, ciphertexts, and signatures strain bandwidth, energy budgets, and protocol limits. The paper also notes a structural risk: the standardised portfolio is dominated by lattice-based constructions, so a cryptanalytic break in that family would be broadly damaging—an argument for continued evaluation of non-lattice alternatives.

The authors position PQC-LEO against five prior frameworks. Rios et al.'s TLS benchmarking lacks non-localhost and Hybrid-PQC support; Commey et al. omit memory metrics and use outdated OQS dependencies; Abbasi et al. publish no implementation and perform no networked-device testing; Hanna et al. rely on outdated libraries; Fitzgibbon et al. provide incomplete handshake timing and deprecated dependencies. PQC-LEO's distinguishing claims are full automation of setup, execution, and parsing; support for PQC, Hybrid-PQC, and classical algorithms within TLS 1.3; and TLS evaluation across physically separate machines rather than localhost only.

Framework design

PQC-LEO is a modular bash-and-Python suite targeting Debian-based x86 and ARM systems. Its computational benchmarking stage invokes Liboqs speed_kem/speed_sig binaries for CPU metrics (cycles, time) and modified test_kem_mem/test_sig_mem binaries under the Valgrind Massif profiler for heap, external heap, and stack usage. The TLS stage automates X.509 certificate generation, then coordinates handshake tests between client and server using OpenSSL s_server and s_time, with session-ID reuse scenarios, plus openssl speed throughput tests. A notable engineering detail is a netcat-based control-signalling channel that synchronises test transitions between physically separate machines, enabling unattended real-world networked testing. Results are parsed into CSV automatically.

The framework builds on OpenSSL 3.5.0's native PQC support alongside Liboqs/OQS-Provider, allowing evaluation of non-standardised candidates (CROSS, MAYO, SNOVA, UOV variants). The authors are candid about upstream limitations: HQC is disabled by default in OQS due to specification non-conformance (re-enableable via an advanced flag); Falcon memory profiling fails under Valgrind on ARM; several signature variants cannot participate in TLS 1.3 handshakes due to protocol non-conformance; ML-DSA works for handshakes but not speed tests; and SLH-DSA is available for certificates but not functional in TLS testing, though SPHINCS+ remains testable. The setup script also dynamically patches hardcoded algorithm-count limits in the OpenSSL speed tool when all OQS-Provider signature schemes are enabled.

Proof-of-concept evaluation

The evaluation used matched hardware pairs per architecture—HP EliteDesk 800 G3 Mini (Intel i5-6500T, Debian 12.11) and Raspberry Pi 4 Model B (Cortex-A72)—connected via Ethernet through a Netgear GS308T switch. Each test category ran three iterations with averaged results; handshake and speed windows were 30 seconds. On ARM, user-level PMU access was enabled via PQAX to obtain cycle counts.

Computational performance: ML-KEM dominates KEM rankings on both architectures—for example, ML-KEM-512 achieves roughly 10–12 µs per operation on x86 versus 41–48 µs on ARM, while FrodoKEM-640-AES keygen takes 409 µs on x86 but nearly 17 ms on ARM. Among signatures, ML-DSA-44 signs in 83 µs on x86 and 923 µs on ARM, and several Additional Signatures candidates (CROSS, MAYO, SNOVA) rank highly at lower security levels. Memory footprints are modest throughout: ML-KEM-512 peak allocations stay below ~13 KB, and SPHINCS+ variants show near-zero heap usage during signing on ARM.

TLS handshakes: On x86, pure-PQC combinations such as MLDSA87/MLKEM512 reach 8,041 connections, exceeding classical prime256v1 baselines (~5,456–5,683). Hybrid combinations halve this (best: p256_OV_Ip_pkc/X25519MLKEM768 at 5,464). On ARM, MLDSA44/MLKEM512 achieves 5,530 handshakes, comparable to classical ECC (~4,937–5,047), but performance degrades sharply with security level: MLDSA87/MLKEM1024 drops to 3,672, a ~34% reduction from the level-1 configuration, whereas the corresponding x86 decrease is comparatively small. This is the paper's central empirical claim: higher-security PQC configurations penalise ARM disproportionately relative to x86, which bears directly on IoT deployment planning where security-critical applications may require level-3 or level-5 parameters.

TLS speed: ML-KEM sustains tens of thousands of operations per second on x86 (e.g., 28,365 keygens/s at level 1) and remains strong on ARM (12,529/s), while code-based and hash-based alternatives fall orders of magnitude behind (FrodoKEM-1344-SHAKE: 39–44 ops/s on ARM). Hybrid KEMs show pronounced overhead from the classical component—p256_mlkem512 manages only 92 keygens/s on x86 because ECDH key generation dominates.

Limitations and open questions

Beyond the upstream library constraints already noted, several caveats qualify the results. The proof-of-concept filters reporting to top-ten performers per category, so tail behaviour of expensive schemes is not characterised in the paper itself. Results derive from single device models per architecture with three-run averaging, without variance reporting, limiting statistical generalisation. The observed ARM-specific degradation at higher security levels is reported as warranting further investigation rather than explained mechanistically—it remains open whether the cause lies in cache behaviour, NEON vectorisation coverage, or memory bandwidth. Energy consumption, handshake byte sizes, and simulated network congestion are explicitly deferred to future work, as is broader OS support and embedded-platform integration via pqm4.

Conclusion

PQC-LEO contributes a reproducible, fully automated benchmarking pipeline spanning CPU, memory, and TLS 1.3 performance for PQC, Hybrid-PQC, and classical algorithms across x86 and ARM, with genuine physical-network testing that prior frameworks lacked. Its proof-of-concept confirms that lattice-based standards deliver consistent cross-architecture performance and can match or exceed classical TLS throughput, while quantifying a disproportionate ARM penalty at elevated security levels and highlighting competitive lower-security multivariate/code-based candidates. The framework's public availability positions it as practical infrastructure for the deployment studies that PQC adoption in constrained environments still requires.