- The paper demonstrates that stdlib-only Python modules, generated with constrained LLM assistance, can meet correctness and performance criteria compared to third-party libraries.
- It employs rigorous empirical benchmarks revealing significant architectural wins, such as a YAML module achieving 6–7x speedup and an HTTP client reaching 18–32x higher throughput.
- It highlights limitations in compute-intensive tasks where pure-Python implementations lag, suggesting subprocess delegation to mitigate performance bottlenecks.
Empirical Evaluation of Stdlib-Only, LLM-Assisted Python Module Reimplementation
Motivation and Research Questions
The study interrogates the necessity and trade-offs of third-party Python libraries versus standard library (stdlib)-only implementations in utility and infrastructure domains. Third-party packages are implicated in substantial dependency bloat, supply-chain risk, and deployment friction, with empirical findings that a single dependency can induce a median of 8-10 additional transitive packages, most of which are unused at runtime. Moreover, code-generating LLMs often hallucinate non-existent package names, introducing latent vulnerabilities. This context motivates the construction of zerodep: a curated repository of single-file, stdlib-only reimplementations of popular Python libraries, created via constrained LLM assistance and validated against reference implementations. The key research questions are: (1) In which scenarios does stdlib suffice for correctness and performance? (2) How effective is LLM-assisted constrained generation under strict and symbolic limitations?
Architecture and Module Taxonomy
Zerodep comprises 44 modules spanning 12 categories and three complexity tiers: simple (<200 LOC), medium (200–800 LOC), and subsystem (>800 LOC). Each module is strictly stdlib-only, single-file, and provides drop-in API compatibility with popular third-party libraries. Specific categories include serialization (YAML, JSON, protobuf), networking (HTTP clients/servers), cryptography, validation, text processing, and agent protocol stacks. Metadata (SemVer versioning, inter-module dependencies) is encoded in PEP 723-style frontmatter and managed by the zerodep CLI, which handles module discovery, dependency resolution, and embedding. Module development adheres to an iterative human-LLM collaboration workflow, where correctness is checked against reference libraries at each stage.
Correctness Evaluation
Each zerodep module is paired with automated correctness tests covering typical usage, edge cases, and format-specific corner cases, executed under identical environments as their reference counterparts. All modules pass their respective test suites, with intentional API simplifications explicitly documented. Behavioral drift emerges as the primary complexity—e.g., nuanced YAML null representation matching, JSON comment handling, protobuf field ordering—resolving to align with reference library documented behavior. These empirical oracle tests are essential for catching subtle semantic mismatches often overlooked by LLMs during generation.
Performance benchmarking is conducted using a standardized hardware and software environment. Benchmarks discriminate three regimes: parity (within 2x reference performance), architectural win (zerodep >2x faster), and C-extension cliff (zerodep <0.5x reference). Of 44 modules, 19 are faster, 11 achieve parity, and 6 are slower than their reference library.
- Architectural wins: Notably, zerodep’s yaml module is 6–7x faster than PyYAML's default pure-Python loader due to reduced AST complexity. jsonc outperforms commentjson by 75–115x, mainly due to a streamlined regex-based comment stripper instead of eval-based parsing. HTTP client achieves 18–32x higher request throughput than httpx by eliminating dispatch and middleware abstraction layers.
- C-extension cliff: Modules relying on computational hot paths (AES cryptography, protobuf serialization, PNG processing) fall behind due to Python interpreter limitations, with pure-Python AES trailing PyCryptodome by 300–17,000x. Subprocess delegation to system libraries (e.g., OpenSSL) can mitigate this, as seen with zerodep’s AES achieving 1.5–8x faster performance than PyCryptodome.
- Parity: Modules corresponding to I/O-bound or string manipulation tasks (dotenv, frontmatter, QR code generation) display practical performance equivalence with their third-party counterparts.
LLM-Assisted Development Workflow
LLM assistance accelerates module development, but effectiveness is tier-dependent. Simple modules are produced in one to three iterations; medium modules require iterative refinement focused on edge-case handling; subsystem modules demand extensive architectural guidance prior to LLM contribution. Across tiers, LLMs consistently leverage core stdlib primitives (json, re, struct, ssl, urllib, asyncio, socket, hashlib, base64). Automated empirical oracle testing is critical, as behavioral divergence is the major source of iteration.
Practical Implications and Guidance
Practitioners are advised that stdlib-only modules can serve as ready substitutes for utility and infrastructure tasks in dependency-sensitive deployments, with roughly two-thirds achieving parity or outperforming their reference library. Subprocess offloading effectively recovers performance in C-extension bottleneck domains without incurring additional dependencies, but introduces new trade-offs related to latency and platform-specific semantics. Architectural overhead in third-party libraries can impose disproportionate penalties, and streamlined stdlib reimplementations frequently yield superior performance. Comprehensive API requirements and hardware acceleration remain the purview of third-party packages.
Limitations and Future Directions
Zerodep intentionally omits compute-intensive frameworks (NumPy, PyTorch) whose performance is intrinsically tied to native code. Some modules lack advanced features present in their reference libraries, so feature completeness must be evaluated in context. Subprocess delegation mitigates throughput gaps but does not generalize to domains such as binary serialization. Benchmarks are conducted in single-threaded, controlled environments; results may diverge in distributed or resource-constrained settings. Memory consumption is not systematically measured and could affect applicability in low-resource deployments.
Future avenues include extending LLM-assisted development to multi-file, cross-module projects, systematic measurement of memory profiles, automated API coverage assessment, and evaluation of zerodep modules under deployment constraints such as FIPS, air-gapped environments, or containerized runtimes.
Conclusion
Empirical evidence demonstrates that stdlib-only, LLM-assisted Python module implementations, validated against reference oracles, suffice for the majority of utility and infrastructure tasks. Performance gaps are driven primarily by architectural overhead or reliance on compiled C/Rust extensions; streamlined stdlib designs frequently surpass their more generalized third-party counterparts. LLMs accelerate development under strict constraints, contingent upon complexity tier and the presence of human-guided architectural input. Dependency-free, correctness-guaranteed modules offer substantial practical benefits in environments where supply-chain risk and deployment friction are paramount. Zerodep advances systematic, benchmarked, and reproducible stdlib-only module engineering, offering actionable guidance at both development and operational scales.
For further details and reproducible artifacts, see https://github.com/Oaklight/zerodep and dashboard at https://oaklight.github.io/zerodep/dev/bench/.
Reference: "Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries" (2605.21405)