- The paper demonstrates that aggressive quantization (e.g., Kimi-K2.5 Q3) enables fully compliant React Native app generation on a single GH200 node.
- The evaluation reveals that synthetic benchmarks often misalign with artifact-level efficacy, highlighting integration and prompt-interpretation errors.
- The study shows efficient architectures and hardware-aware quantization strategies offer superior cost-capability tradeoffs for self-hosted coding deployments.
Summary and Evaluation of "React-ing to Grace Hopper 200: Five Open-Weights Coding Models, One React Native App, One GH200, One Weekend" (2604.17187)
Experimental Design and Model Evaluation
The study systematically evaluates five open-weight coding LLMs—Kimi-K2.5 (Q3 and Q4 quantizations), GLM-5.1, Qwen3-Coder-480B, and DeepSeek-V3.2—on a pragmatic application-generation task. All models are deployed via llama.cpp on a single NVIDIA GH200 576 GB node, using Unsloth's Dynamic 2.0 GGUF quantizations. The task involves generating, from a single prompt, a multi-file React Native app with authentication, per-user per-day counting, and verified web compatibility.
Rather than relying on synthetic benchmarks or patch-level evaluations, the methodology centers real artifact generation and task-completion, with validation based on operational correctness: out-of-the-box runnability, feature-level compliance (credential validation, per-user data isolation, per-day bucketing, history, logout), and adaptation to the web platform (avoiding native-only APIs like Alert.alert).
Key Findings and Numerical Outcomes
The results exhibit a pronounced decoupling between SWE-Bench scores and task-level efficacy. Kimi-K2.5 at aggressive 3-bit quantization (UD-Q3_K_XL, 480 GB) delivered the only fully spec-compliant app, outperforming GLM-5.1 (SOTA on SWE-Bench Pro) and DeepSeek-V3.2 (highest SWE-Bench Verified among scale-school models). GLM-5.1, despite its benchmark dominance, over-engineered the authentication layer (Firebase, mandatory config), rendering the app unusable without manual intervention. DeepSeek-V3.2 suffered from a tooling-induced integration failure, corrupting bundling with misplaced App.js paths due to reasoning token leakage. Qwen3-Coder-480B misread prompt semantics, omitting per-day bucketing.
Generation speeds ranged from 7.9 to 20 tok/s, and output sizes hovered around 4–5k tokens per model. Notably, Kimi-K2.5 Q3 sustained 17 tok/s from a 1T/32B active model on a single 96 GB GPU, demonstrating the viability of aggressive quantization for high-throughput artifact generation.
Deployment Observations and Integration Issues
Three deployment-level phenomena were documented:
- Sampling hangs with default temperature settings: Coding tools like aider, defaulting to temperature=0, resulted in inference stalls for reasoning-model architectures. Properly setting temperature=1.0 and top_p=0.95 was non-negotiable for practical deployment.
- Reasoning-token leakage corrupts integration tools: DeepSeek-V3.2 responses contained reasoning traces (e.g., stray
> tags) that confounded aider’s file-path parsing, leading to structural bundling failures. This intermediary output-format sensitivity is underreported.
- Universal training-data gap in web adaptation: All models relied on Alert.alert for user interaction, which is non-functional on react-native-web. None adapted code for conditional platform dispatch, revealing a compositional gap in training corpora regarding React Native platform divergence.
Hardware-Tier Landscape and Efficiency Analysis
The analysis maps two distinct architectural schools in 2026 open-weight LLMs:
Efficiency-school models (e.g., MiniMax M2.5/M2.7, MiMo-V2-Flash) achieve roughly equivalent SWE-Bench Verified scores at approximately 1/7 the hardware expenditure of scale-school models. The practical implication is that for most self-hosted coding use-cases, architecture selection and quantization strategy dominate parameter-count scaling, and hardware commitments should be guided by artifact-level evaluation rather than leaderboard benchmarks.
Limitations
The study is constrained by:
- Single-prompt, single-seed evaluation for each model, lacking repeated runs or prompt variation
- Task specificity: Only React Native web-compatible generation was tested, limiting generalizability
- Tooling specificity: aider’s orchestration layer influenced the surface manifestation of some failures
- Quantization mix: Only Unsloth Dynamic 2.0 quantizations were used; FP8 and some vLLM-based stacks were omitted due to time and stability constraints
- Hardware limitation: All experiments were conducted during a single weekend of GH200 access
Implications and Future Directions
These findings imply that benchmark rankings (SWE-Bench Verified/Pro) should not be relied upon exclusively for model selection in practice-oriented workloads. Integration and specification-compliance failures predominate over code-generation errors, and model behavior at task boundaries (artifact generation, platform adaptation) often escapes synthetic evaluation. In future AI development, more attention should be paid to product-like validation tasks and to the integration layer between LLMs and downstream engineering tools. Further, corpus curation should address training-data gaps affecting compositional adaptation to platforms like react-native-web.
Hardware-wise, efficient architectures offer superior cost-capability tradeoffs for self-hosting scenarios, and aggressive quantization (dynamic, importance-aware) has matured to the point where high-throughput, large-model inference is feasible on modest systems.
Conclusion
The assessment demonstrates that open-weight coding LLMs have achieved benchmark parity with proprietary models, but leaderboard placement does not correlate with real artifact fitness. For self-hosted coding deployments, direct task-level evaluation is required before hardware and architecture choices are made. Model failures were primarily due to integration and prompt-interpretation errors rather than technical code synthesis. Optimization of sampling parameters and output-format robustness should be prioritized in tool development. The future landscape favors efficiency-school architectures for typical institutional workloads, with aggressive quantization and hardware-aware deployment strategies.