OpenAI o3: Ukrainian Code Gen LLM

• OpenAI o3 is a proprietary large language model that excels in Ukrainian competitive programming code generation using native problem statements.
• The UA-Code-Bench evaluation on Eolymp revealed a 50% pass@1 rate, highlighting both its strengths and performance gaps.
• Benchmarking uncovers limitations such as single language focus and data contamination risks, informing future multilingual enhancement strategies.

OpenAI o3 is a proprietary LLM developed by OpenAI that has demonstrated notable performance in competitive programming-based code generation tasks targeting low-resource languages. In the context of Ukrainian-language code generation, OpenAI o3 was among the top-performing models in the UA-Code-Bench benchmark, successfully solving approximately half of the posed problems. Its competitive status, as evidenced by standardized evaluation on the Eolymp platform, provides insight into the current capabilities and limitations of neural code generation systems in underrepresented linguistic domains.

1. Role in Ukrainian-Language LLM Code Generation

OpenAI o3 was evaluated as part of the UA-Code-Bench framework, which is a large-scale, open-source benchmark designed for Ukrainian-language code generation and competitive programming problem-solving. The benchmark comprises 500 algorithmic problems sampled uniformly from the Eolymp competitive programming platform, spanning five concrete difficulty bands: very easy, easy, medium, hard, and very hard. OpenAI o3, together with a diverse set of 12 other proprietary and open-source models, was required to generate Python solutions for each problem, using a one-shot prompt format that included the full native Ukrainian statement and a single illustrative example.

The UA-Code-Bench evaluation revealed that even top-tier LLMs, including OpenAI o3 and GPT-5, only attained pass@1 rates near 50%, i.e., only half of the problems received fully correct code solutions as judged by Eolymp's rigorous hidden test suite. This performance quantifies the persistent challenges in extending LLM code generation competence to low-resource natural languages beyond English.

2. Evaluation Methodology and Metrics

The benchmarking of OpenAI o3's code generation utilized the automated online judging infrastructure integral to the Eolymp platform. Each code submission was assessed against a diverse suite of hidden tests (typically 20–30 per problem), designed to detect hard-coded responses, random input handling, and adversarial edge cases. Submissions had to comply with strict computational constraints—0.5 to 2 seconds per test and 256 MB to 1 GB memory limits, scaled by problem difficulty.

Two primary evaluation metrics were collected:

• pass@1 (fraction of tasks solved without error on all hidden tests)
• average score (the mean percentage of hidden test cases passed, across all problems)

OpenAI o3’s aggregate scores in these regimes highlighted both its superiority relative to many peer models and the substantial gap remaining for robust low-resource language code synthesis. Additionally, the benchmark examined solution uniqueness and computational efficiency (elapsed time and memory consumption) for each generated solution; however, the main determinant of success was correctness as determined by Eolymp’s private judging.

3. Eolymp Platform Architecture and UA-Code-Bench Integration

The Eolymp platform provided the operational environment underpinning the UA-Code-Bench assessment. Eolymp offers Ukrainian-native problem statements, well-calibrated difficulty gradation, and a fully automated judge accessible via API, making it amenable to large-scale, unattended LLM evaluation workflows. All problems are structured in standardized sections—"Умова" (Statement), "Вхідні дані" (Input), "Вихідні дані" (Output)—and specify algorithmic and input constraints in native language augmented with mathematical notation (including LaTeX and ASCII-math).

UA-Code-Bench leveraged these capabilities via an automated submission toolchain: model outputs were collected, submitted to Eolymp via HTTP POST under distinct accounts, and verdicts were programmatically aggregated for downstream metric computation. All scripts for data parsing, prompt generation, code submission, and performance aggregation were released open source, supporting reproducibility and extensibility.

4. Features and Limitations Exposed by Benchmarking

The competitive evaluation of OpenAI o3 via UA-Code-Bench exposed several salient properties:

Strengths:

• Language coherence: The use of native Ukrainian statements, rather than translation artifacts, ensured that LLMs were tested on authentic linguistic and syntactic constructions.
• Algorithmic diversity: Problems ranged from basic string processing and arithmetic (very easy, easy), to advanced data structures (segment trees, dynamic programming, FFT) and combinatorial algorithms (very hard).
• Hidden test suites: The Eolymp judge prevented overfitting/heavy prompt engineering by including random and adversarial cases not observable by the LLMs.

Limitations:

• Single language and source: All tasks were Python-only and drawn exclusively from Eolymp, limiting immediate generalization to other programming languages and contest ecosystems.
• Systemic grader limitations: 14 out of 500 original problems were excluded due to judge-side failures (concentrated in the most difficult band).
• Data contamination concerns: Some Eolymp problems may have public code solutions, introducing possible overlap with model training data. Adoption of unseen problem generation (as practiced in LiveCodeBench) is suggested to mitigate this risk in future iterations.

A plausible implication is that OpenAI o3's performance on UA-Code-Bench is an upper bound, conditioned by test suite construction and potential data leakage probability.

5. Comparative Status and Implications for Multilingual Code Generation

OpenAI o3, as evaluated in the UA-Code-Bench context, stands as one of the leading models for code generation in Ukrainian, alongside GPT-5. However, the maximal pass@1 rates of only 50% across the entire problem spectrum draw a sharp contrast with performance reported in resource-rich languages and on translated benchmarks. This gap underlines enduring methodological and architectural challenges when developing LLMs for low-resource or morphologically complex languages.

These findings demonstrate the necessity of culturally and linguistically representative, natively-authored programming benchmarks when evaluating multilingual LLMs. Future directions encouraged by the UA-Code-Bench findings include: expanding problem sources beyond Eolymp, supporting a broader set of output languages, reducing contamination through dynamic task generation, and exploring reasoning-augmented models. This suggests that despite OpenAI o3's notable advances, the research community must prioritize data diversity and robust test construction to advance general-purpose code generation for underrepresented languages.