Alvorada-Bench: Brazilian Academic Assessment Benchmark
- Alvorada-Bench is a benchmark that evaluates language models using 4,515 text-only questions drawn from five key Brazilian university entrance exams.
- It employs zero-shot, role-playing, and chain-of-thought prompting to assess models' accuracy, confidence, perceived difficulty, and cognitive complexity.
- Results show models excel in humanities and languages while facing challenges in multi-step quantitative reasoning, highlighting cost–accuracy trade-offs.
Alvorada-Bench is a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations and designed to evaluate whether LLMs can solve academically and culturally grounded assessment items in Brazilian Portuguese (Godoy, 19 Aug 2025). It evaluates twenty models under zero-shot, role-playing, and chain-of-thought prompting, yielding 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level (Godoy, 19 Aug 2025). The benchmark is centered on exams that distill decades of Brazilian educational priorities and assess millions of students yearly, and it is explicitly framed as a test of the intersection of language, culture, and reasoning that defines academic readiness in Brazil (Godoy, 19 Aug 2025).
1. Benchmark scope and corpus composition
Alvorada-Bench aggregates five examination sources, totaling 4,515 text-only multiple-choice questions (Godoy, 19 Aug 2025). The source exams are ENEM with 1,629 questions (36.1%), FUVEST with 1,303 questions (28.9%), UNICAMP with 716 questions (15.9%), ITA with 720 questions (15.9%), and IME with 147 questions (3.3%) (Godoy, 19 Aug 2025). The temporal coverage spans ENEM 2010–2024, FUVEST 1981–2025, UNICAMP 2011–2025, ITA 2008–2024, and IME 2017–2023 (Godoy, 19 Aug 2025).
| Exam | Questions | Share |
|---|---|---|
| ENEM | 1,629 | 36.1% |
| FUVEST | 1,303 | 28.9% |
| UNICAMP | 716 | 15.9% |
| ITA | 720 | 15.9% |
| IME | 147 | 3.3% |
The subject distribution is aligned with Brazil’s BNCC. Natural Sciences account for 1,667 questions (36.9%), Human Sciences for 1,275 (28.2%), Languages for 814 (18.0%), and Mathematics for 759 (16.8%) (Godoy, 19 Aug 2025). Natural Sciences comprise Chemistry, Physics, and Biology; Human Sciences comprise History, Geography, Sociology, and Philosophy; Languages comprise Portuguese, English, and Spanish; Mathematics is described as quantitative problem solving (Godoy, 19 Aug 2025).
A text-only filtering pipeline was used: PDF extraction, regex-based segmentation in a five-alternative format A–E, with four alternatives for UNICAMP, automatic exclusion of diagram- or map-based items, and normalization of formulas and notation (Godoy, 19 Aug 2025). Cognitive taxonomy and perceived difficulty are recorded per question, although per-level counts are not publicly enumerated in the paper; self-reported difficulty uses a 0–10 scale (Godoy, 19 Aug 2025).
The corpus composition is methodologically important because it combines a national standardized test with institution-specific vestibular examinations, including highly quantitative and engineering-oriented sources such as ITA and IME (Godoy, 19 Aug 2025). This suggests that Alvorada-Bench is not merely a Portuguese-language benchmark, but a benchmark with substantial variation in disciplinary profile, selectivity, and reasoning load.
2. Evaluation protocol and prompting design
The evaluation covers twenty models as of August 2025 (Godoy, 19 Aug 2025). The model set includes twelve OpenAI systems, six Anthropic systems, and two DeepSeek systems (Godoy, 19 Aug 2025). The OpenAI group includes GPT-4.1, GPT-4o, o1, o3, o4 mini and their “mini” or “nano” variants, as well as o1 preview; the Anthropic group includes Claude 3.5 Haiku, 3.5 Sonnet, 3.7 Sonnet, 4 Opus, 4 Sonnet, and 3 Opus; the DeepSeek group includes DeepSeek Chat and DeepSeek Reasoner (Godoy, 19 Aug 2025). Context lengths range from 64K to 1M tokens (Godoy, 19 Aug 2025).
Three prompting strategies are used, and all require JSON output with the keys "resposta", "confianca", "dificuldade", and "bloom" (Godoy, 19 Aug 2025). The "resposta" field is restricted to A–E, "confianca" is on a 0–10 scale, "dificuldade" is on a 0–10 scale, and "bloom" must be one of six levels (Godoy, 19 Aug 2025). The three prompting conditions are zero-shot with minimal instruction, role-playing in which the model adopts the persona of a top Brazilian vestibular student and invokes test-taking strategies, and chain-of-thought prompting that instructs step-by-step reasoning by identifying what is asked, decomposing the problem, eliminating options, and justifying the answer (Godoy, 19 Aug 2025).
This protocol makes structured self-report an integral part of the benchmark rather than an auxiliary probe. Because every response includes confidence, perceived difficulty, and a Bloom’s taxonomy label, the benchmark supports calibration analysis, correlation analysis, and cognitive-complexity analysis in addition to raw answer scoring (Godoy, 19 Aug 2025).
Prompt engineering effects are reported as limited. Figure 1 is described as showing under 1.6 percentage points of variation across zero-shot, role-playing, and chain-of-thought, with O3 varying only 0.1 percentage points (Godoy, 19 Aug 2025). A plausible implication is that the benchmark’s difficulty profile is driven less by superficial prompt framing than by underlying model capability.
3. Performance profile across models, subjects, and exams
Accuracy is defined as
The overall leaderboard reported in Table 2 places O3 Pro first at 0.9463, followed by O3 at 0.9455, O1 at 0.9308, DeepSeek Reasoner at 0.9271, and O4 Mini at 0.9150; the lowest reported model is GPT-4.1 Nano at 0.6049 (Godoy, 19 Aug 2025). The corpus mean is 0.8133, and the best–worst gap is 0.3414 (Godoy, 19 Aug 2025). The paper summarizes this as top systems now exceeding 94% overall (Godoy, 19 Aug 2025).
Subject-level performance is uneven. Mean accuracy across all models is 93.9% in Human Sciences, 90.8% in English within the Languages subset, approximately 82% in Natural Sciences, and 62.7% in Mathematics (Godoy, 19 Aug 2025). For mathematics specifically, the top reasoning models are O3 at 93.8% and DeepSeek Reasoner at 93.7%, compared with the baseline average of approximately 62.7% (Godoy, 19 Aug 2025).
Exam-type performance shows a similarly differentiated pattern. Figure 2 reports ENEM at 86.2%, UNICAMP at 86.1%, FUVEST at 82.1%, ITA at 68.1%, and IME at 61.4% (Godoy, 19 Aug 2025). The approximately 24.8 percentage-point drop from ENEM to IME or ITA is presented as highlighting difficulties with technical, multi-step problems (Godoy, 19 Aug 2025).
Figure 3 adds a cognitive-taxonomy perspective. It reports high accuracy on Remember at 92.4% and Understand at 92.0%, a valley at Apply at 69.7%, Evaluate at 87.8%, and Create above 90% for reasoning models (Godoy, 19 Aug 2025). This pattern is consistent with the reported weakness on technical quantitative items and suggests that the principal remaining deficit is not general linguistic decoding but the execution of applied, multi-step transformations.
4. Calibration, self-reported uncertainty, and perceived difficulty
A distinctive feature of Alvorada-Bench is that confidence is elicited directly on a 0–10 integer scale in each JSON response (Godoy, 19 Aug 2025). Calibration is analyzed using Expected Calibration Error and Brier Score. The reported formulas are
and
Exact ECE and Brier values are not tabulated, but the calibration curves show that high-confidence bins 9–10 achieve above 90% empirical accuracy (Godoy, 19 Aug 2025). Figure 4 also shows that responses with self-reported confidence less than or equal to 1 still exceed 90% accuracy, and that accuracy declines monotonically as reported confidence decreases (Godoy, 19 Aug 2025).
Perceived difficulty is also elicited on a 0–10 scale, enabling a direct uncertainty–difficulty correlation analysis (Godoy, 19 Aug 2025). Defining uncertainty as and difficulty as , the Pearson correlation is reported as
The paper states that this correlation is positive and substantial, and the plots suggest (Godoy, 19 Aug 2025). The interpretation given is that models correctly identify harder items (Godoy, 19 Aug 2025).
These results are notable because they do not only indicate high answer accuracy; they indicate that models can accurately assess their own certainty capabilities and flag difficult items (Godoy, 19 Aug 2025). This supports risk-aware educational deployments, although the benchmark itself does not define a deployment protocol (Godoy, 19 Aug 2025). A plausible implication is that abstention or escalation policies could be informed by self-reported uncertainty when models are used in high-stakes educational settings.
5. Cost–accuracy trade-offs and temporal change
Alvorada-Bench includes a cost–accuracy analysis based on price per 1,000 tokens as of August 2025 in USD (Godoy, 19 Aug 2025). Figure 4a identifies several frontier points: DeepSeek Reasoner reaches 92.71% at \$1.82 per 1K tokens, O3 Mini reaches 91.50% at \$1.95 per 1K tokens, and GPT-4.1 reaches 74.99% at \$15.00 per 1K tokens, which the paper describes as diminishing returns [2508.15835]. The break-even summary is that at least 91% accuracy is achievable for under \$2 per 1K tokens (Godoy, 19 Aug 2025).
| Model | Accuracy | Cost per 1K tokens |
|---|---|---|
| DeepSeek Reasoner | 92.71% | \$1.82 |
| O3 Mini | 91.50% | \$1.95 |
| GPT-4.1 | 74.99% | \$15.00 |
Figure 4b presents temporal evolution. The benchmark reports an increase from 73.6% for GPT-4o in May 2024 to 94.6% for O3 Pro in 2025, and it associates this change with reasoning-supervised architectures (Godoy, 19 Aug 2025). In the benchmark’s framing, this makes cost-efficiency and temporal progress inseparable: high accuracy is no longer confined to the most expensive systems (Godoy, 19 Aug 2025).
The broader significance is twofold. First, low-cost models at or below \$2 per 1K tokens are said to democratize access to high accuracy (Godoy, 19 Aug 2025). Second, the same result raises questions about equitable deployment (Godoy, 19 Aug 2025). The paper does not resolve those questions normatively, but it clearly positions them as a consequence of the benchmark’s findings.
6. Educational interpretation, human comparison, and limitations
The benchmark’s most direct educational comparison is on ENEM 2024, which the paper describes as involving more than 4 million students (Godoy, 19 Aug 2025). On that exam, O3 achieved 100% in Languages, while even GPT-4.1 Nano underperforms humans only in Mathematics (Godoy, 19 Aug 2025). Figure 6 is described as showing that all models surpass human baselines in Humanities, Natural Sciences, and Languages (Godoy, 19 Aug 2025).
The paper also reports cultural fluency: above 90% on humanities items rich in Brazilian-Portuguese literature and history (Godoy, 19 Aug 2025). The stated interpretation is that large-scale pre-training captures cultural knowledge without explicit targeting (Godoy, 19 Aug 2025). This is significant because Brazilian Portuguese evaluation has often been secondary to English-centric benchmarking, whereas Alvorada-Bench directly tests culturally specific content in a high-stakes assessment format (Godoy, 19 Aug 2025).
At the same time, the paper is explicit about a persistent bottleneck in multi-step quantitative reasoning. Standard models average approximately 62.7% in Mathematics, whereas reasoning-optimized ones approach 94% (Godoy, 19 Aug 2025). The exam breakdown reinforces the same point: performance is much lower on the engineering-oriented ITA and the highly quantitative IME than on ENEM, UNICAMP, or FUVEST (Godoy, 19 Aug 2025). The benchmark therefore does not support the conclusion that LLMs have solved vestibular-style reasoning in general. Rather, it supports the narrower conclusion that they rival or exceed human performance in most domains while still exhibiting a frontier on symbolic multi-step reasoning (Godoy, 19 Aug 2025).
A common misconception would be to treat the benchmark as measuring only Portuguese-language fluency. Its construction and results indicate a broader target: language, culture, and reasoning jointly (Godoy, 19 Aug 2025). Another misconception would be to infer that chain-of-thought prompting alone explains performance gains; the reported prompt variation is small, whereas the strongest gains are associated with reasoning-optimized models and reasoning-supervised architectures (Godoy, 19 Aug 2025).
7. Position within Brazilian-language evaluation
Alvorada-Bench is presented as a response to the fact that LLMs are increasingly used in Brazil while most evaluation remains English-centric (Godoy, 19 Aug 2025). Its core contribution is therefore not only the size of the corpus, but the specificity of the evaluation setting: five Brazilian university entrance examinations, text-only normalization, BNCC-aligned subject coverage, and structured introspective reporting (Godoy, 19 Aug 2025).
Within that setting, the benchmark establishes several points simultaneously. It shows that top models exceed 94% overall, that humanities and language performance are consistently high, that mathematics and engineering-oriented exams remain comparatively difficult, that confidence is well calibrated and correlates with perceived difficulty, and that high accuracy can be achieved at under \$2 per 1K tokens (Godoy, 19 Aug 2025). Taken together, these findings make Alvorada-Bench a culturally grounded diagnostic for Brazilian academic readiness rather than a generic multilingual benchmark.
The paper’s closing interpretation is that models now rival or exceed human performance on large-scale national exams in most domains, yet symbolic multi-step reasoning remains a frontier for further research and targeted alignment (Godoy, 19 Aug 2025). This suggests that future work will likely focus less on basic linguistic competence in Brazilian Portuguese and more on robustness for quantitative reasoning, technical problem solving, and reliable uncertainty-aware use in educational contexts.
