---
title: 'Alvorada-Bench: Brazilian Academic Assessment Benchmark'
url: https://www.emergentmind.com/topics/alvorada-bench
type: topic
---

# Alvorada-Bench: Brazilian Academic Assessment Benchmark

Alvorada-Bench is a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations and designed to evaluate whether language models can solve academically and culturally grounded assessment items in Brazilian Portuguese [2508.15835]. It evaluates twenty models under zero-shot, role-playing, and chain-of-thought prompting, yielding 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level [2508.15835]. The benchmark is centered on exams that distill decades of Brazilian educational priorities and assess millions of students yearly, and it is explicitly framed as a test of the intersection of language, culture, and reasoning that defines academic readiness in Brazil [2508.15835].

## 1. Benchmark scope and corpus composition

Alvorada-Bench aggregates five examination sources, totaling 4,515 text-only multiple-choice questions [2508.15835]. The source exams are ENEM with 1,629 questions (36.1%), FUVEST with 1,303 questions (28.9%), UNICAMP with 716 questions (15.9%), ITA with 720 questions (15.9%), and IME with 147 questions (3.3%) [2508.15835]. The temporal coverage spans ENEM 2010–2024, FUVEST 1981–2025, UNICAMP 2011–2025, ITA 2008–2024, and IME 2017–2023 [2508.15835].

| Exam | Questions | Share |
|---|---:|---:|
| ENEM | 1,629 | 36.1% |
| FUVEST | 1,303 | 28.9% |
| UNICAMP | 716 | 15.9% |
| ITA | 720 | 15.9% |
| IME | 147 | 3.3% |

The subject distribution is aligned with Brazil’s BNCC. Natural Sciences account for 1,667 questions (36.9%), Human Sciences for 1,275 (28.2%), Languages for 814 (18.0%), and Mathematics for 759 (16.8%) [2508.15835]. Natural Sciences comprise Chemistry, Physics, and Biology; Human Sciences comprise History, Geography, Sociology, and Philosophy; Languages comprise Portuguese, English, and Spanish; Mathematics is described as quantitative problem solving [2508.15835].

A text-only filtering pipeline was used: PDF extraction, regex-based segmentation in a five-alternative format A–E, with four alternatives for UNICAMP, automatic exclusion of diagram- or map-based items, and normalization of formulas and notation [2508.15835]. Cognitive taxonomy and perceived difficulty are recorded per question, although per-level counts are not publicly enumerated in the paper; self-reported difficulty uses a 0–10 scale [2508.15835].

The corpus composition is methodologically important because it combines a national standardized test with institution-specific vestibular examinations, including highly quantitative and engineering-oriented sources such as ITA and IME [2508.15835]. This suggests that Alvorada-Bench is not merely a Portuguese-language benchmark, but a benchmark with substantial variation in disciplinary profile, selectivity, and reasoning load.

## 2. Evaluation protocol and prompting design

The evaluation covers twenty models as of August 2025 [2508.15835]. The model set includes twelve OpenAI systems, six Anthropic systems, and two DeepSeek systems [2508.15835]. The OpenAI group includes GPT-4.1, GPT-4o, o1, o3, o4 mini and their “mini” or “nano” variants, as well as o1 preview; the Anthropic group includes Claude 3.5 Haiku, 3.5 Sonnet, 3.7 Sonnet, 4 Opus, 4 Sonnet, and 3 Opus; the DeepSeek group includes DeepSeek Chat and DeepSeek Reasoner [2508.15835]. Context lengths range from 64K to 1M tokens [2508.15835].

Three prompting strategies are used, and all require JSON output with the keys `"resposta"`, `"confianca"`, `"dificuldade"`, and `"bloom"` [2508.15835]. The `"resposta"` field is restricted to A–E, `"confianca"` is on a 0–10 scale, `"dificuldade"` is on a 0–10 scale, and `"bloom"` must be one of six levels [2508.15835]. The three prompting conditions are zero-shot with minimal instruction, role-playing in which the model adopts the persona of a top Brazilian vestibular student and invokes test-taking strategies, and chain-of-thought prompting that instructs step-by-step reasoning by identifying what is asked, decomposing the problem, eliminating options, and justifying the answer [2508.15835].

This protocol makes structured self-report an integral part of the benchmark rather than an auxiliary probe. Because every response includes confidence, perceived difficulty, and a Bloom’s taxonomy label, the benchmark supports calibration analysis, correlation analysis, and cognitive-complexity analysis in addition to raw answer scoring [2508.15835].

Prompt engineering effects are reported as limited. Figure 8 is described as showing under 1.6 percentage points of variation across zero-shot, role-playing, and chain-of-thought, with O3 varying only 0.1 percentage points [2508.15835]. A plausible implication is that the benchmark’s difficulty profile is driven less by superficial prompt framing than by underlying model capability.

## 3. Performance profile across models, subjects, and exams

Accuracy is defined as

$$
\mathrm{Accuracy} = \frac{\#\text{correct answers}}{\#\text{questions}}.
$$

The overall leaderboard reported in Table 2 places O3 Pro first at 0.9463, followed by O3 at 0.9455, O1 at 0.9308, DeepSeek Reasoner at 0.9271, and O4 Mini at 0.9150; the lowest reported model is GPT-4.1 Nano at 0.6049 [2508.15835]. The corpus mean is 0.8133, and the best–worst gap is 0.3414 [2508.15835]. The paper summarizes this as top systems now exceeding 94% overall [2508.15835].

Subject-level performance is uneven. Mean accuracy across all models is 93.9% in Human Sciences, 90.8% in English within the Languages subset, approximately 82% in Natural Sciences, and 62.7% in Mathematics [2508.15835]. For mathematics specifically, the top reasoning models are O3 at 93.8% and DeepSeek Reasoner at 93.7%, compared with the baseline average of approximately 62.7% [2508.15835].

Exam-type performance shows a similarly differentiated pattern. Figure 7 reports ENEM at 86.2%, UNICAMP at 86.1%, FUVEST at 82.1%, ITA at 68.1%, and IME at 61.4% [2508.15835]. The approximately 24.8 percentage-point drop from ENEM to IME or ITA is presented as highlighting difficulties with technical, multi-step problems [2508.15835].

Figure 9 adds a cognitive-taxonomy perspective. It reports high accuracy on Remember at 92.4% and Understand at 92.0%, a valley at Apply at 69.7%, Evaluate at 87.8%, and Create above 90% for reasoning models [2508.15835]. This pattern is consistent with the reported weakness on technical quantitative items and suggests that the principal remaining deficit is not general linguistic decoding but the execution of applied, multi-step transformations.

## 4. Calibration, self-reported uncertainty, and perceived difficulty

A distinctive feature of Alvorada-Bench is that confidence is elicited directly on a 0–10 integer scale in each JSON response [2508.15835]. Calibration is analyzed using Expected Calibration Error and Brier Score. The reported formulas are

$$
\mathrm{ECE} = \sum_{m=1}^M \frac{|B_m|}{n} \bigl|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\bigr|
$$

and

$$
\mathrm{BS} = \frac{1}{n} \sum_{i=1}^n \left(\mathrm{conf}_i - \mathbf{1}\{y_i=\hat y_i\}\right)^2.
$$

Exact ECE and Brier values are not tabulated, but the calibration curves show that high-confidence bins 9–10 achieve above 90% empirical accuracy [2508.15835]. Figure 5 also shows that responses with self-reported confidence less than or equal to 1 still exceed 90% accuracy, and that accuracy declines monotonically as reported confidence decreases [2508.15835].

Perceived difficulty is also elicited on a 0–10 scale, enabling a direct uncertainty–difficulty correlation analysis [2508.15835]. Defining uncertainty as $u_i = 10-\mathrm{conf}_i$ and difficulty as $d_i = \mathrm{dificuldade}_i$, the Pearson correlation is reported as

$$
r = \frac{\sum (u_i-\bar u)(d_i-\bar d)}{\sqrt{\sum (u_i-\bar u)^2\,\sum (d_i-\bar d)^2}}.
$$

The paper states that this correlation is positive and substantial, and the plots suggest $r \approx 0.7\text{–}0.8$ [2508.15835]. The interpretation given is that models correctly identify harder items [2508.15835].

These results are notable because they do not only indicate high answer accuracy; they indicate that models can accurately assess their own certainty capabilities and flag difficult items [2508.15835]. This supports risk-aware educational deployments, although the benchmark itself does not define a deployment protocol [2508.15835]. A plausible implication is that abstention or escalation policies could be informed by self-reported uncertainty when models are used in high-stakes educational settings.

## 5. Cost–accuracy trade-offs and temporal change

Alvorada-Bench includes a cost–accuracy analysis based on price per 1,000 tokens as of August 2025 in USD [2508.15835]. Figure 4a identifies several frontier points: DeepSeek Reasoner reaches 92.71% at \$1.82 per 1K tokens, O3 Mini reaches 91.50% at \$1.95 per 1K tokens, and GPT-4.1 reaches 74.99% at \$15.00 per 1K tokens, which the paper describes as diminishing returns [2508.15835]. The break-even summary is that at least 91% accuracy is achievable for under \$2 per 1K tokens [2508.15835].

| Model | Accuracy | Cost per 1K tokens |
|---|---:|---:|
| DeepSeek Reasoner | 92.71% | \$1.82 |
| O3 Mini | 91.50% | \$1.95 |
| GPT-4.1 | 74.99% | \$15.00 |

Figure 4b presents temporal evolution. The benchmark reports an increase from 73.6% for GPT-4o in May 2024 to 94.6% for O3 Pro in 2025, and it associates this change with reasoning-supervised architectures [2508.15835]. In the benchmark’s framing, this makes cost-efficiency and temporal progress inseparable: high accuracy is no longer confined to the most expensive systems [2508.15835].

The broader significance is twofold. First, low-cost models at or below \$2 per 1K tokens are said to democratize access to high accuracy [2508.15835]. Second, the same result raises questions about equitable deployment [2508.15835]. The paper does not resolve those questions normatively, but it clearly positions them as a consequence of the benchmark’s findings.

## 6. Educational interpretation, human comparison, and limitations

The benchmark’s most direct educational comparison is on ENEM 2024, which the paper describes as involving more than 4 million students [2508.15835]. On that exam, O3 achieved 100% in Languages, while even GPT-4.1 Nano underperforms humans only in Mathematics [2508.15835]. Figure 3 is described as showing that all models surpass human baselines in Humanities, Natural Sciences, and Languages [2508.15835].

The paper also reports cultural fluency: above 90% on humanities items rich in Brazilian-Portuguese literature and history [2508.15835]. The stated interpretation is that large-scale pre-training captures cultural knowledge without explicit targeting [2508.15835]. This is significant because Brazilian Portuguese evaluation has often been secondary to English-centric benchmarking, whereas Alvorada-Bench directly tests culturally specific content in a high-stakes assessment format [2508.15835].

At the same time, the paper is explicit about a persistent bottleneck in multi-step quantitative reasoning. Standard models average approximately 62.7% in Mathematics, whereas reasoning-optimized ones approach 94% [2508.15835]. The exam breakdown reinforces the same point: performance is much lower on the engineering-oriented ITA and the highly quantitative IME than on ENEM, UNICAMP, or FUVEST [2508.15835]. The benchmark therefore does not support the conclusion that language models have solved vestibular-style reasoning in general. Rather, it supports the narrower conclusion that they rival or exceed human performance in most domains while still exhibiting a frontier on symbolic multi-step reasoning [2508.15835].

A common misconception would be to treat the benchmark as measuring only Portuguese-language fluency. Its construction and results indicate a broader target: language, culture, and reasoning jointly [2508.15835]. Another misconception would be to infer that chain-of-thought prompting alone explains performance gains; the reported prompt variation is small, whereas the strongest gains are associated with reasoning-optimized models and reasoning-supervised architectures [2508.15835].

## 7. Position within Brazilian-language evaluation

Alvorada-Bench is presented as a response to the fact that language models are increasingly used in Brazil while most evaluation remains English-centric [2508.15835]. Its core contribution is therefore not only the size of the corpus, but the specificity of the evaluation setting: five Brazilian university entrance examinations, text-only normalization, BNCC-aligned subject coverage, and structured introspective reporting [2508.15835].

Within that setting, the benchmark establishes several points simultaneously. It shows that top models exceed 94% overall, that humanities and language performance are consistently high, that mathematics and engineering-oriented exams remain comparatively difficult, that confidence is well calibrated and correlates with perceived difficulty, and that high accuracy can be achieved at under \$2 per 1K tokens [2508.15835]. Taken together, these findings make Alvorada-Bench a culturally grounded diagnostic for Brazilian academic readiness rather than a generic multilingual benchmark.

The paper’s closing interpretation is that models now rival or exceed human performance on large-scale national exams in most domains, yet symbolic multi-step reasoning remains a frontier for further research and targeted alignment [2508.15835]. This suggests that future work will likely focus less on basic linguistic competence in Brazilian Portuguese and more on robustness for quantitative reasoning, technical problem solving, and reliable uncertainty-aware use in educational contexts.

Source: https://www.emergentmind.com/topics/alvorada-bench