PoETa v2 Benchmark: Portuguese LLM Evaluation
- PoETa v2 is a comprehensive benchmark featuring 44 tasks (native and translated) to evaluate Portuguese language models.
- It employs task-specific few-shot prompting to systematically measure areas like reasoning, cultural knowledge, exams, and social media.
- Empirical findings reveal that increased compute and Portuguese-specific adaptation enhance performance, though gaps with English persist.
PoETa v2, short for “Portuguese Evaluation Tasks,” is a benchmark for evaluating LLMs in Portuguese. It is presented as the most extensive Portuguese LLM evaluation suite to date, comprising over 40 tasks in Portuguese and used to assess more than 20 models. Its design targets systematic measurement of how base or non-instruction-tuned LLMs handle Portuguese across a broad range of abilities, while exposing language-specific and cultural performance differences that are often obscured by English-centered evaluation practice (Almeida et al., 21 Nov 2025).
1. Definition, scope, and positioning
PoETa v2 is intended as a standardized benchmark for Portuguese language modeling and evaluation. Its central motivation is that LLM performance varies across languages and cultural or regional contexts, Portuguese is underrepresented relative to English in typical pretraining corpora, and existing Portuguese evaluation resources are fragmented and often restricted to narrow domains such as entrance exams or sentiment tasks. The benchmark is therefore positioned as a broader multitask resource for tracking progress in Portuguese and for comparing models trained with different scales, data mixtures, and adaptation strategies (Almeida et al., 21 Nov 2025).
The benchmark is not framed as a generic chatbot evaluation. Instead, it is oriented toward the early stages of model development and pretraining evaluation. For that reason, it prioritizes tasks that can be solved from pretraining knowledge alone and intentionally excludes open-ended chat tasks, code generation benchmarks, complex reasoning challenges, and agentic evaluations. This suggests a deliberate separation between Portuguese language competence as measured through controlled task formats and broader assistant-style behavior, which the benchmark does not attempt to cover.
PoETa v2 also occupies a specific place relative to earlier Portuguese resources. The paper contrasts it with structured exam datasets such as ENEM and university entrance exams, and with classification tasks derived from informal or social-media data such as TweetSentBR, HateBR, and PT Hate Speech. A plausible implication is that PoETa v2 aims to consolidate these previously scattered evaluation traditions into a single benchmark while extending coverage into capability areas that had been undersupplied in Portuguese.
2. Benchmark composition and task design
PoETa v2 contains 44 tasks divided into two high-level groups: 12 native Portuguese tasks and 32 translated tasks adapted from established English benchmarks. Native tasks are intended to capture regional knowledge, culture, idioms, exams, and Portuguese-specific linguistic phenomena, whereas translated tasks expand coverage into broader reasoning, commonsense, and text-understanding domains that currently lack enough Portuguese-native evaluation resources (Almeida et al., 21 Nov 2025).
The benchmark assigns each task a primary task type. The paper defines the following types: regression, multiple-choice, binary QA, sentence entailment, extractive QA, classification, and sentiment analysis. Each task is also annotated with one or more subcategories, including math, brazil, reasoning, common-sense, text-understanding, exams, ethics, code, social-media, general-knowledge, proverbs, and hate-speech. This makes PoETa v2 a heterogeneous benchmark both in supervision format and in capability coverage.
The native Portuguese inventory reported in the appendix includes ASSIN RTE, ASSIN STS, BLUEX, ENEM, ENEM 2022, FaQuAD, TweetSentBR, BRoverbs History to Proverb, BRoverbs Proverb to History, InferBR, RePro, MINA-BR, PT Hate Speech, HateBR Binary, and POSComp. The translated or adapted inventory includes AG News, BoolQ, IMDB, MASSIVE, MKQA, SST-2, WSC-285, several BIG-Bench subsets, ARC Challenge, ARC Easy, StoryCloze, ETHICS Commonsense, Math MC, GSM8K MC, AGIEval SAT Math, Balanced COPA, and LogiQA. The paper notes a discrepancy here: the main text says 12 native tasks, while the appendix lists 15 tasks marked “Translated = No.” The paper does not explain this discrepancy.
Several capability clusters are especially prominent. Text understanding and reading comprehension are represented by tasks such as FaQuAD, StoryCloze, and BIG-Bench VitaminC Fact Verification. Reasoning and logical inference are represented by ASSIN RTE, InferBR, BIG-Bench Analogical Similarity, BIG-Bench Mathematical Induction, Balanced COPA, and LogiQA. Exams and academically grounded knowledge are represented by BLUEX, ENEM, ENEM 2022, POSComp, ARC, AGIEval SAT Math, Math MC, and GSM8K MC. Social-media and offensive-language understanding are represented by TweetSentBR, RePro, MINA-BR, PT Hate Speech, and HateBR. Cultural and Brazil-specific coverage comes from ENEM, BLUEX, BRoverbs, POSComp, and Brazilian social-media and hate-speech datasets.
This combination of native and translated tasks is one of the benchmark’s defining design choices. The paper explicitly treats it as both a strength and a limitation: it broadens coverage, but also shows that Portuguese evaluation still depends heavily on translated benchmarks. This suggests that PoETa v2 functions not only as an evaluation suite but also as a map of what Portuguese-native benchmark development still lacks.
3. Evaluation methodology and aggregate scoring
PoETa v2 is evaluated mainly as a few-shot benchmark. It is designed so that models can be tested without supervised fine-tuning and primarily in a base-model or pretraining-stage setting. The benchmark is therefore prompt-based and task-specific, but the paper does not describe a unified generation-versus-ranking protocol, nor does it specify decoding parameters or exact prompt templates (Almeida et al., 21 Nov 2025).
Few-shot settings vary by task. The paper states that tasks already present in PoETa v1 retain the same few-shot counts, while new tasks in v2 use 5 few-shot examples. Most tasks use 5-shot prompting, but there are notable deviations: ASSIN RTE uses 18, ASSIN STS 15, BLUEX and both ENEM tasks 1, FaQuAD 4, IMDB 2, MASSIVE 36, MKQA 40, SST-2 34, and WSC-285 18. PoETa v2 is therefore neither uniformly zero-shot nor uniformly fixed-shot; it is a task-specific few-shot benchmark.
Per-task evaluation uses a preferred metric. Unless otherwise noted, tasks are translated into Portuguese and evaluated using accuracy as the primary metric. FaQuAD uses F1, and the paper notes that the benchmark’s aggregation framework can accommodate metrics such as accuracy, F1-score, and exact match. For ASSIN 2 STS, which is a regression task, the paper does not explicitly state in the provided text which regression metric is used in the aggregate.
The benchmark’s principal aggregate is the Normalized Preferred Metric, or NPM, originally introduced in PoETa v1. The formula is given as
This normalization maps random performance to 0 and perfect performance to 1. Its purpose is to make task scores comparable despite different scales and random baselines. The paper’s motivating example is that a four-choice multiple-choice task has random accuracy of 25%, whereas open-ended QA may have a random score effectively near 0%. A raw average over such tasks would therefore be misleading.
The benchmark’s overall score is the average NPM across all PoETa v2 tasks. The paper does not state any weighting scheme beyond this task-level average, nor does it report confidence intervals, bootstrap estimates, or statistical significance tests. A plausible implication is that PoETa v2 is optimized for descriptive benchmarking rather than inferential comparison.
The paper also analyzes model quality as a function of estimated training compute. It defines multiply–accumulate operations per token and overall training cost , where is the number of training tokens and the cost formulation is adapted from DeepSeek’s non-embedding FLOPs-per-token metric. However, the formula for appears corrupted in the provided text, and the paper explicitly omits sequence length because recent models are pretrained in multiple stages with different context lengths. The compute analysis is thus conceptually central, even if the exact expression is not fully recoverable from the text.
4. Model coverage and headline empirical findings
PoETa v2 is used to evaluate more than 20 models spanning Portuguese-specialized systems, general open-source multilingual families, and commercial models. The main comparison includes CuriÓ 1.1B, CuriÓ 7B, Sabiá 7B, Sabiá 3, TinyLlama 1T, Llama 1 7B, Llama 2 7B, Llama 2 13B, Llama 3.1 8B, Falcon 3 1B, Falcon 3 3B, Falcon 3 7B, Falcon 3 10B, Qwen 1 1.8B, Qwen 1 7B, Qwen 2 1.5B, Qwen 2 7B, Qwen 2.5 1.5B, Qwen 2.5 3B, Qwen 2.5 7B, Qwen 2.5 14B, Qwen 3 1.7B, Qwen 3 4B, Qwen 3 8B, Qwen 3 14B, GPT-4.1, GPT-4o, and Sabiá 3 (Almeida et al., 21 Nov 2025).
The top overall PoETa v2 average NPM scores reported in Table 2 are:
| Model | Average NPM |
|---|---|
| GPT-4.1 | 76.2 |
| GPT-4o | 75.2 |
| Sabiá 3 | 72.2 |
| Qwen 2.5 14B | 71.0 |
| Qwen 3 14B | 70.5 |
| Qwen 3 8B | 64.7 |
| Qwen 2.5 7B | 63.7 |
| Falcon 3 10B | 63.5 |
| Qwen 3 4B | 59.2 |
| Qwen 2 7B | 58.8 |
| Falcon 3 7B | 58.5 |
| Llama 3.1 8B | 53.5 |
Among open-source models, Qwen 2.5 14B and Qwen 3 14B are the strongest. Among commercial systems, GPT-4.1 is highest overall. Among Portuguese-oriented commercial systems in the table, Sabiá 3 exceeds all reported open-source models. This places PoETa v2 not only as a Portuguese benchmark but also as a platform for comparing multilingual pretraining strategies against language-specific specialization.
One of the paper’s clearest findings is a strong positive correlation between training compute and Portuguese performance. Larger models with more pretraining data generally perform better on PoETa v2. Yet the paper emphasizes that compute is not the whole story. Qwen 1 7B and Llama 2 7B have similar compute, but Qwen 1 7B scores 41.1 while Llama 2 7B scores 29.5. Qwen 2.5 7B scores 63.7 against 53.5 for Llama 3.1 8B despite comparable compute. The paper interprets this as evidence that architecture and training-data composition, especially exposure to Portuguese, matter substantially in addition to raw scale.
The paper also analyzes performance by category. Compute-performance correlations are high for text understanding, exams, ethics, common sense, Brazil, reasoning, and math, while social media and general knowledge scale less predictably. Appendix Figure 1 further indicates that sentiment analysis scales more slowly than multiple-choice tasks, suggesting that some sentiment tasks are relatively easier and less compute-sensitive.
5. Portuguese adaptation, cross-lingual comparison, and benchmark insights
One of PoETa v2’s most important empirical results concerns continued pretraining on Portuguese. The paper reports consistent gains from Portuguese adaptation over base models: TinyLlama 1T to CuriÓ 1.1B yields +4.5 NPM, Llama 2 7B to CuriÓ 7B yields +5.3 NPM, and Llama 1 7B to Sabiá 7B yields +12.5 NPM (Almeida et al., 21 Nov 2025).
These gains are not uniform across subcategories. For TinyLlama 1T to CuriÓ 1.1B, reasoning shows a gain of +15.86, common sense +7.17, and math +5.76, while general knowledge and code decline. For Llama 1 7B to Sabiá 7B, large improvements appear in Brazil (+19.42), code (+19.15), math (+19.15), and common sense (+15.95). For Llama 2 7B to CuriÓ 7B, the strongest improvements are in Brazil (+10.94), common sense (+8.16), and reasoning (+4.22), while code and math decline. The paper treats this as evidence that Portuguese adaptation helps beyond scale but does so unevenly across capabilities.
PoETa v2 also supports direct Portuguese-versus-English comparison on matched BIG-Bench subsets. The reported average gap is 3.5 NPM points, usually favoring English. This gap decreases with model size: models under 5B parameters show an average gap of 5.1, models in the 7B–8B range show 4.3, and models at or above 10B show 3.8. The paper interprets this as evidence that larger models have stronger multilingual capabilities and therefore narrower language gaps.
There are, however, important exceptions. CuriÓ 7B scores 30.9 on BIG-Bench PT versus 27.2 on BIG-Bench EN, Qwen 2.5 14B scores 63.5 versus 62.0, and Sabiá 3 scores 62.8 versus 59.4. By contrast, Qwen 3 8B shows a Portuguese-English gap of 5.5 and Qwen 3 14B a gap of 9.5. The paper suggests possible causes such as different pretraining data composition or possible contamination in English, but it performs no formal contamination check.
The benchmark also yields Portuguese-specific insights that go beyond generic multilingual evaluation. It shows that Brazil-related tasks improve with both compute and adaptation, informal and socially grounded Portuguese remains difficult, and social-media performance scales more weakly than other subcategories. At the same time, the benchmark is heavily weighted toward Brazilian Portuguese in its native tasks, including Brazilian exams, Brazilian proverbs, Brazilian social media, Brazilian NLI, Brazilian reviews, and Brazilian hate-speech data. The paper does not provide a dedicated European-versus-Brazilian Portuguese breakdown.
6. Limitations, caveats, and future directions
The paper presents PoETa v2 as an important step rather than a complete solution. Its first major limitation is reliance on translated tasks. Although it includes native Portuguese resources, the benchmark still depends heavily on translations from English, which limits cultural and linguistic authenticity. The paper explicitly identifies missing native coverage in ethical reasoning, commonsense knowledge, code understanding, and advanced problem solving (Almeida et al., 21 Nov 2025).
A second limitation is scope. PoETa v2 is oriented toward relatively simple tasks suitable for measuring pretraining-stage progress. It does not cover long-text generation, open-ended interaction, agentic skills, or advanced fine-tuned assistant behavior. This is a design choice rather than an omission in the narrow sense, but it constrains the kinds of claims that can be made from benchmark results.
A third limitation is methodological transparency. The paper does not report exact numbers of examples per task used in evaluation, train-dev-test handling, annotation workflows, inter-annotator agreement, deduplication or overlap removal, contamination audits, licensing status, or prompt-format standardization details beyond few-shot counts. It also notes a practical evaluation caveat: some models are sensitive to prompt-following conditions. Appendix discussion states that Tucano 2.4B sometimes produced invalid option letters in few-shot prompting, causing negative NPM on some tasks.
A fourth limitation is the absence of inferential statistics. The reported differences are descriptive: the benchmark does not provide confidence intervals or significance testing. This suggests that PoETa v2 is best used as a broad comparative instrument and for trend analysis, rather than for strong statistical claims about small score differences.
The paper identifies several future directions. These include expanding native Portuguese tasks, especially in ethical reasoning, commonsense, code understanding, and advanced problem solving; analyzing pretraining-data composition more deeply; studying fairness, bias, and robustness in Portuguese contexts; and extending evaluation to fine-tuned models and more advanced capabilities. In that sense, PoETa v2 functions both as a benchmark and as an agenda-setting resource for Portuguese LLM evaluation.
PoETa v2 is publicly available at the repository cited in the paper. Its enduring significance lies in providing a shared, public, standardized framework for evaluating Portuguese LLMs while making visible three recurring facts: scaling improves Portuguese performance, Portuguese-specific adaptation improves it further, and Portuguese still lags English on matched tasks even in strong models.