BRoverbs: Brazilian Proverbs Benchmark

• BRoverbs is a culturally grounded benchmark featuring Brazilian proverbs paired with illustrative narratives to assess figurative and idiomatic understanding in LLMs.
• It implements dual tasks—Proverb-to-Story and Story-to-Proverb—using multiple-choice questions with manually verified distractors to evaluate narrative mapping.
• The benchmark highlights cultural nuances and exposes capacity thresholds by comparing performance across commercial and open-source language models.

BRoverbs is a dataset specifically designed to assess LLM performance through Brazilian proverbs. It is a culturally grounded benchmark that measures whether LLMs truly understand Brazilian Portuguese proverbs—compact, figurative, and culturally embedded expressions that require more than literal word-to-word translation. By pairing proverbs with short, illustrative narratives and evaluating models in proverb-to-story and story-to-proverb multiple-choice settings, BRoverbs targets figurative-language comprehension, idiomaticity, pragmatics, and cultural knowledge in Portuguese, and is presented as a regionally informed evaluation resource in a setting where translated benchmarks often miss authentic cultural signals (Almeida et al., 10 Sep 2025).

1. Motivation and conceptual scope

BRoverbs is motivated by the observation that LLMs exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. For Portuguese, existing evaluations remain limited, often relying on translated datasets that may not fully capture linguistic nuances or cultural references, while native Portuguese-language datasets predominantly focus on structured national exams or sentiment analysis of social media interactions. The benchmark is therefore framed as filling a gap in evaluating broader linguistic understanding, especially figurative and culturally specific understanding (Almeida et al., 10 Sep 2025).

The choice of proverbs is central to that design. Proverbs encode folk wisdom via figurative language, idiomatic expressions, and pragmatics; their interpretation is not compositionally recoverable from literal word meaning. Brazilian Portuguese proverbs further involve metaphor, hyperbole, cultural references and folk sayings, idiomaticity in the sense of fixed forms with conventional meanings, and sometimes syntactic idiosyncrasies. Distinguishing near-synonymous proverbs and mapping them to life scenarios demands pragmatic inference and cultural familiarity.

Within Portuguese evaluation, BRoverbs is positioned against the limitations of translated benchmarks. Translation often loses conventionalized meaning and usage contexts, and translated items may not reflect authentic cultural signals. The paper also notes prior work, such as Sabiá, observing that models pre-trained on Portuguese data improve more on natively designed Portuguese tasks than on translated ones. In that sense, BRoverbs is not only a task resource but also an argument for culturally native benchmarking. It is presented as complementing recent cross-cultural tests such as WorldBench, BLEnD, and TiEBe, as well as figurative-language evaluations such as ProverbEval, with a focused Brazilian Portuguese resource.

2. Dataset construction and curation

The collection pipeline begins with three web searches for Brazilian proverb lists—“lista de provérbios brasileiros”, “ditados populares brasileiros”, and “provérbios populares brasileiros”—which yielded 15 sites. Proverbs were extracted, clustered via fuzzy string matching, and deduplicated. This produced 196 unique proverbs; after manual verification, 3 were dropped, including “Batatinha quando nasce, se esparrama pelo chão” due to disagreement on meaning, yielding 193 proverbs (Almeida et al., 10 Sep 2025).

For each proverb, GPT-4 generated three short stories that implicitly convey the proverb’s meaning, plus a short explanation, labeled “explicação”, of the moral. Human verification and edits ensured that each story accurately reflected the proverb, and in rare cases stories were rewritten. The resulting dataset contains 579 short stories, corresponding to 3 stories for each of the 193 proverbs, together with an explanation per proverb.

The core fields per proverb are proverbio, explicacao, historia_curta_1, historia_curta_2, and historia_curta_3. From these entries, the benchmark constructs multiple-choice items for two tasks, each with 5 options, consisting of 1 correct answer and 4 distractors drawn from other items. Distractors were then manually validated to avoid unintended alternative correct answers. The paper does not report additional metadata such as themes or domains, literal versus figurative tags, dialect markers, or usage contexts beyond the explanation field.

Quality control follows a two-stage validation protocol. First, there was initial manual curation of distractors for every question. Second, there was a post-hoc review of all items that at least one of four commercial models answered incorrectly, approximately 12% of questions, of which approximately 35% were modified to remove ambiguity. Inter-annotator agreement is not reported, and no Cohen’s kappa is provided. The dataset is also characterized statistically: story character counts are fairly consistent across the dataset, and Jaccard distances among triplet stories per proverb are high, with most greater than 0.9, indicating low word overlap and diverse lexicalization of the same underlying meaning. Splits are not reported; the dataset is provided as a single resource in this version.

3. Task design and evaluation protocol

BRoverbs defines two complementary multiple-choice meaning-alignment tasks. In Proverb-to-Story (PtS), the prompt asks: “Qual das histórias melhor corresponde ao provérbio dado?” Given a proverb, the model must select the story that best instantiates its meaning. In Story-to-Proverb (StP), the prompt asks: “Qual dos provérbios melhor corresponde à história dada?” Given a story, the model must select the proverb that best matches it (Almeida et al., 10 Sep 2025).

The prompt templates are given in Portuguese. For PtS, the template is “Qual das histórias melhor corresponde ao provérbio dado? Provérbio: {proverbio} [A–E: cinco histórias curtas]”. For StP, it is “Qual dos provérbios melhor corresponde à história dada? História: {historia} [A–E: cinco provérbios]”. The release focuses on these two tasks and does not include paraphrase, cloze, translation scoring, or theme classification tasks, although the examples provide literal English translations of prompts for illustration.

Scoring is by simple accuracy over multiple-choice selection:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{y}_i = y_i].$$

The paper also lists Precision, Recall, F1, BLEU, ROUGE-L, and BERTScore for reference, but in BRoverbs v1 the reported metric is Accuracy. The evaluation protocol uses a 1-fewshot setting: one randomly selected example is included per test instance as a single-shot illustration, different from the test item to avoid leakage. Open-source models were run on an RTX A6000, whereas commercial models were accessed via API; decoding settings beyond few-shot are not detailed.

The benchmark documentation also includes example items. One PtS example uses the proverb “O que vem fácil, vai fácil.” with the English gloss “What comes easy, goes easy.” Its figurative meaning is that gains obtained without effort tend to be lost quickly or carelessly, and the correct story is the one in which a character wins a large sum of money by chance and quickly squanders it, ending up in debt. An StP example presents the story “Mariana viu um homem com roupas rasgadas… mais tarde descobriu que ele era um renomado artista de rua…” and targets the proverb “Não julgue um livro pela capa.”

4. Empirical results

The empirical evaluation covers large commercial, small commercial, medium open-source, and small open-source systems. The paper reports the following accuracies for PtS, StP, and their average.

Category	Model	Accuracy (PtS / StP / avg)
Large commercial	GPT-4o	0.97 / 0.98 / 0.97
Large commercial	Claude 3.5 Sonnet	0.96 / 0.98 / 0.97
Large commercial	Sabiá-3	0.96 / 0.96 / 0.96
Small commercial	GPT-4o-mini	0.94 / 0.97 / 0.95
Small commercial	Claude 3.5 Haiku	0.95 / 0.94 / 0.95
Small commercial	Sabiazinho-3	0.94 / 0.96 / 0.95
Medium open-source	Qwen 2.5 14B	0.92 / 0.94 / 0.93
Medium open-source	Qwen 2.5 7B	0.83 / 0.83 / 0.83
Medium open-source	Llama-3 8B	0.48 / 0.75 / 0.61
Medium open-source	Llama-2 7B	0.22 / 0.25 / 0.23
Medium open-source	Llama-1 7B	0.21 / 0.24 / 0.22
Medium open-source	Sabiá-7B	0.23 / 0.33 / 0.28
Small open-source	Qwen 2.5 3B	0.51 / 0.68 / 0.59
Small open-source	Qwen 2.5 1.5B	0.41 / 0.57 / 0.49
Small open-source	Tucano 2.4B	0.19 / 0.20 / 0.19
Small open-source	Tucano 1.1B	0.19 / 0.20 / 0.19
Small open-source	TinyLlama 3T (1.1B)	0.19 / 0.20 / 0.19
Small open-source	TinyLlama 1T (1.1B)	0.18 / 0.22 / 0.20
Small open-source	Curiò 1.1B	0.20 / 0.20 / 0.20

Several regularities are emphasized in the paper (Almeida et al., 10 Sep 2025). StP is generally easier than PtS across models, suggesting that proverb recognition among options is simpler than inferring which story instantiates a given proverb. Large commercial models perform near ceiling. Among open-source models, Qwen stands out, with clear gains as size increases. Many models below 7B parameters cluster near random chance, approximately 20%, which the paper interprets as evidence of capacity thresholds and of the importance of training data scale and quality.

These results give BRoverbs a dual role. It functions as a benchmark for ranking systems, but it also serves as a diagnostic for asymmetries between proverb recognition and proverb interpretation. Large gaps between StP and PtS indicate that surface familiarity with proverb strings can outpace robust narrative-to-meaning mapping.

5. Error modes and linguistic implications

The error analysis isolates several recurring failure modes. A primary failure mode is literal versus figurative confusion: models that perform near chance often default to literal interpretations or fail to infer the implicit moral of the story, especially in PtS where the story is indirect. A second source of error is near-synonym proverb confusion, where distractors are semantically close, as in “Quem com ferro fere, com ferro será ferido” versus “Tudo que vai, volta”, producing mis-selections even after manual curation intended to reduce multiple correct answers (Almeida et al., 10 Sep 2025).

A third error type is pragmatic alignment error. Models may misread the consequence or lesson in the story, confusing the intended moral with tangential details, for example focusing on a character’s setting rather than the causal lesson. The benchmark also exposes difficulties with regional variants and idiomaticity: when fixed expressions have conventional meanings, smaller or less-exposed models fail to retrieve the conventionalized mapping. This helps explain why StP is easier: proverb strings are distinctive and more recognizable than indirect story paraphrases. Finally, syntactic ambiguities in complex sentences or stories with implicit causality can cause weaker models to select proverb options tied to surface words instead of deeper situational meaning.

The broader linguistic interpretation advanced by the paper is that figurative language remains nontrivial even for strong models. The PtS gap reveals that deriving a proverb’s implicit meaning from a narrative is harder than recognizing a proverb string. Large models show strong grasp of Brazilian Portuguese idioms, approaching native-level recognition, while the Qwen scaling pattern and the poor performance of many small models suggest that capacity and exposure matter for emergent figurative understanding. A plausible implication is that proverb comprehension in this benchmark depends on a conjunction of lexical familiarity, pragmatic inference, and culturally situated semantic mapping rather than on simple pattern matching alone.

6. Limitations, access, and projected extensions

BRoverbs is explicitly presented as a first version, and several limitations are identified. Coverage is limited to 193 proverbs and is not exhaustive across Brazil’s regions. There are no explicit per-region or dialect annotations, and topic or theme metadata are not reported. Because stories were generated with GPT-4 and then human-verified, the initial generation process could bias style or content. The selection of web sources, together with the lack of explicit regional markers, may skew the dataset toward nationally popular sayings rather than region-specific ones. Generalization is evaluated only in Brazilian Portuguese, not in European Portuguese or other Lusophone varieties (Almeida et al., 10 Sep 2025).

The paper also raises an ethical caution. As a culturally grounded benchmark, BRoverbs should be used carefully so that model behavior is not interpreted in ways that reinforce stereotypes; the stated emphasis is inclusive evaluation rather than prescriptive cultural judgment. This positions the benchmark as an instrument for measuring underrepresentation and model limitations rather than for essentializing cultural competence.

In practical terms, the benchmark is publicly available at https://huggingface.co/datasets/Tropic-AI/BRoverbs. The license is not specified in the paper, so the dataset card is the authoritative place to consult current licensing terms. Splits are not detailed; the benchmark appears as a single release. Evaluation scripts and a leaderboard are not reported, although accuracy computation is straightforward by comparing the model’s selected option to the gold label. The paper estimates the open-source evaluation workload as less than 5 hours on an RTX A6000, with a carbon footprint of approximately 0.85 kg CO₂ via the ML CO₂ Impact calculator.

The proposed extensions are concrete. The paper suggests increasing task difficulty by using more than five options, richer distractor generation, and adversarial paraphrases. It proposes adding metadata such as themes or domains, literal versus figurative markers, regional or dialectal tags, usage contexts, and frequency. It also points to cross-lingual extensions, including combination with ProverbEval and evaluation across Portuguese varieties, as well as corpus audits on resources such as Gigaverbo and ClueWeb22 Portuguese to relate training exposure to proverb performance. Additional directions include evaluating more model families and sizes, exploring finetuning or continued pretraining on idiom or proverb corpora, and adding multimodal or temporal updates through images, videos, or contemporary proverb usage.