Blackbird Language Matrices

• Blackbird Language Matrices are linguistic matrix-completion problems that require inferring missing sentence elements based on structured grammatical rules.
• They utilize multi-level organization—from sentence-internal syntax to matrix-level relational operations—to rigorously probe compositionality and rule abstraction in language models.
• Empirical studies reveal strong within-language performance paired with poor cross-lingual transfer, highlighting both technological advances and current limitations in model generalization.

Blackbird Language Matrices (BLMs) are Raven-inspired linguistic matrix-completion problems in which a system must infer the missing element of a structured sentence sequence from a context set and an answer set of minimally contrastive candidates. They were introduced as a language-native benchmark for generalisation, abstraction, compositionality, and disentangled representation learning, replacing visual objects and attributes with linguistic objects such as noun phrases, verbal arguments, attractors, and paradigm cells, and replacing visual relations with grammatical and paradigmatic rules such as alternation and progression (Merlo et al., 2022, Merlo, 2023, Merlo et al., 24 Feb 2026).

1. Origins and conceptual rationale

The initial BLM benchmark was introduced in 2022 as a linguistic variant of Raven’s Progressive Matrices centered on subject–verb agreement in French. In that formulation, a model receives seven context sentences and must choose the correct eighth sentence from six candidates. The original dataset contains 44,800 BLMs and was designed to investigate generalisation under controlled grammatical variation, including agreement attraction and lexical variation (Merlo et al., 2022).

The 2023 formal-specification paper repositioned BLMs as a task for fine-tuning rule-like generalization in LLMs. Its explicit motivation was that human solvers of Raven-style tests identify relevant objects, identify their relevant attributes, and infer rules over those objects and attributes; BLMs were proposed as a linguistic analogue of that process (Merlo, 2023). The 2026 framework paper generalized the proposal into a curated family of multilingual, multi-phenomenon datasets and described BLMs as “multiple-choice problems, structured at multiple levels: within each sentence, across the input sequence, within each candidate answer” (Merlo et al., 24 Feb 2026).

Across these formulations, the underlying rationale is stable. BLMs are not ordinary sentence-level acceptability tests. They are intended to probe whether a model can recover a structured linguistic system distributed across several related sentences, and whether errors arise from failures on linguistic structure, failures on sequence-level reasoning, or interactions between the two (Merlo et al., 24 Feb 2026, Samo et al., 16 Mar 2026).

2. Formal structure

The framework paper defines the BLM task by the instruction

$\text{find } (a_c \in A) \text{ given } C,$

given a 4-tuple $(LP, C, A, a_c)$, where $LP$ is the linguistic phenomenon, $C$ the context matrices, $A$ the answer set, and $a_c$ the correct item in $A$ (Merlo et al., 24 Feb 2026). A BLM problem is correspondingly written as $(LP, C, A, Aug)$, where $Aug$ specifies the augmentation method (Merlo et al., 24 Feb 2026, Merlo, 2023).

A BLM matrix is formalized as a tuple $(S,R,T)$, where $(LP, C, A, a_c)$0 is the shape of the matrix, $(LP, C, A, a_c)$1 the relational operators connecting its items, and $(LP, C, A, a_c)$2 the set of items. The linguistic phenomenon itself is specified by a grammar

$(LP, C, A, a_c)$3

with $(LP, C, A, a_c)$4 the set of objects, $(LP, C, A, a_c)$5 the set of attributes or features of those objects, $(LP, C, A, a_c)$6 the set of external observed rules, $(LP, C, A, a_c)$7 the set of unobserved internal rules, and $(LP, C, A, a_c)$8 the lexicon of objects, attributes, and operators (Merlo et al., 24 Feb 2026). The 2023 specification uses the same decomposition with attributes written as $(LP, C, A, a_c)$9 rather than $LP$0 (Merlo, 2023).

This formalism is designed to separate three sources of structure. First, there is sentence-internal structure, such as chunk sequence, grammatical number, semantic role, or voice. Second, there is matrix-level structure, implemented by relational operators such as alternation and progression. Third, there is answer-set structure, because distractors are generated so as to violate $LP$1 in $LP$2, $LP$3, or $LP$4 rather than being arbitrary negatives (Merlo et al., 24 Feb 2026, Merlo, 2023). In a later formulation for verb alternations, the authors state that “the relations operations $LP$5 transform a list of sentences into a predictable sequence” (Samo et al., 16 Mar 2026).

The framework also formalizes augmentation as controlled expansion of the item set while keeping matrix shape and relational structure fixed. One formulation states the constraints

$LP$6

and

$LP$7

capturing the requirement that augmented objects and items remain close to the originals under the augmentation criterion (Merlo et al., 24 Feb 2026).

3. Dataset construction and template regimes

BLM datasets are built through a strongly curated pipeline. The framework paper describes the workflow as: identify a linguistic phenomenon, define the BLM problem and template, create or source seed sentences, validate them by hand, and then augment them semi-automatically into larger datasets (Merlo et al., 24 Feb 2026). The original 2022 benchmark already used an attribute-value grammar together with matrix-level progression and alternation rules, and then increased lexical diversity semi-automatically with CamemBERT-based substitutions (Merlo et al., 2022).

A central design variable is lexicalization regime. In the original benchmark, Type I matrices reuse the same lexical items throughout the matrix, Type II matrices introduce partial lexical variation, and Type III matrices use full lexical variation by shuffling sentences instantiating the same template values into matrices (Merlo et al., 2022). The later framework paper retains the same Type I/II/III distinction across agreement and alternation datasets, while the Italian multitask study also uses Type I, II, and III to test increasing lexical novelty (Merlo et al., 24 Feb 2026, Nastase et al., 2024).

A parallel distinction appears in later alternation work as MinLex versus MaxLex. In MinLex, lexical material is largely reused across the matrix; in MaxLex, the same syntactic and semantic structure is preserved while lexical items differ across sentences, making the task unsolvable by lexical repetition alone (Samo et al., 16 Mar 2026). This MaxLex condition is especially important because it isolates structural abstraction from local lexical matching.

Another layer of design concerns the template family itself. A later extension introduces type A, type B, and type C templates. Type A combines the target phenomenon with an operator that is peripheral to that phenomenon, such as presence or absence of a temporal phrase. Type B uses linguistic diagnostics that expose internal argument structure, such as relative clauses. Type C combines both diagnostic structure and extra sequencing complexity, as in Hebrew binyan tasks that jointly vary diagnostics, tense, and PP material (Samo et al., 16 Mar 2026). These template families were explicitly designed to vary the degree to which a model can rely on superficial sequencing cues rather than abstract linguistic structure.

4. Linguistic coverage and multilingual expansion

BLMs were first instantiated for subject–verb agreement, but the framework has since expanded to several classes of formal and syntax-semantics phenomena. The 2026 overview lists seven paradigms: subject–verb agreement, causative alternation with Change-of-State verbs, causative alternation with Roll verbs, object-drop alternation, spray/load alternation, sequence of tense, and a mixture of sequence of tense and agreement (Merlo et al., 24 Feb 2026).

Agreement BLMs have been developed most extensively. The multilingual agreement study constructs parallel datasets for English, French, Italian, and Romanian using chunk templates of the form $LP$8, with no attractor, one attractor, or two attractors, and with grammatical number systematically manipulated across subject and attractor nouns. The sentence-level inventory covers 14 chunk patterns (Nastase et al., 2024). The original French-only benchmark likewise targets subject–verb agreement under attraction and distributes examples across main, completive, and relative clauses (Merlo et al., 2022).

Verb alternation BLMs extend the framework from formal agreement to paradigmatic syntax-semantics mapping. Change-of-state and object-drop datasets were built for English and Italian, with later paradigm-based datasets covering English, German, Italian, and Hebrew. These later resources distinguish change-of-state verbs such as break from object-drop verbs such as paint, and also include Hebrew binyanim as a paradigmatic morphological system involving Paal, Nifal, Hifil, and Hufal (Samo et al., 16 Mar 2026, Merlo et al., 24 Feb 2026).

The spray/load alternation and Roll-verb datasets target finer-grained argument-structure patterns. In the spray/load case, BLMs are organized around Agent, Theme, and Locative mappings and contrast alternants such as spray the paint onto the wall and spray the wall with the paint; the answer set includes errors on voice, prepositions, embedding, and syntax-semantics mapping (Merlo et al., 24 Feb 2026). The Roll-verb datasets similarly probe alternations within a narrowly defined Levin class (Merlo et al., 24 Feb 2026).

Morphological and voice-based extensions further broaden the framework. One study adapts BLMs to Turkish and Modern Hebrew voice paradigms, using four-cell systems: Active, Passive, Causative, and Causative-Passive in Turkish, and Paal, Nifal, Hifil, and Hufal in Hebrew (Samo et al., 5 Feb 2026). Another study instantiates BLMs for passive alternation in French and Italian using natural data from Universal Dependencies as well as synthetic data, with voice, number of overt arguments, and sentence type as the dimensions of the matrix (Samo et al., 26 Mar 2026).

5. Modeling strategies and empirical findings

The original modeling setup paired BLMs with a $LP$9-VAE / variational information bottleneck architecture over FlauBERT sentence embeddings. Each BLM input consisted of seven sentence vectors, and the model was trained to predict a representation close to the correct answer vector. In that setting, the best reported result was 84.8% accuracy in the easiest lexical regime, Type I, with $C$0. Performance decreased from Type I to Type III, larger $C$1 values performed worse than $C$2, and shuffling the sentence order reduced accuracy by about half (Merlo et al., 2022).

Subsequent work introduced simpler and more modular architectures. The framework paper evaluates a feed-forward neural network over concatenated sentence embeddings with a max-margin objective and a more tailored two-level VAE in which a sentence-level module compresses each sentence embedding and a task-level module solves the BLM itself. In chunking experiments, the sentence-level compression achieved high F1 for recovering structural patterns: 0.951 and 0.952 for spray/load, 0.960 for Italian Change-of-State, 0.975 for Italian object-drop, and 0.896 for Romanian agreement (Merlo et al., 24 Feb 2026). These results are presented as evidence that pretrained sentence embeddings contain information about chunk structure, grammatical number, active/passive distinctions, and role configurations relevant to BLM solving (Merlo et al., 24 Feb 2026).

The multilingual agreement study reports strong within-language performance but poor cross-lingual transfer. For monolingual training and testing on Type I BLMs, F1 scores were 0.884 for English, 0.948 for French, 0.845 for Italian, and 0.733 for Romanian. Cross-lingual transfer was generally very low, with the main exception being moderate French–Italian transfer, and multilingual training often underperformed monolingual training. The same paper concludes that syntactic information is not shared, even across closely related languages, in the sentence embeddings it evaluates (Nastase et al., 2024).

The Italian multitask study uses agreement, causative/inchoative alternation, and object-drop alternation to test whether a shared sentence-level compression can support several BLM tasks simultaneously. It reports that single-task learning generally outperforms multitask learning. For agreement under Italian Electra, Type I to Type I testing yields 0.909 in single-task training versus 0.772 in multitask training; for Type I to Type III, the scores are 0.707 versus 0.483. The same study interprets this gap as evidence that the clues needed by the different tasks are encoded differently in pretrained sentence embeddings rather than as a shared abstract representation of constituents or thematic roles (Nastase et al., 2024).

Later BLM studies also use the framework to compare data regimes and tokenization regimes. For French and Italian passive alternation, synthetic training and synthetic testing reached near-ceiling performance—1.00 in French and 0.99 in Italian—while synthetic-to-natural transfer dropped to 0.29 and 0.28, respectively. Natural-to-natural performance was 0.62 in French and 0.77 in Italian, and natural-to-synthetic transfer remained robust, with French multilingual NatSyn reaching 0.99 (Samo et al., 26 Mar 2026). For Turkish and Hebrew verbal paradigms, Turkish performance remained strong for both monolingual and multilingual models, but Hebrew showed a large split: the monolingual model achieved average F1 0.835 on natural-sentence BLMs, whereas the multilingual model achieved 0.333, with Mann–Whitney $C$3, $C$4, $C$5 (Samo et al., 5 Feb 2026).

6. Diagnostic role, error typology, and limitations

A defining property of BLMs is that they support structured error analysis rather than only final accuracy. In agreement datasets, answer options are organized into interpretable error types such as WNA, WN1, WN2, AEV, AEN1, and AEN2, separating sequence-continuation failures from agreement failures (Nastase et al., 2024, Merlo et al., 24 Feb 2026). In alternation datasets, the same principle appears in labels such as I-Int, ER-Pass, IER-Pass, R-Trans, and E-WrBy for causative and object-drop tasks, or Alt-NP, Alt-PP, NoEmb, and LexPrep for spray/load (Merlo et al., 24 Feb 2026). This organization allows BLMs to distinguish whether a model has recovered the grammatical phenomenon, the matrix progression, or neither.

Several recurring findings temper straightforward interpretations of good benchmark performance. First, BLMs are not textual transcriptions of visual RPMs; the framework paper explicitly characterizes them as genuine linguistic analogues built from linguistic paradigms themselves (Merlo et al., 24 Feb 2026). Second, high scores on synthetic data do not guarantee abstraction over natural language, as shown by the synthetic-to-natural collapse in passive-alternation experiments (Samo et al., 26 Mar 2026). Third, strong within-language results do not entail language-agnostic syntactic representations, since cross-lingual agreement transfer remains weak even for closely related languages (Nastase et al., 2024).

The framework papers also emphasize practical limitations. BLM design is expert-dependent and time-consuming, because templates are built by hand, seeds are manually validated, and augmentation is linguistically constrained rather than purely automatic (Merlo et al., 24 Feb 2026). Phenomenon coverage and typological coverage remain limited in several studies, and some evaluations are explicitly baseline-only or intended as diagnostic rather than definitive model comparisons (Samo et al., 16 Mar 2026, Samo et al., 26 Mar 2026). The Turkish–Hebrew morphology study adds that future work should expand language coverage, architectures, validation, and human upper bounds (Samo et al., 5 Feb 2026).

Taken together, these properties position BLMs less as a single benchmark than as a research framework for probing rule-like generalization in LLMs. Their distinctive contribution is to make multi-sentence linguistic structure explicit enough that grammatical competence, paradigmatic competence, and sequence-level reasoning can be studied in a single controlled formalism (Merlo, 2023, Merlo et al., 24 Feb 2026).