---
title: Constituency Parse Extraction from PLMs
url: https://www.emergentmind.com/topics/constituency-parse-extraction-from-pre-trained-language-models-cpe-plm
type: topic
---

# Constituency Parse Extraction from PLMs

Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) refers to the set of methodologies that induce full constituency parse trees directly from the internal representations of frozen large language models without the need for explicit supervised parser training. These methods are typically parameter-free at extraction time or rely on minimal probing layers atop contextual encoders. CPE-PLM leverages features such as self-attention heads, contextual embeddings, and representation perturbation signals to construct binary constituency trees, often achieving performance competitive with unsupervised probabilistic parsers and robust enough for application across multiple languages and grammars.

## 1. Formal Definitions and Core Encodings

The CPE-PLM paradigm encompasses two main families: sequence-labeling probes and chart-based extraction from span or distance scores.

- **Sequence Labeling Encodings**: A bijective mapping is defined between a constituency tree $T$ and a sequence of local labels $\ell_i = (n_i, C_i, u_i)$ for each position $i$, where $n_i$ is the change in shared ancestor depth across word boundaries, $C_i$ is the lowest common ancestor nonterminal, and $u_i$ is a unary chain encoding [2002.01685, 2309.11165]. Such encodings are lossless and reconstruct the tree exactly in $O(n)$ time.
  
- **Span/distance-based chart extraction**: For a sentence of $n$ tokens, all $O(n^2)$ contiguous spans $(i,j)$ are assigned scalar scores $s(i,j)$ measuring the likelihood of constituenthood. These scores derive from pairwise attention-head distances (using metrics like Jensen–Shannon or Hellinger divergence between distributions), average token-level distortions under syntactic perturbations, or cross-span compositionality measures [2002.00737, 2004.13805, 2306.00645]. The resulting score matrix is used to extract the binary parse tree minimizing total span cost via CKY or top-down recursive algorithms.

## 2. Extraction Algorithms: Chart, Greedy, and Pointing

- **Chart-based CKY decoding**: The dominant algorithm is bottom-up dynamic programming: for each span $(i,j)$,
  $$ 
  s_{\text{span}}(i,j) = s_{\text{comp}}(i,j) + \min_{i \le k < j} [s_{\text{span}}(i,k) + s_{\text{span}}(k+1,j)]
  $$
  The full tree $T^*$ is the argument minimizing $\sum_{(i,j)\in T} s_{\text{span}}(i,j)$ [2004.13805, 2211.00479, 2306.00645]. Fixed-length normalization and multi-head ensemble averaging are used to stabilize extraction.

- **Top-K and ensemble selection**: Instead of relying on a single attention head, ensemble extraction averages the span scores from the best $K$ heads chosen by dev-set $F_1$ or by unsupervised ranking criteria ("Heads-up!" dynamic K) [2211.00479, 2010.09517]. Greedy and beam search algorithms select maximally consistent sets of heads to form robust ensemble parses.

- **Pointing-based and fast decoders**: Some architectures reduce parsing to a sequence of pointing or boundary-selection tasks using local cross-entropy losses, yielding competitive $O(n^2)$ greedy decoders [2006.13557].

- **Contextual distortion extraction**: For masked LMs, syntactic constituency can be detected as spans that minimally distort contextual representations under perturbations (substitution, decontextualization, movement), and a CKY variant parses the normalized distortion matrix [2306.00645].

## 3. Empirical Evaluation and Performance

### English Constituency Parsing (PTB, F₁)

| Method                    | Model(s)                | Dev-tuned? | F₁ (%) | Reference      |
|---------------------------|-------------------------|------------|--------|---------------|
| Sequence labeling (ff-ft) | BERT-base               | Yes        | 93.5   | [2002.01685]  |
| Chart-based Top20 Ensemble| RoBERTa/XLNet           | Yes        | 46.4   | [2004.13805]  |
| Chart-based Greedy Beam   | 16 PLMs                 | Yes        | 55.7   | [2211.00479]  |
| Contextual distortion     | BERT-large, RoBERTa     | No         | 48.8   | [2306.00645]  |
| Pointing (fine-tuned)     | BERT-large              | Yes        | 95.48  | [2006.13557]  |
| Heads-up!                 | XLNet-base, RoBERTa     | No         | 42.7   | [2010.09517]  |

Unsupervised CPE-PLM chart ensembles match the F₁ of neural PCFGs and in some configurations outperform tuned approaches on small categories and adverbial phrases [2211.00479, 2002.00737]. The best parameter-free methods reach ~46–56 F₁; probe/fine-tuned architectures approach supervised SOTA (≥95 F₁).

### Multilingual Performance and Transfer

CPE-PLM methods generalize robustly to morphologically rich languages (SPMRL datasets) and low-resource settings, matching or exceeding traditional PCFGs in 8/9 languages [2004.13805, 2211.00479]. Zero-shot ensembles use English-tuned heads for cross-lingual parsing with $<$2 F₁ drop. Multilingual PLMs (mBERT, XLM-R) sustain high bracket recall even on small probes if the language is present in the pretraining corpus [2309.11165, 2010.09623].

## 4. Probing, Interpretability, and Implied Syntactic Biases

- **Probing analyses** demonstrate that a single linear classifier on last-layer LM vectors suffices to recover tree brackets and constituent categories (depth offsets, LCA labels, chunk tag boundaries) [2204.06201, 2002.01685]. Linear separability is high (82–95% bracket F₁), and syntactic signals are distributed across middle layers.

- **Attention head structure**: Certain heads cluster words into constituents (horizontal blocks in heatmaps), and middle-to-upper Transformer layers recurrently host universal phrase detectors [2004.13805]. Attention-distance-based scores are more informative than hidden vector metrics for zero-shot parsing [2002.00737].

- **Contextual distortion**: Low average distortion under perturbation identifies true constituents, with movement-based perturbations most critical for SBAR/PP/ADVP recall [2306.00645].

- **Sequence labeling vs. chart methods**: Both paradigms are viable; no intrinsic formalism bias is observed in LM representations (constituency and dependency structures are recoverable to similar degrees) [2309.11165].

## 5. Biases, Limitations, and Control Experiments

Branching bias—systematic preference for certain tree shapes (right-branching, head-initial)—is a recurring concern. Margin-based chart parsers ("Mart") and prefix-only attention feature definitions induce strong right-branching bias, significantly inflating F₁ on English and suppressing scores in reversed-language controls [2010.02448]. Distance-based (Dist) parsers and symmetric feature scores (full attention or hidden-vector distances) remain unbiased, and reversed-corpus gap $\Delta_{\text{bias}}$ provides an algorithm-agnostic metric of structural directionality. Explicit right-branching bias injection boosts SBAR/VP recall by up to 10 pts if tuned but should be set to zero for interpretable probing [2002.00737].

Mitigation strategies include symmetric chart decoders, feature selection ablations, and universal reversed-language benchmarking [2010.02448, 2211.00479].

## 6. Practical Applications and Use Cases

CPE-PLM trees serve as high-quality pseudo-annotations for downstream models:
- **Bootstrapping unsupervised RNNGs**: Trees induced by CPE-PLM allow training RNNGs/URNNGs to achieve strong parsing and lower language model perplexity [2211.00479].
- **Tree-LSTM text classification**: Classifiers using CPE-PLM trees nearly reach supervised parse accuracy [2211.00479].
- **Silver parser distillation**: Fast supervised parsers trained on CPE-PLM outputs match or exceed their unlabeled F₁ in under 1% of the inference time.
- **Few-shot regimes**: CPE-PLM needs only 1% of dev data for ensemble tuning to outperform vanilla few-shot supervised parsers by $>$30 F₁ [2211.00479].

## 7. Methodological Variants and Future Directions

- **LLM-based parsing**: Modern LLMs (ChatGPT, GPT-4, LLaMA, OPT) can predict linearized bracketed trees using prompt-based or fine-tuned strategies. Zero-shot and 5-shot in-context learning yield limited valid parse accuracy; fine-tuning brings LLM parses to SOTA in-domain F₁, but hallucination errors, domain shift, and invalid-tree rates remain challenging [2310.19462]. Constraint decoding and grammar-aware beam search are recommendations for robust deployment.

- **Language, tokenization, and resource effects**: Language presence in LM pretraining is a stronger determinant of recoverable syntax than labeled treebank size [2309.11165]. Subword tokenization is preferable to character-based models for reliable structure extraction.

- **Open problems**: Extensions to dependency parsing via perturbation, autoregressive LM adaption, and hybrid self-training approaches are promising future avenues [2306.00645, 2002.00737]. Full theoretical characterizations of why MLM objectives encode Inside-Outside marginal probabilities in PCFGs remain active research [2303.08117].

## References

- Sequence-labeling probes: [2002.01685], [2309.11165], [2204.06201]
- Chart-based/attention distance extractors: [2002.00737], [2004.13805], [2010.09517], [2306.00645]
- Ensemble extraction: [2211.00479], [2004.13805], [2010.09517]
- Contextual distortion/perturbation: [2306.00645]
- Pointing decoders: [2006.13557]
- Multilingual and low-resource parsing: [2004.13805], [2010.09623], [2309.11165]
- Branching bias analysis: [2010.02448], [2002.00737]
- LLM-based parsing and prompt methods: [2310.19462]

Source: https://www.emergentmind.com/topics/constituency-parse-extraction-from-pre-trained-language-models-cpe-plm