Inside–Outside Algorithm Overview

• Inside–Outside algorithm is a dynamic programming method that computes inside (content) and outside (context) vector representations for each span in a sequence.
• It uses bottom-up and top-down passes along with neural composition functions to score binary splits and induce latent tree structures efficiently.
• The algorithm’s parallelizable dynamic programming framework underpins neural models like DIORA, achieving competitive results in unsupervised syntactic parsing tasks.

The inside–outside algorithm is a dynamic programming technique central to probabilistic and neural approaches for latent tree induction, structure prediction, and unsupervised syntactic parsing. It computes, for every subsequence (“span”) of an input sequence, two vector-valued representations: an “inside” representation capturing the content of the span itself, and an “outside” representation capturing the remainder of the sequence excluding that span. Classical forms arise in parsing algorithms for probabilistic context-free grammars. Recent neural instantiations, such as the deep inside–outside recursive autoencoder (DIORA) framework, generalize the principle by incorporating highly parameterized composition functions, enabling unsupervised induction of constituent structure while simultaneously learning span representations in a differentiable manner (Drozdov et al., 2019).

1. Vector-Valued Inside and Outside Representations

Given an input sequence $x_0,\dots,x_{T-1}$ with learned or pre-trained embeddings $v_i\in\mathbb R^d$, the inside–outside algorithm works over chart cells $(i,j)$ representing spans $x_i,\dots,x_{j-1}$ ($0\leq i<j\leq T$). For each chart cell, the following are computed:

• Inside representation $I(i,j)=\bar a(i,j)\in\mathbb R^h$: encodes the content of $x_i,\dots,x_{j-1}$.
• Outside representation $O(i,j)=\bar b(i,j)\in\mathbb R^h$: encodes all content outside $x_i,\dots,x_{j-1}$ in the sequence.
• Scalar compatibility scores $\hat e(i,j)$ and $v_i\in\mathbb R^d$0 for inside and outside, respectively.

These representations form the basis for soft-structured learning of trees and for modeling token-level conditional distributions.

2. Dynamic Programming for Inside and Outside Passes

The computation proceeds via two passes:

2.1. Inside Pass (Bottom-Up)

• For length-1 spans: $v_i\in\mathbb R^d$1, $v_i\in\mathbb R^d$2, typically using a one-layer neural network.
• For larger spans $v_i\in\mathbb R^d$3, all binary splits ($v_i\in\mathbb R^d$4) are considered, applying a composition function (e.g., a two-layer MLP or TreeLSTM) to merge subspan representations.
• The score for a split $v_i\in\mathbb R^d$5 is given by

$v_i\in\mathbb R^d$6

where $v_i\in\mathbb R^d$7 is a learned bilinear form and $v_i\in\mathbb R^d$8 contains trainable parameters.

• Normalized weights $v_i\in\mathbb R^d$9 are obtained by softmax over splits.
• The inside representation $(i,j)$0 and expected score $(i,j)$1 are weighted sums over splits.

2.2. Outside Pass (Top-Down)

• The root outside vector $(i,j)$2 is initialized as a learned vector; $(i,j)$3.
• For each non-root span $(i,j)$4, each context pair $(i,j)$5 (where $(i,j)$6 is one child of parent span $(i,j)$7 and $(i,j)$8 is the sibling span) is considered.
• The context score is

$(i,j)$9

followed by a softmax normalization to obtain weights $x_i,\dots,x_{j-1}$0 over valid parent–sibling configurations.

• The outside representation $x_i,\dots,x_{j-1}$1 and score $x_i,\dots,x_{j-1}$2 are computed as expected values over contexts using a composition function.

This framework allows parallel and differentiable computation of all span and context representations.

3. Span-Conditional Training Objectives

After both passes, each terminal cell $x_i,\dots,x_{j-1}$3 has associated $x_i,\dots,x_{j-1}$4 and $x_i,\dots,x_{j-1}$5 vectors. DIORA adopts a masked language modeling loss in which, for every position $x_i,\dots,x_{j-1}$6, the model is trained to distinguish the true token $x_i,\dots,x_{j-1}$7 from sampled negatives $x_i,\dots,x_{j-1}$8, using the dot product of inside and outside vectors:

$x_i,\dots,x_{j-1}$9

An equivalent probabilistic formulation applies a softmax to yield the conditional probability of observing $0\leq i<j\leq T$0 given the outside representation, enabling unsupervised induction of tree structure (Drozdov et al., 2019).

4. CKY-Style Decoding for Tree Recovery

At test time, the algorithm extracts the single highest-scoring binary tree over the sequence, with split scores given by $0\leq i<j\leq T$1. A CKY-style Viterbi dynamic program is employed:

• For every cell $0\leq i<j\leq T$2 and possible split $0\leq i<j\leq T$3,

$0\leq i<j\leq T$4

and the split yielding the maximum is recorded.

• Backtracking from the root yields the optimal binary parse tree.

This method ensures exact or near-gold parses for sentences in unsupervised settings, leveraging the learned composition parameters (Drozdov et al., 2019).

5. Computational Complexity and Parallelization

Critical computational bottlenecks are addressed by efficiently structuring the DP recurrences:

• Chart filling for inside and outside passes over $0\leq i<j\leq T$5 tokens requires $0\leq i<j\leq T$6 serial steps but is highly parallelizable. All “soft” split operations can be batched—permitting $0\leq i<j\leq T$7 wall-clock time with modern GPU hardware.
• The CKY decoding phase also scales as $0\leq i<j\leq T$8 in serial but can achieve $0\leq i<j\leq T$9 parallel time.
• The masked LM loss is $I(i,j)=\bar a(i,j)\in\mathbb R^h$0 for $I(i,j)=\bar a(i,j)\in\mathbb R^h$1 negatives and can be parallelized over batch dimensions.
• Empirically, DIORA with an 800-dimensional composition MLP handles sentences of length up to $I(i,j)=\bar a(i,j)\in\mathbb R^h$2–$I(i,j)=\bar a(i,j)\in\mathbb R^h$3, with full training completed in several days on standard GPUs.

This parallelism is essential for scaling latent tree induction in neural architectures (Drozdov et al., 2019).

6. Significance and Extensions

The inside–outside algorithm, instantiated as in DIORA, provides a foundation for unsupervised syntax discovery and constituent representation learning. By jointly learning to compose, score, and reconstruct all possible tree structures, it can outperform prior approaches in F1 constituency parsing metrics on standard benchmarks such as WSJ and MultiNLI. The approach’s capacity to consider all binary trees and its compatibility with end-to-end neural training distinguish it from earlier non-neural or non-differentiable frameworks.

A plausible implication is that such architectures may generalize to any structured prediction task where latent tree or segmentation structure is relevant, provided suitable composition and scoring functions can be defined. The dynamic programming structure underlying the approach is essential for tractably considering the exponential space of possible trees, demonstrating the durability and significance of the inside–outside algorithm in modern deep learning (Drozdov et al., 2019).