Distribution-Based Semantics Overview

• Distribution-based semantics is a family of approaches that represent meaning via structured statistical distributions derived from linguistic contexts.
• It employs vector-space models, tensor operations, and reweighting schemes (e.g., PMI) to capture lexical relationships and compositional structure.
• It bridges corpus-driven methods with formal semantics by integrating probabilistic logic and scalable language representation frameworks.

Distribution-based semantics denotes a family of semantic frameworks in which meaning, or more generally interpretable content, is represented through distributions rather than by direct symbol-to-world mappings alone. In computational linguistics, the dominant sense is distributional semantics: lexical items are represented by patterns of contextual co-occurrence under the distributional hypothesis that words occurring in similar contexts tend to have similar meanings. In logic and probabilistic programming, by contrast, “distribution semantics” denotes a semantics in which a free random component induces a probability measure over worlds, interpretations, or reductions, often followed by a deterministic construction. The shared motif is representation by structured distributions, but the mathematical objects and explanatory goals differ substantially (Kartsaklis, 2014, Boleda, 2019, Weitkämper, 2022).

1. Lexical meaning as contextual distribution

In the vector-space tradition, a vocabulary $V$ and a set of context features $C$ are fixed, and each word $w$ is represented by its co-occurrence profile with those features. A classical count-based model builds a matrix $M \in \mathbb{R}^{|V|\times |C|}$ with entries $M_{w,c}$ equal to the number of times $w$ appears in context $c$. Equivalently, one may write a word as a vector

$w=\sum_i c_i n_i \in \mathbb{R}^d,$

where $\{n_i\}$ is an orthonormal basis and $c_i$ measures co-occurrence with feature $C$0. Contexts may be defined by sliding windows, dependency relations, or documents; dependency-based schemes also record structured features such as “subject-of-buy” or “object-of-see.” One dependency-path formulation collects all paths $C$1 of bounded length from $C$2, weights each path by $C$3, and aggregates end-word contributions accordingly (Kartsaklis, 2014).

This representation implements the distributional hypothesis in a directly statistical form. The representation is lexical rather than truth-conditional: it abstracts over use in corpora rather than mapping expressions to denotations in a model. Typical modern systems transform and compress $C$4 to obtain dense embeddings, often by weighting and truncated SVD, while neural predictive models such as word2vec and GloVe directly learn dense vectors and are formally related to matrix factorization. The review literature also emphasizes that these representations are multi-dimensional, graded, and empirically induced, which is central to their relevance for semantic change, polysemy, and the syntax-semantics interface (Boleda, 2019).

2. Geometry, weighting, and semantic variation

Raw co-occurrence counts are usually reweighted to mitigate frequency bias. The literature represented here explicitly includes TF-IDF, PMI, PPMI, and log-likelihood $C$5. For PMI,

$C$6

and in one equivalent formulation,

$C$7

Once vectors are built, similarity is commonly measured by cosine,

$C$8

while Euclidean distance remains a standard alternative (Kartsaklis, 2014, Boleda, 2019).

Because the geometry is induced from observed usage, distributional semantics is naturally suited to graded similarity, near-synonymy, and selectional preference. The same geometric machinery has also been extended to lexical ambiguity and semantic change. Multi-prototype models cluster context vectors into sense-specific centroids; contextualized embeddings compute a context-sensitive vector $C$9 for each token occurrence; diachronic models build time-specific embeddings $w$0, align them across periods, and measure change by cosine shift or nearest-neighbor change. The literature further records evaluation practices such as Spearman’s $w$1, AUC, and classification accuracy on changed-versus-stable lexical items (Boleda, 2019).

A recurrent misconception is that the geometric nature of these models makes them merely associative. That criticism is too coarse. The models are indeed sensitive to topical relatedness and corpus bias, but they also support structured notions of plausibility and context refinement. Prototype-based selectional preference, for example, forms role-specific centroids for verb arguments and scores candidate fillers by cosine with the appropriate prototype, thereby linking vector geometry to predicate-argument structure (Boleda, 2019).

3. Compositional operators and syntactic mediation

The central technical problem beyond lexical representation is composition: how to construct phrase and sentence meanings from word meanings. The simplest operators keep all representations in the same space $w$2. For two word vectors $w$3, the additive model forms $w$4, typically with $w$5; the multiplicative model uses the componentwise product $w$6; and circular convolution defines a third commutative composition. These operators are strong baselines, but because they are commutative they ignore word order and grammatical role (Kartsaklis, 2014).

More linguistically informed approaches treat relational words as multilinear maps. Baroni and Zamparelli model adjectives as matrices $w$7 acting on noun vectors via $w$8, where the adjective matrix is learned by regression on corpus-derived compound vectors. Verbs may be modeled as higher-order tensors: a transitive verb is a bilinear map $w$9, and a sentence vector is obtained by tensor contraction,

$M \in \mathbb{R}^{|V|\times |C|}$0

The Coecke-Sadrzadeh-Clark framework supplies a categorical justification for this architecture: pregroup grammar and finite-dimensional vector spaces are both compact closed categories, and a strong monoidal functor maps grammatical reductions to linear maps, so grammatical type reduction becomes tensor contraction in vector space (Kartsaklis, 2014).

Subsequent work broadened the inventory of syntactically mediated composition. The APT framework represents lexemes by anchored packed dependency trees whose features are dependency-typed paths $M \in \mathbb{R}^{|V|\times |C|}$1, and composes them by offsetting anchors and merging aligned structures. This allows composition to operate over full sentential dependency contexts rather than only over local windows; in the reported formulation, $M \in \mathbb{R}^{|V|\times |C|}$2 implements strict mutual compatibility, whereas $M \in \mathbb{R}^{|V|\times |C|}$3 supports union-like enrichment of compatible contexts (Weir et al., 2016). A related line connects additive vector composition to Dependency-based Compositional Semantics: intersection is mapped to addition, projection to multiplication by a matrix $M \in \mathbb{R}^{|V|\times |C|}$4, and inverse projection to multiplication by $M \in \mathbb{R}^{|V|\times |C|}$5, yielding vector compositions that parallel DCS tree constructors (Tian et al., 2016).

Compositional distributional semantics has also been extended beyond the sentence. In categorical compositional distributional discourse analysis, a basic anaphoric discourse is represented as a tensorial effect with open entity wires, a DRS-style mid-level representation, and a corresponding RDF basic graph pattern. Composition then connects word-to-sentence meaning construction with discourse-level operations such as anaphora resolution and question answering (Coecke et al., 2018).

4. Interfaces with formal semantics, logic, and model theory

A major research theme is the relation between distributional semantics and formal semantics. One line proves direct structural compatibility. In the extensional case, an extensional model with primitive types $M \in \mathbb{R}^{|V|\times |C|}$6 and $M \in \mathbb{R}^{|V|\times |C|}$7 is embedded into a vector-space model by an injective mapping $M \in \mathbb{R}^{|V|\times |C|}$8 from entities to basis vectors and $M \in \mathbb{R}^{|V|\times |C|}$9 from truth values to two orthonormal vectors $M_{w,c}$0. Functions become multilinear maps on tensor products, predicates become tensors $M_{w,c}$1, and logical connectives such as negation and conjunction are represented by matrices or tensors. The result is a homomorphism: every semantic function in the extensional model corresponds to a compatible vector-space operation, and composition is preserved (Quigley, 2024).

The same program has been extended to intensional semantics. Kripke-style models with multiple index sorts—such as worlds, times, and locations—are embedded into vector spaces $M_{w,c}$2, and semantic functions lift to linear or multilinear maps. Propositions of type $M_{w,c}$3 become operators $M_{w,c}$4, accessibility relations induce linear operators on the compound index space, and modal conditions reduce to threshold checks after linear accumulation. For uncountable index domains, sums are replaced by integrals; necessity becomes truth almost everywhere, and possibility becomes truth on a set of positive measure (Quigley, 3 Feb 2026).

A different route incorporates distributionality on the level of formal models rather than by directly vectorizing existing denotations. Distributional Formal Semantics samples a finite set of first-order models $M_{w,c}$5 and represents a proposition $M_{w,c}$6 by a binary vector $M_{w,c}$7, with $M_{w,c}$8 iff $M_{w,c}$9. Sub-propositional expressions are represented by real-valued vectors in $w$0; Boolean operations are lifted componentwise, quantifiers are computed by product, min, max, or their De Morgan duals, and the probabilistic interpretation of vectors yields notions of surprisal, entropy, and inference scores. The same framework also describes incremental semantic construction with a Simple Recurrent Network whose intermediate states are themselves sub-propositional meaning vectors (Venhuizen et al., 2021).

Probabilistic model-theoretic hybrids place contextual meaning in posterior distributions rather than in single vectors. Functional Distributional Semantics distinguishes latent entities in a semantic space from semantic functions $w$1 attached to predicates. In the associated graphical model, observed predicates in a DMRS graph induce a posterior over latent “pixies,” and phrase or sentence meaning is the joint posterior conditioned on the observed truth-value variables. Emerson and Copestake explicitly interpret this as a probabilistic version of model theory and use conditional probabilities in the graphical model to recast semantic composition, context dependence, and graded inference (Emerson et al., 2016, Emerson et al., 2017).

Theoretical disagreement persists over what exactly such systems are expected to model. Westera and Boleda argue that standard objections—reference, truth conditions, entailment, and full compositionality—rest on conflating expression meaning with speaker meaning. On that view, a mapping $w$2 can be fully adequate for expression meaning even if truth conditions and referential intentions are only fixed later, at the speaker-meaning level (Westera et al., 2019). This suggests that debates over the adequacy of distributional semantics often turn as much on semantic architecture as on representational form.

5. Distribution semantics in probabilistic logic and computation

Outside computational linguistics, “distribution semantics” has a distinct technical meaning. In generalized distribution semantics, a relational signature is split into probabilistic predicates and intensional predicates, a sample space $w$3 is built from ground probabilistic facts with independent Bernoulli weights, and each sample $w$4 determines a deterministic structure via a logic program or, ավելի abstractly, a family of expansion maps $w$5. The resulting law on interpretations is a push-forward measure. For projective families of distributions, representability is characterized exactly by the Strong Independence Principle and the absence of essential asymmetry; moreover, every representable family can be realized by an acyclic determinate logic program over auxiliary predicates (Weitkämper, 2022).

The classical probabilistic-logic-programming version of this idea is formulated in terms of worlds, selections, and explanations. In Logic Programs with Annotated Disjunctions, a selection picks one head for each ground clause, thereby determining a world $w$6, and the probability of a query $w$7 is obtained by summing the probabilities of worlds in which $w$8 is true under the Well-Founded Semantics. When function symbols are present, well-definedness depends on finite explanations and finite covering sets. Riguzzi and Swift identify bounded term-size programs and queries as a class for which finite, mutually incompatible, covering explanation sets exist, and they operationalize inference with PITA, a transformation to normal logic programming combined with tabling, answer subsumption, and BDDs (Riguzzi et al., 2011).

A closely related construction has been developed for probabilistic term rewriting. There, each probabilistic rewrite rule contributes an atomic choice, a selection induces a world $w$9 consisting of one chosen branch per probabilistic rule plus the regular rules, and the reachability probability $c$0 is defined by summing over the worlds or derivations that reduce $c$1 to $c$2. To avoid explicit summation over all worlds, one collects covering sets of explanations—partial assignments of rule choices—and computes their measure, for example via BDDs or MDDs (Vidal, 2024).

Distribution-based semantics also appears in probabilistic team semantics, where the semantic object is a probabilistic team $c$3 assigning weights to variable assignments. The key distribution-based atoms are marginal identity $c$4, marginal distribution equivalence $c$5, and probabilistic conditional independence $c$6. The associated expressive-power results show that first-order logic with marginal identity alone is strictly weaker than systems enriched by dependence or marginal-distribution-equivalence atoms, while the latter reach the expressive power of probabilistic independence logic (Hannula et al., 2018).

Recent categorical work abstracts these probabilistic semantics even further. In NeSyCat, ULLER’s classical, fuzzy, and probabilistic semantics are recast uniformly in monadic terms; the probabilistic case is modeled by the Giry monad, computational predicates and functions live in its Kleisli category, and quantifiers over arbitrary measurable domains are handled by measure-theoretic aggregators (Schellhorn et al., 27 Apr 2026). This is not a continuation of lexical distributional semantics, but it illustrates how the language of distributions has become a general semantic resource across logic, category theory, and probabilistic programming.

6. Evaluation, limitations, and open problems

The limitations of vector-space distributional semantics are well documented. Static embeddings are context-insensitive; low-frequency words, rare senses, and low-resource languages are under-represented; function words and grammatical phenomena are difficult to model without additional structure; and biases in source corpora propagate directly into embeddings. At the compositional level, tensor models face acute scalability problems because tensor order and parameter count grow rapidly with argument structure, while large parameter spaces intensify data sparsity (Boleda, 2019, Kartsaklis, 2014).

Open problems in compositional distributional semantics remain fundamental rather than merely engineering-oriented. One concerns the validity of the distributional hypothesis at the phrasal level: even with 2.8 billion tokens, composed noun-phrase vectors may fail to rank the corresponding observed compound vectors highly, raising the possibility that the limitation is not only data sparsity but also the linearity assumptions or the representational objective itself. Other unresolved issues include how to model quantifiers, prepositions, determiners, negation, conjunction, and disjunction; how to design sentence spaces $c$7 with interpretable or task-relevant dimensions; and how to evaluate compositional models by benchmarks that test whether they reconstruct genuinely observed phrasal contexts rather than merely achieving downstream accuracy on paraphrase retrieval or textual entailment (Kartsaklis, 2014).

The hybrid and model-theoretic literature adds a second layer of unresolved questions. Injective embeddings from extensional or intensional semantics into vector spaces establish structural compatibility, but they do not by themselves explain how corpus-trained embeddings should be aligned with the distinguished basis vectors required by the proofs. Probabilistic and graphical-model approaches offer context dependence and graded inference, but often at the cost of approximate inference, structured parsing requirements, or expensive posterior computation (Quigley, 2024, Quigley, 3 Feb 2026, Emerson et al., 2016). This suggests that the deepest unresolved issue is not whether distribution-based semantics is possible, but which semantic tasks should be assigned to geometric representation, which to logical structure, and which to probabilistic conditioning.

Taken together, the literature presents distribution-based semantics not as a single doctrine but as a family of mathematically explicit strategies for representing meaning, truth, or uncertainty through distributions. In lexical semantics it begins with corpus co-occurrence and extends through algebraic, tensorial, and categorical composition. In formal and probabilistic semantics it becomes a mechanism for inducing measures over interpretations, structures, or worlds. The continuing significance of the field lies in that dual development: increasingly rich geometric models of linguistic usage, and increasingly precise accounts of how such models interface with grammar, logic, and probabilistic inference.