---
title: Submodular Conditional Mutual Information
url: https://www.emergentmind.com/topics/submodular-conditional-mutual-information-scmi
type: topic
---

# Submodular Conditional Mutual Information

Submodular Conditional Mutual Information (SCMI) is a generalization of classical conditional mutual information in which the additive, monotone, submodular set function replaces the role of Shannon entropy. SCMI enables the quantification of dependence and information flow among arbitrary collections of objects—such as random variables, strings, sets, or more abstract combinatorial entities—by leveraging the structure and properties of submodularity. It unifies entropic, algorithmic, and combinatorial notions of information, and underpins modern algorithmic frameworks for causal inference, summarization, and query-driven subset selection in both classical and non-i.i.d. settings [1002.4020][2107.00717][2006.15412].

## 1. Formal Definition

Given a finite ground set \( V \) and a normalized, monotone, submodular set function \( f: 2^V \to \mathbb{R}_+ \), the submodular conditional mutual information of subsets \( A, B, C \subseteq V \) is defined as:
\[
I_f(A;B | C) = f(A \cup C) + f(B \cup C) - f(C) - f(A \cup B \cup C)
\]
This measures the incremental value of jointly considering \( A \) and \( B \) given \( C \), over and above their individual contributions conditioned on \( C \) [1002.4020][2107.00717][2006.15412].

If \( f \) is the Shannon entropy, this recovers the standard conditional mutual information for random variables:
\[
I(X_A; X_B | X_C) = H(A, C) + H(B, C) - H(C) - H(A, B, C)
\]
If \( f \) is the Kolmogorov complexity (up to additive logarithmic terms), this recovers algorithmic conditional mutual information for finite strings [1002.4020].

## 2. Theoretical Properties

The axiomatic submodularity, monotonicity, and normalization of \( f \) endow SCMI with several key properties [1002.4020][2006.15412][2107.00717]:

- **Non-negativity:** \( I_f(A;B|C) \ge 0 \) for all \( A,B,C \subseteq V \).
- **Symmetry:** \( I_f(A;B|C) = I_f(B;A|C) \).
- **Chain Rule:** For any \( A, B, C, D \subseteq V \):
  \[
  I_f(A; B \cup C | D) = I_f(A; B | D) + I_f(A; C | B \cup D)
  \]
- **Monotonicity/Submodularity:** For any fixed \( B, C \), \( A \mapsto I_f(A;B|C) \) is monotone and, for functions satisfying a third-order supermodularity condition, also submodular [2006.15412].
- **Data-Processing Inequality:** If \( f(A \cup C) - f(C) = I_f(A; C) = 0 \), then for all \( D \subseteq V \), \( I_f(A; D) \le I_f(B; D) \) if \( A \subseteq B \).

Submodular SCMI extends the operational and combinatorial toolkit available for classical mutual information, encompassing settings defined by combinatorial optimization, algorithmic information theory, and non-stochastic data [1002.4020][2006.15412].

## 3. Representative Instantiations

Many widely used submodular functions yield concrete, tractable SCMI expressions. Key examples include [2107.00717][2006.15412]:

| Model                | Submodular Function \( f(X) \) | SCMI \( I_f(A;B|C) \)                    |
|----------------------|---------------------------------|------------------------------------------|
| Facility-Location    | \( \sum_{i\in V} \max_{j\in X} S_{ij} \) | \( \sum_{i\in V} [ \max_{j\in A \cup C} S_{ij} + \max_{j\in B \cup C} S_{ij} - \max_{j\in A \cup B \cup C} S_{ij} - \max_{j\in C} S_{ij} ] \) |
| Set Cover            | \( |\bigcup_{j\in X} \Gamma(j)| \)       | \( |\Gamma(A\cup C)| + |\Gamma(B\cup C)| - |\Gamma(C)| - |\Gamma(A\cup B\cup C)| \)               |
| Graph Cut            | \( \lambda \sum_{i\in V}\sum_{j\in X} s(i,j) - \sum_{j,j'\in X}s(j,j') \) | See [2006.15412] for explicit formula   |

These models permit the application of SCMI to information summarization, feature selection, and batch active learning, among other tasks [2006.15412][2107.00717].

## 4. Computational Estimation and Algorithms

In practice, SCMI is often estimated using surrogate functions tailored to the application domain or empirical data constraints. For algorithmic information, computable proxies based on data compression schemes, such as Lempel–Ziv complexity or grammar-based representations, serve as practical alternatives to uncomputable Kolmogorov complexity [1002.4020]. For combinatorial models, facility-location, set cover, and graph-cut instantiations are directly evaluable from the underlying similarity or coverage structures [2107.00717][2006.15412].

Optimization tasks involving SCMI, such as maximizing \( I_f(A;Q|P) \) under cardinality or knapsack constraints, leverage the monotone submodular structure to enable greedy, lazy-greedy, or stochastic-greedy algorithms with provable \((1-1/e)\)-approximation guarantees when the underlying function admits the appropriate third-order submodularity (second-order supermodular) conditions [2107.00717][2006.15412].

## 5. Connections to Classical and Algorithmic Information Theory

SCMI generalizes both Shannon-theoretic and algorithmic notions of conditional mutual information [1002.4020]. When \( f \) is Shannon entropy, SCMI reduces to ordinary conditional mutual information for random variables. When \( f \) is Kolmogorov complexity, SCMI yields algorithmic CMI for deterministic objects, modulo additive constants. Other choices of \( f \) model gain, redundancy, and coverage over sets, generalizing information-theoretic concepts to arbitrary combinatorial domains [2006.15412].

SCMI thereby provides a unified abstraction encompassing stochastic, deterministic, and combinatorially structured data, admitting direct analogues to chain rules, conditional independence, and data-processing inequalities [1002.4020][2006.15412].

## 6. Applications in Machine Learning and Causal Inference

SCMI underpins a diverse array of optimization-based machine learning tasks [2006.15412][2107.00717]:

- **Causal Discovery:** SCMI-based conditional independence tests can be directly integrated into constraint-based algorithms such as PC or FCI, applicable to arbitrary objects, including strings and time-series, using compression-based estimators [1002.4020].
- **Active Learning:** SIMILAR, an SCMI-based batch active learning framework, targets rare classes, avoids redundancy, and filters out out-of-distribution data by selecting points maximizing SCMI with respect to task-specific label and exclusion sets. The underlying optimization leverages the submodularity and monotonicity of SCMI, ensuring efficient and scalable acquisition [2107.00717].
- **Data Summarization and Diversification:** Maximizing SCMI captures relevance to query sets while enforcing diversity with respect to auxiliary sets; privacy-preserving summarization is formulated by subtracting SCMI terms corresponding to sensitive sets [2006.15412].
- **Partitioning and Clustering:** Multi-way generalizations of SCMI (total correlation, k-way mutual information) support clustering, robust partitioning, and submodular welfare problems by quantifying dependence among groupings of elements [2006.15412].

## 7. Illustrative Examples and Empirical Surrogates

Examples showcase the breadth of SCMI applicability:
- For period-length data, with \( F(S) = \log(\mathrm{lcm}\{x_i: i \in S\}) \), the SCMI expression simplifies to differences in logarithmic GCDs, exposing new and shared periodicities, directly interpretable in the combinatorial domain [1002.4020].
- For sequences of texts (e.g., translations), Lempel–Ziv-based SCMI proxies empirically verify conditional independence chains, facilitating recovery of causal order via constraint-based algorithms [1002.4020].
- In active learning datasets, facility-location and set-cover SCMI instantiations yield closed forms for batch selection that concomitantly balance query relevance, diversity, and redundancy avoidance [2107.00717].

The practical significance of SCMI derives from both its theoretical guarantees—reflecting foundational information-theoretic properties—and its flexibility, enabling scalable, adaptive algorithms across diverse, structurally complex data regimes [1002.4020][2107.00717][2006.15412].

Source: https://www.emergentmind.com/topics/submodular-conditional-mutual-information-scmi