Concept Component Analysis (ConCA)

• Concept Component Analysis (ConCA) is a principled framework that extracts interpretable conceptual structures from high-dimensional representations, enabling enhanced ontology engineering and LLM interpretability.
• It employs a multi-step process combining graph clustering, latent variable modeling, and unsupervised linear unmixing to recover shared conceptual components from complex data.
• ConCA delivers actionable insights on semantic abstraction with strong empirical performance, including high alignment correlations and reduced reconstruction MSE in practical applications.

Concept Component Analysis (ConCA) is a principled framework for extracting interpretable conceptual structure from high-dimensional representations, with prominent applications in both ontology engineering and mechanistic interpretability of LLMs. It formalizes the recovery of domain-general components that correspond to shared patterns (in ontologies) or latent conceptual variables (in neural networks) through structured decomposition and clustering. ConCA’s foundations span relational pattern mining, latent variable modeling, unsupervised linear unmixing, and information-theoretic interpretability metrics.

1. Formal Definition and Problem Statement

ConCA identifies and catalogs Conceptual Components (CCs), abstractions that embody intensional meaning or relational patterns common to structures within a corpus. In ontology analysis, a CC is constructed as a set of Observed Ontology Design Patterns (OODPs), each realized as a fragment from their respective ontology encoding (classes, properties, axioms), all instantiating the same latent pattern. For LLM representations, CCs are instantiated as latent concept variables $\mathbf{z}$ within a generative model:

$$p(x, y) = \sum_{\mathbf{z}} p(x | \mathbf{z})\;p(y | \mathbf{z})\;p(\mathbf{z})$$

with $x$ as input context, $y$ as output token, and concepts $\mathbf{z}$ underlying both.

ConCA systematically computes a mapping

$$\Phi:\bigcup_{i=1}^{n} \left\{\text{communities in } O_i\right\} \to \{CC_1, \ldots, CC_m\}$$

for ontologies, or recovers a linear mixture of log-posteriors over concepts for LLMs: $f(x) \approx A\,\log p(\mathbf{z} \mid x) + b$ where $f(x)$ is the LLM’s internal representation, and $A$ is the mixing matrix.

2. Methodology: Algorithms and Workflow

ConCA implements a multi-step pipeline tailored to the domain:

Ontology Engineering (Asprino et al., 2021)

1. Pre-processing: Transform OWL axioms to intensional undirected graphs for each ontology.
2. Community Detection: Apply Clauset–Newman–Moore clustering; recursively split based on density.
3. OODP Extraction: Retrieve subgraphs as OODPs, collecting domain/range axioms and relations.
4. Virtual Document Construction: Concatenate rdfs:label; perform word-sense disambiguation (UKB, WordNet); annotate with matching FrameNet frames.
5. Clustering: Vectorize documents via tf–idf; cluster using KMeans, optimizing $k$ via the elbow method.
6. CC Labeling and Hierarchy: Assign labels as most frequent synsets/frames; build hierarchy using frame inheritance strength:

$$w(ab) = \frac{|\{f \in F(CC_a) \mid \exists f' \in F(CC_b): f \sqsubseteq f'\}|}{\max_{p,q} |\{f \in F(CC_p) \mid \exists f' \in F(CC_q): f \sqsubseteq f'\}|}$$

LLM Mechanistic Interpretability (Liu et al., 28 Jan 2026)

1. Latent Variable Model: Define context, next token, and latent discrete concepts.
2. Linear Mixture Approximation: Express $f(x)$ as approximately linear in log-marginals over concepts.
3. Unsupervised Linear Unmixing: Train encoder/decoder pairs $(W_e, b_e),\,(W_d, b_d)$ so that

$$\hat{z} = W_e\, f(x) + b_e,\qquad \hat{f}(x) = W_d\,\hat{z} + b_d$$

and minimize

$$\mathbb E_x \left[\| W_d(W_e f(x) + b_e) + b_d - f(x) \|_2^2 + \alpha\,\mathcal S(g(W_e f(x) + b_e))\right]$$

where $g$ is an exp-like activation and $\mathcal S$ enforces sparsity.

1. Sparse ConCA Variants: Combine normalization (LayerNorm, BatchNorm, Dropout, GroupNorm) and activation surrogate (SoftPlus, ELU, SELU), yielding 12 algorithmic variants.

3. Similarity Measures, Aggregation, and Interpretation

ConCA employs domain-appropriate similarity and aggregation mechanisms:

• Ontology: Documents clustered via tf–idf; K-Means with Euclidean distance; hierarchical relations quantified by frame overlap and inheritance.
• LLMs: Concept features recovered as blocks in $W_e$; context activation patterns inspected via $g(\hat{z})$.

Two cluster evaluation metrics for ontologies:

• Overlap Coefficient:

$$\mathrm{Overlap}(CC_p, CC_q) = \frac{|F(CC_p) \cap F(CC_q)|}{\min(|F(CC_p)|, |F(CC_q)|)}$$

• Alignment Correlation:

$$\mathrm{corr}(A,B) = \frac{\sum_{(e_i, e_j) \in A} cs(e_i, e_j)}{\sum_{(e_i, e_j) \in B} cs(e_i, e_j)}$$

where $cs$ denotes confidence score.

For LLMs, evaluation uses:

• Mean Pearson Correlation (MPC) between recovered features and supervised log-posterior probes.
• Reconstruction MSE, rank-based feature stability, downstream classification AUC.

4. Empirical Evaluation and Case Studies

Ontology (Asprino et al., 2021)

Applied to Cultural Heritage ($n=43$) and Conference ($n=16$) ontologies:

• Community Quality: Manual assessment into categories (“bad,” “medium,” “good,” “ideal”).
• Overlap Scores: Low ($0.20$ CH, $0.17$ Conf), indicating strong cluster separation.
• Correlation to Curated Alignments: Up to $0.99$ with expert sets; $0.58$ and $0.36$ with automated or alternative benchmarks.

Illustrative examples:

• Membership CC: Cross-ontology detection of “membership” pattern (object property, n-ary class).
• Event CC: Mapping “event” fragments across heterogenous ontologies (FrameNet frame “Event”).

LLMs (Liu et al., 28 Jan 2026)

Benchmarked on Pythia-70M/1.4B/2.8B, Gemma3-1B, Qwen3-1.7B.

• MPC (Pythia 2.8B): ConCA reaches $0.807$ (LayerNorm), compared to SAE at $0.58$–$0.76$.
• Reconstruction MSE: $3.6$ for ConCA, lower than SAE.
• Rank Stability: Sparse ConCA yields features with minimal rank changes under concept-perturbation.
• Downstream AUC: Up to $0.85$ AUC on binary classification; ConCA maintains advantage in out-of-distribution splits.

5. Application Domains and Potential Impact

ConCA’s cross-domain utility is reflected in diverse workflows:

• Ontology Engineering: Enables conceptual cataloging, pattern-based import, matching, and empirical evaluation of modeling practices. Facilitates semantic interoperability and reuse by abstracting common building blocks.
• LLM Interpretability: Delivers theory-driven mechanisms for extracting human-meaningful concept directions, addressing the ambiguity of sparse autoencoder (SAE) approaches. Supports robust attribution, few-shot learning, and OOD generalization.

A plausible implication is improved transparency and actionable insights for both automated systems (e.g., search space restriction in matching) and interactive domain experts (e.g., browsing conceptual hierarchies).

6. Algorithmic Features, Limitations, and Practical Guidance

Strengths of ConCA include its non-extractive abstraction, scalability (processing $n=43$ ontologies in $\sim$1h15m), and cross-system comparison. For LLMs, sparse ConCA shows theory-backed improvement over SAE, with higher faithfulness (concordance to log-posterior ground truth) and utility (classification metrics).

Guidance for LLM applications:

• Feature Dimension $D$: Set large, e.g., $2^{15}$.
• Regularization: LayerNorm recommended for stability.
• Sparsity Weight $\alpha$: Effective range $5\times10^{-3}$ to $10^{-2}$; $10^{-4}$ for scaling.
• Activation Surrogates: SoftPlus empirically superior for approximating $\exp$.

Limitations:

• Ontology: Anonymous class expressions not handled.
• Clustering: Heuristic parameter tuning.
• Labeling: Partial reliance on manual curation.
• LLMs: Only approximate log-posteriors; underdetermined solutions; dependence on theoretical assumptions (diversity, entropy); discrete concepts only.

7. Related Methodologies and Theoretical Context

ConCA is distinguished by its explicit theoretical grounding (linear mixture of log-posteriors; unsupervised unmixing) versus heuristic decompositions such as SAEs. In ontology engineering (Asprino et al., 2021), it operationalizes pattern mining and frame semantics, enabling conceptual abstraction beyond concrete encodings. In mechanistic LLM interpretability (Liu et al., 28 Jan 2026), it formalizes the link between model activations and conceptual latent variables, directly targeting actionability and attribution.

In summary, Concept Component Analysis provides a robust, extensible methodology for discovering, extracting, and interpreting conceptual structure across knowledge graphs and neural network models, advancing the analytic capabilities in both semantic engineering and machine learning research.