Code Concept Analysis (CoCoA)

• Code Concept Analysis (CoCoA) is a post-hoc framework that discovers latent code concepts via clustering contextualized token embeddings.
• It integrates unsupervised concept discovery, syntactic alignment through static analysis, and LLM hybrid annotation for human-friendly labels.
• CoCoA provides local, concept-grounded explanations that improve the interpretability of code language model predictions.

Searching arXiv for the core CoCoA paper and closely related code-concept analysis works. Code Concept Analysis (CoCoA) denotes a global, post-hoc interpretability framework for code LLMs that uncovers emergent lexical, syntactic, and semantic structures in a model’s representation space by clustering contextualized token embeddings into human-interpretable concept groups (Sharma et al., 1 Oct 2025). In the code-model literature, the term also connects to adjacent lines of work on neuron-level concept localization in code-trained LLMs (Sharma et al., 2023) and concept-supervised reasoning for software engineering tasks (Rahman et al., 24 Mar 2026). In its most specific current usage, however, CoCoA refers to the framework introduced for analyzing latent concepts in code LLMs, combining unsupervised concept discovery, hybrid annotation with static analysis and LLMs, and concept-grounded local explanations (Sharma et al., 1 Oct 2025).

1. Definition and scope

CoCoA is a global, post-hoc interpretability framework for code LLMs that operates on hidden representations rather than on surface tokens alone (Sharma et al., 1 Oct 2025). Its central premise is that contextualized token embeddings in code models contain latent clusters corresponding to recurring code concepts, and that these clusters can be recovered, labeled, and analyzed without modifying model training (Sharma et al., 1 Oct 2025). The recovered concepts span multiple abstraction levels, including lexical regularities, syntactic roles, and higher-level semantic functions (Sharma et al., 1 Oct 2025).

This framing distinguishes CoCoA from local attribution methods such as Integrated Gradients, SHAP, or LIME, which typically highlight individual tokens but often do not explain why a token matters in code-specific terms (Sharma et al., 1 Oct 2025). It also differs from conventional probing, which requires predefined labels and primarily tests whether a property can be decoded from hidden states rather than how the model organizes its internal concept space (Sharma et al., 1 Oct 2025). CoCoA instead treats the model’s representation geometry itself as the primary object of analysis (Sharma et al., 1 Oct 2025).

The term has acquired broader resonance because related work addresses “code concepts” from complementary angles. “Redundancy and Concept Analysis for Code-trained LLMs” studies neuron-level traceability of code-related concepts such as “number” neurons, “string” neurons, and higher-level task features (Sharma et al., 2023). “ConceptCoder” defines code concepts as human-understandable semantic properties of code and uses them as supervised intermediate representations for vulnerability detection and branch prediction (Rahman et al., 24 Mar 2026). This suggests a broader research area in which CoCoA can denote concept-centric analysis of code models, but the canonical framework name corresponds to the latent-concept clustering approach introduced in 2025 (Sharma et al., 1 Oct 2025).

2. Core pipeline

CoCoA consists of three main components: concept discovery, concept alignment and annotation, and concept-based local explanations (Sharma et al., 1 Oct 2025). The pipeline begins by extracting contextualized token embeddings from each layer of a code LLM for a large corpus of code, using approximately 310k token occurrences per layer for clustering (Sharma et al., 1 Oct 2025). The reported models include CodeBERT, UniXCoder, and DeepSeekCoder V2 Lite Instruct, and the data source is approximately 30k snippets from Project CodeNet (Sharma et al., 1 Oct 2025).

Concept discovery is performed per layer using K-means with a fixed number of clusters $K = 350$ (Sharma et al., 1 Oct 2025). The optimization objective is

$$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$

where $C_k$ is cluster $k$, $\mu_k$ its centroid, and $z$ are contextualized token embeddings (Sharma et al., 1 Oct 2025). Before clustering, highly frequent tokens with frequency $> 15k$ are discarded, and clusters with $> 15k$ members are later removed as too broad or uninformative (Sharma et al., 1 Oct 2025). This yields a layer-wise inventory of latent concepts defined by representation-space proximity rather than by programmer-specified taxonomies (Sharma et al., 1 Oct 2025).

Concept alignment and annotation then convert clusters into interpretable units. Static analysis via Tree-sitter ASTs is used to quantify syntactic alignment, while a hybrid manual-plus-LLM annotation process generates lexical, syntactic, and semantic labels (Sharma et al., 1 Oct 2025). The resulting labeled collection is curated as CodeConceptNet (CoCoNet) (Sharma et al., 1 Oct 2025). Finally, CoCoA integrates the discovered concepts with local attribution by mapping salient tokens to latent concept clusters and using cluster labels and descriptions to produce concept-grounded explanations (Sharma et al., 1 Oct 2025).

A plausible implication is that CoCoA treats concept analysis as a layer between raw representations and human interpretation: clustering provides the latent structure, static analysis constrains syntactic plausibility, and LLM annotation supplies descriptive semantics (Sharma et al., 1 Oct 2025).

3. Mathematical formulation and annotation methodology

The formalization of concept alignment in CoCoA is based on overlaps between discovered clusters and ground-truth syntactic categories (Sharma et al., 1 Oct 2025). Let $\mathcal{T}$ denote the set of token occurrences, $\mathcal{C}_E = \{C_e\}$ the discovered clusters, and $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$0 the set of syntactic categories derived from Tree-sitter (Sharma et al., 1 Oct 2025). For a cluster $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$1 and category $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$2, the token-overlap ratio is

$$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$3

A cluster is aligned at threshold $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$4 if

$$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$5

Coverage of a ground-truth category is defined as

$$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$6

(Sharma et al., 1 Oct 2025)

These definitions support aggregate measures such as “Clusters Labeled,” “Tag Coverage,” and an “Overall Alignment Score” (Sharma et al., 1 Oct 2025). For CodeBERT at layer 12 with 350 clusters and a 90% threshold, the reported values are 328/350 clusters labeled, 50% tag coverage, and overall alignment 0.719 in the pre-trained setting; after AST-node-classification fine-tuning, these become 341/350, 76.4%, and 0.869, respectively (Sharma et al., 1 Oct 2025). Such numbers indicate that many latent clusters are syntactically coherent, and that structure-aware fine-tuning sharpens this coherence (Sharma et al., 1 Oct 2025).

Annotation proceeds in two stages. First, senior computer science students with Java experience manually labeled approximately 500 clusters using token lists and example code contexts (Sharma et al., 1 Oct 2025). Second, LLM-based labeling scaled this process using prompts that included token lists, context sentences, and examples of desired output format (Sharma et al., 1 Oct 2025). The reported LLMs include GPT-4o and Gemini-2.0-Flash, and the final annotation scheme consolidates 1,724 raw labels into 43 canonical semantic tags, with “Unclear Behavioral Role” assigned to residual clusters that resist stable interpretation (Sharma et al., 1 Oct 2025). With the improved prompt, 95.8% of LLM-generated labels were rated acceptable; 38.4% of syntactic labels and 83.2% of semantic labels were judged superior to human counterparts (Sharma et al., 1 Oct 2025).

The hybrid annotation design is notable because it does not require the latent concept inventory to be predefined. This suggests that CoCoA occupies a middle ground between unsupervised discovery and ontology-based interpretation: clusters are model-induced, but their final descriptions are constrained by static syntax and curated semantic labeling (Sharma et al., 1 Oct 2025).

4. Concept types and internal organization

CoCoA reports that discovered clusters capture lexical, syntactic, and semantic regularities (Sharma et al., 1 Oct 2025). Lexical clusters include variable-name patterns such as suffix-sharing tokens, numerics, and casing regularities such as CamelCase and PascalCase (Sharma et al., 1 Oct 2025). Syntactic clusters group items such as access modifiers (public, private, protected), comparison operators (<, ==, !=), parentheses, and the method invocation operator (.) (Sharma et al., 1 Oct 2025). Semantic clusters capture broader code functions such as mathematical operations (abs, max, floor, pow), error logging, data-structure operations, and function definition keywords across languages (Sharma et al., 1 Oct 2025).

Layer-wise analyses indicate that lexical patterns are strongest in early layers but persist into deeper layers, where they are embedded in more structured or compositional clusters (Sharma et al., 1 Oct 2025). Syntactic alignment is high in early and middle layers and then plateaus or slightly declines in later layers, consistent with a transition toward more semantic or task-specialized concepts (Sharma et al., 1 Oct 2025). Fine-tuning modulates this structure. AST node classification increases lexical and structural coherence, compile-error detection produces more unaligned clusters suggestive of higher-level semantics, and language classification reduces syntactic alignment while promoting cross-lingual abstractions (Sharma et al., 1 Oct 2025).

The framework’s examples emphasize ambiguity resolution. A salient token such as < may belong to different clusters corresponding to logical comparison, generic type delimiters, or PHP opening tags, depending on context (Sharma et al., 1 Oct 2025). CoCoA’s cluster assignment and annotation disambiguate these roles by grounding each occurrence in a concept group rather than treating the token type as intrinsically meaningful (Sharma et al., 1 Oct 2025). This directly addresses a common limitation of token-level saliency, where punctuation often appears important but its role is underdetermined (Sharma et al., 1 Oct 2025).

Related work complements this cluster-centric notion of concept organization with neuron-centric evidence. Neuron-level analysis identifies “number” neurons, “string” neurons, “identifier” neurons, and higher-level “text” neurons in code-trained LLMs, and reports that over 95% of neurons are redundant with respect to several code intelligence tasks (Sharma et al., 2023). ConceptCoder, by contrast, defines explicit code concepts such as null assignment, null check, pointer dereference, memory allocation, buffer access, bounds check, and memory free for vulnerability detection, as well as abstract-value concepts such as True, False, NULL, Not NULL, and numeric range abstractions for branch prediction (Rahman et al., 24 Mar 2026). Together, these studies suggest that code concepts can be localized at multiple representational scales: clusters of token embeddings, subsets of neurons, and supervised intermediate labels (Sharma et al., 1 Oct 2025, Sharma et al., 2023, Rahman et al., 24 Mar 2026).

5. Robustness, attribution, and human interpretability

A central contribution of CoCoA is the Cluster Sensitivity Index (CSI), which measures how stable latent concept clusters remain under semantic-preserving perturbations (Sharma et al., 1 Oct 2025). If $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$7 and $$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$8 are clusterings before and after perturbation, CoCoA computes pairwise Jaccard similarities

$$\min_{\{C_1,\dots,C_K\}} \sum_{k=1}^K \sum_{z \in C_k} \|z - \mu_k\|^2$$9

and uses the Hungarian algorithm to find a bijection maximizing total similarity (Sharma et al., 1 Oct 2025). The average matched Jaccard is then converted into

$C_k$0

Lower CSI indicates greater stability (Sharma et al., 1 Oct 2025). Reported examples include deterministic identifier renaming with Average Jaccard 0.6751 and CSI 0.3249, identifier casing variation with 0.6723 and 0.3277, minimal casing perturbation with 0.6657 and 0.3343, and statement order randomization with 0.5315 and 0.4685 (Sharma et al., 1 Oct 2025). The paper reports average CSI $C_k$1, indicating that discovered concepts remain moderately stable under semantics-preserving transformations (Sharma et al., 1 Oct 2025).

CoCoA’s local explanation mechanism, termed Latent Concept Attribution, combines Integrated Gradients with concept mapping (Sharma et al., 1 Oct 2025). A model prediction is first explained at token level; top tokens are selected until they account for 50% of cumulative attribution mass (Sharma et al., 1 Oct 2025). Each salient token’s contextual embedding is then mapped to a latent concept cluster using a logistic regression classifier trained to predict cluster IDs from embeddings (Sharma et al., 1 Oct 2025). Cluster labels, word clouds, and LLM-generated descriptions are finally used to express why the salient token mattered in concept terms (Sharma et al., 1 Oct 2025).

This changes the explanatory unit from isolated symbols to structured code roles. In a programming-language-classification example, the token < is not merely highlighted as important; it is mapped to a concept cluster associated with PHP opening tags such as <? and <?php, thereby clarifying its role as a language indicator rather than as a comparator or generic-type delimiter (Sharma et al., 1 Oct 2025). In a user study, only 24.3% of cases were explainable from the salient token alone, while concept cluster context improved human-centric explainability by 37 percentage points compared with token-level attributions using Integrated Gradients (Sharma et al., 1 Oct 2025).

Concept-grounded explanation also aligns with adjacent research. ConceptCoder uses explicit code concepts as intermediate supervision and shows that concept-based fine-tuning improves vulnerability-detection F1 from 66.32 to 72.15 on average over 9 open-source LLMs, while also improving branch prediction (Rahman et al., 24 Mar 2026). This suggests that concept representations are not only interpretable artifacts but can also serve as useful computational bottlenecks or regularizers (Rahman et al., 24 Mar 2026). A plausible implication is that CoCoA-style concept inventories could support both explanation and performance-oriented model design.

6. Relation to adjacent research, limitations, and outlook

CoCoA sits within a broader movement toward concept-centered analysis of code models. Identifier-based program comprehension work predating modern code LMs already treated domain terms and concept location as bridges between source code and human understanding, using AST parsing, identifier splitting, and WordNet-based semantic relations to derive domain vocabularies and locate code fragments implementing particular ideas (Vaclavik et al., 2010). More recent neuron-level work localizes code properties within latent representations and reports highly redundant but partially decomposable concept-bearing subnetworks (Sharma et al., 2023). Concept-supervised modeling extends the concept perspective from post-hoc interpretation to training-time guidance (Rahman et al., 24 Mar 2026). CoCoA’s distinctive contribution is to make the model’s global latent concept structure itself an analyzable object (Sharma et al., 1 Oct 2025).

Several limitations are explicitly identified. K-means assumes spherical clusters and may miss hierarchical or irregular concept structure; the choice $C_k$2 is heuristic; and hierarchical clustering, while potentially more expressive, is computationally prohibitive at the reported scale (Sharma et al., 1 Oct 2025). Syntactic alignment depends on Tree-sitter parsing quality, and LLM-based labeling, despite strong reported acceptability, can hallucinate or drift without careful prompt design (Sharma et al., 1 Oct 2025). The attribution component inherits limitations of Integrated Gradients, including baseline sensitivity and the tendency to emphasize syntactic tokens (Sharma et al., 1 Oct 2025). Coverage is also limited to three models and three fine-tuning tasks in the core CoCoA study (Sharma et al., 1 Oct 2025).

Related works expose complementary limitations. Neuron-level concept analysis relies on probing and selectivity assumptions that may not fully resolve whether a concept is genuinely encoded or merely recoverable (Sharma et al., 2023). ConceptCoder’s supervised concept sets are intentionally narrow—7 concepts for vulnerability detection and 12 for branch prediction—and scaling them across languages and domains would require richer static or dynamic analysis pipelines (Rahman et al., 24 Mar 2026). Earlier identifier-centric approaches assume meaningful naming conventions and largely English-language identifiers, which can degrade in acronym-heavy or multilingual codebases (Vaclavik et al., 2010).

The forward trajectory of CoCoA research is therefore twofold. One path expands latent concept discovery with more expressive clustering, stronger robustness diagnostics, and integration with larger code models (Sharma et al., 1 Oct 2025). The other path connects discovered concepts to intervention, compression, and task supervision, as suggested by neuron-level pruning results and concept-based fine-tuning (Sharma et al., 2023, Rahman et al., 24 Mar 2026). This suggests that Code Concept Analysis may develop into a unifying framework for interpreting, steering, and improving code LLMs through explicit intermediate semantic structure rather than through token-level heuristics alone.