Neo-Grounded Theory (NGT)

• Neo-Grounded Theory (NGT) is a qualitative research framework that uses 1536-dimensional semantic embeddings and unsupervised clustering to replace manual coding.
• It employs high-dimensional vector clustering and hierarchical agglomeration to efficiently organize and reveal patterns in large qualitative datasets.
• The human-in-the-loop design ensures iterative refinement, combining automated multi-agent coding with researcher insight to produce robust and scalable theoretical models.

Neo-Grounded Theory (NGT) is a methodological framework for qualitative research that integrates high-dimensional vector clustering and multi-agent computational collaboration. NGT addresses the scale–depth paradox inherent in qualitative analysis—where increasing scale traditionally erodes interpretive depth—by embedding qualitative text into a 1536-dimensional semantic space, leveraging parallelized multi-agent coding, and incorporating an iterative human-in-the-loop process. NGT replaces manual coding frames with mathematically grounded, unsupervised pattern discovery while preserving the core interpretive commitments of grounded theory through structured researcher–AI collaboration (Wen et al., 26 Sep 2025).

1. Theoretical Foundation and Rationale

Neo-Grounded Theory is defined by three principal methodological advances over classical grounded theory:

• Semantic Vector Embedding: Segments of qualitative data are embedded into a high-dimensional semantic space. Categories emerge through unsupervised clustering, obviating the need for pre-specified code lists.
• Distributed Cognition via Multi-Agent Systems: Specialized computational agents, each responsible for distinct stages of theory construction, operate in parallel. This compresses the timeline for coding large datasets from months to hours.
• Augmented Sensitivity (Human-in-the-Loop): Researchers direct and refine the computational pattern recognition process. Iterative cycles of AI-driven summary and human theoretical interpretation achieve rich, actionable frameworks that neither automation nor manual analysis alone can yield.

NGT directly resolves the scale–depth paradox by delegating low-level pattern detection to machine agents, allowing interpretive effort to scale independently of dataset size.

2. Embedding and Clustering Workflow

2.1 Embedding and Normalization

Each textual segment $t_i$ (50–200 words) is processed as follows:

• Token embeddings are generated using OpenAI’s text-embedding-3-small model:

$e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$

where $E$ is the embedding function for token $w_{ij}$.

• Segment embedding is computed via mean pooling:

$v_i = \frac{1}{N_i} \sum_{j=1}^{N_i} e_{ij}$

where $N_i$ is the number of tokens in $t_i$.

• Each segment vector is L2-normalized:

$\hat v_i = \frac{v_i}{\|v_i\|_2}$

2.2 Similarity, Distance, and Clustering

• Normalized embeddings are compared by cosine similarity:

$$\mathrm{sim}(\hat v_i,\hat v_j) = \hat v_i \cdot \hat v_j$$

• Cosine distance is:

$$D(\hat v_i,\hat v_j) = 1 - \mathrm{sim}(\hat v_i,\hat v_j)$$

NGT employs hierarchical agglomerative clustering with average linkage:

• Each segment starts as its own cluster.
• Pairs of clusters $e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$0 with minimal average cosine distance are merged:

$e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$1

• Merging continues until the similarity threshold ($e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$2) is not met.

The objective is to minimize within-cluster variance: $e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$3

2.3 Clustering Algorithm Summary

Step	Description	Notes
1	Normalize embeddings	e.g., L2-norm
2	Initialize each as singleton cluster
3	Compute all pairwise distances	Cosine-based
4	Merge clusters by $e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$4	Stop when similarity $e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$5
5	Output final clusters

3. Multi-Agent Coding Architecture

NGT orchestrates up to 12 specialized coding agents per dataset, with each agent operating on an assigned cluster:

• Open Coding Agent: Induces preliminary codes and definitions.
• Axial Coding Agent: Identifies interrelations among codes (causal, contextual, consequence, intervening).
• Selective Coding Agent: Integrates concepts to form a core category.
• Cross-Cluster Integration Agent: Aggregates and synthesizes all clusters' output, performing frequency, centrality, and contrast analyses, and constructing the multi-level theory network.

Agents output structured JSON containing cluster identifiers, open codes, axial relationships, core categories, and supporting evidence.

A lightweight message-passing API enables agent communication for automated code requests, response review cycles, and refinement.

4. Human–AI Interaction Protocol

NGT operationalizes researchers’ theoretical sensitivity through iterative, structured human–AI interaction. Two experimental modes structure this interaction:

• Pure Automation: Agents cluster and code with neutral prompts and no human intervention, yielding high-level but abstract conceptual schemas.
• Human-Guided Refinement: Researchers review AI outputs, engineer prompts to probe theoretical tensions, divergent trajectories, and actionable intervention points, then regenerate codes. This cycle repeats until theoretical saturation.

Prompt refinement examples include transitioning from generic framework-building requests to explicit instructions emphasizing adaptive/maladaptive pathways and conceptual tensions.

5. Empirical Evaluation

NGT was benchmarked on 8 Chinese semi-structured interviews (40,000 characters) with these comparators:

Method	Time	Quality Score (0–1)	Cost	Speedup vs. Manual
Manual Coding (NVivo, 2 experts)	504 hours	0.883	\$12,800	1x
ChatGPT-4 Turbo Assisted	24 hours	0.840	\$425	21x
NGT Exp 1 (automation)	0.5 hours	0.812	\$95	1,008x
NGT Exp 2 (human-in-loop)	3 hours	0.904	\$95	168x

• Silhouette coefficient ($e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$6) and Davies–Bouldin Index (DB) evaluated cluster quality, and were used internally for threshold adjustment.
• Inter-evaluator reliability: Krippendorff’s $e_{ij} = E(w_{ij}) \in \mathbb{R}^{1536}$7.
• Inter-method Jaccard similarity for code sets: 75.4% manual vs NGT.

NGT achieved a 168-fold speedup and a 99.3% cost reduction in its human-in-loop variant. The return on investment (ROI) was computed as approximately 13,368%. NGT Exp 2 outperformed both manual and automated methods in theory quality (0.904 vs. 0.883 and 0.812).

6. Empirical and Theoretical Contributions

NGT surfaced novel empirical patterns, including:

• Temporal rhythms in gaming corresponding to stress cycles.
• Identity bifurcation phenomena, with strategic switching between “gamer” and “disabled person” identities.
• Compensatory hierarchies involving emotional, social, and cognitive domains.

The framework established that vector embeddings can mathematically ground semantic relations without erasing interpretive nuance, and that multi-agent parallel coding maintains emergent category formation with substantial scalability. Human–AI collaboration is essential: pure automation yields brittle abstraction, while episodic human guidance produces dialectically robust, testable theories.

The system ensures reproducibility and transparency with comprehensive audit trails covering clusters, code logs, prompts, and agent reasoning traces.

7. Implications, Limitations, and Prospects

Implications:

• Democratization of qualitative research through extreme reductions in time and cost, enabling small teams or communities to analyze their own data contemporaneously with unfolding events.
• Agile theory construction capable of tracking rapid social change, such as during pandemics or in social media phenomena.

Limitations:

• Current empirical validation is domain-specific (Chinese gaming interviews), and efficacy in other areas and languages remains to be shown.
• Potential underrepresentation of narrative flow, metaphorical content, and subtle cultural meaning.
• Despite reduced material barriers, effectiveness depends on theoretical sophistication in prompt engineering and analytical interpretation.

Future Directions:

• Development of locally trained embedding models to mitigate cultural bias.
• Extension to multimodal clustering (images, audio, video) with integrated joint semantic spaces.
• Creation of user-friendly, graphical interfaces for researchers lacking technical backgrounds.
• Incorporation of temporal modeling to trace the evolution of concepts.
• Integration with deep hermeneutic approaches for enhanced single-case interpretive depth (Wen et al., 26 Sep 2025).

Neo-Grounded Theory thus furnishes a technically rigorous and scalable pathway for surfacing, evaluating, and theorizing qualitative patterns, combining the strengths of computational reproducibility with humanistic analytical depth.