C-Mining: Unsupervised Discovery of Seeds for Cultural Data Synthesis via Geometric Misalignment

Abstract: Achieving cultural alignment in LLMs increasingly depends on synthetic data generation. For such synthesis, the most vital initial step is seed curation; however, current methods lack quantifiable standards for selecting these seeds. Existing approaches rely on unscalable manual curation or bias-prone LLM extraction, treating cultural specificity as an abstract concept rather than a measurable signal. In this paper, we address this "quantification gap" by proposing C-Mining, an unsupervised framework that transforms the discovery of cultural seeds from a subjective selection process into a computable data mining formulation. Our approach exploits a novel geometric insight, leveraging the cross-lingual misalignment of cultural concepts within pre-trained embedding spaces as a quantifiable discovery signal. By systematically identifying these regions characterized by pronounced linguistic exclusivity and geometric isolation, while actively filtering out noise, C-Mining automatically extracts high-fidelity Culture Points (CPs) from raw multilingual corpora without reliance on human or LLM supervision, reducing preparation costs by more than 150-fold. We further leverage the mined knowledge to steer the synthesis of diverse instruction-tuning datasets. Extensive experiments demonstrate that this seed-centric approach significantly enhances cultural understanding and reasoning capabilities, achieving a +6.03 point improvement on CulturalBench-Hard and surpassing state-of-the-art baselines, providing a scalable, quantifiable solution for high-quality cultural data synthesis.

• The paper presents a novel geometric mining framework that quantifies cultural specificity by identifying distinct Culture Points in multilingual embedding spaces.
• It leverages robust monolingual filtering and cross-lingual clustering to extract high-fidelity cultural seeds, leading to significant improvements on culture-specific benchmarks.
• The approach achieves over two orders-of-magnitude cost efficiency while preserving core language model capabilities, enabling scalable global cultural instruction tuning.

Unsupervised Discovery of Cultural Seeds via Geometric Misalignment: The C-Mining Framework

Introduction

The dominance of high-resource, English-centric corpora in the pre-training of LLMs induces severe representation collapse for regionally grounded, culture-specific knowledge. Accurate realignment of LLMs toward genuine cultural nuance is critically dependent on high-quality synthetic instruction data tailored to underrepresented domains. The crucial and frequently overlooked stage in this pipeline is the systematic, scalable identification of robust culture-specific "seeds"—lexical or conceptual anchors facilitating authentic data synthesis. Prevailing manual and LLM-based extraction approaches remain unscalable, often perpetuate superficial stereotypes, and are susceptible to bias amplification. The paper "C-Mining: Unsupervised Discovery of Seeds for Cultural Data Synthesis via Geometric Misalignment" (2604.15675) proposes a geometric data mining approach, operationalizing cultural specificity as quantifiable misalignment in multilingual embedding spaces.

Formulation: From Subjectivity to Geometric Quantification

C-Mining transforms the presently subjective, ill-defined notion of cultural specificity into an unsupervised, computable signal, leveraging unique topological features of frozen multilingual representations. The framework identifies Culture Points (CPs)—semantically dense, intra-lingual clusters that exhibit distinct geometric isolation (i.e., low cross-lingual alignment) in latent space. The computational procedure consists of two principal phases: (1) high-quality monolingual data pruning, and (2) cross-lingual geometric misalignment mining.

Figure 1: The C-Mining pipeline locates geometric islands of native homogeneity and cross-lingual exclusivity to extract high-fidelity cultural seeds.

This geometric formalization addresses coverage limitations—by surfacing long-tail, low-frequency knowledge—and bypasses the capacity limits and inherent biases of LLM-based proxies. The theoretical underpinning is rooted in observed stratification of embedding manifolds: universal knowledge converges, whereas culturally grounded concepts remain resistant to unsupervised cross-lingual alignment.

Methodology

Monolingual Filtering and Semantic Cohesion

The initial phase curates monolingual Wikipedia corpora using heuristic pruning (with NER negation filtering) to eliminate non-semantic noise, supplemented by metric-based centrality and intra-document semantic connectivity (Shannon entropy over embedding similarity matrices). The output is a high-quality, semantically cohesive lexicon per language.

Geometric Misalignment as a Mining Signal

The core innovation is unsupervised multigraph clustering in a shared embedding space (via frozen encoders). Clusters are scored for statistical stability and linguistic dominance; those with high intra-lingual purity ($\gamma > \theta$) and sufficient size are retained as seeds. Figure 2 demonstrates the geometric signature of CPs versus non-candidates.

Figure 2: CPs form strongly segregated, language-specific embedding clusters, unlike non-selected points which exhibit significant cross-lingual intermixing.

A radar plot comparison between embedding shifts pre-/post-translation further substantiates the claim: authentic cultural anchors show substantial semantic displacement, confirming their resistance to forced alignment.

Figure 3: CPs exhibit larger embedding distances upon translation, indicating inherent cultural semantic opacity to English-centric models.

Instruction Synthesis Pipeline

Culture Points are converted into synthetic fine-tuning data via cluster-level prompting, incorporating multiple cognitive task formats (e.g., discriminative MCQ, cultural norm reasoning, generative short-answer). This pipeline extends seeds into actionable supervision for robust downstream alignment.

Empirical Evaluation

Intrinsic Topology and Human Judgments

The extracted CPs are evaluated intrinsically through geometric separability and extrinsically via expert native speaker annotation, employing a multi-factor rubric encompassing exclusivity, context dependence, and non-universality. C-Mining CPs score consistently higher than those from LLM or survey-anchored baselines such as CultureAtlas, including in high-context domains.

Figure 4: Expert annotation across six cultural axes shows superior semantic and cultural specificity for C-Mining CPs relative to alternative extraction strategies.

Distributional Analysis & Data Stratification

To avoid linguistic distribution artifacts, the sampled Wikipedia dataset is balanced across languages. Reflecting the efficacy of geometric constraints, CP yield correlates with cultural-linguistic distance rather than corpus size.

Figure 5: Corpus balancing through stratified sampling ensures unbiased density estimation, with English oversampled to capture global and translated knowledge.

Figure 6: The distribution of extracted CPs aligns with linguistic divergence—not merely data presence—highlighting sensitivity to deep cultural signals.

Downstream Model Performance

Instruction fine-tuning with CP-synthesized data on standard and challenging multilateral cultural benchmarks demonstrates decisive improvements. On CulturalBench-Hard, C-Mining outperforms Qwen2.5-7B baseline by +6.03 points and establishes a new SOTA among culture-specific methods (outperforming CultureBank, CultureLLM, CultureSPA). Ablation studies confirm that seed quality, not just the scale of synthetic data, is the principal driver of alignment gains; random or monolingual-only seeds can actively degrade performance.

Sensitivity and Capacity Preservation

The model's performance exhibits an inverted U-curve with respect to the linguistic dominance threshold $\theta$, peaking at intermediate levels and confirming the necessity of strict—but not absolute—geometric purity. Critically, specialization does not entail catastrophic forgetting: Global-MMLU-Lite results show negligible or slightly positive impact on general reasoning, demonstrating that targeted cultural alignment is not antagonistic to core LLM capabilities.

Cost and Scalability

C-Mining’s computational cost ($\sim$ \$11 per million seeds, per language) is negligible compared to manual and API-based LLM curation (>$1500–50,000 for comparable scale), introducing an over two-order-of-magnitude efficiency gain and drastically improving accessibility for comprehensive cultural coverage at global scale.

Qualitative Expressivity

Case studies of generative responses reveal that models fine-tuned with C-Mining data move beyond superficial or stereotypical narratives, incorporating granular, context-relevant details unique to particular cultures and events. This demonstrates an improved ability to reproduce non-trivial, culture-anchored semantics, advancing the evaluation frontier from factual recitation to authentic narrative depth.

Implications and Future Directions

The C-Mining paradigm establishes geometric misalignment in embedding topology as a reliable, scalable, and quantifiable criterion for cultural seed discovery. This framework provides strong empirical evidence that data-centric, unsupervised quantification can efficiently activate latent knowledge in LLMs, closing the coverage and nuance gaps otherwise perpetuated by existing extraction and synthesis strategies.

The approach also sets a precedent for broader alignment protocols: any domain characterized by "non-universality" (minority dialects, analogical domains, subcultural or technical lexica) is theoretically amenable to analogous unsupervised mining under this framework. Extensions could incorporate adaptive sampling for resource-scarce languages, joint optimization with alignment-phase objectives, or bootstrapped domain adaptation for further cross-cultural robustness.

Conclusion

C-Mining redefines culture-aware data synthesis as an unsupervised geometric mining problem, eliminating reliance on subjective human or LLM proxies and solving for both efficiency and fidelity. The evidence provided by intrinsic topology, expert annotation, downstream benchmarks, and practical cost analyses makes a strong case for quantifiable, topology-based supervision as the foundation for next-generation, culturally aligned LLMs. The method’s scalability and lack of regression on core model intelligence support its adoption as a de facto pipeline for robust, global-scale instruction tuning and cultural deployment (2604.15675).