---
title: Semantic Recommendation of Subhāṣitas
url: https://www.emergentmind.com/topics/semantic-recommendation-of-subhasitas
type: topic
---

# Semantic Recommendation of Subhāṣitas

Semantic recommendation of subhāṣitas refers to the application of semantic retrieval and recommendation techniques, leveraging sentence embeddings, vector search, and language models, to match Sanskrit subhāṣita verses (concise, philosophical aphorisms) to user queries not by keyword overlap but by underlying meaning, intent, or contextual relevance. This approach addresses linguistic, cultural, and accessibility barriers, enabling both scholarly study and practical use of the classical corpus through modern AI and information retrieval methodologies [2601.06607], [2410.09560].

## 1. Dataset Construction and Annotation

Semantic recommendation systems for subhāṣitas require high-quality, thematically annotated corpora. In the Pragya framework, a dataset of 200 verses was curated from canonical classical Indian sources (e.g., Hitopadeśa, Pañcatantra) [2601.06607]. Each verse was:

- **Transcribed in Devanagari script.**
- **Accompanied** by a Marathi translation and a simplified English translation, enhancing cross-lingual accessibility.
- **Expert-annotated** with mood or theme tags (e.g., motivation, friendship, compassion, courage), allowing theme-aware retrieval.
- **Stored** in a structured CSV schema: Sanskrit verse, Marathi translation, English translation, theme tags.

This structured, annotated resource is foundational for any system seeking to retrieve not just textually but semantically similar or contextually relevant verses.

## 2. Embedding Computation and Semantic Indexing

Semantic representations were generated using the IndicBERT transformer, a multilingual model trained for Indian languages and adopted in "sentence embedding" mode [2601.06607]. The workflow involves:

- **Text normalization** and tokenization of the Sanskrit verse or query.
- **Embedding extraction:** The [CLS] token embedding (768 dimensions) serves as the fixed-length semantic representation.
- **Similarity measurement:** Cosine similarity between query and verse embeddings quantifies semantic alignment:
  $$
  \operatorname{cosine\_sim}(e_q, e_v) = \frac{e_q \cdot e_v}{\|e_q\| \cdot \|e_v\|}
  $$
- **Indexing:** All corpus embeddings are ingested into a FAISS FlatL2 index for efficient nearest-neighbor retrieval.

An alternative approach under development leverages large language model (LLM)-based embeddings, further quantized and structured via multi-codebook vector quantization à la Mixture-of-Codes (MoC) [2410.09560]. This enhances discriminability and embedding robustness for large-scale recommendations.

## 3. Retrieval and Recommendation Pipeline

The recommendation system architecture comprises distinct retrieval and generation modules [2601.06607]:

- **Retrieval:** On query receipt, its embedding is compared (cosine metric) against the indexed corpus, retrieving the top-$k$ nearest verses (typically $k=3$ in Pragya).
- **Post-processing:** No hard similarity threshold; always return the $k$ best matches. Coverage is computed as the fraction of queries for which at least one result shares the intended theme.
- **Scalability (MoC):** With Mixture-of-Codes, each verse's semantics are distributed across $N$ parallel codebooks (VQ-VAE), yielding a high-dimensional semantic ID while avoiding spectrum collapse and supporting fast approximate nearest-neighbor search [2410.09560].

The table below summarizes retrieval performance reported in Pragya:

| Metric                                 | Keyword Search | Pragya (RAG)  |
|-----------------------------------------|---------------|---------------|
| Top-3 Precision                        | 45 %          | 72 %          |
| Coverage (≥1 relevant in top 3)         | 60 %          | 82 %          |
| Latency per Query (seconds)             | 0.5           | 1.2           |
| User Satisfaction (1–5 scale)           | 2.8           | 4.3           |

Semantic retrieval notably increases both precision and semantic coverage relative to traditional keyword search.

## 4. Generation and Explanation Module

A distinguishing feature is retrieval-augmented generation (RAG): after semantic retrieval, a local Mistral LLM (via Ollama) is invoked to enhance user accessibility [2601.06607]:

- **Inputs:** The target Sanskrit verse, theme metadata, provided translations, and user query.
- **Outputs:**
  1. **Transliteration** (IAST or Latin script).
  2. **Fluent English translation.**
  3. **Contextualized explanation:** A motivational, modern paraphrase linking the verse's theme to the user's context.

Comparison of generated explanations:

| Metric                        | Dictionary Translation | Pragya (Mistral) |
|-------------------------------|-----------------------|------------------|
| Clarity (1–5)                 | 2.5                   | 3.4              |
| Cultural Appropriateness (1–5) | 3.0                   | 4.2              |
| Engagement (1–5)              | 2.2                   | 4.1              |
| Overall Score (avg)           | 2.6                   | 4.4              |

The LLM based generation module significantly improves clarity, cultural resonance, and engagement compared to dictionary-style outputs.

## 5. Advanced Embedding Techniques: Mixture-of-Codes Approach

Scaling classical embedding approaches to large corpora while maintaining semantic granularity and computational tractability is addressed by the Mixture-of-Codes (MoC) paradigm [2410.09560]:

- **Multi-codebook quantization:** High-dimensional LLM embeddings ($z \in \mathbb{R}^{n_z}$) are projected into $N$ low-dimensional codebooks, each with $K$ codewords.
- **ID construction:** Each verse is tagged by $N$ discrete code indices, preserving a distributed semantic representation.
- **Fusion module:** Embeddings from all codebooks are concatenated and processed by a residual MLP bottleneck ("fusion") prior to downstream recommendation (e.g., within DeepFM or AutoInt+).
- **Loss functions:** VQ-VAE-style loss ($L_{rec} + L_{commit} + L_{codebook}$) ensures faithful quantization and diversity of semantic subspaces; task-specific losses supervise downstream recommendation.
- **Retrieval:** Precomputed, fused verse embeddings allow scalable nearest-neighbor and ranking operations for user or context-specific queries.
- **Storage and computation:** Embedding size grows linearly with $N$; quantization and retrieval remain efficient for large $K$ and corpus sizes; periodic retraining maintains adaptation to corpus or model drift.

This approach supports high-dimensional, information-rich semantic matching without the discriminability loss inherent in brute-force projection or single-codebook compression.

## 6. Evaluation, Limitations, and User Studies

Evaluation of semantic recommendation frameworks encompasses both traditional IR metrics and user-centered measures [2601.06607]:

- **Retrieval:** Top-3 Precision@3, Coverage@3, Recall@k, MRR.
- **Explanation Quality:** Scored by users on clarity, cultural appropriateness, engagement, and overall.
- **User studies:** Encompass a demographically mixed cohort (Sanskrit, Marathi, English speakers); tasks include thematic querying and utility rating.
- **Key findings:** Semantic retrieval via IndicBERT + FAISS outperformed keyword search in both system-level metrics and user satisfaction (2.8 → 4.3/5 Likert scale).
- **Generation:** LLM-generated explanations rated ~4.0+ versus <2.6 for literal translations.
- **Usability:** Users noted bridging of the gap between classical content and contemporary motivational or ethical advice.

Limitations include the small dataset (200 verses), absence of domain-specific model fine-tuning, and constrained user study scope. No recall@k or large-scale A/B testing has yet been performed; current deployment relies on local hardware with moderate latency and basic TTS.

## 7. Future Directions and Research Opportunities

The field of semantic recommendation for subhāṣitas is at a nascent stage, with several promising research directions [2601.06607], [2410.09560]:

- **Dataset expansion:** Scaling verse annotations to thousands, incorporating additional Indian and global languages, and enriching theme tagging.
- **Adaptive embeddings:** Fine-tuning LLMs and embedding models on subhāṣita-specific corpora, or integrating multilingual encoders (e.g., LaBSE, mBERT) for cross-lingual capability.
- **System scalability:** Leveraging Mixture-of-Codes and advanced vector quantization to support web-scale retrieval and robust discriminability.
- **Multimodal integration:** Adding automatic speech recognition (ASR), high-quality TTS, and multimodal (image/text/audio) recommendation.
- **Comprehensive evaluation:** Systematic user studies, recall@k and MRR benchmarking, and A/B tests over diverse populations.
- **Deployment:** Development of mobile, web, and edge-computing applications for public engagement, education, and heritage preservation.

A plausible implication is that as annotated corpora and semantic embedding models mature, domain-specific fine-tuning and scalable MoC-based representations will further enhance the accessibility, relevance, and impact of subhāṣita recommendation systems, contributing to the digital preservation and revitalization of classical Sanskrit literature.

Source: https://www.emergentmind.com/topics/semantic-recommendation-of-subhasitas