MahaSBERT-STS-v2: Marathi SBERT for STS

Updated 4 July 2026

MahaSBERT-STS-v2 is a Marathi Sentence-BERT fine-tuned on the human-annotated MahaSTS dataset, achieving high regression accuracy for semantic similarity.
The model employs a two-stage training pipeline with NLI pre-training and subsequent STS fine-tuning using a Marathi backbone and MEAN pooling.
It records strong performance metrics with Pearson 0.9600 and Spearman 0.9523, outperforming earlier monolingual and multilingual models on Marathi STS tasks.

MahaSBERT-STS-v2 is a Marathi Sentence-BERT model for sentence-level semantic similarity, implemented as a siamese/bi-encoder network and optimized for regression-style similarity scoring with cosine similarity (Mirashi et al., 29 Aug 2025). In the most recent formulation, it denotes the model fine-tuned directly on MahaSTS, a human-annotated Marathi Sentence Textual Similarity dataset containing 16,860 sentence pairs with continuous labels from 0 to 5 (Mirashi et al., 29 Aug 2025). Earlier L3Cube work released a Marathi STS model under the identifier l3cube-pune/marathi-sentence-similarity-sbert, referred to in text as “MahaSBERT-STS,” and the “v2” designation is used in later descriptions to connect that STS model to the marathi-bert-v2 backbone or, in the 2025 work, to distinguish the MahaSTS-fine-tuned variant from the earlier NLI-only checkpoint (Deode et al., 2023, Joshi et al., 2022).

1. Nomenclature and model identity

The term MahaSBERT-STS-v2 has two closely related usages in the L3Cube literature. In the 2025 work “L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models,” it denotes a Marathi Sentence-BERT model fine-tuned specifically on MahaSTS, with the base encoder initialized from MahaSBERT, which itself had been trained on IndicXNLI using MahaBERT as the underlying backbone (Mirashi et al., 29 Aug 2025). In the earlier L3Cube sentence-embedding literature, the exact string “MahaSBERT-STS-v2” is not used in the paper text; instead, the released STS model is “MahaSBERT-STS,” available as l3cube-pune/marathi-sentence-similarity-sbert, and “v2” is associated with the marathi-bert-v2 backbone (Deode et al., 2023, Joshi et al., 2022).

This naming overlap is central to interpreting the model lineage. The earlier model family distinguishes between MahaSBERT, the NLI-only checkpoint l3cube-pune/marathi-sentence-bert-nli, and MahaSBERT-STS, the two-step NLI+STS model (Joshi et al., 2022). The 2025 paper adds a new task-specific stage by fine-tuning on MahaSTS, a native human-annotated Marathi STS resource rather than only translated STS data, and explicitly states that the v2 fine-tuning on MahaSTS yields notable performance gains over the NLI-only model for sentence similarity (Mirashi et al., 29 Aug 2025).

A common misconception is to treat all references to MahaSBERT-STS-v2 as pointing to a single invariant checkpoint. The record is more specific: the 2022 and 2023 papers describe the released Marathi STS model corresponding to the two-step NLI→STS recipe, whereas the 2025 paper uses the name for the MahaSTS-fine-tuned regression model and benchmarks it on the new dataset (Deode et al., 2023, Joshi et al., 2022, Mirashi et al., 29 Aug 2025).

2. Architectural design and similarity formulation

MahaSBERT-STS-v2 follows the standard SBERT design. Two sentences are independently encoded into dense vectors by a transformer encoder with shared weights, and a pooling operation converts token-level representations into fixed-length sentence embeddings (Mirashi et al., 29 Aug 2025). The underlying Marathi backbone is MahaBERT, released as l3cube-pune/marathi-bert-v2, which is described as a Marathi BERT derived from google/muril-base-cased and further fine-tuned on L3Cube-MahaCorpus and other Marathi monolingual datasets (Mirashi et al., 29 Aug 2025). In earlier L3Cube descriptions, the backbone is identified as L3Cube-MahaBERT v2 trained or fine-tuned on 752M Marathi tokens from L3Cube-MahaCorpus and other Marathi corpora (Joshi et al., 2022).

Pooling strategy is a salient design choice. The 2025 study evaluates CLS, MEAN, and MAX pooling and reports that MEAN pooling consistently performs best; MahaSBERT-STS-v2 therefore uses MEAN pooling (Mirashi et al., 29 Aug 2025). The 2022 and 2023 papers report the same directional conclusion in slightly different terminology, describing AVG or mean pooling as superior to CLS for Marathi monolingual SBERT (Deode et al., 2023, Joshi et al., 2022). The mean-pooled sentence embedding is described in the earlier work as

$e = \frac{1}{T}\sum_{t=1}^{T} h_t$

where $h_t$ are token representations (Joshi et al., 2022).

Sentence similarity is computed using cosine similarity between the two sentence embeddings:

$\hat{s} = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}$

(Mirashi et al., 29 Aug 2025). The STS objective is CosineSimilarityLoss from Sentence-Transformers, i.e., cosine similarity–based regression to gold similarity scores (Mirashi et al., 29 Aug 2025). The 2022 and 2023 descriptions make explicit that this loss is used after the NLI stage in the two-step pipeline, with the target typically treated as a normalized version of the gold STS score (Deode et al., 2023, Joshi et al., 2022). The 2025 paper, however, does not specify label normalization; it states the use of CosineSimilarityLoss but does not detail the normalization (Mirashi et al., 29 Aug 2025).

The embedding dimensionality is documented only in the earlier work: the model inherits the base BERT hidden size, and for L3Cube “base” BERTs this is 768 dimensions (Deode et al., 2023). By contrast, the 2025 MahaSTS paper does not report parameter count (Mirashi et al., 29 Aug 2025).

3. The MahaSTS dataset and its supervision regime

MahaSTS is a human-annotated Marathi Sentence Textual Similarity dataset with 16,860 sentence pairs labeled on a continuous 0–5 scale (Mirashi et al., 29 Aug 2025). The splits are train 14,328, test 1,692, and validation 840, corresponding to an 85:10:5 ratio (Mirashi et al., 29 Aug 2025). Its defining structural property is a uniform bucket strategy: the full dataset is uniformly distributed across six score-based buckets, each containing exactly 2,810 pairs, spanning score 0 through the interval 4.1–5.0 (Mirashi et al., 29 Aug 2025).

The bucket definitions are given explicitly as qualitative supervision guidelines. Bucket 0 corresponds to score 0 and “no semantic similarity”; bucket 1 covers 0.1–1.0 and “minimal similarity”; bucket 2 covers 1.1–2.0 and “somewhat related”; bucket 3 covers 2.1–3.0 and “moderate similarity”; bucket 4 covers 3.1–4.0 and “highly similar”; and bucket 5 covers 4.1–5.0 and “nearly or fully semantically equivalent” (Mirashi et al., 29 Aug 2025). Per split, each bucket contains 2,388 train pairs, 282 test pairs, and 140 validation pairs (Mirashi et al., 29 Aug 2025).

The source corpus is 1 million real Marathi sentences from L3Cube-MahaCorpus (Mirashi et al., 29 Aug 2025). Preprocessing removed sentences that were too short, defined as fewer than 3 words, too long, defined as more than 20 words, non-Marathi sentences, and duplicates (Mirashi et al., 29 Aug 2025). For candidate selection, the team used an earlier SBERT similarity model, l3cube-pune/marathi-sentence-similarity-sbert, to embed the 1M sentences, randomly sampled 5,000 query sentences, and computed cosine similarity against the full corpus (Mirashi et al., 29 Aug 2025). Candidate pairs were then retrieved by cosine-based bins, with one similar sentence per bucket per query across five nonzero similarity buckets, yielding 25,000 pairs before further filtering (Mirashi et al., 29 Aug 2025). Completely dissimilar pairs were assigned to bucket 0, and human annotation refined the labels to the 0–5 gold scores and enforced uniform distribution across buckets; incomplete or nonsensical pairs were discarded, producing the final 16,860 pairs (Mirashi et al., 29 Aug 2025).

The dataset is Marathi in Devanagari script, and non-Marathi content was filtered out (Mirashi et al., 29 Aug 2025). Beyond the corpus-level filtering and SBERT-based retrieval, no special tokenization or text normalization is reported for the annotations or training (Mirashi et al., 29 Aug 2025). Annotators used prior cosine scores as a reference during annotation, but inter-annotator agreement metrics such as kappa or correlation are not reported (Mirashi et al., 29 Aug 2025).

This supervision design is notable because it differs materially from the translated STSb regime used in earlier Marathi SBERT work. The 2022 and 2023 papers trained and evaluated Marathi STS models on machine-translated STS benchmark data, preserving the original STSb split sizes of 5,749 train, 1,500 dev, and 1,379 test per language (Deode et al., 2023, Joshi et al., 2022). MahaSTS replaces that translated-resource dependency with a human-curated, bucket-balanced Marathi dataset (Mirashi et al., 29 Aug 2025).

4. Training pipeline and optimization

In the 2025 formulation, MahaSBERT-STS-v2 is obtained by fine-tuning the base encoder MahaSBERT on MahaSTS for cosine-similarity regression (Mirashi et al., 29 Aug 2025). The optimization setup is concise and explicitly reported: CosineSimilarityLoss, MEAN pooling over token embeddings, AdamW, learning rate $1e^{-5}$ , batch size 8, and 2 epochs (Mirashi et al., 29 Aug 2025). Warmup steps, max sequence length, dropout, random seed, early stopping, hardware, and training duration are not reported (Mirashi et al., 29 Aug 2025).

The broader L3Cube training lineage is two-stage. In the earlier papers, Marathi SBERT training begins with NLI pretraining on translated IndicXNLI using MultipleNegativesRankingLoss, followed by STS fine-tuning with CosineSimilarityLoss on translated Marathi STSb (Deode et al., 2023, Joshi et al., 2022). For the NLI step, the papers report 1 epoch, batch size 4, AdamW, and learning rate $2e^{-5}$ ; for the STS step, they report 4 epochs with AdamW and learning rate $2e^{-5}$ , with batch size 8 in the two-step STS fine-tuning stage (Deode et al., 2023, Joshi et al., 2022). The NLI step treats entailment pairs as positives and contradiction pairs as hard negatives for the same anchor (Deode et al., 2023).

This establishes a three-level progression in the Marathi sentence-embedding line: MahaBERT as the monolingual backbone, MahaSBERT as the NLI-only sentence encoder, and MahaSBERT-STS or MahaSBERT-STS-v2 as the STS-calibrated model (Joshi et al., 2022, Mirashi et al., 29 Aug 2025). A plausible implication is that the 2025 model inherits the semantic structuring induced by IndicXNLI-based NLI training and then calibrates it to native human similarity judgments through MahaSTS fine-tuning, but the 2025 paper specifically frames the improvement relative to the NLI-only checkpoint rather than reproducing the full earlier two-step translated-STSb pipeline (Mirashi et al., 29 Aug 2025).

5. Empirical performance and comparative evaluation

On the MahaSTS test set, the 2025 MahaSBERT-STS-v2 achieves Pearson correlation $r = 0.9600$ and Spearman rank correlation $\rho = 0.9523$ against human labels (Mirashi et al., 29 Aug 2025). The same paper reports a pooling ablation on the MahaSBERT base showing MEAN $>$ CLS $>$ MAX on the test set: CLS yields Pearson 0.9580 and Spearman 0.9503; MEAN yields Pearson 0.9600 and Spearman 0.9523; MAX yields Pearson 0.9532 and Spearman 0.9444 (Mirashi et al., 29 Aug 2025).

The comparative baselines trained and evaluated on MahaSTS show the following test-set results (Mirashi et al., 29 Aug 2025):

Model	Pearson $h_t$ 0	Spearman $h_t$ 1
MahaSBERT-STS-v2	0.9600	0.9523
MahaBERT	0.9483	0.9386
MuRIL	0.9361	0.9267
IndicBERT	0.7311	0.7004
IndicSBERT	0.9515	0.9441

The 2025 paper further reports that fine-tuning on MahaSTS improves over the original NLI-only MahaSBERT baseline, which records Pearson 0.9355 and Spearman 0.9268 (Mirashi et al., 29 Aug 2025). Statistical significance tests are not discussed (Mirashi et al., 29 Aug 2025).

The earlier L3Cube results provide a complementary historical baseline. On translated Marathi STSb test data, the two-step Marathi SBERT model records Spearman correlation 0.83 in both the 2022 and 2023 descriptions (Deode et al., 2023, Joshi et al., 2022). In that setting, the model outperforms multilingual baselines such as LaBSE, MuRIL, mBERT, and IndicBERT, and improves over NLI-only and STS-only Marathi variants (Joshi et al., 2022). The earlier 2023 paper also reports that the monolingual Marathi SBERT outperforms IndicSBERT-STS on Marathi STS, 0.83 versus 0.81, while IndicSBERT-STS is stronger for cross-lingual settings (Deode et al., 2023).

These result sets are not numerically interchangeable because they are measured on different benchmarks: the older figure of 0.83 Spearman is on translated STSb, whereas the newer Pearson/Spearman values are on MahaSTS (Deode et al., 2023, Joshi et al., 2022, Mirashi et al., 29 Aug 2025). What they jointly support is a consistent pattern: Marathi-specific SBERT models improve markedly when trained with task-aligned similarity supervision rather than relying on generic sentence encoders alone.

MahaSBERT-STS-v2 sits within a broader L3Cube program on Marathi and Indic sentence representations. MahaBERT is the monolingual Marathi BERT backbone; MahaSBERT is the Marathi NLI-trained sentence-BERT checkpoint; MahaSBERT-STS is the earlier STS-tuned Marathi SBERT release; IndicSBERT and IndicSBERT-STS are multilingual sentence-BERT models built on MuRIL and trained on aggregated data from ten Indic languages (Deode et al., 2023, Joshi et al., 2022).

The distinction between monolingual and multilingual objectives is methodologically important. The monolingual Marathi models use Marathi-only data and a Marathi-specific base model, which the papers position as the best choice for Marathi-only applications (Deode et al., 2023). IndicSBERT, by contrast, is intended for a common multilingual representation space and is explicitly described as useful for cross-language similarity and mining tasks (Deode et al., 2023). The 2023 paper reports that IndicSBERT exhibits strong cross-lingual capabilities and performs significantly better than alternatives like LaBSE, LASER, and paraphrase-multilingual-mpnet-base-v2 on Indic cross-lingual and monolingual sentence similarity tasks (Deode et al., 2023).

For Marathi-only semantic similarity, however, the monolingual line remains stronger in the reported evaluations. The 2023 paper states that MahaSBERT-STS at 0.83 Spearman outperforms multilingual models on Marathi STS, including IndicSBERT-STS at 0.81 and LaBSE at 0.70 (Deode et al., 2023). The 2025 MahaSTS study reaches a similar conclusion with a different benchmark and a different comparison set: MahaSBERT-STS-v2 outperforms MahaBERT, MuRIL, IndicBERT, and IndicSBERT on the MahaSTS test set (Mirashi et al., 29 Aug 2025).

This suggests a stable division of labor rather than a contradiction. MahaSBERT-STS-v2 is optimized for high-fidelity Marathi semantic similarity, whereas IndicSBERT-type models serve multilingual and cross-lingual alignment needs (Deode et al., 2023). The evidence presented across the papers supports that interpretation without eliminating the value of multilingual encoders in mixed-language retrieval pipelines.

7. Practical use, limitations, and future directions

For practical inference, the model is used through Sentence-Transformers. The 2025 paper provides the model identifier l3cube-pune/marathi-sentence-similarity-sbert-v2, expects Marathi input in Devanagari, and states that no special preprocessing is required beyond standard transformer tokenization; training relied on MEAN pooling over token embeddings (Mirashi et al., 29 Aug 2025). The minimal usage pattern is to encode two Marathi sentences and compute cosine similarity between their embeddings (Mirashi et al., 29 Aug 2025). The earlier papers provide analogous examples for the earlier checkpoint l3cube-pune/marathi-sentence-similarity-sbert (Deode et al., 2023, Joshi et al., 2022).

The stated application profile includes sentence-level semantic similarity for retrieval, clustering, semantic search, paraphrase detection, retrieval-augmented generation, and question answering (Joshi et al., 2022, Mirashi et al., 29 Aug 2025). The 2025 paper notes that strong sentence embeddings directly benefit downstream tasks such as paraphrase detection, semantic search, retrieval-augmented generation, clustering, and question answering (Mirashi et al., 29 Aug 2025).

Several limitations are explicit. The 2025 paper notes reduced generalization for longer or complex or compound Marathi sentences and states that SBERT-style models perform best on short to moderately long inputs (Mirashi et al., 29 Aug 2025). The earlier work emphasizes limitations stemming from synthetic translation noise, domain mismatch, limited topical coverage, lack of bias audits, and uncertain robustness to informal, dialectal, or code-mixed Marathi, especially Romanized Marathi, which earlier preprocessing intentionally removed (Deode et al., 2023, Joshi et al., 2022). In the MahaSTS setting, non-Marathi content is filtered out and no special normalization beyond standard tokenization is reported, so robustness outside clean Devanagari Marathi remains an open question (Mirashi et al., 29 Aug 2025).

Future directions are stated directly in the 2025 paper: develop separate datasets with sentences of varying lengths to improve generalization for complex structures, pursue more task-specific fine-tuning strategies, and continue emphasizing human-curated resources for Marathi and other Indic languages (Mirashi et al., 29 Aug 2025). This suggests a broader research agenda in which native, bucket-balanced supervision complements or supersedes translated STS resources for low-resource languages.