BanglaTextDistinguish Dataset

• BanglaTextDistinguish is a large-scale, human-annotated dataset of 212,184 Bangla news and blog articles spanning seven genres for NLP research.
• The dataset features detailed preprocessing, manual annotation by five experts, and balanced category statistics to ensure wide representativeness.
• Baseline experiments using deep generative models like LSTM-VAE and AAE demonstrate its effectiveness for feature extraction and document classification.

The BanglaTextDistinguish Dataset is a large-scale, human-annotated corpus of Bangla news and blog documents, curated to facilitate research in text mining, feature extraction, and document classification with a focus on Bangla—a language historically underrepresented in computational linguistics resources. Encompassing 212,184 documents across seven genre-diverse categories, it constitutes the largest publicly available collection of its kind and offers detailed preprocessing routines, rigorous annotation methodology, and baseline feature-extraction benchmarks for advanced NLP applications (Rafi-Ur-Rashid et al., 2023).

1. Corpus Composition and Category Statistics

BanglaTextDistinguish comprises 212,184 news and blog articles sourced from a heterogeneous set of Bangla portals and blogs, collected via a custom Python crawler. Data sources were chosen based on Alexa rankings to balance broad popular sites with narrower niche content, maximizing topical and stylistic diversity. Each document was strictly bounded to ≤200 tokens to prevent bias toward a minority of oversized articles, with long pieces split to ensure uniformity.

Documents are distributed across seven mutually exclusive categories, as shown below:

Category	Document Count	Percentage of Corpus
Government & Politics	62,974	29.7%
Science & Technology	11,004	5.2%
Economics	20,922	9.9%
Health & Lifestyle	13,467	6.3%
Entertainment	34,702	16.4%
Arts & Literature	15,214	7.2%
Sports	53,901	25.4%

Mild class skew is observed, especially toward Government & Politics (~30%) and Sports (~25%). The average document length is approximately 175 words after preprocessing. The dataset is stratified by category into training (70%; ≈148,529 documents), validation (10%; ≈21,218 documents), and test (20%; ≈42,437 documents) partitions:

$$|\mathrm{Train}| = \lfloor 0.7\,N \rfloor,\quad |\mathrm{Val}| = \lfloor 0.1\,N \rfloor,\quad |\mathrm{Test}| = \lfloor 0.2\,N \rfloor.$$

2. Data Collection, Annotation, and Validation

The collection pipeline began with exhaustive scraping of article HTML, followed by deduplication to ensure unique instances. Annotation was performed by five trained annotators, each independently labeling articles into one of the seven prescribed categories—Government & Politics, Science & Technology, Economics, Health & Lifestyle, Entertainment, Arts & Literature, or Sports—based on title, body content, URL, and source metadata.

Final labels were established via majority vote aggregation across the five annotators; no automatic or pre-labeling tools were involved in the process, preserving independent human semantic judgment. No quantitative inter-annotator agreement statistic (e.g., Cohen’s kappa) was reported. This methodology ensures the annotated labels reflect consensus human understanding over automated heuristics (Rafi-Ur-Rashid et al., 2023).

3. Preprocessing Pipeline and Data Formatting

A standardized preprocessing pipeline was applied to all documents post-scraping:

1. Unicode normalization and lowercasing: NFC normalization standardizes code point representations.
2. Character filtering: All punctuation, digits, and non-Bangla special characters were removed.
3. Tokenization: Whitespace-based splitting produced lists of Bangla tokens.
4. Stop-word removal: A manually crafted list comprising 430 Bangla stop-words was used.
5. Light stemming: High-frequency morphological inflections for Bangla were collapsed to their base forms.

The following pseudocode summarizes this procedure:

def PREPROCESS(text): text = UNICODE_NORMALIZE(text) text = LOWERCASE(text) text = REGEX_REMOVE(text, "[0–9]|[^\u0980–\u09FF\s]") tokens = SPLIT(text, "\s+") tokens = [STEM(t) for t in tokens if t not in STOPLIST] return tokens

Processed documents were formatted as JSON Lines, with each entry containing unique identifiers, title, cleaned content (as a token list), source, and category.

4. Dataset Access, Licensing, and Directory Layout

The entire corpus is distributed under a “fair use for research” license and is publicly downloadable at https://cutt.ly/SYTV6Pv. The on-disk organization ensures clear provenance and traceability:

/BanglaTextDistinguish/ ├── raw/ # Original scraped HTML ├── processed/ # One-line JSON records │ ├── train.jsonl │ ├── val.jsonl │ └── test.jsonl └── metadata.csv # id, title, source, date, category

Each processed JSONL row takes the form:

{ "id": "000123", "title": "বাংলাদেশে নতুন নীতি…", "content": ["বাংলাদেশে", "নতুন", "নীতি", …], "source": "prothomalo.com", "category": "Government & Politics" }

The structuring facilitates scalable parsing and reproducibility across experiments (Rafi-Ur-Rashid et al., 2023).

5. Feature-Extraction Baselines and Benchmark Experiments

To validate the corpus for representation learning and classification, three deep generative models—LSTM-based variational autoencoder (LSTM-VAE), auxiliary classifier generative adversarial network (AC-GAN), and adversarial autoencoder (AAE)—were implemented over this dataset. Each model operated on document embedding matrices formed by a 128-dimensional embedding layer applied to the first 200 tokens per document.

After training, document representations were compressed to a 32-dimensional latent feature vector using these generative models. These were then used as input to standard document classifiers, including LSTM, Bi-LSTM with attention, CNN, and C-LSTM. Classic baselines such as PCA and the raw embedding were also evaluated.

Across precision, recall, and F₁ metrics on the held-out test set, the adversarial autoencoder yielded the most compact and discriminative feature space for Bangla document classification. No explicit metric results are reported in the source; this experimental protocol benchmarks the utility of BanglaTextDistinguish for representation learning and downstream tasks (Rafi-Ur-Rashid et al., 2023).

6. Research Implications and Usage Guidelines

The scope and annotation fidelity of BanglaTextDistinguish position it as a pivotal resource for advancing Bangla NLP, closing the documented gap in available large-scale, well-annotated corpora for the language. Its preprocessing rules and baseline results establish de facto standards for Bangla text classification pipelines. Users are encouraged to follow the provided train/val/test splits and leverage the distributional statistics and category balances when reporting new results. The explicit stop-list, stemming conventions, and directory structure support robust reproducibility and comparative evaluation.

While the dataset provides rigorous manual class assignments and a spectrum of textual genres, the absence of inter-annotator agreement scores suggests caution when benchmarking tasks sensitive to subtle semantic boundary distinctions. A plausible implication is the need for complementary human studies to quantify annotation reliability longitudinally. The inclusion of generative model baselines further enables the development and assessment of modality-agnostic and unsupervised text representations in low-resource language settings (Rafi-Ur-Rashid et al., 2023).