TAG2CRED Pipeline Overview

• TAG2CRED pipeline is a URL-agnostic framework that uses a four-axis, 35-category rhetorical tag system to annotate Telegram messages based on theme, framing, call-to-action, and evidence.
• It maps tag vectors to calibrated risk scores using regularized logistic regression and Platt scaling, significantly outperforming traditional TF-IDF and SBERT baselines under strict evaluation conditions.
• The approach enhances interpretability and performance through ensemble models, while also addressing limitations such as MBFC-based supervision biases and tag assignment noise.

The TAG2CRED pipeline is a URL-agnostic credibility scoring framework for short-form, high-risk messages, specifically developed for Telegram. Unlike traditional credibility models reliant on source reputation or lexical features—both ineffective for URL-sparse, high-variance social messaging—TAG2CRED leverages a compact, interpetable tag annotation system to capture the rhetorical attributes of messages. This vectorized representation is mapped, via regularized logistic regression, to calibrated risk scores in the $[0,1]$ interval for downstream risk assessment and monitoring. The pipeline significantly outperforms standard TF-IDF and SBERT baselines in domain-disjoint, URL-masked conditions, and demonstrates superior generalization on infrequent domains (Wang et al., 19 Jan 2026).

1. Rhetorical Tag Label System

TAG2CRED replaces high-dimensional lexical and source-domain cues with a four-axis, 35-category “rhetorical tag” ontology. Messages are annotated along the following orthogonal dimensions:

• Theme (single label, 11 categories): Categories include Finance/Crypto, Public health/medicine, Politics, Crime/public safety, News/Information, Technology, Lifestyle/well-being, Gaming/Gambling, Sports, Conversation/Other, and a rare fallback.
• Claim Type / Framing (multi-label, up to 3 from 11 categories): Categories include No substantive claim, Announcement, Speculative forecast, Promotional hype, Scarcity/FOMO, Misleading context, Emotional appeal, Rumour, Opinion, Verifiable factual statement, and a fallback.
• Call to Action (“CTA,” multi-label, 7 categories): Options are Share/repost/like, Engage/Ask questions, Visit external link, Buy/invest/donate, Join/Subscribe, Attend event/livestream, No CTA.
• Evidence / Support (multi-label, 6 categories): Includes None/assertion only, Link/URL, Quotes, Statistics, Chart/diagram, and Other support.

Assignment proceeds according to a codebook (decision order Q1$\rightarrow$Q2$\rightarrow$Q3$\rightarrow$Q4, 51% rule, tie-breakers). A seed corpus of 633 messages was hand-annotated. Qwen-32B was fine-tuned (Supervised Fine-Tuning, 2 epochs) on 560 examples for JSON-formatted tagging and achieved $\sim$88% mean-field F₁ on held-out messages, exceeding off-the-shelf chat models. For inference, greedy decoding (temperature zero) yields a 35-dimensional, sparse multi-hot tag vector for each message.

2. Feature Representation and Risk Mapping

Feature encoding maps each message $m_i$ to $x_i\in\{0,1\}^{35}$, indicating the presence or absence of each tag. Supervision labels $y_i \in \{0,1\}$ are computed from Media Bias/Fact Check (MBFC) continuous domain risk scores: $R_{msg} \leq 0.3 \rightarrow y=0$, $R_{msg} \geq 0.8 \rightarrow y=1$, with messages in the ambiguous interval dropped.

The risk prediction model is logistic regression with $\ell_2$ regularization:

$$L(w) = \sum_{i=1}^N \log\bigl(1 + \exp(-y_i\,w^T x_i)\bigr) + \lambda \|w\|_2^2,$$

mapping features $x_i$ to uncalibrated scores $s_i = w^T x_i$. The sigmoid $\sigma(s) = 1/(1+e^{-s})$ converts $s_i$ to a probability. Final calibration employs Platt scaling,

$$\hat p_{\text{calib}}(y=1|m_i) = \frac{1}{1 + \exp(A\,\sigma(w^T x_i)+B)},$$

with $(A, B)$ estimated on a held-out validation set, optimizing risk calibration as measured by Brier score and Expected Calibration Error.

3. Corpus Construction and Preprocessing

The raw data set consists of 9.37 million messages from 234 public Telegram channels (2015–2025). Preprocessing steps include deterministic cleaning, near-deduplication (fingerprints, MinHash/LSH, SimHash, embedding clustering), URL extraction and canonicalization (public-suffix rules), and URL/domain masking. Only messages with at least one MBFC-rated domain are eligible for supervised training, yielding 87,936 messages. All URLs and domains are removed from text prior to feature extraction. Domain-disjoint splits assign all messages from 20% of MBFC domains to the test set (rotated over ten random seeds).

4. Empirical Performance

Under rigorous URL-masked, domain-disjoint 10-fold evaluation, logistic regression with tag features attains:

• ROC-AUC: $0.871 \pm 0.092$
• macro-F1: $0.787 \pm 0.104$
• Brier score: $0.167 \pm 0.030$

Baseline comparisons (logistic regression, same splits):

Model	ROC-AUC	macro-F1	Brier Score
TF-IDF (URL-masked)	$0.831\pm0.053$	$0.737\pm0.055$	$0.248$
SBERT	$0.848\pm0.070$	$0.748\pm0.086$	$0.152$
Tag2Cred (tags)	$0.871\pm0.092$	$0.787\pm0.104$	$0.167$

Stacked ensemble models, where a meta-learner combines TF-IDF, TAG2CRED, and SBERT outputs, further improve performance:

• TF-IDF + Tag2Cred: AUC = $0.879 \pm 0.086$, macro-F1 = $0.806$, Brier = $0.138$
• TF-IDF + Tag2Cred + SBERT: AUC = $0.901 \pm 0.075$, macro-F1 = $0.813$, Brier = $0.114$

This establishes that rhetorical tag signals are complementary to both bag-of-words and dense semantic embedding-based models.

5. Signal Analysis and Interpretability

Rhetorical tags capture persuasive maneuvers (e.g., scarcity/FOMO, hype, explicit CTA) largely orthogonal to traditional vocabularic or semantic features. Quantitative improvement in discrimination and calibration when these features are ensembled with SBERT or TF-IDF models confirms their additive informational value.

Analysis reveals that CTA tags—most notably "Join/Subscribe" and "Buy/Invest/Donate"—and framing tags such as "Scarcity/FOMO," dominate the weight spectrum in the learned model. Topical theme tags demonstrate lower domain-transfer robustness, indicating greater variability under cross-domain evaluation. Model linearity and the compact 35-feature representation enable direct interpretability of output risk scores: weights can be directly inspected to explain individual message risk (e.g., prominence of investment CTAs plus speculative framing).

6. Limitations and Prospective Directions

TAG2CRED’s dependency on MBFC-based risk supervision imports all related biases or coverage gaps from this external resource. Tag assignment errors (10–15% field-level F₁) introduce noise, though empirical evidence suggests risk scoring degrades gracefully under plausible tag noise levels. The codebook and tag schema have been tailored to Telegram text; extending the approach to multimodal content (images, video) or new platforms will require redesign of the tag system. Identified directions for further work include semi-supervised drift adaptation, expansion of tag schemata to capture emotional tone and moral framing, and operationalization in real-time or user-facing triage systems.