Contextualized Token Discrimination (CTD)

• Contextualized Token Discrimination (CTD) is a method that integrates tag-dependent attention and fusion mechanisms to refine token classification and enhance language understanding.
• It builds on transformer architectures like BERT by incorporating additional convolutional layers and dual-encoder setups, achieving significant improvements in multi-label tagging and token importance extraction.
• CTD demonstrates practical efficiency with rapid inference and scalability to thousands of tags, applicable in dynamic online Q&A and e-commerce query environments.

TagBERT is the name of two conceptually related but architecturally distinct transformer-based models aiming to improve tag recommendation and token importance extraction in language understanding tasks. The first variant addresses multi-label tag recommendation in online Q&A communities by leveraging a BERT backbone combined with convolutional and dense layers programmed for multi-label classification (Khezrian et al., 2020). The second variant operationalizes a dependency-aware dual-encoder transformer architecture, fusing vanilla BERT representations with attention paths restricted to semantically co-tagged token pairs for extracting critical tokens in e-commerce queries (Kabir et al., 14 Jul 2025). Both designs demonstrate robust empirical gains over their baselines and share the central innovation of making the representation or attention process tag-dependent, but their architectural details, training regimes, and evaluation settings differ according to their respective domains.

1. Model Architectures

1.1 TagBERT for Tag Recommendation (Khezrian et al., 2020)

TagBERT builds upon pre-trained BERT-Base (12 transformer layers, hidden size 768), with input comprising the concatenated title and body of a forum post, prepended by a [CLS] token up to a fixed length $L$ (e.g., 256 tokens). The model extracts the $h_\text{CLS} \in \mathbb{R}^{768}$ embedding for the entire post. Unlike single-label settings, TagBERT processes multi-label outputs for $T$ possible tags.

The architecture extends BERT with stacked 1D CNN layers over token embeddings, using three kernel widths (2, 3, and 4), each with 50 output channels, max-pooled globally. The pooled features from each width ($p_2, p_3, p_4 \in \mathbb{R}^{50}$ each) are concatenated into a single vector $p_{\text{cat}} \in \mathbb{R}^{150}$, then passed through a dense layer ($256$ ReLU units) and a final sigmoid layer of size $T$ for multi-label tag probability outputs. Tags are recommended based on calibrated thresholding ($\tau=0.92$) or top-$K$ selection.

1.2 TagBERT for Token Importance (Kabir et al., 14 Jul 2025)

This variant uses a dual-encoder framework. Each token $s_i$ is processed in parallel by:

• A vanilla BERT encoder (hidden size $h_\text{CLS} \in \mathbb{R}^{768}$0), producing embedding $h_\text{CLS} \in \mathbb{R}^{768}$1.
• A dependency-aware encoder with identical configuration, but whose self-attention is restricted: for a token $h_\text{CLS} \in \mathbb{R}^{768}$2, attention is permitted only across neighboring tokens $h_\text{CLS} \in \mathbb{R}^{768}$3 as defined by a pre-computed (static) or dynamically learned tag-interaction graph.

A fusion gate $h_\text{CLS} \in \mathbb{R}^{768}$4 merges the two streams: $h_\text{CLS} \in \mathbb{R}^{768}$5. The fused embedding is decoded by a linear-softmax classification head into one of three labels per token ("special," "keep," "drop").

2. Tag Dependency Modeling and Integration

2.1 Static and Dynamic Tag Graphs (Kabir et al., 14 Jul 2025)

In the dependency-aware encoder, neighborhood sets $h_\text{CLS} \in \mathbb{R}^{768}$6 are established by either:

• Static graphs: Off-line mined co-occurrence patterns of tag pairs across queries, forming edges for pairs frequently co-occurring above a support threshold, always adding edges for contiguous tokens to ensure graph connectivity.
• Dynamic graphs: Embedded tag-ids and positional encodings are fed through single-head attention to estimate a tag–tag affinity matrix, whose entries are sparsified to yield a runtime-defined communication graph.

Self-attention within the encoder is restricted to these neighbor sets, modifying the canonical transformer's global attention into tag-conditioned local or soft-local attention.

2.2 Token Embedding Fusion

The fusion approach allows each token representation to dynamically gate between standard BERT context and tag-interaction context, with all parameters optimized end-to-end under cross-entropy loss against token-level labels.

3. Training, Optimization, and Data

3.1 TagBERT for Tag Recommendation (Khezrian et al., 2020)

• Backbone: BERT-Base pretrained on BooksCorpus+Wikipedia; full fine-tuning is performed.
• Optimizer: AdamW, weight decay 0.01, batch size 16 (effectively 32 with gradient accumulation).
• Learning rate: warmed up over first 10% of steps to $h_\text{CLS} \in \mathbb{R}^{768}$7, then linear decay.
• Number of epochs: 4; dropout 0.1 on BERT outputs and dense layers, with gradient clipping at norm=1.0.
• Pre-processing: Posts are tokenized, HTML and code removed, lowercased, truncated/padded to length $h_\text{CLS} \in \mathbb{R}^{768}$8.

3.2 TagBERT for Token Importance (Kabir et al., 14 Jul 2025)

• Both encoders: BERT-base scale (12 layers, hidden size 768, 12 heads).
• Tag embedding for dynamic model: dimension 512.
• Optimizer: AdamW, learning rate $h_\text{CLS} \in \mathbb{R}^{768}$9, batch size 32, training for 3 epochs.
• Training objective: token-level cross-entropy loss.

4. Evaluation Datasets and Metrics

4.1 Freecode Tag Recommendation (Khezrian et al., 2020)

• Dataset: 150,000 freecode.com Q&A posts, vocabulary of $T$05,000 tags, average 5–8 tags per post.
• Data split: 140K for training/validation, 10K test.
• Metrics: Precision@K, Recall@K, F1@K for $T$1 tags per post.

4.2 E-Commerce Token Extraction (Kabir et al., 14 Jul 2025)

• Dataset: 10 million eBay queries, each 3–7 tokens, with tags derived from eBay's pipeline.
• Data split: 6M train, 2M dev, 2M test.
• Metrics: Token-level F1, Token-level Accuracy, Exact-Match Accuracy.

5. Empirical Performance and Comparative Results

5.1 Tag Recommendation (Khezrian et al., 2020)

On the freecode test set, TagBERT achieves:

Model	Precision@10	Recall@10	F1@10
TagBERT	40.3%	64.4%	46.5%

In comparison, the next-best model (TagCNN) yields F1@10 of 45.3%. Notably, TagBERT maintains precision stability as $T$2 increases: Precision@5 = 41.8%, Precision@10 = 40.3% (∼1.5% drop), whereas all baselines suffer significant precision decay.

5.2 Token Importance Extraction (Kabir et al., 14 Jul 2025)

• TagBERT-Dynamic: F1 = 0.83 (a 6.0% absolute improvement over BERT), Exact-Match = 0.37 (+27% vs. BERT), Token-Level Acc = 0.76 (+11.1%).
• Baseline BERT F1: 0.783; best previous non-TagBERT model (eBERT TT+Gated): 0.809.
• Token-level F1 remains stable (0.82–0.85) as query length grows, though Exact-Match falls for longer queries.

6. Ablation, Error Analysis, and Practical Considerations

6.1 Ablation and Component Impact (Khezrian et al., 2020)

Removing the CNN head reduces F1@10 by 2.1%; excising the dense layer reduces F1@10 by 1.7%. Freezing BERT leads to an 8.5% absolute drop in F1@10. Manipulating the decision threshold $T$3 trades recall for precision.

6.2 Error Profiles

False positives typically involve generic tags (e.g., "java", "python") suggested for more specialized content; false negatives are associated with infrequent tags ($T$4 training examples).

6.3 Practicalities

• TagBERT (tag recommendation): Training ≈ 3 hours (Tesla V100 GPU, 32GB RAM), inference ≈ 15 ms/post (GPU) or 60 ms/post (CPU). Model size ≈ 420MB.
• TagBERT scales to $T$5 tags via candidate preselection (co-occurrence/TF–IDF) or two-stage filtering.
• Model export supported via TensorFlow or ONNX.

7. Significance and Key Innovations

TagBERT for tag recommendation establishes high precision and robust stability against increasing candidate sizes, outperforming prior neural and deep learning approaches on multi-label Q&A tagging. For token importance in e-commerce queries, TagBERT demonstrates a principled integration of semantic tag-relationships into transformer attention—either statically or dynamically—combined with gated dual-stream fusion, yielding consistent improvements across diverse baseline architectures, including general and domain-adapted BERT, sequence-to-sequence models, and two-tower networks. The tag-conditioned attention and fusion mechanisms represent the central methodological advance, supporting application across tagging, token selection, and domain-specific semantic parsing (Khezrian et al., 2020, Kabir et al., 14 Jul 2025).