Balanced Group Softmax (BAGS) for Long-Tail Detection

• BAGS is a method for long-tail object detection that groups object classes based on frequency and replaces the flat softmax with group-wise normalization.
• The technique addresses classifier imbalance by calibrating competition within groups using background and 'others' nodes, ensuring balanced training signals for both head and tail classes.
• Empirical studies on LVIS demonstrate that BAGS improves overall mAP and rare class performance compared to baseline softmax and other re-balancing methods.

Searching arXiv for the specified BAGS papers to ground the article in the cited research. Balanced Group Softmax (BAGS) is a modification of the standard Softmax-based classification head in object detectors, proposed for long-tail object detection where object categories exhibit extreme frequency imbalance. In the original formulation, BAGS replaces the single softmax over $C$ object classes plus background with group-wise training over disjoint frequency groups, together with a special background group and per-group “others” nodes. Its stated purpose is to balance the classifiers within detection frameworks, to ensure that head and tail classes are both sufficiently trained, and to address the classifier imbalance that arises when standard detectors are trained on highly skewed data such as LVIS (Li et al., 2020).

1. Long-tail detection and classifier imbalance

Modern two-stage detectors such as Faster R-CNN and Mask R-CNN typically use a single softmax classifier over $C$ object classes plus background. Under a long-tailed distribution, abundant head-class proposals repeatedly activate and reinforce their corresponding weight vectors, while tail-class weights remain near zero. Empirically, the norm $\lVert w_j\rVert$ of the classifier weight vector for class $j$ grows roughly in proportion to the number of its training instances. As a result, inference strongly favors head classes and almost never predicts low-shot categories (Li et al., 2020).

In the standard setting, the class posterior takes the form

$$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$

When the dataset is extremely skewed, this competition across all classes produces what the paper describes as “classifier imbalance in terms of parameter magnitude.” The underperformance is not framed as a generic class-imbalance problem alone, but as a detection-specific failure mode: direct adaptation of long-tail classification models to detection frameworks does not solve the problem because of the intrinsic difference between detection and classification (Li et al., 2020).

The original analysis also rejects two common remedies as sufficient on their own. Simple re-sampling risks overfitting and slows convergence, while loss re-weighting is described as brittle to hyper-parameters and as struggling to handle the dominant background class. This establishes BAGS as a response to imbalance at the classifier level rather than only at the data-sampling or scalar-loss level (Li et al., 2020).

2. Group formation and group-wise softmax

BAGS divides the $C$ object classes into $N$ disjoint groups $G_1,\dots,G_N$, each containing classes with similar instance counts, and introduces a special background group $G_0=\{\text{background}\}$. Let $\mathbb{N}(j)$ denote the total number of training instances of class $C$0. With pre-defined thresholds

$C$1

group $C$2 contains all object classes $C$3 satisfying

$C$4

On LVIS, the original paper uses $C$5 with $C$6 (Li et al., 2020).

A distinctive element of BAGS is the “others” calibration. To reduce inter-group false positives and allow each group to veto improbable predictions, one “others” node is appended to each object group. If the detection head produces logits $C$7 for $C$8 plus $C$9 “others” logits $\lVert w_j\rVert$0, then the index sets are defined as

$\lVert w_j\rVert$1

For each group $\lVert w_j\rVert$2, the posterior is

$\lVert w_j\rVert$3

This construction restricts direct softmax competition to classes of similar shot level, rather than forcing rare classes to compete against the full head-dominated vocabulary (Li et al., 2020).

A later exposition of the same idea defines a grouping function $\lVert w_j\rVert$4, where $\lVert w_j\rVert$5 if class $\lVert w_j\rVert$6 belongs to group $\lVert w_j\rVert$7, and partitions the Region-of-Interest logits into group-specific subvectors. In that presentation, each RoI activates only the background group and its own frequency group, and the BAGS loss is the sum of cross-entropy terms over those two active groups (Gaba, 2 Sep 2025).

3. Training objective, “others” sampling, and inference

For a proposal labeled as class $\lVert w_j\rVert$8, the original BAGS formulation assigns cross-entropy losses for exactly two groups: $\lVert w_j\rVert$9 and the object group $j$0 containing $j$1. The per-proposal loss is written as

$j$2

where $j$3 and $j$4 are one-hot labels in each group (Li et al., 2020).

Because the “others” category can encompass many classes, its training examples can dominate unless explicitly controlled. The original method therefore samples at most $j$5 times the total number of regular in-group proposals for each group. On LVIS, if $j$6, the method randomly picks up to $j$7 proposals labeled “others” to train group $j$8, with $j$9 reported to work well (Li et al., 2020).

The training loop described in the original paper is operationally simple. For each image mini-batch, the detector generates region proposals, extracts RoI features $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$0, computes logits

$$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$1

for $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$2 together with $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$3 for each group, determines the group containing the ground-truth label, samples “others” proposals subject to the cap, computes the background and group-specific cross-entropies, and backpropagates only the classification head, or optionally part of the backbone (Li et al., 2020).

Inference is likewise group-wise. For each proposal, BAGS computes all group posteriors, discards the “others” entries, and multiplies each foreground class probability by the background-group probability:

$$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$4

Non-maximum suppression and box regression then proceed as usual. This preserves the overall two-stage detection pipeline while replacing only the classifier normalization structure (Li et al., 2020).

4. Theoretical rationale and diagnostic ablations

The theoretical motivation given for BAGS is that restricting competition in softmax to classes of similar shot level prevents head-class weight vectors from suppressing tail-class weights. The per-group normalization is said to equalize gradient magnitudes across groups, so that each class group receives balanced numbers of positive and negative signals irrespective of its global prevalence. The “others” nodes then maintain calibration against background and out-of-group classes, preserving overall precision (Li et al., 2020).

The ablation results clarify several failure modes and common misconceptions. Groups alone are not sufficient: without “others” calibration, grouping hurts head-class mAP and produces many false positives. Adding only the background group yields only a modest gain, whereas adding both background and “others” yields the full benefit. These observations indicate that BAGS is not merely a coarse binning of categories; its calibration mechanism is integral to its reported behavior (Li et al., 2020).

Group count and “others” sampling also matter. On LVIS, $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$5 is reported as best; fewer groups bias the system toward head classes, while more groups starve tail classes. Performance rises as the “others” sampling ratio increases up to approximately $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$6, after which over-sampling “others” degrades tail performance. This suggests that BAGS depends on balancing two forms of competition simultaneously: within-group class competition and between-group rejection (Li et al., 2020).

A later LVIS v1 study reinforces the importance of semantically meaningful frequency grouping. In that work, “Random BAGS (RAGS),” obtained by shuffling categories before binning, causes rare AP to collapse to $$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$7. This result supports the original claim that the grouping should reflect instance-count structure rather than arbitrary partitioning (Gaba, 2 Sep 2025).

5. Empirical performance on LVIS

The original experiments are conducted on LVIS v0.5, a long-tail large-vocabulary benchmark with 1,230 object categories. The validation split is binned into four shot levels: rare ($$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$8 instances), common ($$p_j=\frac{\exp(w_j^T x)}{\sum_{i=0}^{C}\exp(w_i^T x)}.$$9–$C$0), frequent ($C$1–$C$2), and very frequent ($C$3). Reported metrics include overall mAP, $C$4, $C$5, $C$6, and per-bin $C$7 (Li et al., 2020).

Setting	Key result
Baseline softmax, Faster R50-FPN	mAP $C$8
Repeat-Factor Sampling	mAP $C$9
Class-balanced re-weighting	mAP $N$0
BAGS, Faster R50-FPN	mAP $N$1
Baseline, ResNeXt-101-FPN + Faster R-CNN	$N$2 with BAGS
Baseline, ResNeXt-101-FPN + Cascade R-CNN	$N$3 with BAGS

For the Faster R50-FPN detector, the baseline softmax yields mAP $N$4, with $N$5, $N$6, $N$7, and $N$8. Repeat-Factor Sampling raises mAP to $N$9 and gives $G_1,\dots,G_N$0, $G_1,\dots,G_N$1. Class-balanced re-weighting reaches mAP $G_1,\dots,G_N$2, with $G_1,\dots,G_N$3 and $G_1,\dots,G_N$4. BAGS reaches mAP $G_1,\dots,G_N$5, with $G_1,\dots,G_N$6, $G_1,\dots,G_N$7, $G_1,\dots,G_N$8, and $G_1,\dots,G_N$9. The same paper reports that Mask R-CNN instance segmentation similarly improves by approximately $G_0=\{\text{background}\}$0–$G_0=\{\text{background}\}$1 mAP, and states that BAGS beats all state-of-the-art methods transferred from long-tail image classification and establishes new state-of-the-art (Li et al., 2020).

The significance of these results lies in their distributional profile. The largest gains occur in the low-shot bins, while the head categories remain competitive rather than collapsing. This is consistent with the paper’s claim that BAGS improves rare-class behavior without requiring extra sampling for the instances from tail classes (Li et al., 2020).

6. Later extensions in long-tailed detection

A later study on LVIS v1 uses a two-stage Faster R-CNN architecture with a ResNet-50 backbone and MMDetection defaults, and treats BAGS as the starting point for further improvements. In that setting, the baseline Softmax on the validation set gives overall mAP $G_0=\{\text{background}\}$2, rare AP $G_0=\{\text{background}\}$3, common $G_0=\{\text{background}\}$4, and frequent $G_0=\{\text{background}\}$5, while original BAGS with four bins gives overall mAP $G_0=\{\text{background}\}$6, rare $G_0=\{\text{background}\}$7, common $G_0=\{\text{background}\}$8, and frequent $G_0=\{\text{background}\}$9 (Gaba, 2 Sep 2025).

That work investigates several modifications. A five-bin variant, obtained by splitting the $\mathbb{N}(j)$0 range into $\mathbb{N}(j)$1 and $\mathbb{N}(j)$2, reaches mAP $\mathbb{N}(j)$3. “Clustered BAGS,” with thresholds chosen by k-means clustering on the instance-count histogram, gives mAP $\mathbb{N}(j)$4. “Focal-Loss BAGS,” which replaces the per-group cross-entropy by focal loss with $\mathbb{N}(j)$5, reports rare AP $\mathbb{N}(j)$6 and overall $\mathbb{N}(j)$7. “Class-Weighted BAGS,” using exact inverse-count weights normalized within each group, reaches mAP $\mathbb{N}(j)$8, rare $\mathbb{N}(j)$9, and common $C$00. A hybrid strategy that uses class-weighted softmax in the rare and common bins and standard softmax in the head bins yields the best reported overall mAP, $C$01, described as a new state of the art on LVIS in that paper (Gaba, 2 Sep 2025).

The same study also hypothesizes that tail-class features may form smaller, denser clusters within the feature space of head classes, making classification challenging for regression-based classifiers. To address this, it explores metric learning to obtain embeddings that are well-separated across classes and tightly clustered within each class, and uses a k-Nearest Neighbors approach at inference to improve classification performance, particularly for rare classes. These additions are presented as enhancements to the BAGS framework rather than as part of the original 2020 formulation (Gaba, 2 Sep 2025).

7. Related grouped-softmax formulations beyond detection

The grouped-softmax idea also appears in neural text generation under the name “F$C$02-Softmax,” where the vocabulary is partitioned into frequency classes with roughly equal total corpus frequencies and the output distribution is factorized into a class probability and a within-class token probability. In that work, the posterior is written as

$C$03

and training remains maximum likelihood, implemented through two softmax heads (Choi et al., 2020).

The paper explicitly states that all of F$C$04-Softmax’s ingredients can be viewed as one instance of a more general BAGS mechanism. In that mapping, one pre-computes a partition of the vocabulary into balanced groups, replaces the flat softmax over the full vocabulary by a group softmax plus a per-group softmax, and optimizes the sum of the corresponding log-probabilities. The intended effect is analogous to the detection setting: more uniform training signals by confining competition to subsets with balanced frequency mass (Choi et al., 2020).

This does not imply that the detection and language-model variants are identical. The original BAGS for object detection is defined around frequency-grouped object classes, a dedicated background group, and “others” calibration inside a two-stage detector, whereas F$C$05-Softmax is a factorized generation head over a token vocabulary. A plausible implication is that BAGS is best understood not as a single architecture, but as a design pattern for replacing flat softmax competition with frequency-aware grouped normalization when extreme imbalance distorts learning dynamics (Li et al., 2020).