Grid Token-guided Bidirectional Scan (GTBS)

• The paper demonstrates that restricting scanning to a local K×K grid around projected 2D joint anchors significantly enhances fine joint-level feature extraction in MV-SSM.
• GTBS employs bidirectional state-space scanning over a flattened grid to capture contextual cues from both preceding and succeeding tokens, improving occlusion robustness.
• Integrated within the Projective State Space block, GTBS reduces computational cost and yields strong empirical gains, such as notable improvements in AP25 and PCP metrics.

Searching arXiv for the specified paper and related state-space / Mamba context. Grid Token-guided Bidirectional Scanning (GTBS) is a modification of Mamba’s traditional scanning introduced within MV-SSM, a Multi-View State Space Modeling framework for 3D human pose estimation. In MV-SSM, GTBS is integral to the Projective State Space (PSS) block: instead of scanning the entire image token grid or all patch tokens, it restricts computation to a small $K \times K$ grid of tokens sampled around each projected 2D joint anchor, and then applies bidirectional state-space scanning over that localized grid. The method is designed to model fine joint-level spatial arrangements, improve robustness under occlusion, and reduce over-reliance on fixed camera layouts when generalizing to new camera configurations (Chharia et al., 31 Aug 2025).

1. Motivation and design rationale

GTBS is motivated by limitations attributed to traditional Mamba and related SS2D/VSS-style scanning in the MV-SSM formulation. The full-grid or all-patch scan is described as unselective, with three specific consequences: high compute cost from scanning $H \times W$ tokens per view per joint, poor focus on the local limb context around each keypoint anchor, and failure to exploit the fact that multi-view fusion already localizes a small neighborhood per anchor via projective attention (Chharia et al., 31 Aug 2025).

Within that problem setting, GTBS serves as a locality-constrained alternative. It restricts the scan to a small $K \times K$ grid of tokens sampled around each 2D anchor, applies bidirectional state-space scanning over that local grid, and thereby concentrates modeling capacity on the immediate context of each keypoint. This design directly targets fine spatial arrangement modeling at the joint level rather than full-scene token traversal.

A plausible implication is that GTBS reinterprets state-space sequence modeling as a local geometric operator inside a multi-view pipeline, rather than as a generic image-wide token mixer. That implication follows from the explicit coupling between projective localization and subsequent localized SSM scanning.

2. Grid-token construction around projected anchors

The GTBS procedure begins from the current 3D estimate of a joint. For each joint estimate $j$ of person token $n$, MV-SSM projects its current 3D estimate $k_n^j$ into each view $t$ as a 2D anchor $u_{n,j,t}$ (Chharia et al., 31 Aug 2025). Around that anchor, the method defines a fixed $K \times K$ sampling grid of 2D offsets $\Delta_u^m$, where $H \times W$0 and $H \times W$1.

Feature extraction is then localized by deformable/projective attention. Specifically, GTBS samples feature tokens

$H \times W$2

from the view-$H \times W$3 feature map at positions

$H \times W$4

These tokens are concatenated across views $H \times W$5 and joints $H \times W$6 to form a local grid-token sequence of length $H \times W$7 for each joint.

This construction is central to the method’s semantics. The grid tokens are not arbitrary patches: they are anchored to a projected joint hypothesis and are therefore conditioned on the current geometric estimate. The resulting sequence is local, view-conditioned, and explicitly tied to the person-keypoint structure already established in MV-SSM.

3. Bidirectional state-space scanning

After grid-token generation, GTBS flattens the $H \times W$8 local grid into a one-dimensional sequence $H \times W$9 (Chharia et al., 31 Aug 2025). It then performs two state-space scans.

The forward scan proceeds left-to-right in the flattened order:

$K \times K$0

$K \times K$1

$K \times K$2

The backward scan proceeds right-to-left:

$K \times K$3

$K \times K$4

$K \times K$5

The directional outputs are aggregated per token as

$K \times K$6

The set $K \times K$7 is then reshaped back into a $K \times K$8 grid and optionally pooled or projected to obtain a single updated feature for the joint. The final joint feature is written as

$K \times K$9

The paper also presents the continuous-time SSM form prior to discretization:

$j$0

$j$1

In procedural terms, GTBS first performs projective sampling to obtain the grid tokens, then executes forward and backward state-space passes, and finally collapses the aggregated tokenwise outputs into an updated joint feature (Chharia et al., 31 Aug 2025).

4. Relation to unidirectional scanning and full-map traversal

GTBS is explicitly distinguished from unidirectional Mamba and SS2D. In a unidirectional scan, each token is visited in only one order, which biases context capture to a single direction (Chharia et al., 31 Aug 2025). GTBS addresses this by pairing forward and backward passes so that each grid token is contextualized by both its “preceding” and “succeeding” neighbors in the local patch.

It is also distinguished from global image-level scanning. Rather than scanning the full $j$2 map, GTBS operates on a small $j$3 neighborhood around each projected anchor. The difference is therefore not only directional but also spatial: GTBS changes both the ordering mechanism and the support of the scan.

This distinction matters for interpretation. GTBS is not merely a bidirectional variant of Mamba; it is a projectively localized bidirectional scan embedded inside a multi-view estimation loop. This suggests that its contribution lies as much in where scanning is applied as in how scanning is ordered.

5. Integration within the Projective State Space block

In MV-SSM, GTBS is integrated directly into the PSS block. The SS2D sub-block is replaced by GTBS, and GTBS consumes the $j$4 grid-token sequence per joint that has already been localized by projective attention (Chharia et al., 31 Aug 2025). Its output is an updated set of joint-visual features.

These features are then passed through LayerNorm and a small Feed-Forward Network and added residually to the joint’s token embeddings. The geometry branch uses the updated token to predict 2D residual offsets and confidences, and ultimately a new 3D keypoint via differentiable triangulation, identified in the paper as Eq. 7.

The placement of GTBS in this pipeline clarifies its function. It is neither an isolated backbone module nor a post hoc refinement stage. Rather, it forms the local visual reasoning component of the PSS block, operating after projective localization and before geometric update. A plausible implication is that GTBS couples appearance refinement and geometric refinement through an iterative token update mechanism.

6. Reported advantages and empirical evidence

The paper attributes three principal advantages to GTBS (Chharia et al., 31 Aug 2025). First, in spatial arrangement modeling, scanning a local 2D grid around each anchor explicitly captures spatial patterns of limbs and joints, including edges and corners, that global cross-attention may gloss over. Second, for occlusion robustness, even if the central anchor is occluded in one view, surrounding grid tokens can still inform the state space about plausible joint appearance and position. Third, in efficiency and generalization, scanning only $j$5 tokens per joint is much cheaper than $j$6, and the local, view-specific scan does not rely on any fixed global camera layout, so it generalizes more readily to unseen camera counts or arrangements.

The broader MV-SSM abstract reports strong generalization gains over prior methods, including +10.8 on AP25 (+24%) on the challenging three-camera setting in CMU Panoptic, +7.0 on AP25 (+13%) on varying camera arrangements, and +15.3 PCP (+38%) on Campus A1 in cross-dataset evaluations (Chharia et al., 31 Aug 2025). Those results are reported for the full MV-SSM framework rather than GTBS in isolation, but the ablation study directly evaluates GTBS.

From Table 4 in the paper, the CMU Panoptic ablation is as follows:

Variant	AP$j$7	MPJPE
Full MV-SSM (with GTBS)	93.5	15.7
w/o GTBS + Mamba (i.e. neither scan)	87.7	18.6
w/o Mamba (but still GTBS?)	92.3	16.0

The paper summarizes the principal effect as follows: removing the bidirectional scan causes an approximately $j$8 percentage-point drop in AP$j$9 and a $n$0 mm increase in MPJPE. In the reported interpretation, this confirms that GTBS is a critical contributor to MV-SSM’s in-domain accuracy and its generalization to novel camera setups (Chharia et al., 31 Aug 2025).

7. Conceptual significance and common points of clarification

A common point requiring clarification is whether GTBS should be understood simply as “Mamba on local patches.” The formulation in MV-SSM is narrower and more structured than that description suggests. GTBS operates on grid tokens sampled around projected 2D anchors, not on arbitrary local windows, and those anchors are induced by the current 3D joint estimates within a multi-view pipeline (Chharia et al., 31 Aug 2025). The locality is therefore projective and joint-centric rather than generic.

Another point concerns whether GTBS replaces attention globally. The described role is more specific: it replaces the SS2D sub-block inside each PSS block, while projective sampling and subsequent token and geometry updates remain part of the overall MV-SSM architecture. GTBS should therefore be understood as a specialized state-space component inside a broader multi-view estimation framework.

Finally, GTBS is presented as a mechanism for both efficiency and generalization, but those properties derive from the same structural constraint: the scan is confined to immediate visual context around each anchor and is not tied to any fixed global camera arrangement. This suggests that the method’s generalization benefit is linked to its local, view-specific representation rather than to bidirectionality alone.