Frequency-Aware Multimodal Transformer

• Frequency-Aware Multimodal Transformer is a model that encodes asynchronous RGB and F/T streams using learned frequency-aware embeddings to preserve fine temporal dynamics.
• The model fuses high-frequency force data with lower-rate visual cues through bi-directional cross-attention, enabling robust performance in contact-rich manipulation tasks.
• Empirical results demonstrate significant improvements in tasks such as gear assembly and battery disassembly, highlighting the impact of frequency-sensitive tokenization and fusion.

Searching arXiv for the specified FMT-related papers and closely related work to ground the article. Frequency-Aware Multimodal Transformer (FMT) most specifically denotes the transformer diffusion policy introduced in ManipForce for contact-rich manipulation, where asynchronous RGB and force/torque (F/T) streams are encoded with frequency- and modality-aware embeddings and fused by bi-directional cross-attention (Lee et al., 23 Sep 2025). In a broader technical reading, the term is also useful for describing transformer designs that treat frequency-sensitive structure as a first-class inductive bias during multimodal or multi-branch fusion. The nomenclature is not uniform across the literature, however: “FMT” also names the “Factorized Multimodal Transformer,” a time-domain multimodal sequential model whose novelty is factorized intra- and intermodal attention rather than frequency-aware processing (Zadeh et al., 2019).

1. Definition and scope

In ManipForce, FMT is defined as a Transformer-based diffusion policy that learns a frequency-aware multimodal representation from asynchronous RGB and F/T streams. Its central design objective is to preserve high-frequency contact dynamics while aligning them with lower-rate visual context, thereby avoiding destructive downsampling of force signals. The model therefore treats frequency awareness not as a post hoc feature-engineering step, but as part of the tokenization and embedding interface presented to the transformer backbone (Lee et al., 23 Sep 2025).

This use of the term should be distinguished from two nearby lines of work. First, the earlier Factorized Multimodal Transformer organizes attention into unimodal, bimodal, and trimodal factors with full time-domain receptive field, but it does not implement frequency-domain processing and explicitly clarifies that its acronym does not imply a frequency-aware model (Zadeh et al., 2019). Second, later work on frequency-sensitive transformers sometimes uses the FMT label only as an interpretive lens. For example, FreqMixFormer is presented as a “Frequency-aware Mixed Transformer” for skeleton action recognition, and its spatial-frequency fusion can be read from an FMT perspective even though the original task is not RGB–F/T manipulation (Wu et al., 2024).

A recurrent misconception is that “frequency-aware” necessarily implies explicit spectral transforms such as STFT, wavelets, or Fourier features. In the ManipForce formulation, that is not the case. Frequency awareness is encoded through learned frequency-aware embeddings and timestamp-based interpolation, whereas explicit cross-modal exchange occurs through bi-directional cross-attention rather than a Fourier-domain attention operator. This distinguishes the model from frequency-domain mixers such as the fixed-size Fourier-based operator used in bioFAME’s frequency-aware transformer encoder (Liu et al., 2023).

2. Problem setting and sensing regime

FMT was introduced for contact-rich manipulation tasks such as gear assembly, LAN plug insertion, box flipping, open lid, battery disassembly, and battery insertion. The motivating observation is that precision assembly and sustained-contact behaviors depend on interaction forces at fine temporal scales, while vision-only imitation learning misses transient force spikes, micro-stick-slip, jamming cues, and subtle changes in normal or tangential force. Cameras operate at 30 Hz, whereas contact events can occur at tens to hundreds of Hz; downsampling high-bandwidth F/T signals to image rate erases transient information that is critical for stable contact and precise alignment (Lee et al., 23 Sep 2025).

The ManipForce system records demonstrations with dual hand-eye cameras, specifically an Intel RealSense D455 and D405 capturing RGB at 30 Hz, together with a wrist-mounted AIDIN AFT200 F/T sensor sampling at 200 Hz. Three-dimensional ArUco markers tracked by an Azure Kinect provide viewpoint-independent wrist and jaw pose. The demonstrations are inherently asynchronous and multi-rate, so timestamps are stored for each modality and used downstream for timestamp-based windowing. The IMU on the D455 provides orientation for gravity compensation, which is applied identically during data collection and robot execution.

The compensated wrench is defined by

$\mathbf{g}_{ft} = R^{ft}_{imu}\,\mathbf{g}_{imu},$

$$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$

and

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$

Empirically, rotating the handheld system by $\pm 90^\circ$ yielded residuals within $\pm 1$ N and $\pm 0.15$ N·m after compensation, which validates consistency between handheld demonstrations and robot execution. This preprocessing step is not merely instrumental; it constrains the F/T stream to encode contact rather than gravity-induced orientation artifacts.

3. Frequency-aware representation and tokenization

FMT tokenizes RGB and F/T streams at their native rates and then aligns them within a shared transformer representation. Each image is square-padded and resized to $256 \times 256$, then encoded by DINOv2-B into $L$ patch tokens per frame. There are two cameras, and each observation window retains $T_{\mathrm{img}} = 2$ frames from each camera, so the visual branch contributes $2 \times L \times T_{\mathrm{img}}$ tokens. The F/T stream is encoded by a 1D CNN over a short horizon, with $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$0 F/T timesteps per window. The reported hyperparameters are $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$1, $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$2, $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$3, and $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$4, yielding 1024 visual tokens, 8 F/T tokens, and 1032 total tokens per window (Lee et al., 23 Sep 2025).

The key alignment mechanism is the combination of spatial, frequency-aware, and modality-aware embeddings. Learned spatial embeddings $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$5 encode within-image token positions. Learned frequency-aware embeddings $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$6 are indexed by force timestamps and linearly interpolated to image timestamps. Modality embeddings $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$7, $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$8, and $$\mathbf{F}_g = m_{tool}\,\mathbf{g}_{ft}, \qquad \boldsymbol{\tau}_g = \mathbf{r}_{com} \times \mathbf{F}_g,$$9 explicitly tag the source modality.

For normalized image timestamps $$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$0 and force timestamps $$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$1, the interpolated image-side frequency embedding is

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$2

with

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$3

The final F/T tokens are

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$4

A technically important clarification is that no spectral transforms are used in this architecture: no STFT, wavelets, or Fourier feature mappings are introduced in the ManipForce FMT. Frequency awareness is therefore implemented through learned embeddings that preserve high-frequency F/T relationships and align them to image frames. This makes the architecture frequency-aware without converting the model into a spectral transformer in the stricter sense used by Fourier-based token mixers.

4. Cross-modal fusion and diffusion policy

After embedding, FMT exchanges information between modalities through bi-directional cross-attention. Using the standard scaled dot-product attention

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$5

the model applies visual queries over force keys and values,

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$6

and force queries over visual keys and values,

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$7

The unified observation tokens are then formed by concatenation, projection, and layer normalization:

$$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$8

The design is symmetric in the sense that vision injects spatial context into the F/T stream, while F/T injects contact dynamics into the visual stream (Lee et al., 23 Sep 2025).

These multimodal observations condition a time-series Transformer diffusion policy. At each denoising step, noisy action tokens cross-attend to $$\mathbf{F}_{comp} = \mathbf{F}_{measure} - \mathbf{F}_g, \qquad \boldsymbol{\tau}_{comp} = \boldsymbol{\tau}_{measure} - \boldsymbol{\tau}_g.$$9 while maintaining causal self-attention across past action tokens. The paper adopts a standard DDPM-style noising process,

$\pm 90^\circ$0

with training objective

$\pm 90^\circ$1

where the conditioning variable is $\pm 90^\circ$2.

The action space is defined by demonstration-derived 6-DOF end-effector pose deltas between consecutive frames from ArUco-tracked wrist poses, transformed to the robot TCP frame via CAD-calibrated transformations. Gripper open/close state is inferred from jaw marker velocities. During execution, denoised action sequences are run through a compliance controller on a Franka Panda. The reported token budget and short observation windows permit real-time execution at 30 Hz control with continuous 200 Hz F/T streaming, although exact latency and memory figures are not given.

5. Empirical behavior and ablations

The evaluation covers six real-world tasks with approximately 100 demonstration episodes per task and 20 trials per task under randomized initial robot and object poses. On these tasks, the RGB-only diffusion-policy baseline achieved success rates of gear 0.35, LAN plug 0.40, box flipping 0.05, open lid 0.20, battery disassembly 0.20, and battery insertion 0.10, for an average of approximately 22%. The full FMT achieved gear 0.95, LAN plug 0.85, box flipping 0.90, open lid 1.00, battery disassembly 0.65, and battery insertion 0.60, for an average of approximately 83% (Lee et al., 23 Sep 2025).

These gains are concentrated in tasks that require precise force modulation and tight clearances. The reported interpretation is that FMT maintains stable contact in non-prehensile manipulation such as box flipping and captures transient force spikes that are critical in open-lid manipulation. Sampling-frequency experiments reinforce this reading: in gear assembly, the success rate is 0.40 at 30 Hz and 0.95 at 200 Hz, with monotonic improvement through 60 Hz and 120 Hz. The stated conclusion is that higher-frequency F/T measurements capture short-duration spikes and subtle torque changes that support smoother alignment and more precise corrective actions.

The ablation studies isolate three components. First, replacing high-frequency F/T with low-frequency F/T at 30 Hz sharply degrades performance in tasks dependent on transient cues and fine alignment. Second, removing frequency-aware multimodal embeddings causes a large drop in gear assembly, which indicates that synchronizing heterogeneous streams through learned frequency-aware embeddings is not incidental. Third, removing cross-attention degrades performance across representative tasks, implying that adaptive fusion rather than simple side-channel conditioning is necessary for robust integration of contact dynamics and spatial context. Exact ablation numbers are not provided, but the full model consistently outperformed all ablated variants.

The reported failure modes remain contact-centric. Vision-only baselines often misalign during insertion or fail to maintain stable contact once visual cues become insufficient. FMT reduces such failures, but can still struggle when contact geometry is extremely tight or when force signatures are ambiguous over very short windows. A plausible implication is that the architecture’s short temporal context, specifically $\pm 90^\circ$3 and $\pm 90^\circ$4, trades longer-horizon reasoning for high-rate local responsiveness.

6. Related architectures, acronym ambiguity, and conceptual boundaries

The literature uses closely related terminology for technically distinct transformer families. The following comparison clarifies the main usages.

Paper	Use of “FMT” or FMT relation	Core mechanism
ManipForce (Lee et al., 23 Sep 2025)	Frequency-Aware Multimodal Transformer	Learned frequency-aware and modality-aware embeddings; bi-directional RGB–F/T cross-attention; diffusion policy
Factorized Multimodal Transformer (Zadeh et al., 2019)	FMT means “Factorized Multimodal Transformer”	Unimodal, bimodal, and trimodal time-domain attention factors with full time receptive field
bioFAME (Liu et al., 2023)	FMT terminology mapped to the frequency-aware transformer encoder	Fixed-size Fourier-based global token mixer independent of input length and sampling rate
FreqMixFormer (Wu et al., 2024)	Discussed from an FMT perspective	DCT-based frequency attention, Frequency Operator, spatial-frequency fusion, temporal attention

The Factorized Multimodal Transformer is the clearest source of acronymic ambiguity. It models language, vision, and acoustics through seven factors, namely $\pm 90^\circ$5, uses full-sequence attention throughout, and reports state-of-the-art results on CMU-MOSI, IEMOCAP, and POM, but it explicitly does not implement frequency-aware modules (Zadeh et al., 2019). In other words, it is an FMT by acronym but not by the later frequency-aware meaning.

bioFAME occupies a different position. Its frequency-aware transformer encoder replaces self-attention with a fixed-size Fourier-based global token mixer,

$\pm 90^\circ$6

and combines this with latent-space masked autoencoding to preserve per-channel frequency components during reconstruction. The reported result is an average improvement of $\pm 90^\circ$7 in classification accuracy over the previous state of the art on unimodal transfer tasks, together with robustness to modality mismatch such as dropout or substitution (Liu et al., 2023). Relative to ManipForce, this is a stronger frequency-domain commitment: the frequency mechanism is inside the token mixer rather than inside the embedding layer.

FreqMixFormer, by contrast, uses explicit DCT/IDCT operations to construct a frequency-aware attention branch alongside a spatial attention branch for skeleton action recognition. Its architecture contains Spatial Attention Blocks, Frequency-aware Attention Blocks, a Frequency Operator that enhances high-frequency coefficients by $\pm 90^\circ$8 and reduces low-frequency coefficients by $\pm 90^\circ$9, and a Temporal Attention Block for global inter-frame correlation. On NTU RGB+D, NTU RGB+D 120, and NW-UCLA, it reports joint-only accuracies of 91.5/96.0 on NTU-60, 87.9/89.1 on NTU-120, and 97.7 on NW-UCLA (Wu et al., 2024). Although this is not a multimodal RGB–F/T policy, the spatial-frequency dual-branch construction suggests a more general architectural pattern in which frequency-sensitive representations are fused with structurally distinct streams.

Taken together, these works show that “Frequency-Aware Multimodal Transformer” is not a single canonical blueprint. One lineage uses learned cross-modal frequency-aware embeddings for asynchronous sensor fusion, another uses explicit Fourier-domain token mixing, and a third uses DCT-based frequency branches inside mixed-transformer designs. This suggests that the common denominator is not a particular spectral operator, but the decision to make frequency structure operational in multimodal or multi-branch transformer fusion rather than leaving it implicit in raw time-domain tokens.