FT-FiLM: Functional Tensor Decoder
- The paper introduces FT-FiLM as a novel decoder formulation that replaces fixed multilinear contraction with latent-state conditioned affine modulation, enhancing expressivity over functional Tucker.
- FT-FiLM leverages mode-wise functional coordinate embeddings combined with a nonlinear readout network to reconstruct fields from sparse, irregular, time-ordered observations.
- Empirical results in StreamPhy demonstrate at least 48% improvement in accuracy and up to 20–100× faster inference compared to diffusion-based methods.
Searching arXiv for the FT-FiLM paper and closely related FiLM/FT references mentioned in the source material. Functional Tensor Feature-wise Linear Modulation (FT-FiLM) is a decoder formulation introduced in StreamPhy for continuous-field reconstruction from sparse, irregular, time-ordered observations of high-dimensional physical dynamics (Chen et al., 8 May 2026). Within StreamPhy, FT-FiLM occupies the decoding stage after a data-adaptive observation encoder and a HiPPO-based state-space model (SSM), and is designed to reconstruct full fields at arbitrary spatial coordinates while preserving the continuous-coordinate structure of functional tensor methods (Chen et al., 8 May 2026). Its defining mechanism is to combine mode-wise functional coordinate embeddings with latent-state-conditioned feature-wise affine modulation and a nonlinear readout network, thereby replacing the fixed multilinear contraction used by functional Tucker decoders (Chen et al., 8 May 2026). The resulting construction is positioned as more expressive than functional Tucker, and StreamPhy reports that this decoder contributes to strong streaming performance, including at least improvement in accuracy and up to faster inference than diffusion-based methods on three physical systems under challenging sampling patterns (Chen et al., 8 May 2026).
1. Origin and problem setting
FT-FiLM was introduced in the context of streaming inference of high-dimensional and multi-modal physical fields from incoming sparse measurements (Chen et al., 8 May 2026). StreamPhy addresses a setting in which observations arrive over time, may be irregularly sampled, and are insufficient on their own to specify the full field without a model that integrates temporal history and spatial structure (Chen et al., 8 May 2026).
The architecture described for StreamPhy has three components: an observation encoder, a structured SSM, and the FT-FiLM decoder (Chen et al., 8 May 2026). The observation encoder converts arbitrary observation sets into a latent representation; the SSM updates a latent temporal state online across irregular time intervals; and FT-FiLM generates field values at arbitrary query coordinates (Chen et al., 8 May 2026). The decoder is presented as the main expressivity bottleneck. A standard linear SSM readout,
is described as too weak for complex multiscale physical fields, while functional Tucker is described as limited by multilinear interactions and by the need for large tensor ranks when representing complicated dynamics (Chen et al., 8 May 2026).
This positioning is important for distinguishing FT-FiLM from earlier FiLM-based mechanisms. In FastSVC, FiLM is used as a practical multi-condition affine modulation mechanism inside a waveform generator, with conditioning streams for sine-excitation and loudness (Liu et al., 2020). In Unified Microphone Conversion, FiLM conditions a CycleGAN-style generator on device frequency-response differences to enable many-to-many microphone mapping (Ryu et al., 2024). In FiLM-Ensemble, FiLM creates member-specific predictors from a shared backbone for epistemic uncertainty estimation (Turkoglu et al., 2022). These systems all use channel-wise affine modulation, but none introduces the functional-tensor decoder formulation and expressivity theorem associated with FT-FiLM (Chen et al., 8 May 2026).
2. Architectural formulation
FT-FiLM preserves the continuous-coordinate premise of functional tensor methods but changes the decoding mechanism (Chen et al., 8 May 2026). For a query coordinate , the decoder first constructs a coordinate embedding by concatenating mode-wise functional embeddings: Each mode-specific function maps a scalar coordinate to a rank-sized latent vector,
and is instantiated using INR-style Fourier features: This embedding step retains the “functional tensor” aspect of the method: coordinates are encoded continuously and mode-wise rather than by discrete lookup (Chen et al., 8 May 2026).
Conditioning then enters through modulation parameters derived from the latent state and observation latent: The appendix-level implementation details specify MLPs over the concatenation of the vectorized state and observation latent:
Here 0 is the augmented SSM state, 1 is the observation latent, 2 is a scaling matrix, and 3 is a bias vector (Chen et al., 8 May 2026).
The final prediction is produced by modulating the coordinate embedding and passing it through a nonlinear readout network: 4 This equation is the core FT-FiLM mechanism (Chen et al., 8 May 2026). Relative to ordinary FiLM, the scaling term is matrix-valued rather than a same-shape per-channel vector, and the output of the affine modulation is consumed by a separate nonlinear readout network rather than being treated as a terminal hidden activation (Chen et al., 8 May 2026). A plausible implication is that FT-FiLM should be understood not merely as “FiLM applied to an existing network layer,” but as a decoder family in which functional coordinate embeddings, state-conditioned affine transformation, and nonlinear decoding are co-designed.
3. Relation to functional Tucker
The most direct comparison in StreamPhy is between FT-FiLM and the functional Tucker model (Chen et al., 8 May 2026). Functional Tucker predicts a scalar at coordinate 5 through a multilinear contraction: 6 where 7 is a core tensor and each 8 is a mode-wise factor function (Chen et al., 8 May 2026). The key structural property is that the output is obtained by linear multilinear contraction with a fixed-rank core tensor.
FT-FiLM removes this fixed multilinear bottleneck. Instead of taking Kronecker-style products of the mode embeddings and contracting them with a core, FT-FiLM concatenates the embeddings, applies a latent-conditioned affine transform, and then uses a nonlinear readout MLP (Chen et al., 8 May 2026). The difference can be summarized as follows.
| Decoder | Coordinate combination | Final mapping |
|---|---|---|
| Functional Tucker | Multilinear contraction with fixed core tensor | Linear in the tensorized feature |
| FT-FiLM | Concatenated functional embedding modulated by 9 | Nonlinear readout 0 |
This architectural change is the basis for the expressivity claim. Functional Tucker remains tied to a fixed-rank tensor function class, whereas FT-FiLM decouples coordinate embedding from latent-state conditioning and from the final nonlinear mapping (Chen et al., 8 May 2026). This suggests that FT-FiLM can represent interactions that are awkward or rank-inefficient under multilinear core-tensor formulations.
The distinction also helps separate FT-FiLM from adjacent but different uses of “FT” and “FiLM” in the literature. LaSAFT extends frequency transformation blocks for conditioned source separation and pairs them with GPoCM, an extension of FiLM involving channel mixing and gating (Choi et al., 2020). That work is highly relevant to conditioned internal transformation, but its “FT” refers to frequency transformation in spectrogram separation, not to functional tensor decoding over continuous coordinates (Choi et al., 2020). FastSVC, GNN-FiLM, Unified Microphone Conversion, and FiLM-Ensemble all stay within standard feature-wise affine modulation families and do not introduce a functional Tucker comparison or a universal-approximation result of this type (Liu et al., 2020, Brockschmidt, 2019, Ryu et al., 2024, Turkoglu et al., 2022).
4. Expressivity theorem
StreamPhy states a formal result that FT-FiLM is strictly more expressive than functional Tucker (Chen et al., 8 May 2026). Let 1, let each 2 be a compact interval with nonempty interior, define 3, fix ranks 4 with 5, let 6, and assume FT-FiLM width 7 (Chen et al., 8 May 2026).
The functional Tucker class is defined as
8
while the FT-FiLM class is defined as
9
The theorem states
0
Accordingly, the closure of FT-FiLM equals the space of all continuous functions on 1, whereas the closure of functional Tucker is a strict subset (Chen et al., 8 May 2026).
The proof sketch has three parts (Chen et al., 8 May 2026). First, FT-FiLM is shown to be dense in 2 by choosing factor functions as coordinate selectors,
3
then choosing 4 and 5 so that the affine map embeds 6 into 7, after which the universal approximation theorem applies to the readout MLP (Chen et al., 8 May 2026). Second, every Tucker model is continuous, so the Tucker class lies inside the closure of FT-FiLM (Chen et al., 8 May 2026). Third, strictness is established with a counterexample. For 8,
9
induces a full-rank matrix on a grid, via a Chebyshev-system argument, whereas any Tucker function induces a matrix whose rank is at most 0; choosing grid size 1 prevents uniform approximation by the fixed-rank Tucker family (Chen et al., 8 May 2026).
Within the paper’s framing, this theorem is the central reason FT-FiLM is not merely a practical alternative. It is a decoder family with a strictly larger approximation class than fixed-rank functional Tucker under the stated assumptions (Chen et al., 8 May 2026).
5. Integration into StreamPhy
FT-FiLM is one stage in a streaming pipeline rather than a stand-alone decoder (Chen et al., 8 May 2026). At time 2, the observation set is
3
The observation encoder maps each coordinate 4 to a functional tensor embedding 5, concatenates this embedding with the observed value 6, projects the result into latent features 7, and uses cross-attention with a learnable query 8 to aggregate the set into
9
(Chen et al., 8 May 2026). This design is described as robust to arbitrary observation patterns and varying numbers of measurements.
The SSM then updates a latent state online: 0 Here 1 is the discretized HiPPO/S4 transition matrix and 2 is an outer-product injection term (Chen et al., 8 May 2026). FT-FiLM uses 3 to generate the modulation parameters 4, which then condition the coordinate embeddings for field reconstruction (Chen et al., 8 May 2026).
The full dataflow is therefore
5
This coupling between temporal state and continuous-coordinate decoder is a defining property of FT-FiLM in StreamPhy (Chen et al., 8 May 2026). A plausible implication is that the affine modulation acts as the interface between temporal inference and spatial reconstruction: the SSM does not directly emit field values, but instead shapes how coordinate embeddings are decoded at each time step.
6. Relation to FiLM and adjacent modulation mechanisms
FT-FiLM belongs to the broader FiLM family in that it uses affine modulation conditioned on auxiliary information, but its structure differs materially from standard FiLM variants (Chen et al., 8 May 2026). Standard FiLM typically applies
6
to an intermediate feature map, with 7 and 8 generated from a conditioning input. This pattern appears directly in FastSVC, where the generator uses
9
to fuse sine-excitation and loudness into the upsampled linguistic stream (Liu et al., 2020). It also appears in Unified Microphone Conversion,
0
where 1 is frequency-response difference information (Ryu et al., 2024); in FiLM-Ensemble,
2
where the conditioning variable is the ensemble index 3 (Turkoglu et al., 2022); and in GNN-FiLM, where the target node computes feature-wise scale and shift vectors for incoming graph messages (Brockschmidt, 2019).
By contrast, StreamPhy’s FT-FiLM does not merely modulate an existing hidden tensor inside a conventional network block (Chen et al., 8 May 2026). Its conditioning object is the concatenated functional tensor embedding of a query coordinate, its modulation is driven by the pair 4, and the modulated representation is passed to a nonlinear readout MLP (Chen et al., 8 May 2026). Moreover, the paper explicitly characterizes FT-FiLM as more expressive than the functional Tucker model and provides a theorem for that claim (Chen et al., 8 May 2026).
It is therefore inaccurate to equate FT-FiLM with any use of FiLM on tensor-shaped data. FastSVC, for example, is described as a special case of standard FiLM with multi-condition additive parameter fusion, not as a full Functional Tensor FiLM formulation (Liu et al., 2020). LaSAFT and GPoCM are closely related in spirit because they combine conditioned transformations and enriched modulation for source separation, but LaSAFT operates through source-attentive frequency transformation and GPoCM through channel mixing and gating, rather than through the functional-coordinate decoder form used in StreamPhy (Choi et al., 2020). FiLM-Ensemble is “functional” in the sense that each ensemble member corresponds to a different function 5, but the paper does not develop a functional tensor decoder or an expressivity result against Tucker-type models (Turkoglu et al., 2022).
A useful conceptual distinction is therefore the following. Standard FiLM methods condition hidden activations within an existing architecture; FT-FiLM conditions continuous coordinate embeddings as part of the decoder itself (Chen et al., 8 May 2026).
7. Empirical behavior and significance
The empirical evidence reported for FT-FiLM in StreamPhy comes primarily from an ablation that replaces FT-FiLM with functional Tucker (Chen et al., 8 May 2026). The “StreamPhy with FTM” variant shows substantially higher VRMSE than the full model across multiple datasets and sampling regimes. Examples reported in Table 2 include:
| Setting | StreamPhy | StreamPhy with FTM |
|---|---|---|
| Turbulent Flow, uniform, 3% | 0.0935 | 0.1435 |
| Ocean Sound Speed, uniform, 1% | 0.0628 | 0.1054 |
| Active Matter, slab, 1% | 0.0905 | 0.1522 |
These numbers are presented as direct evidence that replacing functional Tucker with FT-FiLM improves reconstruction accuracy (Chen et al., 8 May 2026). At the system level, StreamPhy reports at least 6 improvement in accuracy and up to 7 faster inference than diffusion-based methods (Chen et al., 8 May 2026). The speed comparison also includes a case where, for Turbulent Flow at 8 sampling, StreamPhy takes 9 versus 0 for SDIFT, which the source text identifies as roughly a 1 speedup in that case (Chen et al., 8 May 2026).
Implementation details that bear on efficiency are also specified. StreamPhy uses HiPPO-LegS matrices with bilinear discretization,
2
which is described as supporting irregular time intervals and stable online updates (Chen et al., 8 May 2026). The paper also notes that FT-FiLM permits smaller latent spaces than SDIFT, giving as an example latent size 3 for active matter versus 4 for SDIFT’s 5 core (Chen et al., 8 May 2026). During training, observations are randomly masked with a rate sampled from 6, which is reported to improve robustness under diverse sparse patterns (Chen et al., 8 May 2026).
Taken together, these results position FT-FiLM as a continuous-field decoder that is both theoretically broader than functional Tucker and practically effective in online reconstruction of physical dynamics (Chen et al., 8 May 2026). This suggests that its significance lies not only in adopting FiLM-style conditioning, but in combining that conditioning with functional coordinate embeddings and nonlinear readout in a way that is specifically matched to streaming scientific inference.