Pooled Feature Summarization (PFS)

• Pooled Feature Summarization (PFS) is a versatile design pattern that converts raw feature streams into invariant, aggregated representations across various domains.
• It employs diverse pooling methods including temporal pooling, featurewise sort pooling, and global average pooling to capture both local and global dynamics.
• PFS enhances performance in video analysis, set autoencoding, self-supervised learning, and federated preprocessing by tailoring pooled summaries to specific downstream tasks.

Pooled Feature Summarization (PFS) denotes a class of representations in which raw feature streams, set elements, spatial feature maps, or local datasets are converted into pooled summaries that can be consumed by downstream learning algorithms. In the cited literature, PFS appears as temporal pooling over descriptor-element time series in first-person video, featurewise sort pooling for sets, global average-pooled subcrop embeddings in self-supervised learning from naturalistic videos, and aggregated feature-level statistics for federated preprocessing (Ryoo et al., 2014, Zhang et al., 2019, Wang et al., 2024, Xu et al., 11 Feb 2026). A related line of work in supervised video summarization combines multiple feature sources with parallel attention and intermediate fusion, but is explicitly characterized as PFS-related rather than as a classical pooled-descriptor method (Ghauri et al., 2021).

1. General abstraction and scope

Across these works, pooled feature summarization replaces direct use of raw observations with a surrogate representation that summarizes structure at the feature level. The pooled object differs by domain. In PoT, each descriptor dimension is treated as a scalar time series and summarized over temporal intervals (Ryoo et al., 2014). In FSPool, each feature channel is sorted across the elements of a set and aggregated by learned rank weights (Zhang et al., 2019). In PooDLe, the pooled representation is the spatial average-pooled feature vector from a flow-aligned subcrop pair (Wang et al., 2024). In FedPS, clients do not share raw records; they share feature-level summaries such as exact statistics, set unions, sketches, or model sufficient statistics (Xu et al., 11 Feb 2026).

This suggests that PFS is better understood as a design pattern than as a single canonical algorithm. What remains constant is the decision to pool feature-level information in a way that preserves a specific invariance or sufficient statistic while avoiding a less desirable representation: an unordered bag of descriptors in first-person video, a fixed ordering of set elements in set autoencoding, semantically inconsistent global crops in naturalistic video SSL, or raw tabular records in federated learning. The main technical variation lies in what is pooled, which operators are used, and which downstream objective the pooled summary supports.

2. Temporal pooled time series in first-person video

"Pooled Motion Features for First-Person Videos" defines the pooled time series (PoT) representation by starting from a per-frame descriptor

$V^t = [v_1^t, v_2^t, \ldots, v_n^t],$

then interpreting the video as $n$ scalar time series,

$f_i(t) = v_i^t,\quad i=1,\ldots,n.$

The video is partitioned into temporal filters $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$, implemented as a temporal pyramid, and multiple pooling operators are computed for every time series on every interval; the resulting values are concatenated into one fixed-length vector for classification with a nonlinear SVM with a $\chi^2$ kernel (Ryoo et al., 2014).

The representation uses four pooling operators. Standard pooling consists of max pooling,

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$

and sum pooling,

$x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$

PoT then adds two gradient-based operators. The histogram of time series gradients counts positive and negative temporal differences,

$$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$

$$x_i^{\Delta_1^-}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) < 0 \wedge t^s \le t \le t^e\}\right|,$$

while the accumulated positive and negative gradients sum the magnitudes of rising and falling changes. These operators are intended to preserve directional temporal evolution rather than only magnitude or accumulation.

The framework is descriptor-agnostic. The paper evaluates HOF, MBH, OverFeat CNN features, and Caffe CNN features, and emphasizes that PoT is particularly well suited to 4096-D CNN descriptors because it does not quantize each frame into a single visual word. This matters in first-person video because useful information comes from both global egomotion and localized motion or appearance changes. The temporal pyramid lets small intervals emphasize short-term dynamics and larger intervals summarize long-term structure.

Empirically, PoT consistently outperforms BoW and IFV with the same underlying descriptors. On DogCentric, the best PoT with combined descriptors is 0.730, compared with the best IFV with pyramid at 0.666 and a previous state-of-the-art of 0.605. Against first-person adaptations of third-person features, DogCentric results are 0.730 for PoT with temporal pyramid, 0.676 for ITF with temporal pyramid, 0.638 for ITF without pyramid, and 0.692 for ITF + CNN mean features. On UEC Park, the same trend holds, although the paper explicitly notes that the videos are poorly temporally aligned, so adding temporal structure is less helpful and can even hurt. CNN-based appearance descriptors also outperform HOG under the same PoT framework, supporting the claim that PoT is a general summarization method for high-level appearance dynamics rather than only for optical flow (Ryoo et al., 2014).

3. Featurewise sort pooling for set representations

"FSPool: Learning Set Representations with Featurewise Sort Pooling" addresses pooled summarization in set-structured data by sorting each feature dimension independently across the elements of a set and then applying a learned weighted sum over ranks (Zhang et al., 2019). If the input set is

$$X = [x^{(1)},\ldots,x^{(n)}] \in \mathbb{R}^{d\times n},$$

FSPool defines the sorted matrix by

$n$0

with descending order applied separately to every feature row. For fixed-size sets, the pooled output is

$n$1

where $n$2. For variable-size sets, the weights are parameterized as a piecewise linear function $n$3, sampled at rank positions $n$4. Max pooling and sum pooling are identified as special cases of this rank-based weighting scheme.

The paper’s central conceptual claim is that many set autoencoders fail because of the responsibility problem. If a decoder must output a set as an ordered list, then as the input set changes smoothly, the mapping from output positions to target elements must sometimes change abruptly. FSPool is paired with an unpooling or unsorting decoder so that the autoencoder becomes permutation-equivariant rather than only permutation-invariant. The decoder first unpools into a sorted feature matrix and then restores the original arbitrary order using the inverse permutation induced by sorting in the encoder. In the autoencoder setting, the method uses continuous sorting networks to obtain a differentiable approximation to the permutation matrix, at $n$5 time and memory.

The empirical evidence is strongest on the polygon toy dataset. The appendix reports direct MSE essentially at zero for the FSPool-FSUnpool autoencoder: set size 2 gives $n$6, 4 gives $n$7, 8 gives $n$8, 16 gives $n$9, 32 gives $f_i(t) = v_i^t,\quad i=1,\ldots,n.$0, and 64 gives $f_i(t) = v_i^t,\quad i=1,\ldots,n.$1. Under Chamfer loss, the MLP baseline has errors around $f_i(t) = v_i^t,\quad i=1,\ldots,n.$2 across set sizes $f_i(t) = v_i^t,\quad i=1,\ldots,n.$3, while under Hungarian loss it has around $f_i(t) = v_i^t,\quad i=1,\ldots,n.$4. The paper states that for set sizes $f_i(t) = v_i^t,\quad i=1,\ldots,n.$5 and above, the baselines collapse to outputting essentially the same constant set regardless of input, matching the random baseline.

Beyond autoencoding, FSPool improves classification and convergence speed in several settings. On CLEVR, the best reported result is $f_i(t) = v_i^t,\quad i=1,\ldots,n.$6, compared with RN at $f_i(t) = v_i^t,\quad i=1,\ldots,n.$7, sum at $f_i(t) = v_i^t,\quad i=1,\ldots,n.$8, mean at $f_i(t) = v_i^t,\quad i=1,\ldots,n.$9, max at $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$0, and Janossy at $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$1. Replacing GIN readout pooling with FSPool improves results on 6 of 9 TU benchmark datasets, with especially large gains on the Reddit datasets. In DSPN set prediction on CLEVR, DSPN-RN-FSPool reaches AP$\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$2 from $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$3 to $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$4, AP$\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$5 up to $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$6, and AP$\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$7 around $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$8 to $\{[t_1^s, t_1^e], \ldots, [t_k^s, t_k^e]\}$9, far above sum and max pooling. A key limitation is that the decoder depends on encoder-side permutation information, so the autoencoder is not a standalone generative model in the usual sense (Zhang et al., 2019).

4. Pooled embeddings in self-supervised learning from naturalistic videos

"PooDLe: Pooled and dense self-supervised learning from naturalistic videos" uses pooled feature summarization as one of two core self-supervised objectives (Wang et al., 2024). The pooled branch operates on small pseudo-iconic subcrops sampled from paired video frames. Because random crops in dense scenes are often semantically mismatched, the method uses flow-informed local cropping to construct aligned subcrop pairs $\chi^2$0. Each subcrop is encoded with a ResNet-50 backbone, the spatial feature map is average pooled over spatial dimensions, and the pooled embeddings are matched with a BYOL-style invariance loss. The paper is explicit that the pooling operator is global average pooling within each subcrop; it is not masked, does not use region proposals, and is not attention-based.

The pooled-projector path follows BYOL, with a pooled projector described as linear for pooled, while the dense path uses $\chi^2$1 convolutions. The projector hidden dimension is 4096, projecting to 256 dimensions, and the predictor uses the same $\chi^2$2 structure. The method uses online/EMA target networks with momentum starting at 0.996 and annealed to 1, and the pooled and dense objectives use separate projector and predictor modules. The default subcrop design is $\chi^2$3 subcrop pairs, area range $\chi^2$4 of the initial dense crop, and subcrops resized to $\chi^2$5.

The dense branch follows FlowE and enforces equivariance to optical flow warping, while the pooled branch supplies semantic invariance. The paper argues that dense SSL alone can overweight large, repetitive backgrounds, whereas pooled SSL alone can lose fine boundary detail. Their combination is meant to balance spatial precision and category-level understanding, especially in naturalistic videos with many objects, heavy clutter, and severe class or size imbalance. The subcrop strategy is therefore important not only for alignment but also for changing which content is emphasized by the pooled objective: the paper reports that for smaller foreground objects, subcrop hit probability exceeds pixel probability, and that subcrops increase the relative prevalence of small foreground classes in the pooled branch.

The reported ablations are central to the interpretation of PFS in this setting. From Table 5, starting from FlowE, adding pooled loss alone gives almost no gain; adding SDM components alone gives only modest gains; using full SDM improves performance; and adding pooled loss on top of full SDM yields the largest jump. The paper further reports that larger subcrops hurt performance, likely because they include multiple subjects and create false invariances, that increasing the number of subcrops helps with $\chi^2$6 chosen as the default, and that class-group breakdowns show strong gains on small and rare classes such as pole, bicycle, traffic sign, and traffic light. This positions pooled summarization not as an auxiliary regularizer but as a mechanism for learning semantic representations from pseudo-iconic local views (Wang et al., 2024).

5. Aggregated pooled statistics in federated preprocessing

"FedPS: Federated data Preprocessing via aggregated Statistics" recasts pooled feature summarization as a federated preprocessing workflow in which clients never share raw samples but do share enough local summaries for the server to reconstruct global preprocessing parameters (Xu et al., 11 Feb 2026). The framework follows five steps: Compute local statistics on each client; Share and aggregate those statistics at the server; Derive preprocessing parameters from the aggregated summaries; Broadcast parameters back to clients; and Apply preprocessing locally using the shared global parameters. The paper’s motivation is that preprocessing is not optional in FL, yet central preprocessing violates privacy and fully local preprocessing can create inconsistent feature spaces, especially under non-IID data.

FedPS distinguishes several summary types: basic additive statistics, set-based summaries, sketch-based summaries, and model-based sufficient statistics. Aggregation depends on the statistic: summation for additive quantities such as sums, counts, and second moments; min or max reduction for range-based preprocessing; set union for categories; sketch merging for approximate quantiles and heavy hitters; and iterative exchange of sufficient statistics for model-based methods such as k-Means, k-NN, and Bayesian Linear Regression. The summaries are chosen so that they are either exact sufficient statistics or communication-efficient approximations.

For StandardScaler, each client sends $\chi^2$7, $\chi^2$8, and $\chi^2$9, and the server aggregates them to $x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$0, then computes

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$1

after which clients apply

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$2

For MinMaxScaler, clients report local minima and maxima, pooled to global $x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$3, followed by

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$4

For RobustScaler, the key pooled summaries are quantiles $x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$5, obtained via merged quantile sketches, and the transform is

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$6

The paper also supports MaxAbsScaler, Normalizer, LabelBinarizer, MultiLabelBinarizer, LabelEncoder, OneHotEncoder, OrdinalEncoder, TargetEncoder, PowerTransformer, QuantileTransformer, SplineTransformer, Binarizer, KBinsDiscretizer, SimpleImputer, KNNImputer, and IterativeImputer.

Model-based pooled summaries are most detailed for BLR. In horizontal FL, the sufficient statistics are additive: $x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$7 In vertical FL, the method avoids direct use of $x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$8 and instead rewrites the posterior mean as

$x_i^{\max}[t^s, t^e] = \max_{t=t_s..t_e} f_i(t),$9

so that

$x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$0

can be aggregated without cross-feature leakage. The paper presents analogous pooled-sufficient-statistics views for federated k-Means and federated k-NN. It also quantifies communication cost: simple aggregates such as min, max, sum, mean, and variance cost $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$1 per client; set union and categorical summaries scale with the number of distinct categories $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$2; sketches cost about $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$3; and iterative methods incur higher but still structured communication costs. In this formulation, PFS is not a representation for prediction directly, but a privacy-preserving surrogate for a global feature distribution (Xu et al., 11 Feb 2026).

6. Attention-based fusion as a related but distinct usage

"Supervised Video Summarization via Multiple Feature Sets with Parallel Attention" is closely related to PFS ideas because it aggregates and fuses multiple feature representations, but the paper states that it is not exactly a classical PFS pipeline (Ghauri et al., 2021). The proposed method, Multi-Source Visual Attention (MSVA), uses three feature types for each video frame or segment: image content features from GoogLeNet pool5, RGB motion-related features from I3D, and optical flow features from I3D, each with dimension 1024. Videos are uniformly sub-sampled at 2 frames per second. A separate attention module is applied to each feature source in parallel, each attended stream is passed through two linear layers and normalization to produce $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$4, $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$5, and $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$6, and the main variant performs intermediate fusion by summing

$x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$7

before subsequent prediction layers.

The supervised target is frame importance scoring rather than pooled-descriptor classification. The method is evaluated on SumMe and TVSum, and the paper identifies methodological issues in prior evaluation protocols: some videos were excluded from validation entirely, and some were repeated across multiple splits. It therefore provides five new non-overlapping splits with equal division, no repetition, and no exclusion. In ablations, intermediate fusion of all three features is best: on SumMe, object only $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$8 gives 50.5, early fusion of all three gives 46.7, intermediate fusion gives 53.4, and late fusion gives 51.0; on TVSum, the corresponding values are 60.1, 57.3, 61.5, and 60.1. The best aperture size is 250. The full model reports 54.5 $x_i^{\sum}[t^s, t^e] = \sum_{t=t_s}^{t_e} f_i(t).$9 and 53.4 $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$0 on SumMe, with 0.20 Kendall $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$1 and 0.23 Spearman $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$2, and 62.8 $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$3 and 61.5 $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$4 on TVSum, with 0.19 Kendall $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$5 and 0.21 Spearman $$x_i^{\Delta_1^+}[t^s, t^e] = \left|\{t ~|~ f_i(t)-f_i(t-1) > 0 \wedge t^s \le t \le t^e\}\right|,$$6.

The distinction drawn by this paper helps delimit the meaning of PFS. It combines multiple modalities, aggregates them into a common latent representation, and uses that representation to score frames, but it does so through parallel attention and learned intermediate fusion rather than by constructing a single pooled summary descriptor. This suggests a useful boundary condition for the term: not every multi-feature aggregation method is a classical pooled feature summarization method, even when its downstream task is video summarization (Ghauri et al., 2021).