Key Frame Proposal Network (K-FPN)

• K-FPN is a neural model that efficiently selects a sparse, informative subset of keyframes to compactly represent video sequences for downstream tasks.
• It leverages differentiable surrogate losses and specialized architectures, like encoder-LSTM-decoder and lightweight ConvNet, to ensure high reconstruction fidelity and robust pose estimation.
• The model achieves significant performance gains in planning and pose estimation, reducing error rates and inference time compared to uniform or random frame sampling.

A Key Frame Proposal Network (K-FPN) is a neural model that efficiently identifies a sparse, maximally informative subset of keyframes within a video sequence to enable compact sequence representation, predictive modeling, and downstream tasks such as pose estimation and long-horizon planning. K-FPN achieves this by learning to select frames whose content, timing, and visual semantics best summarize or explain the original sequence, allowing subsequent modules to reconstruct, inpaint, or interpolate the missing frames using only these keyframes. This paradigm has been instantiated in both hierarchical generative modeling settings for visual planning (Pertsch et al., 2019) and in efficient video-based pose estimation pipelines (Zhang et al., 2020).

1. Architectural Taxonomy of K-FPN Approaches

Published instances of K-FPN exhibit divergent architectures tailored to their application contexts but share a common objective: sparse, informative frame selection through differentiable, data-driven mechanisms.

Application	Core K-FPN Structure	Downstream Module
Visual Prediction/Planning (Pertsch et al., 2019)	Encoder-LSTM-Decoder (Latent Variable)	LSTM-based Sequence Inpainter
Pose Estimation (Zhang et al., 2020)	Lightweight ConvNet + Sigmoid Selector	Dynamics-based Dictionary Interpolator (HPIM)

In "Keyframing the Future" (Pertsch et al., 2019), the K-FPN is a variational encoder-LSTM-decoder that integrates CNN-based feature extraction, temporal recurrence, and a categorical latent variable predicting frame offsets between keyframes. The selected moments are sampled by their likelihood of being keyframes and passed to the inpainter LSTM.

In the context of pose estimation (Zhang et al., 2020), K-FPN consists of a spatial-temporal lightweight ConvNet applied on deep features from a frozen ResNet backbone. A per-frame weighting vector $\mathbf{s} \in [0,1]^T$ is produced via a sigmoid function, yielding a soft (nearly binary after annealing) mask over temporal positions, thresholded or ranked to obtain the subset of keyframes for explicit pose processing.

2. Keyframe Selection Mechanisms

K-FPNs operationalize keyframe discovery through differentiable surrogate losses that obviate the need for non-differentiable combinatorial selection or REINFORCE-style methods.

In (Pertsch et al., 2019), the offset predictor in the LSTM outputs a discrete probability distribution $\delta^n$ over temporal jumps ($1,\ldots,J$), which are convoluted across predictions to yield a soft distribution $\tau^n \in \Delta^T$ over future timesteps for each keyframe: $$\tau^1 = \delta^1,\quad \tau^n_t = \sum_{j=1}^J \tau^{n-1}_{\,t-j} \delta^n_j$$ The keyframe's visual target is then a soft blend of frames, $\widetilde K^n = \sum_t \tau^n_t I_t$. All operations, including the convolutions and soft mixtures, are differentiable.

In (Zhang et al., 2020), unsupervised selection is driven by the ability to reconstruct all deep features $\mathbf{Y}$ using a small subset indexed by $\mathbf{P}_r$, constrained through a differentiable relaxation of binary frame selection. The loss function: $$\mathcal{L}_{\mathrm{K\!-\!FPN}}(\mathbf{D}, \mathbf{s}; \mathbf{Y}) = \left\| \left[ \mathbf{I} + \rho^{-1} \mathbf{D} \mathbf{D}^T \mathrm{diag}(\mathbf{s}) \right]^{-1} \mathbf{Y} \right\|_F^2 + \lambda \sum_{i=1}^T s_i$$ penalizes reconstruction error and the number of frames selected, imparting sparsity and informativeness.

3. Objective Functions and Differentiable Surrogates

Both representative K-FPN pipelines utilize differentiable, variational, and sparsity-inducing losses tailored to the neural selection process.

• Visual prediction K-FPN (Pertsch et al., 2019): The core objective is an ELBO with additive keyframe reconstruction and inpainting losses. The keyframe loss incorporates pixel-wise MSE between proposed and soft ground-truth keyframes, plus a KL penalty over the latent variable. The inpainting loss aggregates pixel-wise errors over interpolated frames. The total loss: $$\mathcal{L}_{\rm total} = \mathcal{L}_{\rm key} + \mathcal{L}_{\rm inp}$$ enables end-to-end stochastic optimization.
• Pose estimation K-FPN (Zhang et al., 2020): The objective, $\delta^n$0, penalizes failing to reconstruct all deep features from the selected subset via dynamic-dictionary coefficients, subject to an $\delta^n$1-like penalty (induced by $\delta^n$2) promoting sparse selection. The design ensures that only the most essential frames are chosen to explain the temporal evolution.

4. Downstream Reconstruction, Inpainting, and Interpolation

Keyframes proposed by K-FPN are leveraged by differentially powerful downstream modules:

• Hierarchical Generative Reconstruction (Pertsch et al., 2019): A weak LSTM-based inpainter, conditioned on two consecutive keyframe embeddings and their temporal offset, generates candidate frames between them. Soft mixing provides differentiability and full sequence estimation. The architecture prioritizes minimal but critical keyframes due to inpainting limitations.
• Human Pose Interpolation Module (HPIM) (Zhang et al., 2020): After selecting keyframes, a standard 2D pose network is applied to these, and the full sequence of skeletons is reconstructed in closed form using a learned (DYAN) dynamics dictionary: $\delta^n$3 This analytic interpolation ensures both computational efficiency and robustness.

5. Training Protocols and Implementation Details

Both pipelines employ staged training strategies to balance module capacity and decouple keyframe discovery from content interpolation:

• KeyIn (Prediction/Planning (Pertsch et al., 2019)):
  • Stage 1: The inpainter LSTM is trained on pairs of real frames, learning short-horizon interpolation.
  • Stage 2: The keyframe predictor is trained end-to-end with the total loss, while freezing inpainter weights; an auxiliary embedding-reconstruction loss is also used.
  • CNNs consist of 3–4 convolutional layers outputting to a 256-d LSTM; LSTMs are 2 layers, 256 hidden units; optimizer is Adam ($\delta^n$4, $\delta^n$5, $\delta^n$6); key parameters include $\delta^n$7, $\delta^n$8, $\delta^n$9; 100k iterations for the inpainter and 200k for the keyframe predictor.
• Pose Estimation (Zhang et al., 2020):
  • Deep features are extracted using frozen ResNet-$1,\ldots,J$0 (for $1,\ldots,J$1).
  • The K-FPN module and the dictionary (poles) are jointly trained via backpropagation into all parameters.
  • Input videos resized and zero-padded; keyframes are selected via sigmoid weighting and binarization.
  • Pose estimation on keyframes is performed using “Simple Baseline” ResNet-101, with HPIM learned independently.

6. Applications, Performance, and Empirical Properties

K-FPNs have demonstrated gains in planning efficiency, robustness, and computational cost:

• Hierarchical Planning (Pertsch et al., 2019): In visual control tasks (e.g., “pushing” problem), hierarchical K-FPN-based planning improves task completion—final-error reduced by $1,\ldots,J$2 and success rates raised to 64% (vs. 15% flat or 58% fixed-jump methods). Subgoal sequential planning is facilitated by using the discovered keyframes as focal subgoals for a low-level controller, making the approach suitable for long-horizon tasks.
• Pose Estimation (Zhang et al., 2020): On the Penn Action dataset, running the pose estimator only on $1,\ldots,J$3 keyframes within a $1,\ldots,J$4 frame window yields $1,\ldots,J$5 speed-up (to $1,\ldots,J$6–$1,\ldots,J$7 ms per frame, ResNet-34/50) with state-of-the-art accuracy (PCK@0.2 = $1,\ldots,J$8) and robust performance under severe input corruption (3–4% degradation under 60% occlusion, blur, or illumination change). K-FPN substantially outperforms uniform or random sampling in terms of end-pose reconstruction error.

7. Design Factors, Ablations, and Theoretical Implications

Ablation studies indicate notable design sensitivities:

• Backbone Network Depth: Reductions from ResNet-101 to ResNet-18 decrease FLOPs and runtime by $1,\ldots,J$9 with only minor PCK losses.
• Keyframe Budget: Empirically, K-FPN selection outperforms both uniform sampling and random subsets by large margins.
• Online vs. Batch Mode: Online variants (with sufficiently large initial window) yield selection quality close to full-batch approaches.
• Sparsity Parameter: The regularization $\tau^n \in \Delta^T$0 in the selection objective directly manages the number of keyframes; higher values lead to sparser selection.

A plausible implication is that these sparse, hierarchically informative frame sets naturally encode semantically salient transitions and are advantageous for both efficient planning and reconstructive inference in videos.

Summary Table: Key Experimental Results

Dataset & Setting	Method	[email protected]	Time (ms/frame)	# Key Frames
Penn Action (Zhang et al., 2020)	K-FPN+HPIM (R50)	98.0	6.8	17.5 (±4.9)
Penn Action	Simple-Baseline (R50)	97.4	11.3	N/A
sub-JHMDB	K-FPN+HPIM (R18)	94.5	4.7	16.3 (±1.8)

These results substantiate the empirical benefits of unsupervised K-FPNs for both representation efficiency and predictive fidelity in diverse video understanding tasks.