Block Vector-Based Prediction

• Block vector-based prediction is a methodology that represents data blocks as vectors to capture local structure and enable efficient mapping from context to target outputs.
• It employs diverse techniques—including convolutional, recurrent, and transformer models—to predict block content in video coding, variational inference, and lexical analysis.
• The approach improves efficiency and calibration by decomposing tasks into block-level operations, reducing complexity and enhancing prediction accuracy across applications.

Searching arXiv for recent and foundational papers on block vector-based prediction across coding, generation, mapping, and statistical inference. arXiv search query: "block vector based prediction OR block-based prediction vector representation intra prediction video coding" Block vector-based prediction denotes a family of predictive formulations in which the basic unit of inference is a block and the operative representation is a vector, tensor, or vectorized latent associated with that block. In the cited literature, blocks appear as luminance coding units, latent feature patches, spatial tiles, sentence-flow segments, token groups, and parameter partitions, while prediction ranges from intra-frame partition trees and image blocks to future token blocks, bathymetric patches, lexical arousal labels, and Bayesian posteriors (Jiang et al., 2023, Ren et al., 11 Feb 2025, Fu et al., 3 Mar 2025). The unifying idea is not a single architecture, but a structural decomposition: represent local or grouped structure at block granularity, then predict block content, block labels, block uncertainty, or block dependence using mappings that are more expressive than scalar or pixelwise rules.

1. Conceptual scope and formal structure

In block-based codecs, a natural formulation is a mapping from a context vector or tensor to a target block vector. For HEVC intra prediction, an $N\times N$ block can be vectorized as $\mathbf{y}\in\mathbb{R}^{N^2}$, its reference context as $\mathbf{r}$, and classical directional prediction written as $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$, while neural network modes replace this by $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$ (Sun et al., 2021). In this sense, block vector-based prediction is a direct generalization of conventional mode-dependent interpolation.

The same abstraction appears in semi-autoregressive video generation. Next-Block Prediction partitions a discrete token map into equal-sized blocks and changes the factorization from next-token prediction to blockwise prediction,

$$p(\mathbf{x})=\prod_{k=1}^{T} p(B_k\mid B_0,\dots,B_{k-1}),$$

so that an entire block is predicted in one step rather than one token at a time (Ren et al., 11 Feb 2025). Here the “vector” is the ordered set of tokens in a block.

A related but probabilistic use appears in variational inference. The parameter vector is partitioned as

$$\theta=(\theta_1^\top,\ldots,\theta_M^\top)^\top,\qquad \theta_j\in\mathbb{R}^{d_j},$$

and a dependent block posterior is constructed by preserving multivariate block marginals while coupling them through a vector copula rather than assuming $q(\theta)=\prod_j q_j(\theta_j)$ (Fu et al., 3 Mar 2025). In that setting, prediction is posterior-predictive rather than geometric, but the operative unit remains the block vector.

Across these works, “block” is therefore a modeling primitive rather than a domain-specific object. It may denote a rectangular image region, a latent patch, a token row, a corpus segment, or a multivariate parameter subset. The vector associated with the block may be explicit, as in edge labels or block-presence indicators, or implicit, as in a learned latent or posterior transform.

2. Forms of block representation

A particularly explicit representation appears in AVS3 intra-frame partitioning. Each $64\times 64$ LCU is converted into a $66\times 66$ single-channel luminance input by appending reconstructed top, left, and top-left reference pixels to the current block, and a scalar QP is concatenated before the final fully connected layer. A ResNet-18-like CNN encodes this input into a floating-point vector of length 480,

$\mathbf{y}\in\mathbb{R}^{N^2}$0

where the 480 dimensions correspond to “basic edges” inside the block and can be interpreted as probabilities that those edges are partition boundaries (Jiang et al., 2023). The same work reshapes the 480-vector into $\mathbf{y}\in\mathbb{R}^{N^2}$1 so that local edge correlations can be exploited by the decoder.

In bathymetric super-resolution, the representation is dual. A VQ-VAE encoder maps a low-resolution bathymetry field $\mathbf{y}\in\mathbb{R}^{N^2}$2 to a latent feature map $\mathbf{y}\in\mathbb{R}^{N^2}$3, which is quantized against a codebook $\mathbf{y}\in\mathbb{R}^{N^2}$4, $\mathbf{y}\in\mathbb{R}^{N^2}$5, yielding a grid of discrete vectors $\mathbf{y}\in\mathbb{R}^{N^2}$6. Separately, the output image is partitioned into fixed non-overlapping spatial blocks $\mathbf{y}\in\mathbb{R}^{N^2}$7, and each block is associated with reconstruction statistics and uncertainty. The method is therefore block-vector-based both in latent space, through codebook vectors, and in image space, through blockwise uncertainty variables (Minoza, 19 Apr 2025).

In lexical arousal prediction, the vector is an inverted-index signature over corpus blocks. The text8 corpus is segmented into overlapping blocks $\mathbf{y}\in\mathbb{R}^{N^2}$8, and for each word $\mathbf{y}\in\mathbb{R}^{N^2}$9 a block-segmentation vector $\mathbf{r}$0 is defined by $\mathbf{r}$1 if $\mathbf{r}$2 and $\mathbf{r}$3 otherwise. Synset vectors are averages of member-word vectors,

$\mathbf{r}$4

This produces a high-dimensional sparse representation in which each coordinate is a block ID rather than a lexical co-occurrence feature (Odaka et al., 2022).

Learned block-based image compression uses a different but closely related formulation. The image is partitioned into pixel-domain blocks $\mathbf{r}$5, each block is predicted from reconstructed top and left neighbors by a Contextual Prediction Module,

$\mathbf{r}$6

and the residual block is mapped to a latent block $\mathbf{r}$7 that is quantized and entropy-coded (Wu et al., 2020). Here both pixel blocks and latent blocks are genuine block vectors.

These examples show that block vectors need not be raw pixel vectors. They may be edge-probability vectors, codebook indices, sparse block-occurrence signatures, latent tensors, or transformed parameter blocks. This suggests that the essential design choice is not vectorization per se, but whether the representation preserves the local structure needed for block-level prediction.

3. Predictive mechanisms and blockwise decision functions

The most direct instance of block vector-based prediction in video coding is the encoder-decoder intra-frame partitioning framework for AVS3. A CNN first compresses an LCU into a 480-D block vector, and a Transformer decoder with 4 stacked layers, hidden dimension $\mathbf{r}$8, and 4 attention heads then transcribes that fixed-length vector into a variable-length sequence of partition tokens $\mathbf{r}$9, corresponding to NS, QT, BT-H, BT-V, EQT-H, and EQT-V. A Block Partitioning Constrain module enforces codec-valid outputs by selecting, at each step, the highest-probability mode among the allowed modes under AVS3 constraints such as maximum depth and aspect-ratio rules (Jiang et al., 2023). The predictive target is not a block of pixels but the entire partition tree of the block.

For intra prediction itself, “Fully Neural Network Mode Based Intra Prediction of Variable Block Size” replaces all 35 standard HEVC intra modes by neural network modes. For $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$0 and $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$1 blocks it uses fully connected networks that map a flattened context vector of length $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$2 to an $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$3-dimensional output; for $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$4 and $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$5 blocks it uses CNNs with two directional paths, fusion, and deconvolution. There are 35 dedicated networks per block size, and mode selection remains RD-based over candidate modes (Sun et al., 2021).

The Progressive Spatial Recurrent Neural Network advances the same idea with a spatial recurrence mechanism. It contains three spatial recurrent units and progressively generates predictions by passing information along from preceding contents to the block to be encoded. Horizontal and vertical sequences are modeled with GRU-based recurrences, then fused, and the network is trained with SATD rather than MSE so that the predicted residue is better aligned with transform-coding cost (Hu et al., 2018). In the cited formulation, this remains a block predictor: context is propagated into an entire block rather than into isolated pixels.

Inter prediction admits another variant. In neural-network-based inter bi-prediction blending, the motion compensation stage remains standard, so two motion-compensated prediction blocks $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$6 and $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$7 are still generated from block motion vectors. The innovation is to replace arithmetic averaging or fixed weighted blending by a learned blockwise function,

$$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$8

implemented as a compact fully convolutional network with fewer than 10k parameters (Galpin et al., 2022). This is still block vector-based prediction, but the vector being predicted is the final blended block conditioned on two block predictors.

Learned block-based image compression adds yet another mechanism. Its Contextual Prediction Module predicts the current block from neighboring blocks, transform-codes only the residual block, and then uses a Boundary-aware Postprocessing Module to attenuate block discontinuities (Wu et al., 2020). In this case the predictive map and the residual coding map are both defined at block granularity.

4. Blockwise selection, uncertainty, and dependence

Block vector-based prediction is not limited to deterministic block reconstruction. In MMVC, feature-domain temporal prediction is performed blockwise using several prediction modes: ConvLSTM-based feature prediction, optical-flow-conditioned feature prediction, feature propagation, and skip. For each residual feature block $$\hat{\mathbf{y}}=\mathbf{W}_M\mathbf{r}+\mathbf{b}_M$$9, the encoder selects the mode that minimizes the coding rate after quantization,

$\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$0

over the available mode-specific residual blocks (Liu et al., 2023). The same system then applies block-wise channel removal and a density-adaptive entropy coder that routes blocks through sparse or dense coding paths using a binary density map.

In ocean floor mapping, uncertainty itself is modeled blockwise. For each spatial block $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$1, the method computes a block-averaged absolute error and updates an exponential moving average,

$\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$2

A normalized uncertainty score

$\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$3

then enters the training loss as a block weight, and blockwise prediction intervals $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$4 are produced at inference (Minoza, 19 Apr 2025). The same framework reports CalErr as a calibration error and UWidth as average uncertainty half-width, so block vectors are coupled to statistically calibrated confidence structure rather than only point prediction.

In variational inference, blockwise dependence is made explicit through vector copulas. If $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$5 denotes a multivariate-uniform transform of block $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$6, the joint approximation takes the form

$\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$7

with $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$8 encoding between-block dependence and the marginals $\hat{\mathbf{y}}=f_{\theta_M}(\mathbf{r})$9 retaining tailored within-block structure (Fu et al., 3 Mar 2025). This construction separates within-block dependence from between-block dependence, in contrast to block-independent approximations.

These works collectively show that block vector-based prediction often requires an explicit policy for cross-block interactions. Depending on the application, that policy may be a mode map, an uncertainty map, or a copula. A plausible implication is that once blocks become the modeling primitive, accuracy depends not only on the quality of each block representation but also on how inter-block dependence is parameterized.

5. Representative realizations and reported outcomes

The literature spans coding, generation, mapping, language processing, and inference. The following examples illustrate how the same structural idea leads to different measurable outcomes.

Domain	Block-vector formulation	Reported outcome
AVS3 intra partitioning	480-D LCU vector + Transformer partition-sequence decoder	87.84% reduction in encoding time with 8.09% coding loss vs AVS3 HPM4.0 (Jiang et al., 2023)
HEVC neural intra modes	Mode-specific regressors from context block vectors to target blocks	BD-rate savings of 3.55%, 3.03%, and 3.27% for Y, U, V (Sun et al., 2021)
HEVC PS-RNN intra	Progressive spatial recurrent block prediction trained with SATD	Average 2.5% bit-rate reduction under variable-block-size settings (Hu et al., 2018)
Bathymetry super-resolution	VQ-VAE codebook vectors + block-based uncertainty	SSIM 0.9433, PSNR 26.88 dB, UWidth 0.1046, CalErr 0.0138 (Minoza, 19 Apr 2025)
Semi-AR video generation	Next-block prediction over token rows or frames	FVD 103.3 on UCF101, 25.5 on K600, 8.89 frames per second, 11x speedup (Ren et al., 11 Feb 2025)
Segmentation-assisted VVC inter prediction	Two region-specific motion vectors inside a block defined by translated segmentation masks	Average BD-rate reduction of 0.82%, 0.49%, 0.37% for LDP, LDB, RA (Li et al., 2024)

The quantitative spread is notable. In coding applications, the principal gains are reduced BD-rate or sharply reduced decision complexity. In generation, the main effect is a better quality-speed tradeoff through fewer decoding steps. In bathymetry, the contribution combines reconstruction accuracy with calibrated uncertainty. In segmentation-assisted inter prediction, arbitrary-shaped regions effectively become sub-block vector supports without explicit partition signaling.

For lexical arousal prediction, the reported advantage is not expressed as BD-rate or FVD but as lower $$p(\mathbf{x})=\prod_{k=1}^{T} p(B_k\mid B_0,\dots,B_{k-1}),$$0-normalized Kendall $$p(\mathbf{x})=\prod_{k=1}^{T} p(B_k\mid B_0,\dots,B_{k-1}),$$1. The best block-segmentation variant, NN-BSeg-Gini, improves arousal regression from 0.300 for NN-GloVe to 0.217, while reducing the feature count from 113,384 initial features to 23,029 after Gini-based selection (Odaka et al., 2022). In variational inference, the reported benefit is more accurate posterior approximation than block-independent or factor-based VI methods across four models and 16 datasets, at limited additional computational cost (Fu et al., 3 Mar 2025).

6. Limitations, misconceptions, and directions of development

A common misconception is to equate block vector-based prediction with classical motion-vector processing alone. The cited works show a broader scope. In some cases the block vector is an edge map or latent codebook vector rather than a motion vector; in others it is a token block, a corpus-block indicator, or a parameter partition (Jiang et al., 2023, Minoza, 19 Apr 2025, Fu et al., 3 Mar 2025). Likewise, the prediction target may be a block of pixels, a partition sequence, a posterior density, or a calibrated interval.

Another misconception is that blockwise methods necessarily use fixed, rectangular, and homogeneous supports. The evidence is mixed. AVS3 partition prediction assumes a $$p(\mathbf{x})=\prod_{k=1}^{T} p(B_k\mid B_0,\dots,B_{k-1}),$$2 LCU and a fixed 480-edge encoding, so its representation is tightly tied to a particular grid and codec rule set (Jiang et al., 2023). By contrast, SAIP derives arbitrary-shaped primary and secondary regions inside a VVC block from translated segmentation masks, and Next-Block Prediction uses rows, clips, or other equal-sized token blocks rather than image tiles (Li et al., 2024, Ren et al., 11 Feb 2025). The “block” is therefore a flexible organizational unit, not necessarily a rigid geometric primitive.

The main technical limitations are also block-specific. In AVS3 partition prediction, training, testing, and evaluation are confined to HD sequences, and the authors state that future work will include other resolutions (Jiang et al., 2023). In bathymetry, block size creates an explicit bias-variance tradeoff: smaller blocks give more localized but noisier uncertainty, while larger blocks stabilize uncertainty estimates but may lose fine detail (Minoza, 19 Apr 2025). In Next-Block Prediction, very large blocks reduce the number of decoding steps but degrade generative quality, while medium row-wise blocks give the best FVD-speed tradeoff (Ren et al., 11 Feb 2025). In block-segmentation vectors for arousal prediction, performance depends strongly on block size, overlap, and feature-selection thresholds, and high overlap increases dimensionality and computational cost (Odaka et al., 2022).

Several development directions recur. The AVS3 encoder-decoder is stated to be easily adaptable to the VVC framework with minor modifications (Jiang et al., 2023). The uncertainty-aware VQ-VAE is presented as applicable to atmospheric fields, land surface models, and spatiotemporal fields (Minoza, 19 Apr 2025). Next-Block Prediction motivates variable or hierarchical blocks, although the cited work itself restricts attention to fixed equal-sized blocks and reports that temporal block size $$p(\mathbf{x})=\prod_{k=1}^{T} p(B_k\mid B_0,\dots,B_{k-1}),$$3 is suboptimal (Ren et al., 11 Feb 2025). In lexical modeling, future work includes adding part-of-speech information and richer emotion dimensions (Odaka et al., 2022).

Taken together, these works suggest that block vector-based prediction is best understood as a design strategy for structured prediction: choose a block partition aligned with the problem, encode each block in a vector-compatible form, and then learn or specify the within-block predictor and the between-block dependency mechanism. Where that partition matches the latent structure of the task, the resulting systems can reduce search, improve calibration, or increase throughput without discarding domain constraints.