Pixel-wise Barlow Twins for Scalable EO

• The paper introduces pixel-wise Barlow Twins to learn invariant representations from annual Sentinel-1/2 pixel time series while reducing redundancy in embedding dimensions.
• It employs a novel view construction method using temporal subsampling and mix-up consistency to maintain full time-series structure for effective pretraining within TESSERA v2.
• The study reveals that pretraining loss poorly predicts downstream performance, emphasizing that model selection should be driven by actual Earth observation task utility.

Pixel-wise Barlow Twins is a pixel-wise self-supervised representation learning scheme for Sentinel-1/2 time series in Earth observation (EO), formulated in TESSERA v2 as a controlled family for studying how pixel-wise EO foundation models scale under fixed pretraining budgets (Feng et al., 4 Jul 2026). In this setting, each training example is a single $10$ m pixel’s annual Sentinel-1 and Sentinel-2 observation sequence, with masks indicating valid timesteps, and training constructs two temporally subsampled views of the same pixel-year. The objective is to learn representations that are invariant across these views while reducing redundancy across embedding dimensions. In TESSERA v2, the method serves a dual role: it is both a practical pretraining recipe and a laboratory for analyzing compute allocation, model selection, and distillation for deployment.

1. Pixel-wise formulation and EO-specific rationale

Pixel-wise Barlow Twins departs from crop-based image self-supervision by operating on annual per-pixel time series rather than on image patches or global composites. For each pixel-year, Sentinel-1 and Sentinel-2 observations are preserved as temporally structured sequences, and masks indicate which timesteps are valid. Two temporally subsampled “views” of the same pixel-year are formed at training time, and the encoder is trained to produce representations that are invariant across these views (Feng et al., 4 Jul 2026).

The method is used because Barlow Twins encourages invariance between two views of the same example, redundancy reduction across representation dimensions, and no need for labels or negative pairs. In EO, that combination is attractive because cloud cover, irregular revisit patterns, and modality sparsity create large input variability that is often irrelevant to downstream tasks. The redundancy-reduction objective is therefore positioned as a way to ignore nuisance variation without supervised labels or contrastive negatives.

TESSERA v2 emphasizes that this choice is motivated by the structure of EO data rather than by an assumption that pretraining loss is itself a reliable target. The paper explicitly states that the pixel-wise Barlow Twins family is used as a controlled laboratory for scaling, not because loss minimization is the ultimate goal. A central caveat follows from this framing: in EO, minimizing the redundancy-reduction loss does not reliably track downstream utility. This directly challenges a common assumption carried over from other domains, namely that better pretraining loss necessarily implies better downstream transfer.

2. View construction and objective functions

The teacher reuses the pixel-wise Barlow Twins-style objective introduced in earlier TESSERA work, but extends it with multi-scale temporal subsampling and additional consistency regularization. Training uses two randomly subsampled annual views per pixel, with view length $L \sim \mathrm{Uniform}\{8,16\}$. The stated purpose is to force the encoder to infer annual phenology from sparse observations while preserving the full time-series structure of EO rather than collapsing the input into cloud-free composites (Feng et al., 4 Jul 2026).

The core pretraining loss is the standard Barlow Twins redundancy-reduction objective on the cross-correlation matrix $C$ between projected embeddings of the two views:

$$\mathcal{L}_{\text{BT}} = \sum_i (1 - C_{ii})^2 + \lambda \sum_{i \neq j} C_{ij}^2.$$

In TESSERA v2, this objective is combined with two practical modifications: random temporal subsampling with $L \in \{8,16\}$, and a mix-up consistency regularizer. The paper also states that the main pretraining objective includes global shuffling.

For distillation, the formulation changes. Each student embedding $s$ is trained against a frozen teacher embedding $t$ with a Matryoshka-style loss:

$$\mathcal{L}(s,t) \;=\; \sum_{k \in \mathcal{K}} \Bigl(1 - \cos\!\bigl(h_k(s_{1:k}),\, t\bigr)\Bigr), \qquad \mathcal{K}=\{16,32,64,128\}.$$

Here, the goal is not to reproduce the teacher’s first $k$ coordinates literally. Instead, each prefix $s_{1:k}$ is trained to act as a shorter approximation of the same semantic representation. The paper reports an important negative result: naive attempts to make Matryoshka-style nested prefixes directly during self-supervised pretraining do not work well. The stated reason is that redundancy-reduction objectives identify a latent subspace only up to rotation, so there is no canonical coordinate ordering for “first 16 dimensions” versus “first 128 dimensions.” Coordinate ordering is imposed later by distillation from a frozen teacher.

3. Encoder, projector, and student heads

The teacher is a dual-branch pixel-wise temporal encoder in which Sentinel-1 and Sentinel-2 are processed in separate modality branches. Valid observations are linearly embedded, a sinusoidal day-of-year positional encoding is added, and each modality branch uses a 4-layer Transformer stack over time. Variable-length sequences are then aggregated with learned attention pooling, after which a 2-layer fusion Transformer merges the two modality tokens into a single pixel embedding (Feng et al., 4 Jul 2026).

The final teacher embedding is

$L \sim \mathrm{Uniform}\{8,16\}$0

A final affine-free LayerNorm is applied, with no learnable scale or shift.

During pretraining only, the encoder output is passed through a batch-normalized projector. TESSERA v2 treats this projector as a training scaffold rather than a deployment object. This architectural distinction becomes important in the scaling study, which concludes that projector size should remain fixed even as the compute budget increases.

For distillation, each student emits a 128-dimensional output embedding. Prefix heads are attached only during distillation:

$L \sim \mathrm{Uniform}\{8,16\}$1

These heads are linear and are used solely for the distillation objective. Their role is to map each prefix of the student embedding into the teacher embedding space. This makes the student representation explicitly storage-adaptive without altering the teacher representation used during pretraining.

4. Scaling laws within a fixed pixel-wise Barlow Twins family

One of the principal contributions of TESSERA v2 is a large controlled scaling study conducted inside a single pixel-wise Barlow Twins family. The authors report 395 controlled pretraining runs on 1,024 GH200 superchips, sweeping encoder size $L \sim \mathrm{Uniform}\{8,16\}$2, projector size $L \sim \mathrm{Uniform}\{8,16\}$3, and training data $L \sim \mathrm{Uniform}\{8,16\}$4. The encoder spans 7–278M parameters across 16 widths, the projector uses four widths, and the training set ranges from $L \sim \mathrm{Uniform}\{8,16\}$5M to $L \sim \mathrm{Uniform}\{8,16\}$6M examples (Feng et al., 4 Jul 2026).

The compute axis is defined in terms of $L \sim \mathrm{Uniform}\{8,16\}$7, $L \sim \mathrm{Uniform}\{8,16\}$8, $L \sim \mathrm{Uniform}\{8,16\}$9, and a reference sequence length $C$0, with the constant $C$1 attributed to $C$2: the standard $C$3 FLOP estimate for forward/backward plus two augmentation views per step. The authors fit compute-optimal vertices within iso-FLOP buckets and then fit power laws through those vertices. The resulting empirical rules are

$C$4

The paper stresses two implications. First, encoder size and dataset size should both grow with compute, roughly in a Chinchilla-style regime where the exponents sum to $C$5. Second, projector size is effectively compute-invariant. The stated recommendation is therefore to spend compute on encoder capacity and matched data, not on a larger projector.

This scaling analysis is also operationalized. Using the resulting allocation rule, the authors train a family of pixel-wise models at 0.5B and 1B, with a 2B model in training, and subsequently distill them into smaller deployment-oriented students. A plausible implication is that pixel-wise Barlow Twins is less significant here as an isolated objective than as a stable experimental substrate for measuring EO-specific scaling behavior.

5. Downstream evaluation and the failure of loss-based selection

The pretraining sweep is evaluated on 15 AlphaEarth tasks drawn from 10 source datasets, covering classification, segmentation, change detection, and regression (Feng et al., 4 Jul 2026). Metrics are adjusted for chance: classification uses $C$6, and regression uses $C$7. Per-task scores are averaged into a composite downstream score. The best models are then further tested on 14 held-out datasets not used in development, yielding a full 29-task suite spanning vegetation-related tasks, urban tasks, and more generally classification, segmentation, regression, and distribution prediction.

All downstream tasks use lightweight frozen-feature heads. Pixel-wise tasks use a 2-layer MLP, while patch-level tasks use a small CNN with fewer than 2M parameters. This design isolates embedding quality from large task-specific finetuning budgets.

The central empirical result is that pretraining loss is a very poor proxy for downstream utility in EO. Across all 395 runs, the Pearson correlation between loss and downstream composite score is about $C$8, and the Spearman correlation is about $C$9. The abstract summarizes this as $$\mathcal{L}_{\text{BT}} = \sum_i (1 - C_{ii})^2 + \lambda \sum_{i \neq j} C_{ij}^2.$$0. The authors attribute the weak relationship to EO-specific nuisances such as clouds, revisit irregularity, and orbital sampling, which dominate the loss but do not necessarily help downstream tasks.

They further estimate that selecting models by pretraining loss rather than downstream score wastes about 254 units of compute to reach the same downstream performance. This result underlies one of the paper’s most explicit prescriptions: EO scaling laws should be fit to downstream metrics rather than assumed from pretraining loss. In that sense, Pixel-wise Barlow Twins in TESSERA v2 is not presented as a doctrine of loss minimization; it is presented as evidence that downstream-driven model selection is necessary in EO.

6. Distillation, Matryoshka representations, and deployment

The large teacher is not intended for direct deployment. TESSERA v2 reports that it costs about 50 H100-years per global annual pass, which is too expensive for routine use (Feng et al., 4 Jul 2026). The deployment strategy is therefore teacher-student distillation into four compact models: N with 1M parameters, S with 7M, M with 21M, and L with 44M. Distillation uses about 200 million pixels on 64 H100s.

Each student produces a 128-dimensional embedding while also supporting usable prefixes with dimensions

$$\mathcal{L}_{\text{BT}} = \sum_i (1 - C_{ii})^2 + \lambda \sum_{i \neq j} C_{ij}^2.$$1

The reported storage-performance tradeoff is one of the defining practical outcomes of the method. A 16-dimensional prefix retains about 92% of the full 128-dimensional composite performance at 1/8 of the storage. A 32-dimensional prefix retains about 97%, and a 64-dimensional prefix retains about 99%. Embedding dimension thereby becomes a user-side control over storage and I/O without requiring retraining.

The best student overall on the full 29-task suite is TESSERA v2-1B-M, with composite score 0.611, outperforming all compared open and proprietary alternatives tested, including some that are much larger. On the 15 AlphaEarth tasks alone, the largest student in the family, v2-1B-L, leads with 0.584. The abstract also states that the 21-million-parameter distilled TESSERA v2-1B-M in aggregate outperforms all open and proprietary models tested, and that the resulting students produce Matryoshka representations that are inexpensive to serve.

The practical conclusions drawn in TESSERA v2 are explicit. Models should not be chosen by pretraining loss alone; encoder and data should be scaled together while the projector stays fixed; large teachers should be trained and then distilled into deployable students; nested embedding dimensions should be learned by distillation rather than by self-supervised prefix losses; and multiple student sizes and embedding prefixes should be offered as a product family. The paper states that, upon completion of training, the authors plan to release v2 global embeddings covering 2017–2025. Within that pipeline, Pixel-wise Barlow Twins functions as the pretraining core of a compute-calibrated EO foundation-model stack rather than merely as a standalone self-supervised objective.