ChunkedPCA: Source-Masked PCA for Radio Imaging

• ChunkedPCA is a chunk-wise, source-masked PCA method designed to subtract common background fluctuations from array detector data.
• It improves on conventional PCA by estimating the background from off-source pixels within short temporal chunks, preserving compact-source flux.
• Simulation and observational studies show that ChunkedPCA achieves lower background RMS and fewer artifacts compared to standard PCA methods.

to=arxiv_search.search 彩神争霸大发json {"query":"(Mandal et al., 22 Aug 2025) ChunkedPCA radio camera principal component analysis", "max_results": 5, "sort_by":"submittedDate", "sort_order":"descending"} to=arxiv_search.search 的天天彩票json {"query":"(Lippi et al., 2019) Incremental Principal Component Analysis Exact implementation and continuity corrections", "max_results": 3, "sort_by":"relevance", "sort_order":"descending"} ChunkedPCA is a chunk-wise, source-masked PCA method for removing common background fluctuations from array-detector radio-camera data. It was introduced to improve on conventional principal component analysis for time-ordered detector data in situations where a strongly correlated, time-varying background from atmosphere, electronics, and related instrumental effects is seen by essentially all pixels, while only a subset of pixels observes the astronomical source at any given time. Its defining idea is to estimate the common fluctuation from off-source pixels only, within short temporal chunks, and then subtract that estimate from all pixels in the chunk. In the reported simulation and MKID-camera observations, this design preserved compact-source flux better than ordinary PCA, reduced artifacts, and lowered background RMS (Mandal et al., 22 Aug 2025).

1. Problem formulation and observational context

ChunkedPCA addresses a standard failure mode of common-mode subtraction in radio and sub-mm imaging with large-format cameras. In such systems, the field of view can be larger than the source, so during a scan many pixels observe blank sky while a smaller subset intersects the source. If the common fluctuation is estimated from all pixels indiscriminately, the source signal contaminates the estimated background. Subtracting that contaminated estimate can distort the source, reduce recovered flux, and generate map-scale artifacts (Mandal et al., 22 Aug 2025).

The method is therefore designed for cameras whose field of view is larger than the target source. The core assumption is not merely that the background is correlated, but that enough off-source detector channels remain available in each time segment to estimate that background robustly. This is why the paper states directly that if a source is larger than the field of view, “we cannot use this method” (Mandal et al., 22 Aug 2025).

The central contrast with ordinary PCA is explicit. Standard PCA explains dominant variance across all detector streams. A strong compact source also contributes dominant variance, so part of the source structure is absorbed into the common mode. The paper characterizes the resulting effect as baseline warping toward the source peak, with source-adjacent distortions and negative artifacts in cleaned time streams, and ring-like or camera-FOV-scale structures in maps. It further notes that both median filtering and ordinary PCA “use each TOD to remove the fluctuation from itself,” which is the mechanism behind source-induced self-subtraction (Mandal et al., 22 Aug 2025).

2. Data model and conventional PCA baseline

The paper frames the data as a matrix of time-ordered data (TOD). If there are $p$ detector pixels and $n$ samples per pixel, the raw data matrix is

$$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$

Here $T_j$ denotes the response of pixel $j$ at sampling time $t_i$ (Mandal et al., 22 Aug 2025).

In conventional PCA, each pixel TOD is mean-centered to form $Z$, and the covariance matrix is computed as

$\Sigma = Z^T Z.$

An eigendecomposition is then performed,

$\Sigma = P D P^{-1},$

where $P$ contains eigenvectors and $n$0 is diagonal with eigenvalues $n$1, sorted in descending order. The corresponding principal-component representation is

$n$2

with $n$3 ordered by decreasing eigenvalue (Mandal et al., 22 Aug 2025).

The background estimate in the conventional approach is reconstructed from the first $n$4 principal components and subtracted from each detector stream. The cleaning equation is given in the paper as

$n$5

Although the notation is informal, the intended model is clear: the leading PCA modes are treated as common background variation and removed from the raw TOD (Mandal et al., 22 Aug 2025).

ChunkedPCA preserves this PCA machinery but changes the admissible data entering the background estimate. The methodological novelty is not a new eigensolver or a new objective; it is the restriction of PCA, within each time chunk, to detector channels believed to be off-source.

3. Chunk-wise, source-masked estimation procedure

The implemented workflow begins with a rough flattening of the raw TOD so that source-contaminated samples can be identified. The paper emphasizes that masking cannot be performed directly on raw data because the large common background fluctuation would obscure the source peaks. A first-pass baseline removal is therefore applied using conventional PCA or another simpler method such as a median filter. This first pass is not the final science reduction; it is used only to create a mask (Mandal et al., 22 Aug 2025).

On these approximately flattened TOD, sigma clipping is applied using Astropy’s sigma-clipping routine. Samples above the threshold are flagged as source-contaminated, producing a masked array $n$6. The paper also mentions a geometric alternative: if telescope pointing, source position, and array geometry are already known, on-source times could be predicted directly. The implemented workflow, however, is data-driven (Mandal et al., 22 Aug 2025).

The observation is then divided into temporal chunks. If the full TOD length is $n$7 and the chunk length is $n$8, the number of chunks is approximately $n$9. For each chunk, if any sample for a given pixel is masked, that entire pixel-chunk combination is excluded from the chunk’s PCA. This selection is represented by a chunk matrix $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$0 with dimensions

$$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$1

indicating whether a pixel is usable in a given chunk (Mandal et al., 22 Aug 2025).

For chunk $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$2, the off-source data matrix is formed from the unmasked detector streams only. If $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$3 denotes the off-source pixel set in chunk $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$4, then

$$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$5

This chunk-specific off-source data is mean-centered and processed with PCA exactly as in the standard case. In faithful mathematical form, if the centered off-source matrix is $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$6 and the first $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$7 principal components are retained with loadings $$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$8, the rank-$$X = \begin{bmatrix} x_{1,1} & \cdots & x_{1,p}\ \vdots & \ddots & \vdots\ x_{n,1} & \cdots & x_{n,p} \end{bmatrix}^{n\times p}, \qquad X_{i,j} = T_j(t_i).$$9 reconstruction of the chunk background is

$T_j$0

Equivalently, under an SVD $T_j$1,

$T_j$2

For unmasked pixels, the background estimate comes directly from this PCA reconstruction. For masked pixels, which do not participate in the eigensystem, the paper introduces an assignment step: either the background fluctuation is averaged over the unmasked pixels, or, more conservatively, it is transferred from unmasked pixels that are highly correlated with the masked one (Mandal et al., 22 Aug 2025).

With a time-dependent chunk background $T_j$3, the cleaned data are represented conceptually as

$T_j$4

or, under the simplest common-mode interpretation,

$T_j$5

After chunkwise processing, the cleaned chunks are concatenated back into full-length cleaned TOD. An important practical point is that each chunk is mean-centered before PCA, so the chunk outputs stitch together naturally without special seam correction. The paper notes that the chunked background may not visually resemble the original common mode because of this mean-centering, and presents that behavior as advantageous for reassembly (Mandal et al., 22 Aug 2025).

4. Parameterization, implementation, and operating regime

The main free parameter is chunk size. The paper gives both a formal lower bound and a practical tradeoff. The chunk must contain at least as many datapoints as needed to estimate the desired number of principal components; more importantly, chunk size controls how many detector channels remain available for off-source PCA. If chunks are too large, many pixels may encounter the source within the same chunk, so many pixel-chunk combinations are masked out. That leaves too few off-source pixels for robust PCA and increases noise in the estimated background. Smaller chunks mitigate this because the source occupies fewer pixels per chunk (Mandal et al., 22 Aug 2025).

In the examples, the number of retained principal components is set to $T_j$6. The stated motivation is the explained-variance curve: the first three principal components explain $T_j$7 of the variance, and adding more components yields diminishing returns. PCA was implemented with scikit-learn (Mandal et al., 22 Aug 2025).

The method is explicitly intended for compact sources, or at least for sources smaller than the array field of view. Its performance depends on maintaining a sufficient number of unmasked pixels in each chunk. It also depends on the quality of the source mask. If the rough flattening and sigma clipping fail to identify source-contaminated samples, source leakage can re-enter the chunkwise PCA and degrade the background estimate (Mandal et al., 22 Aug 2025).

A common misconception is that ChunkedPCA is simply PCA applied to smaller blocks for computational convenience. The paper supports a narrower interpretation. The “chunking” is used because the set of source-contaminated pixels changes during the scan, so dynamic source exclusion must be local in time. This suggests that ChunkedPCA is best understood as a source-masking strategy embedded inside a chunkwise common-mode estimator, rather than as a generic large-scale PCA algorithm.

5. Simulation study and observational validation

The simulation study uses a 30-pixel camera producing 30 TODs, each with 30,000 time samples. A common-mode fluctuation is added to all pixels as

$T_j$8

together with additional Gaussian noise in each pixel. To mimic a compact source during a raster scan, one Gaussian peak is added to each TOD with amplitude five times the Gaussian-noise RMS. The peaks are uniformly spaced in time and do not overlap, so at any instant only one pixel observes the source while the others see blank sky. The paper presents this as the ideal use case for ChunkedPCA (Mandal et al., 22 Aug 2025).

Under these conditions, ordinary PCA produces visible artifacts in the reconstructed common mode around source passages, and these distortions propagate into the cleaned TOD. ChunkedPCA removes these distortions because the relevant pixel-chunks are excluded before PCA. Polynomial fitting avoids source-copy artifacts of the PCA type, but tracks the slowly varying background less well and leaves a noisier baseline. In cropped peak comparisons, ChunkedPCA preserves source amplitude better than both ordinary PCA and polynomial fitting (Mandal et al., 22 Aug 2025).

The reported simulation RMS values after cleaning are:

Method	RMS
Polynomial	0.1112
PCA	0.1088
ChunkedPCA	0.1065

These values show the lowest average RMS for ChunkedPCA over the 30 simulated pixels (Mandal et al., 22 Aug 2025).

The paper also tests sensitivity to source amplitude. Ordinary PCA worsens as source peak amplitude increases, because stronger peaks bias the global PCA baseline estimate more strongly. ChunkedPCA remains relatively stable with source amplitude because those peaks are masked before background estimation. This suggests a key conceptual distinction: ChunkedPCA largely decouples baseline estimation from source strength within its intended operating regime (Mandal et al., 22 Aug 2025).

The real-data validation uses a microwave kinetic inductance detector (MKID) camera on the Nobeyama 45-m telescope in the 100 GHz band. The camera has 109 pixels, of which 82 produced good data in the analyzed Mars observation, with an average beam size of about $T_j$9. The observation is a raster scan of a $j$0 field centered on Mars, while the array field of view is about $j$1. The cleaned TOD are gridded into maps using HEALPix (Mandal et al., 22 Aug 2025).

In combined maps from all pixels, clipped at $j$2 dB from the peak, the background RMS is reported as

$j$3

The ordinary PCA map also shows a circular artifact with size approximately equal to the MKID camera field of view ($j$4), which the paper attributes to source-induced PCA contamination across the array. ChunkedPCA suppresses this structure. The same study states that ChunkedPCA preserves data scaling, so calibration steps such as chopper-wheel scaling can be applied either before or after higher-order baseline subtraction (Mandal et al., 22 Aug 2025).

6. Relation to other PCA variants and methodological boundaries

ChunkedPCA belongs to a broader family of PCA adaptations, but its objective differs from the usual motivations for scalable or incremental PCA. In exact incremental PCA, for example, the transformation coefficients matrix is updated on-line for each new sample without keeping all samples in memory, and the final result is formally equivalent to batch PCA on the same data (Lippi et al., 2019). ChunkedPCA does not seek equivalence to batch PCA on all channels; instead, it deliberately changes which detector streams are allowed to define the common mode in each local time segment.

Other large-scale PCA methods address different bottlenecks. “HPCA” combines structured and unstructured random projections to compute top principal components when both the number of examples and the number of features are very large, using two data-parallel passes and reducing the working state from $j$5 to $j$6 (Karampatziakis et al., 2013). Adaptive randomized dimension reduction similarly emphasizes randomized low-rank approximation through repeated multiplications by $j$7 and $j$8, with the goal of scalable approximation and implicit regularization rather than source masking (Darnell et al., 2015). Streaming kernel PCA extends the same low-memory logic to nonlinear structure by combining random feature maps with sketching, maintaining a compact basis in a stream instead of forming the full kernel matrix (Ghashami et al., 2015).

There are also PCA methods in which “chunking” refers to partitioning the feature axis rather than time. “FeatPCA,” for instance, divides genes into multiple feature subspaces, applies PCA independently in each subspace, and concatenates the reduced representations for clustering in scRNA-seq data (Islam et al., 8 Feb 2025). This suggests that the term “chunked PCA” is not intrinsically tied to a single algorithmic pattern; ChunkedPCA in radio astronomy is specifically a temporal, source-masked common-mode subtraction method.

From a numerical perspective, ChunkedPCA also differs from iterative deflation approaches such as NIPALS-PCA and GS-PCA, which compute components sequentially and emphasize orthogonality preservation and GPU acceleration through BLAS kernels (0811.1081). It likewise differs from Bayesian nonparametric PCA, whose main contribution is principled latent-dimension inference via an Indian buffet process and orthogonal-basis priors, rather than time-local background estimation (Elvira et al., 2017).

These comparisons delimit the method’s scope. ChunkedPCA is lightweight and fast on a regular computer, but its gain does not come from a more sophisticated PCA objective. It comes from changing the admissible support of the background model: standard PCA computes dominant modes from the entire array at once, whereas ChunkedPCA computes them locally in time and only from source-free channels. That narrow change is precisely what makes it useful for compact-source observations with array fields of view larger than the target (Mandal et al., 22 Aug 2025).