---
title: 'Caracal: Astronomy & ML Systems'
url: https://www.emergentmind.com/topics/caracal
type: topic
---

# Caracal: Astronomy & ML Systems

Searching arXiv for the two provided Caracal-related papers to ground the article in the cited literature.
Caracal denotes two distinct technical systems in recent literature. In radio astronomy, **CARACal** is the **Containerized Automated Radio Astronomy Calibration** pipeline, a calibration/reduction framework used to transform raw MeerKAT interferometric visibilities into science-ready H I products for analysis of Hickson Compact Groups. In machine learning, **Caracal** is a **decoder-only language model architecture** that replaces global self-attention with a Fourier-based token mixer called **Multi-Head Fourier (MHF)** in order to obtain autoregressive long-sequence modeling with $\mathcal{O}(L \log L)$ sequence mixing. The shared name therefore spans two unrelated technical contexts: end-to-end radio data reduction and efficient causal sequence modeling [2502.09691] [2605.00292].

## 1. Disambiguation and nomenclature

The two usages differ in capitalization, domain, and technical role.

| Form | Domain | Definition |
|---|---|---|
| **CARACal** | Radio astronomy | **Containerized Automated Radio Astronomy Calibration** pipeline |
| **Caracal** | Machine learning | **Causal Architecture via Spectral Mixing** |

In the MeerKAT Hickson Compact Group study, CARACal is described as a **calibration/reduction pipeline** that integrates several radio-astronomy software packages and runs them through a **Stimela** workflow. The paper explicitly states that CARACal is not a single algorithm; it is the orchestrator that automates the full MeerKAT reduction chain. In the long-sequence modeling paper, Caracal is presented as a decoder-only architecture that is structurally close to a standard GPT-style Transformer but replaces masked multi-head self-attention with the **Multi-Head Fourier** module and removes explicit positional encodings [2502.09691] [2605.00292].

This distinction matters because the same surface term can otherwise obscure fundamentally different technical objects: one is an end-to-end software framework for H I imaging, whereas the other is a neural architecture for autoregressive language modeling.

## 2. CARACal as a radio-astronomy calibration and reduction pipeline

CARACal was the main end-to-end software framework used in the MeerKAT Hickson Compact Group study to produce the science-ready H I data products analyzed in the paper. Its documented functions include flagging of bad data and radio-frequency interference, cross-calibration on standard calibrators, continuum imaging and self-calibration, continuum subtraction, spectral-line imaging, and diagnostic plotting and quality checks. The pipeline integrates multiple packages within a Stimela workflow, providing a scripted and reproducible reduction path from raw visibilities to calibrated, continuum-subtracted line data [2502.09691].

The reduction chain was organized into three main workers. The first worker handled flagging and cross-calibration. Auto-correlations and shadowed antennas were flagged with CASA `flagdata` inside CARACal, and RFI was flagged using **AOFlagger**. Cross-calibration on the primary calibrator used the sequence **KGBKGB**, corresponding to time-dependent delay calibration (**K**), gain calibration (**G**), and bandpass calibration (**B**). Those solutions were then transferred to the secondary calibrator: **KB** was applied, gain amplitude and phase were solved for, spurious calibrated visibilities were flagged, and the solution was repeated with flux-scale bootstrapping using **GAF** and `calmode: [ap, null, ap]`. For the target, CARACal performed on-the-fly application of the primary delay/bandpass and secondary gains while also flagging auto-correlations, shadowed antennas, and RFI [2502.09691].

The second worker covered continuum imaging and self-calibration. This stage used **WSClean** for imaging and **Cubical** for self-calibration. Imaging and self-calibration were repeated iteratively; each iteration used the previous clean mask, multi-scale cleaning was enabled, and scales were selected automatically. The reported parameter choices were an image size of **2° × 2°**, a pixel size of **1 arcsec**, **Briggs -1** weighting, **0 arcsec** tapering, and **three calibration iterations**. WSClean clean-mask thresholds were **30, 20, 10, and 5 × rms**, and the WSClean auto-threshold parameter was **0.5**. These steps produced continuum models for later subtraction from the line data [2502.09691].

The third worker implemented continuum subtraction and line imaging. The continuum model was subtracted from the target visibilities, and CARACal additionally used **CASA `mstransform`**, Doppler-tracking corrections, and a **first-order polynomial fit** to the real and imaginary parts of the line-free channels. The `mstransform` outputs were re-flagged with **AOFlagger**, and line cubes were made with **WSClean** using multi-scale cleaning and a major-cycle clean gain of **0.2**. The authors first created a cube at **98 arcsec**, then used **SoFiA** outside CARACal to make a better clean mask, reran CARACal using that mask to create the final lowest-resolution cube, and repeated this mask-refinement process stepwise for higher-resolution cubes [2502.09691].

## 3. CARACal-generated products, observing setup, and scientific role

The MeerKAT study used CARACal to produce **data cubes** at multiple angular resolutions, **moment maps** including moment 0 column density maps and moment 1 velocity fields, **integrated spectra** for each Hickson Compact Group, and the calibrated, continuum-subtracted line data needed for source finding. From the CARACal-generated cubes, the authors derived RA–velocity slices, channel maps, moment maps, position–velocity diagrams, flux measurements, and H I masses. These cubes formed the basis for comparison with previous **VLA** and **GBT** observations [2502.09691].

Additional imaging details are explicit. Cubes were produced at **60 arcsec** and at higher resolution down to about **20 kpc linear resolution**, with intermediate-resolution cubes also provided. The **60 arcsec cube** was the default unless stated otherwise. Primary-beam attenuation was corrected using the model of **Mauch et al. (2020)**, frequency was converted to velocity with **MIRIAD `VELSW`**, and the paper used the **optical velocity definition**. For comparison to GBT fluxes, the authors modeled the GBT beam as a 2D Gaussian with FWHM **9.1 arcmin**:
$$
f(R) = C3 + (C4 - C3)\,\exp[-((0.707/C5)^2\,R^2)]
$$
with $C3 = 0$, $C4 = 1$, and $C5 = 9.1 \times 60 / 2.355$ arcsec. This beam-response map was multiplied with each MeerKAT channel before extracting the integrated spectrum. H I mass was computed from the primary-beam-corrected integrated flux as
$$
M_{\mathrm{HI}}[M_{\odot}] = 235600 \times D^2 \times \sum S_i \Delta v,
$$
where $D$ is distance in Mpc and $\sum S_i \Delta v$ is the integrated flux in Jy km s$^{-1}$. For visual comparison of MeerKAT and GBT spectra, both were smoothed to **20 km s$^{-1}$** with a **boxcar kernel** [2502.09691].

The observing configuration is also specified: the MeerKAT L-band receiver covered **856–1711.974 MHz** in **32k** correlator mode with central observing frequency **1389.1322 MHz**, channel width **26.123 kHz**, velocity resolution **5.5 km s$^{-1}$** at 1420.4 MHz, **32768** total channels, **8 s** integration time, about **5.13 h per group** on source, and **61–63** antennas used [2502.09691].

The scientific conclusions that depended on these reduced products were sharply comparative. Relative to previous VLA observations, MeerKAT revealed much more extended tidal features in phase 2 groups and some new high surface brightness features in phase 3 groups, but no diffuse H I component was found in phase 3 groups. The difference between phase 2 and phase 3 groups therefore remained substantial, supporting previous findings that the transition between the two phases must be abrupt. The reduced data also enabled the identification of tidal tails, bridges, clumps, surrounding galaxies, and disturbed kinematics in several groups. The paper explicitly notes a known issue with UVLIN-like continuum subtraction—**vertical stripes in RA–velocity slices** where line channels are excluded from the fit—and reports that this effect was checked in the CARACal products [2502.09691].

## 4. Caracal as a decoder-only sequence model

In machine learning, Caracal is introduced as a **decoder-only language model architecture that replaces global self-attention with a Fourier-based token mixer** called the **Multi-Head Fourier (MHF)** module. The model is proposed to address three stated limitations of long-context Transformers: the quadratic cost of attention, $\mathcal{O}(L^2)$; the need for explicit positional encodings; and the non-causality of prior FFT-based token mixers such as FNet. Caracal is therefore designed to be efficient, autoregressive, and portable across environments because it uses standard FFT and convolution operators rather than hardware-specialized kernels [2605.00292].

The architecture remains close to a GPT-style Transformer with two major changes: masked multi-head self-attention is replaced by MHF, and explicit positional encodings are removed. To improve local pattern capture, the model retains a small number of **sliding-window attention (SWA)** layers. In the default configuration, one SWA layer is interleaved after every two MHF layers, giving a **2:1 MHF:SWA ratio**. This makes Caracal a hybrid architecture in which global sequence mixing is handled by Fourier-domain operations while local precision is supported by occasional windowed attention [2605.00292].

The motivation is framed in explicitly comparative terms. Transformer attention is expensive at long lengths; positional encoding schemes are treated as sophisticated fixes for a permutation-equivariant mechanism; and prior Fourier models are typically non-causal, which prevents direct autoregressive generation. Caracal is proposed as a single response to these issues: FFT-based mixing for efficiency, removal of explicit positional encodings, and a causal frequency-domain procedure for generation [2605.00292].

## 5. Multi-Head Fourier mechanics and causal sequence mixing

The MHF forward pass begins with a **local causal pre-convolution**,
$$
\tilde{x} = \text{CausalConv1d}(x),
$$
described as a lightweight depthwise causal convolution, typically with kernel size 3 and left padding. This injects local inductive bias and compensates for the removal of positional encoding. After normalization,
$$
x_{\text{norm}} = \text{LayerNorm}(\tilde{x}),
$$
the module forms a value stream
$$
x_v = \text{Linear}_V(x_{\text{norm}})
$$
and a gate stream
$$
x_g = \text{Conv1d}_{G2}\left(\sigma(\text{Linear}_{G1}(x_{\text{norm}}))\right),
$$
where $\sigma$ is SiLU and $\text{Conv1d}_{G2}$ is a 1x1 grouped convolution with $n_{\text{head}}$ groups. The paper identifies this gate stream as the source of content dependence in the mixing rule [2605.00292].

Causal mixing is then performed by padding the sequence to length $N = 2L$, taking FFTs of the value and gate streams,
$$
V_{\text{fft}} = \mathcal{F}(x_v), \qquad G_{\text{fft}} = \mathcal{F}(x_g),
$$
multiplying them elementwise,
$$
X_{\text{fft}} = V_{\text{fft}} \odot G_{\text{fft}},
$$
and applying inverse FFT followed by truncation,
$$
x_{\text{mixed}} = \text{Truncate}\left(\mathcal{F}^{-1}(X_{\text{fft}})\right),
$$
with a final output projection
$$
y = \text{Linear}_O(x_{\text{mixed}}).
$$
The paper characterizes this as a “pad-FFT-multiply-iFFT-truncate” pipeline. Its complexity is $\mathcal{O}(L \log L)$ because the dominant operations are FFT and inverse FFT rather than the $\mathcal{O}(L^2)$ matrix formation of dense attention [2605.00292].

The multi-head formulation makes the convolutional interpretation explicit. After reshaping, $\mathbf{x}_g, \mathbf{x}_v \in \mathbb{R}^{B \times H \times L \times d}$, and for each batch $b$, head $h$, and channel $c$,
$$
r_t^{(b,h,c)} = \sum_{j=0}^{t} \alpha_{tj}^{(b,h,c)} v_j^{(b,h,c)},
$$
with
$$
\alpha_{tj}^{(b,h,c)} = x_g[b,h,t-j,c], \qquad v_j^{(b,h,c)} = x_v[b,h,j,c].
$$
The resulting matrix is described as a **lower-triangular Toeplitz matrix** for each channel:
$$
\mathbf{A}_{mhf} =
\begin{pmatrix}
x_g[0] & 0 & \dots & 0 \\
x_g[1] & x_g[0] & \dots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
x_g[L-1] & x_g[L-2] & \dots & x_g[0]
\end{pmatrix}.
$$
This means the mixing depends on relative offsets $t-j$ rather than absolute positions. The paper therefore places MHF between two familiar constructions: Fourier mixing with static structured weights and attention with input-dependent weighted sums [2605.00292].

The key causal mechanism is not a literal mask in frequency space but an **asymmetric padding and truncation strategy**. Without zero-padding, FFT multiplication corresponds to circular convolution, and future terms can wrap into earlier outputs. Padding both sequences to $N = 2L$ moves those wrap-around terms outside the causal region; truncating to the first $L$ positions yields the desired causal convolution,
$$
r_t = \sum_{j=0}^{t} v_j g_{t-j}.
$$
This directly addresses the standard objection that Fourier token mixers are non-causal [2605.00292].

## 6. Empirical performance, portability, and limitations of Caracal

The Caracal paper evaluates the architecture on language modeling, commonsense reasoning, long-context extraction/retrieval, and efficiency scaling. Across Tiny, Small, Medium, and Large scales, Caracal is described as **competitive** with Transformer and SSM baselines rather than uniformly dominant. On the **Large** setting, the reported results are **Lambada perplexity: 29.39**, **Lambada accuracy: 35.26**, and **average benchmark accuracy: 49.01**. In a broader comparison using a **345M-parameter** configuration and the prior-work evaluation protocol, **Caracal (Default)** attains the best average accuracy in the comparison table with **46.62** [2605.00292].

The long-context retrieval results are more mixed. On **SWDE** and **FDA**, the paper states that **Llama** performs better on exact retrieval-style tasks, with **Llama avg: 9.55** and **Caracal avg: 5.37**. The authors describe this as a **“resolution gap”**, and dense attention is said to retain an advantage in fine-grained retrieval. This is one of the principal limitations explicitly acknowledged by the paper [2605.00292].

Efficiency is the clearest empirical strength. Training time and token throughput were measured from context lengths 256 to 8192. At context length **8192**, the reported training times are **Llama: 94,258 s**, **Caracal default: 34,071 s**, and **Mamba2: 33,956 s**. Throughput at the same length is **Llama: 106,091 tokens/s**, **Caracal default: 293,504 tokens/s**, and **Mamba2: 294,498 tokens/s**. The hybrid Caracal with SWA is stated to be slightly faster than pure MHF in some settings because the SWA layers benefit from highly optimized FlashAttention kernels [2605.00292].

Ablation studies isolate the contributions of local modeling and gating design. The tested variants are **w/o SWA**, **w/o SWA + PC**, **with PE**, and **SSLP**. The reported conclusions are that SWA and pre-convolution help local modeling, that adding positional encodings does not help much, and that the two-stage gating design outperforms a simpler single-stage projection. The best hybrid ratio in the parameter study is **2:1 MHF:SWA**; increasing SWA beyond that slightly hurts performance [2605.00292].

Portability is framed as a major architectural distinction from Mamba-like SSMs. Caracal uses standard FFT operators, standard convolution, standard linear layers, and standard FlashAttention only for the small SWA component. By contrast, the paper characterizes Mamba and Mamba-2 as architectures whose best implementations often rely on custom CUDA kernels, hardware-aware scan algorithms, and specialized tuning for block sizes and memory layout. This suggests that Caracal seeks a specific balance between scalability and engineering simplicity rather than asymptotic optimality alone [2605.00292].

The paper is explicit about remaining limitations: retrieval precision is weaker than dense attention on some tasks, large-scale validation remains future work, and inference optimization would require more specialized engineering for production-grade throughput. Proposed future directions include **multi-scale gating**, **hierarchical mixing**, and **better retrieval-oriented refinements** [2605.00292].

Source: https://www.emergentmind.com/topics/caracal