---
title: 'RadarLLM: Adapting LLMs for Radar Inference'
url: https://www.emergentmind.com/topics/radarllm
type: topic
---

# RadarLLM: Adapting LLMs for Radar Inference

Searching arXiv for the specified RadarLLM papers and closely related radar–LLM work.
RadarLLM denotes a line of work in which radar measurements are adapted to pretrained large language models or vision-language models for downstream semantic inference. In the current literature, the label refers not to a single canonical architecture but to several task-specific frameworks: a marine radar target detector built on GPT-2 with a preference-aware loss [2509.12089], a related marine detection pipeline that combines feature preprocessing, patch selection, and partial LayerNorm fine-tuning of GPT-2 [2509.12110], a millimeter-wave human-motion understanding system that couples a motion-guided radar tokenizer with a radar-aware T5 model [2504.09862], and a vision-aligned 4D radar captioning pipeline for weather-robust scene semantics that aligns radar embeddings to frozen SigLIP and decodes structured captions with Qwen2.5-VL [2605.07367]. Across these variants, the unifying idea is to cast radar signals, point clouds, or radar tensors into token sequences or embedding sets that can be processed by pretrained sequence models while keeping most backbone parameters frozen.

## 1. Conceptual scope and historical emergence

RadarLLM emerged from the observation that pretrained language models can be repurposed as generic sequence learners for radar-derived representations. In marine radar target detection, the central claim is that a pre-trained LLM can serve as a backbone for small-target detection once radar echoes are converted into “language-like” token streams and adapted through lightweight fine-tuning [2509.12089]. In a parallel formulation for marine echo analysis, the framework integrates feature preprocessing with large language models by tokenizing radar sequence features, filtering uninformative segments, projecting them into the pretrained embedding space, and fine-tuning only normalization layers [2509.12110]. In millimeter-wave motion understanding, Radar-LLM is presented as the first framework that leverages large language models for human motion understanding using millimeter-wave radar as the sensing modality, with discrete radar tokens aligned to text in a shared embedding space [2504.09862]. In weather-robust automotive semantics, the term is used for a pipeline in which a radar encoder is aligned to frozen SigLIP vision embeddings and a frozen Qwen2.5-VL decoder generates structured scene captions from radar tokens [2605.07367].

A plausible implication is that RadarLLM is best understood as a research paradigm rather than a single model family. The shared premise is consistent across the papers: radar is not treated as a modality requiring entirely bespoke architectures, but as a source of structured sequences or tokens that can be mapped into pretrained model spaces. The major variation lies in what is tokenized, which pretrained backbone is used, and whether the downstream task is binary detection, motion-to-text generation, or open-vocabulary scene captioning.

## 2. Common architectural patterns

Despite targeting different radar modalities and tasks, the RadarLLM papers exhibit a recurring pipeline structure: radar-specific preprocessing, tokenization or projection, lightweight adaptation of a largely frozen pretrained model, and task-specific decoding. The concrete realizations differ substantially.

| Variant | Radar representation | Backbone adaptation |
|---|---|---|
| Marine detection with preference-aware loss | Five sequence features split into non-overlapping patches | Smallest GPT2, first four layers, LoRA on $W_Q$ and $W_V$, LayerNorm fine-tuning [2509.12089] |
| Marine detection with patch selection | Five sequence features, top-$y\%$ selected patches | GPT-2 small truncated to first 6 layers, only LayerNorm and classification head trainable [2509.12110] |
| Human motion understanding | Aggregate VQ-VAE tokens from mmWave point-cloud sequences | T5-Small with extended radar-token vocabulary and cross-modal attention [2504.09862] |
| Weather-robust scene semantics | 4D radar tokens from a ResNet-18 encoder and projector | Frozen Qwen2.5-VL-3B with LoRA adapters in all Q/K projections [2605.07367] |

In the marine detection setting with preference-aware loss, each radar return $x(n)$ is converted into five complementary sequence features: instantaneous phase, Doppler spectrum entropy, short-time Fourier magnitude spectrum, amplitude, and Doppler phase. Each feature vector of length $N$ is split into non-overlapping patches of length $L$, yielding $K=5\lceil N/L\rceil$ tokens per sample; a linear projection maps each patch into a GPT2-compatible embedding, and a sinusoidal positional encoding is added before feeding the tokens into GPT2 [2509.12089]. In the related marine framework, the same five-feature decomposition is used, but a small reference transformer computes global patch importance scores, and only the top $y\%$ patches are retained before projection into the pretrained LLM space [2509.12110].

In the motion-understanding formulation, the tokenization problem is substantially more elaborate. A Motion-guided Radar Tokenizer based on Aggregate VQ-VAE converts a point-cloud sequence $\mathbf P_{1:T}\in\mathbb R^{T\times N\times4}$ into a shorter discrete token sequence $\mathbf z_{1:L}\in\{1,\dots,K\}^L$. The tokenizer uses template-prior grouping, masked trajectory modeling, and aggregated quantization with a codebook $\mathcal Z$ [2504.09862]. By contrast, the weather-robust scene semantics pipeline does not quantize to discrete radar symbols. It begins from a 4D radar “tesseract” $X_0\in\mathbb R^{64\ \mathrm{Doppler}\times256\ \mathrm{range}\times37\ \mathrm{elevation}\times107\ \mathrm{azimuth}}$, collapses elevation by max pooling, forms either a 5-channel or 66-channel representation, and encodes the result with ResNet-18 plus PETR positional embedding before token pooling and projection into the VLM token space [2605.07367].

A consistent design principle across these systems is parameter efficiency. The marine preference-aware detector trains 2.46 M out of 69.4 M parameters by freezing almost all GPT2 weights and adapting the model through LoRA plus LayerNorm updates [2509.12089]. The patch-selection marine framework trains only LayerNorm parameters and a final classification head in the truncated GPT-2 [2509.12110]. The 4D radar captioning system uses approximately 7M trainable parameters while keeping the SigLIP vision encoder and Qwen2.5-VL-3B frozen, aside from LoRA adapters of rank 16 inserted into all Q/K projections [2605.07367].

## 3. Marine radar target detection formulations

Marine radar target detection is the earliest and most explicit setting in which RadarLLM is framed as adapting a pretrained LLM to radar sequence analysis. The preference-aware variant uses the smallest GPT2 with hidden size $F=768$ and activates its first four layers as a feature extractor. Through Low-Rank Adaptation on the multi-head attention’s query and value matrices, plus fine-tuning LayerNorm parameters, the architecture remains largely frozen while a layer-norm and two fully connected layers produce a $B\times K\times2$ token-level output [2509.12089]. The training protocol specifies batch size 64, evaluation batch 400, 500 epochs for LLM fine-tuning, Adam with $(\beta_1=0.9,\beta_2=0.999)$, learning rate $10^{-4}$, patch length $L=48$, and early stopping on validation.

The distinctive methodological contribution of that paper is the preference-aware loss. Two models are maintained: a lightweight reference model $\theta_r$ trained on validation samples only and the target LLM-based detector $\theta_t$. For each feature token, the token importance score is
$$
s_{b,k}=\mathrm{ReLU}\bigl(\mathcal{L}_{t,b,k}-\alpha\,\mathcal{L}_{r,b,k}\bigr),\quad \alpha=0.9,
$$
and the final mini-batch loss is
$$
\mathcal{L}_{\mathrm{final}}=\frac{1}{B\,K}\sum_{b=1}^B\sum_{k=1}^K s_{b,k}\,\mathcal{L}_{t,b,k}.
$$
The stated interpretation is that tokens on which the larger LLM still errs more than the reference model are the most worthy of further training, whereas trivial or overly noisy tokens receive zero weight [2509.12089].

The experimental setting uses eight real-world X-band marine radar sequences from the IPIX Archive, each with 14 range cells and 131 072 echoes at 1 kHz; Data1–3 are low-SCR scenes with $<8$ dB, and Data4–8 are high-SCR scenes with $>8$ dB. The split uses the first 20% of time for training, the next 15% for validation, and the remaining 65% for testing. At a fixed false alarm rate $10^{-3}\times 6{,}000\approx0.005$, RadarLLM achieves an average detection rate of 81.96%, compared with 80.82% for ADN18. Under limited data with 10% training, RadarLLM attains 78.91%, compared with 77.18% for ADN18 and 69.56% for GRU; this is described as a +9.9 pp improvement over LLM+CE fine-tuning. On Data1 with SCR approximately 7 dB, preference-aware loss lifts average detection from 34.7% to 52.1%, with further reported improvements of +5.9 pp on Data2 and +7.5 pp on Data3 [2509.12089].

The related paper “When marine radar target detection meets pretrained large language models” uses a different optimization strategy but reaches a similar conclusion about LLM transfer. Here, a randomly initialized transformer computes global patch importance by averaging $\boldsymbol a^{(n,h)}$ over training samples and attention heads, retaining only the top $y\%$ patches. Those selected patches are projected into the GPT-2 feature space, standard sinusoidal positional encodings are added, and GPT-2 small is truncated to its first 6 transformer layers [2509.12110]. All self-attention and feed-forward weights are frozen; only LayerNorm scale and shift parameters, together with the final classification head, are trainable.

On nine IPIX datasets in HH polarization, with the first 10% of time for training, the next 5% for validation, and the remainder for testing, the full system LLM4TS(S) achieves an average detection rate of 72.06% at $P_{fa}=0.002$. The best non-LLM baseline, ADN18, reaches 66.18%; PatchTST reaches 64.35%, and PatchTST(S) 69.16%; LLM4TS(0) reaches 62.06%; LLM4TS(F) reaches 67.93%; and LLM4TS with partial fine-tuning but no patch selection reaches 69.49% [2509.12110]. The ablations report that patch selection alone adds up to +15.5% on IPIX #17 for the reference model and +3.7% for the LLM pipeline, that partial fine-tuning outperforms full fine-tuning by +1.56%, and that pretraining improves over random initialization by +7.43%. The same paper also reports 1,334 samples/s for LLM4TS(S), compared with 1,074 samples/s for the standard Transformer, attributing the throughput gain to token sparsification and GPT inference optimizations.

Taken together, these marine studies establish two recurrent claims. First, pretrained GPT-2 backbones can outperform conventional RNN, Transformer, and CNN baselines on radar detection. Second, overfitting in low-SCR or limited-data regimes is a central failure mode, addressed either by preference-aware token reweighting [2509.12089] or by reference-model-driven patch selection with minimal parameter adaptation [2509.12110].

## 4. Millimeter-wave human-motion understanding

In human-motion understanding, RadarLLM addresses a substantially different radar regime: sparse millimeter-wave point-cloud sequences rather than marine echo sequences. The framework is organized into three stages: physics-aware synthesis of a radar–text corpus from motion–text data, a Motion-guided Radar Tokenizer based on Aggregate VQ-VAE, and a Radar-aware Language Model based on T5 that aligns radar tokens and words in a shared embedding space [2504.09862].

The synthesis stage is designed to compensate for the lack of paired radar–text data. The system simulates realistic mmWave returns from SMPL X motion sequences from HumanML3D and inherits their captions. Ray tracing with RF adaptive sampling uses a Sobel-style edge detector to focus rays on high-curvature regions; for each path, the intermediate frequency is approximated using the Physical Optics Integral:
$$
\mathrm{IF}(t)\approx \sum_p \Gamma_p\exp\bigl(-j\,\tfrac{4\pi f_c R_p}{c}\bigr)/R_p^2.
$$
A 1D Range-FFT is computed per chirp, followed by a 1D Doppler-FFT across slow time, static clutter is removed by subtracting the temporal mean of each range bin, and the top $K=128$ peaks are extracted from each heatmap to form point clouds $\mathbf P_t\in\mathbb R^{128\times4}$ [2504.09862]. Real-world validation uses 375 sequences, corresponding to 125 motions with 3 trials, collected on TI AWR1843BOOST.

The Motion-guided Radar Tokenizer converts noisy point-cloud sequences into discrete token sequences through four steps. Template-prior grouping defines $N_g$ 3D anchor points on a deformable bounding-box grid aligned to the human body template, and a P4Conv encoder produces group features. Masked trajectory modeling randomly masks 50% of anchor tubes across time and uses a transformer decoder with cross-attention to reconstruct them. Aggregated quantization assigns each time-slice feature to the nearest codebook vector:
$$
z_t = \arg\min_{z_k\in\mathcal Z}\bigl\|\mathbf F_{\text{all}}^t-z_k\bigr\|_2.
$$
The optimization combines reconstruction, embedding alignment, and commitment losses, where $\mathcal L_{\mathrm{rec}}$ is based on Chamfer distance on masked points, $\mathcal L_{\mathrm{emb}}=\|\mathbf F_{\mathrm{all}}-\mathbf F_{\mathrm{mot}}\|_2^2$, and $\mathcal L_{\mathrm{commit}}$ stabilizes the codebook [2504.09862].

The language model stage extends T5-Small’s vocabulary with radar tokens and the special markers `/som` and `/eom`. Radar and text share a 512-d embedding table and are concatenated before stacked cross-modal attention layers. Pre-training combines unsupervised mask reconstruction on radar tokens with supervised radar-to-text and text-to-radar translation losses:
$$
\mathcal L_{\mathrm{pre}}=\lambda_1\mathcal L_{\mathrm{pred}}+\lambda_2\mathcal L_{\mathrm{r2t}}+\lambda_3\mathcal L_{\mathrm{t2r}}.
$$
Some variants also add a symmetric InfoNCE alignment loss between global radar and text embeddings [2504.09862].

On the synthetic test set, RadarLLM achieves ROUGE-1 38.4, ROUGE-L 36.0, BLEU-4 11.4, METEOR 33.7, and CIDEr 8.3. On real data, it obtains ROUGE-1 31.7, ROUGE-L 28.8, BLEU-1 44.2, and BLEU-4 5.0. The comparisons include MotionGPT, AvatarGPT, and Video-LLaMA2. The ablations report that removing the template-prior anchors drops ROUGE-L from 36.0 to 25.1, training without masked-trajectory modeling cuts BLEU-4 by 42%, omitting the embedding alignment loss reduces CIDEr from 5.9 to 2.1, and T5-Small outperforms T5-Base and GPT-2 Medium when learning radar–text alignment [2504.09862].

This formulation broadens the scope of RadarLLM beyond detection. Instead of a classifier over clutter and target, the system generates natural-language motion descriptions such as “The subject stands upright, then bends forward slowly and returns to standing” and “The person takes two steps forward, stops, and turns 90° to the left” [2504.09862]. A plausible implication is that the RadarLLM concept can support not only sequence discrimination but also generative semantic grounding when a sufficiently structured tokenizer and alignment regime are available.

## 5. Vision-aligned 4D radar scene semantics

The weather-robust scene semantics pipeline extends RadarLLM-like ideas to 4D automotive radar and open-vocabulary scene description. Its starting point is the empirical observation that cameras and LiDAR degrade in rain, fog, and snow, whereas millimeter-wave radar remains largely unaffected. The model aligns a radar encoder to frozen SigLIP vision embeddings and decodes structured scene captions through a frozen Qwen2.5-VL-3B with approximately 7M trainable parameters [2605.07367].

The encoder consumes a radar “tesseract” $X_0\in\mathbb R^{64\times256\times37\times107}$, collapses elevation to form $X_1\in\mathbb R^{64\times256\times107}$, and augments the representation with R⁴-compensation and coordinate channels. Two input variants are defined: a 5-channel representation containing power, mean Doppler, peak Doppler, range, and azimuth, and a 66-channel representation containing the full 64-bin Doppler plus range and azimuth [2605.07367]. A ResNet-18 backbone with its first convolution resized to $C_{in}\in\{5,66\}$ processes the input. After Layer2, a PETR positional embedding maps normalized $(\mathrm{range},\mathrm{azimuth})$ through a two-layer MLP into $\mathbb R^{128}$ and adds it to the feature map. After Layer4, the feature map is $F\in\mathbb R^{512\times8\times4}$.

Token pooling and projection are explicit design axes. Average pooling over $4\times4$ produces $N=16$ tokens, whereas cross-attention pooling with 32 learned queries produces $N=32$ tokens. A two-layer MLP projector maps 512 to 2048 and then 2048, with GELU nonlinearity, and an output LayerNorm produces aligned tokens in the VLM embedding dimension [2605.07367]. The structured prompt prepends radar tokens to the template
`{"objects":[{"class","azimuth_deg","range_m"}]}`
and Qwen2.5-VL autoregressively decodes structured captions.

A central contribution is the diagnosis of token-norm mismatch. The paper reports empirical norms $\|f_{\mathrm{proj}}(T_{\mathrm{in}})\|_2\approx20$–200 versus native VLM word-embedding norms $\|e_{\mathrm{word}}\|_2\approx1$. The claimed mechanism is that large norms saturate softmax attention weights on radar tokens, so no gradient flows and the VLM ignores radar. The remedy is projector-output LayerNorm:
$$
\hat u_i=\mathrm{LN}(u_i)=\gamma\odot(u_i-\mu_i)/\sqrt{\sigma_i^2+\epsilon}+\beta,
$$
which enforces $E[\|\hat u_i\|_2]\approx O(1)$ and restores proper cross-modal attention [2605.07367]. In Stage 1, radar tokens are aligned to frozen SigLIP image embeddings using
$$
L_{\mathrm{align}}=\lambda_1\|\mathrm{mean}(\hat T_{\mathrm{proj}})-z_{\mathrm{vis}}\|_2^2+\lambda_2\,[1-\cos(\mathrm{mean}(\hat T_{\mathrm{proj}}),z_{\mathrm{vis}})],
$$
typically with $\lambda_1=\lambda_2=1$. In Stage 2, the ResNet-18 and projector are frozen, and LoRA adapters in Qwen2.5-VL are fine-tuned using caption-token cross-entropy.

The reported results on held-out fog, light snow, and heavy snow sequences from K-RADAR are notable for weather robustness. Aggregated over those weather conditions, the camera baseline has Class F1 0.336, Range MAE 17.0 m, Azimuth MAE 9.3°, and Hallucination 0.543. Radar 5ch+align+prose reaches F1 0.527, Range MAE 13.9 m, and Hallucination 0.296. Radar 66ch+align+prose reaches F1 0.473, Range MAE 10.9 m, and Hallucination 0.376. Radar 5ch+align+JSON reaches F1 0.540, Range MAE 16.6 m, Azimuth MAE 7.0°, and Hallucination 0.329. Radar 5ch+align+JSON+cross-attn reaches F1 0.491, Range MAE 13.2 m, Azimuth MAE 7.9°, and Hallucination 0.336 [2605.07367].

The design ablations make the tradeoffs unusually explicit. JSON decoding yields higher F1, better recall, and best azimuth MAE because digits directly supervise numeric outputs, whereas prose yields lower hallucination and better minority-class recall for bicycles and motorcycles. The 66-channel input improves frozen-encoder linear-probe object count RMSE from 2.20 to 2.03, but under approximately 8 k frames the 5-channel pipeline outperforms the 66-channel version in caption F1, 0.527 versus 0.473, with lower hallucination. Average pooling yields the highest F1 and lowest hallucination, while cross-attention pooling improves range MAE from 16.6 m to 13.2 m but worsens F1 and azimuth MAE. Without Stage 1 alignment, spatial grounding fails entirely; hallucination remains high at approximately 0.54 and bearing accuracy is poor. Replacing Qwen2.5-VL with text-only Qwen2.5 gives 8% worse validation loss, which the paper interprets as evidence that vision-conditioned attention benefits radar tokens [2605.07367].

## 6. Misconceptions, limitations, and research directions

A common misconception is that RadarLLM refers to a single standardized model. The published record instead shows several incompatible implementations under the same or closely related name: GPT-2-based binary detectors for marine radar [2509.12089; 2509.12110], a T5-based motion-to-text system for mmWave point clouds [2504.09862], and a SigLIP-plus-Qwen2.5-VL captioning pipeline for 4D radar scene semantics [2605.07367]. The term therefore identifies a family of strategies for adapting pretrained sequence models to radar, not a fixed architecture or benchmark protocol.

Another misconception is that the main benefit comes simply from replacing conventional radar networks with larger models. The papers assign the decisive role to modality adaptation and optimization details rather than raw model scale. In marine detection, directly fine-tuning pretrained LLMs tends to suffer from pronounced overfitting in low-SCR scenarios, motivating preference-aware token weighting [2509.12089]. In the patch-selection marine framework, partial fine-tuning outperforms full fine-tuning, and patch selection is a major contributor to performance [2509.12110]. In 4D radar captioning, the dominant failure mode is not insufficient backbone capacity but token-norm mismatch between projected radar tokens and native VLM embeddings; projector-output LayerNorm is identified as critical because otherwise the VLM ignores radar [2605.07367]. In human-motion understanding, performance depends strongly on the tokenizer, synthetic data pipeline, and radar–text alignment losses rather than on substituting a larger language model [2504.09862].

The limitations are likewise modality- and task-specific. For marine detection, the model size remains large at 69 M total parameters, and quantization, pruning, or knowledge distillation are explicitly highlighted as important for embedded platforms; the reference-model choice is also left open, with Mamba or iterative validation splits suggested as alternatives [2509.12089]. The patch-selection marine pipeline points to dynamic or hierarchical patch sizing, joint reconstruction anomaly detection, adaptation to varying sea states and hardware constraints, interpretability of which physical features the LLM attends to, and multimodal fusion with AIS or optical data [2509.12110]. The motion-understanding system notes device gap across radars with different antenna counts or resolutions, the restriction to single-person scenes, the absence of human-object interactions and environment context, and the possibility of long-term forecasting or retrieval-based prompting [2504.09862]. The weather-robust scene-semantics pipeline suggests that a regression or detection head on aligned embeddings may yield more precise metric outputs, and proposes LiDAR-as-teacher or cross-dataset radar alignment to enrich supervision [2605.07367].

A plausible synthesis is that RadarLLM research is converging on a common problem statement: pretrained language or vision-language models can be effective radar backbones only when the radar side supplies an appropriate tokenization, embedding scale, and supervision signal. The specific mechanisms differ—patch selection, preference-aware loss, vector quantization, or SigLIP alignment—but all four systems treat cross-modal compatibility as the central engineering and scientific challenge.

Source: https://www.emergentmind.com/topics/radarllm