RadarLLM: Adapting LLMs for Radar Inference

• RadarLLM is a paradigm that transforms radar measurements into language-like tokens for processing by largely frozen pretrained language and vision-language models.
• It leverages specialized preprocessing, tokenization, and lightweight adaptations (e.g., LoRA, LayerNorm) to optimize detection, motion understanding, and scene captioning.
• Experimental results demonstrate improved detection rates and semantic accuracy across marine, human-motion, and weather-robust automotive tasks while mitigating overfitting.

Searching arXiv for the specified RadarLLM papers and closely related radar–LLM work. RadarLLM denotes a line of work in which radar measurements are adapted to pretrained LLMs or vision-LLMs for downstream semantic inference. In the current literature, the label refers not to a single canonical architecture but to several task-specific frameworks: a marine radar target detector built on GPT-2 with a preference-aware loss (Hu, 15 Sep 2025), a related marine detection pipeline that combines feature preprocessing, patch selection, and partial LayerNorm fine-tuning of GPT-2 (Hu et al., 15 Sep 2025), a millimeter-wave human-motion understanding system that couples a motion-guided radar tokenizer with a radar-aware T5 model (Lai et al., 14 Apr 2025), and a vision-aligned 4D radar captioning pipeline for weather-robust scene semantics that aligns radar embeddings to frozen SigLIP and decodes structured captions with Qwen2.5-VL (Hamilton et al., 8 May 2026). Across these variants, the unifying idea is to cast radar signals, point clouds, or radar tensors into token sequences or embedding sets that can be processed by pretrained sequence models while keeping most backbone parameters frozen.

1. Conceptual scope and historical emergence

RadarLLM emerged from the observation that pretrained LLMs can be repurposed as generic sequence learners for radar-derived representations. In marine radar target detection, the central claim is that a pre-trained LLM can serve as a backbone for small-target detection once radar echoes are converted into “language-like” token streams and adapted through lightweight fine-tuning (Hu, 15 Sep 2025). In a parallel formulation for marine echo analysis, the framework integrates feature preprocessing with LLMs by tokenizing radar sequence features, filtering uninformative segments, projecting them into the pretrained embedding space, and fine-tuning only normalization layers (Hu et al., 15 Sep 2025). In millimeter-wave motion understanding, Radar-LLM is presented as the first framework that leverages LLMs for human motion understanding using millimeter-wave radar as the sensing modality, with discrete radar tokens aligned to text in a shared embedding space (Lai et al., 14 Apr 2025). In weather-robust automotive semantics, the term is used for a pipeline in which a radar encoder is aligned to frozen SigLIP vision embeddings and a frozen Qwen2.5-VL decoder generates structured scene captions from radar tokens (Hamilton et al., 8 May 2026).

A plausible implication is that RadarLLM is best understood as a research paradigm rather than a single model family. The shared premise is consistent across the papers: radar is not treated as a modality requiring entirely bespoke architectures, but as a source of structured sequences or tokens that can be mapped into pretrained model spaces. The major variation lies in what is tokenized, which pretrained backbone is used, and whether the downstream task is binary detection, motion-to-text generation, or open-vocabulary scene captioning.

2. Common architectural patterns

Despite targeting different radar modalities and tasks, the RadarLLM papers exhibit a recurring pipeline structure: radar-specific preprocessing, tokenization or projection, lightweight adaptation of a largely frozen pretrained model, and task-specific decoding. The concrete realizations differ substantially.

Variant	Radar representation	Backbone adaptation
Marine detection with preference-aware loss	Five sequence features split into non-overlapping patches	Smallest GPT2, first four layers, LoRA on $W_Q$ and $W_V$, LayerNorm fine-tuning (Hu, 15 Sep 2025)
Marine detection with patch selection	Five sequence features, top-$y\%$ selected patches	GPT-2 small truncated to first 6 layers, only LayerNorm and classification head trainable (Hu et al., 15 Sep 2025)
Human motion understanding	Aggregate VQ-VAE tokens from mmWave point-cloud sequences	T5-Small with extended radar-token vocabulary and cross-modal attention (Lai et al., 14 Apr 2025)
Weather-robust scene semantics	4D radar tokens from a ResNet-18 encoder and projector	Frozen Qwen2.5-VL-3B with LoRA adapters in all Q/K projections (Hamilton et al., 8 May 2026)

In the marine detection setting with preference-aware loss, each radar return $x(n)$ is converted into five complementary sequence features: instantaneous phase, Doppler spectrum entropy, short-time Fourier magnitude spectrum, amplitude, and Doppler phase. Each feature vector of length $N$ is split into non-overlapping patches of length $L$, yielding $K=5\lceil N/L\rceil$ tokens per sample; a linear projection maps each patch into a GPT2-compatible embedding, and a sinusoidal positional encoding is added before feeding the tokens into GPT2 (Hu, 15 Sep 2025). In the related marine framework, the same five-feature decomposition is used, but a small reference transformer computes global patch importance scores, and only the top $y\%$ patches are retained before projection into the pretrained LLM space (Hu et al., 15 Sep 2025).

In the motion-understanding formulation, the tokenization problem is substantially more elaborate. A Motion-guided Radar Tokenizer based on Aggregate VQ-VAE converts a point-cloud sequence $\mathbf P_{1:T}\in\mathbb R^{T\times N\times4}$ into a shorter discrete token sequence $\mathbf z_{1:L}\in\{1,\dots,K\}^L$. The tokenizer uses template-prior grouping, masked trajectory modeling, and aggregated quantization with a codebook $W_V$0 (Lai et al., 14 Apr 2025). By contrast, the weather-robust scene semantics pipeline does not quantize to discrete radar symbols. It begins from a 4D radar “tesseract” $W_V$1, collapses elevation by max pooling, forms either a 5-channel or 66-channel representation, and encodes the result with ResNet-18 plus PETR positional embedding before token pooling and projection into the VLM token space (Hamilton et al., 8 May 2026).

A consistent design principle across these systems is parameter efficiency. The marine preference-aware detector trains 2.46 M out of 69.4 M parameters by freezing almost all GPT2 weights and adapting the model through LoRA plus LayerNorm updates (Hu, 15 Sep 2025). The patch-selection marine framework trains only LayerNorm parameters and a final classification head in the truncated GPT-2 (Hu et al., 15 Sep 2025). The 4D radar captioning system uses approximately 7M trainable parameters while keeping the SigLIP vision encoder and Qwen2.5-VL-3B frozen, aside from LoRA adapters of rank 16 inserted into all Q/K projections (Hamilton et al., 8 May 2026).

3. Marine radar target detection formulations

Marine radar target detection is the earliest and most explicit setting in which RadarLLM is framed as adapting a pretrained LLM to radar sequence analysis. The preference-aware variant uses the smallest GPT2 with hidden size $W_V$2 and activates its first four layers as a feature extractor. Through Low-Rank Adaptation on the multi-head attention’s query and value matrices, plus fine-tuning LayerNorm parameters, the architecture remains largely frozen while a layer-norm and two fully connected layers produce a $W_V$3 token-level output (Hu, 15 Sep 2025). The training protocol specifies batch size 64, evaluation batch 400, 500 epochs for LLM fine-tuning, Adam with $W_V$4, learning rate $W_V$5, patch length $W_V$6, and early stopping on validation.

The distinctive methodological contribution of that paper is the preference-aware loss. Two models are maintained: a lightweight reference model $W_V$7 trained on validation samples only and the target LLM-based detector $W_V$8. For each feature token, the token importance score is

$W_V$9

and the final mini-batch loss is

$y\%$0

The stated interpretation is that tokens on which the larger LLM still errs more than the reference model are the most worthy of further training, whereas trivial or overly noisy tokens receive zero weight (Hu, 15 Sep 2025).

The experimental setting uses eight real-world X-band marine radar sequences from the IPIX Archive, each with 14 range cells and 131 072 echoes at 1 kHz; Data1–3 are low-SCR scenes with $y\%$1 dB, and Data4–8 are high-SCR scenes with $y\%$2 dB. The split uses the first 20% of time for training, the next 15% for validation, and the remaining 65% for testing. At a fixed false alarm rate $y\%$3, RadarLLM achieves an average detection rate of 81.96%, compared with 80.82% for ADN18. Under limited data with 10% training, RadarLLM attains 78.91%, compared with 77.18% for ADN18 and 69.56% for GRU; this is described as a +9.9 pp improvement over LLM+CE fine-tuning. On Data1 with SCR approximately 7 dB, preference-aware loss lifts average detection from 34.7% to 52.1%, with further reported improvements of +5.9 pp on Data2 and +7.5 pp on Data3 (Hu, 15 Sep 2025).

The related paper “When marine radar target detection meets pretrained LLMs” uses a different optimization strategy but reaches a similar conclusion about LLM transfer. Here, a randomly initialized transformer computes global patch importance by averaging $y\%$4 over training samples and attention heads, retaining only the top $y\%$5 patches. Those selected patches are projected into the GPT-2 feature space, standard sinusoidal positional encodings are added, and GPT-2 small is truncated to its first 6 transformer layers (Hu et al., 15 Sep 2025). All self-attention and feed-forward weights are frozen; only LayerNorm scale and shift parameters, together with the final classification head, are trainable.

On nine IPIX datasets in HH polarization, with the first 10% of time for training, the next 5% for validation, and the remainder for testing, the full system LLM4TS(S) achieves an average detection rate of 72.06% at $y\%$6. The best non-LLM baseline, ADN18, reaches 66.18%; PatchTST reaches 64.35%, and PatchTST(S) 69.16%; LLM4TS(0) reaches 62.06%; LLM4TS(F) reaches 67.93%; and LLM4TS with partial fine-tuning but no patch selection reaches 69.49% (Hu et al., 15 Sep 2025). The ablations report that patch selection alone adds up to +15.5% on IPIX #17 for the reference model and +3.7% for the LLM pipeline, that partial fine-tuning outperforms full fine-tuning by +1.56%, and that pretraining improves over random initialization by +7.43%. The same paper also reports 1,334 samples/s for LLM4TS(S), compared with 1,074 samples/s for the standard Transformer, attributing the throughput gain to token sparsification and GPT inference optimizations.

Taken together, these marine studies establish two recurrent claims. First, pretrained GPT-2 backbones can outperform conventional RNN, Transformer, and CNN baselines on radar detection. Second, overfitting in low-SCR or limited-data regimes is a central failure mode, addressed either by preference-aware token reweighting (Hu, 15 Sep 2025) or by reference-model-driven patch selection with minimal parameter adaptation (Hu et al., 15 Sep 2025).

4. Millimeter-wave human-motion understanding

In human-motion understanding, RadarLLM addresses a substantially different radar regime: sparse millimeter-wave point-cloud sequences rather than marine echo sequences. The framework is organized into three stages: physics-aware synthesis of a radar–text corpus from motion–text data, a Motion-guided Radar Tokenizer based on Aggregate VQ-VAE, and a Radar-aware LLM based on T5 that aligns radar tokens and words in a shared embedding space (Lai et al., 14 Apr 2025).

The synthesis stage is designed to compensate for the lack of paired radar–text data. The system simulates realistic mmWave returns from SMPL X motion sequences from HumanML3D and inherits their captions. Ray tracing with RF adaptive sampling uses a Sobel-style edge detector to focus rays on high-curvature regions; for each path, the intermediate frequency is approximated using the Physical Optics Integral:

$y\%$7

A 1D Range-FFT is computed per chirp, followed by a 1D Doppler-FFT across slow time, static clutter is removed by subtracting the temporal mean of each range bin, and the top $y\%$8 peaks are extracted from each heatmap to form point clouds $y\%$9 (Lai et al., 14 Apr 2025). Real-world validation uses 375 sequences, corresponding to 125 motions with 3 trials, collected on TI AWR1843BOOST.

The Motion-guided Radar Tokenizer converts noisy point-cloud sequences into discrete token sequences through four steps. Template-prior grouping defines $x(n)$0 3D anchor points on a deformable bounding-box grid aligned to the human body template, and a P4Conv encoder produces group features. Masked trajectory modeling randomly masks 50% of anchor tubes across time and uses a transformer decoder with cross-attention to reconstruct them. Aggregated quantization assigns each time-slice feature to the nearest codebook vector:

$x(n)$1

The optimization combines reconstruction, embedding alignment, and commitment losses, where $x(n)$2 is based on Chamfer distance on masked points, $x(n)$3, and $x(n)$4 stabilizes the codebook (Lai et al., 14 Apr 2025).

The LLM stage extends T5-Small’s vocabulary with radar tokens and the special markers /som and /eom. Radar and text share a 512-d embedding table and are concatenated before stacked cross-modal attention layers. Pre-training combines unsupervised mask reconstruction on radar tokens with supervised radar-to-text and text-to-radar translation losses:

$x(n)$5

Some variants also add a symmetric InfoNCE alignment loss between global radar and text embeddings (Lai et al., 14 Apr 2025).

On the synthetic test set, RadarLLM achieves ROUGE-1 38.4, ROUGE-L 36.0, BLEU-4 11.4, METEOR 33.7, and CIDEr 8.3. On real data, it obtains ROUGE-1 31.7, ROUGE-L 28.8, BLEU-1 44.2, and BLEU-4 5.0. The comparisons include MotionGPT, AvatarGPT, and Video-LLaMA2. The ablations report that removing the template-prior anchors drops ROUGE-L from 36.0 to 25.1, training without masked-trajectory modeling cuts BLEU-4 by 42%, omitting the embedding alignment loss reduces CIDEr from 5.9 to 2.1, and T5-Small outperforms T5-Base and GPT-2 Medium when learning radar–text alignment (Lai et al., 14 Apr 2025).

This formulation broadens the scope of RadarLLM beyond detection. Instead of a classifier over clutter and target, the system generates natural-language motion descriptions such as “The subject stands upright, then bends forward slowly and returns to standing” and “The person takes two steps forward, stops, and turns 90° to the left” (Lai et al., 14 Apr 2025). A plausible implication is that the RadarLLM concept can support not only sequence discrimination but also generative semantic grounding when a sufficiently structured tokenizer and alignment regime are available.

5. Vision-aligned 4D radar scene semantics

The weather-robust scene semantics pipeline extends RadarLLM-like ideas to 4D automotive radar and open-vocabulary scene description. Its starting point is the empirical observation that cameras and LiDAR degrade in rain, fog, and snow, whereas millimeter-wave radar remains largely unaffected. The model aligns a radar encoder to frozen SigLIP vision embeddings and decodes structured scene captions through a frozen Qwen2.5-VL-3B with approximately 7M trainable parameters (Hamilton et al., 8 May 2026).

The encoder consumes a radar “tesseract” $x(n)$6, collapses elevation to form $x(n)$7, and augments the representation with R⁴-compensation and coordinate channels. Two input variants are defined: a 5-channel representation containing power, mean Doppler, peak Doppler, range, and azimuth, and a 66-channel representation containing the full 64-bin Doppler plus range and azimuth (Hamilton et al., 8 May 2026). A ResNet-18 backbone with its first convolution resized to $x(n)$8 processes the input. After Layer2, a PETR positional embedding maps normalized $x(n)$9 through a two-layer MLP into $N$0 and adds it to the feature map. After Layer4, the feature map is $N$1.

Token pooling and projection are explicit design axes. Average pooling over $N$2 produces $N$3 tokens, whereas cross-attention pooling with 32 learned queries produces $N$4 tokens. A two-layer MLP projector maps 512 to 2048 and then 2048, with GELU nonlinearity, and an output LayerNorm produces aligned tokens in the VLM embedding dimension (Hamilton et al., 8 May 2026). The structured prompt prepends radar tokens to the template {"objects":[{"class","azimuth_deg","range_m"}]} and Qwen2.5-VL autoregressively decodes structured captions.

A central contribution is the diagnosis of token-norm mismatch. The paper reports empirical norms $N$5–200 versus native VLM word-embedding norms $N$6. The claimed mechanism is that large norms saturate softmax attention weights on radar tokens, so no gradient flows and the VLM ignores radar. The remedy is projector-output LayerNorm:

$N$7

which enforces $N$8 and restores proper cross-modal attention (Hamilton et al., 8 May 2026). In Stage 1, radar tokens are aligned to frozen SigLIP image embeddings using

$N$9

typically with $L$0. In Stage 2, the ResNet-18 and projector are frozen, and LoRA adapters in Qwen2.5-VL are fine-tuned using caption-token cross-entropy.

The reported results on held-out fog, light snow, and heavy snow sequences from K-RADAR are notable for weather robustness. Aggregated over those weather conditions, the camera baseline has Class F1 0.336, Range MAE 17.0 m, Azimuth MAE 9.3°, and Hallucination 0.543. Radar 5ch+align+prose reaches F1 0.527, Range MAE 13.9 m, and Hallucination 0.296. Radar 66ch+align+prose reaches F1 0.473, Range MAE 10.9 m, and Hallucination 0.376. Radar 5ch+align+JSON reaches F1 0.540, Range MAE 16.6 m, Azimuth MAE 7.0°, and Hallucination 0.329. Radar 5ch+align+JSON+cross-attn reaches F1 0.491, Range MAE 13.2 m, Azimuth MAE 7.9°, and Hallucination 0.336 (Hamilton et al., 8 May 2026).

The design ablations make the tradeoffs unusually explicit. JSON decoding yields higher F1, better recall, and best azimuth MAE because digits directly supervise numeric outputs, whereas prose yields lower hallucination and better minority-class recall for bicycles and motorcycles. The 66-channel input improves frozen-encoder linear-probe object count RMSE from 2.20 to 2.03, but under approximately 8 k frames the 5-channel pipeline outperforms the 66-channel version in caption F1, 0.527 versus 0.473, with lower hallucination. Average pooling yields the highest F1 and lowest hallucination, while cross-attention pooling improves range MAE from 16.6 m to 13.2 m but worsens F1 and azimuth MAE. Without Stage 1 alignment, spatial grounding fails entirely; hallucination remains high at approximately 0.54 and bearing accuracy is poor. Replacing Qwen2.5-VL with text-only Qwen2.5 gives 8% worse validation loss, which the paper interprets as evidence that vision-conditioned attention benefits radar tokens (Hamilton et al., 8 May 2026).

6. Misconceptions, limitations, and research directions

A common misconception is that RadarLLM refers to a single standardized model. The published record instead shows several incompatible implementations under the same or closely related name: GPT-2-based binary detectors for marine radar (Hu, 15 Sep 2025, Hu et al., 15 Sep 2025), a T5-based motion-to-text system for mmWave point clouds (Lai et al., 14 Apr 2025), and a SigLIP-plus-Qwen2.5-VL captioning pipeline for 4D radar scene semantics (Hamilton et al., 8 May 2026). The term therefore identifies a family of strategies for adapting pretrained sequence models to radar, not a fixed architecture or benchmark protocol.

Another misconception is that the main benefit comes simply from replacing conventional radar networks with larger models. The papers assign the decisive role to modality adaptation and optimization details rather than raw model scale. In marine detection, directly fine-tuning pretrained LLMs tends to suffer from pronounced overfitting in low-SCR scenarios, motivating preference-aware token weighting (Hu, 15 Sep 2025). In the patch-selection marine framework, partial fine-tuning outperforms full fine-tuning, and patch selection is a major contributor to performance (Hu et al., 15 Sep 2025). In 4D radar captioning, the dominant failure mode is not insufficient backbone capacity but token-norm mismatch between projected radar tokens and native VLM embeddings; projector-output LayerNorm is identified as critical because otherwise the VLM ignores radar (Hamilton et al., 8 May 2026). In human-motion understanding, performance depends strongly on the tokenizer, synthetic data pipeline, and radar–text alignment losses rather than on substituting a larger LLM (Lai et al., 14 Apr 2025).

The limitations are likewise modality- and task-specific. For marine detection, the model size remains large at 69 M total parameters, and quantization, pruning, or knowledge distillation are explicitly highlighted as important for embedded platforms; the reference-model choice is also left open, with Mamba or iterative validation splits suggested as alternatives (Hu, 15 Sep 2025). The patch-selection marine pipeline points to dynamic or hierarchical patch sizing, joint reconstruction anomaly detection, adaptation to varying sea states and hardware constraints, interpretability of which physical features the LLM attends to, and multimodal fusion with AIS or optical data (Hu et al., 15 Sep 2025). The motion-understanding system notes device gap across radars with different antenna counts or resolutions, the restriction to single-person scenes, the absence of human-object interactions and environment context, and the possibility of long-term forecasting or retrieval-based prompting (Lai et al., 14 Apr 2025). The weather-robust scene-semantics pipeline suggests that a regression or detection head on aligned embeddings may yield more precise metric outputs, and proposes LiDAR-as-teacher or cross-dataset radar alignment to enrich supervision (Hamilton et al., 8 May 2026).

A plausible synthesis is that RadarLLM research is converging on a common problem statement: pretrained language or vision-LLMs can be effective radar backbones only when the radar side supplies an appropriate tokenization, embedding scale, and supervision signal. The specific mechanisms differ—patch selection, preference-aware loss, vector quantization, or SigLIP alignment—but all four systems treat cross-modal compatibility as the central engineering and scientific challenge.