VLMaterial: Vision-Language Model-Based Camera-Radar Fusion for Physics-Grounded Material Identification

Abstract: Accurate material recognition is a fundamental capability for intelligent perception systems to interact safely and effectively with the physical world. For instance, distinguishing visually similar objects like glass and plastic cups is critical for safety but challenging for vision-based methods due to specular reflections, transparency, and visual deception. While millimeter-wave (mmWave) radar offers robust material sensing regardless of lighting, existing camera-radar fusion methods are limited to closed-set categories and lack semantic interpretability. In this paper, we introduce VLMaterial, a training-free framework that fuses vision-LLMs (VLMs) with domain-specific radar knowledge for physics-grounded material identification. First, we propose a dual-pipeline architecture: an optical pipeline uses the segment anything model and VLM for material candidate proposals, while an electromagnetic characterization pipeline extracts the intrinsic dielectric constant from radar signals via an effective peak reflection cell area (PRCA) method and weighted vector synthesis. Second, we employ a context-augmented generation (CAG) strategy to equip the VLM with radar-specific physical knowledge, enabling it to interpret electromagnetic parameters as stable references. Third, an adaptive fusion mechanism is introduced to intelligently integrate outputs from both sensors by resolving cross-modal conflicts based on uncertainty estimation. We evaluated VLMaterial in over 120 real-world experiments involving 41 diverse everyday objects and 4 typical visually deceptive counterfeits across varying environments. Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks while eliminating the need for extensive task-specific data collection and training.

• The paper introduces a training-free camera–radar fusion framework that extracts dielectric constant from 60 GHz mmWave echoes and uses a vision-language model for material reasoning.
• VLMaterial achieves 96.08% accuracy across 120+ real-world experiments, outperforming vision-only, radar-only, and prior training-free methods while identifying all four tested counterfeits.
• The paper shows that PRCA normalization, context-augmented electromagnetic references, and uncertainty-aware fusion improve robustness to reflections, dim lighting, ambiguous appearances, and object geometry.

VLMaterial addresses a persistent failure mode of vision-LLMs (VLMs): they recognize object categories reliably but cannot determine what an object is made of when appearance is ambiguous, reflective, or deliberately deceptive. The paper proposes a training-free camera–radar fusion framework that grounds material identification in physics — specifically, the relative dielectric constant $\varepsilon_r$ extracted from 60 GHz mmWave radar echoes — and delegates semantic reasoning to a VLM augmented with domain-specific electromagnetic knowledge. Evaluated on 41 everyday objects plus four counterfeit items across three realistic environments, the system achieves 96.08% recognition accuracy without any task-specific training.

Motivation and preliminary findings

The authors first quantify how poorly current VLMs perform at material identification. Using SAM for segmentation followed by Gemini-3-Pro, GPT-5, and Qwen3-VL on 41 everyday objects spanning six materials and seven categories, all models achieved 100% category accuracy but only 75.61% (Gemini-3-Pro), 73.17% (GPT-5), and 68.29% (Qwen3-VL) material accuracy. A notable finding from repeated querying is that VLM errors are deterministic rather than random: a transparent plastic cup was misclassified as glass in all 100 trials. The authors categorize visual-only failures into specular reflections (metal mistaken for mirror glass), visual similarity (frosted glass vs. plastic), and visually deceptive objects (paper resembling plastic).

On the radar side, direct feeding of raw data, point clouds, or heatmaps into a VLM fails due to modality gap and sparsity, while prior reflection-intensity methods are closed-set classifiers requiring training. This motivates four design challenges: extracting intrinsic material properties from reflections, identifying universally interpretable electromagnetic parameters, bridging the semantic gap between raw radar signals and VLMs, and implementing adaptive multimodal fusion.

Electromagnetic characterization pipeline

The core signal-processing contribution converts raw FMCW radar returns (60 GHz carrier, 3.96 GHz bandwidth) into physically meaningful parameters. After range-Doppler and range-angle map generation via 2D-FFT and digital beamforming, the system calibrates RCS using a metal sphere reference target ($d = 63$ mm), yielding a one-time calibration constant $K$ that removes distance dependence via the radar equation:

$$\sigma = \sigma_c \cdot \frac{SNR}{SNR_c} \cdot \left( \frac{R}{R_c} \right)^4.$$

Two practical geometric deviations — angular misalignment and surface curvature dispersion — degrade naive SNR estimates. To address these, the authors introduce weighted vector synthesis: each antenna's focused signal component is phase-calibrated against a metallic sphere reference, coherently summed, and weighted by its projection onto the total coherent sum. Antennas receiving phase-coherent specular reflections receive large positive weights; those receiving noise or multipath are suppressed. The resulting weighted vector's magnitude indicates reflected strength and its direction reveals surface normal orientation. A coherence factor $$c(\mathbf{v}) = |S(\mathbf{v})|^2 / (N \sum_j |I_j(\mathbf{v})|^2)$$ validates signal quality and suppresses ghost targets before final SNR computation.

To decouple material from geometry, the paper introduces the peak reflection cell area (PRCA): instead of total RCS, which conflates shape with composition, the system computes a power reflection coefficient $\rho = \sigma / A_r$ over the dominant −3 dB beam footprint, analogous to the backscattering coefficient $\sigma_0$. Normalizing $\rho$ yields the Fresnel reflection coefficient $r_p$, which maps to dielectric constant through the p-polarization Fresnel relation; at normal incidence this simplifies to $\varepsilon_r = ((1 + r_p)/(1 - r_p))^2$, assuming $d = 63$0 for non-magnetic everyday materials.

Validation against ITU-R P.2040 theoretical baselines shows physically consistent ordering: metal ($d = 63$1), glass ($d = 63$2–10.0), wood ($d = 63$3), paper ($d = 63$4). The authors concede a slight positive bias for porous hygroscopic materials (wood, paper) attributed to environmental moisture absent from dry-condition ITU models — a deviation they argue justifies empirical rather than purely theoretical references.

Context-augmented generation

Because public knowledge lacks 60 GHz electromagnetic characterizations, the authors construct a context-augmented generation (CAG) knowledge base by recording mmWave echoes from seven flat material boards under ideal vertical incidence. Eight radar-derived parameters (distance, velocity, angle, SNR, $d = 63$5, $d = 63$6, $d = 63$7, $d = 63$8) are supplied to the VLM alongside this reference set. CAG proves essential: the measured dielectric constant of a metal cup is 30.51, far from the theoretical infinity a VLM would expect from priors; without CAG, the model misclassifies it as ceramic. Tracing outputs to knowledge-base references also provides transparency and mitigates hallucination on out-of-domain queries.

Adaptive fusion mechanism

Fusion operates on candidate sets from both pipelines. When $d = 63$9, the intersection is selected. For disjoint sets, an uncertainty-aware conflict resolution assigns per-modality confidence scores grounded in physical degradation factors: visual uncertainty combines photon-shot-noise-driven luminance loss, scene clutter, and VLM epistemic entropy; radar uncertainty combines inverse SNR, beam divergence $K$0, and oblique-incidence loss $K$1. Notably, the scaling coefficients are never computed numerically; instead, the mathematical relationships are encoded in prompts, letting the VLM realize them as attention weights — an approach the authors justify by citing evidence that LLMs can articulate their own uncertainty when contextually grounded. Softmax-style negative exponential gating then produces fusion weights $K$2 and $K$3, and the VLM arbitrates over the resulting weighted scores.

An ablation comparing this against naive prompt-guided fusion (packing everything into one prompt) shows marked instability: the unstructured approach exhibits both over-reliance on vision (ceramic as glass) and blind trust in radar (frosted glass as plastic). This supports the paper's claim that explicit signal-quality evaluation is necessary for reliable fusion.

Results

Using an IWR6843ISK radar co-mounted with a RealSense D435i, with Gemini-3-Pro as the reasoning backbone, VLMaterial achieves 96.08% overall accuracy across 120+ real-world experiments — matching the trained closed-set CRFUSION benchmark (96%) and exceeding mSense (93%), while dramatically outperforming the training-free LLMaterial baseline (19.69%), VLM-only (78.74%), and radar-only operation (41.73%). Per-material accuracy reaches 100% for five of seven categories, with plastic at 94.44% and paper at 80.00%; the paper failures trace to a single bottle whose smooth processed surface generates strong reflections mimicking metal.

Micro-benchmarks establish robustness along several axes. Recognition is invariant to board size and thickness, attributable to PRCA normalization. Accuracy is stable across distances of 25 cm and 50 cm, with one exception (a small heart-shaped box whose weak echo yields low radar confidence). Halving ADC samples per frame from 512 to 256 degrades distance resolution and causes systematic misclassifications of small objects, indicating the system operates at the hardware's resolution limit. Under dim lighting, VLM-only drops from 75.61% to 70.73%, whereas VLMaterial remains robust because radar corrects most visual errors — though dim lighting can occasionally help by suppressing specular reflections. Increasing camera resolution 25-fold with an iPhone 15 yields only marginal gains, confirming that resolution alone cannot resolve visual ambiguity.

Most strikingly, on four counterfeit objects (aluminum-plastic panel imitating ceramic, ceramic board imitating wood, two plastics imitating wood and ceramic finishes), the VLM alone failed on every item, yet VLMaterial identified all four correctly by resolving cross-modal conflicts — demonstrating penetration of deceptive appearances to intrinsic composition.

Limitations and open questions

The authors identify several constraints. SAM-based segmentation restricts the system to single-object scenes; multi-object settings require sequential radar measurement, limiting throughput. Under extreme visual degradation such as total darkness, the VLM cannot provide usable cues, so fusion cannot recover — motivating future integration of lighting-invariant modalities like tactile or acoustic sensing. Reliance on SOTA-scale VLMs introduces inference latency incompatible with real-time deployment, and the authors propose knowledge distillation as a remedy without demonstrating it. The framework assumes relatively smooth surfaces oriented toward the radar at close range (~25 cm); rough surfaces causing diffuse scattering already produce radar-side errors (e.g., ceramic misread due to scattering). Finally, the system is static; robotic-arm integration for active perception remains untested. An open question the results leave unresolved is whether the qualitative, prompt-encoded uncertainty weighting generalizes beyond the tested object distribution, given that the single consistent failure (the processed-paper bottle) arose precisely from misplaced trust in a high-SNR radar signal.

Conclusion

VLMaterial demonstrates that a VLM can interpret object materials when supplied with physics-grounded electromagnetic measurements rather than raw sensor data. Its contributions — PRCA-based dielectric extraction robust to geometry, CAG grounding against mismatched theoretical priors, and uncertainty-aware adaptive fusion — yield 96.08% training-free accuracy on realistic everyday objects, including perfect detection of visually deceptive counterfeits that defeat vision-only inference entirely. The result establishes that complementary physical sensing can compensate for the representational limits of RGB-based perception, while leaving scalability to multi-object scenes, extreme illumination conditions, and real-time operation as open problems.