Phy3DAdvGen: Text-to-3D Adversarial Generation

• The paper introduces Phy3DAdvGen, a detector-aware attack that optimizes text prompts to generate human-object 3D compositions which evade LiDAR-based pedestrian detection.
• It systematically studies pedestrian detectability in CARLA, revealing that multi-object semantic combinations can reduce detection success to as low as 9.9% across various detectors.
• The method achieves high attack success rates in simulation and physical tests by leveraging latent prompt optimization, real object pools, and differentiable Gumbel-Softmax weighting.

Physically-Informed Text-to-3D Adversarial Generation (Phy3DAdvGen) is a detector-aware attack paradigm in which text prompts, rather than point clouds or abstract meshes, are adversarially optimized so that a text-to-3D generator synthesizes semantically plausible, physically deployable human-object compositions that evade LiDAR-based 3D pedestrian detectors. It was introduced in "OBJVanish: Physically Realizable Text-to-3D Adv. Generation of LiDAR-Invisible Objects" (Li et al., 8 Oct 2025), which treats the problem as a safety-critical robustness issue for autonomous driving and defines "LiDAR-invisible objects" as objects that remain present in the scene and return LiDAR points, yet fail to produce a valid pedestrian detection under the paper’s detection criteria.

1. Threat Model and Operational Definition

Phy3DAdvGen targets LiDAR-based 3D object detectors used in autonomous driving. The attacker does not need access to internal detector weights and does not tamper with the raw LiDAR stream through sensor spoofing. Instead, the attacker places or realizes a carefully designed 3D adversarial object in the scene, typically a pedestrian combined with additional objects in a particular pose, so that the detector fails to output a valid pedestrian box (Li et al., 8 Oct 2025).

The paper explicitly distinguishes detector invisibility from literal sensor invisibility. A detection is considered successful if the predicted box has IoU $> 0.5$ with ground truth. An attack is successful if no box is detected, or if the best detection has IoU $< 0.1$. Within this framing, a "LiDAR-invisible object" is not an object that ceases to generate returns, but an object that causes downstream failure of the LiDAR 3D detection stack. This distinction is central because the method attacks the perception model’s pedestrian prior rather than the sensing physics itself.

The motivation is a critique of two earlier families of LiDAR adversarial attacks. Point perturbation and mesh-based attacks are described as rarely achieving complete object disappearance and as weakly physically realizable. Phy3DAdvGen responds by moving the optimization target from sensor-space perturbations to fully generated 3D content. A plausible implication is that the attack surface shifts from low-level signal corruption to high-level semantic structure: the adversary no longer asks how to perturb a captured scan, but what physically plausible object configuration a detector is least likely to recognize as a pedestrian.

2. Empirical Basis for the "Physically-Informed" Design

A major contribution of the underlying work is the first systematic empirical study of pedestrian detectability factors in CARLA. The study manipulates topology, connectivity, intensity / reflectance, and human-object combinations, while varying viewing angle, distance, and ego-vehicle speed. Detectability is formalized as

$$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$

where $\gamma$ is detector confidence, $\tilde{\mathrm{Ped}}$ is the manipulated pedestrian, $\mathrm{AddObj}$ is the additional combined object, and $\mathcal{A}, \mathcal{D}, \mathcal{V}$ denote viewing angle, distance, and ego velocity (Li et al., 8 Oct 2025).

The central empirical finding is that isolated attribute manipulations have only moderate and detector-dependent effect, whereas human-object combinations produce the strongest and most consistent degradation. For a normal single pedestrian, detection success remains high across the six evaluated detectors, roughly $81.3\%-91.0\%$ in Scene 1, $76.4\%-95.3\%$ in Scene 2, and $83.0\%-96.3\%$ in Scene 3. Topology changes can reduce performance substantially for some models, such as PointRCNN from 89.6 to 52.1 in Scene 1 and IA-SSD to 23.4, but models such as SAFDNet and VoxelNeXt often remain comparatively strong. Connectivity and intensity changes are weaker still, with Scene 1 detection-success values remaining around 67.0–83.3 for connectivity and 69.5–84.3 for intensity.

By contrast, pedestrian-object combinations reduce detection success across all six detectors and all three scenes: Scene 1 values are 37.6, 31.7, 36.1, 26.2, 30.1, and 21.1 for PointRCNN, IA-SSD, SAFDNet, VoxelNeXt, HEDNet, and PDV; Scene 2 values are 30.8, 25.0, 21.8, 16.4, 17.7, and 9.9; Scene 3 values are 36.4, 36.2, 32.0, 21.8, 18.7, and 11.4. This result is the empirical basis for the method’s emphasis on multi-object semantic composition rather than direct geometric perturbation.

The "physically-informed" qualifier refers to the use of domain knowledge extracted from this study. The prompt grammar is constrained to semantically meaningful verb-object-pose interactions, the object set is restricted to real deployable props, and pose semantics are chosen to target vulnerable body regions. Appendix experiments reinforce this rationale: torso occlusion achieves 80.4% average attack success rate, feet occlusion 78.1%, and head occlusion 52.3%. This suggests that successful adversarial prompts work in part by disturbing torso and lower-body structure that LiDAR pedestrian detectors rely on.

Distance is identified as the strongest environmental factor, whereas view angle and ego motion have milder effects than expected. The paper concludes that generated adversarial objects show comparatively better viewpoint and motion robustness than simple point addition or removal attacks. In encyclopedic terms, the empirical study functions as the design layer that converts qualitative intuition about occlusion into a prompt-search prior.

3. Prompt Space, Objective, and Latent Optimization

Phy3DAdvGen defines a discrete prompt space using a verb-object-pose decomposition. The full prompt is composed as

$< 0.1$0

The base sentence may be "a full body-shot of a person standing," while $< 0.1$1, $< 0.1$2, and $< 0.1$3 contain verbs such as "hold," "carry," and "push," objects such as "umbrella," "box," and "board," and poses such as "on the head," "in front of the body," and "on the back" (Li et al., 8 Oct 2025).

The end-to-end optimization problem is formulated as

$< 0.1$4

where $< 0.1$5 is the text-to-3D generator, $< 0.1$6 inserts the generated object into environment $< 0.1$7 and outputs the point cloud $< 0.1$8, $< 0.1$9 is the LiDAR 3D detector, and $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$0 is the ground-truth box set. The adversarial loss is

$$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$1

with $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$2 in the description and fallback $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$3. The loss suppresses the confidence of any predicted box that sufficiently overlaps the target pedestrian.

Because prompts are discrete, the implemented optimization occurs in the latent embedding space of the frozen text encoder. If $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$4 is the number of prompt triplets and $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$5 is the embedding of the $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$6-th prompt, the combined embedding is approximated as

$$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$7

The paper further decomposes the weights into verb, object, and pose components using Gumbel-Softmax, so the formal mechanism is a differentiable latent optimization over VOP logits rather than a hand-coded discrete search algorithm.

The practical loop is fixed and fully differentiable: build the VOP prompt pool, encode all prompts using the frozen text encoder, initialize verb/object/pose logits, form the combined latent embedding with Gumbel-Softmax weights, generate a 3D object through a Gaussian Splatting-based text-to-3D model such as LGM, insert the object into a LiDAR scene, run the frozen detector, compute $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$8, and update only the prompt logits. The stopping condition is a fixed budget of 300 steps with Adam at learning rate $$\gamma = \mathcal{D}\left(\mathrm{Obj}\big(\tilde{\mathrm{Ped}} \cup \mathrm{AddObj}\big),\; \mathrm{Env}(\mathcal{A}, \mathcal{D}, \mathcal{V})\right),$$9; the generator, detector, and text encoder remain frozen. The reported implementation uses 4 NVIDIA L40 GPUs.

Conceptually, the paper describes the attack as iteratively refining verbs, objects, and poses. Operationally, however, the refinement is continuous in latent space. This distinction matters because it places Phy3DAdvGen closer to differentiable prompt optimization than to explicit combinatorial search.

4. Physical Realizability and Object-Pool Constraints

Phy3DAdvGen’s physical realizability claim rests on constraining generation to combinations of real objects rather than arbitrary free-form geometry. The method constructs an object pool containing 13 3D models of real objects and constrains optimization to compositions based on that set. The excerpt names backpack and umbrella as examples, but does not provide the full list (Li et al., 8 Oct 2025).

The approximation of a generated adversarial embedding by real deployable objects is formulated as

$\gamma$0

where $\gamma$1 is the real object pool and $\gamma$2 are object embeddings. Operationally, these weights determine how real props are combined through positions, rotations, and scales to mimic the generated human-object composition.

This construction is presented as a practical alternative to full human-sized 3D printing. The attack no longer depends on fabricating an arbitrary optimized mesh; instead it requires assembling existing objects in specific configurations around a human subject. That constraint is restrictive, but it is also the source of physical credibility: material and reflectivity are inherited from real props, and the attack becomes deployable with everyday objects rather than specialized fabrication.

The realism guarantee remains partial. Geometry must be approximable by combinations of the 13 real objects, object pose must be replicable using multi-view guidance, and the person must be able to carry or wear the assembly. The appendix also notes spatial grounding errors and stochasticity in text-to-3D models, so a given prompt may not always yield a perfectly aligned or reproducible 3D composition. In that sense, Phy3DAdvGen is physically informed by object availability and semantic deployability, not by a full mechanics-based simulator.

5. Evaluation Protocol and Main Results

The evaluation spans both simulation and physical deployment. Six LiDAR 3D detectors from OpenPCDet are tested: the point-based PointRCNN and IA-SSD, the voxel-based SAFDNet and VoxelNeXt, and the point-voxel-based HEDNet and PDV. Digital experiments use CARLA with a KITTI-style sensor configuration over 3 CARLA maps/scenes. Physical experiments use an autonomous-driving test field with a stationary ego vehicle equipped with a Robosense RS-Ruby 128-beam LiDAR (Li et al., 8 Oct 2025).

The digital protocol generates 100 adversarial 3D objects, with 50 for single-pedestrian attribute changes and 50 with object combinations. The physical protocol collects about 5,000 real-world point cloud frames, fine-tunes detectors using SUSTech POINTS, involves five participants in outdoor scenes, assigns one participant to carry the adversarial object under Phy3DAdvGen, and evaluates over ten adversarial poses while the subject walks toward the vehicle. Metrics are Detection Success Rate (DSR) and Attack Success Rate (ASR), defined exactly by the IoU thresholds stated earlier.

In CARLA, the headline comparison is on PointRCNN. Reported ASR values are 64.2% for AdvPC, 62.9% for NI-FGSM, 79.3% for the Tu et al. 2020 reimplementation, 68.4% for the Zheng et al. 2025 reimplementation, 47.5% for ScAR, and 94.6% for Phy3DAdvGen. The result is significant because the stronger baseline classes include both point perturbation and geometry-aware or physically realizable attacks.

Ablation on prompt strategy, optimized on PointRCNN and averaged across the six detectors, shows 71.0 $\gamma$3 6.6 for Random Prompt, 78.0 $\gamma$4 11.2 for LatentPerturb, 62.8 $\gamma$5 4.7 for Phy3DAdvGen using verb-only optimization, 77.3 $\gamma$6 8.8 for verb + object, and 87.7 $\gamma$7 7.9 for full verb + object + pose optimization. The pattern is structurally important: verbs alone are weak, object semantics materially improve attack strength, and pose semantics produce the final large increase. This indicates that adversarial efficacy depends not only on what object is added, but also on where and how it is positioned relative to the body.

Physical-world results remain high under the baseline 128-beam, longitudinal, sunny, slow condition: PointRCNN reaches 84.6, IA-SSD 90.3, SAFDNet 86.7, the table row labeled VoxelNet / VoxelNeXt reports 90.6, HEDNet 85.3, and PDV 81.6. Beam count has a clear effect: lower beam counts make attacks easier, as illustrated by PDV dropping from 93.0% ASR at 32-beam to 81.6% at 128-beam. Longitudinal motion generally gives higher ASR than lateral motion, while weather and speed have relatively minor effect; for example, PointRCNN is 85.3% in cloudy conditions versus 84.6% in sunny conditions, and VoxelNet is 90.3% at fast speed versus 90.6% at slow speed.

The paper also reports a real-world example in which a pedestrian on a zebra crossing is not detected by SAFDNet. More broadly, the physical ASR range of roughly 81.6% to 90.6% across six detectors under the baseline setting indicates that the digital-to-physical transfer is not merely nominal. The main degradation factors in practice are distance and beam density rather than weather or moderate speed variation.

6. Position Within Text-to-3D, Adversarial Generation, and Physics-Informed 3D Research

Phy3DAdvGen occupies a distinct position at the intersection of adversarial robustness, text-to-3D generation, and physical deployment. Earlier text-conditioned 3D GAN work such as "Generation High resolution 3D model from natural language by Generative Adversarial Network" uses a two-stage conditional WGAN pipeline for colored voxel generation, with a low-resolution semantic stage and a high-resolution refinement stage, but includes no explicit physical knowledge, no simulation-based supervision, and no adversarial attack objective against downstream perception systems (Fukamizu et al., 2019).

"IT3D: Improved Text-to-3D Generation with Explicit View Synthesis" introduces a different adversarial element: explicit multi-view image synthesis with Stable Diffusion and ControlNet, followed by GAN-style refinement in which synthesized views are treated as real and current renderings as fake. IT3D is geometry-aware and uses an adversarial distribution-matching stage, but it does not model calibrated lighting, BRDFs, collision, stability, or rigorous multi-view geometric consistency, and it is not designed as a physically deployable attack framework (Chen et al., 2023).

Physics-aware text-to-3D work takes a different route. "Precise-Physics Driven Text-to-3D Generation" introduces Phy3DGen, which refines text-conditioned 3D priors using a differentiable solid-mechanics surrogate supervised by FEM and optimized for reduced stress concentration under linear elasticity, but it provides no discriminator, no min-max training, and no attack objective (Xu et al., 2024). "Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication" adds differentiable rigid-body simulation to SDS-based refinement and optimizes standability and stable equilibrium of generated rigid objects, including real-world 3D-printed validation, yet it remains a physical-validity module rather than an adversarial generator (Chen et al., 2024). "PAT3D: Physics-Augmented Text-to-3D Scene Generation" uses a VLM-guided object-centric pipeline, a hierarchical scene tree, and differentiable rigid-body simulation to produce physically plausible, simulation-ready, and intersection-free scenes, but it explicitly includes no GAN-style training, discriminators, or adversarial losses (Lin et al., 26 Nov 2025).

Against this backdrop, Phy3DAdvGen is distinguished by the object of optimization and the deployment setting. It does not adversarially alter detector parameters, does not attack raw LiDAR returns directly, and does not merely regularize a generator for physical plausibility. Instead, it adversarially optimizes semantically structured text prompts so that a text-to-3D model yields human-object assemblies that are hard for LiDAR detectors to classify as pedestrians while remaining deployable with real props. This suggests that Phy3DAdvGen is less a general-purpose physics-based text-to-3D framework than a task-specific adversarial generation system whose "physical" content is grounded in object availability, pose semantics, and demonstrable real-world realization.

7. Limitations, Safety Implications, and Defensive Directions

The paper presents the method as dual-use research intended for safety auditing. Its central implication is that a pedestrian carrying or arranged with everyday objects can become effectively invisible to LiDAR-based 3D detectors, exposing a realistic failure mode for autonomous driving systems (Li et al., 8 Oct 2025).

Several limitations are explicit. The attack is sensitive to distance, and distance is identified as the largest challenge for robust deployment. Higher-beam LiDARs reduce ASR. Text-to-3D backbones can exhibit spatial grounding errors and stochasticity, so the same prompt may not reliably produce the same geometry or object placement. Physical realizability is constrained by the semantic and geometric coverage of the 13-object pool. The work focuses on LiDAR-only detection rather than camera-LiDAR fusion, and some detector families remain relatively insensitive to single-attribute manipulations, especially intensity-based changes.

The paper does not provide a full defense method, but several directions are implied. These include training detectors on richer human-object compositions, incorporating intensity-aware or material-aware cues, strengthening robustness to atypical pedestrian geometries, exploiting multi-sensor fusion with RGB cameras, and using adversarially generated compositions during training. A plausible implication is that future autonomous-driving evaluation will need to test not only sensor-level spoofing and point-cloud perturbation, but also semantically plausible, physically deployable compositions generated through text-conditioned 3D models.

A common misconception is that such attacks require exotic fabrication or unrealistic geometry. Phy3DAdvGen is notable precisely because it rejects that assumption: its attack model is based on semantically interpretable human-object compositions, constrained by a real object pool and demonstrated outdoors. The corresponding controversy is therefore not whether the examples are physically possible, but how widely such detector-invisibility modes generalize once multi-sensor perception, denser LiDAR, and richer training distributions are introduced.