NatGenAI: Natural Generative AI Systems

• NatGenAI is defined as a family of generative AI approaches that leverage natural modalities—such as language, speech, diagrams, and gestures—to steer complex system interactions.
• Its architecture typically composes data encoders, generative models, and retrieval/storage modules, enabling iterative processing from natural input to refined outputs.
• Applications span design ideation, relational database and hardware synthesis, and embodied fabrication, highlighting practical metrics, controllability challenges, and multimodal integration.

Natural Generative AI (NatGenAI) denotes a family of generative-AI approaches in which natural modalities and human-facing abstractions are treated as the primary medium for generation, control, or interaction. In the literature, this includes keyword-driven design ideation over patent corpora (Zhu et al., 2022), natural-language interfaces to relational and multimodal databases (Fotso, 2024, Erfanian et al., 2024), natural-language and diagram-driven hardware synthesis (Yang et al., 28 Mar 2025, Chang et al., 2024), speech-to-object fabrication (Kyaw et al., 2024), and XR systems driven by speech, gesture, gaze, and scene understanding (Zhu et al., 13 Jan 2026). A systems formulation describes such systems as compositions of Data Encoders, a GenAI Model, and a Retrieval/Storage module linked by natural language and modality-specific I/O interfaces (Tomczak, 2024). At the same time, the label is also used in broader theoretical ways: “NatGen” for code defines naturalness as idiomatic developer-written code and trains models to rewrite semantically equivalent “un-natural” programs into more natural originals (Chakraborty et al., 2022), while a separate line reframes evolutionary computation itself as NatGenAI governed by exploratory search under natural selection (Shi et al., 4 Oct 2025).

1. Conceptual scope and meanings of “natural”

The literature uses NatGenAI in several distinct but connected senses. In interface-centric systems, “natural” typically refers to everyday modalities such as free-form language, speech, diagrams, gaze, or gesture, and to low-friction interaction with complex backends. In output-centric work on source code, “natural” refers instead to the regular, predictable, idiomatic forms that human developers actually write. In the evolutionary-computation reformulation, “natural” refers to natural selection as the driver of generation rather than to natural language or human-facing interfaces. This suggests that NatGenAI is better understood as a family resemblance concept than as a single fixed architecture.

Sense of “natural”	Primary emphasis	Representative papers
Natural modalities and interfaces	Keywords, speech, diagrams, gesture, gaze	(Zhu et al., 2022, Kyaw et al., 2024, Zhu et al., 13 Jan 2026, Chang et al., 2024)
Natural system composition	Data Encoders + GenAI Model + Retrieval/Storage	(Tomczak, 2024, Fotso, 2024, Lange et al., 23 Sep 2025)
Natural outputs	Rewrite “un-natural” but equivalent code into idiomatic code	(Chakraborty et al., 2022)
Natural selection	Exploratory search under natural selection	(Shi et al., 4 Oct 2025)

One systems-based articulation is especially influential for the interface-centric strand. It defines a Generative AI System as comprising Data Encoders, a GenAI Model, and a Retrieval/Storage module, with natural language functioning as the communication means and modality encoders serving as I/O interfaces for text, images, audio, and other sources (Tomczak, 2024). Within this view, NatGenAI is not a single model but a composite system whose generative behavior depends on how encoders, retrieval, tools, memory, and output channels are composed.

2. Architectural patterns and control loops

A recurring architectural pattern is the conversion of natural input into a structured intermediate representation, followed by retrieval, generation, execution, validation, and user-visible refinement. In the relational-database query engine, natural-language questions are embedded, matched against schema metadata and business rules stored in Milvus, translated into SQL by LLama 3, executed on IBM watsonx.data, and then iteratively repaired through syntactic and semantic validation before a natural-language answer is generated by Mixtral (Fotso, 2024). In YAC, a multi-agent pipeline uses a GPT-4.1 orchestrator, a GPT-4.1 filter agent, and a fine-tuned Llama-3.1-8B-Instruct visualization agent to emit structured declarative output that the front end interprets as filters, linked visualizations, and widgets (Lange et al., 23 Sep 2025).

NatGenAI systems also use generation inside the query engine itself rather than only at the user interface. Needle answers complex natural-language image queries by generating synthetic “guide tuples” from a foundation model, embedding those guides with multiple embedders, and averaging distances to database items. Its central estimator is

$$\bar{\delta}_{Q,i} = \frac{1}{m\,l}\sum_{j=1}^m \sum_{\ell=1}^l \delta\big( E^\ell(\bar{g}_j), \vec{v}^\ell_i\big),$$

where the query is represented not by a single text embedding but by a Monte Carlo average over generated interpretations and multiple embedders (Erfanian et al., 2024). This is a strong NatGenAI pattern: generation is used to construct the internal semantics of retrieval, not merely to polish outputs.

The systems perspective makes these examples legible as instances of the same design logic. Encoders translate natural modalities into internal representations; the central generative model plans, explains, or generates; retrieval and storage provide long-term memory and external knowledge; and validation or visualization layers constrain outputs before they reach the user or the environment (Tomczak, 2024).

3. Modalities, interaction, and the claim that natural language is not enough

Several works treat natural language as the main design medium. In knowledge-based design ideation, a 355M-parameter GPT-2 is fine-tuned on USPTO patent titles organized by InnoGPS, and users provide a target keyword such as “rolling toy” to receive concise, patent-title-like idea descriptions. The system treats natural language as the design medium and positions itself as a design assistant rather than an autonomous designer (Zhu et al., 2022). In NLS, natural language specifications are wrapped into a System Prompt, Updated Prompt, and Input Question, and general-purpose LLMs generate Verilog or SystemVerilog source files in a “natural-language-in, HDL-out” workflow (Yang et al., 28 Mar 2025).

At the same time, an important correction appears in the hardware literature: natural language alone is often insufficient for spatially complex artifacts. The multimodal Verilog benchmark argues that block diagrams, state diagrams, and brief textual specifications jointly encode design intent more effectively than text alone, because language is sequential and comparatively weak at specifying exact topologies, dense interconnections, and multi-level spatial structure (Chang et al., 2024). The proposed VLMQL framework therefore couples visual hardware graphs with text descriptions of module functions and port semantics.

Other NatGenAI systems further expand the notion of “natural” beyond text. “Speech to Reality” maps speech to text, extracts an object phrase with GPT-4 Turbo, generates a low-poly mesh through Meshy.ai, voxelizes and filters the geometry under fabrication constraints, and executes the resulting plan with a UR10 robotic arm (Kyaw et al., 2024). In XR, “natural interaction” is characterized by everyday modalities—free-form speech, simple hand gestures, and gaze—combined with scene awareness, personalization, and dynamic 3D content generation (Zhu et al., 13 Jan 2026). Taken together, these works make a general point: NatGenAI is often multimodal by construction, and the adjective “natural” usually refers to the fit between human intent expression and system control rather than to text alone.

4. Representative application domains

In early-stage design, NatGenAI appears as controllable ideation over structured knowledge bases. The design-ideation system fine-tunes one GPT-2 model per source domain and uses InnoGPS knowledge-distance rankings to create “virtual experts” at different distances from a target domain. For rolling-toy design, near-field domains such as Weapons, Agriculture, and Lighting yield more structurally familiar ideas, while far-field domains such as Drilling & Mining, Grinding & Polishing, and Fuels & Lubricants yield more novel recombinations (Zhu et al., 2022).

Over structured data, NatGenAI functions as a mediator between human language and formal backends. The relational database engine accepts free-form questions, retrieves schema fragments and business rules, generates SQL, validates it against actual execution errors and semantic criteria, and returns natural-language summaries rather than raw rows (Fotso, 2024). Needle handles multimodal databases by generating synthetic images that capture complex textual descriptions and then retrieving real images through approximate nearest-neighbor search over guide-tuple embeddings (Erfanian et al., 2024). YAC extends this pattern to biomedical discovery by turning chat requests into declarative visualization specifications, linked filters, and editable widgets over C2M2-based metadata packages (Lange et al., 23 Sep 2025).

In hardware design, NatGenAI ranges from code generation to end-to-end synthesis workflows. NLS is implemented as a Visual Studio Code extension that lets engineers specify algorithms, architectures, and constraints in English, generate HDL, package .v files, and refine prompts after synthesis or simulation feedback (Yang et al., 28 Mar 2025). The multimodal Verilog benchmark shows the same general problem from a different angle: diagrams plus text improve synthesis accuracy for FSMs, pipelines, systolic arrays, and other multi-module designs that are hard to express linearly in natural language alone (Chang et al., 2024).

In fabrication and embodied generation, NatGenAI closes the loop from description to physical artifact. “Speech to Reality” uses Google Speech-to-Text, GPT-4 Turbo, Meshy.ai, Rhino 3D scripts, feasibility filtering, and Python-URX control of a UR10 arm to assemble objects such as stools, shelves, and letters from a reusable voxel inventory (Kyaw et al., 2024). In XR, LLMs, VLMs, diffusion-based text-to-3D systems, and text-to-speech support VR education, AR assistance, and MR training by generating assets, overlays, behaviors, and feedback from natural user instructions and scene context (Zhu et al., 13 Jan 2026).

5. Evaluation, metrics, and controllability

NatGenAI systems are evaluated with a mixture of generative, task-level, and human-centered metrics. In design ideation, each domain-specific GPT-2 generated 500 rolling-toy ideas; uniqueness ranged from 35.8% for Weapons to 76.4% for Grinding & Polishing, and novelty was proxied by the minimum TechNet term-term relevancy score, with far-field models tending toward lower minima than near-field models (Zhu et al., 2022). These results operationalize a controllability claim central to that line of work: knowledge distance can steer the novelty–feasibility trade-off.

In multimodal retrieval, Needle reports strong gains over CLIP on complex queries. On targeted caption-style retrieval, average mAP rose from 0.18 to 0.36 and hit rate from 0.63 to 0.75 across nocaps and Senticap; on object-detection-style retrieval, average mAP rose from 0.62 to 0.70 overall and from 0.12 to 0.35 on hard subsets (Erfanian et al., 2024). In database querying, the IBM watsonx-based engine reports that over 50% of the queries were correctly translated into SQL on the first attempt, more than 58% for simple queries, and that a dozen non-technical users rated responses as “excellent” in 90% of test cases (Fotso, 2024).

Hardware-oriented NatGenAI work introduces domain-specific evaluation axes. NLS defines Quality of Generated Hardware through Performance, Power, and Area metrics and Required Design Efforts through prompt length, number of adjustments, and related measures; in the AGRNN-based dynamic QP solver case, the number of prompt adjustments reached 93, indicating that natural-language steering is itself a measurable design effort (Yang et al., 28 Mar 2025). The multimodal Verilog benchmark reports that GPT-4V with vision plus text achieved 84.38% syntax success and 71.81% functional success, compared with 68.75% and 46.88% for GPT-4 text-only, supporting the claim that multimodal natural specifications improve correctness for complex hardware tasks (Chang et al., 2024).

Embodied systems add timing and fabrication constraints to the evaluation picture. In “Speech to Reality,” Meshy.ai averaged 1 min 16 s across ten runs, while discrete robotic assembly averaged 3m 10s for the benchmarked objects, compared with 2d 12h 37m for large-format FFF and 4h 24m for parallel FFF (Kyaw et al., 2024). XR systems emphasize different constraints—latency, scene grounding, and trustworthiness—and explicitly note that some current speech-to-action or text-to-3D loops are still too slow for fully natural real-time interaction (Zhu et al., 13 Jan 2026).

6. Alternative theoretical lineages: naturalization and natural selection

One distinct lineage grounds NatGenAI in output naturalness rather than in interface design. “NatGen: Generative pre-training by ‘Naturalizing’ source code” exploits code’s bimodal, dual-channel nature by applying six classes of semantics-preserving transformations—Loop Transformation, Dead Code Injection, Block Swap, Operand Swap, Confusing Code Insertion, and Variable Renaming—and training a seq2seq model to reconstruct the original human-written program from its “un-natural” variant (Chakraborty et al., 2022). On the naturalization task, the model reports 70.39% Exact Match, 98.78% Syntax Match, 97.69% Dataflow Match, and 97.31% CodeBLEU on the full validation set, and it transfers competitively to code generation, code translation, and code refinement (Chakraborty et al., 2022). Here, “natural” refers to idiomatic, predictable, developer-like code rather than to natural-language interaction.

A second theoretical lineage reframes evolutionary computation itself as NatGenAI. In this view, NatGenAI is “a generative paradigm governed by exploratory search under natural selection,” with a dynamic population distribution $p_e(x,t)$ shaped by variation and selection rather than by maximum-likelihood fitting to a fixed dataset (Shi et al., 4 Oct 2025). The paper distinguishes parent-centric operators, which preserve the structure of parent distributions and mirror conventional within-distribution generation, from disruptive operators such as Occurrence-Based Scanning, which enable structured evolutionary leaps and out-of-distribution artifacts. Evolutionary multitasking and moderated selection are presented as mechanisms that allow novel cross-domain combinations to survive long enough to become useful. The car–airplane aerodynamic case study exemplifies this interpretation by producing hybrid designs that combine traits from both domains while improving aerodynamic performance (Shi et al., 4 Oct 2025).

These two lineages materially broaden the concept. They imply that NatGenAI need not be limited to LLM-mediated human interfaces. It can also denote generative systems that naturalize artifacts toward human conventions or that generate novelty through population dynamics and natural selection.

7. Limitations, misconceptions, and future directions

A common misconception is that NatGenAI implies autonomous replacement of expert workflows. Several papers explicitly reject that framing. The design-ideation system is presented as a design assistant that generates stimuli for humans to interpret, select, and elaborate (Zhu et al., 2022). NLS likewise operates as an AI-in-the-loop workflow in which engineers run simulation and synthesis externally, inspect errors, update prompts, and regenerate HDL (Yang et al., 28 Mar 2025). Even in XR, explainability, user verification, and override are treated as necessary conditions for trust rather than optional embellishments (Zhu et al., 13 Jan 2026).

Another misconception is that NatGenAI is equivalent to text-only prompting. The hardware benchmark states the opposite in its title and argument: natural language is not enough for many spatially structured tasks, and diagrams materially improve Verilog generation accuracy (Chang et al., 2024). XR work makes the same point in a different domain by defining natural interaction through speech, simple hand gestures, gaze, and scene awareness rather than through text alone (Zhu et al., 13 Jan 2026). YAC, similarly, assigns complementary roles to chat and direct manipulation: natural language creates views and filters, while widgets and brushing provide fast, precise adjustment (Lange et al., 23 Sep 2025).

The practical limitations are substantial. Database-oriented NatGenAI still struggles with ambiguous language and domain knowledge absent from the vector store (Fotso, 2024). YAC currently supports only direct inter-entity relationships and combines filters only by intersection, not union (Lange et al., 23 Sep 2025). “Speech to Reality” is limited by a 40-voxel inventory, 10 cm resolution, coarse cantilever heuristics, and vibration-induced failures at higher robot speeds (Kyaw et al., 2024). XR systems face hallucination, privacy exposure from continuous multimodal capture, high latency, and resource contention on mobile devices (Zhu et al., 13 Jan 2026). Hardware generation remains prompt-sensitive and verification-heavy, with no integrated formal verification or automatic testbench generation in NLS (Yang et al., 28 Mar 2025).

The future directions are correspondingly diverse. Design ideation proposes BERT-based sentence embeddings for better evaluation and investigation of GPT-3 few-shot learning (Zhu et al., 2022). The database query engine points to domain-specific model adaptation, dynamic business-rule generation, and extension beyond relational databases (Fotso, 2024). NLS proposes dedicated system-level HDL datasets, domain-specific model training, AI-driven system partitioning, support for locally trained models, and broader HDL refactoring support (Yang et al., 28 Mar 2025). “Speech to Reality” suggests multiple component types, multi-robot assembly, stronger language handling, and tighter fabrication-aware generation (Kyaw et al., 2024). Needle is explicitly designed to absorb advances in foundation models and embedders and to extend from images to audio and video (Erfanian et al., 2024). Taken together, these trajectories indicate that NatGenAI is evolving toward more strongly grounded, multimodal, tool-integrated, and verifiable systems, while continuing to debate whether its defining property is natural interaction, naturalized outputs, or generation under natural selection.