GenAI-Powered Inference: Methods & Applications

• GenAI-Powered Inference (GPI) is an umbrella term for systems that embed generative models within inference pipelines to interpret, plan, and control application behavior.
• GPI approaches span inference-time control, AI-driven Bayesian methods, and representation-based causal inference to enhance decision-making and predictive performance.
• GPI architectures are designed for distributed, edge, and orchestrated environments, optimizing latency, resource use, and security in human-AI interactions.

Searching arXiv for papers on "GenAI-Powered Inference" and closely related formulations. arXiv search query: "GenAI-Powered Inference" GenAI-Powered Inference (GPI) denotes a set of research programs in which generative models participate directly in inference-time computation rather than serving only as offline pretraining artifacts. In one formulation, a GenAI engine is called at run time to interpret user inputs, plan sequences of actions, select and parameterize tools, and emit outputs that an application treats as authoritative instructions (Cohen et al., 2024). In others, GPI refers to nonparametric Bayesian inference with an AI-driven prior (O'Hagan et al., 26 Feb 2025), causal and predictive inference over unstructured data using pretrained generative models as representation extractors (Imai et al., 5 Jul 2025), and constrained analytical systems in which GenAI translates natural-language intent into formal statistical specifications executed by a deterministic backend (Koonchanok et al., 2 Sep 2025). Taken together, these lines of work suggest that GPI is best understood as an umbrella term for inference pipelines in which generative models act as inferential components, controllers, priors, interfaces, or schedulers, rather than as a single standardized architecture.

1. Scope and principal formulations

Recent papers use GPI in several technically distinct but structurally related senses. What unifies them is the placement of a generative model inside the inferential path of a system, so that the model affects either the representation of inputs, the selection of actions, the formation of posterior beliefs, or the orchestration of computation itself.

Formulation	Core role of GenAI	Representative paper
Inference-time application control	Planner, tool selector, control oracle	(Cohen et al., 2024)
Distributed and edge execution	Runtime model tiering, offloading, pruning, routing	(Zhang et al., 2024)
Bayesian statistical inference	Prior over the data-generating distribution	(O'Hagan et al., 26 Feb 2025)
Causal and predictive inference from unstructured data	Fixed representation extractor plus learned deconfounder	(Imai et al., 5 Jul 2025)
Statistical interface design	Natural-language-to-formal-specification compiler	(Koonchanok et al., 2 Sep 2025)
Reliable evaluation of GenAI systems	Noisy surrogate integrated with valid uncertainty	(Martinon et al., 29 May 2026)

This diversity is substantive rather than terminological. The distributed-systems literature emphasizes latency, communication cost, and model placement; the statistical literature emphasizes posterior validity, uncertainty quantification, and identification; the application-security literature emphasizes the fact that inference-time model outputs can directly steer control flow; and the HCI-oriented literature emphasizes constrained interaction, interpretability, and user agency (Zhang et al., 2024).

2. Inference-time control, tool use, and the security model

A central systems interpretation of GPI treats a “GenAI-powered application” as any software system whose behavior at run time depends on a GenAI engine: it receives user inputs, incorporates those inputs into prompts, and determines execution flow based on the GenAI output (Cohen et al., 2024). The paper’s canonical examples are an email assistant that plans how to respond, drafts a reply, checks whether the reply is safe, and rephrases it if not, and an e-commerce chatbot that translates user requests into SQL queries via an LLM and executes them against a product database. In this setting, the model is not merely generating text; it is participating in a finite-state control process.

For Plan & Execute, function-calling, and ReAct-like architectures, the control logic can be written conceptually as an FSM whose transition function is induced by model outputs: $s_{t+1} = f_{\text{app}(s_t, G(\text{prompt}(s_t, \text{history}))).$ The security consequence is that a party controlling part of the prompt indirectly influences $G$, and hence the transition $s_t \to s_{t+1}$. This is the basis of PromptWare, described as a “0-click polymorphic malware” whose “binary” is a user input, and of the more sophisticated Advanced PromptWare Threat (APwT), which in black-box settings instructs the model to execute a six-step kill chain at inference time: privilege escalation, context analysis, asset identification, reasoning about malicious activities, decision, and execution (Cohen et al., 2024).

The attack literature makes explicit that GPI systems inherit a distinctive code/data confusion problem. Morris-II shows that when communication among GenAI-powered applications relies on RAG-based inference, an attacker can construct an adversarial self-replicating prompt satisfying

$G(x)\rightarrow x$

or

$$G(w \mathbin\Vert x \mathbin\Vert y)\rightarrow \text{payload} \mathbin\Vert x,$$

thereby turning retrieved context into executable prompt logic for subsequent inferences (Cohen et al., 2024). In the RAG case, ingestion and later retrieval create persistence; in flow-steering cases, model outputs such as "classification": "forward" immediately trigger application actions. The resulting harms are operational rather than purely textual: the paper demonstrates a DoS loop in a ReWOO-style assistant and SQL-table modification in an e-commerce chatbot, while Morris-II demonstrates multi-hop confidential data extraction in an email-assistant ecosystem (Cohen et al., 2024).

A recurring misconception is that prompt injection and jailbreaks only concern unsafe text generation. These papers indicate a broader risk: GPI can flip the model’s role from serving the application to attacking the host application, precisely because the surrounding system treats model outputs as actionable control signals (Cohen et al., 2024).

3. Bayesian and nonparametric statistical inference with AI-driven priors

A second major formulation treats GPI as a formal statistical pipeline in which a generative model supplies prior information. “AI-Powered Bayesian Inference” makes this explicit by interpreting the variability of a generative model’s answers to repeated prompts as a distribution over plausible outputs and using that distribution as the base measure of a Dirichlet process prior (O'Hagan et al., 26 Feb 2025). With observed labeled data

$\mathcal D_n = \{(Y_i, \mathbf X_i)\}_{i=1}^n$

from an unknown joint distribution $F_0$, and an AI system $\hat\mu_{\text{AI}}(x)$ that stochastically generates labels, the method constructs an AI baseline $F_{\text{AI}}$ and places

$F \sim \text{DP}(\alpha, F_{\text{AI}}).$

The posterior remains conjugate: $G$0 with

$G$1

where $G$2. Here $G$3 functions as an effective prior sample size, and the predictive mixture combines AI-induced pseudo-data with observed human-labeled data. Posterior simulation proceeds via a posterior bootstrap on an augmented dataset of real and AI-imputed pseudo-observations: $G$4 The method is parallelizable and produces i.i.d. posterior draws by optimization rather than MCMC (O'Hagan et al., 26 Feb 2025).

The empirical studies situate this statistical version of GPI in concrete data-analysis tasks. In skin disease diagnosis, the paper uses the UCI dermatology dataset with 366 cases, 6 classes, 12 clinical features, 20% labeled training data, 20% labeled test data, and 60% extra unlabeled data prompted through GPT-4o. With a small AI prior weight, posterior predictive accuracy improves, with best performance around $G$5 and about 5 percentage points gain over using only human-labeled data; as $G$6 grows large, performance degrades toward pure ChatGPT predictions at about 60% (O'Hagan et al., 26 Feb 2025). In the galaxy example, the task is to estimate the proportion of spiral galaxies using $G$7 human labels and $G$8 AI-imputed labels from a computer vision model; a moderate $G$9 around 500 yields 90% credible intervals similar in width to PPI intervals while maintaining near-nominal coverage (O'Hagan et al., 26 Feb 2025). This line of work defines GPI as probabilistically coherent uncertainty quantification that uses GenAI predictions as noisy prior information rather than as oracle labels.

4. Representation-based causal and predictive inference from unstructured data

Another formalization of GPI uses pretrained generative models as fixed encoders for unstructured covariates, treatments, or confounders. In this framework, text or images $s_t \to s_{t+1}$0 are mapped into internal representations

$s_t \to s_{t+1}$1

where $s_t \to s_{t+1}$2 is a pretrained open-source LLM or diffusion model, and then transformed into a low-dimensional deconfounder

$s_t \to s_{t+1}$3

The objective is not to fine-tune the foundation model, but to learn a task-specific low-dimensional representation sufficient for downstream causal or predictive analysis (Imai et al., 5 Jul 2025).

For causal inference, the paper emphasizes the ATT,

$s_t \to s_{t+1}$4

under an unconfoundedness condition with respect to $s_t \to s_{t+1}$5. Estimation uses double machine learning with two-fold cross-fitting, separate outcome and propensity models, influence-function-based standard errors, and propensity truncation. For prediction tasks, the framework estimates flexible functions such as

$s_t \to s_{t+1}$6

The paper stresses that the generative model is deterministic and fixed from the standpoint of statistical inference; inferential uncertainty is attached to the learned $s_t \to s_{t+1}$7 and downstream nuisance models, not to the pretrained encoder itself (Imai et al., 5 Jul 2025).

Three applications illustrate the framework. In the Chinese social-media censorship study, the data comprise 75,324 Weibo posts from 4,155 users; LLaMA3-8B and Gemma3-1B embeddings are used to learn deconfounders, and GPI finds that prior censorship significantly reduces subsequent posting volume and strongly increases the probability of future posts being censored. In the electoral image study, the data include more than 7,000 candidate photographs from Danish elections, with Stable Diffusion v1.5 and v2.1 encoders used to estimate predictive effects of facial traits on standardized vote counts; GPI finds no significant predictive effect of facial dominance and produces estimates more robust to covariate inclusion than OLS. In the rhetoric application, the data include 336 political arguments, 14 rhetorical elements, and 3,317 respondents; a semiparametric ordered-logit-style structural model using LLaMA3-8B, LLaMA3.3-70B, and Gemma3-1B representations finds appeals to authority most persuasive and ad hominem attacks significantly reducing persuasiveness (Imai et al., 5 Jul 2025). This version of GPI is therefore a representation-learning-and-inference pipeline for settings where the substantively relevant information lives in text or images.

5. Distributed, edge, and orchestrated inference systems

The infrastructure literature uses GPI to denote execution architectures for running generative models under latency, communication, and resource constraints. A representative formulation is the three-tier “cloud–edge–end” collaborative inference system, in which end devices run small models, edge servers run medium-sized models, and the cloud hosts large models. The core latency decomposition is

$s_t \to s_{t+1}$8

with total latency

$s_t \to s_{t+1}$9

and the system objective is to minimize latency subject to a correctness requirement $G(x)\rightarrow x$0 (Zhang et al., 2024). The architecture combines calibrated confidence-based probabilistic offloading, attention-based token pruning,

$G(x)\rightarrow x$1

stability-based early exit within a single model, and weighted ensembling across participating tiers. On IMDB sentiment analysis with bert-base-uncased, bert-large-uncased, and bertweet, the authors report up to 17% reduction in inference time without sacrificing inference accuracy relative to existing work (Zhang et al., 2024).

Edge execution studies sharpen the systems picture. On a Raspberry Pi 5 K3s cluster, CPU-only deployment of lightweight models such as Yi, Phi, and Llama3 achieves a generation throughput of 5 to 12 tokens per second with less than 50% CPU and RAM usage, supporting localized inference in remote or bandwidth-constrained environments (Nezami et al., 2024). On-device GPU work extends this agenda to mobile and laptop platforms: ML Drift is reported to enable on-device execution of generative workloads containing 10 to 100x more parameters than existing on-device generative AI models and to achieve an order-of-magnitude performance improvement relative to existing open-source GPU inference engines (Tang et al., 1 May 2025). At the orchestration level, Kubernetes-native components such as Kueue, DAS, and GAIE form a unified platform for mixed batch and online GenAI workloads, with reported gains of up to 15% lower makespan for batch ASR, 36% shorter mean job completion time with GPU slicing, and 82% improvement in Time to First Token for online summarization (Malleni et al., 3 Feb 2026). A further extension places LLMs in the control plane itself: a multi-agentic framework with a long-term planning agent, a short-term prompt scheduling agent, and per-node deployment agents reduces average latency by over 80% and improves Normalized Jain index to 0.90 in a real-world mobile edge network (Li et al., 6 Feb 2026). These works indicate that, in the systems literature, GPI encompasses not only model execution but also runtime routing, deployment, and scheduling policy.

6. Interfaces, evaluation, and human-guided inferential workflows

A distinct line of work constrains GenAI to a high-level inferential role while delegating formal computation to verifiable backends. In a lightweight visual data-analysis system, GPT-4 via API receives a natural-language query, maps it to a predefined task description, and generates formal statistical specifications such as an R formula or hypothesis structure, while all statistical computation is executed by an R backend using packages such as gamlss and emmeans (Koonchanok et al., 2 Sep 2025). The architecture is explicitly tripartite: GenAI layer for intent classification and translation, R-based backend for deterministic inference, and visualization layer for model criticism. The formal intermediate representation is the key safeguard: the LLM is a compiler from natural language to executable statistical language, not an autonomous analyst.

Reliable evaluation of GenAI and agentic systems introduces yet another inferential role: GenAI outputs are treated as noisy surrogates that can be combined with human labels through prediction-powered inference. GLIDE implements mean-estimation estimators such as PPI++, stratified PPI, Predict-Then-Debias, and Active Statistical Inference under a common API (Martinon et al., 29 May 2026). The canonical estimator is

$G(x)\rightarrow x$2

and the power-tuned version is

$G(x)\rightarrow x$3

Here the LLM is neither the target of evaluation nor the final decision-maker; it is a noisy measurement device whose predictions can reduce labeling cost while preserving valid uncertainty. In an R-Judge case study with 568 user/agent conversations, 100 human labels plus proxy labels yield effective sample sizes of about 143 for PPI++, about 148 for Active Statistical Inference, and about 157 for Stratified PPI++, while proxy-only intervals remain biased (Martinon et al., 29 May 2026).

Applied educational systems further show how GPI can mediate dialogue and assessment. A dataset collected from 279 postgraduate students using the FLoRA platform records student–GenAI dialogue transcripts, writing logs, final project proposals, and surveys, with GPT-4o embedded as a constrained chatbot that supports information problem solving rather than producing complete proposal content (Li et al., 19 Jan 2026). In “Socratic Mind,” a GenAI-powered formative assessment tool uses multi-turn, adaptive dialogues, ASR, instructor-configured question sets, and LLM-generated session summaries; students who engaged with the tool showed significant gains in quiz scores relative to non-users, with stronger benefits for lower baseline achievement, while qualitative evidence emphasized gains in problem solving, critical thinking, and self-reflection (Lee et al., 18 Sep 2025). These interface-oriented systems suggest a broader principle: GPI can be designed so that GenAI mediates specification, questioning, or surrogate evaluation while leaving formal inference, grading, or high-impact actuation to constrained or human-auditable components.