Domain-Informed Text Gradients

• Domain-informed text gradients are optimization techniques that incorporate domain-specific criteria, such as aesthetic preferences, scientific constraints, or language model priors, into gradient calculations.
• They enable controlled generation in models by updating embeddings or token sequences directly, as seen in text-to-image diffusion and multi-agent LLM guidance frameworks.
• Empirical results demonstrate improved alignment with domain goals, increased validity of outputs in scientific discovery, and heightened vulnerability in text reconstruction via gradient inversion.

Domain-informed text gradients are a class of optimization and guidance methodologies that steer generative models or reconstruction procedures by incorporating domain-specific knowledge, constraints, or feedback directly within the gradient computation over text or text-conditional models. These techniques have emerged as pivotal in tasks ranging from controlled text-to-image generation and privacy attacks on textual models to conditional chemical composition discovery with LLMs. Central to the concept is the integration of domain priors—whether from human preferences, scientific constraints, or learned surrogates—into gradients computed with respect to discrete or continuous text representations, thereby shifting outputs toward regions of higher domain-aligned value.

1. Mathematical Formulation of Domain-Informed Text Gradients

Domain-informed text gradients can be formalized within various modalities that take text as input or conditioning. Canonical settings include text-to-image diffusion models, sequence generation with LLMs, and inference-time optimization of discrete token representations.

Text-to-Image Gradient Conditioning

A prototypical example is aesthetic gradients in CLIP-based diffusion models (Gallego, 2022). Given:

• Text prompt $y$
• Aesthetic reference images $\{x_i\}_{i=1}^K$
• CLIP text encoder $\mathrm{CLIP}_{\theta,\mathrm{txt}}$ and vision encoder $\mathrm{CLIP}_{\theta,\mathrm{vis}}$ The process defines unit-normalized prompt embedding $c$ and reference embedding $e$, then maximizes their cosine similarity $S(c,e) = c^\top e$ by performing gradient ascent on the CLIP text encoder weights $\theta$:

$$L_{\mathrm{aest}}(\theta) = -\mathrm{CLIP}_{\theta,\mathrm{txt}}(y)^\top e$$

and updates

$$\theta^{(t+1)} = \theta^{(t)} + \epsilon \nabla_{\theta^{(t)}} S(\mathrm{CLIP}_{\theta^{(t)},\mathrm{txt}}(y), e)$$

After $N$ steps, the personalized text embedding $c'$ is used as the sole condition for the downstream diffusion model.

Multi-Agent LLM Guidance

For domain-conditioned generation, such as double perovskite (DP) composition discovery, Lee et al. (Lee et al., 27 Nov 2025) define gradients in discrete composition (token) space:

• LLM gradient: $$\Delta x_{\mathrm{LLM}} = -\nabla_x \mathcal{L}_{\mathrm{text}}(x)$$ with $\mathcal{L}_{\mathrm{text}}(x) = -\log p_\theta(x)$
• Domain knowledge gradient: $$\Delta x_{\mathrm{DK}} = -\nabla_x \mathcal{L}_{\mathrm{domain}}(x)$$, where $$\mathcal{L}_{\mathrm{domain}}(x) = \max\{0, T(x)-T_0\}$$ encodes stability via a physics-based tolerance factor $T(x)$
• ML surrogate gradient: $$\Delta x_{\mathrm{ML}} = -\nabla_x \mathcal{L}_{\mathrm{ML}}(x)$$ for a surrogate stability predictor

Gradients are linearly combined:

$$\Delta x_{\mathrm{total}} = \alpha \Delta x_{\mathrm{LLM}} + \beta \Delta x_{\mathrm{DK}} + \gamma \Delta x_{\mathrm{ML}}$$

Gradient-Guided Text Reconstruction

In the context of privacy research, LAMP (Balunović et al., 2022) reconstructs text from gradients by minimizing

$$\min_x \left[ L_{\mathrm{rec}}(\theta, x) - \lambda \log P_{\mathrm{LM}}(t) \right]$$

where $L_{\mathrm{rec}}$ matches input gradients, and $P_{\mathrm{LM}}$ is a LLM prior. Optimization alternates between gradient descent in embedding space and discrete token search.

2. Integration with Generative Model Architectures

Domain-informed text gradients are introduced into generative frameworks at points where text interacts with the model, altering sampling, personalization, or optimization trajectories.

CLIP-Conditioned Diffusion

In aesthetic gradient personalization, the prompt embedding supplied to the Stable Diffusion model is replaced with the personalized $c'$, shifting the entire diffusion run into a domain aligned with the user’s aesthetic (Gallego, 2022). No changes to the diffusion model architecture or its sampling procedure are necessary; only the CLIP text encoder parameters are updated (for a few steps), and the resulting embedding is used at all conditioning points.

Multi-Agent LLM Generation

In the knowledge-guided double perovskite discovery framework (Lee et al., 27 Nov 2025), domain gradients are computed and communicated between agents operating as LLM prompts. No architectural modifications to the LLM are made; instead, gradients are interpreted as textual suggestions or edits to the discrete token sequence, and the ProposalAgent iteratively refines output candidates.

Textual Data Privacy Attacks

LAMP physically reconstructs input sentences by matching the observed gradients, augmented with a LLM prior that enforces naturalness in the discrete solution. Alternating continuous optimization (over embeddings) and discrete transformations enhances the attack’s ability to recover long and coherent text (Balunović et al., 2022).

3. Algorithmic Workflows and Pseudocode

Algorithmic deployment of domain-informed text gradients is typically structured as follows:

Stage	Operation	Example Paper
Domain signal encoding	Compute or identify domain-specific criterion (aesthetic embedding, tolerance factor, etc.)	(Gallego, 2022, Lee et al., 27 Nov 2025)
Gradient computation	Differentiate loss w.r.t. text representation (embedding, token, or encoder weights)	(Gallego, 2022, Lee et al., 27 Nov 2025, Balunović et al., 2022)
Update/Proposal	Update embedding, token, or next candidate proposal using gradient-informed guidance	(Lee et al., 27 Nov 2025, Balunović et al., 2022)
Model sampling or output	Generate samples or predictions, possibly using modified embeddings or candidate proposals	(Gallego, 2022, Lee et al., 27 Nov 2025)

In CLIP-based diffusion, the full pseudocode involves five to twenty gradient ascent steps on the CLIP text encoder with respect to the cosine similarity with the domain embedding, then running standard SD sampling conditioned on the resulting vector (Gallego, 2022).

In knowledge-guided LLM-driven generation, proposals and their evaluations are iteratively refined via multiple specialized agents, each returning scalar losses and editable suggestions interpreted as gradients in token space (Lee et al., 27 Nov 2025).

In gradient-inversion attacks, the optimization alternates between continuous embedding-space descent and discrete token sequence search, always guided both by the match to true gradients and LLM likelihood (Balunović et al., 2022).

4. Domain Knowledge Injection Mechanisms

The efficacy of domain-informed text gradients hinges on the explicit encoding of domain knowledge in the loss or feedback functions.

Scientific and Physical Constraints

For double perovskites, stability is encoded as a loss based on the tolerance factor $T(x)$, calculable from ionic radii and oxidation states, with stability enforced via $T(x) \leq 4.18$ (Lee et al., 27 Nov 2025). The gradient is computed with respect to the token sequence representing the composition, and proposed edits (e.g., “swap I− for Br−”) are interpreted as descent steps toward constraint satisfaction.

Human or Aesthetic Preferences

Aesthetic gradients utilize a reference pool of images to define a domain of target “style” in CLIP latent space; matching the prompt embedding to the centroid of these reference visuals re-aligns text-to-image generations with user-defined aesthetic sub-domains (Gallego, 2022).

LLM Priors

In adversarial contexts, the probability of a sentence under a pretrained LLM is used as a prior to steer reconstructions toward plausible text, thereby imposing domain constraints corresponding to natural language (Balunović et al., 2022).

5. Empirical Evaluation and Practical Impact

Quantitative and qualitative evaluations across these applications underscore the effectiveness and limitations of domain-informed text gradients.

Controlled Generation

In personalized text-to-image generation, aesthetic gradients yield images with distinct alignment to reference styles, as measured by automated aesthetic scoring, outperforming stylistic keyword prompt engineering (Gallego, 2022). Mean aesthetic scores increase substantially across prompts compared to unpersonalized baselines.

Scientific Knowledge Integration

The multi-agent perovskite generation framework demonstrates over 98% validity for compositional constraints and lifts the proportion of stable/metastable candidates to 54%, surpassing LLM-only (43%) and GAN-based (27%) methods. Addition of an ML-surrogate gradient further improves results in in-distribution chemical families but degrades them out-of-distribution, highlighting the crucial importance of domain relevance in gradient construction (Lee et al., 27 Nov 2025).

Data Privacy

LAMP achieves a fivefold increase in bigram recovery rate compared to previous gradient-matching attacks, and remains uniquely effective for batch sizes greater than one (e.g., ROUGE-2 ≈29.5% for $B=2$) (Balunović et al., 2022), revealing severe privacy vulnerabilities in natural language federated learning setups.

6. Extensions to Latent and Multimodal Domains

Domain-informed gradient strategies are increasingly applied in multimodal generative architectures, such as text-to-3D generation. DreamPolish introduces Domain Score Distillation (DSD), linearly combining unconditional diffusion model guidance, classifier-free guidance, and “variational domain” guidance from fine-tuned models on user-specific renderings. The resulting gradients regularize texture optimization in text-to-3D tasks, enhancing photorealism and geometric consistency by confining updates to well-behaved domains in the latent manifold (Cheng et al., 3 Nov 2024).

7. Limitations and Scope of Applicability

Empirical analyses indicate that domain-informed gradients optimize toward domain-consistent and high-value regions only when the encoded domain signal remains reliable and within-distribution (Lee et al., 27 Nov 2025). ML-based surrogates can introduce instability or bias in unconstrained or poorly-represented chemical spaces. Similarly, the degree of alignment afforded by human aesthetic embeddings is contingent on the representational capacity of the embedding model and the specificity of reference images (Gallego, 2022). In privacy-oriented attacks, reliance on LLM priors may bias reconstructions toward generic text when the original data is domain-specific (Balunović et al., 2022).

---

Domain-informed text gradients thus constitute a versatile methodological paradigm for embedding scientific, aesthetic, or linguistic priors directly into gradient-based optimization pipelines over text representations, with demonstrated impact across generative modeling, scientific discovery, and adversarial data reconstruction (Gallego, 2022Lee et al., 27 Nov 2025Balunović et al., 2022Cheng et al., 3 Nov 2024).