Dual-Classifier Prompt-Guided Search

• Dual-Classifier Prompt-Guided Search is an evolutionary optimization framework that uses two independent multi-label classifiers to evaluate and optimize generated images based on dual semantic criteria.
• The method employs an NSGA-II style evolutionary process, balancing prompt fidelity with two distinct user objectives to trace a clear and interpretable 2D Pareto frontier.
• Experimental results with Stable Diffusion and classifiers like Artemis and CLIP demonstrate efficient convergence and diverse applications in art design, emotion-guided advertising, and data augmentation.

Dual-Classifier Prompt-Guided Search refers to an evolutionary optimization framework for generative models conditioned on prompts, in which two (dual) independent multi-label image classifiers guide the search for outputs that simultaneously satisfy two distinct user-preference objectives. This dual-objective procedure forms a special case of classifier-guided prompt evolution, tailored for environments where user intent or requirements are best represented as a pair of measurable semantic properties or classes, such as emotive state and visual scene element, jointly optimized within a constraint on prompt fidelity (Wong et al., 2023).

1. Formal Optimization Problem: Dual Objectives

The objective is to produce a set of images $X \subseteq \mathbb{R}_+^{n \times k \times c}$ that satisfy two user-specified preference labels %%%%1%%%%. Each label is a target concept expressible as a classifier query (for example, $\lambda_1 =$ ‘person reading book’, $\lambda_2 =$ ‘awe’). Given:

• Prompt $\theta$: A conditioning variable, typically a natural language string or reference image.
• Generative model $\mathcal{G}$: Frozens text-conditional diffusion models, e.g., Stable Diffusion.
• Two multi-label classifiers $\mathcal{F}_1$ and $\mathcal{F}_2$: $x \mapsto [0,1]$, each outputting $f_i(x) = p_i(x) = P(\lambda_i|x)$.

The search seeks to maximize both classifier probabilities under a prompt-consistency constraint:

$$\begin{align*} \text{maximize}\quad & (f_1(x), f_2(x)) \ \text{subject to}\quad & d(x, \theta) \leq b, \end{align*}$$

where $$d(x, \theta) = \tau [1 - \cos(x_{\text{CLIP}}, \theta_{\text{CLIP}})]$$, $\tau > 0$ controls the constraint tightness, and $b$ is a threshold set for prompt adherence.

For two objectives ($Q=2$), Pareto dominance reduces to: $x \succeq x'$ iff $f_1(x) \geq f_1(x')$ and $f_2(x) \geq f_2(x')$, with at least one strict inequality.

2. Integration of Dual-Classifier Objectives

Dual-classifier prompt-guided search employs two pre-trained, multi-label classifiers as independent evaluators of generated images. Each classifier measures adherence to a different label, supporting simultaneous optimization along both semantic axes. In the methodology described by (Wong et al., 2023), classifiers are chosen from:

• Artemis: Transformer-based, fine-tuned on art emotion labels.
• CLIP-based classifier: ViT-text dual-encoder, trained on ~400M image–text pairs.
• Tresnet: CNN architecture specialized for multi-label object recognition.

For any image $x$, the system defines $f_i(x) = p_i(x)$, where each classifier probability is considered conditionally independent of the other (multi-label assumption). Explicit label correlations are not modeled; emergent co-occurrences may nonetheless materialize through generative model $\mathcal{G}$ during search.

3. Evolutionary Search Mechanism

The evolutionary process is an NSGA-II–style multi-objective optimization, adapted to leverage generative models for mutation. The algorithm can be summarized as follows:

• Initialization: Sample an initial population $P_0$ of $M$ images from $\mathcal{G}(\cdot|\theta)$.
• Loop (for T generations):
  • Evaluate both classifier scores $(f_1(x), f_2(x))$ on all images $x \in P_{t-1}$.
  • Generate offspring by producing $\lambda$ “mutated” child images per parent, using the pre-trained generative model $\mathcal{G}(x'|\theta, x)$, where $x$ is optionally provided as additional conditioning.
  • Pool parents and offspring into $U_t$.
  • Sort $U_t$ into Pareto fronts via non-dominated sorting.
  • Construct the new population $P_t$ up to size $M$, using descending crowding distance to break ties.
  • For the dual-objective case ($Q=2$), crowding distance is computed as:

  $$d_{\text{crowd}}(x) = \frac{f_1^{i+1} - f_1^{i-1}}{f_1^{\max} - f_1^{\min}} + \frac{f_2^{j+1} - f_2^{j-1}}{f_2^{\max} - f_2^{\min}}$$

• Output: The union of all non-dominated individuals from all generations summarizes the discovered Pareto frontier.

No explicit crossover is employed. Prompt mixing (combining two prompts) remains proposed as a potential extension but is not present in the described work.

4. Empirical Setup, Metrics, and Results

Experimental Configuration

• Prompts: Ten natural-language proverbs (e.g., “A journey of a thousand miles begins with a single step”).
• Objectives: $Q=2$–$3$ per proverb; for dual-classifier runs, $Q=2$.
• Model: Stable Diffusion v1.4 (pre-trained, frozen).
• Classifiers: Artemis, CLIP-based, Tresnet (all pre-trained and fixed).
• Key hyperparameters: $M=50$ ($\text{pop size}$), $\lambda=3$ (offspring per parent), $T=20$ (generations), $b=0.3$, $\tau=100$.

Evaluation Metrics

• Classifier-adherence: Mean $f_i$ over Pareto front.
• Diversity: Hypervolume (HV) with respect to the reference point $(0,0)$.

Key Findings

• Initial generations (t=0) contain many images with low adherence to at least one objective (low HV).
• By $t \approx 6$, HV of the front surpasses a brute-force baseline of $1\,550$ independently sampled candidates, both with and without prompt engineering.
• The final non-dominated set ($t=20$) traces a clear 2D Pareto frontier, demonstrating trade-offs between objectives (e.g., “person reading book” and “awe” emotion).
• Visualization and computation of fronts for $Q=2$ are straightforward ($O(N \log N)$); a plausible implication is that the dual-classifier setting is computationally efficient and interpretable relative to higher-dimensional objectives.

5. Interpretation and Limits of the Dual-Classifier Approach

Dual-classifier (two-objective) prompt-guided search offers several salient properties:

• Interpretability: The Pareto front is a 2D curve, which facilitates direct user inspection of trade-offs between preferences.
• Convergence: Focusing search on two objectives accelerates optimization relative to $(Q > 2)$, due to lower-dimensional trade-off space.
• Coverage: The method still explores diverse image solutions spanning the semantic space controlled by the two classifiers and the generative prior.

Constraints arise from classifier coverage and quality. Only the axes defined by the chosen classifiers are actively optimized, and dependencies or higher-order semantic relations between labels are not directly captured. The scheme is also computationally intensive compared to single-pass generation, as each evolutionary iteration invokes the generative model multiple times per prompt.

6. Applications, Extensions, and Future Directions

Notable application areas include art design, emotion-guided advertising, and data augmentation for classifier training, where trade-offs between two core user intents can be specified as dual objectives. Potential future research includes:

• Joint evolution of prompts and generated outputs (“hybrid prompt- and output-space evolution”).
• Modeling label correlations explicitly, for example, by introducing multi-output Gaussian process surrogates.
• Extending methodology to other modalities such as video or audio, with temporal extension of the mutation mechanism.
• Empirical studies on user perception of fidelity–diversity trade-offs, particularly under dual-classifier guidance (Wong et al., 2023).

A plausible implication is that while dual-classifier search balances tractability and interpretability, richer semantic customization may require extending the approach to higher-dimensional multi-objective spaces or modeling inter-label dependencies explicitly.