LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings (2510.08338v1)

Abstract: Consumer research costs companies billions annually yet suffers from panel biases and limited scale. LLMs offer an alternative by simulating synthetic consumers, but produce unrealistic response distributions when asked directly for numerical ratings. We present semantic similarity rating (SSR), a method that elicits textual responses from LLMs and maps these to Likert distributions using embedding similarity to reference statements. Testing on an extensive dataset comprising 57 personal care product surveys conducted by a leading corporation in that market (9,300 human responses), SSR achieves 90% of human test-retest reliability while maintaining realistic response distributions (KS similarity > 0.85). Additionally, these synthetic respondents provide rich qualitative feedback explaining their ratings. This framework enables scalable consumer research simulations while preserving traditional survey metrics and interpretability.

• The paper presents the SSR method that maps free-text responses from LLMs to Likert-scale distributions, reaching 90% human test–retest reliability.
• It demonstrates that SSR outperforms direct and follow-up Likert rating approaches with high distributional similarity (KS > 0.85) and robust demographic alignment.
• The approach provides scalable, interpretable synthetic consumer surveys that capture both quantitative fidelity and qualitative insights for market research.

Semantic Similarity Elicitation Enables LLMs to Reproduce Human Purchase Intent Distributions

Introduction

The paper "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings" (2510.08338) addresses a critical limitation in the application of LLMs for synthetic consumer research: the inability of LLMs to generate realistic Likert-scale response distributions when directly prompted for numerical ratings. The authors introduce the semantic similarity rating (SSR) method, which leverages free-text responses from LLMs and maps them to Likert-scale distributions using embedding-based semantic similarity to reference anchor statements. This approach is evaluated on a substantial dataset of 57 personal care product surveys (9,300 human responses), demonstrating that SSR achieves 90% of human test–retest reliability and high distributional similarity (KS similarity > 0.85) to real survey data.

Methodology

Synthetic Consumer Construction and Response Elicitation

Synthetic consumers are instantiated by prompting LLMs (GPT-4o and Gemini-2.0-flash) with demographic attributes and product concept stimuli (text and/or image). Three response generation strategies are compared:

1. Direct Likert Rating (DLR): LLMs are asked to respond with an integer (1–5) directly.
2. Follow-up Likert Rating (FLR): LLMs first generate a free-text statement, which is then mapped to a Likert score by a second LLM instance acting as a "Likert rating expert."
3. Semantic Similarity Rating (SSR): Free-text responses are embedded and compared to five reference anchor statements (one per Likert category) using cosine similarity. The resulting similarities are normalized to produce a probability mass function (pmf) over the Likert scale.

Figure 1: Overview of response generation procedures and SSR mapping, illustrating the construction of synthetic consumers and the embedding-based mapping to Likert distributions.

Success Metrics

Two primary metrics are used to evaluate synthetic panels:

• Distributional Similarity: Kolmogorov–Smirnov (KS) similarity between synthetic and real Likert distributions.
• Correlation Attainment ($\rho$): Pearson correlation between mean purchase intents of synthetic and real surveys, normalized by the maximum attainable correlation from human test–retest reliability.

Results

Direct Likert Rating Baseline

DLR yields high correlation attainment ($\rho \approx 80\%$) but poor distributional similarity (KS similarity: 0.26 for GPT-4o, 0.39 for Gem-2f). LLMs predominantly regress to the center of the scale (response '3'), rarely producing extreme ratings ('1' or '5'), resulting in unrealistic, narrow distributions.

Figure 2: Comparison of real and synthetic survey distributions for GPT-4o, showing the limited dynamic range of DLR responses.

Figure 3: KS similarity comparison for GPT-4o, highlighting the distributional mismatch of DLR versus SSR and FLR.

SSR and FLR Performance

SSR markedly improves both metrics: for GPT-4o, SSR achieves $\rho = 90\%$ and KS similarity $xy = 0.88$; for Gem-2f, $\rho = 90\%$ and $xy = 0.8$. FLR also improves over DLR but is consistently outperformed by SSR in distributional similarity. SSR-generated distributions closely match human data, and product concept rankings by mean purchase intent are robust.

Figure 4: Gem-2f results showing SSR and FLR distributions compared to real data, with SSR achieving superior alignment.

Figure 5: KS similarity for Gem-2f, demonstrating SSR's advantage in distributional fidelity.

Demographic and Product Feature Conditioning

SSR enables LLMs to replicate demographic and product feature effects observed in human data. For example, mean purchase intent exhibits a concave relationship with age, and income level conditioning produces lower purchase intent for budget-constrained personas. Product category and price tier effects are also mirrored.

Figure 6: Mean purchase intent stratified by demographic and product features, showing SSR's ability to capture nuanced subgroup effects.

Figure 7: Gender and region stratification, indicating weaker alignment for these features but overall low influence on purchase intent.

Qualitative Feedback

SSR preserves the richness of free-text responses, enabling qualitative analysis. Synthetic consumers provide detailed rationales for their ratings, often surpassing the depth of human survey responses. This qualitative data can be mined for actionable insights in product development.

Generalization and Ablation

SSR generalizes to other Likert-based constructs (e.g., concept relevance) with high correlation attainment and distributional similarity. Removing demographic conditioning increases distributional similarity but reduces correlation attainment, indicating that persona prompts are essential for meaningful product differentiation.

Comparison to Supervised ML

A LightGBM classifier trained on demographic and product features achieves lower correlation attainment ($\rho = 65\%$) than SSR ($\rho = 88\%$), despite moderate distributional similarity. This demonstrates the superiority of zero-shot LLM elicitation for capturing human-like response behavior without training data.

Implementation Details

SSR Algorithm

The SSR mapping is implemented as follows:

import numpy as np def ssr(response_text, reference_texts, embedding_model, temperature=1.0, epsilon=0.0): # Embed response and reference statements response_vec = embedding_model.embed(response_text) ref_vecs = [embedding_model.embed(ref) for ref in reference_texts] # Compute cosine similarities sims = np.array([np.dot(response_vec, ref_vec) / (np.linalg.norm(response_vec) * np.linalg.norm(ref_vec)) for ref_vec in ref_vecs]) # Subtract minimum similarity for normalization sims -= sims.min() # Add epsilon to avoid zero probabilities sims += epsilon # Apply temperature scaling probs = sims ** (1.0 / temperature) # Normalize to obtain pmf pmf = probs / probs.sum() return pmf

Reference anchor sets should be carefully constructed to span the semantic range of Likert categories. Averaging over multiple anchor sets mitigates mapping variance.

Resource and Scaling Considerations

SSR requires access to a high-quality embedding model (e.g., OpenAI's text-embedding-3-small). The computational cost is dominated by embedding inference, which is tractable for survey-scale applications. SSR is model-agnostic and does not require fine-tuning, making it suitable for rapid deployment and scaling across domains with sufficient LLM training coverage.

Trade-offs and Limitations

• Reference Set Design: Manual optimization of anchor statements is required; dynamic or LLM-generated anchors may further improve alignment.
• Demographic Conditioning: SSR captures some but not all subgroup effects; caution is warranted in interpreting synthetic subgroup analyses.
• Domain Coverage: SSR's validity depends on LLM exposure to relevant product categories in training data.
• Embedding Model Choice: Alternative embedding spaces may yield improved results; benchmarking is recommended for new domains.

Implications and Future Directions

SSR establishes a robust framework for synthetic consumer research, enabling scalable, interpretable, and cost-effective simulation of human survey panels. The method preserves both quantitative and qualitative fidelity, facilitating richer product concept evaluation. Potential extensions include:

• Generalization to other survey constructs (e.g., satisfaction, trust)
• Automated optimization of SSR parameters (temperature, epsilon)
• Multi-stage LLM pipelines for enhanced interpretability
• Hybrid approaches combining SSR with light fine-tuning or calibration

SSR can accelerate early-stage product screening, reduce research costs, and democratize access to consumer insights. However, it should be viewed as an augmentation rather than a replacement for human panels, especially in domains with limited LLM training data.

Conclusion

Semantic similarity elicitation via SSR enables LLMs to reproduce human purchase intent distributions and product rankings with high fidelity, overcoming the limitations of direct Likert-scale elicitation. The approach is computationally efficient, interpretable, and preserves qualitative richness, making it a valuable tool for synthetic consumer research. While further optimization and validation are warranted, SSR represents a significant advance in the practical application of LLMs for survey simulation and market research.