Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, LLaMA (2512.12812v1)

Abstract: Prompt engineering has emerged as a critical factor influencing LLM performance, yet the impact of pragmatic elements such as linguistic tone and politeness remains underexplored, particularly across different model families. In this work, we propose a systematic evaluation framework to examine how interaction tone affects model accuracy and apply it to three recently released and widely available LLMs: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama 4 Scout (Meta). Using the MMMLU benchmark, we evaluate model performance under Very Friendly, Neutral, and Very Rude prompt variants across six tasks spanning STEM and Humanities domains, and analyze pairwise accuracy differences with statistical significance testing. Our results show that tone sensitivity is both model-dependent and domain-specific. Neutral or Very Friendly prompts generally yield higher accuracy than Very Rude prompts, but statistically significant effects appear only in a subset of Humanities tasks, where rude tone reduces accuracy for GPT and Llama, while Gemini remains comparatively tone-insensitive. When performance is aggregated across tasks within each domain, tone effects diminish and largely lose statistical significance. Compared with earlier researches, these findings suggest that dataset scale and coverage materially influence the detection of tone effects. Overall, our study indicates that while interaction tone can matter in specific interpretive settings, modern LLMs are broadly robust to tonal variation in typical mixed-domain use, providing practical guidance for prompt design and model selection in real-world deployments.

• The paper demonstrates that prompt tone can shift accuracy, with very rude prompts reducing performance by up to 3.2% in humanities tasks.
• The study employs a rigorous empirical design using three LLMs and six tasks, with multiple trial runs to ensure 95% confidence in the findings.
• The research highlights that while Gemini is largely tone-insensitive, model-specific variations affect GPT and LLaMA, particularly in interpretive domains.

Tone Sensitivity in LLMs: A Cross-Model Evaluation of Politeness Effects

Introduction

The research paper "Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, LLaMA" (2512.12812) offers a detailed examination of how prompt tone—specifically degrees of politeness—affects the accuracy of state-of-the-art LLMs. Despite the prominence of prompt engineering in optimizing LLM use, the nuanced influence of linguistic tone remains only partially understood. This paper provides an empirical framework for quantifying such effects, focusing on three production-scale models: GPT-4o mini (OpenAI), Gemini 2.0 Flash (Google DeepMind), and Llama4 Scout (Meta).

Methodological Framework

The study systematically measures accuracy shifts induced by three prompt tones: Very Friendly, Neutral, and Very Rude. These tone variants are applied across six representative MMMLU tasks—comprising both STEM and Humanities—enabling robust cross-domain analyses. The experiment design includes ten runs per question-tone-model combination, minimizing stochasticity and allowing accurate estimation of tone-induced effects with 95% confidence intervals. Both per-task and aggregated domain-level results are examined.

The selected models represent small-to-mid-scale, public, and industry-stable LLMs from distinct architectural and alignment families. While direct equivalence is not guaranteed due to unknowns in proprietary architectures (e.g., Gemini's parameter count, effects of mixture-of-experts in Llama4), the comparison provides high external validity for widely deployed LLM classes.

Empirical Findings

General Tone Effects

A clear directional trend is observed: across most (27/36) model-task pairings, Neutral or Very Friendly prompts yield higher accuracy than Very Rude ones. This trend prevails in both STEM and Humanities, but statistical significance is primarily confined to select Humanities tasks. Notably, statistically significant accuracy degradation (up to +3.2% mean difference; 95% CI nonzero) is observed for Very Rude prompts in Philosophy (GPT, Llama) and Professional Law (Llama) tasks. In contrast, Very Friendly prompts do not always outperform Neutral—occasional significant drops in accuracy are observed for overly friendly prompt variants, indicating non-monotonic sensitivity in some cases.

Model-Specific Robustness

Gemini 2.0 Flash consistently exhibits minimal tone sensitivity, with all pairwise differences falling within non-significant ranges across tasks and tone comparisons. In contrast, GPT-4o mini and Llama4 Scout show measurable, and occasionally significant, sensitivity in specific Humanities domains. This divergence suggests that tone effects likely result from interaction effects between training corpus properties, alignment protocols, and decoder sampling strategies, rather than intrinsic architectural limitations.

Domain-Specificity

Tone-induced accuracy differences are much more pronounced in Humanities (interpretive) tasks than in fact-based STEM categories. STEM effects are typically minor and not statistically significant, supporting the hypothesis that contextually ambiguous or subjectively evaluative questions amplify susceptibility to tone-variant adversarial prompting, whereas deterministic tasks are robust.

Domain Aggregation

When model performance is aggregated across multiple tasks per domain, the effect sizes attenuate and lose statistical significance. This indicates that, in real-world mixed-domain deployments, the risk of material accuracy degradation due to prompt rudeness is low—robustness to stereotype or affective prompt variation is an emergent property of modern LLMs when applied broadly.

Contradiction With Prior Work

Contrary to the findings by Dobariya and Kumar (who observed impolite prompting improves GPT-4o accuracy in smaller-scale tests), this study demonstrates that effect detection is highly contingent on task coverage and data size. The discrepancy highlights the crucial role of experimental scale in assessing prompt engineering claims, and cautions against overgeneralizing from limited-question datasets.

Implications and Directions for Future Research

Practical Implications

For practitioners, the results guide prompt engineering strategies: in most typical mixed-domain settings, LLM predictions are largely invariant to pragmatic tone variation, and adverse effects of user rudeness are rare and domain-limited. For highly specialized, interpretive tasks—especially in Humanities—neutral tone is optimal for accuracy, while highly rude phrasing may be deleterious for GPT and Llama.

For model selection, deployment contexts where linguistic robustness is critical (e.g., open-domain chat, public-facing APIs) may prefer models such as Gemini, which show empirical insensitivity to input tone across benchmarks.

Theoretical Implications

The findings reinforce that fine-grained behavioral properties of LLMs (e.g., pragmatic tone sensitivity) are strongly shaped by data, alignment protocols, and possibly the extent of instruction-following tuning in the underlying architectures. The non-significant tone sensitivity of Gemini suggests that alignment data and policy reinforcement in Gemini-like models generalize better across pragmatic context.

Future Trajectories

The research emphasizes the need to:

• Expand comparative analyses to a broader array of open-source LLMs with reproducible training protocols.
• Extend evaluations to non-English, open-ended, and multimodal input paradigms, as cue-driven tone effects may be amplified or modified in non-MCQA settings.
• Investigate alternative social cues (formality, affect, indirectness) and use auxiliary evaluation metrics such as calibration, safety, and user satisfaction.

Exploring architectural, corpus, and alignment variants will help disentangle the causal mechanisms behind observed sensitivity differences.

Conclusion

This paper demonstrates that, while prompt tone can induce limited and domain-specific shifts in LLM accuracy, modern LLMs—especially when assessed across large, representative benchmarks and diverse families—are broadly robust to user-imposed pragmatic variations. The statistically significant effects of rudeness on Humanities tasks for GPT and Llama underscore localized risks, but these do not generalize under domain aggregation. Model choice remains consequential in safety-critical or open-domain environments, where tone-invariant performance is desirable. The results strengthen the empirical foundation for prompt engineering recommendations and model robustness analysis, and advocate for large-scale, systematic behavioral evaluations in future LLM research (2512.12812).