PromptPrism: Analytic Framework for LLM Prompts

• PromptPrism is a linguistically-inspired analytic framework and taxonomy that systematically dissects prompts into structural, semantic, and syntactic components.
• It leverages linguistic theories such as Rhetorical Structure Theory and pragmatics to enable reproducible prompt profiling and controlled sensitivity analyses.
• Empirical evaluations reveal that taxonomy-guided refinement can boost LLM text generation performance by up to 112%, evidencing its practical impact in prompt engineering.

PromptPrism is a linguistically-inspired analytic framework and taxonomy for prompts targeting LLMs, designed to enable systematic prompt dissection, profiling, refinement, and principled sensitivity analysis. It formally treats prompts as structured discourse sequences and provides a multi-level annotation scheme capturing structural, semantic, and syntactic properties essential for analyzing LLM behavior, optimizing prompt performance, and conducting controlled experiments. PromptPrism advances prompt engineering from an artisanal practice to a rigorous, reproducible discipline, with demonstrated empirical gains across diverse LLM architectures and instruction-following tasks (Jeoung et al., 19 May 2025).

1. Theoretical Foundations and Motivation

PromptPrism arises from the recognition that prompt design for LLMs, in contrast to human-to-human discourse, has lacked the benefit of established linguistic theory. The framework explicitly draws on Rhetorical Structure Theory, discourse models (Grosz & Sidner, 1986), pragmatics (Grice, 1975; Levinson, 1983), and morphological analysis (Matthews, 1972; Aronoff, 1976). Unlike prior taxonomies that rely on coarse distinctions such as "instruction vs. data," PromptPrism enables fine-grained annotation of prompt intentions, context, and formatting by integrating concepts from discourse analysis (hierarchy of roles and purposes), speech-act theory (illocutionary force), and morphosyntactic cues.

The motivating hypothesis is that a linguistically-principled taxonomy will allow systematic decomposition of prompt structure and function, reproducible cross-model/dataset profiling, and automated tools for prompt refinement and sensitivity quantification. This is supported by empirical findings that LLM output is strongly affected by prompt component ordering and semantic composition, but relatively robust to surface-level syntactic style (Jeoung et al., 19 May 2025).

2. Hierarchical Taxonomy: Structural, Semantic, and Syntactic Levels

PromptPrism characterizes prompts as ordered sequences of role–content pairs $P = \{(r_i, c_i)\}_{i=1}^n$, where each role $r_i$ belongs to a finite inventory $$R = \{\mathsf{system}, \mathsf{user}, \mathsf{assistant}, \mathsf{tools}, ...\}$$ and each content $c_i$ inhabits a modality space (in practice, restricted to text).

2.1 Structural Level

The structural axis records the macro-organization of the prompt as interlocutor turns or system instructions. Roles include:

• system: global instructions or persona setup
• user: the main query or command
• assistant: generated (LLM) output
• tools: constraints or parameters for API/function calls

Properties such as the number of turns, role-sequence patterns (e.g.\ system→user→tools), and role-switch complexity are critical analytic features.

2.2 Semantic Level

PromptPrism parses each content $c_i$ into nested semantic components based on discourse purpose, captured via XML-style tags:

• Instruction ($\mathcal{I}$): task directives, guidelines, chain-of-thought cues
• Contextual/Reference ($\mathcal{C}$): few-shot exemplars, retrieval context, background knowledge
• Output Constraints ($\mathcal{O}$): prescribed formats, label sets, tone/style limitations
• Tools ($\mathcal{T}$): explicit tool names, function parameters
• User Request ($\mathcal{Q}$): explicit user query/command
• Response ($r_i$0): expected output (used in annotation)
• Other ($r_i$1): adversarial or distractor material

Semantic coverage, as $r_i$2, and tree depth/width encode prompt complexity.

2.3 Syntactic Level

The syntactic stratum logs morphological and positional properties at the component level:

• Component indices and token/character spans
• Delimiter types (e.g., double newline, tab, mixed)
• Morphological directive markers (prefixes/suffixes), such as "#", ":", bullets, or list numbering
• Special model tokens, e.g., <|begin_of_text|>.

Systematic tracking of these features enables controlled perturbation experiments and quantification of stylistic complexity.

3. Taxonomy-Guided Prompt Refinement

PromptPrism enables the construction of pipeline algorithms for automated prompt enhancement. The refinement procedure first annotates a base prompt using taxonomy tags, inserts canonical templates for missing semantic components (ensuring, e.g., that Instruction, Context, OutputConstraints are present), reorders segments to conform to best practices (Instruction→Context→Query→Constraints), and converts tagged content back to natural language.

Algorithmic workflow (pseudocode, (Jeoung et al., 19 May 2025)): $r_i$7 Empirical evaluations on Super-NaturalInstructions v2.8 show that taxonomy-guided prompt refinement yields substantial performance gains, with up to 29% F1 improvement for text generation over standard chain-of-thought prompting in two-shot settings and up to 112% improvement in zero-shot (Jeoung et al., 19 May 2025).

4. Dataset Profiling and Multidimensional Corpus Analysis

PromptPrism extracts, for each prompt $r_i$3 in a dataset, its structural (turn count, role patterns), semantic (component frequencies, tree width/depth), and syntactic (delimiter, marker) features, along with token length and inferred task type. Aggregation over a dataset $r_i$4 allows for corpus-level statistics, facilitating the discovery of dominant interaction patterns and annotation gaps.

For example, dataset profiling of function-calling UIs (high Instruction/Tools coverage, deep semantic trees, strong delimiter regularity) versus chat-style logs (multi-turn, context-emphasized, shallow trees, and free-text structure) uncovers systematic structural differences. This enables targeted dataset improvement and benchmark construction (Jeoung et al., 19 May 2025).

5. Controlled Experimental Framework for Prompt Sensitivity

PromptPrism operationalizes sensitivity analysis by defining semantic and syntactic perturbations:

• Semantic: reordering, addition, or deletion of core components (Instruction, Request, Few-Shot exemplars)
• Syntactic: global substitutions among delimiter types (e.g., switching between \n\n, tab, and whitespace)

Paired with statistical hypothesis testing (ANOVA, effect sizes), PromptPrism reveals that LLM performance is highly sensitive to the semantic ordering of prompt segments, while surface syntactic changes (delimiter types) show no statistically significant effect ($r_i$5). For instance, placing Instruction last can yield a +12% gain for certain LLMs, while out-of-order composition may degrade accuracy by up to 76% (Jeoung et al., 19 May 2025).

6. Empirical Results, Limitations, and Future Directions

PromptPrism's framework improves LLM instruction following, especially in zero-shot and generative contexts, outperforms baseline prompt paradigms (e.g., naïve or CoT), and supports corpus-level profiling to guide data design. Automated annotation exhibits high agreement with human validation ($r_i$6), but scaling to larger manual-annotated benchmarks would further strengthen reliability.

Key limitations include a top-down bias that may miss emergent prompt phenomena discoverable via unsupervised analysis and restriction to textual inputs (multi-modal extension is a future goal). Prospective work includes integrating bottom-up corpus induction, expanding to non-text modalities, and deploying interactive tooling for real-time prompt design and diagnostics (Jeoung et al., 19 May 2025).

---

PromptPrism provides a linguistically-grounded, multi-level analytic framework that subsumes prompt representation, profiling, refinement, and controlled experimentation, delivering scientifically principled tools to transform prompt engineering into a systematic discipline.