Prompt2Constraints: Analyzing Student-LLM Prompts

• Prompt2Constraints is a logic-based framework that converts natural language prompts into formal propositional constraints, enabling automated and scalable analysis.
• It employs a multi-step pipeline including data collection, few-shot prompt engineering, automated constraint extraction, and change categorization to track semantic evolution.
• Empirical evaluations demonstrate over 95% extraction accuracy and reveal that specific prompt refinement patterns correlate with faster convergence in computational problem solving.

Prompt2Constraints is a logic-based framework for the systematic analysis of natural language prompts in student–LLM interactions, particularly in the context of computational problem solving in computing education. The framework provides a scalable technique for mapping the content of student prompts to formal propositional logic constraints, enabling precise quantification and tracking of prompt intent and its semantic evolution over iterative refinement cycles. Prompt2Constraints advances prior qualitative and quantitative prompt analysis approaches by supporting automated extraction, change categorization, and statistical analysis across large datasets while preserving semantic interpretability (Alfageeh et al., 25 Apr 2025).

1. Mapping Natural-Language Prompts to Propositional Constraints

Prompt2Constraints operates by interpreting each student prompt $P_i$ as implicitly asserting a set of atomic requirements, which are represented as propositional symbols $C_j$. For example, the prompt “Write me a Python function that returns how many times 0 appears in a list” is decomposed as:

• $C_1$: output must be a Python function
• $C_2$: function returns a count
• $C_3$: count of integer 0

The core mapping produced by the framework for each prompt $P_i$ is a logic formula of the form:

$$P_i \rightarrow (C_{a} \wedge C_{b} \wedge \ldots \wedge C_{k})$$

For a session comprising prompts $P_1,\ldots,P_n$, the evolution of constraints can be classified as (a) preserved, (b) newly added, (c) removed/modified, or (d) reworded. This symbolic formalism makes it straightforward to compare prompts in terms of their logical content.

2. Formal Syntax and Semantics

Prompt2Constraints models all constraints in propositional (zero-order) logic. The set of constraints for a given task is $V = \{C_1, C_2, \ldots, C_m\}$. The permitted syntax includes:

• atomic constraints: $C_j$
• connectives: $C_j$0, $C_j$1, $C_j$2, $C_j$3

The semantics are standard: any interpretation $C_j$4 determines the truth of $C_j$5 via the usual truth tables. In practical application, constraints are reduced to a conjunction of positive literals (CNF):

$C_j$6

This form is sufficient to express the intent of prompts in standard programming exercises and supports efficient change tracking.

3. The Prompt2Constraints Analytical Pipeline

The end-to-end pipeline comprises the following steps:

1. Data Collection: Sequences of student prompts ($C_j$7) are gathered for each programming exercise.
2. Few-shot Prompt Engineering: A small "primer" for the LLM (e.g., GPT-4), consisting of two consecutive prompts with human-annotated constraint decompositions, is constructed.
3. Automated Constraint Extraction: The few-shot examples and analysis prompts are sent to the LLM, which outputs, for each prompt:
   • a list of human-readable constraint descriptions,
   • a formal conjunction $C_j$8,
   • a summary of differences from the previous step.
4. Change Categorization: For each turn, additions and deletions between $C_j$9 and $C_1$0 are identified; turns are classified as “adding” (additions$C_1$1, deletions$C_1$2), “modifying” (deletions$C_1$3), “rewording” (text changed, no constraint change), or “resubmission” (unchanged).
5. Aggregation and Analysis: Student-level statistics, heatmaps, correlation matrices, and spider charts are constructed; statistical measures such as Pearson correlation and Mann–Whitney U are applied as appropriate.

4. Worked Examples

The framework’s logic is illustrated with the following sequence (summarized from Table 1 of the source):

Prompt (English)	Constraints Added / Modified	$C_1$4 Form
P1: "Write me a Python function that returns how many elements in a given list is the integer 0."	C₁, C₂, C₃	$C_1$5
P2: "Write me a Python function called counter(test_input) that returns how many elements ... integer 0."	add C₄ (function and parameter name)	$C_1$6
P3: same as P2 but parameter is “user_input”	replace C₄ with C₅ (parameter renamed)	$C_1$7

Constraint evolution is thus directly and transparently captured in the propositional formalism.

5. Dataset and Empirical Evaluation

Prompt2Constraints was evaluated on 1,872 prompts from 203 undergraduate CS1 students across three "prompt problems" (counter, initials, repeat). Key statistics:

• Average constraints per prompt:
  • Task 1: $C_1$8 $C_1$9
  • Task 2: $C_2$0 $C_2$1
  • Task 3: $C_2$2 $C_2$3
• Constraint extraction accuracy: Out of a random sample of 234 prompts, 225 were correctly decomposed (96.2%). The remaining 9 prompts had one fewer atomic constraint than an expert would assign.
• Constraint-evolution analysis: Change types (adding, modifying, rewording, resubmission) are robustly classified. Heatmaps demonstrate that “adding constraints” occurs primarily in early session turns, “modifying” spikes during backtracking or strategy shifts.
• Correlation measures: For example, the percentage of “adding” correlates inversely with session duration ($C_2$4, $C_2$5), indicating that prompt sessions with more additions converge faster. Similar positive correlations are observed for “modifying,” “rewording,” and “resubmission” with length.
• No significant difference was found in the average constraint-difference per turn between successful and unsuccessful sessions (mean $C_2$6 vs $C_2$7, $C_2$8).

6. Key Empirical Findings

Empirical analysis demonstrates:

• Successful and unsuccessful sessions use a similar number of constraints; unsuccessful sessions are characterized by larger, more erratic shifts, particularly in the form of “modifying constraints,” suggesting strategy volatility.
• Sessions dominated by “adding constraints” converge more rapidly.
• Abrupt jumps in constraint trajectories (as seen in spider-chart visualizations) correspond to student frustration or major strategy changes, identifying potential intervention points.
• Reductions in prompt length post-solution are primarily surface-level rewording, not logic-level refinement.

These findings provide scalable markers for prompt quality and student struggle, leading to the potential for automated, context-sensitive interventions.

7. Integration Into Real-Time Educational Tooling

Prompt2Constraints can be modularly embedded in programming environments (IDEs, LLM playgrounds, Jupyter plug-ins):

• For each prompt submission, extract $C_2$9 and compare to a library of gold-standard solutions $C_3$0.
• Compute constraint distance $C_3$1 as a measure of student progress.
• If $C_3$2 stagnates or deletions spike, the tool can inject contextually-targeted hints (e.g., “It looks like you've dropped the requirement to return an integer. Try adding back ‘return the count’.").
• A dedicated side panel can visualize constraint coverage and contradiction.
• Complete session logs can provide instructors and learning analytics systems with detailed trajectory data on strategic evolution and common stumbling blocks.

Prompt2Constraints thus operationalizes the collection, interpretation, and intervention of student-LLM interactions in a scalable, logic-centered workflow, yielding $C_3$3 semantic extraction accuracy and supporting both summative research and real-time pedagogical scaffolding (Alfageeh et al., 25 Apr 2025).