Indirect and Direct AI Scaffolding for Computational Problem Posing: A Pilot Experience Report

Abstract: Problem posing is a valuable learning activity in computing education, encouraging learners to actively construct, refine, and reflect on problems rather than simply solving them. This experience report presents the design and pilot deployment of two LLM-powered scaffolding systems for supporting problem posing across two computational scenarios with different levels of task openness. Both systems assessed student-generated problems using Bloom's Taxonomy-based criteria and applied the same assessment framework, differing only in output modality: one provided guiding questions (Indirect scaffolding), while the other offered worked examples (Direct scaffolding). We conducted a within-subjects, counterbalanced pilot study with 20 graduate students and collected problem-quality ratings, user-experience surveys, and post-session interviews. Our deployment showed that both systems supported problem refinement in complementary ways, each offering distinct benefits. Direct scaffolding produced greater immediate improvements, while interviews showed that participants valued Indirect scaffolding for promoting deeper reflection on their own problem design. Based on these findings, we suggest sequencing the two modalities by beginning with Indirect scaffolding to promote reflection, then shifting to Direct scaffolding when learners become stuck. These lessons offer an initial practical strategy for integrating LLM-based scaffolding into computing classrooms.

• The paper demonstrates that both LLM scaffolding modes significantly improved computational problem quality, but Direct scaffolding produced larger gains than Indirect scaffolding (median improvement 1.83 vs. 0.67; p=.005).
• Direct scaffolding was most effective in the open-ended Making Change scenario and maintained stable gains across repeated tasks, while Indirect scaffolding better supported reflection but declined in effectiveness over time.
• The findings support an adaptive sequence that begins with reflective questions and shifts to worked examples after two or three attempts, while noting limits from the 20-participant graduate-student sample and lack of an unassisted control group.

Overview

This experience report presents the design and pilot evaluation of two LLM-powered scaffolding systems for computational problem posing, a learning activity in which students construct and refine their own problems rather than solve pre-defined ones (2607.09628). The two systems shared an identical assessment framework grounded in Bloom's Taxonomy but differed in output modality: Indirect scaffolding responded to a student's posed problem with guiding questions intended to prompt reflection, while Direct scaffolding provided worked-example revisions adapted from the student's submission. The authors deployed both systems in a within-subjects, counterbalanced pilot study with 20 computer science graduate students across two problem-posing scenarios of differing task openness, collecting problem-quality ratings, user-experience surveys (UEQ-S, NASA-TLX), and semi-structured interviews.

The central design motivation is the tension between cognitive support and cognitive offloading. Prior work shows that LLM assistance can reduce learner engagement when students accept generated outputs uncritically, yet overly open-ended guidance can leave students uncertain how to improve. The paper frames the relevant design question as how help is distributed between system and learner, rather than whether help is given at all.

System design

Both systems were implemented as high-fidelity web interfaces backed by GPT-4o via the OpenAI API. On submission, the model evaluated the posed problem along three dimensions — Conceptual Accuracy, Structural Clarity, and Cognitive Demand (the latter mapped to four Bloom's Taxonomy levels from Remember/Understand through Evaluate/Create) — identified the weakest dimension, and generated condition-specific feedback constrained to exactly two items. The Indirect condition produced two reflective questions without hints; the Direct condition produced two revised example problems positioned approximately one Bloom level above the original submission. Prompts explicitly forbade solving the problem or introducing new mathematical content. Four target qualities (accuracy, clarity, cognitive challenge, originality) were displayed throughout the activity.

The two scenarios were adapted from prior mathematical problem-posing research: Handshake, a combinatorial/graph-reasoning task with a narrow problem space, and Making Change, a greedy coin-change task offering richer numerical relationships. A third "Food Truck" scenario served only for training.

Study method

Twenty CS graduate students (17 of whom were teaching assistants) completed both scenarios under both conditions in a Latin-square counterbalanced design, three sequential tasks per scenario, each consisting of an initial problem, one scaffold request, and one revision. Problem quality was scored on a 0–2 rubric per dimension (maximum 8), with independent second-rater agreement at Cohen's $\kappa > 0.7$. Manipulation checks confirmed participants perceived the systems as stylistically distinct as intended (Indirect rated higher on prompting questions, $p=.016$; Direct rated higher on showing how to improve, $p=.031$).

Quantitative results

Both modalities improved problem quality significantly relative to original submissions (Wilcoxon signed-rank, $Z=-3.83$, $p<.001$, $r=.88$). However, Direct scaffolding produced substantially larger improvements than Indirect ($Mdn=1.83$ vs. $Mdn=0.67$; $W=25.5$, $p=.005$, $p=.016$0), an effect that held in mixed-effects regression controlling for participant variance ($p=.016$1, $p=.016$2, $p=.016$3). A ceiling effect was observed: participants with stronger initial problems showed smaller gains ($p=.016$4, $p=.016$5).

Scenario structure moderated these effects. In the open-ended Making Change scenario, Direct scaffolding significantly outperformed Indirect on novelty ($p=.016$6, $p=.016$7), cognitive demand ($p=.016$8, $p=.016$9), and structural clarity ($p=.031$0, $p=.031$1). In the constrained Handshake scenario, no dimension differed significantly between conditions, implying that narrow tasks limit scaffolding gains regardless of modality.

Temporal dynamics also diverged: improvement under Indirect scaffolding declined significantly across the three sequential tasks (Spearman's $p=.031$2, $p=.031$3), dropping from $p=.031$4 at Q1 to $p=.031$5 at Q2 before partially recovering, while Direct scaffolding remained stable over time. This suggests Indirect scaffolding is most useful early in an activity, whereas Direct scaffolding sustains performance across repeated attempts.

User-experience results favored Direct on pragmatic quality ($p=.031$6 vs. $p=.031$7, $p=.031$8) and overall UEQ scores ($p=.031$9 vs. $Z=-3.83$0, $Z=-3.83$1), but hedonic quality, ownership, agency, satisfaction, and NASA-TLX workload did not differ significantly between conditions — indicating that the reflective demands of Indirect scaffolding did not impose a measurable cognitive cost.

Qualitative findings and interpretation

Interviews revealed that participants viewed the two modalities as complementary rather than competing. One participant summarized the contrast directly: "Direct was easier to act on, but Indirect made me think more." Several participants noted that Direct scaffolding helped them progress when stuck, while Indirect scaffolding better supported metacognitive engagement; some explicitly connected their preferences to their TA experience, distinguishing what benefits themselves versus novice learners.

A notable finding concerns authorship. Because students generated their own problems before receiving scaffolding, ownership, agency, and perceived originality were preserved even under Direct scaffolding, where worked examples were described as aids for moving forward rather than replacements for students' ideas. This sequencing — student-created problem, LLM scaffold, student-led revision — appears to mitigate the passivity risk documented in prior generative-AI problem-posing studies.

The paper also reports what did not work: some participants misinterpreted Indirect guiding questions as prompts for solution-finding rather than problem-quality revision, suggesting future prompts should explicitly direct attention to structure, clarity, and cognitive demand. The Handshake scenario's narrow problem space compounded this by limiting opportunities for meaningful revision, underscoring that prompt design and task design interact and cannot compensate for each other independently.

Practical guidance

From these results the authors derive concrete recommendations for instructors: choose open-ended tasks with multiple solution paths; use Direct scaffolding when rapid, concrete revision is the goal; use Indirect scaffolding when reflection or metacognitive growth is the goal; and escalate from Indirect to Direct after roughly 2–3 attempts, since Indirect's benefit fades with repetition. The headline recommendation is to sequence the modalities — begin with Indirect scaffolding to promote reflection, then shift to Direct when learners become stuck.

Limitations and open questions

The paper concedes several constraints plainly. The sample comprised 20 graduate students, most of them TAs, whose teaching experience may limit generalizability to undergraduates. There was no no-scaffolding control condition, so improvements reflect differences between scaffolding types rather than absolute gains over unassisted revision. Data came from only two scenarios in a single session, leaving long-term learning and retention unexamined. Open questions include whether behavioral or cognitive signals can be used to dynamically personalize scaffolding modality in real time, and whether the observed effects extend to other computational contexts such as test-case design or algorithmic requirement specification.

Conclusion

This pilot study demonstrates that pedagogically distinct LLM scaffolding modalities — reflective questioning and worked-example revision — both improve the quality of computationally posed problems, but through different mechanisms and with different temporal profiles. Direct scaffolding yields larger immediate gains and higher pragmatic usability; Indirect scaffolding fosters deeper reflection at no significant additional workload cost, though its benefit attenuates across repeated tasks. The evidence supports treating the two modalities as complementary components of an adaptive scaffolding strategy rather than alternatives, contingent on the caveats of a small, expert sample and the absence of a baseline condition.