Design & Development Research
A Type 1 design and development research study for a Scaffolded LLM Tutoring Design Framework
The study
Intelligent tutoring systems deliver measurable gains, but building one demands infrastructure and technical depth out of reach for most individual designers. Tutoring built on large language models is far more accessible, yet it has lacked a structured framework connecting prompt design to learning theory. This study is a formative validation of the Scaffolded LLM Tutoring Design Framework (SLTDF): a design model that translates established learning science into auditable tutoring behaviors that can be prompt‑engineered into an LLM.
The framework's first instantiation is the Math Tutor GPT (v4.1), the latest of eight development iterations, built for community college students preparing for math placement testing. Eight expert reviewers, selected through purposive maximum‑variation sampling across AI/LLM prompting, mathematics education, and instructional design, evaluated the Tutor through seven scenario‑based interactions.
Three ways in
The project is presented in three formats: a recorded presentation, a 16‑page interactive graphic novel that retells the study, and the full written report.
A guided walkthrough of the study: the problem, the framework, the expert review, and where the Tutor goes next.
The complete study: theoretical framework, methodology, instrument, per‑behavior results, discussion, and the five design principles, with full reviewer response data in the appendices.
What the experts found
Expert reviewers
8
Maximum-variation sample; math education, instructional design, AI/LLM design
Behaviors reliably present
6 of 7
Judged fully or partially exhibited by ≥75% of reviewers
Grand mean appropriateness
3.02
On a 1–4 scale, across all seven behaviors
Moderate-or-high fidelity
87.5%
Of reviewers, in overall design fidelity ratings
Behavioral appropriateness by tutoring behavior
Mean expert rating on a 1–4 appropriateness scale (n = 8; n = 7 for spaced retrieval)
The scaffolded hint ladder was the most reliably present behavior (100% of reviewers), yet its appropriateness rating shows presence and quality are not the same thing: hints sometimes revealed answers instead of guiding toward them. Spaced retrieval is the one clear fidelity gap.
| Behavior | n | Mean | SD | Median | % fully exhibited |
|---|---|---|---|---|---|
| B7 · Motivational framing | 8 | 3.50 | 0.76 | 4.0 | 87.5% |
| B3 · Worked examples with fading | 8 | 3.25 | 0.89 | 3.5 | 75.0% |
| B4 · Mastery check gating | 8 | 3.25 | 1.04 | 4.0 | 62.5% |
| B1 · Diagnostic questioning | 8 | 3.13 | 0.64 | 3.0 | 50.0% |
| B2 · Scaffolded hint ladder | 8 | 2.88 | 0.83 | 3.0 | 100% |
| B6 · Error classification & remediation | 8 | 2.88 | 1.13 | 3.0 | 62.5% |
| B5 · Spaced retrieval prompts | 7 | 2.29 | 0.95 | 2.0 | 12.5% |
Reviewers converged on clear refinement priorities: hint scaffolding that occasionally gave answers away (flagged by five of eight reviewers), spaced retrieval prompts that rarely surfaced, and error feedback that treated conceptual and procedural mistakes the same. Each is logged for the next Tutor revision, with spaced retrieval first in the modification triage.
What the field can take away
Stated in the Richey and Klein format for design and development research, each principle pairs a design move with its theoretical rationale and the evaluation evidence behind it.
Scaffolded hint ladders should withhold answer‑revealing structure until the learner has attempted the problem: Vygotskian scaffolding requires productive struggle within the zone of proximal development.
Evidence: five of eight reviewers flagged hints that handed over the answer (hint‑ladder M = 2.88).
Spaced retrieval cannot rely on the model remembering prior content. The system should engineer explicit session state, because the spacing effect depends on deliberate reintroduction of material over time.
Evidence: spaced retrieval rated fully exhibited by only 12.5% of reviewers and absent by 62.5% (M = 2.29).
Tutor feedback should classify each error as conceptual, procedural, or careless before responding, and route to type‑specific remediation: process‑level feedback yields larger gains than uniform correction.
Evidence: three of eight reviewers flagged undifferentiated error feedback (M = 2.88).
Self‑explanation prompts should require learner input before the tutor proceeds: the self‑explanation effect depends on active generation by the learner, not a rhetorical pause.
Evidence: three of eight reviewers reported strategy checks continuing without waiting for a response.
Designers should produce a written behavior specification before prompt engineering begins: Type 1 fidelity assessment requires a design contract mapping each principle to an auditable behavior.
Evidence: overall fidelity of M = 3.00, moderate but not full, with several flagged issues tracing to implicit specification.
The framework
The SLTDF is a three‑tier model. It synthesizes ten learning science constructs into seven operationalizable tutoring behaviors, then acts as a session manager deciding which behaviors surface, and which recede, at any moment in a chat based on the learner's actions and session state.
The theoretical base
Synthesizes the constructs and arbitrates which behaviors are evidenced or reduced, turn by turn
The auditable design contract evaluated in this study
Type 1 research asks whether the design is adequate, not yet whether learners gain from it. This study contributes the first piece of formative evidence that the framework's behaviors are recognizable to experts as theoretically grounded, and it maps the road to the learner‑facing research that follows.