Data-Driven Classroom Interviews

• Data Driven Classroom Interviews (DDCIs) are a method that integrates real-time learning analytics with brief, targeted in-situ interviews to capture critical learning moments.
• The approach uses automated behavioral and affective detectors and the open-source Quick Red Fox system to trigger ultra-short interviews (3–5 minutes) immediately after significant classroom events.
• DDCIs bridge quantitative log analysis with qualitative student insights, validating educational inferences and informing adaptive teaching practices.

Data Driven Classroom Interviews (DDCIs) are a methodological innovation for educational research that leverage real-time learning analytics in digital environments to coordinate brief, targeted, in-situ interviews with students. By integrating automated behavioral and affective detection systems with analytic tools, DDCIs enable the efficient capture of students’ reasoning, cognitive strategies, and emotional states at precisely those moments deemed pedagogically or scientifically significant, while minimizing classroom disruption. The approach is operationalized through an open-source software suite known as Quick Red Fox (QRF), which automates trigger detection, interviewer notification, and workflow management within authentic classroom settings (Ocumpaugh et al., 17 Nov 2025).

1. Rationale and Theoretical Foundations

DDCIs are motivated by the need to contextualize and validate log- and model-based inferences in learning analytics. While educational logs and detectors can flag episodes of interest (e.g., error bursts, signs of disengagement), they lack access to the student’s subjective perspective. DDCIs provide a means to bridge this gap by eliciting qualitative, self-reported accounts synchronized to those “interesting” events. The technique is informed by:

• Research on affective dynamics using BROMP detectors and frameworks for capturing cycles of boredom, confusion, and concentration (D’Mello & Graesser, 2012).
• Interest development frameworks, which distinguish situational from individual forms of interest, probed during tool use or social interaction (Hidi & Renninger, 2010).
• Microethnography and design-based research traditions, which use iterative fieldwork to refine both learning technologies and research instruments in situ (Au & Mason, 1983; Design-Based Research Collective, 2003).
• Addressing the “one-shot” and “needle-in-haystack” problems: DDCIs facilitate the paper of rare, critical learning events (e.g., spontaneous help-seeking after repeated errors).

2. Distinctive Features and Methodological Principles

DDCIs differ fundamentally from traditional interview methods:

• Temporal Precision: Interviews are initiated immediately (typically within 1–2 minutes) following a system-detected event of interest, in contrast to post-session debriefs.
• Economy of Time: Individual interviews are ultra-short (3–5 minutes, rarely exceeding 10), reducing disruption to teaching and maximizing interviewer coverage.
• Ecological Validity: DDCIs are conducted in authentic classroom environments, not in laboratory or pull-out settings, preserving naturalistic student behavior.
• Analytic Triggering: Interviews are initiated based on automated triggers—pre-specified by the research team—using detectors for behavior, self-regulated learning (SRL), or affect, thus reducing observer bias and idle time.

A central objective is to elicit student reasoning, feelings, and strategies “in moment,” situated within the flow of digital learning activity. Typical questions might include “Why did you delete that map node?” or “How did you decide to use that tool?” to directly probe the inferred process behind detected events.

3. Quick Red Fox System Architecture

The QRF architecture has several interlocking components optimized for continuous, scalable deployment in classrooms:

Component	Function	Key Technologies
Learning System (Client)	Logs student actions, runs custom detectors (affect, SRL, behavior)	Integrated into digital platform
QRF Polling Script (Python)	Pulls logs at configurable intervals, applies trigger rules	Python, triggered every ~10s
QRF Dispatcher (Server)	Receives triggers, applies prioritization/cooldown, dispatches to app	WebSocket/Firebase API
Firebase Realtime DB	Stores triggers, student IDs, metadata, notes, audio	Firebase (cloud)
QRF Android App	Receives/display triggers, records interviews, logs notes	Android, WebSocket-secured
QRF Dashboard (Web)	Visualizes triggers, notes, and interviews for monitoring	Web interface

The polling script applies rules such as:

while True: data = pull_student_logs() for student in data: if data[student].error_count > ERROR_THRESHOLD: triggers.append((student, 'high_error_rate')) send_triggers(triggers) sleep(POLL_INTERVAL)

Triggers are prioritized and dispatched according to pre-set rules relating to frequency, recency, and alignment with core research questions. Expiration and per-student cooldown parameters ensure interviews remain timely and non-disruptive.

4. Trigger Design and Prioritization Logic

Triggers define the circumstances in which interviews are launched. They must be tightly aligned with research objectives, operationalized using analytic or detector pipelines embedded in the digital platform. Triggers may be:

• Threshold-based: Quantitative cut-points set on observed behaviors (e.g., trigger if a student places more than $\theta$ blocks in a defined window, where $\theta$ reflects the 95th percentile of historical data).
• Probabilistic: Trigger fires if $$P(\text{target\_behavior} \mid \text{features}) > p_0$$, as determined by logistic regression or classifier output.
• Event-structured: Social or tool-use events (e.g., distance from peers, rapid tool repeats).

The system enforces constraints on latency (targeting $<$1s for delivery), expiration (triggers removed after e.g., 2 minutes), and cooldown (minimum time $\Delta_{\text{cool}}$ between interviews with the same student). Prioritization among live triggers is performed as:

$$\text{Priority}(T_j) = w_1\frac{1}{\text{freq}(T_j)} + w_2\mathbb{I}(\text{aligned to core Q}) + w_3\frac{1}{\text{time\_since}(T_j)}$$

where $w_1, w_2, w_3$ are weights set by the research team.

5. Interview Protocol and Best Practices

The DDCI protocol is governed by a “Big Sister” interviewing stance—friendly and non-authoritative, avoiding teaching or grading, to promote rapport and candor. The interviewer workflow follows:

1. Greet the student by username, establishing a supportive tone.
2. Use an open-ended, student-centered ice-breaker (e.g., “How’s your day going?”), then pivot to the event trigger.
3. Employ minimal encouragers (“Mmm?”, “Oh!”) and reflective echoing to elicit elaboration.
4. Record a focused 3–5 minute conversation, using the dedicated audio recording function in the QRF app.
5. Conclude with gratitude, maintaining rapport for subsequent interactions.

Researchers are advised to explain the process to students, emphasize voluntary participation (assent), and demonstrate recording equipment. Interviews should be conducted promptly ($<$2 min after trigger), be kept brief, and skipped if the student declines, is distressed, or is deeply engaged in a critical task.

6. Data Handling, Transcription, and Analysis

Collected data include audio, metadata (trigger details, timestamps, student IDs), and interviewer notes. Extraction is performed via USB or secure transfer. Transcription protocols are specified to capture pauses, overlap, slang, jargon, and de-identification, ensuring analytic fidelity.

Analysis follows an iterative deductive-inductive coding process, with structured codebook development and reliability assessment. Interrater reliability (IRR) is quantified via:

• Cohen’s $\kappa$ for binary coding: $\kappa = \frac{p_o - p_e}{1 - p_e}$, with a target of $\kappa \geq 0.8$.
• Krippendorff’s $\alpha$ for multi-rater, mixed type data.

Ambiguous codes are resolved through social moderation and re-coded for reliability. Quantitative analyses include:

• Correlations $(\rho)$ between coding categories and quantitative measures,
• Ordered Network Analysis (ONA) for transitions among codes/behaviors,
• Sequential Pattern Mining (SPM) for identifying frequent code/rule sequences,
• Mixed-effects models linking code frequencies to learning outcomes, e.g., $$\text{Outcome}_i = \beta_0 + \beta_1 \,\#\text{explanations}_i + u_{\text{student}_i} + \epsilon_i$$.

7. End-to-End Fieldwork Procedure

Implementation proceeds in the following stages:

1. Planning and Integration: Define research questions, simulate and select triggers, configure QRF components (server, dashboard, app).
2. Pre-Fieldwork Training: Prepare interviewers with Big Sister/assent training, mock interviews, and technical system checks.
3. Field Deployment: Live test equipment, brief students, obtain assent, monitor dashboard, conduct interviews, and adjust protocols as necessary.
4. Post-Fieldwork & Data Processing: Extract and transcribe data, code interviews, merge with logs/surveys via IDs and timestamps.
5. Analysis and Reporting: Conduct descriptive/correlational/sequential analyses, visualize with ONA/sequence charts, evaluate method efficacy (interviewer idle time, event coverage, IRR), and reflect on methodological performance and improvement (Ocumpaugh et al., 17 Nov 2025).

By systematically integrating learning analytics with in-situ qualitative inquiry, DDCIs provide a scalable, theory-anchored protocol for uncovering student reasoning and affect at critical moments, with the potential to inform both basic research and the iterative design of educational technologies.