---
title: 'RIOT: Real-time Instructor Observing Tool'
url: https://www.emergentmind.com/topics/real-time-instructor-observing-tool-riot
type: topic
---

# RIOT: Real-time Instructor Observing Tool

Searching arXiv for RIOT and closely related observation-tool papers to ground the article in the cited literature.
The Real-time Instructor Observing Tool (RIOT) is a computerized real-time instructor observation tool developed to systematically observe, document, and quantify instructor-student interactions in active-learning science courses, particularly the CLASP (Collaborative Learning through Active Sense-making in Physics) introductory physics course at UC Davis [1212.1494]. Its stated purposes are both research and professional development: research, by describing and illustrating pedagogical variation among instructors implementing a common interactive-engagement curriculum, and professional development, by supporting training and reflection on enacted teaching practice [1212.1494]. Subsequent work used RIOT to characterize teaching assistant pedagogies in traditional laboratories, recitations, labs, and workshops, extending its role as a fine-grained observational protocol for comparing instructional settings and documenting default instructional behavior in the early stages of reform [1602.07740] [2509.09136].

## 1. Origins and instructional context

RIOT emerged from efforts to study physics instruction at UC Davis for life science majors in a long-standing reformed large-enrollment physics course in which discussion/lab instructors, primarily graduate student teaching assistants, implement the interactive-engagement elements of the course [1212.1494]. Because many different instructors participate in disseminating the course elements, the developers considered it essential to observe and document student-instructor interactions within the classroom [1212.1494].

The motivating premise was that even in a standardized interactive-engagement curriculum, instructors can have widely different classroom practices, which may influence student experience and outcomes [1212.1494]. RIOT was therefore designed not merely as a descriptive checklist, but as a moment-by-moment instrument capable of capturing sustained classroom interaction patterns with sufficient granularity to compare instructors teaching identical curricular activities across different sections [1212.1494].

Later studies preserved this comparative function while shifting the instructional setting. One study at San José State University used RIOT to understand what graduate teaching assistants may “default to” with little to no intervention in introductory physics labs and workshops [1602.07740]. A further replication-and-expansion study used RIOT to observe graduate student teaching assistants facilitating introductory physics labs and recitations, with the explicit goal of comparing those settings to the CLASP curriculum studied by West et al. [2509.09136]. This suggests that RIOT became not only a local observation instrument for UC Davis, but also a transportable protocol for cross-context analysis of pedagogical enactment.

## 2. Operational design and real-time logging

RIOT operates through real-time, continuous observation of the instructor during class, with the observer recording the type of interaction taking place at each moment [1212.1494]. In the UC Davis implementation, observers used a computer interface to record both what the instructor was doing and with whom: individual, group, or whole class [1212.1494]. The interface is described as a color-coded grid of buttons on a laptop [1212.1494]. In SJSU deployments, RIOT likewise coded the interaction type, who was involved, start time, and duration with one-second resolution [1602.07740].

A central design choice is that each instructor can only be coded as engaging in one interaction type at a time [1212.1494]. In the original study, observers implemented a self-imposed 10-second delay before coding a shift in interaction type, so that actions less than 10 seconds were ignored on the premise that only sustained behaviors impact classroom culture [1212.1494]. By contrast, later descriptions emphasize second-level or one-second resolution in logging every change in instructor behavior by interaction type and time [2509.09136] [1602.07740]. A plausible implication is that the protocol’s temporal precision can be adapted across studies while preserving the same categorical framework.

The data structure is explicitly described as storing each event with a timestamp, category, and interaction target, effectively of the form $(\mathrm{Time}, \mathrm{Action}, \mathrm{Target})$ [1212.1494]. The observer focused only on actions performed by or with the teaching assistant; student activity independent of the TA was not coded [1602.07740]. Resulting logs support color-coded graphs and aggregated breakdowns for comparing data across instructors, sections, and time periods [1212.1494].

## 3. Coding architecture and interaction taxonomy

RIOT differentiates four major interaction types, each subdivided for specificity, and each may occur with different targets such as individuals, groups, or the whole class [1212.1494]. The same broad scheme is retained in later studies [1602.07740] [2509.09136].

| Major category | Subcategories | Core focus |
|---|---|---|
| Talking at Students | Explaining; Clarifying Instructions | Instructor-centered talk |
| Dialoguing with Students | Listening to Question; Engaging in Closed Dialogue; Engaging in Open Dialogue; Open Dialogue with Ideas Being Shared / Ideas Being Shared | Conversational interaction |
| Observing | Passive Observing; Active Observing; Student Presentation / Students Presenting; Students Talking Serially | Monitoring or listening |
| Not Interacting | Administrative/Grading; Working on Apparatus or Material; Chatting; Class Prep/Reading Notes; Out of Room | Non-interactive activity |

Within **Talking at Students**, “Explaining” or “Explaining Content” refers to explaining physics concepts, answers, or processes, whereas “Clarifying Instructions” refers to reviewing instructions, logistics, transitions, or procedural guidance [1212.1494] [1602.07740] [2509.09136].

Within **Dialoguing with Students**, “Listening to Question” captures the instructor attending to a student’s query [1212.1494]. “Engaging in Closed Dialogue” denotes instructor-led, short-answer questioning or scaffolded questioning leading students to correct answers [1212.1494] [2509.09136]. “Engaging in Open Dialogue” denotes extended conversation in which students contribute complete sentences, though not necessarily making sense of ideas [1212.1494]. “Open Dialogue with Ideas Being Shared,” or in later formulations “Ideas Being Shared,” marks student-directed or student-led discussion in which concepts are actively developed, challenged, and discussed [1212.1494] [1602.07740].

Within **Observing**, RIOT distinguishes “Passive Observing,” such as scanning the room or briefly checking student work, from “Active Observing,” such as intently listening to a group or individual [1212.1494] [2509.09136]. Additional categories include “Student Presentation” or “Students Presenting,” and “Students Talking Serially,” where students build on one another’s ideas with the instructor listening [1212.1494] [1602.07740].

Within **Not Interacting**, RIOT codes administrative work, grading, class preparation, chatting, apparatus-related help devoid of physics discussion, and absence from the room [1212.1494] [1602.07740]. The original UC Davis study also notes that categories such as “Chatting” and “Ideas” were added mid-study when the initial coding scheme proved insufficient for some observations [1212.1494]. This suggests that RIOT’s taxonomy is stable enough for comparison but capable of refinement when field use reveals missing distinctions.

## 4. Implementation, observation procedure, and reliability

The original RIOT software was programmed as a custom application using File Maker Pro database software scripts, enabling quick and reliable time-stamping and categorization in real time [1212.1494]. Its implementation centered on rapid event entry rather than post-hoc coding, which was intended to support continuous observation during live instruction [1212.1494].

In the UC Davis study, two researchers performed all observations, and prior to data collection they established inter-rater reliability of 85–95% through joint observations and agreement on category definitions, with ongoing calibration during the study [1212.1494]. Each of 29 different instructors, mostly graduate teaching assistants and some faculty, was observed twice for five hours total over one quarter, for about 150 hours of class time, typically coding a full 135–140 minute discussion/lab period per observation [1212.1494].

At SJSU, RIOT was used as the primary observational protocol over one semester to observe five out of eight departmental teaching assistants, each observed at least twice, mostly in Physics 50 and 50W [1602.07740]. Reliability was reported as 88.5% agreement between two trained observers on parallel sessions, with Cohen’s $\kappa = 0.825$ [1602.07740]. The same $\kappa$ value appears in the later labs-and-recitations study, where inter-rater reliability was ensured and characterized as “almost perfect” agreement [2509.09136].

The 2016 study provides the reliability equations in LaTeX form [1602.07740]:

$$
N_\mathrm{agreement} = \sum\limits_i^n a_{ii}
$$

$$
N_\mathrm{chance} = \frac{\sum\limits_i^n\left(\sum\limits_j^n a_{ij}\sum\limits_j^n a_{ji}\right)}{\sum\limits_i^n\sum\limits_j^n a_{ij}}
$$

$$
\kappa = \frac{N_\mathrm{agreement} - N_\mathrm{chance}}{\sum\limits_i^n\sum\limits_j^n a_{ij} - N_\mathrm{chance}}
$$

The 2025 study also states that the total number of minutes devoted to each interaction type was summed across observations and gives the percent-time calculation as [2509.09136]:

$$
\text{Percent Time} = \frac{\text{Minutes in Interaction Category}}{\text{Total Observed Minutes}} \times 100\%
$$

These details position RIOT as a quantitative observational protocol with explicit reliability procedures and a time-based aggregation model rather than as a purely qualitative field-note system.

## 5. Empirical findings on pedagogical variation

The foundational RIOT study reported that the range of instructor behaviors was more extreme than previously assumed, even though instructors worked within the same interactive-engagement curriculum and received common professional development and instructor notes [1212.1494]. Variation appeared in both the allocation of time between Small Group and Whole Class Discussion and in the types and frequencies of interactions [1212.1494].

In that study, some classes spent as little as 10 and as much as 91 minutes of a 135–140 minute session in whole-class format [1212.1494]. During small-group time, minutes spent “explaining” varied from 0 to 28, while dialoguing varied from 1.2 to 77 minutes [1212.1494]. Proportionally, time spent in “dialogue” ranged from 2.7% to 63% of small-group time [1212.1494]. The authors also found that instructors who interacted similarly during small-group time did not necessarily do so in whole-class discussion and vice versa [1212.1494].

No single “best practice” was identified in the original RIOT paper [1212.1494]. The curriculum discouraged excessive “explaining,” yet some instructors predominantly lectured in both small-group and whole-class contexts [1212.1494]. The study argued that instructors interpret and enact the interactive-engagement philosophy differently, influenced by their prior experiences and teaching models [1212.1494]. This suggests that a common curriculum does not by itself normalize enacted pedagogy.

The later SJSU work extended these findings by documenting default TA practices in less reformed settings. Across all courses in the 2016 study, less than a third of observed time was “talking at” students, about 17% was “talking with” students or observing them, and about 45% of time TAs were “Not Interacting” [1602.07740]. In Physics 50, about 55% of observed time was “Not Interacting,” and observing students occupied very little time [1602.07740]. In Physics 50W, there was greater variety in interaction types, but still about 30% “Not Interacting,” with considerable variation between TAs [1602.07740].

The 2025 study similarly confirmed large variation between TAs’ interactions during recitation sessions, while finding that TAs facilitating traditional labs displayed fairly similar interaction profiles to each other [2509.09136]. The amount of time instructors spend observing students was identified as a key distinguishing characteristic between traditional settings and the CLASP curriculum [2509.09136].

## 6. Comparative use across instructional settings

RIOT has been used not only to document within-course variation but also to compare instructional environments that differ in reform status, structure, and pedagogical intent. The 2025 study is especially explicit in using RIOT to provide a detailed, moment-by-moment, and quantitative profile of instructor-student interactions in traditional labs and recitations and to allow direct comparison with prior studies of active-learning environments, notably the CLASP curriculum [2509.09136].

In traditional lab settings, TAs spent a majority of observed time not interacting with students: 60.5% of total observation time [2509.09136]. “Talking at Students” accounted for 27.9%, with “Clarifying Instruction” at 16% and “Explaining Content” at 11.9% [2509.09136]. “Dialoguing with Students” totaled 8.8%, and observing students was only about 3% [2509.09136]. During small-group work, 69.6% of the TA’s time was “Not Interacting” [2509.09136].

In recitation settings, the interaction profile was more balanced but remained explanation-centric: 29.8% “Not Interacting,” 33.4% “Explaining Content,” and 20.6% “Closed Dialogue” [2509.09136]. Even during small-group work, about 40.6% of small-group time was “Not Interacting” [2509.09136]. Time was almost evenly split among whole class, individuals, and small groups, at about 20–25% each [2509.09136].

By comparison, the CLASP TAs in West et al. interacted with students more than 70% of session time, allowed far more student presentation and idea sharing, spent more time observing students as they worked, and had much lower “Not Interacting” time, as low as 20% of small-group time [2509.09136]. Student presentations and deep dialogue were virtually absent in the SJSU traditional labs and recitations [2509.09136]. A plausible implication is that RIOT’s observational categories reveal not merely stylistic variation but structural differences in how learning environments distribute attention across explanation, dialogue, observation, and disengagement.

The 2016 study makes a related comparison between SJSU labs and workshops and UC Davis CLASP, reporting that SJSU TAs spent more time not interacting with students and that CLASP TAs observed more, conducted more dialogue, and facilitated more student presentations or peer discussions [1602.07740]. In this sense, RIOT functions as a bridge between local program evaluation and comparative pedagogical analysis across institutional contexts.

## 7. Significance, limitations, and relation to newer real-time systems

RIOT’s central contribution is its granularity. The original paper argues that it provides fine-grained, time-resolved, quantitative documentation of instructor actions in interactive-engagement classrooms, capturing nuances lost in standard checklists or summary protocols such as RTOP [1212.1494]. It also provides a mechanism for generating comparative visualizations that expose the spectrum of implementations among instructors sharing a uniform curriculum and training [1212.1494]. Across studies, it is presented as a reflective tool for instructors and curriculum designers and as a foundation for future correlation analyses between interaction patterns and student learning [1212.1494].

The limitations stated in the RIOT-based literature are equally important. The original study does not directly link instructor actions to student learning [1212.1494]. The 2016 and 2025 studies similarly frame their findings as snapshots of teaching practice that inform professional development and reform, rather than as causal demonstrations of effectiveness [1602.07740] [2509.09136]. Common misconceptions can therefore arise if RIOT is treated as a direct measure of instructional quality. The available studies do not identify a single “best practice” interaction profile, nor do they claim that any specific distribution of categories is universally optimal [1212.1494].

Within the provided literature, RIOT also appears as a comparator for newer real-time classroom systems. The DLOT paper cites RIOT among existing education tools and states that existing tools in education, including HART and RIOT, are either rigid in their configuration or lack key functionalities for flexible, real-time qualitative and mixed-methods research [2312.16361]. However, the same source also notes that it does not elaborate on RIOT in detail [2312.16361]. ClassAid, a 2026 real-time orchestration system for programming classrooms, states that both RIOT and ClassAid provide real-time dashboards for monitoring student progress and classroom dynamics and support identification of at-risk students and critical teaching moments using live alerts, while presenting ClassAid as extending beyond observation to per-student and class-wide AI control, student–AI interaction visibility, and automated intervention [2602.06734]. This suggests a broader trajectory in which RIOT represents an earlier generation of real-time classroom instrumentation focused on human-coded instructor-student interaction, whereas later systems incorporate orchestration, dashboards, and AI-mediated intervention.

In physics education research and related observational studies, RIOT remains notable for making classroom enactment measurable at high temporal resolution while preserving categorical distinctions that are pedagogically interpretable. Its empirical legacy is the demonstration that instructors operating under common curricular structures can create significantly different course experiences, and that those differences can be documented systematically enough to inform curriculum design, TA professional development, and comparative analysis of instructional environments [1212.1494].

Source: https://www.emergentmind.com/topics/real-time-instructor-observing-tool-riot