TaskTracker: Activity & Drift Analysis Toolkit

• TaskTracker is an open‐source toolkit that systematically records fine‐grained programming activity and monitors LLM task drift.
• It features an IDE plugin, centralized server, and visualization suite to capture and analyze keystroke-level events in educational settings.
• The LLM module leverages activation delta extraction and probing mechanisms to robustly detect adversarial task shifts in RAG workflows.

TaskTracker refers to two distinct open-source toolkits broadly aimed at the systematic recording, analysis, and inspection of fine-grained activity data—one in the context of student code authoring in IDEs (Lyulina et al., 2020), and the other in the introspective detection of task drift in LLMs in retrieval-augmented generation (RAG) workflows (Abdelnabi et al., 2024). While these domains differ, both systems share a focus on capturing detailed process snapshots and supporting high-resolution behavioral or model inspection at scale.

1. TaskTracker for Programming Activity: System Architecture

The TaskTracker-tool for programming education is structured around three primary components: an IntelliJ-based IDE plugin, a central collection server, and a post-processing/visualization toolkit (Lyulina et al., 2020).

• IDE Plugin: Deployed in PyCharm, IntelliJ IDEA, or CLion, the plugin registers file edit events and hooks into the Activity Tracker utility to collect both code-change and IDE interaction events. Every text modification (including single-character edits) of predefined “task files” triggers a snapshot, capturing the file’s entire content at that instant. Concurrently, Activity Tracker records granular IDE commands (e.g., RUN, DEBUG, COPY, PASTE).
• Server: Acts as the RESTful endpoint that serves task metadata and localization data over HTTPS, and receives per-task data uploads. Data are organized in a hierarchical directory by anonymized user UUID and task, and are provided as raw CSVs with optional demographics in JSON.
• Post-processing Suite: Implemented in Python or Kotlin, this standalone pipeline supports merging, scoring (running code against test suites), snapshot filtering, and a suite of visualization modules including annotated action plots, score dynamics heatmaps, and histograms.

This modular architecture enables extensibility, robustness against data corruption, and efficient scaling across cohorts and classrooms.

2. Types of Recorded Events and Data Artifacts

TaskTracker’s instrumentation yields two synchronized data streams:

• Code Snapshots (snapshots.csv): Each entry includes a UTC ISO8601 timestamp, full file content (escaped multi-line), and length in characters. Snapshots are appended strictly on every IDE-level file edit event, supporting extremely fine-grained temporal resolution.
• IDE Interaction Events (ide_events.csv): Each record comprises timestamp, event_type (e.g., RUN, DEBUG), and action_detail metadata (such as configuration names or caret position). These events are filtered, deduplicated, and chronologically ordered.

Supplementary metadata (demographics) are logged in demographics.json per user once per session, including age, gender, country, and years_of_experience.

3. Data Storage, Supported Tasks, and Dataset Properties

On the backend, TaskTracker uses a simple yet robust file organization scheme:

$S_{t+1}$5

This allows independent, atomic retrieval of all activity data per (user, task), facilitating parallel analyses and secure anonymization. Out-of-the-box, TaskTracker supports Java, Kotlin, Python, and C++ tasks. Multi-language support is managed at the project and directory level, preserving language-specific behavior, with extension possible via additional test suite definitions and file watchers.

The published dataset aggregates 148 participants (age 11–40, mean 19, programming experience 0–6+ years) completing six “toy” tasks. It contains 474 submitted solutions (326 correct, 148 incorrect) with comprehensive per-language breakdowns (Python: 328, Java: 90, Kotlin: 48, C++: 8, total) and several thousand code snapshots and IDE event logs. Every keystroke triggers a new recorded snapshot, yielding temporally dense trace data.

4. Metrics, Analyses, and Visualization Paradigms

The toolkit’s design prioritizes process-oriented metrics. Key analyses include:

• Edit Distance Between Snapshots: For consecutive snapshots $S_t$, $S_{t+1}$ of lengths $|S_t|$, $|S_{t+1}|$,

$$d(S_t,\,S_{t+1}) = \sum_{i=1}^{\max(|S_t|,|S_{t+1}|)} \mathbf{1}[S_t(i) \neq S_{t+1}(i)]$$

• Active Programming Time: For keystrokes at $t_1, t_2, ..., t_n$, sum intervals with adjacent timing below a threshold (e.g., 5 s):

$$T_{\mathrm{active}} = \sum_{j=1}^{n-1} (t_{j+1} - t_{j}) \quad \text{if } t_{j+1} - t_{j} < \theta$$

• Other Measures: code churn (sum of characters inserted or deleted), long/unusual pause durations, refactoring ratio (snapshots with identical correctness score and distinct code), and feature-use profiles (debug/run/navigate per time unit).

Visualization modules generate:

• Line plots of file length over time (annotated with specific IDE actions).
• Score dynamics heatmaps (mapping snapshots × time with score-encoded color).
• Participant/task histograms (completion times, event counts, etc.).

These facilities support both individual and aggregate behavioral analysis.

5. TaskTracker for LLM Task Drift Detection: Methodology and Architecture

In the context of LLM task drift detection, TaskTracker refers to a toolkit and methodology for identifying deviations between a user’s intended instruction and the LLM’s response when external (“retrieved”) data is appended (Abdelnabi et al., 2024). The core concept is the “activation delta”—the difference in internal activations (hidden states) at selected model layers after ingesting the external context.

• Prompt Template: All inputs are wrapped with a standardized template $T(\cdot)$ to control prompt structure and facilitate robust activation comparison.
• Activation Extraction: The hidden state at the final token position is recorded at each selected transformer layer for both the “primary-only” input $x_{\mathrm{pri}}$ and the combined “full” input $x = (x_{\mathrm{pri}} \Vert \text{data block})$.
  • $S_{t+1}$0
  • $S_{t+1}$1
  • The residual is $S_{t+1}$2
• Probing Mechanisms:
  • Linear Classifier: Logistic regression on the concatenated activation delta vectors from key layers, trained on “clean” vs. “poisoned” pairs.
  • Metric-Learning Triplet Probe: A triplet network (per-layer 1D convolutions, concatenation, linear → 1024D $S_{t+1}$3-normalized) trained such that “clean” activations cluster near the anchor, “poisoned” drift apart, optimized by standard triplet loss with semi-hard/hard mining.

No LLM fine-tuning is required, and only white-box forward passes are needed at inference.

6. Dataset Scale and Evaluation Protocols

The LLM TaskTracker dataset comprises approximately 836 K training pairs, with balanced validation and test sets (clean vs. poisoned). Task sources are broad: SQuAD, SEP generic NLP, Alpaca, Code Alpaca, AdvBench, BIPA, TrustLLM, JailbreakBench. “Poisoned” data includes prompt-injection/jailbreak attempts and adversarially crafted triggers.

Four LLMs—Phi-3 3.8B, Mistral 7B, Llama-3 8B, Mixtral 8×7B—are instrumented for activation extraction. Evaluation centers on ROC AUC for clean-vs-poisoned discrimination. The linear probe achieves ROC AUC ≥ 0.99, with the triplet approach yielding 0.93–0.99 (robustness dependent on layer choice and noise).

Temporal analysis of activation distance trajectories enables pinpointing the onset of injected instructions, as $S_{t+1}$4 (embedding distance over tokens) reveals characteristic spikes.

7. Research Applications, Deployment, and Released Tooling

TaskTracker enables a spectrum of research directions and applications:

• In Programming Education: Studies of IDE feature adoption versus experience, debugging/refactoring behavior post-solution, error pattern clustering, personalized hint generation, adaptive curriculum analysis (Lyulina et al., 2020).
• In LLM Task Drift: Zero-shot generalization to emerging attack types not seen during training (prompt injection, jailbreaking, adversarial question insertion). The probe approach is harder to subvert than text-based detectors, robust to evasive prompt engineering.

Both toolkits are released open-source, with TaskTracker for LLMs available at https://github.com/microsoft/TaskTracker. Key modules include loaders for precomputed activations, extraction routines, probing and evaluation utilities, and code for end-to-end integration in RAG pipelines or coding classroom experiments.

A plausible implication is that the “activation delta” paradigm could generalize to other contexts where model introspection is critical, including security, explainability, and behavioral profiling tasks.

---

References:

• "TaskTracker-tool: a Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks" (Lyulina et al., 2020)
• "Get my drift? Catching LLM Task Drift with Activation Deltas" (Abdelnabi et al., 2024)