---
title: 'OtterDataset: Bug Detection in Notebooks'
url: https://www.emergentmind.com/topics/otterdataset
type: topic
---

# OtterDataset: Bug Detection in Notebooks

Searching arXiv for the cited papers to ground the article and disambiguate the term.
{"query":"2606.23877 JupOtter OtterDataset", "max_results": 5}
{"query":"2505.18303 optical microrobot dataset OTMR", "max_results": 3}
{"query":"2503.03734 OTTER Vision-Language-Action dataset DS-PnP DS-ALL", "max_results": 3}
{"query":"2105.13710 OTTers One-turn Topic Transitions", "max_results": 3}
{"query":"2510.00652 OTTER Open-Tagging Otter dataset", "max_results": 3}
OtterDataset is a labeled corpus of Python Jupyter notebooks introduced as the central dataset contribution of JupOtter for supervised cell-level bug detection. It was created to address the absence of public notebook datasets with fine-grained labels identifying which code cells are buggy, in contrast to earlier resources that were unlabeled, targeted other notebook properties, or supported only file-level analysis. In OtterDataset, each notebook carries both a notebook-level bug label and a per-cell binary label sequence, so the dataset supports localization of implementation bugs to individual code cells rather than only coarse file-level detection [2606.23877].

## 1. Definition and scope

OtterDataset is designed around a notebook-specific formulation of bug detection. A Jupyter notebook is treated not as a flat source file but as an ordered sequence of executable code cells, often interleaved with markdown and subject to non-linear execution histories. The dataset therefore targets **cell-level bug prediction**, where the goal is to predict, for each code cell, whether that cell contains an implementation bug, while also deriving a notebook-level label indicating whether the notebook contains at least one buggy cell [2606.23877].

Its scale is substantial for this setting. After collection, cleaning, labeling, and deduplication, OtterDataset contains **21,303 total notebooks**, of which **9,937** are buggy and **11,366** are non-buggy. At cell granularity it contains **661,909 total cells**, including **43,789 buggy cells**, yielding a **7%** buggy-cell ratio and an average of **2.1 buggy cells per notebook**. The paper also reports a **mean lines of code per cell** of **6.43**, which supports the authors’ claim that cell-level localization is materially more actionable than notebook-level warning alone [2606.23877].

The dataset’s held-out test portion, after tokenization-related processing, contains **4,113 files**, **2,006 buggy files (49%)**, **124,364 cells**, and **8,598 buggy cells (7%)**. The discrepancy between a nominal 20% split of 21,303 notebooks and 4,113 test notebooks is attributed in the paper to tokenization, truncation, or filtering constraints in the experimental pipeline. This suggests that the benchmark should be understood not only as a corpus release but also as a dataset-plus-representation pipeline [2606.23877].

## 2. Corpus construction and label inference

OtterDataset was collected from **GitHub** using the **GitHub Search API**. The retrieval strategy used **32 search queries** intended to provide broad topical and bug-type coverage, including statistics and machine learning notebooks and notebooks associated with common Python exception names such as **`IndexError`** and **`TypeError`**. To avoid overrepresentation from a small number of queries, the collection process imposed a cap of **1,000 notebooks per query** [2606.23877].

Deduplication and leakage control were handled explicitly. Each notebook was assigned a unique identifier combining **author name**, **repository name**, and **file name**, and duplicates across search queries were removed using that identifier. The paper additionally reports that **83.9% of test notebooks** come from **author-repository combinations not present in training**, which reduces but does not eliminate source overlap between train and test. The dataset was not assembled from version histories or buggy/fixed commit pairs, so it avoids the particular leakage mode in which a test-time buggy file might have a near-identical fixed revision in training [2606.23877].

The labeling pipeline is based primarily on **stored outputs already embedded in notebook files**, not on large-scale re-execution. For each executed code cell, the pipeline inspects the cell’s output field and extracts the **error type**, **error message**, **cell number**, and **line number** where the error occurred. Those artifacts are then converted into binary cell labels. To capture syntax errors that might never have appeared in saved outputs, each code cell is also parsed independently with Python’s **`ast`** module; cells that raise parsing exceptions are treated as syntactically erroneous. Because notebook cells may contain IPython-specific syntax such as **`%magic`** and **`!shell`**, the preprocessing step uses IPython’s **`TransformerManager`** to normalize notebook-specific constructs before parsing [2606.23877].

The dataset excludes several error categories considered too dependent on runtime environment or user intervention: **`ImportError`**, **`FileNotFoundError`**, **`KeyboardInterrupt`**, **`SystemExit`**, and **`ConnectionError`**. **Memory errors** were considered for exclusion but retained, because the paper argues they may also arise from faulty implementation, including memory leaks or poor code design. A direct consequence of the stored-output strategy is that runtime-buggy cells that were never executed will often be labeled non-buggy. The paper identifies this as an important supervision limitation rather than a corner case [2606.23877].

Quality control combined automatic construction with targeted manual verification. A sample of **100 notebooks** was manually inspected to determine whether the search process had preferentially collected pedagogical “bug-demo notebooks”; none were found to focus explicitly on demonstrating specific bugs. For label validation, **30 notebooks** were manually checked. In **14 of the 30 notebooks**, some cells could not be executed during verification because of missing dependencies or unavailable data, so labels were assigned by manual inspection and, when possible, dummy input data. The results were **24 of 30 notebooks** with identical manual and automated labels and **22 disagreements across 970 cells (2.3%)**, with a reported **±0.94% margin of error at 95% confidence** using a Wald interval. The paper also reports a **bootstrap McNemar test** with **10,000 resamples** of size **100**; only **139/10,000 resamples (1.39%)** were significant at **\(p<0.05\)**, with median **\(p=0.5000\)** [2606.23877].

## 3. Schema and notebook representation

OtterDataset stores each notebook as a structured record with seven features. The fields are concise but semantically rich enough to support cell-level supervision, notebook-level supervision, and error-type analysis.

| Field | Description |
|---|---|
| `notebook_name` | Unique identifier combining author, repo, and file name |
| `notebook_structure` | Concatenated code cells only, with cell boundary markers |
| `is_buggy` | Binary notebook-level bug label |
| `error_locations` | List of cell and line numbers where errors occurred |
| `error_type` | Type(s) of detected errors |
| `error_message` | Detected error message(s) |
| `buggy_cells` | Per-cell binary label list |

At annotation granularity, the core target is a binary cell-label vector
\[
\mathbf{y} = [y_1, y_2, \dots, y_n], \qquad y_i \in \{0,1\},
\]
where \(y_i=1\) indicates that the \(i\)-th code cell is buggy. A notebook is labeled buggy if at least one code cell is buggy. The dataset also stores error-type metadata and error messages, but those are used analytically rather than as supervised training targets [2606.23877].

The representation used by JupOtter is notebook-aware. Only **code cells** are included in `notebook_structure`; **markdown cells are omitted**. Special tokens mark the **start and end of each cell**, cells are packed sequentially into **non-overlapping chunks**, and each chunk is limited to **2,500 tokens**. The only truncation occurs when a **single cell** itself exceeds the chunk length. Such very long cells are rare: **987 of 661,909 cells**, or **0.15%**. During training, the model uses up to the **first 4 chunks** per notebook, which still includes all code cells from **19,728 notebooks**, or **93% of the dataset** [2606.23877].

This tokenization scheme addresses a length mismatch between ordinary code models and real notebooks. The paper reports that a baseline using the original CodeT5 tokenizer and file-level classification cannot fully tokenize **84% of notebooks**, whereas JupOtter with notebook-aware chunking fails on only **7% of notebooks**. A cell-boundary-token model without multi-segment splitting also fails to fully process **84% of notebooks**. This suggests that OtterDataset is not only a labeled benchmark but also a stress test for long-context notebook modeling [2606.23877].

## 4. Task formulation and benchmark protocol

The benchmark is framed as binary classification at cell granularity, with notebook-level aggregation as a secondary evaluation mode. For each notebook, logits from all chunks are concatenated into a single per-cell prediction vector
\[
\mathbf{z} = [z_1, z_2, \dots, z_n].
\]
Training uses **BCEWithLogitsLoss**, i.e., sigmoid plus binary cross-entropy over cells:
\[
\mathcal{L} = - \frac{1}{n} \sum_{i=1}^{n} \left( y_i \log \sigma(z_i) + (1-y_i)\log(1-\sigma(z_i)) \right),
\]
with
\[
\sigma(z_i) = \frac{1}{1 + e^{-z_i}}.
\]
The paper states that this formulation ensures each cell contributes equally to the notebook’s total loss [2606.23877].

The data split uses random seed **42** with **80% training** and **20% testing**. The training set contains **46.6% buggy notebooks**, and the testing set contains **49% buggy notebooks**. The paper does not define a distinct three-way train/validation/test partition; instead, it describes validation behavior using what is elsewhere called the testing set. This implies that the release is primarily documented as an 80/20 split rather than a conventional train/validation/test benchmark [2606.23877].

Evaluation uses **precision**, **recall**, **F1**, and **accuracy**, with two aggregation schemes. In **cell-aggregated metrics**, true positives, false positives, true negatives, and false negatives are accumulated over all cells in the test set. In **file-aggregated metrics**, metrics are computed per notebook and then averaged across notebooks. The benchmark therefore permits analysis of both localization quality and coarse notebook-level screening [2606.23877].

The principal supervised models are **JupOtter-small**, built on **CodeT5-small (60M parameters)**, and **JupOtter-base**, built on **CodeT5-base (220M parameters)**. Baselines include **Flake8**, adapted from script-level analysis to notebooks by converting cells into a Python script and mapping lines back to cells, as well as **GPT-4o-mini** and **Gemini 3 Flash** under an incremental cell-by-cell prompting setup. For the LLM baselines, due to cost and latency, evaluation is performed on a random sample of **100 notebooks from each dataset**, totaling **7,628 cells** with a **6% buggy-cell rate** [2606.23877].

## 5. Empirical behavior and benchmark results

On the held-out **OtterDataset Test** split, the reported **cell-level** results are as follows: **Flake8** achieves **precision 0.75**, **recall 0.69**, **F1 0.72**, **accuracy 0.80**; **JupOtter-small** achieves **0.93**, **0.69**, **0.79**, **0.95**; **JupOtter-base** achieves **0.88**, **0.75**, **0.81**, **0.95**; **GPT-4o-mini** achieves **0.26**, **0.85**, **0.40**, **0.74**; and **Gemini 3 Flash** achieves **0.45**, **0.88**, **0.59**, **0.85**. Thus, **JupOtter-base** attains the best cell-level **F1 of 0.81**, while the smaller variant remains close at **0.79**. The paper further reports that the difference between the two JupOtter variants is statistically significant under a **Wilcoxon Signed-Rank Test, \(p<0.01\)** across all three benchmark datasets used in the study [2606.23877].

At **file level** on the same test split, **Flake8** reports **precision 0.61**, **recall 0.77**, **F1 0.68**, **accuracy 0.64**; **JupOtter-base** reports **0.88**, **0.67**, **0.76**, **0.80**; **GPT-4o-mini** reports **0.55**, **0.87**, **0.67**, **0.58**; and **Gemini 3 Flash** reports **0.61**, **0.88**, **0.72**, **0.66**. Even when reduced to notebook-level decisions, the cell-trained model remains strongest by **F1** [2606.23877].

The paper also analyzes error-type sensitivity. The five most frequent error categories in OtterDataset are **`TypeError`**, **`NameError`**, **`SyntaxError`**, **`ValueError`**, and **`AttributeError`**. On an external CodeParrot-based analysis filtered to notebooks containing at least one such error, **JupOtter-base** achieves **F1 0.87** for **`SyntaxError`**, **0.64** for **`TypeError`**, **0.63** for **`ValueError`**, **0.38** for **`AttributeError`**, and **0.34** for **`NameError`**. This pattern is consistent with the dataset construction process: syntax errors are directly exposed to both stored-output parsing and explicit AST checking, whereas other runtime errors are only partially observed through saved execution artifacts [2606.23877].

Ablation results further characterize the dataset’s difficulty. A baseline using standard CodeT5 tokenization plus file-level classification achieves **F1 = 0.73**, compared with **F1 = 0.76** for **JupOtter-base** at file level, but lacks cell localization and cannot fully process most notebooks. A cell-boundary-token model without multi-segment chunking scores **F1 = 0.78** versus **0.81** for the full model but still fails on **84% of notebooks**. This indicates that the dataset’s defining challenge is not merely classification but classification under long, structured, notebook-shaped contexts [2606.23877].

## 6. Terminological ambiguity, related artifacts, and limitations

The name **OtterDataset** is unusually easy to confuse with other recent “Otter” or “OTTER” artifacts on arXiv. It is distinct from **OTMR**, the **OpTical MicroRobot dataset** for microscopy-based microrobot pose and depth perception [2505.18303]; from **DS-PnP** and **DS-ALL**, the real-robot demonstration datasets used by the vision-language-action model **OTTER** [2503.03734]; from **OTTers**, the dialogue dataset for one-turn topic transitions [2105.13710]; and from the proprietary multimodal **Otter** benchmark for hybrid predefined-tag and open-tag prediction [2510.00652]. The term is therefore most precisely reserved for the Jupyter-notebook bug-detection dataset introduced in JupOtter [2606.23877].

Several limitations are explicit. First, runtime-buggy cells that were never executed are often labeled non-buggy because the pipeline relies on stored outputs. Second, the labels are primarily automatically inferred, with manual verification on samples rather than exhaustive human annotation. Third, **markdown is omitted** from the model representation, so explanatory natural language context is not part of the benchmark input. Fourth, the exclusion of **`ImportError`**, **`FileNotFoundError`**, **`KeyboardInterrupt`**, **`SystemExit`**, and **`ConnectionError`** makes the corpus cleaner for implementation-bug detection but less representative of the full space of notebook failures. Fifth, the **4-chunk** training cap is a hardware-driven constraint, and the paper notes that files are considered buggy only if they contain an error within the first four **2,500-token** chunks in that setup [2606.23877].

Taken together, these properties position OtterDataset as a specialized benchmark for **notebook-aware, cell-localized bug detection** rather than a general corpus of all notebook failures. Its main technical contribution is not only the scale of the labeled notebook collection but also the coupling of that collection to a representation regime that preserves cell boundaries and supports long-context processing. A plausible implication is that future work using OtterDataset will need to treat notebook structure, partial observability, and chunked inference as first-class aspects of the problem rather than incidental implementation details [2606.23877].

Source: https://www.emergentmind.com/topics/otterdataset