OtterDataset: Bug Detection in Notebooks

Updated 5 July 2026

OtterDataset is a labeled corpus of 21,303 Jupyter notebooks with detailed cell-level bug annotations for precise bug localization.
It employs a notebook-aware representation that preserves code cell boundaries and uses chunk-based tokenization for handling long contexts.
The dataset supports both file- and cell-level evaluations using metrics like precision, recall, and F1, driving advances in bug detection research.

Searching arXiv for the cited papers to ground the article and disambiguate the term. {"query":"(Ottenhof et al., 22 Jun 2026) JupOtter OtterDataset", "max_results": 5} {"query":"(Wei et al., 23 May 2025) optical microrobot dataset OTMR", "max_results": 3} {"query":"(Huang et al., 5 Mar 2025) OTTER Vision-Language-Action dataset DS-PnP DS-ALL", "max_results": 3} {"query":"(Sevegnani et al., 2021) OTTers One-turn Topic Transitions", "max_results": 3} {"query":"(Ouyang et al., 1 Oct 2025) OTTER Open-Tagging Otter dataset", "max_results": 3} OtterDataset is a labeled corpus of Python Jupyter notebooks introduced as the central dataset contribution of JupOtter for supervised cell-level bug detection. It was created to address the absence of public notebook datasets with fine-grained labels identifying which code cells are buggy, in contrast to earlier resources that were unlabeled, targeted other notebook properties, or supported only file-level analysis. In OtterDataset, each notebook carries both a notebook-level bug label and a per-cell binary label sequence, so the dataset supports localization of implementation bugs to individual code cells rather than only coarse file-level detection (Ottenhof et al., 22 Jun 2026).

1. Definition and scope

OtterDataset is designed around a notebook-specific formulation of bug detection. A Jupyter notebook is treated not as a flat source file but as an ordered sequence of executable code cells, often interleaved with markdown and subject to non-linear execution histories. The dataset therefore targets cell-level bug prediction, where the goal is to predict, for each code cell, whether that cell contains an implementation bug, while also deriving a notebook-level label indicating whether the notebook contains at least one buggy cell (Ottenhof et al., 22 Jun 2026).

Its scale is substantial for this setting. After collection, cleaning, labeling, and deduplication, OtterDataset contains 21,303 total notebooks, of which 9,937 are buggy and 11,366 are non-buggy. At cell granularity it contains 661,909 total cells, including 43,789 buggy cells, yielding a 7% buggy-cell ratio and an average of 2.1 buggy cells per notebook. The paper also reports a mean lines of code per cell of 6.43, which supports the authors’ claim that cell-level localization is materially more actionable than notebook-level warning alone (Ottenhof et al., 22 Jun 2026).

The dataset’s held-out test portion, after tokenization-related processing, contains 4,113 files, 2,006 buggy files (49%), 124,364 cells, and 8,598 buggy cells (7%). The discrepancy between a nominal 20% split of 21,303 notebooks and 4,113 test notebooks is attributed in the paper to tokenization, truncation, or filtering constraints in the experimental pipeline. This suggests that the benchmark should be understood not only as a corpus release but also as a dataset-plus-representation pipeline (Ottenhof et al., 22 Jun 2026).

2. Corpus construction and label inference

OtterDataset was collected from GitHub using the GitHub Search API. The retrieval strategy used 32 search queries intended to provide broad topical and bug-type coverage, including statistics and machine learning notebooks and notebooks associated with common Python exception names such as IndexError and TypeError. To avoid overrepresentation from a small number of queries, the collection process imposed a cap of 1,000 notebooks per query (Ottenhof et al., 22 Jun 2026).

Deduplication and leakage control were handled explicitly. Each notebook was assigned a unique identifier combining author name, repository name, and file name, and duplicates across search queries were removed using that identifier. The paper additionally reports that 83.9% of test notebooks come from author-repository combinations not present in training, which reduces but does not eliminate source overlap between train and test. The dataset was not assembled from version histories or buggy/fixed commit pairs, so it avoids the particular leakage mode in which a test-time buggy file might have a near-identical fixed revision in training (Ottenhof et al., 22 Jun 2026).

The labeling pipeline is based primarily on stored outputs already embedded in notebook files, not on large-scale re-execution. For each executed code cell, the pipeline inspects the cell’s output field and extracts the error type, error message, cell number, and line number where the error occurred. Those artifacts are then converted into binary cell labels. To capture syntax errors that might never have appeared in saved outputs, each code cell is also parsed independently with Python’s ast module; cells that raise parsing exceptions are treated as syntactically erroneous. Because notebook cells may contain IPython-specific syntax such as %magic and !shell, the preprocessing step uses IPython’s TransformerManager to normalize notebook-specific constructs before parsing (Ottenhof et al., 22 Jun 2026).

The dataset excludes several error categories considered too dependent on runtime environment or user intervention: ImportError, FileNotFoundError, KeyboardInterrupt, SystemExit, and ConnectionError. Memory errors were considered for exclusion but retained, because the paper argues they may also arise from faulty implementation, including memory leaks or poor code design. A direct consequence of the stored-output strategy is that runtime-buggy cells that were never executed will often be labeled non-buggy. The paper identifies this as an important supervision limitation rather than a corner case (Ottenhof et al., 22 Jun 2026).

Quality control combined automatic construction with targeted manual verification. A sample of 100 notebooks was manually inspected to determine whether the search process had preferentially collected pedagogical “bug-demo notebooks”; none were found to focus explicitly on demonstrating specific bugs. For label validation, 30 notebooks were manually checked. In 14 of the 30 notebooks, some cells could not be executed during verification because of missing dependencies or unavailable data, so labels were assigned by manual inspection and, when possible, dummy input data. The results were 24 of 30 notebooks with identical manual and automated labels and 22 disagreements across 970 cells (2.3%), with a reported ±0.94% margin of error at 95% confidence using a Wald interval. The paper also reports a bootstrap McNemar test with 10,000 resamples of size 100; only 139/10,000 resamples (1.39%) were significant at $p<0.05$ , with median $p=0.5000$ (Ottenhof et al., 22 Jun 2026).

3. Schema and notebook representation

OtterDataset stores each notebook as a structured record with seven features. The fields are concise but semantically rich enough to support cell-level supervision, notebook-level supervision, and error-type analysis.

Field	Description
`notebook_name`	Unique identifier combining author, repo, and file name
`notebook_structure`	Concatenated code cells only, with cell boundary markers
`is_buggy`	Binary notebook-level bug label
`error_locations`	List of cell and line numbers where errors occurred
`error_type`	Type(s) of detected errors
`error_message`	Detected error message(s)
`buggy_cells`	Per-cell binary label list

At annotation granularity, the core target is a binary cell-label vector

$\mathbf{y} = [y_1, y_2, \dots, y_n], \qquad y_i \in \{0,1\},$

where $y_i=1$ indicates that the $i$ -th code cell is buggy. A notebook is labeled buggy if at least one code cell is buggy. The dataset also stores error-type metadata and error messages, but those are used analytically rather than as supervised training targets (Ottenhof et al., 22 Jun 2026).

The representation used by JupOtter is notebook-aware. Only code cells are included in notebook_structure; markdown cells are omitted. Special tokens mark the start and end of each cell, cells are packed sequentially into non-overlapping chunks, and each chunk is limited to 2,500 tokens. The only truncation occurs when a single cell itself exceeds the chunk length. Such very long cells are rare: 987 of 661,909 cells, or 0.15%. During training, the model uses up to the first 4 chunks per notebook, which still includes all code cells from 19,728 notebooks, or 93% of the dataset (Ottenhof et al., 22 Jun 2026).

This tokenization scheme addresses a length mismatch between ordinary code models and real notebooks. The paper reports that a baseline using the original CodeT5 tokenizer and file-level classification cannot fully tokenize 84% of notebooks, whereas JupOtter with notebook-aware chunking fails on only 7% of notebooks. A cell-boundary-token model without multi-segment splitting also fails to fully process 84% of notebooks. This suggests that OtterDataset is not only a labeled benchmark but also a stress test for long-context notebook modeling (Ottenhof et al., 22 Jun 2026).

4. Task formulation and benchmark protocol

The benchmark is framed as binary classification at cell granularity, with notebook-level aggregation as a secondary evaluation mode. For each notebook, logits from all chunks are concatenated into a single per-cell prediction vector

$\mathbf{z} = [z_1, z_2, \dots, z_n].$

Training uses BCEWithLogitsLoss, i.e., sigmoid plus binary cross-entropy over cells: $\mathcal{L} = - \frac{1}{n} \sum_{i=1}^{n} \left( y_i \log \sigma(z_i) + (1-y_i)\log(1-\sigma(z_i)) \right),$ with

$\sigma(z_i) = \frac{1}{1 + e^{-z_i}}.$

The paper states that this formulation ensures each cell contributes equally to the notebook’s total loss (Ottenhof et al., 22 Jun 2026).

The data split uses random seed 42 with 80% training and 20% testing. The training set contains 46.6% buggy notebooks, and the testing set contains 49% buggy notebooks. The paper does not define a distinct three-way train/validation/test partition; instead, it describes validation behavior using what is elsewhere called the testing set. This implies that the release is primarily documented as an 80/20 split rather than a conventional train/validation/test benchmark (Ottenhof et al., 22 Jun 2026).

Evaluation uses precision, recall, F1, and accuracy, with two aggregation schemes. In cell-aggregated metrics, true positives, false positives, true negatives, and false negatives are accumulated over all cells in the test set. In file-aggregated metrics, metrics are computed per notebook and then averaged across notebooks. The benchmark therefore permits analysis of both localization quality and coarse notebook-level screening (Ottenhof et al., 22 Jun 2026).

The principal supervised models are JupOtter-small, built on CodeT5-small (60M parameters), and JupOtter-base, built on CodeT5-base (220M parameters). Baselines include Flake8, adapted from script-level analysis to notebooks by converting cells into a Python script and mapping lines back to cells, as well as GPT-4o-mini and Gemini 3 Flash under an incremental cell-by-cell prompting setup. For the LLM baselines, due to cost and latency, evaluation is performed on a random sample of 100 notebooks from each dataset, totaling 7,628 cells with a 6% buggy-cell rate (Ottenhof et al., 22 Jun 2026).

5. Empirical behavior and benchmark results

On the held-out OtterDataset Test split, the reported cell-level results are as follows: Flake8 achieves precision 0.75, recall 0.69, F1 0.72, accuracy 0.80; JupOtter-small achieves 0.93, 0.69, 0.79, 0.95; JupOtter-base achieves 0.88, 0.75, 0.81, 0.95; GPT-4o-mini achieves 0.26, 0.85, 0.40, 0.74; and Gemini 3 Flash achieves 0.45, 0.88, 0.59, 0.85. Thus, JupOtter-base attains the best cell-level F1 of 0.81, while the smaller variant remains close at 0.79. The paper further reports that the difference between the two JupOtter variants is statistically significant under a Wilcoxon Signed-Rank Test, $p<0.01$ across all three benchmark datasets used in the study (Ottenhof et al., 22 Jun 2026).

At file level on the same test split, Flake8 reports precision 0.61, recall 0.77, F1 0.68, accuracy 0.64; JupOtter-base reports 0.88, 0.67, 0.76, 0.80; GPT-4o-mini reports 0.55, 0.87, 0.67, 0.58; and Gemini 3 Flash reports 0.61, 0.88, 0.72, 0.66. Even when reduced to notebook-level decisions, the cell-trained model remains strongest by F1 (Ottenhof et al., 22 Jun 2026).

The paper also analyzes error-type sensitivity. The five most frequent error categories in OtterDataset are TypeError, NameError, SyntaxError, ValueError, and AttributeError. On an external CodeParrot-based analysis filtered to notebooks containing at least one such error, JupOtter-base achieves F1 0.87 for SyntaxError, 0.64 for TypeError, 0.63 for ValueError, 0.38 for AttributeError, and 0.34 for NameError. This pattern is consistent with the dataset construction process: syntax errors are directly exposed to both stored-output parsing and explicit AST checking, whereas other runtime errors are only partially observed through saved execution artifacts (Ottenhof et al., 22 Jun 2026).

Ablation results further characterize the dataset’s difficulty. A baseline using standard CodeT5 tokenization plus file-level classification achieves F1 = 0.73, compared with F1 = 0.76 for JupOtter-base at file level, but lacks cell localization and cannot fully process most notebooks. A cell-boundary-token model without multi-segment chunking scores F1 = 0.78 versus 0.81 for the full model but still fails on 84% of notebooks. This indicates that the dataset’s defining challenge is not merely classification but classification under long, structured, notebook-shaped contexts (Ottenhof et al., 22 Jun 2026).

The name OtterDataset is unusually easy to confuse with other recent “Otter” or “OTTER” artifacts on arXiv. It is distinct from OTMR, the OpTical MicroRobot dataset for microscopy-based microrobot pose and depth perception (Wei et al., 23 May 2025); from DS-PnP and DS-ALL, the real-robot demonstration datasets used by the vision-language-action model OTTER (Huang et al., 5 Mar 2025); from OTTers, the dialogue dataset for one-turn topic transitions (Sevegnani et al., 2021); and from the proprietary multimodal Otter benchmark for hybrid predefined-tag and open-tag prediction (Ouyang et al., 1 Oct 2025). The term is therefore most precisely reserved for the Jupyter-notebook bug-detection dataset introduced in JupOtter (Ottenhof et al., 22 Jun 2026).

Several limitations are explicit. First, runtime-buggy cells that were never executed are often labeled non-buggy because the pipeline relies on stored outputs. Second, the labels are primarily automatically inferred, with manual verification on samples rather than exhaustive human annotation. Third, markdown is omitted from the model representation, so explanatory natural language context is not part of the benchmark input. Fourth, the exclusion of ImportError, FileNotFoundError, KeyboardInterrupt, SystemExit, and ConnectionError makes the corpus cleaner for implementation-bug detection but less representative of the full space of notebook failures. Fifth, the 4-chunk training cap is a hardware-driven constraint, and the paper notes that files are considered buggy only if they contain an error within the first four 2,500-token chunks in that setup (Ottenhof et al., 22 Jun 2026).

Taken together, these properties position OtterDataset as a specialized benchmark for notebook-aware, cell-localized bug detection rather than a general corpus of all notebook failures. Its main technical contribution is not only the scale of the labeled notebook collection but also the coupling of that collection to a representation regime that preserves cell boundaries and supports long-context processing. A plausible implication is that future work using OtterDataset will need to treat notebook structure, partial observability, and chunked inference as first-class aspects of the problem rather than incidental implementation details (Ottenhof et al., 22 Jun 2026).