JupOtter: Bug Detection for Jupyter Notebooks

• JupOtter is a bug detection system that leverages notebook-specific tokenization and cell-level predictions to enhance error identification.
• It utilizes transformer encoders with special cell markers and chunking techniques to maintain essential context within notebook cells.
• Empirical evaluations show that JupOtter outperforms traditional static analyzers and large language models in cell-level bug detection benchmarks.

Searching arXiv for recent and related papers on Jupyter notebook analysis and code bug detection. {"query":"arXiv Jupyter notebook bug detection notebook analysis CodeT5", "max_results": 10} {"query":"Jupyter notebook analysis arXiv", "max_results": 10} {"query":"all:JupOtter (Ottenhof et al., 22 Jun 2026)", "max_results": 5} JupOtter is a bug detection system designed specifically for Jupyter Notebooks, motivated by the increasing use of notebooks for more complex programs and the corresponding rise of buggy notebooks on platforms such as GitHub. Its design centers on three contributions: a notebook-specific tokenization strategy that preserves cell structure, a cell-level bug prediction technique, and OtterDataset, a labeled corpus containing over 21,000 notebooks annotated for fine-grained cell-level bug detection. In evaluation, JupOtter achieves cell-level bug detection F1 scores that surpass static analyzers and LLMs in two out of three evaluation datasets (Ottenhof et al., 22 Jun 2026).

1. Problem setting and system scope

Jupyter Notebooks differ from conventional source files in that program logic is distributed across ordered cells rather than a single contiguous file. JupOtter formalizes a notebook as a sequence of $N$ code cells, $C = (c_1, c_2, \dots, c_N)$, and targets the prediction of whether each individual cell is buggy. This cell-level objective is central: the system is not limited to notebook-level classification, although notebook-aggregated predictions can be derived by marking a file as buggy when any constituent cell is predicted buggy.

The underlying motivation is that standard code tokenizers either truncate long inputs or ignore notebook cell structure, thereby losing context that is intrinsic to the notebook execution model. JupOtter addresses this by preserving cell boundaries explicitly during tokenization and by decoding predictions back to cells rather than only to files or tokens. A common simplification is to treat notebooks as ordinary source files; JupOtter is explicitly constructed against that assumption, using the notebook as its native unit of representation.

The target bug signal is operational rather than purely syntactic. OtterDataset labels cells with runtime errors found in stored outputs, while also detecting syntax errors by parsing each code cell with Python’s ast module through IPython TransformerManager. Environment-only errors—specifically ImportError, FileNotFoundError, KeyboardInterrupt, SystemExit, and ConnectionError—are excluded from the labeling process, defining a narrower notion of buggy behavior than a general-purpose execution-failure detector.

2. Cell-aware tokenization and chunk construction

JupOtter’s tokenization strategy augments each cell with two special markers, [CLS_CELL] and [SEP_CELL], so that a cell with token sequence $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$ is transformed into

$$\widetilde{\mathrm{Tok}}(c_i)=([\text{CLS\_CELL}], t_{i,1}, \dots, t_{i,\ell_i}, [\text{SEP\_CELL}]).$$

The effective cell length becomes $\ell_i' = \ell_i + 2$. This encoding makes cell boundaries explicit to the downstream transformer and later enables token-level outputs to be mapped back to notebook cells (Ottenhof et al., 22 Jun 2026).

Notebook inputs are then partitioned by a chunking procedure that scans cells in order and groups them into non-overlapping segments of complete cells. A maximum chunk length $L$ is fixed—for example, 2,500 tokens—and cells are appended to the current chunk while the cumulative length remains within $L$; otherwise a new chunk is started. The resulting notebook is represented as $K$ chunks, each a flattened token sequence of length at most $L$.

After padding, each notebook is encoded as three two-dimensional tensors:

• $T \in \mathbb{Z}^{K \times L}$ for input token IDs
• $C = (c_1, c_2, \dots, c_N)$0 for attention masks
• $C = (c_1, c_2, \dots, c_N)$1 for cell labels, where $C = (c_1, c_2, \dots, c_N)$2 is the maximum number of cells in any chunk

This representation is notebook-specific in a precise sense: chunking preserves complete cell boundaries rather than arbitrary sliding windows, and the special markers provide a reversible interface between token-space computation and cell-space prediction. The ablation study indicates that both ingredients matter. A default CodeT5 file-level setup with no cell markers and no chunking attains F1 = 0.73 and fails to fully process 84% of notebooks due to input length limits; adding cell boundary markers raises F1 to 0.78 but still leaves 84% unprocessed; the full configuration with markers and chunking reaches F1 = 0.81 and reduces the unprocessed fraction to 7%.

3. Prediction architecture and optimization regime

JupOtter uses a pre-trained transformer encoder—either CodeT5-small or CodeT5-base—to process notebook chunks in parallel. For chunk $C = (c_1, c_2, \dots, c_N)$3, the encoder outputs hidden states

$C = (c_1, c_2, \dots, c_N)$4

For each cell within a chunk, the model identifies the token span corresponding to that cell and computes a cell embedding by average-pooling the hidden states over the span. A shared linear layer then maps each pooled vector $C = (c_1, c_2, \dots, c_N)$5 to a scalar logit $C = (c_1, c_2, \dots, c_N)$6, and the buggy-cell probability is obtained with a sigmoid:

$C = (c_1, c_2, \dots, c_N)$7

Predictions across all chunks are concatenated in execution order to recover a notebook-level vector of length $C = (c_1, c_2, \dots, c_N)$8 (Ottenhof et al., 22 Jun 2026).

Training uses binary cross-entropy with logits loss (BCEWithLogitsLoss) over all cells in a batch of notebooks:

$C = (c_1, c_2, \dots, c_N)$9

Here $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$0 is the number of cells in chunk $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$1, and $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$2.

Although chunks are processed independently, the self-attention layers can attend across tokens of neighboring cells within a chunk, capturing cross-cell dependencies without assuming a fixed execution order. This design yields contextualization at the cell neighborhood level while avoiding the memory costs of full-notebook transformer passes.

To control memory consumption on long notebooks, JupOtter treats each notebook’s $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$3 chunks as $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$4 sub-samples in a per-sample backward-pass loop, accumulating gradients across all chunks before updating parameters. Mixed-precision training via torch.cuda.amp is used to further reduce GPU memory and speed up training. The paper notes that training was limited to the first 4 chunks per notebook, corresponding to approximately 93% coverage, due to GPU memory rather than an architectural restriction.

4. OtterDataset: collection, annotation, and validation

OtterDataset is the corpus introduced alongside JupOtter for fine-grained cell-level bug detection. The collection procedure queried GitHub for Python notebooks using 32 search keywords, including examples such as TypeError and IndexError, with a cap of at most 1,000 notebooks per query to ensure diversity. The final raw set contains 21,303 unique notebooks, of which 9,937 are labeled buggy and 11,366 non-buggy (Ottenhof et al., 22 Jun 2026).

Annotation proceeds in three stages. First, the stored outputs of executed cells are extracted, and cells with runtime errors are labeled buggy, excluding the environment-only error classes listed above. Second, each code cell is independently parsed with Python’s ast module through IPython TransformerManager to capture syntax errors not visible in outputs. Third, the process produces a binary vector $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$5 per notebook together with metadata fields error_type, error_message, and error_locations.

Resource	Notebooks	Cells
OtterDataset (total)	21,303	661,909
OtterDataset test split	4,113	124,364
CodeParrot subset	4,769	78,313
Jupyter Errors dataset	9,313	282,371

Within OtterDataset as a whole, 47% of notebooks are buggy. The corpus contains 43,789 buggy cells, approximately 7% of all cells and approximately 2.1 buggy cells per buggy notebook. The train/test partition is 80%/20% by notebooks, with no author–repo overlap; 83.9% of test notebooks come from unseen authors.

Label-quality validation is reported through a manual audit of 30 notebooks comprising 970 cells. Automated and manual annotation agree at 97.7%, with a $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$6 margin at 95% confidence interval. A bootstrap McNemar test with 10,000 resamples shows strong agreement, with median $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$7 and only 1.39% of resamples significant at $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$8. This suggests that the automated labeling pipeline is reliable at the granularity required for cell-level supervision.

5. Empirical performance and comparative evaluation

JupOtter is evaluated on three benchmarks: the OtterDataset test split, a CodeParrot subset, and the Jupyter Errors dataset. Baselines include Flake8, mapped back to cells, and two LLMs—GPT-4o-mini and Gemini 3 Flash—used with incremental cell-by-cell prompting on samples of 100 notebooks per benchmark due to API cost. The principal cell-level metrics are file-aggregated precision, recall, F1, and accuracy (Ottenhof et al., 22 Jun 2026).

On the OtterDataset test split, the reported cell-level results are:

Technique	F1	Acc.
Flake8	0.72	0.80
JupOtter_small	0.79	0.95
JupOtter_base	0.81	0.95
GPT-4o-mini	0.40	0.74
Gemini 3 Flash	0.59	0.85

JupOtter_base achieves cell-level F1 = 0.81 on OtterDataset and F1 = 0.89 on CodeParrot, outperforming Flake8, GPT-4o-mini, and Gemini on two of three benchmarks. On the Jupyter Errors dataset, it maintains high precision of 0.82 but lower recall of 0.22, which the paper attributes to sparser training signal for un-executed runtime errors.

The error-type breakdown on the CodeParrot subset shows particularly strong performance on SyntaxError, with recall 0.85, precision 0.90, F1 0.87, and accuracy 0.96. Performance is lower on TypeError and ValueError (both recall 0.51), and substantially lower on NameError (recall 0.23, F1 0.34) and AttributeErr (recall 0.28, F1 0.38). This distribution indicates that the model is not uniformly effective across bug classes.

Although JupOtter is designed for cell-level prediction, its notebook-aggregated output also performs competitively at file level. On the OtterDataset test split, JupOtter_base attains precision 0.88, recall 0.67, F1 0.76, and accuracy 0.80, compared with Flake8 at F1 0.68, GPT-4o-mini at F1 0.67, and Gemini 3 Flash at F1 0.72. The system outperforms all baselines on two of three file-level benchmarks by F1.

Statistical testing is reported for two comparisons. The difference between JupOtter_small and JupOtter_base is significant under the Wilcoxon Signed-Rank test with $\mathrm{Tok}(c_i) = (t_{i,1}, \dots, t_{i,\ell_i})$9. The comparison between JupOtter_base and Flake8 at file level is significant under McNemar’s test with $$\widetilde{\mathrm{Tok}}(c_i)=([\text{CLS\_CELL}], t_{i,1}, \dots, t_{i,\ell_i}, [\text{SEP\_CELL}]).$$0 on all three benchmarks.

6. Practical significance, limitations, and prospective extensions

JupOtter is intended for integration into interactive development environments such as JupyterLab or VS Code as a live assistant that lints cells on the fly. Its cell-level granularity is operationally significant because it yields immediate, actionable feedback at the level where notebook authors actually edit and execute code. The paper identifies several natural extensions: outputting multi-class logits to predict error type, incremental inference on edited cells, confidence scores, and inline explanations (Ottenhof et al., 22 Jun 2026).

The system also has explicit limitations. Runtime errors that were never executed in the original notebook are labeled non-buggy, so recall for such errors is a lower bound rather than a complete estimate of defect-detection capability. Training was limited to the first 4 chunks per notebook because of GPU memory, even though the method itself supports more chunks on larger hardware. Environment-only errors were excluded during annotation to avoid spurious labels, but real deployments may need to detect missing imports or file-path failures as well.

The paper discusses data-leakage mitigation measures: CodeT5 pre-training did not include notebooks; duplicates and author–repo overlaps were removed; and no version history was used. At the same time, it states that zero leakage cannot be guaranteed for proprietary LLM baselines. This is a methodological caveat rather than a claim of observed contamination.

Future work is framed along four axes: incorporation of markdown context such as heading structure into chunking; optimization of model size and latency for client-side inference; augmentation of OtterDataset with synthetic or rare error types to improve recall on underrepresented bugs; and joint modeling of environment and implementation errors. Taken together, these directions suggest an emerging research program in notebook-native program analysis, where the notebook cell is treated as a first-class unit for representation, supervision, and feedback rather than as a formatting inconvenience appended to conventional source-code pipelines.