JupOtter: Cell-Level Bug Detection in Jupyter Notebooks

Abstract: Jupyter Notebooks are an increasingly popular coding environment used across many domains, especially in Python-based data science and scientific computing. Originally used for prototyping and interactive exploration, notebooks are increasingly used to develop more complex programs, leading to a rapid rise in buggy notebooks on platforms like GitHub. To address this trend, we present JupOtter, a bug detection system designed specifically for Jupyter Notebooks. JupOtter features three novel contributions: (1) a notebook-specific tokenization strategy that preserves cell structure, (2) a cell-level bug prediction technique, and (3) a new labeled dataset, OtterDataset, containing over 21,000 notebooks annotated for fine-grained cell-level bug detection. JupOtter achieves cell-level bug detection F1 scores that surpass static analyzers and LLMs in two out of three evaluation datasets.

• The paper introduces a transformer-based method that detects buggy cells in Jupyter Notebooks using a novel, notebook-aware tokenization strategy.
• The approach leverages the OtterDataset of over 21,000 annotated notebooks, achieving cell-level F1 scores up to 0.81 and outperforming traditional static analysis tools.
• The study offers practical implications for automated debugging in notebook environments and paves the way for real-time, IDE-integrated bug detection.

JupOtter: Cell-Level Bug Detection in Jupyter Notebooks

Introduction and Motivation

The paper "JupOtter: Cell-Level Bug Detection in Jupyter Notebooks" (2606.23877) proposes JupOtter, a transformer-based system for cell-level bug detection in Jupyter Notebooks. With the widespread adoption of notebooks for data science and scientific computing, their inherent cell-based and non-linear execution patterns exacerbate the bug incidence rate. The paper addresses the deficit of accurate, actionable bug detection tooling adapted to the notebook paradigm, especially given the rapid growth of implementation-related errors. JupOtter is architected to accommodate the notebook-specific challenges of variable cell lengths, mixed code and markdown, and complex inter-cell dependencies, surpassing previous methodologies that were hampered by limited granularity and file/function-level focus.

OtterDataset Construction

Central to JupOtter is OtterDataset, a large-scale, cell-level annotated corpus comprising over 21,000 Python notebook files sourced from GitHub. The dataset was curated via domain-specific search queries targeting prevalent bug types, followed by rigorous validation protocols to ensure label accuracy and eliminate data leakage. Cells were annotated for implementation-related errors based on explicit output traces and direct code parsing, intentionally excluding environment-related errors to minimize unreproducible labeling artifacts. Manual validation of automated labels demonstrated high fidelity, with a label error margin below 1% on a cell sample size of nearly 1,000, ensuring suitability for supervised learning.

Notebook-Aware Tokenization and Model Architecture

JupOtter introduces a notebook-tailored, chunking-based tokenization strategy. Notebooks are sliced into non-overlapping chunks bounded by special cell markers, preserving cell boundary semantics. This design enables processing of large notebooks, eliminating truncation pitfalls and supporting multi-chunk training. The architecture leverages transformer encoders (CodeT5-small/base), with each cell's representation extracted by mean-pooling the token embeddings within its boundaries (including markers). A linear classifier then produces a binary prediction per cell—buggy or not—allowing direct localization rather than ambiguous file-level signaling.

Figure 1: JupOtter's end-to-end pipeline, from cell extraction and chunking to transformer-based prediction.

This structure is especially relevant for notebooks with variable-length cells and pervasive cross-cell dependencies, ensuring robust, scalable inference unconstrained by token limits.

Prediction Granularity and Output

JupOtter's approach to cell-level bug prediction is exemplified by its capacity to output a binary label per cell, thereby precisely highlighting buggy code regions without diluting actionable quality feedback. Each notebook is tokenized, passed into the encoder, and all cells in processed chunks are independently classified.

Figure 2: Cell-wise predicted labels, illustrating direct assignment of bug probabilities per notebook cell.

This granularity not only enhances developer productivity by providing specific repair targets but also aligns with the practical realities of notebook development, where file-level bug indicators tend to be uninformative.

Empirical Evaluation

JupOtter was empirically validated against three datasets: OtterDataset's held-out test split, a cell-labeled subset of CodeParrot, and Jupyter Errors. Both small and base versions of the model were trained, and performance was benchmarked against Flake8, GPT-4o-mini, and Gemini 3 Flash, adapted to cell-level settings. On the OtterDataset test set, JupOtter-base outperformed baselines with a cell-level F1 of 0.81 (Precision: 0.88; Recall: 0.75), while JupOtter-small achieved almost equivalent F1 at 0.79. On CodeParrot, cell-level F1 reached 0.89 for JupOtter-base, compared to 0.74 for Flake8 and significantly lower scores for LLM baselines.

In Jupyter Errors, despite overall high precision ($>0.8$), recall dropped (0.22–0.17), indicating runtime bugs remain harder to capture without execution traces—an area for future enhancement. Importantly, statistical testing (Wilcoxon Signed-Rank) demonstrated JupOtter's superiority over baselines at high confidence ($p<0.01$).

Error-Type Sensitivity and Ablation Study

Error-type stratified analysis showed JupOtter particularly excels at SyntaxError detection (F1: 0.87; Recall: 0.85), with moderate efficacy for TypeError and ValueError (F1: 0.63–0.64), but lower for NameError and AttributeError (F1: 0.34–0.38). This illustrates both strong capabilities in static pattern detection and limitations driven by the dataset's error distribution and intrinsic semantic complexity of certain bug types.

Ablation studies confirmed that the notebook-aware tokenization strategy is critical for both cell-level and file-level prediction. Without multi-chunk processing, only 16% of notebooks could be tokenized, whereas JupOtter's approach scales to 93% coverage using four chunks, reinforcing the necessity of the method for real-world deployment.

Comparative Performance and Practical Implications

JupOtter consistently outperformed Flake8 and LLM baselines at cell-level detection and exhibited reduced false positive rates in file-level bug detection. On datasets with lower bug density (CodeParrot, OtterDataset), JupOtter-base yields a file-level F1 up to 0.76—substantially higher than LLM methods (F1 $<0.48$ on CodeParrot). However, in settings dominated by runtime bugs (Jupyter Errors), recall is less competitive, motivating augmentation with dynamic analysis.

The model's predictive conservatism translates to fewer spurious bug indications, aligning with developer needs for actionable, high-confidence localization. In notebook-centric workflows, where rapid iteration and refactoring predominate, JupOtter's cell-level granularity is highly advantageous for automated quality feedback in live environments (e.g., JupyterLab, VS Code).

Theoretical and Practical Implications; Future Directions

JupOtter advances the state-of-the-art in fine-grained defect detection in notebook environments. The OtterDataset enables further development of cell-level bug prediction systems, fostering benchmark reproducibility and comparative research. The scalable tokenization strategy and cell-wise classification unlock new opportunities for automated debugging, educational feedback, and notebook refactoring not previously possible with script- or function-oriented models.

Model integration into IDEs for real-time inference is a logical next step. Extensions to error-type multi-classification, markdown context integration, and incremental inference for latency reduction are promising avenues. Combining cell-level static models with runtime data could substantially elevate recall for elusive errors.

Conclusion

JupOtter establishes a robust pipeline for cell-level bug detection in Jupyter Notebooks, coupling a novel tokenization method with a transformer-based classifier and releasing a high-quality annotated dataset. Evaluations demonstrate superior performance to conventional static analysis and mainstream LLMs in both precision and actionable granularity. The research lays the groundwork for practical AI-driven notebook quality tools and signals future progress toward real-time, context-aware notebook debugging environments.