Plan-Quiz: Code Understandability Pipeline

• Plan-Quiz is a data-driven framework that recommends code edits to improve understandability by analyzing empirical code reviews.
• It employs multi-stage annotations, feature extraction from code diffs, and a sequence-to-sequence Transformer to incorporate reviewer expertise.
• Evaluations demonstrate significant improvements over baselines with metrics like 37% Precision@1 and integration into IDEs for dynamic feedback.

A Plan-Quiz is a data-driven framework for recommending code understandability improvements, instantiated as a sequential pipeline that extracts reviewer expertise, encodes human feedback, and trains a sequence-to-sequence Transformer to suggest readability-preserving code edits. The methodology is grounded in empirical code review practices and leverages multi-stage annotation, code–diff feature extraction, and formalized evaluation to build machine learning models that operationalize reviewer knowledge for practical code-edit recommendation.

1. Dataset Construction and Annotation

The foundation is a manually curated dataset of code understandability improvements sourced from code reviews on mature pull-request–driven repositories (e.g., apache/beam, apache/skywalking). For each merged request, review comments are fetched via GitHub APIs. Two expert annotators independently label each comment+diff pair for suggestions explicitly targeting readability, naming, structure, or proposing a more “neat,” “clear,” or “concise” alternative. The label “Understandability” is applied if these criteria are met. Disagreements are resolved via discussion, with Cohen’s κ enforced at κ ≥ 0.7 to ensure inter-annotator reliability.

Dataset parameters include an initial labeled corpus of approximately 2,000 comments (equal split between positive and negative), sampled from five popular repositories covering the latest two years of pull requests. Filtering heuristics exclude trivial comments (e.g., less than five characters, emoji-only), remarks unconnected to code diffs, and limit the programming languages to Java and Python to control vocabulary and structural variance (Oliveira, 2021).

2. Feature Engineering and Representation

Code snippets and associated review comments are jointly encoded through a combination of natural language and program analysis features. Review comments are tokenized and embedded using pre-trained LLMs (e.g., CodeBERT) to vectors of dimension m ≈ 768. Code versions before (v) and after (v′) the edit are parsed into abstract syntax trees (ASTs), and their difference (Δ = AST(v′) − AST(v)) is linearized and embedded similarly.

Custom features quantify readability and structural change:

• Identifier length mean: IL(v), IL(v′)
• Halstead volume: HV(v), HV(v′)
• Cyclomatic complexity: CC(v), CC(v′)

Δ-readability features are computed as ΔIL, ΔHV, ΔCC, while semantic similarity is assessed by cosine similarity between code embeddings before and after edit. These features ensure functional similarity post-edit and inform the model about the readability impact of proposed changes.

3. Model Architecture and Formalization

The recommendation engine is a sequence-to-sequence Transformer (e.g., T5 or CodeT5) trained to “translate” input code (v) to its improved, readable form (v′):

• Input: x = tokens(v) ∈ V^L
• Output: y = tokens(v′) ∈ V^L′
• Parameterization: θ, with conditional log-likelihood pθ(y | x)

The loss function optimizes token-level cross-entropy:

$$\ell(f_\theta(x), y) = -\sum_{t=1}^{L'} \log p_\theta(y_t | y_{<t}, x)$$

The global objective is empirical risk minimization (including regularization):

$$R(\theta) = \frac{1}{N} \sum_{i=1}^{N} \ell(f_\theta(x^{(i)}), y^{(i)}) + \lambda \|\theta\|^2$$

The dataset is split by project for stratified 80%/10%/10% train/dev/test sets, and 5-fold cross-validation is performed. Hyperparameters, such as layer depth, hidden size, dropout, and learning rate, are optimized by Bayesian search with early stopping determined by minimum development perplexity.

4. Evaluation Metrics and Quantitative Results

Model effectiveness is measured using rigorous retrieval and generation metrics:

• Precision@k: fraction of correct edits in top-k suggestions (k=1,5)
• Recall@k: fraction of ground-truth edits found in top-k
• F1 over exact-match predictions
• Mean Reciprocal Rank (MRR) for ranked suggestion quality

On held-out test pairs (N ≈ 200): Precision@1 = 37%, Precision@5 = 62%, Recall@5 = 56%, F1@1 = 35%, MRR = 0.49. Bootstrap-based confidence intervals and paired t-tests demonstrate that gains over heuristic baselines are statistically significant at p < .05 (Oliveira, 2021).

Exemplars show the system’s fidelity to reviewer suggestions, such as consolidating verbose Java conditionals (“if (status == ACTIVE) ... else ...”) to concise expressions (“return status == ACTIVE;”), and converting imperative Python loops to comprehensions (“result.append(f(x))” ⇒ “result = [f(x) for x in xs if cond(x)]”).

5. Integration Workflow and Feedback Loops

The model is deployable as a REST-backed microservice integrated into IDEs (e.g., VS Code, IntelliJ) or code-review toolchains. Users can invoke “Suggest Understandability Edit” on highlighted code, receiving top-3 candidate edits displayed in an inline diff panel. User feedback—including acceptance, rejection, or modification of suggestions—is logged for dynamic retraining, ensuring adaptation to project-specific idioms and evolving coding styles. The framework supports periodic retraining to incorporate newly approved edits as additional supervision signals.

6. Limitations, Challenges, and Future Research

Existing limitations include an over-simplification tendency that can compromise corner-case semantics, poor handling of rarely encountered language constructs (macros, metaprogramming), and an observed preference for brevity that sometimes reduces clarity. The representational focus on Java and Python restricts broader language coverage.

Future work directions emphasize:

• Scaling datasets across more languages and diverse projects (e.g., leveraging MSR 2020 giants dataset)
• Incorporating reviewer profiles for personalized recommendations
• Integrating a reranking stage maximizing predicted readability gain
• Conducting comprehensive offline and online A/B testing to assess real-world developer productivity impact

These improvements aim to generalize the model’s recommendations, boost robustness in diverse development contexts, and empirically quantify productivity gains in end-user settings (Oliveira, 2021).

---

The Plan-Quiz methodology constitutes a principled pipeline for code understandability improvement via code-edit recommendation, demonstrating how code review expertise and formalized code–diff features can yield measurable, statistically significant improvements in automated refactoring and readability enhancement.