LLM-Integrated Scoring Adjustment (LISA)

• LISA is a family of methods that integrate LLM outputs into scoring pipelines to improve accuracy and capture rich qualitative signals.
• It employs multi-stage procedures—elicitation, alignment, and adjustment—to recalibrate scores using expert or LLM-generated rubrics.
• Empirical results show LISA boosts performance in educational grading, psychometric assessment, regression scoring, speech recognition, and recommendation systems.

LLM-Integrated Scoring Adjustment (LISA) encompasses a family of techniques for leveraging the outputs or implicit knowledge of LLMs to refine, adjust, or augment the assignment of scores across a variety of assessment, recommendation, and evaluation tasks. LISA methods formally integrate LLM-derived information or inference rules—whether as analytic grading rubrics, response scoring functions, text-based item augmentations, regression-aware inference rules, or cross-modal rescoring steps—into algorithmic pipelines, thereby aligning model outputs with specific target criteria, improving measurement accuracy, and capturing richer signal from qualitative or multimodal inputs. The following sections synthesize the core methodologies, evaluation protocols, and empirical properties across recent LISA instantiations in educational grading, psychological measurement, regression scoring, silent speech recognition, and natural language recommendation.

1. Core LISA Methodologies

LISA is implemented as a structured, multi-stage procedure that embeds LLM outputs or judgments directly into the scoring or selection process. The general workflow encompasses:

1. Rule or Score Elicitation: An LLM is prompted to generate analytic rubrics, scoring criteria, or response-level relevance judgments. For constructed response assessment, this involves tailored prompts that elicit minimal, rule-based criteria—a set R of non-overlapping grading rules (Wu et al., 2024). For rating-scale augmentation or label refinement, LLMs assign numerical or class scores to open-ended responses based on prompt variants (Watson et al., 9 Oct 2025).
2. Alignment or Calibration: Generated rubrics or scores are evaluated for alignment with “gold” human rules, expert labels, or baseline model values. Alignment metrics such as $F_1$ or $Δ=1-F_1$ (semantic overlap via SBERT) quantify correspondence, enabling a closed loop for rubric selection (Wu et al., 2024), or information gain via IRT co-calibration in psychometrics (Watson et al., 9 Oct 2025).
3. Adjustment Mechanism: The highest-alignment rubric, the most informative LLM-derived item, or the regression-optimal inference rule is reintegrated into the LLM’s subsequent scoring prompt (“score strictly by rules $R^*$”) or directly into final output computation (mean/median for regression; GP regression for relevance estimation; LLM re-ranking in speech recognition) (Wu et al., 2024, Watson et al., 9 Oct 2025, Lukasik et al., 2024, Benster et al., 2024, Liu et al., 24 Oct 2025).

The following table summarizes representative LISA pipelines across domains:

Domain	LISA Adjustment Mechanism	Key Alignment Metric / Rule
Educational grading	Rubric injection + F₁-based selection	Rubric–human rule overlap (Δ)
Psychometric test augmentation	IRT item info gain selection	ΔI (information gain)
Regression scoring	Bayes-optimal mean/median aggregation	Risk minimization under metric m
Silent speech recognition	LLM rescoring of beam outputs	Best hypothesis via LLM + acoustic score
NL recommendation	GPR posterior with LLM relevance labels	RBF kernel on LLM-adjusted GP

2. Analytic Rubric Generation and Alignment in Constructed Response Scoring

A canonical LISA instance is the fully documented three-stage pipeline for LLM-based automatic scoring of written science responses (Wu et al., 2024). For each new item, a human-crafted analytic rubric $\hat R = \{h_1, ..., h_m\}$ specifies a minimal checklist of mutually independent reasoning steps. The LLM is role-play prompted to generate its own rubric $R = \{r_1, ..., r_n\}$, with “full-shot” (all previous items’ rubrics as context), “one-shot”, “no rubric”, “+graded responses”, and “+holistic rubric” prompt variants rigorously evaluated.

Rubric alignment is quantified using SBERT-based semantic matching (threshold ≥ 0.6), computing:

$$p = \frac{|\{r_i : \exists\, h_j,\ \text{sim}(r_i, h_j) \geq 0.6\}|}{n},\quad r = \frac{|\{h_j : \exists\, r_i,\ \text{sim}(h_j, r_i) \geq 0.6\}|}{m}$$

$F_1 = \frac{2pr}{p+r},\quad \Delta = 1 - F_1$

Empirically, the LISA scoring pipeline—eliciting rubrics, aligning, and then re-injecting the best-matching rubric into the scoring prompt—raises exact agreement accuracy from 34.8% (no rubric) to 50.4% (human rubric) and 54.6% (LLM full-shot + holistic, i.e., LISA), with a high Spearman correlation between rubric alignment $F_1$ and scoring accuracy ($\rho=0.943$, $p < 0.01$) (Wu et al., 2024).

3. LISA for Integrating LLM-Scored Text in Psychometric Measurement

In the psychometric context, LISA augments rating-scale assessments using LLM-scored open-ended text, bypassing reliance on pre-labeled data or manual rubric construction (Watson et al., 9 Oct 2025). The procedure is as follows:

• Pool Generation: Open-text responses (e.g., sentence completions, essays) are paired with multiple simple LLM scoring prompts, producing a candidate pool (e.g., 52 LLM-scored items across 13 texts × 4 prompts).
• Item Co-Calibration and Selection: Each candidate item $j$ is co-calibrated with baseline binary items in a unidimensional 2PL item response theory model, and its information gain $\Delta I_j$ is calculated:

$\Delta I_j = \int I_j(\theta)w(\theta)d\theta$

where $w(\theta)$ is the empirical trait distribution.

• Test Construction: Candidacy is restricted to LLM-scored items utilizing the full response scale (1–5). Final augmented forms (e.g., Best_All, Top_5) maximize total added information.

On real student data, LISA-based augmentation results in improvements equivalent to adding an average of 6.3 rating-scale items (Best_All) in terms of information gain and a statistically significant reduction in trait estimate standard error ($p<.001$). On synthetic data, equivalent gain rises to 16.0 items (Watson et al., 9 Oct 2025). LISA remains agnostic to content domain, requiring only existing rating tests plus accompanying open-text.

4. Metric-Aware Inference in LLM Regression and Scoring

LISA also describes regression-aware adjustments in LLM inference (Lukasik et al., 2024). Standard greedy or sampling-based decoding strategies are suboptimal for most evaluation metrics except exact-match. For regression ($-(y-\hat y)^2$), the Bayes-optimal output is $\mathbb{E}[y]$; for ordinal absolute-error, the median; for ranking AUC, the positive-class frequency; and for $F_1$, empirical search over candidates solves:

$$\hat y^* = \arg\max_{y' \in Y}\; \mathbb{E}_{y\sim p(\cdot|x)}\left[m(y, y')\right]$$

The LISA/MALI pipeline samples $K$ LLM outputs given a prompt, aggregates them according to the closed-form Bayes-optimal rule for the chosen metric, and outputs the result. This reduces error relative to both greedy and argmax self-consistency decoding, with improvements ranging from 5–50% in typical regression settings (e.g., STSB RMSE reduced from 0.685 to 0.636 with metric-weighted LISA) (Lukasik et al., 2024). The computational overhead matches self-consistency sampling due to transformer prefix caching.

5. LLM-Integrated Rescoring in Silent Speech Recognition

LISA is utilized as a post-beam-search rescoring module in silent speech recognition, where task knowledge and semantic context encoded by an LLM disambiguate acoustically plausible but incorrect hypotheses (Benster et al., 2024). The pipeline proceeds as:

• Acoustic decoding yields a beam of $K$ candidate text hypotheses $H=\{h_1,\ldots,h_K\}$.
• The LLM, prompted with the full list, selects or re-ranks candidates, returning a final best hypothesis or softmaxed relevance probabilities.
• (Optional) Fine-tuning on in-domain beam lists further specializes the LLM for rescoring.
• Final output is either the LLM’s top choice or the argmax of $$S_\text{acoustic}(h_j) + \lambda S_\mathrm{LLM}(h_j)$$, with $\lambda$ typically absorbed by prompt design.

Empirically, this approach reduces word error rate (WER) from 22.2% to 12.2% on the Gaddy silent speech benchmark, and to 3.7% for vocal EMG, with further reductions via ensemble and fine-tuned LLM rescoring (Benster et al., 2024). The primary computational limitation is increased inference latency from LLM calls and ensemble decoding.

6. Multimodal Item Scoring and Relevance Calibration in Recommendation

In natural language recommendation, LISA denotes the use of Gaussian Process Regression (GPR) with LLM-labeled relevance judgments to adaptively recalibrate and de-unimodalize DR-based passage scores per query (Liu et al., 24 Oct 2025). The procedure is:

• Dense retrieval produces candidate passage scores $s_{DR}(q,p_j)$.
• A small set of $R$ passages per query is labeled by an LLM (discrete logits converted to expected relevance).
• These (embedding, LLM-score) pairs define training data for a GPR model with an RBF kernel, yielding posterior mean scores $m(\mathbf{v}^{(p)})$ over all passages.
• Final item relevance is aggregated from the top-$T$ passages ranked by $m(\cdot)$.

The RBF kernel is found to outperform linear and cosine baselines, and even with as few as $R=25–50$ LLM labels, the method improves re-ranking precision by up to 65% versus pointwise LLM scoring or DR alone. Parameter selection (e.g., $\epsilon$-greedy passage sampling, kernel bandwidth) is robust across datasets (Liu et al., 24 Oct 2025).

7. Comparative Insights, Scalability, and Limitations

LISA strategies consistently demonstrate substantial empirical gains across application domains—raising scoring accuracy, measurement precision, and recommendation recall without requiring additional human-labeled data or new model training.

Key comparative findings include:

• Alignment between LLM and human rubrics or expert scoring rules (as measured by semantic overlap or information gain) is tightly coupled with downstream performance, motivating closed-loop selection and calibration (Wu et al., 2024, Watson et al., 9 Oct 2025).
• Simple injection of LLM-generated rules or scores, if carefully aligned or selected, bridges the gap between domain-expert interpretation and model output, accommodating cases where standard inference or DR scores are misspecified or unimodal (Lukasik et al., 2024, Liu et al., 24 Oct 2025).
• Performance gains saturate at moderate label budgets (e.g., $R=50$ in GP-LISA), and computation is tractable; most LISA instances involve inference-only modifications with no additional model training.

Reported limitations include reliance on the quality and interpretability of LLM outputs, context and prompt drift, and evaluation restrictions to specific metrics or tasks. Future extensions include generalization to more diverse LLMs, active learning for label selection, improved prompt engineering for rubric alignment, and integration into low-latency pipelines.

Overall, LISA represents a unifying framework for formally integrating LLM-derived information into scoring mechanisms across multiple domains, yielding robust, interpretable, and empirically superior assessment strategies.