---
title: Natural Language-Based Assessment (NLA)
url: https://www.emergentmind.com/topics/natural-language-based-assessment-nla
type: topic
---

# Natural Language-Based Assessment (NLA)

Searching arXiv for recent and relevant papers on Natural Language-Based Assessment.
Natural Language-Based Assessment (NLA) denotes a family of assessment paradigms in which natural language is not merely an auxiliary interface but the primary representational medium for eliciting, expressing, and evaluating judgments. Across the literature, NLA appears in several technically distinct forms: as natural-language explanation of low-level perceptual judgments in speech quality assessment, as theory-driven measurement directly from language embeddings, as rubric-guided scoring with generated rationales, and as automated inference from naturally occurring discourse in clinical, educational, software-engineering, and safety settings [2503.20290] [2605.04873] [2507.10200]. A common shift across these formulations is from scalar prediction alone toward assessment artifacts that remain linguistically interpretable: descriptors, rationales, structured comments, or semantically grounded projections. This suggests that NLA is best understood not as a single model class but as a design principle for assessment systems in which language itself carries the construct definition, the evidence, or the explanatory trace.

## 1. Conceptual scope and formalization

In one explicit formulation, NLA is contrasted with conventional numerical assessment by the mapping from input to text rather than input to a scalar score: numerical quality assessment is written as \(f(x) \rightarrow y \in \{1, \dots, 5\}\), whereas natural language assessment is written as \(f(x) \rightarrow \text{text} \in \mathcal{L}\) [2503.20290]. In that formulation, scalar scores and natural language are described as complementary rather than competing representations: a score gives a rough overall assessment, while language identifies what is wrong, where, and why [2503.20290].

A second formulation treats NLA as direct measurement from language rather than prediction of an external questionnaire. In the semantic-projection framework for psychological assessment, a response is embedded as a vector \(x\), a construct is encoded as a semantic axis \(a\), and the assessment score is the projection
\[
\text{score}(x) = \frac{x \cdot a}{\lVert a \rVert}.
\]
This converts natural language into a continuous psychological score without supervised training on individual-level labels [2605.04873]. In that setting, NLA is explicitly positioned against supervised text-to-score pipelines by defining the construct directly in embedding space through theory-based axes derived from lexical anchors or validated scale items [2605.04873].

In language assessment for L2 speaking, NLA is defined as using instructions expressed as can-do descriptors, originally intended for human examiners, to determine whether LLMs can interpret and apply them comparably to human assessment [2507.10200]. Here the assessment object is neither a latent neural score nor a post hoc explanation, but the descriptor-grounded judgment itself. The same general idea appears in rubric-guided SpeechLLMs for L2 pronunciation, where the model jointly predicts ordinal labels and a free-text rationale in a single response [2606.09470].

Across these variants, several recurrent structural motifs appear. One is descriptor-grounding: CEFR can-do descriptors, quality rubrics, or clinically motivated semantic axes define the target construct in language [2605.04873] [2507.10200] [2606.09470]. Another is reasoning-rich output: comments, chain-of-thought-style assessments, or narrative rationales are treated as part of the assessment object rather than as ancillary explanation [2503.20290] [2606.09470]. A third is task-oriented evaluation of language outputs by mapping them back to structured targets, such as scores, temporal intervals, or clinically relevant dimensions [2503.20290].

## 2. From scores to descriptors, comments, and semantic axes

A central theme in NLA research is dissatisfaction with assessment systems that output only a number. In speech quality assessment, the critique is directed at MOS-style scalar ratings: they summarize perceived quality but do not reveal the reasoning behind the score and give limited guidance on how to improve a system [2503.20290]. QualiSpeech addresses this by collecting seven scored aspects—noise, distortion, speed, continuity, listening effort, naturalness, and overall quality—together with four description-based aspects: noise description, distortion description, unnatural pause description, and voice characterization [2503.20290]. The descriptive comment then integrates these low-level aspects into an overall judgment in natural language [2503.20290].

In psychological assessment, the analogous critique is that supervised language models trained to predict questionnaire scores may have limited interpretability and generalizability across contexts [2605.04873]. The semantic-projection alternative operationalizes constructs such as depression-related affect and worry/anxiety-related affect as contrasts between low- and high-symptom content. The axis vector is defined as
\[
a = \frac{1}{m} \sum_{j=1}^{m} p_j \;-\; \frac{1}{n} \sum_{k=1}^{n} q_k,
\]
with positive anchors \(p_j\) and negative anchors \(q_k\) derived either from lexical items or validated scale items such as CES-D, Zung, and STAI-Y [2605.04873]. This makes the assessment scale itself linguistically inspectable.

A third instantiation appears in L2 oral proficiency. There, CEFR analytic descriptors for ten aspects—general linguistic range, vocabulary range, vocabulary control, grammatical accuracy, sociolinguistic appropriateness, flexibility, thematic development, coherence and cohesion, fluency, and propositional precision—are presented to an LLM, which selects the best-fitting descriptor for each aspect [2507.10200]. Holistic scores are then reconstructed from those analytic decisions. This suggests a form of NLA in which natural language descriptors supply both the scoring rubric and the interpretive frame.

Rubric-guided SpeechLLM assessment extends this principle to joint multi-granular output. On SpeechOcean762, the model predicts sentence-level labels for accuracy, fluency, and prosody, word-level and phoneme-level accuracy labels, and a free-text rationale in a single generated response [2606.09470]. The rationale is not external to the assessment; it is part of the output format and conditioned on the same rubric definitions [2606.09470].

These threads point to a broader distinction between two kinds of NLA. One kind uses language as the output format for assessment, often with reasons, descriptors, or comments. Another uses language as the measurement substrate itself, as in semantic projection. A plausible implication is that NLA spans both “assessment in language” and “assessment from language,” and the most mature systems increasingly combine the two.

## 3. Methodological architectures

NLA systems in the cited literature fall into several technical families.

One family is auditory-LLM generation. QualiSpeech uses fine-tuned SALMONN-7B, with Whisper and BEATs audio encoders, a Q-former connector, and a Vicuna-v1.5-7B backbone; only the speech Q-former connector and LoRA adapters on the LLM are fine-tuned in most experiments [2503.20290]. The model is trained to generate either scores, descriptions, or full comments from audio plus prompt, and the natural-language output is then parsed back into structured items for evaluation [2503.20290].

A second family is text-only descriptor application by a general LLM. In L2 oral proficiency assessment, a 4-bit quantized Qwen 2.5 72B model is used in a zero-shot setting to assess ASR transcripts against CEFR descriptors [2507.10200]. For each aspect, logits over descriptor options are converted into a softmax distribution and then into a continuous Fair Average Score:
\[
\text{FairAvg} = \sum_{k=1}^{K} p_k \cdot v_k.
\]
These analytic scores are averaged, and optionally combined via Ridge regression, to predict holistic performance [2507.10200].

A third family is hybrid supervised generative modeling. The rubric-guided SpeechLLM based on Qwen2-Audio-7B-Instruct is trained with a hybrid objective that combines supervised fine-tuning and Bounded Direct Preference Optimization:
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{BDPO}} + \lambda \cdot \mathcal{L}_{\text{SFT}}.
\]
The model consumes speech waveform, transcript, target phoneme sequence, and rubric definitions, and generates structured labels plus a rationale [2606.09470]. The BDPO component is used only on label tokens, while rationales are trained through the SFT objective [2606.09470].

A fourth family is unsupervised semantic geometry. In psychological assessment, Sentence-BERT embeddings and semantic projection yield continuous scores without fitting to subject-level labels [2605.04873]. For long free-text, sentence-level mean and max-absolute aggregation are introduced:
\[
\text{score}_{\text{mean}(T)} = \frac{1}{k} \sum_{i=1}^{k} \text{score}(s_i),
\qquad
\text{score}_{\text{maxabs}(T)} = \text{score}(s_j),
\]
where \(j = \arg\max_i |\text{score}(s_i)|\) [2605.04873]. This gives a distinct NLA architecture in which interpretability derives from the construct axis rather than from generated explanations.

A fifth family is structured text classification or sequence labeling for downstream assessment. In classroom discussion assessment, BERT-based sentence classifiers, hierarchical classifiers, and BERT–BiLSTM–CRF sequence labeling are used to infer Analyzing Teaching Moves, which are then mapped to discussion-level IQA rubric scores [2306.14918]. In rehabilitation fidelity assessment, a BERT classifier labels utterances as guided, directed, or none, enabling automated fidelity scoring from therapist language [2209.06727]. In requirements satisfaction assessment, Sat-BERT and its multitask variants assess whether design texts satisfy requirement texts by semantic coverage [2312.04463]. In software testing, CCG-based semantic parsing maps English descriptions to executable property-based tests [2202.03616]. These are less generative than the descriptor- and rationale-based systems, but they remain NLA insofar as the assessment target is inferred from language artifacts.

## 4. Evaluation paradigms and metrics

Evaluation in NLA research is heterogeneous because the outputs themselves differ. Several papers explicitly avoid relying only on surface text similarity.

In QualiSpeech, the natural language output is the primary assessment object, but evaluation parses it back into structured forms. Numeric aspects are scored with Pearson Correlation Coefficient:
\[
\text{PCC}(y, \hat{y}) = \frac{\sum_i (y_i - \bar{y})(\hat{y}_i - \bar{\hat{y}})}{\sqrt{\sum_i (y_i - \bar{y})^2} \sqrt{\sum_i (\hat{y}_i - \bar{\hat{y}})^2}},
\]
while temporal descriptions are scored by Intersection over Union:
\[
\text{IoU} = \frac{\text{length}([t_1, t_2] \cap [t_1', t_2'])}{\text{length}([t_1, t_2] \cup [t_1', t_2'])}.
\]
Descriptive fidelity is additionally measured with precision, recall, and a GPT-based correlation score [2503.20290]. This establishes a task-oriented evaluation protocol for NLA: language is judged by the structured information it encodes.

In semantic-projection assessment, psychometric evaluation is central. Correlations with clinical scales provide convergent validity; split-half reliability is computed with odd–even partitioning and Spearman–Brown correction:
\[
r_{\text{SB}} = \frac{2r_{\text{half}}}{1 + r_{\text{half}}}.
\]
Observed correlations are partially or fully disattenuated using scale and projection reliability:
\[
r_{\text{partial}} = \frac{r_{\text{observed}}}{\sqrt{r_{\text{scale}}}},
\qquad
r_{\text{corrected}} = \frac{r_{\text{observed}}}{\sqrt{r_{\text{projection}} \cdot r_{\text{scale}}}}.
\]
Distributional similarity is assessed using Wasserstein distance between standardized projection and clinical-score distributions [2605.04873]. This is an unusually explicit psychometric treatment of NLA.

In L2 speech assessment with natural-language descriptors, evaluation uses Pearson and Spearman correlations between predicted and human holistic scores [2507.10200]. In multi-granular SpeechLLM assessment, sentence-, word-, and phoneme-level outputs are evaluated with PCC, RMSE, and Matthews Correlation Coefficient, while rationale quality is further analyzed through sentiment consistency and mention-based agreement [2606.09470].

For classroom discussion scoring, Quadratic Weighted Kappa is used for rubric prediction, and macro-\(F_1\) for sentence-level talk-move classification [2306.14918]. In ASD spoken language assessment, macro-\(F_1\) is used for child/adult and speech/vocalization classification, and one-way ANOVA links derived behavioral measures to clinically defined language levels [2305.14117]. In rehabilitation fidelity assessment, utterance-level cue classification is evaluated with \(F_1\), with the BERT model reaching 0.8075 internally and 0.8259 on external validation [2209.06727]. In requirements satisfaction assessment, macro \(F_\beta\) with \(\beta=2\) and MAP are used to reflect safety-critical recall requirements [2312.04463].

A plausible implication is that NLA has no single canonical metric family. Instead, the evaluation regime tends to mirror the role language plays: psychometrics when language is the measured signal, structured extraction metrics when language is the primary output, and task-specific classification or ranking metrics when language is the evidence base for downstream assessment.

## 5. Domain-specific applications

NLA has been instantiated across a notably broad range of domains.

In speech and audio, QualiSpeech reframes low-level speech quality assessment as natural-language reasoning over perceptual factors such as noise, distortion, continuity, and naturalness [2503.20290]. In L2 oral proficiency, CEFR descriptor application yields competitive zero-shot text-only assessment [2507.10200], while rubric-guided SpeechLLMs extend this to joint sentence-, word-, and phoneme-level scoring with rationales [2606.09470]. Both lines suggest that auditory assessment is increasingly moving from scalar prediction toward descriptor-rich, explainable judgments.

In mental health, semantic projection offers an unsupervised, theory-driven framework for depression- and worry-related assessment from open-ended language [2605.04873]. A separate design-oriented study converts SCL-90 into conversational dialogue with an LLM, using natural inquiry questions, a symptom matrix, and three-phase interaction design for non-invasive mental health assessment [2510.18158]. Open Brain AI operationalizes discourse- and feature-based automatic language assessment in neurogenic conditions, combining ASR, linguistic metrics, and GPT-3-class discourse analysis for multilingual spoken and written assessment [2306.06693]. Computational Language Assessment in neurodegenerative disease similarly treats language as the medium for diagnosis, prognosis, and treatment monitoring [2305.20046].

In developmental and rehabilitation contexts, speech embeddings support automated assessment of spoken language development in children with ASD by classifying child versus adult speech and speech versus nonverbal vocalization; these automatically derived counts and durations are statistically linked to clinically defined language levels [2305.14117]. In inpatient rehabilitation, therapist utterances are classified as guided or directed cues to automate strategy training fidelity assessment [2209.06727]. In both cases, NLA operates not by scoring content quality but by inferring behaviorally meaningful constructs from naturalistic language streams.

In education, automated classroom discussion assessment uses transcripts to infer instructional quality rubric scores via intermediate discourse-code prediction [2306.14918]. In argumentation, GAQCorpus supports theory-based assessment of cogency, effectiveness, reasonableness, and overall argument quality from online arguments [2006.00843]. A plausible implication is that education-focused NLA increasingly favors multi-dimensional, rubric-structured evaluation rather than monolithic scoring.

In software and systems engineering, requirements satisfaction assessment evaluates whether linked design texts semantically cover requirement texts, using transformer encoders over raw language artifacts [2312.04463]. CCG-based translation of natural-language properties into executable property-based tests makes natural language itself the source of software assessment procedures [2202.03616]. In these cases NLA functions as compliance checking or executable specification, rather than human-performance scoring.

In responsible AI evaluation, TEAL operationalizes ethical assessment of language generation models through prompt-based behavioral probing and automated scoring of outputs for toxicity, profanity, insult, threat, and group-disaggregated harms [2207.10576]. Pairwise LLM-based comparative assessment of NLG outputs similarly treats natural language as both object and instrument of evaluation [2307.07889]. These lines extend NLA beyond human subjects to machine-generated language.

Finally, in healthcare decision support, Natural Language-Assisted Multi-modal Medication Recommendation uses patient-side textual descriptions and drug-side textual descriptions, together with molecular graphs, in a joint alignment framework for combinatorial medication recommendation [2501.07166]. Although framed as recommendation rather than assessment, it exemplifies language-based assessment of both patient state and medication properties.

## 6. Reliability, interpretability, and recurring limitations

A consistent theme across NLA is that interpretability is often improved, but reliability and faithfulness remain domain-dependent.

QualiSpeech shows that fine-tuned auditory LLMs can generate accurate, temporally localized descriptions of noise and distortion, and that full comment generation can encode structured information nearly as well as specialized models while remaining human-readable [2503.20290]. Yet the same work reports that off-the-shelf auditory LLMs perform poorly on low-level quality tasks, with near-zero or negative correlations on several aspects [2503.20290]. It also notes that explicit reasoning improves assessment only when low-level features are accurate and the reasoning backbone is strong enough; Vicuna-7B remains a bottleneck [2503.20290].

In rubric-guided L2 assessment, rationales are plausible and highly self-consistent at sentence level, but word- and phoneme-level faithfulness is weak. Aspect-level mention agreement is high with respect to the model’s own labels, but phoneme-level mention correlations are very low, and token-level references are sparse [2606.09470]. This directly challenges a common assumption that generated rationales are faithful explanations. A plausible implication is that NLA systems can produce useful holistic explanations before they can produce reliable fine-grained diagnostics.

Psychological semantic projection yields strong correlations and high split-half reliability for structured response formats such as selected words, written words, and phrases, but whole-text embeddings of free-text responses perform much worse unless sentence-level aggregation is used [2605.04873]. This indicates that response format and text processing strategy are not peripheral choices; they define the measurement quality of the resulting NLA instrument.

Single-annotator or limited-annotator designs are another recurring issue. QualiSpeech uses one annotator per sample due to annotation complexity, then later studies multi-annotator subsets and finds only moderate mutual PCC for some aspects [2503.20290]. The classroom discussion study is based on 90 discussions and severe class imbalance in talk-move labels [2306.14918]. ASD spoken-language assessment uses 45 sessions, with weaker performance for minimally verbal children and some gender differences [2305.14117]. Rehabilitation cue assessment depends on careful annotator training to reach Krippendorff’s alpha around 0.73–0.79 [2209.06727]. These findings suggest that NLA often replaces cheap labels with rich labels, but at significant annotation cost.

Bias, privacy, and cultural generalization are also recurrent concerns. TEAL emphasizes dependence on imperfect toxicity classifiers and warns that scorer bias propagates into ethical assessment [2207.10576]. Mental-health conversational assessment explicitly flags privacy protection, algorithmic bias, and cross-cultural applicability as critical challenges [2510.18158]. Open Brain AI notes multilingual consistency and interpretability as unresolved issues in clinical deployment [2306.06693]. The semantic-projection framework warns that construct axes built from English-language Western instruments may not generalize cross-culturally without re-anchoring [2605.04873].

## 7. Research trajectory and open problems

Several research directions recur across the literature. Stronger backbones are an obvious one. QualiSpeech argues that reasoning-based gains are currently limited more by LLM capacity and error propagation than by the idea of reasoning itself [2503.20290]. SpeechLLM work suggests that better constraints between token-level labels and rationale text are needed if faithful explanations are desired [2606.09470]. Mental-health conversational systems similarly require more robust empathy modeling and crisis handling before clinical use [2510.18158].

A second direction is reasoning-aware evaluation. QualiSpeech explicitly critiques reliance on BLEU-, ROUGE-, or BERTScore-style metrics and instead uses content-fidelity measures extracted from natural language [2503.20290]. The same issue surfaces in rationale assessment for L2 pronunciation, where plausibility and faithfulness must be separated [2606.09470]. This suggests that NLA needs evaluation protocols that distinguish whether language sounds appropriate from whether it encodes the right evidence.

A third direction is improved construct modeling. Semantic projection already shows one path: axes grounded in psychometric theory, evaluated with reliability and attenuation corrections [2605.04873]. Theory-based argument assessment shows another: decomposing holistic quality into cogency, effectiveness, and reasonableness and exploiting relations among them in multitask models [2006.00843]. Descriptor-based oral assessment suggests that analytic descriptors can serve as a general-purpose interface between human rating theory and machine assessment [2507.10200]. A plausible implication is that future NLA systems may increasingly combine explicit construct definitions, descriptor-grounded prompting, and latent alignment methods.

A fourth direction is scale and multilinguality. QualiSpeech explicitly calls for more languages, more recording environments, and more TTS architectures [2503.20290]. Open Brain AI frames multilingual analysis as central to equitable deployment [2306.06693]. Computational Language Assessment in neurodegenerative disease likewise emphasizes multilingual NLP and acoustic features as a route beyond major-language test batteries [2305.20046]. Yet the literature also makes clear that “more languages” is not merely a data-expansion issue; it may require re-anchoring descriptors, revalidating constructs, and testing cultural invariance.

A fifth direction is integrating NLA into operational workflows. In education, classroom-discussion scoring is explicitly motivated by scalable observation and teacher feedback [2306.14918]. In rehabilitation, automated cue detection is intended for multi-site pragmatic trials where manual fidelity coding is infeasible [2209.06727]. In software engineering, transformer-based requirements satisfaction assessment is aimed at safety- and mission-critical toolchains [2312.04463]. In mental health, conversational assessment is positioned as pre-screening or adjunctive support rather than replacement for clinician judgment [2510.18158]. These cases suggest that NLA is moving from proof-of-concept toward workflow-specific decision support, but usually with explicit human oversight.

Taken together, the literature portrays Natural Language-Based Assessment as a convergent development across multiple fields: language becomes not only the object assessed, but also the scaffold for defining constructs, eliciting evidence, expressing judgments, and evaluating explanations. The strongest current results arise where this linguistic scaffolding is tied to a clear theoretical construct, a task-oriented evaluation protocol, and a domain in which natural language genuinely encodes the relevant evidence [2503.20290] [2605.04873] [2507.10200] [2606.09470]. The most persistent open problems concern faithfulness of rationales, psychometric validation, annotation cost, and the portability of descriptor- or axis-based constructs across domains, languages, and populations.

Source: https://www.emergentmind.com/topics/natural-language-based-assessment-nla