Natural Language-Based Assessment (NLA)

• Natural Language-Based Assessment is a paradigm that uses natural language to elicit, express, and interpret judgments, moving beyond simple scalar scores.
• It encompasses applications in speech quality, L2 proficiency, psychology, education, and more by leveraging descriptors, semantic projections, and generated rationales.
• NLA systems combine linguistic outputs with structured evaluation metrics to provide interpretable, theory-driven assessments across diverse domains.

Searching arXiv for recent and relevant papers on Natural Language-Based Assessment. Natural Language-Based Assessment (NLA) denotes a family of assessment paradigms in which natural language is not merely an auxiliary interface but the primary representational medium for eliciting, expressing, and evaluating judgments. Across the literature, NLA appears in several technically distinct forms: as natural-language explanation of low-level perceptual judgments in speech quality assessment, as theory-driven measurement directly from language embeddings, as rubric-guided scoring with generated rationales, and as automated inference from naturally occurring discourse in clinical, educational, software-engineering, and safety settings (Wang et al., 26 Mar 2025, Luongo et al., 6 May 2026, Bannò et al., 14 Jul 2025). A common shift across these formulations is from scalar prediction alone toward assessment artifacts that remain linguistically interpretable: descriptors, rationales, structured comments, or semantically grounded projections. This suggests that NLA is best understood not as a single model class but as a design principle for assessment systems in which language itself carries the construct definition, the evidence, or the explanatory trace.

1. Conceptual scope and formalization

In one explicit formulation, NLA is contrasted with conventional numerical assessment by the mapping from input to text rather than input to a scalar score: numerical quality assessment is written as $f(x) \rightarrow y \in \{1, \dots, 5\}$, whereas natural language assessment is written as $f(x) \rightarrow \text{text} \in \mathcal{L}$ (Wang et al., 26 Mar 2025). In that formulation, scalar scores and natural language are described as complementary rather than competing representations: a score gives a rough overall assessment, while language identifies what is wrong, where, and why (Wang et al., 26 Mar 2025).

A second formulation treats NLA as direct measurement from language rather than prediction of an external questionnaire. In the semantic-projection framework for psychological assessment, a response is embedded as a vector $x$, a construct is encoded as a semantic axis $a$, and the assessment score is the projection

$$\text{score}(x) = \frac{x \cdot a}{\lVert a \rVert}.$$

This converts natural language into a continuous psychological score without supervised training on individual-level labels (Luongo et al., 6 May 2026). In that setting, NLA is explicitly positioned against supervised text-to-score pipelines by defining the construct directly in embedding space through theory-based axes derived from lexical anchors or validated scale items (Luongo et al., 6 May 2026).

In language assessment for L2 speaking, NLA is defined as using instructions expressed as can-do descriptors, originally intended for human examiners, to determine whether LLMs can interpret and apply them comparably to human assessment (Bannò et al., 14 Jul 2025). Here the assessment object is neither a latent neural score nor a post hoc explanation, but the descriptor-grounded judgment itself. The same general idea appears in rubric-guided SpeechLLMs for L2 pronunciation, where the model jointly predicts ordinal labels and a free-text rationale in a single response (Parikh et al., 8 Jun 2026).

Across these variants, several recurrent structural motifs appear. One is descriptor-grounding: CEFR can-do descriptors, quality rubrics, or clinically motivated semantic axes define the target construct in language (Luongo et al., 6 May 2026, Bannò et al., 14 Jul 2025, Parikh et al., 8 Jun 2026). Another is reasoning-rich output: comments, chain-of-thought-style assessments, or narrative rationales are treated as part of the assessment object rather than as ancillary explanation (Wang et al., 26 Mar 2025, Parikh et al., 8 Jun 2026). A third is task-oriented evaluation of language outputs by mapping them back to structured targets, such as scores, temporal intervals, or clinically relevant dimensions (Wang et al., 26 Mar 2025).

2. From scores to descriptors, comments, and semantic axes

A central theme in NLA research is dissatisfaction with assessment systems that output only a number. In speech quality assessment, the critique is directed at MOS-style scalar ratings: they summarize perceived quality but do not reveal the reasoning behind the score and give limited guidance on how to improve a system (Wang et al., 26 Mar 2025). QualiSpeech addresses this by collecting seven scored aspects—noise, distortion, speed, continuity, listening effort, naturalness, and overall quality—together with four description-based aspects: noise description, distortion description, unnatural pause description, and voice characterization (Wang et al., 26 Mar 2025). The descriptive comment then integrates these low-level aspects into an overall judgment in natural language (Wang et al., 26 Mar 2025).

In psychological assessment, the analogous critique is that supervised LLMs trained to predict questionnaire scores may have limited interpretability and generalizability across contexts (Luongo et al., 6 May 2026). The semantic-projection alternative operationalizes constructs such as depression-related affect and worry/anxiety-related affect as contrasts between low- and high-symptom content. The axis vector is defined as

$$a = \frac{1}{m} \sum_{j=1}^{m} p_j \;-\; \frac{1}{n} \sum_{k=1}^{n} q_k,$$

with positive anchors $p_j$ and negative anchors $q_k$ derived either from lexical items or validated scale items such as CES-D, Zung, and STAI-Y (Luongo et al., 6 May 2026). This makes the assessment scale itself linguistically inspectable.

A third instantiation appears in L2 oral proficiency. There, CEFR analytic descriptors for ten aspects—general linguistic range, vocabulary range, vocabulary control, grammatical accuracy, sociolinguistic appropriateness, flexibility, thematic development, coherence and cohesion, fluency, and propositional precision—are presented to an LLM, which selects the best-fitting descriptor for each aspect (Bannò et al., 14 Jul 2025). Holistic scores are then reconstructed from those analytic decisions. This suggests a form of NLA in which natural language descriptors supply both the scoring rubric and the interpretive frame.

Rubric-guided SpeechLLM assessment extends this principle to joint multi-granular output. On SpeechOcean762, the model predicts sentence-level labels for accuracy, fluency, and prosody, word-level and phoneme-level accuracy labels, and a free-text rationale in a single generated response (Parikh et al., 8 Jun 2026). The rationale is not external to the assessment; it is part of the output format and conditioned on the same rubric definitions (Parikh et al., 8 Jun 2026).

These threads point to a broader distinction between two kinds of NLA. One kind uses language as the output format for assessment, often with reasons, descriptors, or comments. Another uses language as the measurement substrate itself, as in semantic projection. A plausible implication is that NLA spans both “assessment in language” and “assessment from language,” and the most mature systems increasingly combine the two.

3. Methodological architectures

NLA systems in the cited literature fall into several technical families.

One family is auditory-LLM generation. QualiSpeech uses fine-tuned SALMONN-7B, with Whisper and BEATs audio encoders, a Q-former connector, and a Vicuna-v1.5-7B backbone; only the speech Q-former connector and LoRA adapters on the LLM are fine-tuned in most experiments (Wang et al., 26 Mar 2025). The model is trained to generate either scores, descriptions, or full comments from audio plus prompt, and the natural-language output is then parsed back into structured items for evaluation (Wang et al., 26 Mar 2025).

A second family is text-only descriptor application by a general LLM. In L2 oral proficiency assessment, a 4-bit quantized Qwen 2.5 72B model is used in a zero-shot setting to assess ASR transcripts against CEFR descriptors (Bannò et al., 14 Jul 2025). For each aspect, logits over descriptor options are converted into a softmax distribution and then into a continuous Fair Average Score: $\text{FairAvg} = \sum_{k=1}^{K} p_k \cdot v_k.$ These analytic scores are averaged, and optionally combined via Ridge regression, to predict holistic performance (Bannò et al., 14 Jul 2025).

A third family is hybrid supervised generative modeling. The rubric-guided SpeechLLM based on Qwen2-Audio-7B-Instruct is trained with a hybrid objective that combines supervised fine-tuning and Bounded Direct Preference Optimization: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{BDPO}} + \lambda \cdot \mathcal{L}_{\text{SFT}}.$$ The model consumes speech waveform, transcript, target phoneme sequence, and rubric definitions, and generates structured labels plus a rationale (Parikh et al., 8 Jun 2026). The BDPO component is used only on label tokens, while rationales are trained through the SFT objective (Parikh et al., 8 Jun 2026).

A fourth family is unsupervised semantic geometry. In psychological assessment, Sentence-BERT embeddings and semantic projection yield continuous scores without fitting to subject-level labels (Luongo et al., 6 May 2026). For long free-text, sentence-level mean and max-absolute aggregation are introduced: $f(x) \rightarrow \text{text} \in \mathcal{L}$0 where $f(x) \rightarrow \text{text} \in \mathcal{L}$1 (Luongo et al., 6 May 2026). This gives a distinct NLA architecture in which interpretability derives from the construct axis rather than from generated explanations.

A fifth family is structured text classification or sequence labeling for downstream assessment. In classroom discussion assessment, BERT-based sentence classifiers, hierarchical classifiers, and BERT–BiLSTM–CRF sequence labeling are used to infer Analyzing Teaching Moves, which are then mapped to discussion-level IQA rubric scores (Tran et al., 2023). In rehabilitation fidelity assessment, a BERT classifier labels utterances as guided, directed, or none, enabling automated fidelity scoring from therapist language (Osterhoudt et al., 2022). In requirements satisfaction assessment, Sat-BERT and its multitask variants assess whether design texts satisfy requirement texts by semantic coverage (Poudel et al., 2023). In software testing, CCG-based semantic parsing maps English descriptions to executable property-based tests (Gordon, 2022). These are less generative than the descriptor- and rationale-based systems, but they remain NLA insofar as the assessment target is inferred from language artifacts.

4. Evaluation paradigms and metrics

Evaluation in NLA research is heterogeneous because the outputs themselves differ. Several papers explicitly avoid relying only on surface text similarity.

In QualiSpeech, the natural language output is the primary assessment object, but evaluation parses it back into structured forms. Numeric aspects are scored with Pearson Correlation Coefficient: $f(x) \rightarrow \text{text} \in \mathcal{L}$2 while temporal descriptions are scored by Intersection over Union: $f(x) \rightarrow \text{text} \in \mathcal{L}$3 Descriptive fidelity is additionally measured with precision, recall, and a GPT-based correlation score (Wang et al., 26 Mar 2025). This establishes a task-oriented evaluation protocol for NLA: language is judged by the structured information it encodes.

In semantic-projection assessment, psychometric evaluation is central. Correlations with clinical scales provide convergent validity; split-half reliability is computed with odd–even partitioning and Spearman–Brown correction: $f(x) \rightarrow \text{text} \in \mathcal{L}$4 Observed correlations are partially or fully disattenuated using scale and projection reliability: $f(x) \rightarrow \text{text} \in \mathcal{L}$5 Distributional similarity is assessed using Wasserstein distance between standardized projection and clinical-score distributions (Luongo et al., 6 May 2026). This is an unusually explicit psychometric treatment of NLA.

In L2 speech assessment with natural-language descriptors, evaluation uses Pearson and Spearman correlations between predicted and human holistic scores (Bannò et al., 14 Jul 2025). In multi-granular SpeechLLM assessment, sentence-, word-, and phoneme-level outputs are evaluated with PCC, RMSE, and Matthews Correlation Coefficient, while rationale quality is further analyzed through sentiment consistency and mention-based agreement (Parikh et al., 8 Jun 2026).

For classroom discussion scoring, Quadratic Weighted Kappa is used for rubric prediction, and macro-$f(x) \rightarrow \text{text} \in \mathcal{L}$6 for sentence-level talk-move classification (Tran et al., 2023). In ASD spoken language assessment, macro-$f(x) \rightarrow \text{text} \in \mathcal{L}$7 is used for child/adult and speech/vocalization classification, and one-way ANOVA links derived behavioral measures to clinically defined language levels (Xu et al., 2023). In rehabilitation fidelity assessment, utterance-level cue classification is evaluated with $f(x) \rightarrow \text{text} \in \mathcal{L}$8, with the BERT model reaching 0.8075 internally and 0.8259 on external validation (Osterhoudt et al., 2022). In requirements satisfaction assessment, macro $f(x) \rightarrow \text{text} \in \mathcal{L}$9 with $x$0 and MAP are used to reflect safety-critical recall requirements (Poudel et al., 2023).

A plausible implication is that NLA has no single canonical metric family. Instead, the evaluation regime tends to mirror the role language plays: psychometrics when language is the measured signal, structured extraction metrics when language is the primary output, and task-specific classification or ranking metrics when language is the evidence base for downstream assessment.

5. Domain-specific applications

NLA has been instantiated across a notably broad range of domains.

In speech and audio, QualiSpeech reframes low-level speech quality assessment as natural-language reasoning over perceptual factors such as noise, distortion, continuity, and naturalness (Wang et al., 26 Mar 2025). In L2 oral proficiency, CEFR descriptor application yields competitive zero-shot text-only assessment (Bannò et al., 14 Jul 2025), while rubric-guided SpeechLLMs extend this to joint sentence-, word-, and phoneme-level scoring with rationales (Parikh et al., 8 Jun 2026). Both lines suggest that auditory assessment is increasingly moving from scalar prediction toward descriptor-rich, explainable judgments.

In mental health, semantic projection offers an unsupervised, theory-driven framework for depression- and worry-related assessment from open-ended language (Luongo et al., 6 May 2026). A separate design-oriented study converts SCL-90 into conversational dialogue with an LLM, using natural inquiry questions, a symptom matrix, and three-phase interaction design for non-invasive mental health assessment (Cai et al., 20 Oct 2025). Open Brain AI operationalizes discourse- and feature-based automatic language assessment in neurogenic conditions, combining ASR, linguistic metrics, and GPT-3-class discourse analysis for multilingual spoken and written assessment (Themistocleous, 2023). Computational Language Assessment in neurodegenerative disease similarly treats language as the medium for diagnosis, prognosis, and treatment monitoring (Themistocleous et al., 2023).

In developmental and rehabilitation contexts, speech embeddings support automated assessment of spoken language development in children with ASD by classifying child versus adult speech and speech versus nonverbal vocalization; these automatically derived counts and durations are statistically linked to clinically defined language levels (Xu et al., 2023). In inpatient rehabilitation, therapist utterances are classified as guided or directed cues to automate strategy training fidelity assessment (Osterhoudt et al., 2022). In both cases, NLA operates not by scoring content quality but by inferring behaviorally meaningful constructs from naturalistic language streams.

In education, automated classroom discussion assessment uses transcripts to infer instructional quality rubric scores via intermediate discourse-code prediction (Tran et al., 2023). In argumentation, GAQCorpus supports theory-based assessment of cogency, effectiveness, reasonableness, and overall argument quality from online arguments (Lauscher et al., 2020). A plausible implication is that education-focused NLA increasingly favors multi-dimensional, rubric-structured evaluation rather than monolithic scoring.

In software and systems engineering, requirements satisfaction assessment evaluates whether linked design texts semantically cover requirement texts, using transformer encoders over raw language artifacts (Poudel et al., 2023). CCG-based translation of natural-language properties into executable property-based tests makes natural language itself the source of software assessment procedures (Gordon, 2022). In these cases NLA functions as compliance checking or executable specification, rather than human-performance scoring.

In responsible AI evaluation, TEAL operationalizes ethical assessment of language generation models through prompt-based behavioral probing and automated scoring of outputs for toxicity, profanity, insult, threat, and group-disaggregated harms (Rasekh et al., 2022). Pairwise LLM-based comparative assessment of NLG outputs similarly treats natural language as both object and instrument of evaluation (Liusie et al., 2023). These lines extend NLA beyond human subjects to machine-generated language.

Finally, in healthcare decision support, Natural Language-Assisted Multi-modal Medication Recommendation uses patient-side textual descriptions and drug-side textual descriptions, together with molecular graphs, in a joint alignment framework for combinatorial medication recommendation (Tan et al., 13 Jan 2025). Although framed as recommendation rather than assessment, it exemplifies language-based assessment of both patient state and medication properties.

6. Reliability, interpretability, and recurring limitations

A consistent theme across NLA is that interpretability is often improved, but reliability and faithfulness remain domain-dependent.

QualiSpeech shows that fine-tuned auditory LLMs can generate accurate, temporally localized descriptions of noise and distortion, and that full comment generation can encode structured information nearly as well as specialized models while remaining human-readable (Wang et al., 26 Mar 2025). Yet the same work reports that off-the-shelf auditory LLMs perform poorly on low-level quality tasks, with near-zero or negative correlations on several aspects (Wang et al., 26 Mar 2025). It also notes that explicit reasoning improves assessment only when low-level features are accurate and the reasoning backbone is strong enough; Vicuna-7B remains a bottleneck (Wang et al., 26 Mar 2025).

In rubric-guided L2 assessment, rationales are plausible and highly self-consistent at sentence level, but word- and phoneme-level faithfulness is weak. Aspect-level mention agreement is high with respect to the model’s own labels, but phoneme-level mention correlations are very low, and token-level references are sparse (Parikh et al., 8 Jun 2026). This directly challenges a common assumption that generated rationales are faithful explanations. A plausible implication is that NLA systems can produce useful holistic explanations before they can produce reliable fine-grained diagnostics.

Psychological semantic projection yields strong correlations and high split-half reliability for structured response formats such as selected words, written words, and phrases, but whole-text embeddings of free-text responses perform much worse unless sentence-level aggregation is used (Luongo et al., 6 May 2026). This indicates that response format and text processing strategy are not peripheral choices; they define the measurement quality of the resulting NLA instrument.

Single-annotator or limited-annotator designs are another recurring issue. QualiSpeech uses one annotator per sample due to annotation complexity, then later studies multi-annotator subsets and finds only moderate mutual PCC for some aspects (Wang et al., 26 Mar 2025). The classroom discussion study is based on 90 discussions and severe class imbalance in talk-move labels (Tran et al., 2023). ASD spoken-language assessment uses 45 sessions, with weaker performance for minimally verbal children and some gender differences (Xu et al., 2023). Rehabilitation cue assessment depends on careful annotator training to reach Krippendorff’s alpha around 0.73–0.79 (Osterhoudt et al., 2022). These findings suggest that NLA often replaces cheap labels with rich labels, but at significant annotation cost.

Bias, privacy, and cultural generalization are also recurrent concerns. TEAL emphasizes dependence on imperfect toxicity classifiers and warns that scorer bias propagates into ethical assessment (Rasekh et al., 2022). Mental-health conversational assessment explicitly flags privacy protection, algorithmic bias, and cross-cultural applicability as critical challenges (Cai et al., 20 Oct 2025). Open Brain AI notes multilingual consistency and interpretability as unresolved issues in clinical deployment (Themistocleous, 2023). The semantic-projection framework warns that construct axes built from English-language Western instruments may not generalize cross-culturally without re-anchoring (Luongo et al., 6 May 2026).

7. Research trajectory and open problems

Several research directions recur across the literature. Stronger backbones are an obvious one. QualiSpeech argues that reasoning-based gains are currently limited more by LLM capacity and error propagation than by the idea of reasoning itself (Wang et al., 26 Mar 2025). SpeechLLM work suggests that better constraints between token-level labels and rationale text are needed if faithful explanations are desired (Parikh et al., 8 Jun 2026). Mental-health conversational systems similarly require more robust empathy modeling and crisis handling before clinical use (Cai et al., 20 Oct 2025).

A second direction is reasoning-aware evaluation. QualiSpeech explicitly critiques reliance on BLEU-, ROUGE-, or BERTScore-style metrics and instead uses content-fidelity measures extracted from natural language (Wang et al., 26 Mar 2025). The same issue surfaces in rationale assessment for L2 pronunciation, where plausibility and faithfulness must be separated (Parikh et al., 8 Jun 2026). This suggests that NLA needs evaluation protocols that distinguish whether language sounds appropriate from whether it encodes the right evidence.

A third direction is improved construct modeling. Semantic projection already shows one path: axes grounded in psychometric theory, evaluated with reliability and attenuation corrections (Luongo et al., 6 May 2026). Theory-based argument assessment shows another: decomposing holistic quality into cogency, effectiveness, and reasonableness and exploiting relations among them in multitask models (Lauscher et al., 2020). Descriptor-based oral assessment suggests that analytic descriptors can serve as a general-purpose interface between human rating theory and machine assessment (Bannò et al., 14 Jul 2025). A plausible implication is that future NLA systems may increasingly combine explicit construct definitions, descriptor-grounded prompting, and latent alignment methods.

A fourth direction is scale and multilinguality. QualiSpeech explicitly calls for more languages, more recording environments, and more TTS architectures (Wang et al., 26 Mar 2025). Open Brain AI frames multilingual analysis as central to equitable deployment (Themistocleous, 2023). Computational Language Assessment in neurodegenerative disease likewise emphasizes multilingual NLP and acoustic features as a route beyond major-language test batteries (Themistocleous et al., 2023). Yet the literature also makes clear that “more languages” is not merely a data-expansion issue; it may require re-anchoring descriptors, revalidating constructs, and testing cultural invariance.

A fifth direction is integrating NLA into operational workflows. In education, classroom-discussion scoring is explicitly motivated by scalable observation and teacher feedback (Tran et al., 2023). In rehabilitation, automated cue detection is intended for multi-site pragmatic trials where manual fidelity coding is infeasible (Osterhoudt et al., 2022). In software engineering, transformer-based requirements satisfaction assessment is aimed at safety- and mission-critical toolchains (Poudel et al., 2023). In mental health, conversational assessment is positioned as pre-screening or adjunctive support rather than replacement for clinician judgment (Cai et al., 20 Oct 2025). These cases suggest that NLA is moving from proof-of-concept toward workflow-specific decision support, but usually with explicit human oversight.

Taken together, the literature portrays Natural Language-Based Assessment as a convergent development across multiple fields: language becomes not only the object assessed, but also the scaffold for defining constructs, eliciting evidence, expressing judgments, and evaluating explanations. The strongest current results arise where this linguistic scaffolding is tied to a clear theoretical construct, a task-oriented evaluation protocol, and a domain in which natural language genuinely encodes the relevant evidence (Wang et al., 26 Mar 2025, Luongo et al., 6 May 2026, Bannò et al., 14 Jul 2025, Parikh et al., 8 Jun 2026). The most persistent open problems concern faithfulness of rationales, psychometric validation, annotation cost, and the portability of descriptor- or axis-based constructs across domains, languages, and populations.