AROA: Argument Rarity-based Originality Assessment

• AROA is a computational framework that quantifies originality by measuring the statistical rarity of argumentative elements within a reference corpus.
• It employs retrieval-augmented LLMs and semantic clustering to assess claims, evidence, structure, and cognitive depth in creative ideas and essays.
• The framework is validated through robust psychometric metrics, scalability tests, and cross-model comparisons, providing actionable insights for novelty evaluation.

Argument Rarity-based Originality Assessment (AROA) is a computational framework for quantifying the originality of arguments, ideas, or essays by measuring the statistical rarity of their constituent elements in relation to a population reference corpus. This paradigm treats originality as infrequency—how seldom a given claim, evidence, structure, or reasoning “move” appears within a relevant population sample—thereby operationalizing originality in a scalable, automated, and psychometrically interpretable manner. AROA has been instantiated both as a general pipeline for creative idea assessment using retrieval-augmented LLMs (Bangash et al., 22 May 2025) and as a targeted suite for evaluating argumentative originality in writing, with component-wise rarity and orthogonal quality adjustment (&&&1&&&).

1. Theoretical Basis and Formal Definition

AROA defines originality as the relative rarity of argumentative elements within a domain-constrained reference corpus. Let $x$ denote a text (idea, essay, or argument) and $\mathcal{D} = \{x_i\}_{i=1}^N$ a corpus of peer texts on the same topic or task. For essays, four principal dimensions are extracted and scored: structural rarity ($S_{\mathrm{struct}}$), claim rarity ($S_{\mathrm{claim}}$), evidence rarity ($S_{\mathrm{evid}}$), and cognitive depth ($S_{\mathrm{move}}$). Each is computed and then z-score normalized:

$$\tilde S_{\mathrm{comp}}(x) = \frac{S_{\mathrm{comp}}(x) - \mu_{\mathrm{comp}}}{\sigma_{\mathrm{comp}}}$$

for each component. The overall originality score combines these (with equal weights $\lambda_i = 0.25$):

$$\tilde S_{\mathrm{orig}}(x) = \sum_{i=1}^4 \lambda_i \tilde S_i(x)$$

A logical quality score $Q(x)$ (the average of LLM-graded coherence and logicality) is applied multiplicatively:

$$S_{\mathrm{final}}(x) = \tilde S_{\mathrm{orig}}(x) \times Q(x)$$

This coupling ensures that only high-quality, rare argumentative phenomena are rewarded as original (Inoshita et al., 2 Feb 2026).

In alternative instantiations (e.g., divergent thinking tasks, MuseRAG), originality is derived from the frequency of semantically clustered ideas (buckets), supporting a direct count-based approach (Bangash et al., 22 May 2025).

2. Methodological Pipeline

AROA is implemented through explicit computational pipelines, most notably in MuseRAG and the AROA essay framework.

a. MuseRAG: Idea Bucketing via Retrieval-Augmented Generation

The MuseRAG system receives a stream of ideas $x$ and constructs a task-specific codebook of buckets $B_k$, where each bucket aggregates semantically equivalent ideas:

• Embedding & Retrieval: Each idea and existing bucket centroid are embedded (e.g., via pre-trained sentence transformers) and a $K$-nearest-neighbor search retrieves a candidate set $\mathcal{D}_x$ for possible bucket assignment.
• LLM-as-Judge: An LLM (zero-shot, optionally with chain-of-thought, CoT) is tasked with determining if $x$ paraphrases any candidate—or warrants a new bucket. Assignments update counts $m_k$.
• Algorithm: Iteratively, all ideas are bucketed, enabling downstream calculation of rarity metrics based on bucket frequencies.

Sample pseudocode (from (Bangash et al., 22 May 2025)):

for each x in X: D_x = retrieve_K_nearest_buckets(x) k_star = LLM_judge(x, D_x) if k_star != -1: assign x to B_{k_star} else: K += 1 create new bucket B_K with x

b. Argument Extraction and Rarity Computation in Essays

The AROA writing framework leverages LLMs and embedding models as follows (Inoshita et al., 2 Feb 2026):

• Argument Extraction: Segmentation of text into claims ($C(x)$), evidence ($E(x)$), reasons, counterarguments, rebuttals—using rule-based or LLM methods.
• Feature Engineering: Construction of a 12-dimensional structure vector $\boldsymbol{\phi}(x)$; semantic embeddings via Sentence-BERT for claims/evidence.
• Density Estimation: For structural rarity, reduction to three principal components, followed by Gaussian KDE; for semantic rarity, local density via cosine similarity among nearest neighbors (e.g., $K = 5$).
• Cognitive Depth: Based on counter-argument/rebuttal matching, argumentative depth, and coverage.
• Normalization and Integration: Z-score normalization and weighted aggregation, orthogonally coupled with LLM-based logical quality.

3. Rarity-based Metrics and Mathematical Formulations

The core measurement principle is infrequency as a proxy for originality. Several complementary rarity metrics are defined:

a. Frequency-based Scores (MuseRAG)

• Relative Rarity: $\mathrm{rarity}(B_k) = 1 - \frac{m_k}{N}$
• Shapley-inspired: $\mathrm{shapley}(B_k) = \frac{1}{m_k}$
• Singleton Bonus (Uniqueness): $$\mathrm{uniqueness}(x) = \mathbf{1}\{m_{k(x)} = 1\}$$
• Thresholded: Tiered scoring by frequency rank.

b. Information-Theoretic Alternative

• For bucket $k$ with frequency $p_k = \frac{m_k}{N}$:

$O(x) = -\log p_{k(x)}$

c. Essay-Specific Semantic Rarity

• Claim/Evidence Density: For embedding $\mathbf{v}(c_k)$:

$$\mathrm{density}(\mathbf{v}(c_k)) = \frac{1}{K} \sum_{j=1}^K \mathrm{sim}(\mathbf{v}(c_k), \mathbf{v}^{(j)})$$

• Rarity Score: $$\mathrm{Rar}_C(c_k) = 1 - \mathrm{density}(\mathbf{v}(c_k))$$
• Essay-level Claim Rarity: $$S_{\mathrm{claim}}(x) = \frac{1}{|C(x)|} \sum_{c_k\in C(x)} \mathrm{Rar}_C(c_k)$$

4. Empirical Findings and Psychometric Validation

a. Agreement with Human Judgment

MuseRAG achieves adjusted mutual information (AMI) of $0.59 \pm 0.05$ against human clusterings (human-human AMI ≈ 0.66) and strong participant-level scoring agreement (Pearson $r = 0.88 \pm 0.04$, ICC $=0.88$) (Bangash et al., 22 May 2025).

b. Quality–Originality Trade-off

In argumentative writing, there is a robust negative correlation between logical quality and the rarity of claims ($\mathrm{corr}(Q, S_{\mathrm{claim}}) = -0.67$) and evidence ($\mathrm{corr}(Q, S_{\mathrm{evid}}) = -0.50$). Structural and cognitive depth components show near-zero correlation with quality ($\mathrm{corr}$ ≈ –0.10, –0.08, respectively) (Inoshita et al., 2 Feb 2026). This indicates that high-quality texts tend to employ familiar argumentative content.

c. Human vs. AI Comparison

AI-generated essays (GPT-4.1-mini, Gemini 2.5, Claude 3) achieve similar or higher structure and depth scores but have much lower claim/evidence rarity than human essays (Cohen’s $d = 2.24$ for claims, $d = 1.34$ for evidence; $p < 0.001$ for both). Quality scores for AI are near-perfect ($0.998 \pm 0.021$) versus humans ($0.609 \pm 0.141$), but this does not translate to argument originality (Inoshita et al., 2 Feb 2026).

d. Cross-Model Robustness

Evidence rarity is found to be the most transfer-stable across LLMs (Pearson $r = 0.62$), while structural and quality judgments show greater variance (structure $r = 0.23$; quality $r = 0.25$), suggesting that rarity-based metrics anchored in semantic content are more consistent than those based on rhetorical features or subjective coherence (Inoshita et al., 2 Feb 2026).

5. Scalability, Efficiency, and Extensions

AROA pipelines scale to large corpora: MuseRAG processes over 16,000 ideas in approximately 100 GPU-days (single RTX 3070 Ti plus CPUs), with per-idea LLM-call latency of ∼1–3 seconds; retrieval and embedding steps are orders of magnitude faster (Bangash et al., 22 May 2025). The overall architecture is stateless with respect to the LLM, supporting distributed scaling.

The methodology has demonstrated robustness to corpus size (Spearman $\rho > 0.80$ at $n \approx 300$ essays; $\rho \approx 0.50$ at classroom scale $n=30$–50) (Inoshita et al., 2 Feb 2026).

Potential improvements include batching multiple ideas per LLM prompt, adaptive candidate extraction, and multi-step reasoning for ambiguous assignments (Bangash et al., 22 May 2025).

AROA is directly extendable to argument mining (analyzing rarity of rhetorical function in debates), design ideation, and qualitative thematic analysis, by replacing domain elements and clustering strategies accordingly (Bangash et al., 22 May 2025).

6. Interpretive Implications and Research Directions

The rarity-based paradigm recalibrates educational and creative assessment from a strictly quality-centric model to a dual-axis regime that simultaneously values logical soundness and originality. Human raters incur bias and fatigue; automated AROA provides consistent, large-scale, and cost-effective measurements (\$0.0024 per essay, ∼1.5s per instance) (Inoshita et al., 2 Feb 2026).

Both studies highlight that LLMs are proficient at emulating human-like argumentative structure but are less proficient at generating original claims or evidential support. The statistical decoupling of quality and originality allows for nuanced feedback and the potential for formative assessment targeting truly novel argumentation strategies.

Future research avenues include validating AROA against expert-generated originality annotations, testing domain and genre generalization, developing formative feedback outputs indicating rare elements or structural novelties, ensembling multiple LLMs for more robust scoring, and conducting longitudinal studies tracking the development of originality over time (Inoshita et al., 2 Feb 2026).

7. Related Frameworks and Positioning

While frequency-based originality assessment has roots in creativity research, AROA’s innovations lie in the full automation, psychometric validation, and fine-grained decomposition of argumentative originality across both structural and semantic dimensions. Unlike prior manual or purely clustering-based approaches, AROA leverages retrieval-augmented LLM pipelines for semantically aligned bucketing and fine component-wise rarity analysis (Bangash et al., 22 May 2025), integrating orthogonal quality adjustment to robustly differentiate between logically coherent and genuinely original argumentation (Inoshita et al., 2 Feb 2026).