OCSAI Tool: Scoring Linguistic Creativity

• OCSAI Tool is an automatic scoring system for creativity tasks that evaluates responses on originality, elaboration, and flexibility using semantic distances and GPT-4 integration.
• It normalizes originality scores from a 1–5 scale to 0–1 and measures elaboration based on word count while flexibility is computed manually from pairwise comparisons.
• Empirical results indicate that large language models outperform human participants in originality and flexibility, highlighting the tool’s effectiveness in structured creativity assessment.

Searching arXiv for the OCSAI Tool and related paper. OCSAI Tool, expanded in the cited study as Open Creativity Scoring with Artificial Intelligence, is an automatic scoring system for creativity tasks that was used to evaluate linguistic responses from humans and LLMs across word-formation and metaphor tasks. In the reported application, OCSAI v1.6 serves as the operational mechanism for quantifying Originality, Elaboration, and Flexibility, with the study describing it as a tool “specifically designed to score creativity tasks,” built on semantic distances and integration of GPT-4, and cited as having a high correlation with human judgment (Dinu et al., 16 Jul 2025).

1. Definition and scoring function

In the study, OCSAI is introduced as the central automatic evaluator for a comparative linguistic creativity benchmark. Its role is not to generate answers, but to score them according to three creativity criteria derived from the classical four-part framework of Originality, Flexibility, Elaboration, and Fluency. The paper states that Fluency is excluded in this setup because each item requires exactly three answers, so OCSAI is used only for the other three dimensions (Dinu et al., 16 Jul 2025).

The three scored dimensions are defined as follows. Originality concerns the uniqueness of the given answers. Flexibility concerns how semantically distant the given answers are. Elaboration concerns the degree of detail of the responses. In this formulation, OCSAI functions as a semantic creativity rater rather than a purely surface-form evaluator. The study explicitly states that originality is scored with the full prompt option and the metaphor task type, and that OCSAI returns originality scores on a 1–5 scale, later normalized to the 0–1 interval, where 0 denotes the least original output.

The paper further states that elaboration is assessed “by OCSAI via word count limit.” This indicates that elaboration is operationalized through response length or detail rather than through an independent semantic richness formalism. A plausible implication is that, within this experimental design, elaboration is the most structurally constrained of the three criteria, because each participant is limited to short responses of up to five words.

2. Operationalization of Originality, Elaboration, and Flexibility

Originality is computed directly by OCSAI from prompt–answer pairs. The study reports that the evaluators uploaded an Excel sheet containing the prompt and corresponding answer, after which OCSAI produced a score on the 1–5 scale. The normalization to 0–1 is part of the study’s analytic pipeline, not a separate scoring model. The paper does not reproduce OCSAI’s internal originality formula, but it explicitly links the tool to semantic distance and GPT-4 integration, indicating that originality is treated as a semantic rather than merely lexical property (Dinu et al., 16 Jul 2025).

Elaboration is handled differently. The paper states that OCSAI evaluates elaboration “via word count limit.” Since each response is restricted to up to five words, the measurable range of elaboration is intentionally narrow. This helps explain why elaboration behaves differently from originality and flexibility in the comparative results discussed later.

Flexibility is not directly output by OCSAI. Instead, the study constructs flexibility manually from OCSAI pairwise scores. For each item, each respondent provides three answers, and all answer pairs are scored. If the three answers are $a_1, a_2, a_3$, the paper describes the pair set as

$\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.$

This suggests that the effective flexibility score can be written as

$F=\frac{1}{3}\sum_{i<j}\mathrm{OCSAI}(a_i,a_j),$

with flexibility interpreted as the mean semantic distance across the three within-item answer pairs. The study presents this as a manual construction from OCSAI outputs rather than as a native OCSAI metric.

3. Task design and experimental use

The linguistic creativity benchmark to which OCSAI was applied consists of 8 tasks total, divided into two major domains: word formation and metaphorical language. Each domain contains four tasks, each task contains two items, and participants are required to produce three original answers per item, with each answer consisting of up to five words (Dinu et al., 16 Jul 2025).

The word-formation section includes four task types: combining two words to form a new one describing a concept; completing a given word with another word to form a compound; continuing a series of derived words using the same formation pattern; and inventing words that fit a semantic field. The metaphorical-language section likewise includes four task types: describing familiar ideas in new metaphorical ways; filling in blanks to create an original metaphorical meaning; continuing words with unusual or funny opposites; and producing three versions of an expression with different connotative tone, namely a harsher or pejorative expression, an equivalent expression, and a milder or appreciative expression.

The study applies OCSAI to responses from 24 human respondents and 24 LLMs. The human participants are described as volunteer students in Humanities, all English majors, all non-native English speakers, with B2 and above proficiency, aged 19–25, and distributed as 8 males and 16 females. The test was administered in English, through Google Forms, in controlled classroom settings, and under the supervision of the tutor. The LLM group consisted of 24 models, of which 20 were run with default settings and 4 with maximum top-p and temperature, where available.

The dataset size is reported explicitly as

$$8 \text{ tasks} \times 2 \text{ items} \times 3 \text{ answers} \times 24 \text{ individuals} = 1152$$

per group, and thus

$1152 \times 2 = 2304$

total answers. The study also reports 3904 total words, with 2010 from LLMs and 1894 from humans. Before scoring, the authors removed extraneous punctuation, quotation marks, numbers, and em-dash lines.

4. Comparative results obtained with OCSAI

The main empirical finding reported with OCSAI is that LLMs outperformed humans overall, and did so on most tasks. The overall mean total score is 0.52 for LLMs and 0.48 for humans. At the criterion level, the LLM group outperformed humans in Originality, Flexibility, and Total score, while the difference in Elaboration was smaller and not statistically significant (Dinu et al., 16 Jul 2025).

The reported inferential statistics are as follows. For Total score, the paper gives $t=-3.03$ and $p=0.0044$. For Originality, it gives $t=-4.21$ and $p=0.00014$. For Elaboration, it gives $t=-1.15$ and $\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.$0. For Flexibility, it gives $\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.$1 and $\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.$2. The study also reports an OLS regression over all 1152 scores, with a significant group effect at $\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.$3, and states that LLMs scored on average 0.037 points higher than humans.

At the task level, LLMs performed better in six of the eight tasks. The paper identifies two exceptions. In the Semantic Field task, humans and LLMs were essentially tied. In the Euphemism/Dysphemism task, humans slightly outperformed LLMs, with 0.33 versus 0.29.

The results therefore show that OCSAI, in the study’s configuration, favors the LLM group on measures of semantically structured novelty and response diversity. The non-significant elaboration difference is consistent with the experimental constraint that each answer was limited to five words and that elaboration was measured through a word-count-based mechanism.

5. Uniqueness, interpretation, and creativity types

The paper supplements OCSAI scores with a separate uniqueness analysis, explicitly distinguished from OCSAI’s own scoring. For each of the 48 individuals, all answers were grouped into a separate file; NLTK was used for stop-word removal; spaCy embeddings were used to compute pairwise semantic similarity; and uniqueness was defined as the inverse of average semantic similarity with all other individuals. The study reports that there is no strong global separation between humans and LLMs on this measure (Dinu et al., 16 Jul 2025).

The ranking distribution is described as slightly skewed: 15 humans and 9 LLMs appear in the top half, while 15 LLMs and 9 humans appear in the lower half. The most unique and least unique individuals were both humans. The paper highlights an especially important case: Human 18 was the most unique by this metric, yet was not among the highest OCSAI scorers. This distinction is central to the study’s interpretation. OCSAI originality measures creativity under the tool’s semantic scoring framework, whereas uniqueness measures divergence from the rest of the dataset.

The manual qualitative analysis in the paper introduces Sampson’s distinction between F-creativity and E-creativity. The study suggests that LLMs tend toward F-creativity, characterized as fixed, rule-based creativity within the language system, whereas humans tend toward E-creativity, characterized as extending creativity that bends or stretches the language system. The authors describe LLM outputs as more likely to follow morphological rules closely, produce systematic compounds and derivations, and reuse common templates, while human outputs more often combine semantically distant items, violate or stretch morphological expectations, and display playful, pragmatic, or humorous constructions.

This yields a nuanced interpretation of OCSAI’s outcome. The automatic scores favor LLMs in originality and flexibility, but the manual analysis suggests that humans may still display more boundary-pushing or pragmatically loaded forms of creativity. The paper therefore does not treat OCSAI as exhausting the concept of creativity; rather, it treats OCSAI as a semantically driven instrument that captures one structured aspect of creative performance.

6. Methodological constraints and reuse considerations

Several constraints in the study are important for interpreting OCSAI Tool as an evaluation instrument. First, Fluency was intentionally excluded, because every participant had to provide exactly three answers per item. Second, Flexibility is not directly provided by OCSAI and had to be constructed manually from pairwise answer comparisons. Third, originality scoring depends on the specific OCSAI configuration: the evaluators used the full prompt option and the metaphor task type (Dinu et al., 16 Jul 2025).

Fourth, the reported originality values were normalized from 1–5 to 0–1, so direct comparison with other uses of OCSAI requires matching that normalization. Fifth, elaboration is described as essentially length-based, which means it should not be conflated with a broader theory of semantic richness. Sixth, the human participants were non-native English speakers, a fact that limits the generality of human–LLM comparison. Seventh, the human condition involved controlled classroom administration without digital aid, which strengthens experimental control but also distinguishes human production conditions from unconstrained model inference.

The paper also makes clear that OCSAI-based scoring is semantically driven, not purely form-driven, because the tool is described as relying on semantic distances and GPT-4 integration. This suggests that OCSAI is sensitive to semantic relations among answers, but also that its judgments depend on the properties of the underlying semantic model. A plausible implication is that OCSAI is better understood as an automatic evaluator of semantically structured creativity than as a complete theory of linguistic creativity.

In this sense, OCSAI Tool occupies a specific methodological position. It provides a scalable and automatic mechanism for scoring Originality, Elaboration, and indirectly Flexibility in constrained creativity tasks. At the same time, the reported study treats its outputs as analytically informative but not exhaustive, supplementing them with uniqueness analysis and manual interpretation to distinguish systematic creativity from more idiosyncratic, rule-bending forms of invention.