---
title: 'OCSAI Tool: Scoring Linguistic Creativity'
url: https://www.emergentmind.com/topics/ocsai-tool
type: topic
---

# OCSAI Tool: Scoring Linguistic Creativity

Searching arXiv for the OCSAI Tool and related paper.
OCSAI Tool, expanded in the cited study as **Open Creativity Scoring with Artificial Intelligence**, is an automatic scoring system for creativity tasks that was used to evaluate linguistic responses from humans and large language models across word-formation and metaphor tasks. In the reported application, OCSAI v1.6 serves as the operational mechanism for quantifying **Originality**, **Elaboration**, and **Flexibility**, with the study describing it as a tool “specifically designed to score creativity tasks,” built on **semantic distances** and **integration of GPT-4**, and cited as having a **high correlation with human judgment** [2507.12039].

## 1. Definition and scoring function

In the study, OCSAI is introduced as the central automatic evaluator for a comparative linguistic creativity benchmark. Its role is not to generate answers, but to score them according to three creativity criteria derived from the classical four-part framework of **Originality, Flexibility, Elaboration, and Fluency**. The paper states that **Fluency is excluded** in this setup because each item requires exactly **three answers**, so OCSAI is used only for the other three dimensions [2507.12039].

The three scored dimensions are defined as follows. **Originality** concerns the uniqueness of the given answers. **Flexibility** concerns how semantically distant the given answers are. **Elaboration** concerns the degree of detail of the responses. In this formulation, OCSAI functions as a semantic creativity rater rather than a purely surface-form evaluator. The study explicitly states that originality is scored with the **full prompt option** and the **metaphor task type**, and that OCSAI returns originality scores on a **1–5 scale**, later **normalized to the 0–1 interval**, where 0 denotes the least original output.

The paper further states that elaboration is assessed “by OCSAI via word count limit.” This indicates that elaboration is operationalized through response length or detail rather than through an independent semantic richness formalism. A plausible implication is that, within this experimental design, elaboration is the most structurally constrained of the three criteria, because each participant is limited to short responses of up to five words.

## 2. Operationalization of Originality, Elaboration, and Flexibility

Originality is computed directly by OCSAI from prompt–answer pairs. The study reports that the evaluators uploaded an Excel sheet containing the prompt and corresponding answer, after which OCSAI produced a score on the **1–5** scale. The normalization to **0–1** is part of the study’s analytic pipeline, not a separate scoring model. The paper does not reproduce OCSAI’s internal originality formula, but it explicitly links the tool to **semantic distance** and **GPT-4 integration**, indicating that originality is treated as a semantic rather than merely lexical property [2507.12039].

Elaboration is handled differently. The paper states that OCSAI evaluates elaboration “via word count limit.” Since each response is restricted to **up to five words**, the measurable range of elaboration is intentionally narrow. This helps explain why elaboration behaves differently from originality and flexibility in the comparative results discussed later.

Flexibility is not directly output by OCSAI. Instead, the study constructs flexibility manually from OCSAI pairwise scores. For each item, each respondent provides three answers, and all answer pairs are scored. If the three answers are \(a_1, a_2, a_3\), the paper describes the pair set as
\[
\{(a_1,a_2), (a_1,a_3), (a_2,a_3)\}.
\]
This suggests that the effective flexibility score can be written as
\[
F=\frac{1}{3}\sum_{i<j}\mathrm{OCSAI}(a_i,a_j),
\]
with flexibility interpreted as the mean semantic distance across the three within-item answer pairs. The study presents this as a manual construction from OCSAI outputs rather than as a native OCSAI metric.

## 3. Task design and experimental use

The linguistic creativity benchmark to which OCSAI was applied consists of **8 tasks total**, divided into **two major domains**: **word formation** and **metaphorical language**. Each domain contains **four tasks**, each task contains **two items**, and participants are required to produce **three original answers per item**, with each answer consisting of **up to five words** [2507.12039].

The word-formation section includes four task types: combining two words to form a new one describing a concept; completing a given word with another word to form a compound; continuing a series of derived words using the same formation pattern; and inventing words that fit a semantic field. The metaphorical-language section likewise includes four task types: describing familiar ideas in new metaphorical ways; filling in blanks to create an original metaphorical meaning; continuing words with unusual or funny opposites; and producing three versions of an expression with different connotative tone, namely a harsher or pejorative expression, an equivalent expression, and a milder or appreciative expression.

The study applies OCSAI to responses from **24 human respondents** and **24 LLMs**. The human participants are described as volunteer **students in Humanities**, all **English majors**, all **non-native English speakers**, with **B2 and above** proficiency, aged **19–25**, and distributed as **8 males** and **16 females**. The test was administered in **English**, through **Google Forms**, in **controlled classroom settings**, and under the **supervision of the tutor**. The LLM group consisted of **24 models**, of which **20** were run with **default settings** and **4** with **maximum top-p and temperature**, where available.

The dataset size is reported explicitly as
\[
8 \text{ tasks} \times 2 \text{ items} \times 3 \text{ answers} \times 24 \text{ individuals} = 1152
\]
per group, and thus
\[
1152 \times 2 = 2304
\]
total answers. The study also reports **3904 total words**, with **2010** from LLMs and **1894** from humans. Before scoring, the authors removed extraneous punctuation, quotation marks, numbers, and em-dash lines.

## 4. Comparative results obtained with OCSAI

The main empirical finding reported with OCSAI is that **LLMs outperformed humans overall**, and did so on most tasks. The overall mean total score is **0.52** for LLMs and **0.48** for humans. At the criterion level, the LLM group outperformed humans in **Originality**, **Flexibility**, and **Total score**, while the difference in **Elaboration** was smaller and **not statistically significant** [2507.12039].

The reported inferential statistics are as follows. For **Total score**, the paper gives \(t=-3.03\) and \(p=0.0044\). For **Originality**, it gives \(t=-4.21\) and \(p=0.00014\). For **Elaboration**, it gives \(t=-1.15\) and \(p=0.258\). For **Flexibility**, it gives \(t=-3.06\) and \(p=0.0040\). The study also reports an **OLS regression** over all **1152 scores**, with a significant group effect at **\(p=0.016\)**, and states that LLMs scored on average **0.037 points higher** than humans.

At the task level, LLMs performed better in **six of the eight tasks**. The paper identifies two exceptions. In the **Semantic Field** task, humans and LLMs were essentially tied. In the **Euphemism/Dysphemism** task, humans slightly outperformed LLMs, with **0.33** versus **0.29**.

The results therefore show that OCSAI, in the study’s configuration, favors the LLM group on measures of semantically structured novelty and response diversity. The non-significant elaboration difference is consistent with the experimental constraint that each answer was limited to five words and that elaboration was measured through a word-count-based mechanism.

## 5. Uniqueness, interpretation, and creativity types

The paper supplements OCSAI scores with a separate **uniqueness** analysis, explicitly distinguished from OCSAI’s own scoring. For each of the 48 individuals, all answers were grouped into a separate file; **NLTK** was used for stop-word removal; **spaCy** embeddings were used to compute pairwise semantic similarity; and uniqueness was defined as the inverse of average semantic similarity with all other individuals. The study reports that there is **no strong global separation** between humans and LLMs on this measure [2507.12039].

The ranking distribution is described as slightly skewed: **15 humans and 9 LLMs** appear in the top half, while **15 LLMs and 9 humans** appear in the lower half. The most unique and least unique individuals were **both humans**. The paper highlights an especially important case: **Human 18** was the most unique by this metric, yet was **not among the highest OCSAI scorers**. This distinction is central to the study’s interpretation. OCSAI originality measures creativity under the tool’s semantic scoring framework, whereas uniqueness measures divergence from the rest of the dataset.

The manual qualitative analysis in the paper introduces Sampson’s distinction between **F-creativity** and **E-creativity**. The study suggests that LLMs tend toward **F-creativity**, characterized as fixed, rule-based creativity within the language system, whereas humans tend toward **E-creativity**, characterized as extending creativity that bends or stretches the language system. The authors describe LLM outputs as more likely to follow morphological rules closely, produce systematic compounds and derivations, and reuse common templates, while human outputs more often combine semantically distant items, violate or stretch morphological expectations, and display playful, pragmatic, or humorous constructions.

This yields a nuanced interpretation of OCSAI’s outcome. The automatic scores favor LLMs in originality and flexibility, but the manual analysis suggests that humans may still display more boundary-pushing or pragmatically loaded forms of creativity. The paper therefore does not treat OCSAI as exhausting the concept of creativity; rather, it treats OCSAI as a semantically driven instrument that captures one structured aspect of creative performance.

## 6. Methodological constraints and reuse considerations

Several constraints in the study are important for interpreting OCSAI Tool as an evaluation instrument. First, **Fluency was intentionally excluded**, because every participant had to provide exactly **three answers** per item. Second, **Flexibility is not directly provided by OCSAI** and had to be constructed manually from pairwise answer comparisons. Third, originality scoring depends on the specific OCSAI configuration: the evaluators used the **full prompt option** and the **metaphor task type** [2507.12039].

Fourth, the reported originality values were **normalized from 1–5 to 0–1**, so direct comparison with other uses of OCSAI requires matching that normalization. Fifth, elaboration is described as essentially **length-based**, which means it should not be conflated with a broader theory of semantic richness. Sixth, the human participants were **non-native English speakers**, a fact that limits the generality of human–LLM comparison. Seventh, the human condition involved **controlled classroom administration without digital aid**, which strengthens experimental control but also distinguishes human production conditions from unconstrained model inference.

The paper also makes clear that OCSAI-based scoring is **semantically driven, not purely form-driven**, because the tool is described as relying on **semantic distances** and **GPT-4 integration**. This suggests that OCSAI is sensitive to semantic relations among answers, but also that its judgments depend on the properties of the underlying semantic model. A plausible implication is that OCSAI is better understood as an automatic evaluator of semantically structured creativity than as a complete theory of linguistic creativity.

In this sense, OCSAI Tool occupies a specific methodological position. It provides a scalable and automatic mechanism for scoring **Originality**, **Elaboration**, and indirectly **Flexibility** in constrained creativity tasks. At the same time, the reported study treats its outputs as analytically informative but not exhaustive, supplementing them with uniqueness analysis and manual interpretation to distinguish systematic creativity from more idiosyncratic, rule-bending forms of invention.

Source: https://www.emergentmind.com/topics/ocsai-tool