Normalized Collaboration Index (NCI)

• Normalized Collaboration Index (NCI) is a quantitative metric measuring academic-industry collaboration in AI preprints using a random-mixing baseline.
• It corrects for team size differences and author pool imbalances to isolate genuine structural collaboration or avoidance patterns.
• Empirical results reveal persistent under-collaboration, with NCI values consistently below 1 across key AI subfields.

The Normalized Collaboration Index (NCI) is a quantitative metric introduced to assess the degree of cross-sector collaboration—in particular, the joint authorship of academic and industrial researchers—within the AI arXiv preprint ecosystem. By measuring the observed prevalence of mixed academic–industry co-authorship relative to a principled random-mixing baseline, the NCI isolates genuine structural collaboration (or avoidance) patterns from mechanical effects induced by varying team sizes and sectoral workforce imbalances. Persistent suppression of the NCI below the random-mixing expectation signals a continuing institutional divide, even in the face of a massive expansion in preprint output and research team sizes following the emergence of LLMs and generative AI systems (Magnur et al., 3 Feb 2026).

1. Formal Definition and Mathematical Formulation

Let each paper $i$ published in calendar month $t$ have $k_i$ authors. Global proportions for author sectors are defined as

$$p_A = \frac{A_{\text{auth}}}{A_{\text{auth}}+I_{\text{auth}}}, \qquad p_I = \frac{I_{\text{auth}}}{A_{\text{auth}} + I_{\text{auth}}}$$

where $A_{\text{auth}}$ and $I_{\text{auth}}$ are the total numbers of academic and industry author “slots,” respectively. Mixed affiliation papers contribute half to each sector. Authors with unknown affiliations are excluded from the denominator.

Under the random-mixing model, the probability that a team of size $k$ contains at least one academic and at least one industrial author is

$$P_{\mathrm{mixed}}(k) = 1 - \left[(1-p_A)^{k} + (1-p_I)^{k} - (1-p_A-p_I)^{k}\right]$$

For each month $t$, the observed fraction of mixed affiliation papers is

$$R^{\mathrm{obs}}_t = \frac{n_{\mathrm{mixed},t}}{n_t}$$

where $t$0 is the total number of papers and $t$1 the count of papers with both academic and industry authors. The expected random-mixing baseline is

$t$2

The Normalized Collaboration Index for month $t$3 is given by

$t$4

An $t$5 indicates parity with the random-mixing expectation. Values below 1 indicate suppressed cross-sector collaboration; values above 1 would correspond to greater-than-expected teaming.

2. Rationale for the Random-Mixing Baseline

The random-mixing baseline serves as a neutral reference, correcting for the structural effects of team size heterogeneity and global author pool composition. As the mean team size increases or the proportions of academic versus industry authors drift, the chance of forming mixed-sector teams by random assembly from the global pool changes mechanically. The baseline isolates true institutional preferences—or aversions—by holding these mechanical factors fixed.

For instance, if industry authors are rare relative to academics, larger teams have increased combinatorial likelihood of being mixed, independent of collaborative intent. Dividing the observed mixed-paper fraction by the expected fraction under random-mixing distinguishes genuine institutional divides from artifacts of sampling and scale (Magnur et al., 3 Feb 2026).

3. Data Collection, Affiliation Inference, and Metric Computation

The NCI calculation relies on comprehensive and accurate author affiliation labeling for arXiv cs.AI preprints from January 2021 through December 2025. The methodological pipeline included:

• Corpus assembly: All cs.AI arXiv papers (∼12.5K in 2021 to ~44.8K in 2025).
• Metadata retrieval: arXiv API harvested title, authors, abstracts, and primary category, in five-day increments.
• Structured institution mapping: OpenAlex queried for author-institution links; missing affiliations filled via scraping of ar5iv mirror HTML for institutional names and email addresses.
• LLM-based affiliation classification: Aggregate “author + affiliation” blocks submitted to GPT-4o-mini using a tuned JSON-schema prompt, outputting institution lists, sectoral types, and cross-sector indicators with ∼87%–91% first-pass accuracy.
• Email domain inference: Secondary pass for LLM-missed papers, using domain heuristics (e.g., “.edu” ⟶ academic) and LLM mini-prompts, raising total labeling reliability to 91%–94%.
• Unknown affiliation imputation: Annual manual audit (50 papers per year) to empirically determine academic/industry/mixed proportions in the “unknown” set, followed by probabilistic reassignment.
• Metric stratification: Computation of NCI at monthly granularity, with subfield stratification (cs.LG, cs.CL, cs.HC) based on arXiv category tags (Magnur et al., 3 Feb 2026).

4. Empirical Results and Subfield Stratification

Between January 2021 and late 2025, every monthly $t$6 value remained well below unity. The main findings are summarized as follows:

Subfield	Median NCI	Mean NCI	Notable Trend
cs.CL (Comp Ling)	0.285	0.284	No significant trend
cs.HC (HCI)	0.209	0.225	Modest upward trend
cs.LG (ML)	0.266	0.270	No significant trend

Typical pre-imputation values lay in the $t$7–$t$8 range. After unknown-affiliation imputation, the NCI increased uniformly by about $t$9–$k_i$0 (i.e., to $k_i$1–$k_i$2), but still well below one. The sole subfield displaying a significant positive drift was cs.HC, though it nonetheless remained far from unity.

A December 2025 crash in NCI to 0.006 was attributed to right-censoring and excluded from interpretation.

5. Interpretation: Institutional Divide and Structural Suppression

The persistent suppression ($k_i$3) indicates that academic and industrial researchers in the AI arXiv preprint landscape co-author papers at rates substantially lower than would be expected by random assembly, even as both mean and variance in team size increase. The “ChatGPT effect” and the rise of generative AI have led to explosive growth in publication volume and larger research teams (academic mean team size from 4.4 to 5.5 authors, industry teams growing faster), but this volume expansion has not translated into a closing of institutional divides.

This pattern underlines the ongoing “compute divide” in generative AI research. The capital- and resource-intensive structure of contemporary AI work appears to reinforce, not erode, the boundary between academic and industrial research output. Subfield exceptions (e.g., a modest upward trend in cs.HC) are quantitatively minor (Magnur et al., 3 Feb 2026).

6. Methodological Limitations and Interpretive Caveats

Several caveats pertain to the computation and interpretation of the NCI:

• Affiliation labeling accuracy: Despite dual-pass (LLM and email domain) classification yielding over 90% accuracy, multi-affiliated and multi-national authors introduce residual misclassification error.
• Author counting: Free-text parsing to estimate team size does not deduplicate recurring individual authors; precision is sufficient only for aggregate monthly statistics.
• Random-mixing assumptions: The benchmark posits independent sampling of authors from a fixed global pool, ignoring network structure, elite lab concentration, or reputation-driven collaboration patterns.
• Temporal right-censoring: The final month in the dataset is incomplete, producing artifactual dips; only fully observed months included in trend analysis.
• Subfield assignment ambiguity: Reliance on arXiv category tags for stratification may not fully capture interdisciplinary or nuanced thematic alignments.
• Probabilistic imputation impact: Reassignment of unknown-labeled papers smooths over individual heterogeneity, possibly underestimating context-dependent collaboration patterns.

7. Significance and Future Directions

The NCI framework, grounded in size-corrected random-mixing expectations and robust affiliation inference, reveals a stable and pronounced under-collaboration between academia and industry in the generative AI preprint ecosystem. Despite the unprecedented acceleration of research activity post-ChatGPT, structural and resource barriers appear to maintain sectoral separation. A plausible implication is that, in the absence of targeted policy action or stronger institutional frameworks for cross-sector integration, collaborative shortfalls are likely to persist and possibly intensify as computational requirements scale (Magnur et al., 3 Feb 2026).