ASC Analyzer: L2 Proficiency Toolkit

• ASC Analyzer is an open-source Python toolkit for systematic identification and quantification of argument structure constructions in English texts.
• It employs a RoBERTa-based neural tagger and computes 50 indices (diversity, proportion, frequency, and strength of association) using corpus-driven metrics.
• Designed for L2 proficiency research, it provides actionable insights for curriculum development and language assessment through objective empirical metrics.

Argument Structure Construction (ASC) Analyzer is an open-source Python toolkit designed for the systematic identification and quantification of argument structure constructions (ASCs) within English texts. Developed to fill the gap in scalable computational analysis of constructional usage in second language (L2) writing, the ASC Analyzer employs neural tagging, corpus-driven reference statistics, and specialized indices to measure diversity, frequency, proportion, and association strength of ASC usage. It provides robust empirical metrics intended for L2 proficiency research, curriculum development, and language assessment.

1. Core Functionality and Measurement Indices

The ASC Analyzer automatically tags occurrences of ASCs within provided English texts using a RoBERTa-based ASC tagger. The toolkit computes 50 distinct indices, which fall into four principal categories:

• Diversity: Quantifies the variety of ASC usage via the moving-average type-token ratio (MATTR) on a token sequence $X$ of length $N$:

$$\text{MATTR}_w(X) = \frac{1}{N - w + 1} \sum_{i=1}^{N-w+1} \left( \frac{|\text{types}(X[i : i+w-1])|}{w} \right)$$

Variants include calculations on ASC tokens, ASC-verb lemma pairs, and exclusion of the verb "be".

• Proportion: Calculates the relative frequency of each targeted ASC type $c$ as

$\text{Prop}_{c}(X) = \frac{f_c}{N_\text{ASC}}$

where $f_c$ is the count for type $c$ and $N_\text{ASC}$ is the total ASC tokens. Nine ASC types are included (e.g., ATTR, DITRAN).

• Frequency: Indexes the frequency of each ASC token $t_i$ or ASC-verb lemma pair external to the text, using corpus-derived statistics:

$$\text{Freq} = \sum_{i=1}^{M} l(t_i), \quad l(t_i) = \ln(f_\text{ref}(t_i))$$

where $f_\text{ref}(t_i)$ is the count from a reference corpus.

• Strength of Association (SOA): Uses contingency tables to measure the association strength of ASC–verb lemma pairs via mutual information (MI) and t-score:

$E(c, v) = \frac{(a + b)(a + c)}{N}$

$$\text{MI}(c, v) = \log_2 \left( \frac{a}{E(c, v)} \right) \quad \text{and} \quad T(c, v) = \frac{a - E(c, v)}{\sqrt{a}}$$

Multiple association probability metrics (APLemma, AP Structure) are also computed for finer discriminability.

2. Technical Architecture and Tagging Pipeline

The ASC Analyzer is implemented in Python. Its pipeline consists of:

• Neural Tagging: Uses a RoBERTa-based tagger pretrained then fine-tuned on a combined L1+L2 gold-standard ASC treebank (cf. Sung and Kyle, 2024b). The tagger achieves F1 scores ranging from 0.915 to 0.928 in both writing and speaking domains.
• NLP Integration: Employs spaCy and spaCy-transformers for preprocessing, tokenization, and model inference. Installation and deployment is facilitated through pip and spaCy model downloads (e.g., “en_core_web_trf”).
• Corpus Reference Integration: Incorporates type and frequency data from the EnCOW web corpus or SUBTLEX-US frequency lists for external normalization and association metrics.
• Index Calculation: Applies sliding window algorithms for MATTR, direct counting for proportions, and mathematical aggregation for frequency and association metrics as specified above.

This integration enables both efficient tagging and comprehensive, corpus-informed quantification.

3. Empirical Applications in L2 Proficiency Research

The primary use case documented for ASC Analyzer is the empirical investigation of L2 writing proficiency, with concrete applications to the ELLIPSE corpus (6,482 ESL essays):

• Diversity and Complexity Profiling: Indices such as ascMATTR provide measures of constructional diversity, which correlate positively (r = +0.26) with proficiency.
• Prototypicality and Association Strength: Frequency-based indices (e.g., ascAvFreq, r = –0.22) capture overreliance on frequent, entrenched form-verb pairings, often characteristic of lower proficiency.
• Granular ASC Type Analysis: The analyzer distinguishes usage patterns for individual construction types (e.g., passive, ditransitive, copula), facilitating targeted curriculum design and proficiency diagnostics.
• Normative Comparison: By leveraging native speaker norms from reference corpora, deviations in ASC usage (e.g., atypical verb-association pairings) can be operationalized as indicators of development.

A plausible implication is that ASC-based indices yield richer explanatory power for L2 development than conventional syntactic metrics.

4. Analytical Methods and Statistical Correlates

Demonstrated methodologies include both:

• Bivariate Analysis: Pearson correlations between individual indices and L2 writing proficiency, confirming predictive relationships highlighted above.
• Multivariate Regression and Model Selection: An AIC-based procedure selected a subset of 12 ASC predictors that jointly explained approximately 14.3% of variance in proficiency scores (adjusted $R^2 \approx 0.143$). Comparison with competing models (syntactic complexity, lexicogrammatical features) showed ASC-based indices as complementary and sometimes superior in explanatory power.

The operationalized indices thus serve not only in descriptive analysis, but also in statistical modeling and proficiency prediction.

5. Limitations and Considerations

Documented limitations include:

• Training Corpus Imbalances: Underrepresentation of some ASC types (notably intransitive resultatives) in the tagger's training data can affect the reliability and completeness of the indices.
• Reference Corpus Coverage: Association and frequency measures rely on limited reference corpora, which may not capture the full diversity of registers and genres encountered in target analyses.
• Interpretive Depth: The tool, as presented, focuses on metric calculation, not detailed linguistic interpretation; further research is needed for pedagogically grounded, qualitative insights.

It is suggested that future enhancements could include expansion of training datasets and reference corpora, fine-tuning to less frequent constructions, and broad integration with complementary assessment frameworks.

6. Open-Source Accessibility and Practical Value

The ASC Analyzer is publicly available as a Python package and is designed for transparent integration into NLP pipelines. The tool supports both command-line invocation and programmatic use, requiring minimal configuration for researchers familiar with Python-based text analytics. Its output consists of detailed, scalar indices compatible with both exploratory and modeling workflows in corpus linguistics, language assessment, and educational research.

The comprehensive design and empirical evidence from large-scale L2 datasets position the ASC Analyzer as a significant resource for the quantitative paper of argument structure construction usage, with direct implications for proficiency assessment, feedback generation, and linguistic research.