Multidomain Role Identification Accuracy

• Role identification accuracy is the measure of how effectively computational systems assign or infer roles from various evidence sources across domains such as social deduction games, air traffic, and social media.
• It encompasses diverse methodologies including rule-based, supervised neural, and unsupervised clustering methods, using metrics like accuracy, F1 scores, and internal cluster-validity indices.
• This concept is crucial for applications ranging from air traffic communication to social media analysis, where different evidence types and role granularities significantly affect performance and interpretability.

Role identification accuracy denotes the quality with which a computational system assigns or infers roles from observable evidence. In the recent literature, the target roles range from concealed identities in Social Deduction Games, to controller–pilot speaker roles in air traffic communication, to male-related, female-related, and brand-related Twitter accounts, to structural roles of nodes in graphs, to semantic roles in predicate–argument structure, and to fine-grained roles in cyberbullying scenarios (Xu et al., 9 Nov 2025, Guo et al., 2021, Li et al., 2018, Ma et al., 2019, Nakamura et al., 2021, Wang et al., 2024). The notion of “accuracy” is correspondingly heterogeneous: in some settings it is ordinary supervised classification accuracy, in others it is Micro-F$_1$ or Macro-F$_1$, and in unsupervised settings it is operationalized through internal cluster-validity indices rather than labeled correctness (Prasad et al., 2021, Wang et al., 2024).

1. Problem formulations and role spaces

Role identification problems differ first in whether the roles are hidden, explicit, or induced. In Social Deduction Games such as Avalon, Mafia, and Werewolf, players conceal their identities and deliberately mislead others, so hidden-role inference is a central task. CSP4SDG formulates this setting as a probabilistic, constraint-satisfaction problem in which game events and dialogue are mapped to four linguistically-agnostic constraint classes—evidence, phenomena, assertions, and hypotheses—and the output is a posterior over roles that is fully interpretable and updates in real time (Xu et al., 9 Nov 2025).

In air traffic communication, the role space is binary: controller versus pilot. One line of work treats speaker role identification as a knowledge-based text classification problem over ASR transcripts generated from noisy VHF recordings; another formulates it as a binary classification problem over text, speech, or both, with $y=1$ meaning “controller” and $y=0$ meaning “pilot” (Prasad et al., 2021, Guo et al., 2021). In both cases, role identification is operational rather than latent: the target is to assign the utterance to one of two speaker roles.

In social media and linguistic analysis, the role inventory can be multiclass or label-sequence based. TWiRole classifies Twitter users as male-related, female-related, or brand-related, explicitly extending prior work that did not consider brand-related users (Li et al., 2018). Japanese semantic role labeling treats role identification at the argument level: a shared BERT encoder, dependency-parsing branch, and SRL branch ultimately predict BIO-tagged semantic-role labels such as B-Agent, I-Agent, and O (Nakamura et al., 2021).

In graph mining and cyberbullying analysis, the role taxonomy is partly emergent from structure. RiWalk studies structural roles of nodes and evaluates them by downstream role classification, including across-network transfer (Ma et al., 2019). The cyberbullying study is fully unsupervised: after clustering into $K=9$ by the SSE elbow, clusters are manually assigned one of nine role names, including Zealous Perpetrator, Calm Observer Analyst, and Sympathetic Bystander (Wang et al., 2024).

Domain	Roles	Evaluation framing
Social Deduction Games	concealed player identities	posterior inference
Air traffic communication	controller, pilot	binary classification
Twitter user classification	male-related, female-related, brand-related	multiclass classification
Japanese SRL	BIO-tagged semantic-role labels	argument identification and classification
Structural graphs	node roles	within- and across-network classification
Cyberbullying analysis	nine clustered roles	clustering validity

2. What counts as “accuracy”

The literature does not use a single universal definition of role identification accuracy. In standard supervised settings, accuracy is given explicitly. The grammar-based air-traffic system defines $$\mathrm{Accuracy} = N_{\mathrm{correct}}/N_{\mathrm{total}}$$ and also reports true-positive rate and true-negative rate (Prasad et al., 2021). The multimodal ATC study reports $\mathrm{ACC} = (\mathrm{TP}+\mathrm{TN})/(P+N)$ together with precision, recall, F$_1$, and AUC (Guo et al., 2021). TWiRole defines overall accuracy as the sum of classwise true positives divided by the total number of users, alongside per-role recall, precision, and F$_1$ (Li et al., 2018).

In role classification on graphs, accuracy is partially replaced by F-score conventions tailored to the task. RiWalk evaluates within-network classification by Micro-F$_1$, which is stated to be identical numerically to accuracy when classes are disjoint, and evaluates across-network transfer by Macro-F$_1$0 (Ma et al., 2019). This distinction matters because transfer tasks emphasize balanced performance across classes rather than aggregate correctness alone.

In semantic role labeling, the metrics separate two stages of the task. Nakamura et al. report Argument-ID F$_1$1 and Argument-Classification Accuracy, showing that gains in role identification need not coincide with gains in role classification (Nakamura et al., 2021). This makes “role identification accuracy” more specific than an undifferentiated end-to-end score.

The cyberbullying study uses the term in a different sense altogether. Rather than supervised accuracy, precision, recall, or F$_1$2, it evaluates DEK by internal clustering validity indices: SSE for choosing $_1$3, Davies–Bouldin Index, Silhouette Coefficient, and Dunn Index (Wang et al., 2024). In that setting, tighter and better-separated clusters are treated as evidence of better role identification even though no ground-truth role labels are used during clustering.

3. Evidence sources and representational choices

Performance in role identification is strongly tied to what evidence is made available to the model. CSP4SDG is explicitly multimodal at the symbolic level: it combines game events and dialogue, imposes hard constraints to prune impossible role assignments, and uses weighted soft constraints to score the remainder. Its information-gain weighting links each hypothesis to its expected value under entropy reduction, and its scoring rule is designed so that truthful assertions converge to classical hard logic with minimum error (Xu et al., 9 Nov 2025).

The two ATC studies illustrate a contrast between shallow domain knowledge and learned multimodal representations. The grammar-based system accepts text as input, either manually verified annotations or automatically generated transcripts, and exploits callsign position, 31 ATCO trigger words, 20 pilot trigger words, and an optional Naive-Bayes bag-of-words score with equal priors (Prasad et al., 2021). The deep-learning study instead uses variable-length token sequences and acoustic inputs. Text models use a vocabulary of 1,284 tokens and 512-dimensional embeddings; speech models use 80-bin log-Mel spectrograms or normalized raw waveform; MMSRINet fuses a BiLSTM textual encoder and a CRNN acoustic encoder through modal attention and self-attention pooling (Guo et al., 2021).

TWiRole combines three orthogonal sources of evidence: profile text and metadata, tweet content, and profile imagery. Its basic features include name scores, description-derived scores, a follower/friend statistic $_1$4, image brightness in HSV space, and tweet-based scores; its advanced features are $_1$5-top-word statistics; and its visual component is a fine-tuned ResNet-18 whose softmax output provides per-class probabilities (Li et al., 2018).

In graph and clustering settings, the evidence is structural or mixed-attribute rather than linguistic. RiWalk-SP relabels nodes using discounted degree and shortest-path information, while RiWalk-WL uses rooted Weisfeiler–Lehman-style count vectors over distances from an anchor node (Ma et al., 2019). DEK is designed for mixed continuous and categorical features and uses Gower distance together with a centroid-separation term in its modified distance function; in the Weibo scenarios the feature space contains 17 continuous and 4 categorical dimensions (Wang et al., 2024). In Japanese SRL, the evidence includes shared BERT representations, a predicate indicator, a Bi-LSTM sequence encoder, and multitask supervision from dependency parsing (Nakamura et al., 2021).

4. Methodological families

Three broad methodological families recur in the literature. The first is rule-based or knowledge-based role identification. The air-traffic grammar system is exemplary: it uses a two-stage rule hierarchy in which the callsign-position rule has absolute priority, followed by a trigger-word rule, with probabilistic fallback only when the rules are ambiguous (Prasad et al., 2021). Its design assumption is that ICAO phraseology carries role-discriminative regularities that can be captured without a rich grammar formalism.

The second family is supervised neural or hybrid classification. MMSRINet belongs to this category, as do the text-only and speech-only ATC baselines built from BiLSTM, TextCNN, Transformer, CRNN, X-vector, and SincNet backbones (Guo et al., 2021). TWiRole is hybrid rather than purely neural: BF, AF, and CNN probability vectors are concatenated and sent to a final multi-classifier, with Random Forest yielding the strongest reported overall accuracy (Li et al., 2018). The Japanese SRL model is also multitask and hierarchical: a shared BERT encoder feeds a shallow dependency-parsing branch and a deeper SRL branch, and the results suggest that multitasking with dependency parsing is mainly effective for argument identification (Nakamura et al., 2021).

The third family replaces direct classification with structured inference or unsupervised induction. CSP4SDG is a probabilistic constraint-satisfaction framework in which hard constraints enforce consistency and soft constraints modulate plausibility (Xu et al., 9 Nov 2025). RiWalk decouples structural embedding into role identification and network embedding, using relabeling to construct rooted kernels before random-walk-based embedding (Ma et al., 2019). DEK embeds K-means within a Differential Evolution search over centroid matrices to reduce local-minimum sensitivity in mixed spaces (Wang et al., 2024).

A plausible implication is that role identification accuracy depends not only on model capacity but also on whether the method’s inductive bias matches the evidence structure. Phraseology-driven rules are effective when discourse conventions are rigid; multimodal neural fusion is effective when acoustic and textual cues are complementary; and constraint or clustering methods are appropriate when interpretability or lack of supervision is primary.

5. Empirical performance across domains

Reported performance varies sharply by domain, supervision regime, and evaluation protocol. In Social Deduction Games, CSP4SDG is reported to outperform LLM-based baselines in every inference scenario and to boost LLMs when supplied as an auxiliary “reasoning tool,” though the available abstract does not provide numerical tables (Xu et al., 9 Nov 2025).

In air traffic communication, the gap between lightweight grammar and multimodal deep learning is large under their respective experimental setups. The grammar-based system provides an average speaker-role identification accuracy of about 83%, with NATS at 85%, ISAVIA at 82.5%, and LiveATC at 73% (Prasad et al., 2021). By contrast, the multimodal MMSRINet achieves 98.56% accuracy on the test set and 98.08% on the unseen test-s set, with F$_1$6 scores of 98.87% and 98.39%, respectively (Guo et al., 2021). Within the unimodal baselines, Pretrained-T reports 97.13% and 97.46% accuracy on test and test-s, while Pretrained-S reports 98.13% and 97.56% (Guo et al., 2021).

On Twitter, TWiRole reaches an overall multiclass accuracy of 0.899 on the balanced Kaggle subset. Its per-class F$_1$7 values are 0.903 for male-related, 0.908 for female-related, and 0.885 for brand-related users, indicating comparatively balanced performance. Feature ablation shows that removing CNN features reduces accuracy to 0.837, the largest reported drop, while removing name features reduces it to 0.870 (Li et al., 2018).

In Japanese semantic role labeling, the multitask DP+SRL model improves Argument-ID F$_1$8 from 72.87% to 74.46%, a statistically significant gain of +1.59 with $_1$9, while Argument-Classification Accuracy changes from 80.35% to 80.74% with $y=1$0. In the span-given setting, accuracy rises from 75.19% to 77.48% (Nakamura et al., 2021). These results isolate identification gains from classification gains.

RiWalk reports within-network Micro-F$y=1$1 and across-network Macro-F$y=1$2. At 80% training, RiWalk-WL attains 66.17% on USA, while RiWalk-SP attains 61.44% on Film and 46.05% on Actor; in across-network transfer, RiWalk-SP and RiWalk-WL reach values such as 81.98% on USA$y=1$3Europe, 80.07% on Europe$y=1$4USA, 80.90% on Actor$y=1$5USA, and 73.97% on USA$y=1$6Actor (Ma et al., 2019).

In cyberbullying role induction, DEK records the strongest overall clustering-validity profile. Across 15 experiments, it achieves 11 wins in DBI, 15 wins in SC, and 12 wins in DVI, whereas the next best method, hierarchical clustering, achieves 4, 0, and 3 wins respectively (Wang et al., 2024).

Setting	Primary metric	Reported result
ATC grammar-based SRI	Accuracy	about 83% average
ATC MMSRINet	Accuracy	98.56% test, 98.08% test-s
TWiRole	Accuracy	0.899
Japanese DP+SRL	Argument-ID F$y=1$7	74.46%
RiWalk transfer	Macro-F$y=1$8	up to 81.98% / 80.90%
DEK cyberbullying	DBI / SC / DVI wins	11 / 15 / 12

6. Interpretation, limitations, and recurrent issues

A persistent source of confusion is that “role identification accuracy” is not a single measurement construct. In ATC and Twitter studies it is literal supervised correctness; in RiWalk it is tied to downstream classification F-scores; in cyberbullying it is a property of cluster compactness and separation rather than labeled prediction (Guo et al., 2021, Li et al., 2018, Ma et al., 2019, Wang et al., 2024). Comparisons across papers are therefore meaningful only after aligning the evaluation regime.

Another recurrent issue is domain robustness. The grammar-based ATC system is designed for VHF audio with SNR ratios below 15 dB and is described as robust under noise because it operates at the text level, yet its performance drops on the LiveATC test relative to NATS and ISAVIA (Prasad et al., 2021). The deep ATC study identifies different failure modes: text-only models are affected by transcripts that deviate from ICAO grammar or contain OOV tokens, while speech-only models are affected by domain mismatch, noise, channel distortions, and class imbalance (Guo et al., 2021). Pretraining improves robustness on unseen data in that setting (Guo et al., 2021).

Role granularity is also consequential. The cyberbullying paper explicitly argues that focusing only on victims, perpetrators, and bystanders is insufficient and instead analyzes nine roles (Wang et al., 2024). TWiRole likewise expands beyond male/female binary gender classification to include brand-related accounts (Li et al., 2018). This suggests that reported accuracy can rise or fall with the granularity of the ontology, and that changes in role inventory alter the task rather than merely its difficulty.

Interpretability is treated differently across methods. CSP4SDG makes it explicit: the posterior over roles is fully interpretable, real-time, and grounded in hard and soft constraints (Xu et al., 9 Nov 2025). By contrast, multimodal deep systems such as MMSRINet emphasize predictive performance and robustness (Guo et al., 2021). In Japanese SRL, the main finding is more specific: multitasking with dependency parsing is mainly effective for argument identification, not for argument classification (Nakamura et al., 2021). Taken together, these results indicate that role identification accuracy is best understood as a family of domain-dependent evaluation outcomes shaped by role ontology, evidence type, inductive bias, and metric choice.