- The paper introduces a hybrid system that integrates a symbolic parser and an LLM to achieve both fine- and coarse-grained word sense disambiguation.
- It employs a multi-step pipeline where a CNLU system generates semantic candidates that are verbalized and refined by an LLM based on contextual cues.
- Empirical results on a COPA dataset show significant improvements over BERT baselines, demonstrating the method's robustness and data efficiency.
Integrating Symbolic Natural Language Understanding and LLMs for Word Sense Disambiguation
Introduction
This paper addresses the persistent problem of fine-grained word sense disambiguation (WSD) in knowledge-rich NLU systems, specifically focusing on integrating symbolic semantic parsing with neural LLMs for accurate semantic interpretation. Existing systems are constrained by either limited domain adaptability, dependence on annotated corpora, or inadequate capacity for subtle distinctions critical for advanced reasoning tasks. The presented approach leverages a structured, rule-based parser (CNLU) to generate ontologically grounded semantic candidates, coupled with LLM-based contextual selection, circumventing the need for hand-annotated training data and enabling robust fine- and coarse-grained semantic disambiguation.
Hybrid Methodology
The approach employs a multi-step pipeline, initiating with the CNLU symbolic NLU system. For every ambiguous lexical item, CNLU generates multiple detailed candidate interpretations, each mapped onto NextKB (an OpenCyc superset ontology). These logical forms are verbalized—using a template-based system—into discriminative natural language fragments interpretable by an LLM (Phi4). The disambiguation process consists of presenting the ambiguous context, tokenizing each interpretation as a numbered option, and querying the LLM, which selects the optimal candidate based on sentential context. Selected meanings are then reverse-mapped into the symbolic system, maintaining consistent semantic propagation via a truth maintenance mechanism.
This setup capitalizes on the structured breadth of symbolic representations (critical for downstream reasoning and inference) while exploiting the contextual generalization strengths of LLMs to adjudicate among subtle candidate meanings. The technique is agnostic to annotated data post-ontology/lexicon construction and does not involve supervised learning from task-specific examples.
Empirical Evaluation
Evaluation is performed on a curated subset of the COPA dataset, targeting sentences containing an average of 2.3 semantic ambiguities each, annotated by an expert. Performance is compared against two BERT baselines: a fine-tuned BERT frame classifier (coarse-grained) and a pipeline that makes random fine-grained choices within predicted frames (BERT Random). Accuracy is measured at both frame (coarse-grained) and predicate (fine-grained) levels:
| Method |
Coarse-Grained Accuracy |
Fine-Grained Accuracy |
| BERT |
69.3% |
20.2% |
| CNLU+Phi4 |
84.2% |
82.5% |
The results demonstrate robust performance by CNLU+Phi4, achieving similar accuracy at both semantic granularities, in stark contrast to BERT, which suffers catastrophic drop-off at the predicate level. Nearly all frames correctly identified by Phi4 yield correct fine-grained selection, illustrating reliability within the frame-constrained candidate set. This supports the core claim: the hybrid system is effective for precise semantic modeling required by sophisticated AI systems, while obviating supervised disambiguation datasets.
Analysis of Error Patterns
Error analysis indicates that the main failure modes involve frame boundary confusion or predicate selection errors traceable to insufficiently context-sensitive verbalizations and boundary cases where LLM world knowledge is misapplied (such as domain mismatches for physical entities). However, within correctly identified frames, fine-grained predicate errors are comparatively rare. The analysis points to the critical role of improved verbalization schemas and suggests the need for robust validation layers to mitigate physically implausible assignments.
Implications and Theoretical Impact
The implications of this work are twofold. Practically, the method enables immediate applicability to domains with large, fine-grained symbolic ontologies without requiring costly annotation investment. This has direct value for knowledge-based agents, scientific information extraction, and any task requiring explicit, semantically precise representations for inference. Theoretically, the work delineates an effective architecture for modular NLU systems: using LLMs as context-sensitive oracles while retaining full symbolic interpretability and transparency. The separation of candidate generation (symbolic) from selection (neural) is preserved, facilitating systematic error analysis and controlled integration.
Furthermore, the hybrid pipeline is broadly transferable—any symbolic NLU that can verbalize logical forms can harness this architecture, as is possible with systems like Cyc. It also signals a route for broader symbolic-neural integration in NLU, with parallel extensions possible for co-reference, event coreference, or structural parsing tasks.
Limitations
The system’s reliance on the quality of linguistic verbalizations is a non-trivial limitation, particularly for domains with ambiguous or under-specified ontology-to-language mappings. The present evaluation uses single-annotator gold standards and a relatively small dataset; results would benefit from larger-scale, multi-annotator validation. Another constraint is system dependence on the engineering capabilities of the underlying symbolic parser and template system, though these elements are expected to scale to similarly structured knowledge bases. Robustness to world knowledge and selectional preferences remains an open challenge, suggesting future work on hybrid symbolic-neural validation and back-off strategies.
Relation to Prior Art
This method contrasts with end-to-end LLM or neural-only approaches by yielding explicit symbolic outputs, suitable for subsequent inference, and with classical symbolic-ontology WSD by eschewing the need for hand-crafted rules for context resolution. It extends the tradition of modular cognitive architectures by operationalizing neural modules as APIs for semantic choice within transparent symbol-manipulation pipelines.
Conclusion
The paper presents a robust, data-efficient hybrid system for word sense disambiguation, combining symbolic NLU with LLM-based selection to circumvent annotated-data bottlenecks while preserving fine-grained semantic coverage and transparency. Empirical results establish strong performance at multiple levels of semantic granularity, with the approach readily extensible to analogous NLU problems. Future directions lie in scaling to larger corpora, integrating more nuanced context modeling, and broadening the class of symbolic ambiguity resolution problems handled by analogous architectures.