Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

Abstract: Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting physical quantities, numbers, units, and symbolic expressions into lexically arbitrary subwords. We present TOTEN, a knowledge-based ontological tokenization framework that replaces statistical derivation with declarative classification grounded in a formal ontology of engineering entities (OEE). We formalize TOTEN as the triple <O, classify, {inst_tau}>: the ontology gathers types, structural principles, composition relations, and preservable invariants; the classification function maps raw text into typed regions; and the instantiator family yields a self-descriptive structured representation. Robustness derives from deterministic coupling with three external oracles: Pint (dimensional), Unicode Character Database (typographic), and RSLP (Portuguese morphology). Intrinsic evaluation covers four properties verifiable by construction -- ontological atomicity, dimensional equivalence, typographic robustness, and numerical reconstruction -- over an internal, physically validated benchmark (EngQuant, N=800) and four Brazilian Portuguese external corpora (N=1771 eligible cases). We also report detection recall, distinguishing coverage from conditional atomicity. Against eight state-of-the-art baselines, TOTEN achieves unit ontological atomicity in all contrasts and numerical reconstruction of 0.775-0.904 on external corpora, vs. 0.627-0.703 for the best baseline (Quantulum3); on EngQuant, 0.780 vs. 0.340. Differences are statistically significant (McNemar with Holm correction). Spearman correlation between internal and external rankings confirms concurrent validity of the control benchmark. Dimensional equivalence shows statistical parity with Pint, the oracle from which the system inherits dimensional authority.

• The paper introduces TOTEN, a framework that replaces statistical tokenization with a declarative, ontology-driven approach for precise handling of technical quantities in PT-BR.
• It leverages external oracles like Pint and Unicode UCD to ensure robust dimension validation, typographic normalization, and accurate numerical reconstruction.
• The findings demonstrate significant improvements in ontological atomicity and numerical metrics, paving the way for enhanced preprocessing in neurosymbolic and regulatory NLP applications.

Knowledge-Based Ontological Tokenization of Technical Quantities in Brazilian Portuguese: The TOTEN Framework

Motivation and Theoretical Context

Tokenization remains a foundational yet often problematic stage in NLP pipelines, particularly when operating on technical domains and non-English languages. Statistical tokenizers such as BPE, WordPiece, and SentencePiece prioritize vocabulary compression and distributional statistics, resulting in subword sequences that fracture semantically atomic entities—physical quantities, composite units, locale-specific numerics, and symbolic expressions. This fragmentation undermines downstream semantic preservation, notably in scientific texts and technical corpora where accuracy in entity recognition is critical. Contemporary research demonstrates that the input structure influences numerical reasoning and arithmetic accuracy in LMs, independently of model scale [3, 4]. Existing solutions in quantitative extraction for English (e.g., Quantulum3, CQE, GLiNER) do not adequately model normative Brazilian Portuguese (PT-BR) technical vocabulary or preserve domain-specific invariants.

TOTEN addresses these deficiencies by architecting an ontological tokenization framework, trading statistical vocabulary induction for declarative ontological commitment. The core shift is from statistical, semantically blind segmentation to a knowledge-based system explicitly representing engineering entities, structural principles, and composition relations, extensible for technical evolutions and verifiable by construction. This approach orients tokenization toward the epistemic demands of technical semantics, dimensional authority, and robustness to varied notation.

Formal Ontology and Tokenization Architecture

TOTEN is grounded in a heavyweight ontology of engineering entities (OEE), formalized as a quadruple $O = \langle T, P, R, I \rangle$—a finite set of primary types ($T$), structural principles ($P$), explicit composition relations ($R$), and invariants ($I$) preserved across representations. The ontology enumerates key types: physical quantity, technical prose, technical identifier, formal operator, universal constant, structural relation, symbolic expression, pure number, and hierarchical reference. Each type is characterized by a signature and invariants (e.g., dimensional homogeneity for physical quantities). The OEE is open to extension but closed for modification, supporting incremental domain adaptation.

Ontological tokenization in TOTEN is explicitly decoupled from statistical approaches. Raw input is mapped via a classification function to typed regions, followed by instantiation: the indexed family $\{ \text{inst}_\tau \}$ formats these regions into structured tags while preserving literal residuals. The system's architecture leverages coupling with authoritative external oracles: Pint for SI units and dimensional analysis, Unicode UCD for typographic identity, and RSLP for robust morphological analysis in Portuguese. This orchestration achieves domain-aligned, semantically typed segmentation, admitting only compositions validated by ontological relations and invariants.

Evaluation Protocol and Comparative Results

TOTEN’s intrinsic evaluation is built on four verifiable properties: ontological atomicity, dimensional equivalence, typographic robustness, and numerical reconstruction. Each property is formalized for rigorous cross-system comparison, with statistical significance measured via McNemar tests, effect sizes by Cohen’s $h$, and multiple comparisons corrected by the Holm procedure. The evaluation spans an internally generated benchmark corpus (EngQuant) validated by OpenSeesPy simulation and further public PT-BR corpora: MMMLU PT_BR, BLUEX, ENEM Maritaca, Alvorada-Bench.

TOTEN achieves categorical superiority in atomicity ($H_1 = 1.000$), demonstrating unit recall and unit atomic emission for all annotated entities. It matches Pint in conditional dimensional equivalence ($H_2 = 0.968$ vs Pint’s $0.985$), as expected given Pint’s role as dimensional oracle. Typographic robustness reaches $T$0, outperforming comparatives by systematic normalization of notational variants via Unicode UCD. Numerical reconstruction is the most discriminative metric: TOTEN achieves $T$1 internally and $T$2—$T$3 externally, compared to Quantulum3's $T$4—$T$5. The system ranking by these metrics correlates strongly across internal and external corpora, confirming benchmark validity.

Ablation studies demonstrate that dimensional coverage and numerical reconstruction are not mere wrappers over Pint. Disabling Pint, Unicode UCD, or RSLP produces orthogonal and significant drops in performance, highlighting the architectural necessity and complementarity of these oracles.

Implications and Limitations

TOTEN’s categorical advantage in ontological atomicity fundamentally distinguishes it from statistical and hybrid extractors. The explicit separation of classification and instantiation enables verifiability, auditability, and extensibility in technical domains—properties aligned with reproducibility requirements in knowledge-based systems [11, 53]. Dimensional equivalence with Pint reinforces the design’s correctness; any deviation would indicate methodological inconsistency. The preservation of IEEE 754 canonicalized values, locale, and typographic identity in the output tags addresses empirical findings that input structure materially impacts downstream reasoning [3, 4].

Practically, TOTEN opens the prospect of input preprocessing layers for neurosymbolic and LLMs, especially for PT-BR technical applications, where tokenization bias, fragmentation, or underrepresented vocabulary can substantially degrade performance. The framework supports integration with monolingual LMs and NER pipelines and lays the foundation for extensible ontological annotation in normative/legal corpora (e.g., NBR, ABNT), pending benchmark availability.

Limitations exist by design: literal preservation is prioritized, leaving typographically degraded notation unresolved. Dimensionless coverage is whitelisted to avoid lexical collision. Ambiguity resolution is architected but deferred to future implementation via the output tag’s attributes. Non-SI dimensional units are subject to explicit inclusion strategies.

Theoretical and Future Directions

TOTEN formalizes ontological tokenization as a triple $T$6, amenable to evaluation by properties and compatible with knowledge-based reproducibility standards. This shift from distributional compression to declarative encoding aligns with the neurosymbolic synthesis program and responds to contemporary critiques regarding the limits of statistical learning absent explicit structure [31, 32, 33, 34, 35].

Future development targets: (i) human-in-the-loop ambiguity resolution, (ii) external validation on normative/legal PT-BR corpora, (iii) training of native LMs on ontologically typed vocabulary, and (iv) quantification of downstream impact on consumer models’ accuracy and robustness.

Conclusion

TOTEN establishes a rigorous, knowledge-based ontological tokenization regime for technical notation in Brazilian Portuguese. By architecting a formal ontology, leveraging authoritative oracles, and defining an output language that preserves semantic entity structure, TOTEN achieves categorical gains in atomicity and numerical reconstruction with dimensional parity relative to external oracles. The framework has immediate practical utility for technical and regulatory NLP workflows in PT-BR and offers a principled foundation for further research in ontologically grounded preprocessing and neurosymbolic integration.