DITING Framework: Multi-Domain Evaluation

• The DITING framework is a multi-domain methodology addressing secure TEE analysis, reference-free code evaluation, and culturally sensitive translation benchmarking.
• It employs rule-based static analysis, distilled reasoning for LLMs, and multi-agent deliberation to detect vulnerabilities, assess code, and evaluate translations.
• Empirical results demonstrate high precision, efficiency, and alignment with human judgment, making it valuable for secure software, code generation, and translation research.

The term "DITING framework" refers to three distinct, high-impact frameworks in the fields of secure software, generative code evaluation, and machine translation, all of which appear in separate research contributions on arXiv (Ma et al., 21 Feb 2025, Yang et al., 26 May 2025, Zhang et al., 10 Oct 2025). Each instantiation of DITING is independently motivated and domain-specific: (1) DITING for TEE security partitioning analysis, (2) CODE-DITING for explainable code evaluation without references or test oracles, and (3) DITING for culturally nuanced, multi-dimensional benchmarking of web novel translation. This article summarizes each framework’s core architecture, methodologies, formal elements, empirical results, and implications.

1. DITING for Static Analysis of TEE Partitioning

The DITING static-analysis framework targets identification of partitioning vulnerabilities in Trusted Execution Environment (TEE) applications, such as OP-TEE and Intel SGX. Its central objective is to discover “bad partitioning” issues—where boundaries between untrusted ("normal world") and trusted ("secure world") code are improperly configured, exposing applications to data leakage, unvalidated input, or unsafe use of shared memory. DITING leverages CodeQL for program analysis, constructs explicit Input/Output/Shared-memory data-flow graphs (IDF, ODF, MDF), and applies a rule-based engine to detect security violation patterns (Ma et al., 21 Feb 2025).

Three Security Rules

The rule engine implements three key inference rules:

1. Encryption of Output: Data emanating from the TEE must be encrypted before use as output. Formally, if $buf$ flows into output parameter $o$, the encryption operation $e(buf)$ must precede any write or assignment to $o$.
2. Input Validation: Input parameters $i$ must be validated (isChecked) before being used in array indexing or as arguments to memory operations, except for special cases such as direct TEE_Malloc followed by copy.
3. Deep Copy for Shared Memory: Shared-memory ($shm$) values must be deep-copied before any memory operation within the TEE; direct operations on untrusted references are prohibited.

Implementation and Data-Flow Analysis

DITING’s pipeline parses C/C++ TEE code to extract ASTs, call graphs, and parameter usage, constructing directed graphs tracing the propagation of input/output/shared memory through code paths. Paths are analyzed to check adherence to the formal rules, with violations flagged alongside source locations. Each analysis routine (ODF, IDF, MDF) is a specialized form of taint analysis, propagating sensitivity labels across procedure calls and control-flow edges.

Empirical Evaluation

Evaluation on the PartitioningE-Bench benchmark (110 cases: 90 vulnerable, 20 safe) yields an F1 score of 0.90 overall, with precision and recall for unencrypted output at 97.06% and 94.29%, respectively; for input validation, F1 = 0.82; for shared memory, F1 = 0.89. Application to open-source TEE projects found that among 68 alarms, 55 signaled real flaws such as unchecked TEE_MemMove and unencrypted debug outputs. DITING’s static analysis is efficient, with end-to-end analysis time of 21–35 seconds per 300–5000 LoC codebase, and millisecond-level rule-checking (Ma et al., 21 Feb 2025).

2. CODE-DITING: Reasoning-Based Metric for Code Evaluation

CODE-DITING is a reference- and test-suite-free code evaluation method, providing functionally aligned assessment of code generation models. It addresses limitations in both traditional (test-driven, reference-based) and LLM-as-Judge paradigms by distilling stepwise reasoning from a large teacher model (DeepSeek-R1-671B, 671B parameters) into much smaller student models (1.5B/7B parameters), yielding explainability, accuracy, and computational efficiency (Yang et al., 26 May 2025).

Model and Training Pipeline

CODE-DITING models are standard autoregressive Transformers, fine-tuned using a distillation objective that combines Kullback-Leibler divergence ($KL$) between soft teacher outputs and cross-entropy on ground-truth functional correctness, with $\alpha\approx0.7$ weighting:

$$\mathcal{L}(\theta) = \alpha \cdot KL\big(p_{\mathrm{teacher}}(\cdot|x) \,\|\, p_{\mathrm{student}}(\cdot|x;\theta)\big) + (1-\alpha)\cdot CE(y, p_{\mathrm{student}}(\cdot|x;\theta))$$

Training data is filtered for high teacher accuracy and logical reasoning traces, resulting in the curated CodeJudge-17K dataset.

Inference and Majority Voting

The student model generates a binary label with a 3–6 step reasoning trace per input. To increase robustness, majority voting across $M$ stochastic passes is employed; the final output is the most frequent label, with the corresponding majority explanation. If single-pass accuracy $P(A)>0.5$, binomial bounds guarantee rapid convergence to correct majority with increasing $M$.

Evaluation and Results

CODE-DITING 1.5B and 7B outperform similarly sized LLM-as-judge models and even surpass GPT-4o and DeepSeek-V3-671B, given only 1–2% of their parameter volume. For instance, CODE-DITING 7B achieves accuracy 0.806 and F1 0.782 on judge datasets, with empirical runtimes of 1–2 seconds per sample at $M=7$. The method demonstrates robustness to preference leakage (e.g., >93% label agreement, $\kappa>0.86$ across unseen code generators and paraphrased NL specs). Explainability is intrinsic, with the model supplying reasoning traces supporting its functional alignment determinations (Yang et al., 26 May 2025).

3. DITING for Multi-Agent Evaluation of Web Novel Translation

The DITING benchmark for web novel translation is the first comprehensive, culturally and narratively grounded evaluation framework for CN→EN machine translation in the highly informal web-novel genre. It decomposes translation quality into six fine-grained, phenomenon-driven dimensions, each assessed by custom metrics over 18,745 expert-annotated sentence pairs (the DiTing-Corpus) (Zhang et al., 10 Oct 2025).

Six Phenomenon-Driven Dimensions

1. Idiom Translation: Evaluates figurative/narrative equivalence for idiomatic expressions.
2. Lexical Ambiguity: Disambiguates polysemous/novel terms in situ, measuring sense selection accuracy.
3. Terminology Localization: Ensures genre/world-specific terms are contextually adapted for the target audience.
4. Tense Consistency: Assesses temporal and aspectual alignment between source and translation.
5. Zero-Pronoun Resolution: Requires the explicit recovery of omitted pronouns in English.
6. Cultural Safety: Detects introduction of harmful, biased, or unsafe content post-translation.

AgentEval and MetricAlign

AgentEval is a multi-agent architecture simulating expert deliberation: two scorer agents produce rationales and fine-grained (0/1/2) dimensional scores; a judge agent arbitrates, orchestrating debate and consensus. If unresolved, a final decision is rendered based on argument soundness. MetricAlign provides a meta-evaluation corpus of 300 pairs with scalar expert-quality labels, used to benchmark both traditional MT metrics (BLEU, chrF, BLEURT, COMET) and LLM-based evaluators.

AgentEval’s multi-agent debate variant achieves the highest correlation with human scores (Spearman $\rho=0.669$, 44.8% variance explained), more than doubling the explained variance of BLEU and COMET. Closed-source Chinese-trained LLMs (DeepSeek-V3, Qwen3) outperform both large generalist LLMs and commercial MT on idioms, terminology localization, and overall narrative style. Tense consistency is generally handled well, but zero-pronoun and cultural safety remain open challenges.

Corpus and Annotation Protocol

The DiTing-Corpus is sourced from major online literature platforms and annotated by two professional CN→EN translators and a third annotator, with labels for each dimension under calibrated, cross-checked guidelines. The annotation procedure follows a hybrid MQM+SQM protocol, with high inter-annotator agreement (simple agreement $\approx0.90$–$0.96$).

4. Integration, Adaptation, and Practical Deployment

Each DITING instantiation provides clear protocols for integration:

• DITING (TEE analysis): Requires a compilable TEE code base and CodeQL environment; the rule engine is extensible via Python routines with new taint source/sink definitions and rules.
• CODE-DITING: Distributed as a Python library/REST service for CI/CD integration, enabling high-throughput, explainable judgment of generated code. Supports batched and multi-GPU deployments, with no need for curated test cases or references.
• DITING (translation evaluation): All data, code, and scripts are open-source; AgentEval can be used for systematic benchmarking or as a training signal for future scoring LLMs.

5. Limitations and Prospective Directions

Domain-specific limitations are actively acknowledged:

• TEE analysis: Relies on complete CodeQL coverage; highly macro-driven codebases or legacy code can reduce analysis fidelity. Some false positives (e.g., copying constant data) and false negatives (complex path patterns) remain. Planned improvements include LLM-guided data-flow recovery and adaptation to new TEE frameworks (RISC-V, AMD SEV, IoT).
• CODE-DITING: Dataset coverage is limited by test and teacher filtering; majority voting cannot fully resolve borderline cases, though it significantly increases robustness.
• Translation benchmark: Presently limited to sentence-level evaluation and a 300-parallel meta-evaluation set. Document-level coherence, reinforcement learning of scoring agents, and user-feedback-based refinement are identified as future directions (Zhang et al., 10 Oct 2025).

6. Comparative Summary Table

DITING Instantiation	Application Domain	Core Technical Approach
DITING (TEE Partitioning)	Secure Software/TEE Static Analysis	Rule-based IR with data-flow analysis, taint propagation
CODE-DITING	Code Generation Evaluation	Distilled reasoning LLM, explainable, reference/test-free
DITING (Web Novel Translation)	CN→EN Machine Translation Benchmarking	Multi-agent evaluation, dimension-decomposition, human-aligned metrics

Each DITING framework addresses domain-specific challenges—security boundary correctness, explainable code assessment, or culturally sensitive translation quality—using tailored static analysis, distilled reasoning, or expert multi-agent deliberation strategies.

7. Impact and Significance Across Domains

The DITING collective frameworks exemplify new paradigms for: (1) automating detection of subtle partitioning flaws in secure enclaves, (2) supporting explainable, test-agnostic evaluation of code generation, and (3) setting multi-dimensional, debate-driven standards for translation quality. These approaches consistently foreground formal rules, human-aligned metrics, extensibility, and empirical rigor, with released benchmarks and code bases supporting reproducibility and future research (Ma et al., 21 Feb 2025, Yang et al., 26 May 2025, Zhang et al., 10 Oct 2025).