ObfusQA: Robustness Under Obfuscation

Updated 3 July 2026

ObfusQA is a suite of evaluation methodologies, benchmarks, and frameworks that rigorously investigates how semantic-preserving obfuscation impacts LLM and program analysis performance.
It systematically categorizes obfuscation strategies—including lexical, structural, semantic, and linguistic transformations—to generate challenging, real-world testing scenarios for program understanding and security analysis.
ObfusQA defines robust evaluation protocols and metrics, such as compile pass rates, exact match accuracy, and quantum obfuscation measures, to effectively assess adversarial robustness and model vulnerabilities.

ObfusQA refers to a suite of evaluation methodologies, benchmarks, and frameworks that systematically investigate the robustness and limitations of LLMs and program analysis systems in the presence of semantic-preserving obfuscation transforms. The term, introduced and popularized in leading literature, encompasses both the generation of obfuscated artifacts across multiple abstraction layers and the assessment of model or tool performance on question-answering, comprehension, and reasoning tasks over such artifacts. ObfusQA targets critical domains such as program understanding, reverse engineering, security analysis, and factual question answering, providing rigorous empirical baselines for adversarial and adversarially robust evaluation.

1. Formal Obfuscation Taxonomies in ObfusQA

ObfusQA frameworks categorize obfuscation strategies along several orthogonal axes:

Lexical (Symbol) Obfuscation: Systematic renaming of identifiers in code or question entities (e.g., variable, function, macro, or named-entity tokens) using bijective maps, erasing all surface-level hints present in the training data (Zhang et al., 2024, Nikiema et al., 14 Apr 2025).
Structural (AST-level) Obfuscation: Application of control-flow restructuring (e.g., dead-code injection, control-flow flattening, call inlining), increasing McCabe’s cyclomatic complexity and producing variants with altered execution paths but unchanged semantics (Zhang et al., 2024, Nikiema et al., 14 Apr 2025).
Semantic (Literal/Implementation) Obfuscation: Replacement of literals (strings, integers) or algorithmic fragments by semantic-equivalent encrypted or algorithmically distinct variants, often involving runtime decryption or alternative implementation logic (Zhang et al., 2024, Nikiema et al., 14 Apr 2025, Ghosh et al., 10 Aug 2025).
Linguistic Obfuscation in QA: Transformations on textual input such as Named-Entity Indirection (NEI), Distractor Indirection (DI), and Contextual Overload (CO), which increase the indirection or noise in factual questions while preserving answer content (Ghosh et al., 10 Aug 2025).

For quantum software, circuit-level and code-level obfuscation primitives are also formalized, utilizing quantum gates, identities, superposition, and quantum-based opaque predicates to cloak functionality (Parayil et al., 22 Dec 2025, Bartake et al., 31 Mar 2025).

2. Benchmark Construction and Dataset Design

Benchmarks in ObfusQA are designed to rigorously control for training-set exposure and overfitting, using multi-layered obfuscation. Notable procedures and datasets include:

Program ObfusQA Benchmarks: Selection of function-level artifacts from post-training PRs in real open-source repositories with official test coverage; application of three-level (symbol, structure, semantic) obfuscation, accompanied by context dependencies and noisy declarations (Zhang et al., 2024).
Factual QA ObfusQA (ObfusQAte Dataset): Construction of base-question corpora (e.g., from TriviaQA), then LLM- and human-verified pipeline to generate NEI, DI, and CO variants, with annotator agreement (Cohen’s κ) tracked for data quality (Ghosh et al., 10 Aug 2025).
IoC Recovery Benchmarks: Instrumentation of JavaScript code to plant a single ground-truth IoC (e.g., IP address) per program, followed by a pipeline of twelve cumulative transformation phases—Base64 encoding, renaming, structural obfuscation, XOR/AES encryption, and all combinations thereof—for a total of thousands of challenging instances (Morales et al., 7 May 2026).

A recurring emphasis is on preserving functional correctness (i.e., all observables preserved under transformation), with test suites and, in code settings, compile/test pass rates used to ensure equivalence (Zhang et al., 2024, Nikiema et al., 14 Apr 2025).

3. Evaluation Protocols and Metrics

ObfusQA protocols assess model robustness through tailored tasks and quantitative metrics, including:

Code Comprehension and Generation:
- Description Generation: Free-form explanation of an obfuscated code snippet, with correctness judged via a semantic similarity oracle S(·,·); thresholding on S<0.5 to distinguish significant semantic drift (Nikiema et al., 14 Apr 2025).
- Deobfuscation (Reconstruction): Model tasked to emit deobfuscated code; outputs are compiled and evaluated on ground-truth test cases, yielding Compile Pass Rate (CPR) and Test Pass Rate (TPR), and pass@k as primary metrics (Zhang et al., 2024, Nikiema et al., 14 Apr 2025).
Factual QA Under Obfuscation:
- Exact Match Accuracy (EM): Normalized model answer is checked for presence in the official answer set, with token-level normalization to ensure fairness (Ghosh et al., 10 Aug 2025).
- Failure/Hallucination Analysis: Rates of spurious or off-domain responses in the presence of distractors and overload in the question.
IoC Extraction under Adversarial Transformations:
- Detection Rate (DR): Fraction of queries for which the model flags the presence of an IoC.
- Extraction Accuracy (Acc): Fraction of correct value extractions among positive detections.
- False Positive/Negative Rates, Uncertainty (“Don’t Know”) Rate: Explicit tracking of errors and model abstentions, essential for risk-sensitive applications (Morales et al., 7 May 2026).

Secondary analyses include layer-wise probing of hidden-state norms, model membership-inference AUROC under obfuscation, and error-type breakdown (e.g., hallucination vs. failure to trigger).

4. Algorithmic and Theoretical Frameworks

ObfusQA systems leverage both empirical and formal apparatus:

Adversarial Optimization for Obfuscation: Bilevel formulations for adversarial code perturbation using obfuscation transforms. Given model θ, the attacker solves $\min_{z,u} \ell_\text{attack}(P'(z,u);θ)$ under budget and equivalence constraints, with discrete control of edit locations (z) and transform choices (u). Projected gradient-descent (PGD) and alternating optimization (AO), with randomized smoothing, achieve substantial gains in model evasion (Srikant et al., 2021).
Obfuscation Construction Detection: For reverse-engineering, semantic-feature extraction by IR normalization and symbolic execution enables multi-label, multi-class classification of both transform types and their concrete implementations (constructions), using chain-ensemble methods (Random Forests, Extremely Randomized Trees) to detect multi-layer obfuscation with up to 91% accuracy (Tofighi-Shirazi et al., 2019).
Quantum Obfuscation via Circuit and Predicate Primitives: Quantum-exclusive transformations—such as random gate conjugation, cloaked gate mappings, and quantum-opaque predicates—are selected to maximize indistinguishability or enforce correct behavior on all density matrices while yielding polynomially bounded overhead (Parayil et al., 22 Dec 2025, Bartake et al., 31 Mar 2025).

5. Empirical Findings and Theoretical Implications

Consistent empirical results reveal:

Model Vulnerability to Obfuscation: Even state-of-the-art models (GPT-4o, Claude 3.5 Sonnet, DeepSeek-Coder-V2) exhibit steep drops in comprehension, description, and generation accuracy under layered obfuscation. In OBFUSEVAL, the steepest collapse (up to 62.5% decrease in TPR) occurs when symbol and structural obfuscation are combined (Zhang et al., 2024). Lexical/symbolic transformations have more adverse effect than semantic obfuscation alone in certain regimes (Nikiema et al., 14 Apr 2025).
Obfuscation Axes Exhibit Differential Hardness: Variable renaming and entity indirection cause more substantial degradation than structural noise (dead code, contextual overload); literal encryption, XOR/AES cryptographic wrapping, and semantic implementation changes are most resilient to all tested LLM architectures (Nikiema et al., 14 Apr 2025, Ghosh et al., 10 Aug 2025, Morales et al., 7 May 2026).
LLMs’ Deobfuscation Weaknesses: LLMs rely heavily on surface lexical patterns for inference and consistently struggle to reconstruct program logic or extract IoCs once syntactic or structural cues are removed or encrypted. Prompt engineering (few-shot, chain-of-thought) only marginally recovers this deficit (Ghosh et al., 10 Aug 2025, Morales et al., 7 May 2026).
Quantum Obfuscation Resists Both Static and LLM Reverse Engineering: Randomized conjugation and quantum gate obfuscations render even advanced LLM-based analysis incapable of detecting inserted malicious code, as demonstrated by a 0% detection rate post-obfuscation of a keylogger (Bartake et al., 31 Mar 2025, Parayil et al., 22 Dec 2025).

6. ObfusQA System Design and Future Directions

Based on empirical and methodological findings, recommended directions and open challenges include:

Hybrid Analysis Pipelines: Integration of static analysis (e.g., IR extraction, control/data-flow graphs) and symbolic execution with LLM postprocessing is necessary to recover hidden semantics or decrypt embedded payloads (Tofighi-Shirazi et al., 2019, Morales et al., 7 May 2026).
Adversarial and Curriculum Fine-Tuning: Exposing models to progressively obfuscated examples (lexical → structural → semantic/cryptographic) can improve robustness, though effectiveness against encryption-induced “hard walls” is structurally limited (Nikiema et al., 14 Apr 2025, Morales et al., 7 May 2026, Ghosh et al., 10 Aug 2025).
Automated, Renewable Benchmarks: Dynamically obfuscated datasets, rigorously validated for functional equivalence and with noise injected into both code and dependency context, are essential to prevent training leakage and maintain challenge intensity over time (Zhang et al., 2024).
Formal Metrics for Obfuscation Intensity: Beyond token length or cyclomatic complexity, entropy or mutual information-based measures may better quantify challenge hardness (Ghosh et al., 10 Aug 2025).
Quantum-Safe Security Arguments: For quantum software, transitioning from heuristic security to formal games based on quantum indistinguishability obfuscation (quantum iO) and reductions under standard assumptions (e.g., LWE) remains an important open line (Parayil et al., 22 Dec 2025, Bartake et al., 31 Mar 2025).

ObfusQA thus defines both a methodology and a research agenda for the rigorous evaluation of program and question-answering systems under semantics-preserving adversarial transformations, establishing necessary empirical and theoretical scaffolding for advancing software security, robust LLMs, and quantum program protection.