NetMasquerade: Securing SNI & ML Evasion

• NetMasquerade is a dual-strategy framework that protects network privacy by encrypting the SNI in a two-phase TLS handshake and employing reinforcement learning to evade ML-based detectors.
• It uses a protocol-level approach where the initial handshake omits host details, followed by an AEAD-encrypted renegotiation that securely conveys the true SNI while maintaining TLS integrity.
• The system integrates Traffic-BERT and SAC to craft minimally altered traffic flows that achieve high evasion rates, with experimental results showing negligible overhead and robust performance.

NetMasquerade refers to distinct advanced frameworks in network security, traffic privacy, and adversarial evasion, denoting (1) a protocol-level mechanism for concealing host identity by encrypting the Server Name Indication (SNI) in TLS/SSL handshakes, and (2) a black-box evasion attack methodology for subverting ML-based malicious traffic detectors through reinforcement learning-guided traffic manipulation. Both paradigms exemplify contemporary approaches to undermining in-network traffic classification and detection at different layers of the protocol and security stack (Khandkar et al., 2021, Liu et al., 16 Oct 2025).

1. Protocol-Level SNI Encryption (Two-Phase TLS Handshake)

The first paradigm of NetMasquerade addresses the privacy exposure of the SNI field in the TLS/SSL protocol. SNI, transmitted in cleartext in standard TLS handshakes, discloses the intended hostname to on-path observers and middleboxes, thus enabling censorship, discrimination, and surveillance. NetMasquerade achieves host privacy by splitting the TLS handshake into two distinct phases:

Phase 1: Initial "Blind" Handshake

A conventional TLS handshake is executed without the server_name extension in the ClientHello. The server responds with a default certificate, establishing an encrypted tunnel while leaking no host-specific information at the handshake metadata level.

Phase 2: Encrypted SNI Renegotiation

Within the AEAD-protected channel of phase 1, the client initiates a second handshake (TLS renegotiation or inner handshake in TLS 1.3), embedding the true SNI—encrypted via the session key—into the server_name extension. The server then supplies the authentic certificate corresponding to the intended destination.

This approach renders the underlying SNI invisible to all on-path entities, preserving standard TLS handshaking cryptographic guarantees.

Message Flow and Structure

The protocol-level exchange is as follows:

Client Server ------ ------ TCP SYN --------------------> <-------------------- SYN,ACK TCP ACK --------------------> ClientHello --------------------> ServerHello (default cert, no SNI) ...key-exchange... ...ServerKeyExchange, ServerHelloDone ClientKeyEx,ChangeCipher,F. -> ChangeCipherSpec,Finished --- AEAD tunnel established --- (enc) ClientHello (encrypted SNI) ---> (enc) ServerHello,Certificate (real host) (enc) key exchange, Finished <--- (enc) key exchange, Finished -- True secure channel (real host) established --

LaTeX description of critical structures:

• ClientHello (phase 1, no SNI):

$$\mathit{HandshakeType}=0x01 \,\Vert\, \mathit{ClientHelloLength} \,\Vert\, \bigl[\mathit{Version}\, \mathit{Random}_C\, \mathit{Extensions}\ (\text{no server\_name})\bigr]$$

• ClientHello (phase 2, encrypted SNI):

$$\mathit{server\_name\_ext} = \mathit{extension\_type}=0x00 \,\Vert\, \mathit{encrypted\_SNI\_blob}$$

Where

$$\mathit{encrypted\_SNI\_blob} = \mathsf{AEAD}_{K_{C\to S}}(\text{"hostname"})$$

2. Cryptography Underpinning Protocol Operation

Key exchange and key derivation use standard TLS (1.2/1.3) mechanisms without modification. ECDHE-based ephemeral keys derive a master secret via

$$ms = \mathrm{PRF}(pms,\, "master\ secret",\, R_C\Vert R_S)$$

From which traffic keys, IVs, and MAC keys are expanded. Phase 2 uses AEAD encryption algorithms (e.g., AES-GCM, ChaCha20-Poly1305) from this key block. All handshake messages after phase 1, including the true SNI, are AEAD-protected.

No custom cryptosystems or trust anchors are introduced. All cryptographic proofs and assurances of TLS/SSL remain intact (Khandkar et al., 2021).

3. Implementation: Protocol and System Modifications

Client Modifications:

On new connections, the client omits the server_name extension in the first handshake and includes it (as the desired hostname) in the second (encrypted) handshake. Pseudocode:

on new_tls_connection(): if handshake_count == 1: omit_extension("server_name") else: include_extension("server_name", desired_hostname) proceed_with_standard_tls_handshake() handshake_count += 1

Server Modifications:

Servers must possess a CA-signed default certificate and handle handshakes lacking the SNI extension, replying with the default cert on phase 1 and processing the real SNI only after decryption in phase 2.

on incoming_ClientHello(ch): if new_tls_connection and handshake_count == 1: if not ch.has_extension("server_name"): use_default_certificate() else: standard_SNI_processing(ch) else: standard_tls_processing(ch)

No adjustments are required for routers or middleboxes.

4. Experimental Evaluation: Overhead, Compatibility, and Compliance

Experiments on live Internet connections between custom client and fronting server demonstrated:

Scenario	Total Transfer Time	Overhead (%)
Baseline TLS (100 MiB)	3.204 s	0 %
NetMasquerade (two handshakes)	3.215 s	0.34 %

• Handshake round-trip times for both phases average ~45 ms, with negligible difference between baseline and NetMasquerade.
• All aspects of the method align with IETF TLS/SSL (RFC 5246/8446) requirements—no new extensions, no protocol-level incompatibility.
• Middleboxes and Deep Packet Inspection observe only two standard-looking TLS handshake bursts; the real SNI is never exposed in cleartext (Khandkar et al., 2021).

5. Security Properties and Threat Mitigation

NetMasquerade, in the TLS context, preserves TLS’s integrity, confidentiality, and authentication guarantees for all application data and the SNI extension. Comparative properties relative to legacy TLS:

Property	Legacy TLS	NetMasquerade
Confidentiality	yes	yes (in phase 2)
Integrity	yes	yes (in phase 2)
Authenticity	yes	yes (phase 1 & 2)
SNI privacy	no	yes
Middlebox view	SNI always in cleartext	only default cert visible

No SNI leakage occurs; only benign, non-identifying default certs are visible on the wire. The method depends entirely on established TLS cryptographic analyses; there is no introduction of proprietary cryptosystems.

6. NetMasquerade as a Black-Box ML Evasion Attack

A distinct application of NetMasquerade targets ML-based malicious traffic detection, as described in (Liu et al., 16 Oct 2025). It achieves high-confidence evasion under the hard-label black-box threat model—that is, detectors yield only pass/block binary feedback, with no exposure of model architecture, confidence, or feature weights.

The methodology includes:

• Training Traffic-BERT, a specialized BERT-style encoder, on Internet-scale benign packet traces (MAWI, 1M flows). Tokenization encompasses packet sizes and inter-packet delays (IPD), mapped to 1606 and 56 vocabulary items, respectively.
• Representing adversarial flow crafting as a sequential decision process in a finite-horizon MDP, with packet-level action space—mask, insert chaff, or perturb delay.
• Using SAC (Soft Actor-Critic) reinforcement learning, with Traffic-BERT as a frozen oracle, to synthesize minimally perturbed, benign-appearing flows.

Experimental results highlight:

• Attack Success Rate (ASR) of 96.65%–99.90% across 80 attack scenarios and six ML detectors, all with AUC & F1 ≥ 0.92.
• Median evasion achieved with <5 modifications per flow; KL-divergence of 0.009–0.013 for bandwidth preservation in DoS flows.
• Throughput of 4.2K packets/sec in generation—substantially outpacing prior GAN- or RL-based methods.
• Resilience to certified-robust (ℓ_p-bounded) defenses, achieving >80% ASR even under such controls.

7. Assumptions, Limitations, and Defenses

Both NetMasquerade instantiations operate under specific assumptions:

• Protocol-SNI defense: Assumes SNI exposure is the principal vector for censorship/classification; not designed for traffic analysis, flow fingerprinting, or complex DPI.
• ML-Evasion: Assumes detectors operate on per-packet statistical summaries, not deep semantic payload checks. Relies on probe-and-response (pass/block) for reward feedback and does not require knowledge of model internals or parameters.

Limitations for the ML-attack paradigm include:

• Dependence on allowed probe budget (1k–2k probes).
• Susceptibility to detection in systems employing payload semantic checks or strict protocol state verification.

Potential defenses include adversarial training with traffic-space perturbations, robustness certification, randomized inference, and payload-verification-based rejection.

---

NetMasquerade thus encapsulates both a TLS protocol modification for SNI privacy (Khandkar et al., 2021) and a traffic-pattern manipulation strategy for ML detector evasion (Liu et al., 16 Oct 2025). Both exploit protocol or statistical blind spots in classification and detection systems, demonstrating the evolving arms race in traffic privacy and adversarial resilience.