Embedding-Based Protocol Analysis

Updated 5 July 2026

Embedding-based protocol analysis is a representational strategy that transforms diverse protocol artifacts into analyzable latent spaces.
It employs techniques such as learned vector embeddings, graph factorization, and formal modeling for tasks like anomaly detection and state-machine extraction.
The approach drives practical applications in network security, protocol debugging, and vulnerability discovery with impressive empirical outcomes.

Embedding-based protocol analysis denotes a family of methods that transform protocol artifacts into representations that can be compared, clustered, searched, or executed in an alternative analytical space. In recent work, those artifacts include fused SS7/Diameter/GTP signalling records, heterogeneous DNS entity graphs, protocol-implementation source code, black-box packet traces, run-time memory states, and formal execution models; the resulting analyses support anomaly triage, malicious-domain and IP-reputation inference, state-machine extraction, implementation differencing, temporal property checking, and safety-preserving protocol execution on alternative substrates (Garg et al., 16 Mar 2026, Qin, 2024, Wei et al., 2024, Ferreira et al., 2021, Schett et al., 2021).

1. Scope and meanings of “embedding”

The literature uses “embedding” in several technically distinct senses. In the most direct data-driven sense, a protocol artifact is mapped into a learned vector space: mobile-core signalling records are serialized into text and embedded by six models before unsupervised anomaly detection (Garg et al., 16 Mar 2026); passive DNS entities are embedded jointly by random-walk-based matrix factorization on a heterogeneous graph (Qin, 2024); and protocol source code is chunked, embedded, and retrieved through a vector store for large-language-model-guided state-machine inference (Wei et al., 2024). In creative-process protocol analysis, design moves are embedded into a semantic space so that pairwise similarities can be converted into a fuzzy linkograph and metrics such as entropy, link density, and critical moves (Lee et al., 8 Mar 2026).

A second usage is representational rather than strictly vectorial. Prognosis learns abstract Mealy-machine models and richer extended Mealy-like models with registers from closed-box packet I/O, while StateInspector represents protocol state through a minimal set of candidate memory locations whose values uniquely determine the current state (Ferreira et al., 2021, Stone et al., 2021). BinPRE and the mixed-protocol PSM framework similarly rely on symbolic or fuzzy representations—operator-sequence signatures, format-cluster sequences, and fuzzy membership vectors—rather than neural embeddings (Jiang et al., 2024, Yang et al., 2024).

A third usage is formal. The block-DAG work proves an embedding of any deterministic Byzantine fault tolerant protocol $\mathcal{P}$ into a block DAG while maintaining all safety and liveness properties of $\mathcal{P}$ (Schett et al., 2021). In enrich-by-need security analysis, minimality is defined with respect to the homomorphism preorder or the embedding preorder, namely the preorder of injective homomorphisms (Dougherty et al., 2018). This suggests that “embedding-based protocol analysis” is not a single method class but a broader representational paradigm.

A separate line uses “embedding” literally as protocol placement in a carrier. The micro-protocol engineering work studies the embedding of steganographic control protocols into digitized audio transmissions, with covertness, capacity, reliability, synchronization, and timing as the main design constraints (Naumann et al., 2015). That line is about protocol engineering inside a carrier, not about latent-space analysis.

2. Representational substrates

The choice of representation determines both the observable semantics and the feasible downstream analyses.

Protocol artifact	Representation	Representative use
Per-subscriber, per-minute fused SS7/Diameter/GTP record	Single-line serialized text embedded by six models	Cross-protocol anomaly consensus (Garg et al., 16 Mar 2026)
Passive DNS domains and IPs	Similarity-enhanced heterogeneous graph with joint embeddings	MDD and IRE (Qin, 2024)
FSM-related protocol source code	Syntax-preserving code chunks in a vector store	RAG-based FSM inference (Wei et al., 2024)
Closed-box protocol I/O	Deterministic Mealy machine or extended Mealy-like model with registers	Model learning and differencing (Ferreira et al., 2021)
Run-time implementation state	Candidate state memory and state-defining memory	Grey-box state learning (Stone et al., 2021)
Message formats and field behavior	Operator-sequence similarity, fuzzy membership vectors, clustered format traces	PRE and PSM inference (Jiang et al., 2024, Yang et al., 2024)

In the mobile-core setting, the unit of analysis is a per-subscriber, per-minute fused record. Messages are grouped by a canonical subscriber identifier and a UTC minute timestamp; only windows containing at least two protocol families are retained. The record is normalized by removing newline characters, collapsing repeated whitespace, and converting the record into a single-line token sequence, which is then embedded without truncation; records that exceed a model’s context window are simply not embedded by that model (Garg et al., 16 Mar 2026).

In DNS analysis, the representation is a heterogeneous weighted graph whose nodes are domains and IPs. Direct domain–IP interactions are augmented with domain–domain and IP–IP similarities derived from shared hosts and shared resolved neighbors, so that higher-order homogeneous and heterogeneous proximity is retained before factorization (Qin, 2024). The final embeddings place domains and IPs in a shared latent space.

In source-based protocol analysis, ProtocolGPT first filters out non-FSM code, then splits FSM-related code with syntax awareness so that functions, classes, and syntactic integrity are preserved. The resulting chunks are embedded and stored in a vector store; the query prompt is likewise embedded, and the most similar chunks are retrieved for LLM analysis (Wei et al., 2024). The motivation is scale: the reported implementations range from 84,763 tokens for RTSP feng to 5,556,945 tokens for IKEv2 strongSwan, well beyond a single context window (Wei et al., 2024).

Grey-box and binary-analysis methods expose an important boundary condition: useful embeddings need not be continuous vectors. StateInspector defines a candidate state memory location as any memory location that takes the same value after the same inputs, for any run of the protocol, and a set of state-defining memory as a minimal subset of candidate state memory locations whose values during a protocol run uniquely determine the current state (Stone et al., 2021). BinPRE approximates field semantics by the sequence of instruction operators that access the bytes, and the mixed-protocol PSM framework represents messages by fuzzy membership values over frequent contiguous substrings (Jiang et al., 2024, Yang et al., 2024).

3. Workflow patterns and core computations

A recurrent workflow is representation construction followed by a task-specific analysis stage. In the mobile-core setting, each embedding model produces a vector for the fused text, an Isolation Forest with feature standardization outputs a binary anomaly decision, and the ensemble assigns a consensus score $S(r)$ equal to the number of embedding models that flag record $r$ as anomalous. If some models cannot process a long record, the coverage $N(r)$ records the number of available decisions, so that $0 \le S(r) \le N(r) \le 6$ . The score is explicitly a count of agreement, not a probability (Garg et al., 16 Mar 2026).

In DNS analysis, the unsupervised core is a random-walk-based matrix factorization derived from DeepWalk’s matrix-factorization view. With adjacency ${\bf A}$ , degree matrix ${\bf D}$ , graph volume $w$ , maximum random-walk length $K$ , and number of negative samples $\mathcal{P}$ 0, the factorized matrix is

$\mathcal{P}$ 1

and the embedding objective is

$\mathcal{P}$ 2

A semi-supervised extension adds graph regularization and an auxiliary classifier, preserving the shared domain/IP embedding space while injecting label information (Qin, 2024).

ProtocolGPT instantiates a retrieval-augmented generation workflow. The LLM first identifies FSM-related code paths, then extracts the set of protocol states $\mathcal{P}$ 3 and message types $\mathcal{P}$ 4, and finally infers state transitions one current state at a time. The process is stepwise because direct one-shot inference of the full FSM is reported as unreliable; the output is constrained to machine-readable JSON-like structures containing current_state, receive_message, and next_state (Wei et al., 2024).

Prognosis follows an active model-learning workflow in the Angluin-style Minimally Adequate Teacher setting. The learner operates over an abstract alphabet and uses the SUL as a query oracle through membership queries $\mathcal{P}$ 5 and equivalence queries $\mathcal{P}$ 6, while the Adapter translates between abstract traces and concrete packets through abstraction and concretization functions

$\mathcal{P}$ 7

with the requirement that $\mathcal{P}$ 8 (Ferreira et al., 2021). Instead of hand-coding concretization, Prognosis instruments a trusted reference implementation so that it can serve as a concretization oracle.

These workflows differ in observability—closed-box packet interaction, passive traffic analysis, source-level retrieval, and grey-box memory inspection—but they share a structural pattern: a representation compresses raw protocol complexity into an analyzable substrate, and the analysis stage exploits that substrate for ranking, classification, or model induction.

4. State-machine inference, reverse engineering, and implementation semantics

State-machine extraction is the most prominent downstream use of embedding-like representations. Prognosis defines a deterministic Mealy machine as a tuple

$\mathcal{P}$ 9

and extends this with registers $S(r)$ 0 and parametric packet symbols so that a transition has the form

$S(r)$ 1

The restricted update language—copying a register, copying an input value, incrementing by $S(r)$ 2—is sufficient to recover sequence-number-like behavior in TCP and QUIC (Ferreira et al., 2021). The same learned models support temporal property checking, equivalence or difference analysis between implementations, and model-based test generation.

ProtocolGPT recovers a finite state machine directly from implementation source code. The formal target is the quintuple

$S(r)$ 3

where $S(r)$ 4 is the set of message types, $S(r)$ 5 the set of protocol states, $S(r)$ 6 the set of initial states, $S(r)$ 7 the transition function, and

$S(r)$ 8

the transition relation (Wei et al., 2024). The methodology is source-grounded rather than trace-grounded, and the paper emphasizes that implementation FSMs can diverge substantially from RFC-level expectations.

StateInspector remains within automata learning but changes the state classifier. It learns deterministic, complete Mealy machines, yet merges and distinguishes states using concrete program state observed through run-time memory snapshots and I/O. A merge is allowed when states are I/O-equivalent to depth $S(r)$ 9 and their differing memory is shown not to be state-defining by taint and concolic analysis; the paper uses a forward exploration bound $r$ 0 instructions for this check (Stone et al., 2021). This yields models that are deeper and cheaper to learn than purely black-box I/O models.

Binary reverse engineering extends the same logic to message formats and mixed traffic. BinPRE performs format extraction by comparing the operator sequences that access adjacent byte ranges and merging them when the normalized Needleman–Wunsch score exceeds $r$ 1; it then infers field type and function through atomic semantic detectors and a cluster-and-refine stage (Jiang et al., 2024). The mixed unknown-protocol framework first clusters messages into protocol format clusters using fuzzy membership vectors and auto-converging DBSCAN, then clusters sessions by protocol type with Needleman–Wunsch and K-Medoids, and finally infers a probabilistic PSM with transition filtering based on

$r$ 2

(Yang et al., 2024). The common objective is semantic recovery: states, transitions, and message structure are inferred from representations that are simpler than the underlying execution.

5. Empirical applications and reported outcomes

Setting	Main reported outcome	Paper
SS7/Diameter/GTP fused anomaly analysis	96,819 / 219,294 = 44.15% flagged by at least one model; 2,120 / 219,294 = 0.97% flagged by all six; for $r$ 3 all flagged records are synthetic	(Garg et al., 16 Mar 2026)
Passive DNS security analysis	Best average improvement over the second-best baseline: 7.8% for MDD and 4.94% for IRE	(Qin, 2024)
Closed-box TCP/QUIC model learning	Reproduced a 6-state, 42-transition TCP model and found critical bugs in multiple QUIC implementations	(Ferreira et al., 2021)
Source-based FSM inference	Average precision/recall exceeding 90%; coverage improvement by more than 20%; two 0-day vulnerabilities	(Wei et al., 2024)
Binary PRE for field inference	Perfection 0.73; semantic inference F1 0.74 for types and 0.81 for functions; 5–29% higher branch coverage; one zero-day vulnerability	(Jiang et al., 2024)
Grey-box protocol state learning	Deep transition requiring 12 consecutive init messages exposed; black-box learning needed around 360k queries and still failed to terminate within 1M queries	(Stone et al., 2021)

The mobile-core study is notable because the anomaly is defined by fused consistency rather than per-message validity. Synthetic anomalies are created by swapping one field group at a time between fused records, preserving individual message validity while making the joint view contradictory. The reported odds ratios for thresholded consensus $r$ 4 are roughly $r$ 5, $r$ 6, $r$ 7, and $r$ 8 for $r$ 9 to $N(r)$ 0, and effectively $N(r)$ 1 for $N(r)$ 2 and $N(r)$ 3, because no original fused record reaches those thresholds (Garg et al., 16 Mar 2026). The practical effect is workload reduction: stricter consensus shrinks the inspection set from nearly half the corpus to under $N(r)$ 4.

In DNS, joint embedding of domains and IPs matters because prior methods usually represented only one entity type. The reported JDE(U) already outperforms baselines using only one type of entity or only low-order structure, while JDE(S) improves further through supervised regularization (Qin, 2024). The nearest-neighbor and visualization studies indicate that the embedding preserves protocol semantics such as IoT-related, gambling-related, pornography-related, and cloud-service-related groups.

In protocol implementation analysis, Prognosis demonstrates that learned models can expose specification ambiguity and concrete bugs. The QUIC case studies on Cloudflare Quiche, Google QUIC, and Facebook mvfst uncovered RFC imprecision around retry and packet-number-space reset, a Facebook mvfst inconsistency after connection close with potential denial-of-service implications, a QUIC-Tracker retry-handling problem involving a new UDP socket and random port, and a Google QUIC bug in STREAM_DATA_BLOCKED where the “Maximum Stream Data” field was always 0 and never updated (Ferreira et al., 2021). The mixed-protocol PSM framework likewise reports strong results on TLSv1.2 and SMTP, including $N(r)$ 5 for format clustering on mixed data, $N(r)$ 6 for session clustering with $N(r)$ 7, and final PSM matching coefficients of $N(r)$ 8 for TLSv1.2 and $N(r)$ 9 for SMTP (Yang et al., 2024).

Source-grounded and grey-box approaches turn protocol representations directly into security leverage. ProtocolGPT reports, for example, $0 \le S(r) \le N(r) \le 6$ 0 precision/recall on TLS 1.3, TLS 1.2, and L2TP, and identifies large FSM differences among IKEv2 implementations such as strongSwan, libopenikev2, Libreswan, and Openswan (Wei et al., 2024). StateInspector discovers a high-impact vulnerability in IWD 1.6 because message 4 of the WPA/2 4-way handshake does not move the state machine into a new state, allowing a key reinstallation attack (Stone et al., 2021). BinPRE’s inferred fields, in turn, were sufficient to uncover CVE-2024-31504 in FreeMODBUS, a buffer overflow involving a Length field and variable-length payload (Jiang et al., 2024).

6. Limits, misconceptions, and formal extensions

A common misconception is that embedding similarity is a neutral proxy for protocol significance. The creative-process literature argues the opposite: fixed embedding similarity can overestimate conceptual continuity and miss creative pivots, as in the shift from “stackable chair modules for compact storage” to “stackable wall modules for reconfigurable room layouts,” which may be treated as continued elaboration despite a substantial change in design intent (Lee et al., 8 Mar 2026). The paper identifies three open challenges: aligning similarity measures with creative significance, segmenting and representing multimodal design traces, and evaluating agentic systems in which embedding-based metrics enter the generation loop.

Practical limitations recur across network-protocol applications. Prognosis depends on a carefully chosen abstraction; if the abstract alphabet hides too much, distinct behaviors can collapse into the same abstract trace, causing apparent nondeterminism and potentially hiding bugs. The framework also does not yet handle richer environment quantities such as congestion, latency, or memory usage (Ferreira et al., 2021). The mobile-core anomaly study faces scarce real labels and therefore evaluates on synthetic cross-protocol-plausible anomalies (Garg et al., 16 Mar 2026). The DNS method uses static time-windowed graphs and therefore does not fully model dynamic DNS evolution (Qin, 2024). ProtocolGPT depends on code filtering, chunking, prompt design, and retrieval quality, while StateInspector assumes a finite state machine, a known input alphabet, bootstrap coverage of state-defining memory, and bounded loop depth and concolic exploration (Wei et al., 2024, Stone et al., 2021).

Another misconception is that embedding-based protocol analysis is necessarily neural and opaque. Formal work shows otherwise. The block-DAG theorem establishes that a deterministic BFT protocol can be embedded into a block DAG and that the resulting shim maintains all safety and liveness properties of the original protocol; the key insight is that the DAG acts as an efficient reliable point-to-point channel and simultaneously enables message compression (Schett et al., 2021). Enrich-by-need security analysis distinguishes homomorphism-minimal models from embedding-minimal models and provides algorithms for computing minimal models and set-of-support summaries through SMT-solver interaction (Dougherty et al., 2018). The hybrid analysis of stateful security protocols embeds mutable state into state-bearing messages so that CPSA can analyze message flow, then uses PVS, compatibility conditions, and bridge lemmas to rule out spurious state splits and recover genuine linear state evolution (Ramsdell et al., 2014).

Taken together, these results indicate that embedding-based protocol analysis is best understood as a representational strategy rather than a single algorithmic family. Its core operation is to move protocol evidence—messages, sessions, code, memory, or formal executions—into a representation that preserves enough semantics for the target task while making comparison and inference tractable. The resulting trade-off is always between representational compression and semantic faithfulness; the current literature shows that the trade-off can be highly productive, but never cost-free.