--- title: 'VHDL-Xform: Assessing Functional Equivalence' url: https://www.emergentmind.com/topics/vhdl-xform type: topic --- # VHDL-Xform: Assessing Functional Equivalence VHDL-Xform is a benchmark dataset and methodology developed to rigorously evaluate language models' ability to recognize and reason about functional equivalence in VHDL (VHSIC Hardware Description Language) code. It targets the specific challenge of distinguishing functionally identical yet lexically dissimilar VHDL snippets, addressing limitations of existing models on tasks related to code generation and summarization for hardware description languages [2507.12308]. ## 1. Formal Definition and Equivalence Semantics VHDL-Xform is formally defined over the set $\mathcal{C}$ of all valid VHDL code snippets. Three classes of transformation functions, $\mathcal{T} = \{T_2, T_3, T_4\}$, are considered: - $T_2$: Type-2 transformations (identifier renaming) - $T_3$: Type-3 transformations (statement-level inert variations) - $T_4$: Type-4 transformations (semantic rewriting via cross-language back-translation) For any $c \in \mathcal{C}$ and $t \in \mathcal{T}$, a transformed $c' = t(c)$ maintains semantic equivalence: $\mathrm{sem}(c) = \mathrm{sem}(c')$, where $\mathrm{sem}(\cdot)$ represents the mapping from input vectors to output vectors over all valid VHDL test sequences. The dataset is defined as: \[ D_{\text{xform}} = \left\{ (c, c') \mid c \in \mathcal{C},\; c' = T_k(c)\;\text{for some}\; T_k \in \mathcal{T},\; \mathrm{sem}(c)=\mathrm{sem}(c') \right\} \] This construction enforces that all (original, transformed) pairs are functionally indistinguishable at the behavior level, a key feature for evaluating the robustness of code understanding in models [2507.12308]. ## 2. Construction Methodology and Transformation Categories VHDL-Xform is built from permissively licensed public VHDL code repositories. Three transformation pipelines are applied: - **Type-2 (Identifier Renaming):** Each code entity is parsed to extract names, then identifiers are systematically renamed using patterns (e.g., variable-to-single-character, snake to camel case), typically with LLM-assisted suggestions. - **Type-3 (Statement-Level Variations):** Functionality-preserving changes are made by inserting inert statements (e.g., comments, unused signals) and reordering declarations that do not affect design semantics or compilation. - **Type-4 (Semantic Rewriting via Back-Translation):** The snippet is compiled into Verilog (using GHDL and Icarus Verilog), then re-synthesized back to VHDL. This process introduces deeper structural and naming variations, often creating new temporary state variables and minimizing direct lexical overlap compared to the original. Functional equivalence for all pairs is verified either through sequential equivalence checking or via internal testbenches, ensuring no behavioral divergence is introduced [2507.12308]. ## 3. Representative Transformation Examples Three transformation classes are realized concretely within the VHDL-Xform corpus: - **Type-2 Example:** - *Original:* ```vhdl entity adder is port( A : in std_logic_vector(3 downto 0); B : in std_logic_vector(3 downto 0); SUM : out std_logic_vector(3 downto 0)); end adder; ``` - *Transformed:* ```vhdl entity four_bit_sum is port( X : in std_logic_vector(3 downto 0); Y : in std_logic_vector(3 downto 0); Z : out std_logic_vector(3 downto 0)); end four_bit_sum; ``` - **Type-3 Example:** - *Original:* ```vhdl signal tmp : std_logic; tmp <= A(0) and B(0); ``` - *Transformed:* ```vhdl -- added for clarity variable unused_flag : boolean; unused_flag := false; signal tmp : std_logic; tmp <= B(0) and A(0); ``` - **Type-4 Example:** - *Original:* ```vhdl process(clk) begin if rising_edge(clk) then Q <= D; end if; end process; ``` - *Transformed:* (after VHDL→Verilog→VHDL) ```vhdl always @(posedge clk) reg_q <= reg_d; ... process(clk) begin if clk'event and clk='1' then Q <= reg_q; -- uses temporary reg_q/reg_d end if; end process; ``` These exemplify systematic code drift while preserving semantics [2507.12308]. ## 4. Evaluation Protocols and Metrics Two principal metrics are used to assess VHDL code summarization consistency over VHDL-Xform: - **ROUGE-L ($R_L$):** Measures the longest common subsequence (LCS) between generated ($G$) and reference ($R$) summaries, using the $F_1$ score: \[ R_L = \frac{(1+\beta^2)\,P_{lcs}\,R_{lcs}}{R_{lcs} + \beta^2\,P_{lcs}},\text{ with } \beta=1 \] Where $P_{lcs} = LCS(G,R)/|G|$ and $R_{lcs} = LCS(G,R)/|R|$. - **LLM Preference Rate (PR):** For each (summary, reference) pair, a judge LLM (Llama-3-70B-Instruct) is queried: "Which summary better captures the functionality?" PR is the fraction of times the generated summary is preferred. Performance for various models under zero-shot conditions and with Chain-of-Descriptions (CoDes) improvements is summarized below: | Model | ROUGE-L (Zero-Shot) | PR (%) (Zero-Shot) | |-------------------|--------------------:|-------------------:| | CodeLlama-34B | 36.63 | 36.6 | | Granite-Code-34B | 38.60 | 35.7 | | Deepseek-33B | 35.10 | 28.6 | | Granite-Code-20B | 31.70 | 28.1 | | Mixtral-8x7B | 19.60 | 14.8 | | Mistral-7B | 18.20 | 6.6 | | Granite-Chat-13B | 28.10 | 19.5 | | Llama-2-Chat-13B | 26.60 | 18.5 | The ROUGE-L range is typically 20–40%, showing significant challenges for current models [2507.12308]. ## 5. Chain-of-Descriptions (CoDes) and Performance Gains CoDes is a multi-step prompting paradigm designed to inject a series of intermediate, descriptive steps into LLM workflows. For code summarization on VHDL-Xform, this involves decomposing the task into a series of explanations or plans—which are then concatenated with the input and supplied to the model. Empirical results demonstrate notable, though model-dependent, improvements: | Model | ROUGE-L (ZS→CoDes) | Δ$R_L$ | PR (ZS→CoDes) | ΔPR | |-------------------|-------------------:|-------:|--------------:|----:| | CodeLlama-34B | 36.63→39.20 | +2.57 | Source: https://www.emergentmind.com/topics/vhdl-xform