FairyFuse: Diverse Fusion Mechanisms

• FairyFuse is a term describing three distinct fusion methods used in machine learning, each addressing verifier ensembling, anchor-based video attention, and ternary CPU inference.
• It leverages unsupervised pseudo-label learning, cross-frame anchor caching, and multiplication-free SIMD kernels to meet constraints in supervision, temporal coherence, and memory bandwidth.
• The shared label reflects an engineering pattern where complex global tasks become tractable through structured decomposition and tailored algorithmic innovations.

Searching arXiv for the cited papers to ground the article in current records. FairyFuse denotes three distinct constructs in recent machine-learning literature: a practical instantiation of Fully Unsupervised Score Ensembling for verifying large-language-model outputs without labeled correctness data; an Editor’s term for the anchor-based cross-frame attention mechanism introduced in Fairy for instruction-guided video-to-video synthesis; and a multiplication-free CPU inference system for ternary LLMs based on fused widely-linear kernels (Lee et al., 20 Apr 2026, Wu et al., 2023, Zuo et al., 22 Apr 2026). The shared label therefore does not identify a single method, but a set of technically unrelated mechanisms whose commonality lies in structured fusion under constraints of supervision, temporal coherence, or memory bandwidth.

1. Nomenclature and scope

The name “FairyFuse” is used in three non-equivalent senses across the cited material. In the verifier-ensembling setting, it is a practical instantiation of FUSE, “Fully Unsupervised Score Ensembling,” for selecting correct responses from multiple LLM generations using only verifier outputs. In the video-editing setting, it is not the paper’s own term, but an Editor’s term for Fairy’s anchor-based cross-frame attention module. In the CPU-inference setting, it is the paper title and refers to a concrete execution system for ternary, widely-linear transformer layers (Lee et al., 20 Apr 2026, Wu et al., 2023, Zuo et al., 22 Apr 2026).

Usage	Domain	Core mechanism
FairyFuse as FUSE instantiation	LLM verification	Triplet-conditional independence, spectral MoM, pseudo-label ensemble learning
FairyFuse as Editor’s term in Fairy	Video-to-video diffusion editing	Anchor-based cross-frame attention with cached anchor key/value maps
FairyFuse as CPU inference system	Ternary LLM decoding	Fused AVX-512 masked add/sub kernels for widely-linear layers

A recurrent source of confusion is terminological rather than methodological. The video-editing usage does not correspond to an author-designated module name; the details explicitly state that the paper “does not name it ‘FairyFuse.’” By contrast, the verifier-ensembling and CPU-inference usages are direct names or instantiations appearing in the supplied material.

2. FairyFuse as fully unsupervised verifier ensembling

In the verification setting, FairyFuse operates on a single-query or multiple-query problem in which a generator produces $N$ i.i.d. candidate responses $r_1,\ldots,r_N$ for a query $q$, and $m$ verifiers $v_1,\ldots,v_m$ assign scores to each response. Writing $v_{ij} := v_j(q,r_i)$, the verifier outputs form a score matrix $V \in \mathbb{R}^{N \times m}$, and the task is to select an index $i^\star \in [N]$ such that $y(q,r_{i^\star}) = 1$, where $y(q,r_i)\in\{\pm 1\}$ denotes ground-truth correctness. The procedure can be run query-conditionally or in batched mode by stacking rows across queries (Lee et al., 20 Apr 2026).

The statistical formulation begins with binary verifiers or binarized verifier outputs. Sensitivity and specificity for verifier $r_1,\ldots,r_N$0 are defined as

$r_1,\ldots,r_N$1

Balanced accuracy is $r_1,\ldots,r_N$2, and the core identifiability condition is that more than half of the verifiers are “better than random,” i.e. have balanced accuracy greater than $r_1,\ldots,r_N$3. The dependence assumption is triplet conditional independence (TCI): any triplet of distinct verifiers is conditionally independent given $r_1,\ldots,r_N$4. The method does not assume joint conditional independence (JCI); that distinction is central to the construction.

The TCI assumption is stated as

$r_1,\ldots,r_N$5

The class-imbalance parameter is $r_1,\ldots,r_N$6.

In its method-of-moments step, FairyFuse works in a binarized space using verifier-specific thresholds $r_1,\ldots,r_N$7, with transformations $r_1,\ldots,r_N$8. The paper also gives an Appendix extension to real-valued verifiers in $r_1,\ldots,r_N$9, where

$q$0

and $q$1.

3. Spectral identification, pseudo-label construction, and empirical profile

FairyFuse estimates verifier quality from unlabeled outputs through first-, second-, and third-order empirical moments. If $q$2, $q$3, and $q$4 denote the mean vector, second-order covariance matrix, and third-order covariance tensor of verifier predictions, then under TCI and the majority-better-than-random assumption, the off-diagonals of $q$5 and $q$6 admit rank-one structure. The Jaffe–Nadler identification theorem states that the off-diagonals of $q$7 equal those of $q$8 and the off-diagonals of $q$9 equal those of $m$0, with

$m$1

In practice, $m$2, $m$3, and $m$4 are estimated from $m$5 at $m$6 rates, the leading eigenvector of the off-diagonal covariance gives $m$7 up to sign, a rank-one fit to the off-diagonal tensor gives $m$8, and the sign of $m$9 is fixed by the majority-better-than-random assumption (Lee et al., 20 Apr 2026).

A distinctive component is adaptive control of conditional dependence. The paper defines a TCI-violation statistic whose variance vanishes under TCI, and uses it to search over monotone per-verifier transformations. The default family is threshold binarization:

$v_1,\ldots,v_m$0

with $v_1,\ldots,v_m$1 chosen by approximately minimizing the empirical TCI-violation measure $v_1,\ldots,v_m$2 via coordinate descent. The implementation notes specify clipping small denominators for numerical stability, using a sigmoid approximation for faster gradient-based optimization, and rescaling real-valued scores to $v_1,\ldots,v_m$3 before transformation. Additional measures include dropping verifiers estimated to have balanced accuracy below $v_1,\ldots,v_m$4 and optionally merging verifiers into “views.”

Once $v_1,\ldots,v_m$5, $v_1,\ldots,v_m$6, and $v_1,\ldots,v_m$7 are obtained, FairyFuse forms pseudo-labels through triplet posteriors. For any triplet $v_1,\ldots,v_m$8,

$v_1,\ldots,v_m$9

Triplet posteriors are averaged over all $v_{ij} := v_j(q,r_i)$0 triplets to form $v_{ij} := v_j(q,r_i)$1, and the final ensemble is learned by maximizing

$v_{ij} := v_j(q,r_i)$2

In experiments, $v_{ij} := v_j(q,r_i)$3 is logistic regression on the raw, real-valued verifier outputs $v_{ij} := v_j(q,r_i)$4, not on the binarized $v_{ij} := v_j(q,r_i)$5. The final decision is $v_{ij} := v_j(q,r_i)$6.

The algorithmic flow consists of: selecting $v_{ij} := v_j(q,r_i)$7 by minimizing the TCI-violation objective; transforming $v_{ij} := v_j(q,r_i)$8 to $v_{ij} := v_j(q,r_i)$9; applying spectral MoM to estimate per-verifier qualities; constructing pseudo-labels by triplet posterior averaging; fitting the final ensemble; and selecting the top-scoring response. Reported complexity is $V \in \mathbb{R}^{N \times m}$0 to construct $V \in \mathbb{R}^{N \times m}$1, $V \in \mathbb{R}^{N \times m}$2 to construct $V \in \mathbb{R}^{N \times m}$3 and average over triplets, and $V \in \mathbb{R}^{N \times m}$4 per logistic-regression iteration. Subsampling triplets is suggested when $V \in \mathbb{R}^{N \times m}$5 is large.

Empirically, FairyFuse is reported as competitive with semi-supervised baselines despite using zero labels, and as robustly better than naive ensemble and majority vote across the Weaver settings. On Humanity’s Last Exam, Best-of-50 accuracy is reported as Pass@1 $V \in \mathbb{R}^{N \times m}$6, Naive Ensemble $V \in \mathbb{R}^{N \times m}$7, Oracle best verifier $V \in \mathbb{R}^{N \times m}$8, Logistic $V \in \mathbb{R}^{N \times m}$9, Weaver $i^\star \in [N]$0, and FUSE $i^\star \in [N]$1. On IMO Shortlist, Naive Ensemble and FUSE are both $i^\star \in [N]$2, while Weaver is $i^\star \in [N]$3, Logistic $i^\star \in [N]$4, and Oracle best verifier $i^\star \in [N]$5. In the mixed-data ablation, FUSE is reported to win across both label configurations, with $i^\star \in [N]$6 versus $i^\star \in [N]$7 for 70B mixed labels and $i^\star \in [N]$8 versus $i^\star \in [N]$9 for 8B GPQA-only labels. The paper attributes the IMO Shortlist tie to near-conditional independence and homogeneous verifier strength, where naive averaging is effectively oracle.

The main limitations are also explicit. Strong conditional dependencies that remain after transformation can break the off-diagonal rank-one structure and degrade MoM estimates; highly biased verifiers can distort pseudo-labels; and very small $y(q,r_{i^\star}) = 1$0 or $y(q,r_{i^\star}) = 1$1 increases estimation noise. The paper further notes that if verifiers are heavily dependent or adversarial, small labeled sets such as $y(q,r_{i^\star}) = 1$2 of queries can stabilize weight estimation in methods such as WEAVER or supervised logistic regression.

4. FairyFuse as anchor-based cross-frame attention in video editing

In the Fairy video-editing system, “FairyFuse” is an Editor’s term for the anchor-based cross-frame attention mechanism that replaces every self-attention block in the U-Net with cross-frame attention against a cache built from a small set of anchor frames. The base model is an instruction-based image-editing diffusion pipeline similar to InstructPix2Pix. The rest of the U-Net—convolutions, residual blocks, up/down-sampling, and text conditioning—remains unchanged (Wu et al., 2023).

The data flow is two-stage. First, $y(q,r_{i^\star}) = 1$3 anchor frames are selected uniformly from the input video and processed once through denoising. At every attention layer $y(q,r_{i^\star}) = 1$4 and diffusion step $y(q,r_{i^\star}) = 1$5, the method caches each anchor’s key/value maps $y(q,r_{i^\star}) = 1$6 and $y(q,r_{i^\star}) = 1$7, then concatenates them into

$y(q,r_{i^\star}) = 1$8

Second, for any target frame, the model computes its own $y(q,r_{i^\star}) = 1$9 and replaces self-attention by anchor-based cross-frame attention:

$y(q,r_i)\in\{\pm 1\}$0

This causes each frame to attend to itself and to all anchor tokens, so semantically corresponding anchor features are fused into the current representation.

The paper interprets cross-frame attention as implicit correspondence estimation and feature propagation. A correspondence from position $y(q,r_i)\in\{\pm 1\}$1 in one frame to position $y(q,r_i)\in\{\pm 1\}$2 in another can be read from the attention map by

$y(q,r_i)\in\{\pm 1\}$3

Empirically, early and late U-Net layers achieve more than $y(q,r_i)\in\{\pm 1\}$4 TAP-Vid position accuracy at $y(q,r_i)\in\{\pm 1\}$5 on DAVIS, which the paper presents as evidence for the tracking interpretation.

The mechanism is also the basis of Fairy’s parallelization strategy. Once the anchor cache has been built, each non-anchor frame depends only on cached anchor features and not on other non-anchor frames. The resulting decoupling allows all non-anchor frames to be generated independently and sharded across GPUs, with numerically identical results to serial execution. The per-layer complexity is $y(q,r_i)\in\{\pm 1\}$6, where $y(q,r_i)\in\{\pm 1\}$7 is the number of tokens in the current frame and $y(q,r_i)\in\{\pm 1\}$8 the number of tokens per anchor, and this cost is independent of the total number of frames $y(q,r_i)\in\{\pm 1\}$9.

Reported inference settings use $r_1,\ldots,r_N$00 anchors by default and 10 Euler ancestral steps. With 8 A100 GPUs at $r_1,\ldots,r_N$01 resolution, Fairy generates 120 frames in $r_1,\ldots,r_N$02–$r_1,\ldots,r_N$03 seconds; on a single A100, the latency is 78 seconds. On a 4-second, 30 FPS, $r_1,\ldots,r_N$04 video, CLIP-based metrics are reported as Tem-Con $r_1,\ldots,r_N$05 and Frame-Acc $r_1,\ldots,r_N$06 for Fairy, compared with $r_1,\ldots,r_N$07 s for TokenFlow and $r_1,\ldots,r_N$08 s for Rerender. In a 1000-sample A/B user study, Fairy wins overall quality $r_1,\ldots,r_N$09 versus $r_1,\ldots,r_N$10 against Rerender, $r_1,\ldots,r_N$11 versus $r_1,\ldots,r_N$12 against TokenFlow, and $r_1,\ldots,r_N$13 versus $r_1,\ldots,r_N$14 against Gen-1.

Ablations isolate the effect of the cross-frame module. Temporal consistency rises from Tem-Con $r_1,\ldots,r_N$15 without anchors and without equivariant fine-tuning, to $r_1,\ldots,r_N$16 with anchors, and to $r_1,\ldots,r_N$17 with anchors plus equivariant fine-tuning. The anchor count matters: $r_1,\ldots,r_N$18 is often insufficient, while quality degrades for $r_1,\ldots,r_N$19; the reported best setting is $r_1,\ldots,r_N$20. The model is also fine-tuned with affine augmentations so that $r_1,\ldots,r_N$21 without an explicit temporal loss. Limitations include carry-over artifacts from the base image editor, over-stabilization of dynamic effects such as lightning or rain, difficulty with instructions that require camera motion, and degradation under extreme non-affine motion or heavy occlusion.

5. FairyFuse as multiplication-free ternary LLM inference on CPUs

In the CPU-systems setting, FairyFuse is an inference runtime for ternary LLMs designed for the autoregressive decode regime, where batch size is approximately 1 and matrix–vector products dominate per-token latency. The paper’s motivation is that dense FP32 GEMV has arithmetic intensity of about $r_1,\ldots,r_N$22 FLOP/byte and is therefore deeply memory-bound on CPUs with roughly $r_1,\ldots,r_N$23 GB/s DRAM bandwidth. Conventional 4-bit CPU pipelines still dequantize weights and execute floating-point multiplications, leaving part of the compression benefit unrealized (Zuo et al., 22 Apr 2026).

FairyFuse exploits ternary weights $r_1,\ldots,r_N$24, for which multiplication by an activation $r_1,\ldots,r_N$25 reduces to add, subtract, or skip. At the matrix level, with binary selectors $r_1,\ldots,r_N$26 and $r_1,\ldots,r_N$27,

$r_1,\ldots,r_N$28

Weights are stored using two indicator bits per element: $r_1,\ldots,r_N$29, $r_1,\ldots,r_N$30, $r_1,\ldots,r_N$31, and $r_1,\ldots,r_N$32 unused. Sixteen ternary weights are packed into one 32-bit word, giving a $r_1,\ldots,r_N$33 reduction relative to FP32 weights.

The targeted model family is Fairy2i’s complex-valued ternary LLM, whose widely-linear layer is

$r_1,\ldots,r_N$34

Writing $r_1,\ldots,r_N$35, $r_1,\ldots,r_N$36, $r_1,\ldots,r_N$37, with per-row real scales $r_1,\ldots,r_N$38, the layer decomposes into eight real GEMVs:

$r_1,\ldots,r_N$39

$r_1,\ldots,r_N$40

Each block matrix is ternary, so the $r_1,\ldots,r_N$41 inner loop becomes multiplication-free; only the final $r_1,\ldots,r_N$42 application of per-row scales uses multiplications.

The kernel implementation fuses these eight real GEMV streams into a single AVX-512 loop. It uses masked FP32 additions and subtractions (vaddps/vsubps with masks), BMI2 pext to decode masks from packed 32-bit words, FP32 accumulators, and no weight dequantization in the inner loop. The paper identifies four key optimizations: mask reuse, input reuse, conjugate sign-swap via a precomputed $r_1,\ldots,r_N$43, and register-resident accumulation. Assembly inspection reportedly shows eight pext, eight masked vaddps, eight masked vsubps, and no vmulps or vfmadd in the inner loop for each 16-lane chunk.

The roofline analysis is central to the performance claim. The paper derives $r_1,\ldots,r_N$44 ops/byte for FP32 GEMV, $r_1,\ldots,r_N$45 ops/byte for a single ternary GEMV, and $r_1,\ldots,r_N$46 ops/byte for the fused widely-linear kernel. On an Intel Xeon 8558P, with $r_1,\ldots,r_N$47 GB/s and $r_1,\ldots,r_N$48 TFOP/s-equivalent add/sub, the ridge point is about $r_1,\ldots,r_N$49 ops/byte, so the fused ternary kernel approaches the compute regime more closely than dense FP32 GEMV.

Measured results include a DRAM-cold $r_1,\ldots,r_N$50 GEMV time of $r_1,\ldots,r_N$51 for FP32 at 1 thread and $r_1,\ldots,r_N$52 for FairyFuse at 48 threads, corresponding to a $r_1,\ldots,r_N$53 speedup; across sizes, the reported speedup range is $r_1,\ldots,r_N$54 to $r_1,\ldots,r_N$55 at 48 threads. End-to-end on LLaMA-2-7B, FairyFuse reaches 32.4 tokens/s on a single Intel Xeon 8558P and is reported as $r_1,\ldots,r_N$56 faster than llama.cpp Q4_K_M, which runs at 26.15 tok/s. Quality is characterized as near-lossless: WikiText-2 perplexity is 5.52 for FairyFuse versus 5.47 for FP16, and average downstream accuracy over ARC-E, ARC-C, HellaSwag, PIQA, and WinoGrande is $r_1,\ldots,r_N$57 versus $r_1,\ldots,r_N$58 for FP16. The fused GEMV accounts for about $r_1,\ldots,r_N$59 of per-layer time in decode.

The paper also states why direct GPU transfer is ineffective. On NVIDIA H200, FP16 GEMV with cuBLAS already saturates HBM bandwidth, the operational intensity remains below the GPU ridge, and the architecture lacks a pext-like primitive and suitable masked add/sub execution path. A direct CUDA port of the masked-add ternary kernel is reported to regress by about $r_1,\ldots,r_N$60 relative to cuBLAS FP16 for $r_1,\ldots,r_N$61 GEMV. Hardware requirements are therefore explicit: AVX-512F and BMI2 are needed for the reported performance, and prompt-processing GEMM is left as future work.

6. Comparative interpretation, misconceptions, and constraints

Several misconceptions are resolved by reading the three usages together. First, FairyFuse is not a unified framework spanning verification, video editing, and CPU inference; it is a shared label attached to unrelated mechanisms. Second, in the verifier-ensembling formulation, the method does not assume JCI for the final ensemble; it assumes TCI and then uses pseudo-label optimization to avoid the full JCI requirement (Lee et al., 20 Apr 2026). Third, in the video-editing formulation, “FairyFuse” is not an author-specified module name but an editorial shorthand for anchor-based cross-frame attention (Wu et al., 2023). Fourth, in the CPU-inference formulation, “zero floating-point multiplications” applies to the $r_1,\ldots,r_N$62 inner loop; the $r_1,\ldots,r_N$63 per-row scale applications still use scalar multiplications (Zuo et al., 22 Apr 2026).

The three usages also impose different bottlenecks and failure modes. In verifier ensembling, the critical issue is residual conditional dependence among verifiers after transformation, together with weak or adversarial verifiers and small-$r_1,\ldots,r_N$64 or small-$r_1,\ldots,r_N$65 estimation noise. In Fairy video editing, the bottlenecks are the quality limits of the base image editor, over-stabilization of dynamic phenomena, and difficulty with camera-motion instructions. In CPU ternary inference, the constraints are ISA support, the focus on decode rather than GEMM-heavy prompt processing, and dependence on Fairy2i’s widely-linear ternary format.

A plausible implication is that the repeated reuse of the name reflects a common engineering pattern rather than a common algorithm: each FairyFuse isolates a structured interaction subproblem and constrains it so that a difficult global operation becomes tractable. In the verifier case, this is achieved through triplet-conditional structure and pseudo-label learning; in the video case, through attention restricted to anchors; and in the CPU case, through bitpacked ternary masks fused into a single SIMD loop. That implication, however, concerns naming and design style rather than formal methodological unity.

For reproducibility, the verifier-ensembling work reports released raw Humanity’s Last Exam and IMO Shortlist data at https://huggingface.co/FUSE-verifiers, Weaver replication code at https://github.com/HazyResearch/scaling-verification, and an experimental setup using vLLM 0.13.0, an 8× H100 (80GB) node for local generations and verifications, and API calls for OpenAI, Google, and DeepSeek services (Lee et al., 20 Apr 2026). The video work specifies preprocessing to longer side 512, 10 Euler ancestral steps, and 8 A100 GPUs for the headline timing (Wu et al., 2023). The CPU work specifies a single-socket Intel Xeon 8558P, g++ -O3 with -mavx512f -mbmi2, OpenMP, and numactl binding for the reported results (Zuo et al., 22 Apr 2026).