ECMH: Incremental Multiset Hashing

• Incremental multiset hashing is a technique that assigns fixed-size digests to multisets, allowing algebraic updates through group operations.
• ECMH employs binary elliptic curve groups and a weak-encoding map to achieve high collision resistance with compact digest sizes.
• The method supports efficient insertions, deletions, and parallel aggregation, making it ideal for streaming data and database integrity verification.

Incremental multiset hashing is the problem of assigning a fixed-length digest to an unordered collection with multiplicities so that updates compose algebraically: if two multisets are combined, their digests combine by a corresponding group operation. The construction introduced in "Elliptic Curve Multiset Hash" (Maitin-Shepard et al., 2016) instantiates this primitive with binary elliptic-curve groups, yielding a homomorphic multiset hash whose digest size is essentially optimal for its collision resistance target while remaining practical in software. The resulting Elliptic Curve Multiset Hash (ECMH) supports insertions, deletions, parallel aggregation, and streaming updates by expressing the digest of a multiset as a sum of per-element curve points.

1. Algebraic model and problem setting

A multiset over a base set $A$ is modeled as a finitely supported function

$M \in \mathbb{Z}^{(A)},$

where $M(a)$ is the multiplicity of element $a$. This formulation explicitly allows negative multiplicities, so removals can be represented within the same algebraic domain. A homomorphic multiset hash is then a group homomorphism

$H : \mathbb{Z}^{(A)} \to G$

from multiset addition to the group law on a finite group $(G,+_G)$, satisfying

$H(M_1+M_2)=H(M_1)+_G H(M_2).$

If the singleton hash is defined by

$\hat H(a)=H(\{a\}),$

then the entire construction is determined by the accumulation rule

$H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$

Repeated elements contribute through scalar multiplication in the group, and negative multiplicities correspond to inverses (Maitin-Shepard et al., 2016).

This homomorphic law is the source of the incremental property. A digest can be updated by adding the hashes of inserted elements and subtracting the hashes of removed elements, rather than recomputing from scratch. The same structure enables parallelization: independent workers hash disjoint portions of a collection and combine the resulting partial digests by the group operation. The paper highlights database integrity verification, streaming set or multiset comparison, network coding or content distribution, and parallel or incremental hashing of structured data represented as sets of index-value pairs as representative applications.

2. Security notions and the weak-encoding reduction

For a group-homomorphic hash $H:(X,+)\to(G,+)$, a collision is equivalent to finding a nonzero kernel element,

$M \in \mathbb{Z}^{(A)},$0

The ideal generic target is collision resistance about $M \in \mathbb{Z}^{(A)},$1 by the birthday bound and preimage resistance about $M \in \mathbb{Z}^{(A)},$2. The paper also emphasizes that second-preimage security is inherently weaker for homomorphic hashes than for ordinary hashes, because once a collision is known, the group structure yields arbitrary second preimages.

The generic Bellare–Micciancio randomize-then-combine paradigm takes a group $M \in \mathbb{Z}^{(A)},$3, a random oracle $M \in \mathbb{Z}^{(A)},$4, and defines

$M \in \mathbb{Z}^{(A)},$5

In practice, however, the main difficulty is instantiating the map into the group. The paper relaxes the requirement that the map behave like a fully uniform or indifferentiable hash-to-group construction and instead introduces the notion of an $M \in \mathbb{Z}^{(A)},$6-weak encoding. A map $M \in \mathbb{Z}^{(A)},$7 is such a weak encoding if preimages can be efficiently computed and uniformly sampled, every fiber satisfies

$M \in \mathbb{Z}^{(A)},$8

and the average normalized fiber size satisfies

$M \in \mathbb{Z}^{(A)},$9

The central reduction states that if $M(a)$0 is an $M(a)$1-weak encoding into an abelian group $M(a)$2 containing a prime-order cyclic direct factor $M(a)$3 of order $M(a)$4, and if $M(a)$5 is a random oracle, then collisions in the resulting multiset hash imply an algorithm for computing discrete logarithms in $M(a)$6. More precisely, if an algorithm finds

$M(a)$7

with probability $M(a)$8, then one can compute discrete logarithms with probability

$M(a)$9

in expected time

$a$0

where $a$1 is the collision finder’s time, $a$2 is the number of oracle queries, and $a$3 bounds the output size (Maitin-Shepard et al., 2016). The security basis of ECMH is therefore the random-oracle model for the intermediate hash, the weak-encoding properties of the map-to-curve procedure, and the hardness of the discrete logarithm problem in the chosen elliptic-curve group.

3. Earlier constructions and their limitations

The paper contrasts ECMH with two main earlier families. MuHash, also discussed as MSet-Mu-Hash, uses the multiplicative group $a$4. Its security reduces to the discrete logarithm problem in $a$5, but that problem is subject to subexponential Number Field Sieve attacks of heuristic complexity

$a$6

As a consequence, achieving about 128-bit security requires a prime $a$7 of around $a$8 bits, so the digest is several thousand bits long. The paper identifies two practical drawbacks: very large digests and slow modular arithmetic, especially at higher security levels.

The second family includes AdHash, VAdd, and LtHash-style additive constructions over groups such as $a$9, particularly $H : \mathbb{Z}^{(A)} \to G$0 for AdHash. These constructions are simple and fast, but their multiset security is substantially weaker. For set hashing, assuming Wagner’s generalized birthday attack is optimal, a group $H : \mathbb{Z}^{(A)} \to G$1 provides only about

$H : \mathbb{Z}^{(A)} \to G$2

bits of security. For multisets, the paper shows that AdHash collisions reduce to finding short vectors in a lattice through relations of the form

$H : \mathbb{Z}^{(A)} \to G$3

The resulting requirement for $H : \mathbb{Z}^{(A)} \to G$4 bits of multiset security is

$H : \mathbb{Z}^{(A)} \to G$5

where $H : \mathbb{Z}^{(A)} \to G$6 is the Hermite factor constant of the lattice reduction algorithm. Using realistic $H : \mathbb{Z}^{(A)} \to G$7 values, the paper concludes that $H : \mathbb{Z}^{(A)} \to G$8 bits would require at least $H : \mathbb{Z}^{(A)} \to G$9-bit modulus and that even $(G,+_G)$0 bits would require more than $(G,+_G)$1 bits (Maitin-Shepard et al., 2016).

These comparisons motivate the central design objective of ECMH: to retain true multiset homomorphism while avoiding the oversized digests of MuHash and the ineffective multiset security of practical AdHash parameters.

4. ECMH construction over binary elliptic curves

ECMH chooses as digest group the group of rational points on a binary elliptic curve

$(G,+_G)$2

with curve equation

$(G,+_G)$3

where $(G,+_G)$4. The group law is elliptic-curve point addition in additive notation. The use of characteristic $(G,+_G)$5 is implementation-driven: the map-to-curve step can be realized through efficient linear-algebraic operations such as trace, half-trace or quadratic solving, squaring, and table-based transforms, avoiding expensive full exponentiation.

The per-element map follows a randomize-then-combine pattern. A standard bitstring hash

$(G,+_G)$6

first condenses an arbitrary input to a field element, and an optimized characteristic-$(G,+_G)$7 variant of Shallue–van de Woestijne then maps that field element to a point on the curve. The paper defines

$(G,+_G)$8

and the multiset digest is

$(G,+_G)$9

Removal is immediate: $H(M_1+M_2)=H(M_1)+_G H(M_2).$0 The implementation uses BLAKE2 as the intermediate hash: BLAKE2s for $H(M_1+M_2)=H(M_1)+_G H(M_2).$1, truncated to $H(M_1+M_2)=H(M_1)+_G H(M_2).$2 bits; BLAKE2b for $H(M_1+M_2)=H(M_1)+_G H(M_2).$3, truncated; and repeated BLAKE2b calls for larger $H(M_1+M_2)=H(M_1)+_G H(M_2).$4.

The SWChar2 encoding takes $H(M_1+M_2)=H(M_1)+_G H(M_2).$5, sets

$H(M_1+M_2)=H(M_1)+_G H(M_2).$6

and if $H(M_1+M_2)=H(M_1)+_G H(M_2).$7 returns the unique point with $H(M_1+M_2)=H(M_1)+_G H(M_2).$8,

$H(M_1+M_2)=H(M_1)+_G H(M_2).$9

Otherwise it computes $\hat H(a)=H(\{a\}),$0, forms three candidate $\hat H(a)=H(\{a\}),$1-coordinates

$\hat H(a)=H(\{a\}),$2

and for each candidate evaluates

$\hat H(a)=H(\{a\}),$3

At least one candidate satisfies

$\hat H(a)=H(\{a\}),$4

where

$\hat H(a)=H(\{a\}),$5

For the first such $\hat H(a)=H(\{a\}),$6, the encoding returns

$\hat H(a)=H(\{a\}),$7

in $\hat H(a)=H(\{a\}),$8-affine coordinates $\hat H(a)=H(\{a\}),$9, where $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$0 when $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$1, and

$H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$2

The added $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$3 term is described as a small paper-specific tweak to halve collisions at negligible extra cost (Maitin-Shepard et al., 2016).

5. Correctness, digest size, and collision security

Because ECMH is defined by summing per-element group elements, correctness is immediate from linearity: $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$4 For any two multisets $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$5 and $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$6,

$H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$7

This is the exact incremental law required for streaming updates, parallel composition, and deletions. The hash is homomorphic in the elliptic-curve group itself; serialization therefore requires a canonical compressed point encoding, but the algebraic law is independent of the bitstring representation.

The paper’s digest-size claim follows from the generic collision bound for elliptic-curve groups. If the group size is about $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$8, generic collision resistance is about $H(M)=\sum_{a\in A} M(a)\cdot \hat H(a).$9. A compressed point on a binary curve over $H:(X,+)\to(G,+)$0 requires about

$H:(X,+)\to(G,+)$1

since one stores $H:(X,+)\to(G,+)$2 together with one extra bit to distinguish the two possible $H:(X,+)\to(G,+)$3-coordinates. Accordingly, the paper summarizes the size-security tradeoff as

$H:(X,+)\to(G,+)$4

At the 128-bit security level, ECMH uses a curve such as GLS254, so the digest size is around $H:(X,+)\to(G,+)$5 bits, in contrast with roughly $H:(X,+)\to(G,+)$6 bits for MuHash at the same security level.

The security argument depends on the weak-encoding analysis of SWChar2. The map is not perfectly uniform, but every point has at most $H:(X,+)\to(G,+)$7 preimages, so $H:(X,+)\to(G,+)$8, and the preimage-size distribution is approximately $H:(X,+)\to(G,+)$9 of points with $M \in \mathbb{Z}^{(A)},$00 preimages, $M \in \mathbb{Z}^{(A)},$01 with $M \in \mathbb{Z}^{(A)},$02, $M \in \mathbb{Z}^{(A)},$03 with $M \in \mathbb{Z}^{(A)},$04, and $M \in \mathbb{Z}^{(A)},$05 with $M \in \mathbb{Z}^{(A)},$06. Hence

$M \in \mathbb{Z}^{(A)},$07

so SWChar2 is an $M \in \mathbb{Z}^{(A)},$08-weak encoding with

$M \in \mathbb{Z}^{(A)},$09

For a suitable elliptic-curve group $M \in \mathbb{Z}^{(A)},$10 with a large prime-order subgroup, the paper states that collision resistance is roughly

$M \in \mathbb{Z}^{(A)},$11

which for binary curves is approximately $M \in \mathbb{Z}^{(A)},$12 (Maitin-Shepard et al., 2016).

Several nuances qualify this reduction. The proof works even though the encoding maps into the full elliptic-curve group rather than only into a prime-order subgroup, which avoids explicit subgroup-membership filtering. The theorem assumes efficient sampling from the small cofactor subgroup complement, and the paper notes that this is easy because the cofactor is tiny. It also discusses heuristic attacks claimed for binary elliptic curves and takes the position that for curves up to about $M \in \mathbb{Z}^{(A)},$13 bits, including the 128-bit-security curve used in the experiments, generic attacks remain dominant in practice; moreover, it observes that ECMH is still preferable to MuHash, whose underlying discrete logarithm problem already admits a stronger subexponential attack.

6. Implementation strategy and empirical performance

A substantial part of ECMH’s practicality derives from implementation choices tailored to binary fields. The software uses carry-less multiplication support, specifically PCLMULQDQ, to accelerate arithmetic in $M \in \mathbb{Z}^{(A)},$14 under polynomial-basis representations. Field inversion is performed with Itoh–Tsujii inversion together with precomputed multi-squaring tables, automatically generated optimal addition chains, and machine-specific cost modeling. This emphasis is crucial because the dominant cost inside SWChar2 is the single field inversion.

Other linear operations—squaring, multi-squaring $M \in \mathbb{Z}^{(A)},$15, square root, and half-trace or quadratic solving—are linear over $M \in \mathbb{Z}^{(A)},$16, so the implementation accelerates them with lookup tables that split coefficients into blocks. SWChar2 outputs points in $M \in \mathbb{Z}^{(A)},$17-affine coordinates $M \in \mathbb{Z}^{(A)},$18, while accumulation uses $M \in \mathbb{Z}^{(A)},$19-projective coordinates

$M \in \mathbb{Z}^{(A)},$20

so point addition and doubling avoid inversions. For even $M \in \mathbb{Z}^{(A)},$21, the paper represents $M \in \mathbb{Z}^{(A)},$22 as a quadratic extension of $M \in \mathbb{Z}^{(A)},$23, a choice described as especially effective for GLS254.

The principal optimization is batch inversion via Montgomery’s trick: $M \in \mathbb{Z}^{(A)},$24 field elements can be inverted with one inversion and $M \in \mathbb{Z}^{(A)},$25 multiplications. Since inversion is much more expensive than multiplication, batching substantially reduces the per-element cost of SWChar2. The fastest implementation uses tables and branches and is therefore vulnerable to timing or cache leakage, but the paper also implements a fully blinded version with branch-free SWChar2, blinded inversion, and blinded quadratic solving. For GLS254 on Haswell, the paper reports $M \in \mathbb{Z}^{(A)},$26 cycles per element for batch ECMH and $M \in \mathbb{Z}^{(A)},$27 cycles per element for blinded batch ECMH.

The benchmark platforms are an Intel Westmere i7-970 @ 3.2 GHz and an Intel Haswell i7-4790K @ 4.0 GHz. The standout result is on Haswell at about 127-bit security using GLS254: single ECMH update requires $M \in \mathbb{Z}^{(A)},$28 cycles per element, batch ECMH $M \in \mathbb{Z}^{(A)},$29 cycles per element, and blinded batch ECMH $M \in \mathbb{Z}^{(A)},$30 cycles per element. At 4.0 GHz, $M \in \mathbb{Z}^{(A)},$31 cycles per element corresponds to approximately

$M \in \mathbb{Z}^{(A)},$32

elements per second, matching the abstract’s statement that the implementation processes over $M \in \mathbb{Z}^{(A)},$33 million set elements per second. For comparison on Haswell at the same security level, ECMH with GLS254 in batch mode requires $M \in \mathbb{Z}^{(A)},$34 cycles, MuHash $M \in \mathbb{Z}^{(A)},$35 cycles, and AdHash $M \in \mathbb{Z}^{(A)},$36 cycles, making ECMH about $M \in \mathbb{Z}^{(A)},$37 faster than MuHash and about $M \in \mathbb{Z}^{(A)},$38 faster than AdHash in those experiments (Maitin-Shepard et al., 2016).

7. Applicability, advantages, and limitations

ECMH is designed for settings in which composable digests are required rather than merely fixed-message hashing. Its homomorphic law,

$M \in \mathbb{Z}^{(A)},$39

supports incremental updates and deletions, parallel aggregation, streaming summaries, and digest comparison for multisets. The paper presents this as a direct fit for database integrity verification, streaming set or multiset comparison, network coding or content distribution, and incremental hashing of structured data.

The construction’s main advantages are a true multiset homomorphism, natural support for insertions and deletions, parallelizability through point addition, near-optimal digest size, and a security reduction based on the elliptic-curve discrete logarithm problem under only a weak-encoding assumption. The paper also notes a practical advantage in not requiring subgroup-only encoding, which avoids some complexity and patent concerns raised around earlier work.

Its limitations are equally explicit. ECMH depends on the random-oracle model for the intermediate hash $M \in \mathbb{Z}^{(A)},$40, and its security rests on the hardness of the elliptic-curve discrete logarithm problem in binary elliptic curves. The implementation is significantly more complex than MuHash’s modular multiplication or AdHash’s integer addition, and its best performance relies on carry-less multiplication support. Side-channel protection also requires care, since the fastest version uses lookup tables and branches and therefore needs blinding or constant-time countermeasures. Finally, the paper does not present ECMH as a replacement for ordinary hashing when homomorphism is unnecessary: ordinary cryptographic hashes remain simpler and often faster for one-shot hashing of an ordered sequence.

Taken together, these properties position ECMH as a specialized primitive for unordered collections with multiplicities. Within that domain, the paper’s contribution is to show that binary elliptic-curve groups, combined with an efficient weak map-to-curve encoding and optimized finite-field arithmetic, make incremental multiset hashing both compact and practically efficient (Maitin-Shepard et al., 2016).