HyperLogLog

HyperLogLog is an algorithm for the count-distinct problem, approximating the number of distinct elements in a multiset. Calculating the exact cardinality of a multiset requires an amount of memory proportional to the cardinality, which is impractical for very large data sets. Probabilistic cardinality estimators, such as the HyperLogLog algorithm, use significantly less memory than this, at the cost of obtaining only an approximation of the cardinality. The HyperLogLog algorithm is able to estimate cardinalities of > 10⁹ with a typical accuracy of 2%, using 1.5 kB of memory. HyperLogLog is an extension of the earlier LogLog algorithm, itself deriving from the 1984 Flajolet–Martin algorithm.

Algorithm

The basis of the HyperLogLog algorithm is the observation that the cardinality of a multiset of uniformly distributed random numbers can be estimated by calculating the maximum number of leading zeros in the binary representation of each number in the set. If the maximum number of leading zeros observed is n, an estimate for the number of distinct elements in the set is 2ⁿ.

In the HyperLogLog algorithm, a hash function is applied to each element in the original multiset to obtain a multiset of uniformly distributed random numbers with the same cardinality as the original multiset. The cardinality of this randomly distributed set can then be estimated using the algorithm above.

The simple estimate of cardinality obtained using the algorithm above has the disadvantage of a large variance. In the HyperLogLog algorithm, the variance is minimised by splitting the multiset into numerous subsets, calculating the maximum number of leading zeros in the numbers in each of these subsets, and using a harmonic mean to combine these estimates for each subset into an estimate of the cardinality of the whole set.

Operations

The HyperLogLog has three main operations: add to add a new element to the set, count to obtain the cardinality of the set and merge to obtain the union of two sets. Some derived operations can be computed using the Inclusion–exclusion principle like the cardinality of the intersection or the cardinality of the difference between two HyperLogLogs combining the merge and count operations.

The data of the HyperLogLog is stored in an array M of counters called registers with size m that are set to 0 in their initial state.

Add

The add operation consists in computing the hash with a hash function h of the input data v getting the first b bits (where b is ${\textstyle \log _{2}(m)}$) and adding 1 to them to obtain the address of the register to modify. With the remaining bits compute ${\textstyle \rho (w)}$ which returns the position of the leftmost 1. The new value of the register will be the maximum between the current value of the register and ${\textstyle \rho (w)}$.

${\begin{aligned}x&:=h(v)\\j&:=1+\langle x_{1},x_{2},..,x_{b}\rangle _{2}\\w&:=x_{b+1}x_{b+2}...\\M[j]&:=\max(M[j],\rho (w))\\\end{aligned}}$

Count

The count algorithm consists in computing the harmonic mean of the m registers.

$Z={\Bigg (}\sum _{j=1}^{m}{2^{-M[j]}}{\Bigg )}^{-1}$

${\text{count}}=\alpha _{m}m^{2}Z$

The intuition is that being n the unknown cardinality of M. Each subset ${\textstyle M_{j}}$ will have ${\textstyle n/m}$ elements. Then ${\textstyle \max _{x\in M_{j}}\rho (x)}$ should be close to ${\textstyle \log _{2}(n/m)}$ the harmonic mean of 2 to these quantities is ${\textstyle mZ}$ which should be near ${\textstyle n/m}$. Thus, ${\textstyle m^{2}Z}$ should be n approximately.

Finally, the constant ${\textstyle \alpha _{m}}$ is introduced to correct a systematic multiplicative bias present in ${\textstyle m^{2}Z}$ due to hash collisions.

This technique, though, is biased for small cardinalities below a threshold of ${\textstyle {\frac {5}{2}}m}$. The original paper proposes using a different algorithm for small cardinalities: Linear Counting

Merge

The merge operation for two HLLs (${\textstyle hll_{1},hll_{2}}$) consists in obtain the maximum for each pair of registers ${\textstyle j:1..m}$

$hll_{union}[j]=\max(hll_{1}[j],hll_{2}[j])$

Complexity

To analyze the complexity, the data streaming $(\epsilon ,\delta )$ model is used, which analyses the space necessary to get a $1\pm \epsilon$ approximation with a fixed success probability $1-\delta$. The relative error of HLL is $1.04/{\sqrt {m}}$ and it needs $O(\epsilon ^{-2}\log \log n+\log n)$ space, where n is the set cardinality and m is the number of registers (usually less than one byte size).

The add operation depends on the size of the output of the hash function. As this size is fixed, we can consider the running time for the add operation to be $O(1)$.

The count and merge operations depend on the number of registers m and have a theoretical cost of $O(m)$. In some implementations (Redis) the number of registers is fixed and the cost is considered to be $O(1)$ in the documentation.

HLL++

The HLL++ is a practical improvement of the HyperLogLog with some modifications to make it useful for real environments . Among other improvements it proposes:

• Use 64 bit hash functions instead of the 32 bits used in the original paper. This reduces the hash collisions for typical ranges allowing to remove the $\alpha _{m}$ correction factor.
• Some bias is found for small cardinalities when switching from linear counting to the HLL counting. They propose to do an empirical bias correction.
• Normally, it is not known the order of magnitude of the cardinality of the set one wants to estimate. Sometimes, it might be that this cardinalities are very small making the $O(m)$ size of the HLL similar to $O(n)$ or larger. In that case many registers are not used and memory is wasted. For this reason, in HLL++ they propose a sparse representation of the registers (using a dictionary to store $(id_{j},\rho (w))$ for the register j) which can be later transformed to a dense representation if the cardinality grows.

References

• "Probabilistic Data Structures for Web Analytics and Data Mining | Highly Scalable Blog". highlyscalable.wordpress.com. Retrieved 2014-04-19.
• "New cardinality estimation algorithms for HyperLogLog sketches" (PDF). Retrieved 2016-10-29.