This article relies too much on references to primary sources. (September 2017) (Learn how and when to remove this template message)
|Original author(s)||Yann Collet|
|Developer(s)||Yann Collet, Przemysław Skibiński (inikep)|
|Initial release||23 January 2015|
1.3.8 / 27 December 2018
|License||Dual: BSD License + GPLv2|
Zstandard (or Zstd) is a lossless data compression algorithm developed by Yann Collet at Facebook. The name also refers to the reference implementation in C. Version 1 of the implementation was released as free software on 31 August 2016.
Zstandard has been designed to give compression comparable to that of the DEFLATE algorithm (developed in 1991, used in original ZIP, gzip and others) with higher compression and especially decompression speeds. Zstandard combines the use of a dictionary-type algorithm (LZ77) with a large search window and fast implementations of entropy coding stage, using either very fast Finite State Entropy (tANS) or Huffman coding. A feature of the Zstandard implementations is backward direction of reading on entropy decompression.
Zstandard also implements several compression levels for additional flexibility, ranging from 1 (fastest) to 22 (slowest in compression speed, but best compression ratio). There are also parallel (multi-threaded) implementations of both compression and decompression. Starting from 1.3.2 version (October 2017), Zstandard optionally implements very long range search and deduplication similar to rzip or lrzip.
According to the LTCB compression benchmark compressing a single 1 GB text file, zstd at maximum levels shows compression ratio close to boz, yxz, tornado archivers, and performs better than lza, brotli, or bzip2. With any compression level it has fast decompression. Zstandard reaches the Pareto frontier, which means it decompresses faster than any other currently-available algorithm with similar or better compression ratio.
Zstandard can use any user-defined pre-populated compression dictionary. It also offers a training mode, able to generate a dictionary from any set of samples. Dictionaries can have a large impact on the compression ratio of small files.. In particular, one dictionary can be loaded to process large amounts of data with redundancy between, but not necessarily within, a set of files, e.g., log files.
Zstandard method is supported in the Linux kernel since version 4.14 (released November 2017) for usage as compression method for file systems like btrfs and squashfs. It was also tested for FreeBSD with integration into OpenZFS file system.
The algorithm is also deployed in datacenters, such as AWS Redshift, and in databases, such as RocksDB.
Canonical has plans to change default deb package compression to zstd in version 18.10 of the Ubuntu Linux distribution to speed up installation by around 10 percent. Zstd compression at level 19 has larger packages than the previously used xz compression, but faster decompression speed.
- LZFSE – a similar algorithm by Apple used since iOS 9 and OS X 10.11 made open source on 1 June 2016
- LZ4 (compression algorithm) – a fast member of the LZ77 family
- "Releases - facebook/zstd". Retrieved 27 December 2018 – via GitHub.
- Sergio De Simone, Facebook Open-Sources New Compression Algorithm Outperforming Zlib / InfoQ, 2 September 2016
- "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 2016-08-31. Retrieved 2016-09-06.
- "facebook/zstd". GitHub.
- Matt Mahoney (2016-08-29). "Large Text Compression Benchmark, .2157 zstd". Retrieved 2016-09-01.
- TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
- Matt Mahoney, Silesia Open Source Compression Benchmark
- https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf "Facebook developers report massive speedups and compression ratio improvements when using dictionaries"
- "Smaller and faster data compression with Zstandard". Facebook. 31 August 2016.
- "The rest of the 4.14 merge window [LWN.net]". lwn.net.
- "Linux_4.14 - Linux Kernel Newbies". Kernelnewbies.org. Retrieved 2018-08-16.
- "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix". www.phoronix.com.
- "Info" (PDF). open-zfs.org. 2017.
- Collet, Yann; Kucherawy, Murray (2018), RFC 8478: Zstandard Compression and the application/zstd Media Type, Internet Engineering Task Force Request for Comments, Menlo Park, CA: IETF Trust.
- "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. Mar 12, 2018. Retrieved 13 August 2018.
- "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix. 12 March 2018. Retrieved 13 August 2018.
- "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. August 31, 2016. Retrieved 2016-09-01.
- zstd/PATENTS "Additional Grant of Patent Rights Version 2", Facebook
- "Zstd v1.3.1 release", GitHub "facebook/zstd"
- "New license", GitHub "facebook/zstd"
- Official website
- zstd on GitHub
- "Smaller and faster data compression with Zstandard", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
- The Guardian is using ZStandard instead of zlib