This article relies too much on references to primary sources. (September 2017) (Learn how and when to remove this template message)
|Original author(s)||Yann Collet|
|Developer(s)||Yann Collet, Przemysław Skibiński (inikep)|
|Initial release||23 January 2015|
1.3.5 / 28 June 2018
|License||Dual: BSD License + GPLv2|
Zstandard (or Zstd) is a lossless data compression algorithm developed by Yann Collet at Facebook. The name also refers to the reference implementation in C. Version 1 of the implementation was released as free software on 31 August 2016.
Zstandard was designed to give compression comparable to that of DEFLATE algorithm (developed in 1991, used in original ZIP, gzip and other) with higher compression / decompression speeds. Zstandard combines use of a dictionary-type algorithm (LZ77) with large search window and fast implementations of entropy coding stage, using either very fast Finite State Entropy (tANS) or Huffman coding. A feature of Zstandard implementations is backward direction of reading on entropy decompression.
Zstandard implements several compression levels for additional flexibility from -1 (fastest) to -22 (slowest in compression speed, but best compression ratio). There are also parallel (multi-threaded) implementations of both compression and decompression. Starting from 1.3.2 version from October 2017, Zstandard optionally implements very long range search and deduplication like rzip or lrzip.
According to LTCB compression benchmark compressing a single 1 GB text file, zstd at maximum levels shows compression ratio close to boz, yxz, tornado archivers, better than lza, brotli, or bzip2. With any compression level it has fast decompression. Zstandard reaches the Pareto frontier, which means it decompresses faster than any other algorithm with similar or better compression ratio.
Zstandard can use any user-defined pre-populated compression dictionary. It also offers a training mode, able to generate a dictionary from any set of samples. Dictionaries can have a large impact on the compression ratio of small files.
Zstandard method is supported in Linux Kernel since version 4.14 (released November 2017) for usage as compression method for file systems like btrfs and squashfs. It was also tested for FreeBSD with integration into OpenZFS file system.
The algorithm is also deployed in datacenters, such as AWS Red Shift , and in databases, such as RocksDB.
Method of Zstandard and MIME type "application/zstd" were submitted to IETF as an Internet draft.
- LZFSE – a similar algorithm by Apple used since iOS 9 and OS X 10.11 made open source on 1 June 2016
- LZ4 (compression algorithm) – a fast member of the LZ77 family
- "Releases - facebook/zstd". Retrieved 1 July 2018 – via GitHub.
- Sergio De Simone, Facebook Open-Sources New Compression Algorithm Outperforming Zlib / InfoQ, 2 September 2016
- "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 2016-08-31. Retrieved 2016-09-06.
- Matt Mahoney (2016-08-29). "Large Text Compression Benchmark, .2157 zstd". Retrieved 2016-09-01.
- TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
- Matt Mahoney, Silesia Open Source Compression Benchmark
- https://indico.fnal.gov/event/15154/contribution/5/material/slides/0.pdf "Facebook developers report massive speedups and compression ratio improvements when using dictionaries"
- Collet, Yann; Kucherawy, Murray S. "Zstandard Compression and The application/zstd Media Type". IETF. Retrieved 2018-07-13.
- "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. August 31, 2016. Retrieved 2016-09-01.
- zstd/PATENTS "Additional Grant of Patent Rights Version 2", Facebook
- "Zstd v1.3.1 release", GitHub
- "New license", GitHub
- Official website
- zstd on GitHub
- "Smaller and faster data compression with Zstandard", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
- The Guardian is using ZStandard instead of zlib