Jump to content

XZ Utils: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
ItzSwirlz (talk | contribs)
270 edits
Temporarily remove stable release field due to malware (5.6.1 is compromised)
Line 1: Line 1:
{{short description|Lossless data compression software}}
{{short description|Lossless data compression software}}
{{Infobox software
{{Infobox software
| author = Lasse Collin
| author = Lasse Collin
| developer = The Tukaani Project
| developer = The Tukaani Project
| latest release version = {{wikidata|property|edit|reference|P348}}
| latest release version = N/A due to ongoing supply chain attack investigation.
| latest release date = {{wikidata|qualifier|P348|P577}}
| latest release date =
| operating system = [[Cross-platform]]
| operating system = [[Cross-platform]]
| programming language = [[C (programming language)|C]]
| programming language = [[C (programming language)|C]]
| genre = [[Data compression]]
| genre = [[Data compression]]
| license = [[public-domain software|Public domain]].<ref name="licensing"/> (but see details in [[XZ Utils#Development and adoption|Development and adoption]])
| license = [[public-domain software|Public domain]].<ref name="licensing"/> (but see details in [[XZ Utils#Development and adoption|Development and adoption]])
| website = <!--{{URL|https://tukaani.org/xz/}}{{Dead link|date=March 2024}}-->{{Web archive|url=https://web.archive.org/web/20240325040631/https://xz.tukaani.org/xz-utils/}}
| website = <!--{{URL|https://tukaani.org/xz/}}{{Dead link|date=March 2024}}-->{{Web archive|url=https://web.archive.org/web/20240325040631/https://xz.tukaani.org/xz-utils/}}
}}
}}


{{Infobox file format
{{Infobox file format
| name = .xz
| name = .xz
| icon =
| icon =
| iconcaption =
| iconcaption =
| icon_size =
| icon_size =
| screenshot =
| screenshot_size =
| screenshot =
| screenshot_size =
| caption =
| caption =
| extension = .xz
| extension = .xz
| mime = application/x-xz
| uniform_type =
| mime = application/x-xz
| uniform_type =
| conforms_to =
| conforms_to =
| magic = <code>FD 37 7A 58 5A 00</code>
⚫
| developer = Lasse Collin<br />[[Igor Pavlov (programmer)|Igor Pavlov]]
| magic = <code>FD 37 7A 58 5A 00</code>
⚫
| released = {{start date and age|2009|01|14}}
⚫
| developer = Lasse Collin<br />[[Igor Pavlov (programmer)|Igor Pavlov]]
⚫
| released = {{start date and age|2009|01|14}}
| latest_release_version = 1.1.0
| latest_release_version = 1.1.0
| latest_release_date = {{start date and age|2022|12|11}}
| latest_release_date = {{start date and age|2022|12|11}}
| type = [[Data compression]]
| type = [[Data compression]]
| extended_from =
| extended_from =
| standard = <!-- or: | standards = -->
| standard = <!-- or: | standards = -->
| free = Yes
| free = Yes
| open = Yes
| open = Yes
| url = <!--{{URL|https://tukaani.org/xz/xz-file-format.txt}}{{Dead link|date=March 2024}}-->{{Web archive|url=https://web.archive.org/web/20231123141839/https://tukaani.org/xz/xz-file-format.txt}}
| url = <!--{{URL|https://tukaani.org/xz/xz-file-format.txt}}{{Dead link|date=March 2024}}-->{{Web archive|url=https://web.archive.org/web/20231123141839/https://tukaani.org/xz/xz-file-format.txt}}
}}
}}


Revision as of 00:06, 1 April 2024

XZ Utils
Original author(s)Lasse Collin
Developer(s)The Tukaani Project
Stable release
N/A due to ongoing supply chain attack investigation.
Repository
Written inC
Operating systemCross-platform
TypeData compression
LicensePublic domain.[1] (but see details in Development and adoption)
WebsiteArchived 2024-03-25 at the Wayback Machine
.xz
Filename extension
.xz
Internet media type
application/x-xz
Magic numberFD 37 7A 58 5A 00
Developed byLasse Collin
Igor Pavlov
Initial releaseJanuary 14, 2009; 15 years ago (2009-01-14)
Latest release
1.1.0
December 11, 2022; 22 months ago (2022-12-11)
Type of formatData compression
Open format?Yes
Free format?Yes
WebsiteArchived 2023-11-23 at the Wayback Machine

XZ Utils (previously LZMA Utils) is a set of free software command-line lossless data compressors, including the programs lzma and xz, for Unix-like operating systems and, from version 5.0 onwards, Microsoft Windows. For compression/decompression the Lempel–Ziv–Markov chain algorithm (LZMA) is used. XZ Utils started as a Unix port of Igor Pavlov's LZMA-SDK that has been adapted to fit seamlessly into Unix environments and their usual structure and behavior.

Features

In most cases, xz achieves higher compression rates than alternatives like gzip and bzip2. Decompression speed is higher than bzip2, but lower than gzip. Compression can be much slower than gzip, and is slower than bzip2 for high levels of compression, and is most useful when a compressed file will be used many times.[2][3]

XZ Utils consists of two major components:

Various command shortcuts exist, such as lzma (for xz --format=lzma), unxz (for xz --decompress; analogous to gunzip) and xzcat (for unxz --stdout; analogous to zcat)

XZ Utils can compress and decompress both the xz and lzma file formats, but since the LZMA format is now legacy,[4] XZ Utils compresses by default to xz.

Usage

Both the behavior of the software as well as the properties of the file format have been designed to work similarly to those of the popular Unix compressing tools gzip and bzip2.

Just like gzip and bzip, xz and lzma can only compress single files (or data streams) as input. They cannot bundle multiple files into a single archive – to do this an archiving program is used first, such as tar.

Compressing an archive: 

xz   my_archive.tar    # results in my_archive.tar.xz
lzma my_archive.tar    # results in my_archive.tar.lzma

Decompressing the archive: 

unxz    my_archive.tar.xz      # results in my_archive.tar
unlzma  my_archive.tar.lzma    # results in my_archive.tar

Version 1.22 or greater of the GNU implementation of tar has transparent support for tarballs compressed with lzma and xz, using the switches --xz or -J for xz compression, and --lzma for LZMA compression.

Creating an archive and compressing it: 

tar -c --xz   -f my_archive.tar.xz   /some_directory    # results in my_archive.tar.xz
tar -c --lzma -f my_archive.tar.lzma /some_directory    # results in my_archive.tar.lzma

Decompressing the archive and extracting its contents: 

tar -x --xz   -f my_archive.tar.xz      # results in /some_directory
tar -x --lzma -f my_archive.tar.lzma    # results in /some_directory

Single-letter tar example for archive with compress and decompress with extract using short suffix: 

tar cJf keep.txz keep   # archive then compress the directory ./keep/ into the file ./keep.txz
tar xJf keep.txz        # decompress then extract the file ./keep.txz creating the directory ./keep/

xz has supported multi-threaded compression (with the -T flag)[5] since 2014, version 5.2.0.;[6] since version 5.4.0 threaded decompression has been implemented. Threaded decompression requires multiple compressed blocks within a stream which are created by the threaded compression interface.[5] The number of threads can be less than defined if the file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit.[5]

The xz format

The xz format improves on lzma by allowing for preprocessing filters. The exact filters used are similar to those used in 7z, as 7z's filters are available in the public domain via the LZMA SDK.

Development and adoption

Development of XZ Utils took place within the Tukaani Project, which was led by Mike Kezner, by a small group of developers who once maintained a Linux distribution based on Slackware.

All of the source code for xz and liblzma has been released into the public domain. The XZ Utils source distribution additionally includes some optional scripts and an example program that are subject to various versions of the GPL.[1]

Specifically, the full list of GPL scripts and sources distributed with the XZ Utils software include:

  • An optional implementation of a common libc function, getopt (GNU LGPL v2.1)
  • An m4 script for pthread detection (GNU GPL v3)
  • Some nonessential wrapper scripts (xzgrep, etc) (GNU GPL v2)
  • And the example program scanlzma, which is not integrated with the build system

The resulting software xz and liblzma binaries are public domain, unless the optional LGPL getopt implementation is incorporated.[7]

Binaries are available for FreeBSD, NetBSD, Linux systems, Microsoft Windows, and FreeDOS. A number of Linux distributions, including Fedora, Slackware, Ubuntu, and Debian use xz for compressing their software packages. Arch Linux previously used xz to compress packages,[8] but as of December 27, 2019, packages are compressed with Zstandard compression.[9] Fedora Linux also switched to compressing its RPM packages with Zstandard with Fedora Linux 31[10]. The GNU FTP archive also uses xz.

Supply chain attack

Further information: Supply chain attack

On 29 March 2024, a thread was published on Openwall's oss-security mailing list showing that the code of liblzma was potentially compromised.[11] The thread author, Andres Freund, identified compressed test files which have been added to the code for setting up a backdoor via additions to the configure script in the tar files. He started his investigation because sshd was using a high amount of CPU.[12] The issue is tracked under the Common Vulnerabilities and Exposures ID CVE-2024-3094 with a Common Vulnerability Scoring System (CVSS) score of 10 (the highest), which was issued by Red Hat following the disclosure of the backdoor.[13]

The malicious code is known to be in version 5.6.0 and 5.6.1. While the exploit remains dormant unless a specific third-party patch of the SSH server is used, under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.[14] The malicious mechanism consists of two compressed test files that contain the malicious binary code. These files are available in the git repository, but remain dormant unless extracted and injected into the program.[15] The code uses the glibc IFUNC mechanism to replace an existing function in OpenSSH called RSA_public_decrypt with a malicious version. OpenSSH normally does not load liblzma, but a common third-party patch used by several Linux distributions cause it to load libsystemd, which in turn loads lzma.[15] A modified version of build-to-host.m4 was included in the release tar file uploaded on GitHub, which extracts a script that performs the actual injection into liblzma. This modified m4 file was not present in the git repository; it was only available from tar files released by the maintainer separate from git.[15] The script appears to only perform the injection when the system is being built on an x86-64 Linux system that uses glibc and GCC and is being built via dpkg or rpm.[15] GitHub has terminated the repository for XZ due to terms of service violation, presumably because of the supply chain attack.[16] While the backdoor was added by "JiaT75", a maintainer of the xz project, it is not certain whether this backdoor was intentionally placed by the maintainer or whether the maintainer was compromised.[17]

The list of affected Linux distributions includes Debian unstable and testing,[18] Fedora Rawhide,[19] Kali Linux,[20] and OpenSUSE Tumbleweed and Zorin OS.[21] Most Linux distributions that followed a stable release update model were not affected, since they were carrying older versions of xz.[15] Arch Linux issued an advisory for users to update immediately, although it also noted that Arch's OpenSSH package does not include the common third-party patch necessary for the backdoor.[22] FreeBSD is not affected by this attack, as all supported FreeBSD releases include versions of xz that predate the affected releases and the attack targets Linux's glibc.[23]

References

  1. ^ a b Licensing on tukaani.org "The most interesting parts of XZ Utils (e.g. liblzma) are in the public domain. You can do whatever you want with the public domain parts. Some parts of XZ Utils (e.g. build system and some utilities) are under different free software licenses such as GNU LGPLv2.1, GNU GPLv2, or GNU GPLv3."
  2. ^ Henry-Stocker, Sandra (2017-12-12). "How to squeeze the most out of Linux file compression". Network World. Retrieved 2020-02-09.
  3. ^ "Gzip vs Bzip2 vs XZ Performance Comparison". RootUsers. 2015-09-16. Retrieved 2020-02-09.
  4. ^ LZMA Utils, retrieved 2011-01-25
  5. ^ a b c "Linux Manpages Online - man.cx manual pages".
  6. ^ XZ Utils Release Notes
  7. ^ "In what cases is the output of a GPL program covered by the GPL too?". GNU.org. Retrieved 21 August 2019.
  8. ^ Pierre Schmitz (2010-03-23). "News: Switching to xz compression for new packages".
  9. ^ "Arch Linux - News: Now using Zstandard instead of xz for package compression". www.archlinux.org. Retrieved 2020-01-07.
  10. ^ Mach, Daniel. "Changes/Switch RPMs to zstd compression". Fedora Project Wiki. Retrieved 30 March 2024.
  11. ^ Freund, Andres (2024-03-29). "backdoor in upstream xz/liblzma leading to ssh server compromise". oss-security mailing list.
  12. ^ "A backdoor in xz". lwn.net. Retrieved 2024-03-30.
  13. ^ "NVD - CVE-2024-3094". nvd.nist.gov. Retrieved 2024-03-30.
  14. ^ "Urgent security alert for Fedora 41 and Rawhide users". www.redhat.com. Retrieved 2024-03-29.
  15. ^ a b c d e James, Sam. "xz-utils backdoor situation". Gist.
  16. ^ Larabel, Michael (29 March 2024). "GitHub Disables The XZ Repository Following Today's Malicious Disclosure". www.phoronix.com. Retrieved 31 March 2024.
  17. ^ Goodin, Dan (2024-03-29). "Backdoor found in widely used Linux utility breaks encrypted SSH connections". Ars Technica. Retrieved 2024-03-29.
  18. ^ "CVE-2024-3094". security-tracker.debian.org. Retrieved 2024-03-30.
  19. ^ "Urgent security alert for Fedora 41 and Fedora Rawhide users". www.redhat.com. Retrieved 2024-03-30.
  20. ^ "All about the xz-utils backdoor | Kali Linux Blog". Kali Linux. 2024-03-29. Retrieved 2024-03-30.
  21. ^ "openSUSE addresses supply chain attack against xz compression library". openSUSE News. 2024-03-29. Retrieved 2024-03-30.
  22. ^ "Arch Linux - News: The xz package has been backdoored". archlinux.org. Retrieved 2024-03-30.
  23. ^ "Disclosed backdoor in xz releases - FreeBSD not affected". Retrieved 2024-03-30.
Retrieved from "https://en.wikipedia.org/w/index.php?title=XZ_Utils&oldid=1216602142"