||It has been suggested that SmartPAR be merged into this article. (Discuss) Proposed since March 2014.|
|Type of format||Erasure code|
Parchive (a contraction of parity archive) is an erasure code system that produces par files for checksum verification of data integrity, with the capability to perform data recovery operations that can repair or regenerate corrupted or missing data. Parchive was originally written to solve the problem of reliable file sharing on Usenet, but it is now commonly used for protecting any kind of data from data corruption, bit rot, and accidental or malicious damage. Despite the name, Parchive uses more advanced techniques that do not utilize simplistic parity methods of error detection and correction.
The original SourceForge Parchive project has been inactive since November 9, 2010. As of 2014, PAR3 is the third generation of the technology, and is being actively developed by MultiPar author Yutaka Sawada.
Parchive was intended to increase the reliability of transferring files via Usenet newsgroups. Usenet was originally designed for informal conversations, and the underlying protocol, NNTP was not designed to transmit arbitrary binary data. Another limitation, which was acceptable for conversations but not for files, was that messages were normally fairly short in length and limited to 7-bit ASCII text.
Various techniques were devised to send files over Usenet, such as uuencoding and Base64. Later Usenet software allowed 8 bit Extended ASCII, which permitted new techniques like yEnc. Large files were broken up to reduce the effect of a corrupted download, but the unreliable nature of Usenet remained.
With the introduction of Parchive, parity files could be created that were then uploaded along with the original data files. If any of the data files were damaged or lost while being propagated between Usenet servers, users could download parity files and use them to reconstruct the damaged or missing files. Parchive included the construction of small index files (*.par in version 1 and *.par2 in version 2) that do not contain any recovery data. These indexes contain file hashes that can be used to quickly identify the target files and verify their integrity.
Because the index files were so small, they minimized the amount of extra data that had to be downloaded from Usenet to verify that the data files were all present and undamaged, or to determine how many parity volumes were required to repair any damage or reconstruct any missing files. They were most useful in version 1 where the parity volumes were much larger than the short index files. These larger parity volumes contain the actual recovery data along with a duplicate copy of the information in the index files (which allows them to be used on their own to verify the integrity of the data files if there is no small index file available).
In July 2001, Tobias Rieper and Stefan Wehlus proposed the Parity Volume Set specification, and with the assistance of other project members, version 1.0 of the specification was published in October 2001. Par1 used Reed-Solomon error correction to create new recovery files. An end user could use any of the recovery files to rebuild a missing file from an incomplete download.
Version 1 became widely used on Usenet, but it did suffer some limitations:
- It was restricted to handle at most 255 files.
- The recovery files had to be the size of the largest input file, so it did not work well when the input files were of various sizes. (This limited its usefulness when not paired with the proprietary RAR compression tool.)
- The recovery algorithm had a bug, due to a flaw in the academic paper on which it was based.
- It was strongly tied to Usenet and it was felt that a more general tool might have a wider audience.
In January 2002, Howard Fukada proposed that a new PAR2 specification should be devised with the significant changes that data verification and repair should work on blocks of data rather than whole files, and that the algorithm should switch to using 16 bit numbers rather than the 8 bit numbers that PAR 1 used. Michael Nahas and Peter Clements took up these ideas in July 2002, with additional input from Paul Nettle and Ryan Gallagher (who both wrote Par1 clients). Version 2.0 of the Parchive specification was published by Michael Nahas in September 2002.
Peter Clements then went on to write the first two PAR2 implementations, QuickPar and par2cmdline. Abandoned since 2004, Paul Houle created phpar2 to supersede par2cmdline, and Yutaka Sawada created MultiPar to supersede QuickPar.
Versions 1 and 2 of the file format are incompatible. (However, many clients support both.)
For version 1, given files f1, f2, ..., fn, the Parchive consists of an index file (f.par), which is CRC type file with no recovery blocks, and a number of "parity volumes" (f.p01, f.p02, etc.). Given all of the original files except for one (for example, f2), it is possible to create the missing f2 given all of the other original files and any one of the parity volumes. Alternatively, it is possible to recreate two missing files from any two of the parity volumes and so forth.
Version 1 supports up to 256 recovery files. Each recovery file must be the size of the largest input file.
Version 2 files generally use this naming/extension system: filename.vol000+01.PAR2, filename.vol001+02.PAR2, filename.vol003+04.PAR2, filename.vol007+06.PAR2, etc. The +01, +02, etc. in the filename indicates how many blocks it contains, and the vol000, vol001, vol003 etc. indicates the number of the first recovery block within the PAR2 file. If an index file of a download states that 4 blocks are missing, the easiest way to repair the files would be by downloading filename.vol003+04.PAR2. However, due to the redundancy, filename.vol007+06.PAR2 is also acceptable. There is also an index file filename.PAR2, it is identical in function to the small index file used in PAR1.
Version 2 supports up to 32768 (215) recovery blocks. Input files are split into multiple equal-sized blocks so that recovery files do not need to be the size of the largest input file.
Directory support is provided in MultiPar's implementation of PAR2.
PAR3 is a planned improvement over PAR2. The authors intend to fix problems related to creating or repairing when the block count or block size is very high. PAR3 also adds support for including directories (file folders) in a parchive and Unicode characters in file names. In addition, the authors plan to enable the PAR3 algorithm to identify files that have been moved or renamed.
- MultiPar — Builds upon QuickPar's features and GUI, and phpar2 as the PAR2 backend. It has support for PAR3, multithreading, multiple processors, and the ability to recurse subfolders, GPL. MultiPar is able to add recovery data to ZIP and 7-Zip files, with a few minor caveats. MultiPar is also verified to work with Wine under PCBSD, and may work with other operating systems too. Although the par2 and par3 components are (or will be) open source, the MultiPar GUI on top of them is currently not open source. See also: MultiPar Alternatives and Similar Software at AlternativeTo and multipar.eu for BitTorrent Sync download, screenshots, etc.
- QuickPar — freeware, unmaintained since 2004, superseded by MultiPar.
- par2+tbb is a concurrent (multithreaded) version of par2cmdline 0.4 using TBB, GPLv2, or later.
- Par-N-Rar (GPL)
- phpar2 — advanced par2cmdline with multithreading and highly optimized assemblercode (about 66% faster than QuickPar 0.9.1)
- Rarslave (GPLv2)
- SmartPAR (no support for PAR2)
- Mirror — First PAR implementation, unmaintained since 2001
- Mac OS X
- POSIX-conforming operating systems
- Bit rot
- Comparison of file archivers - Some file archivers are capable of integrating parity data into their formats for error detection and correction:
- Data corruption
- RAID - RAID levels at and above RAID 5 make use of parity data to detect and repair errors.
- SnapRAID uses drives filled with parity files to pool drives together with redundancy and recovery capabilities. It is similar to Parchive in that it can repair damaged files and restore deleted files, but it is much more automated. In some use cases, it can supplement or replace Parchive, and it compares favorably with other tools like FlexRAID, ZFS, Btrfs, etc.
- "Parchive: Parity Archive Volume Set". Retrieved 2009-10-29. "The original idea behind this project was to provide a tool to apply the data-recovery capability concepts of RAID-like systems to the posting and recovery of multi-part archives on Usenet."
- "Parchive: Parity Archive Tool". Retrieved 2012-09-02.
- possibility of new PAR3 file
- Question about your usage of PAR3
- Risk of undetectable intended modification
- PAR3 specification proposal not finished as of April 2011
- Kantor, Brian; Lapsley, Phil (February 1986). "Character Codes". Network News Transfer Protocol. IETF. p. 5. sec. 2.2. RFC 977. https://tools.ietf.org/html/rfc977#section-2.2. Retrieved 2009-10-29.
- Nahas, Michael (2001-10-14). "Parchive: Parity Volume Set specification 1.0". Retrieved 2009-04-07.[dead link]
- Plank, James S.; Ding, Ying (April 2003). "Note: Correction to the 1997 Tutorial on Reed-Solomon Coding". Retrieved 2009-10-29.
- Plank, James S. (September 1997). "A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems". Retrieved 2009-10-29.
- Nahas, Michael; Clements, Peter; Nettle, Paul; Gallagher, Ryan (2003-05-11). "Parity Volume Set Specification 2.0". Retrieved 2009-10-29.
- Wang, Wallace (2004-10-25). "Finding movies (or TV shows): Recovering missing RAR files with PAR and PAR2 files". Steal this File Sharing Book (1st ed.). San Francisco, California: No Starch Press. pp. 164 – 167. ISBN 1-59327-050-X. Retrieved 2009-09-24.
- http://www.quickpar.co.uk/forum/viewtopic.php?id=1065 QuickPar forum posting
- http://hp.vector.co.jp/authors/VA021385/ Beta release from MultiPar with PAR3 beta functionality
- http://www.quickpar.org.uk/forum/viewtopic.php?id=1264 QuickPar forum posting - status PAR3
- http://www.quickpar.co.uk/forum/viewtopic.php?id=1047 QuickPar forum posting - PAR3 specifications
- http://hp.vector.co.jp/authors/VA021385/par3_spec_prop.htm PAR3 proposal
- http://www.livebusinesschat.com/smf/index.php?topic=4751.0 PAR3 move/rename brainstorming
- http://sourceforge.net/tracker/?func=detail&aid=3141214&group_id=14481&atid=364481 7-Zip Parchive feature request
- How to add recovery record to ZIP or 7-Zip archive
- MultiPar works with PCBSD 9.0
- contacted you, asking about sourcecode
- SnapRAID comparison
- Parchive project - full specifications and math behind it
- Introduction to PAR and PAR2
- Slyck's Guide To The Usenet Newsgroups: PAR & PAR2 Files
- Guide to use PAR and PAR2 under QuickPar
- Another introduction to PAR and PAR2 and more information from the same site
- Guide to repair files using PAR2
- Tutorial on how to reconstruct files using PAR (Google translation from original Polish)