zpaq is a journaling, incremental, deduplicating, multi-threaded archiver for Windows and Linux. "Journaling" means that when you update a file or directory, both the old and new versions are saved. You can extract from the archive as it existed at any time in the past. "Incremental", means that when you back up your entire hard drive, for example:
zpaq add e:backup.zpaq c:\*that only those files whose last-modified date has changed since the previous backup are added. For 100 GB of files, this typically takes 1-2 minutes, vs. a few hours to create the first version. "Deduplicating" means that identical files or fragments are stored only once to save time and space.
zpaq is open source (GPL v3) and not known to be encumbered by patents. Files are compressed in the open standard ZPAQ format, which is supported by a specification, reference decoder, and a test case (which should decode to the Calgary corpus), and a public domain API in C++ that provides compression and decompression services. ZPAQ stores the decompression algorithm descripion in the archive, which means that when the program is updated to use a better compression algorithm, older extractors will still be able to read the new archives. zpaq ranks highly by speed and compression ratio in many benchmarks:
The download zpaq v6.42 (zip) includes the following files:
Utilities for compression algorithm developers (sample config files and the zpaqd tool to run, test, and debug them).
You might need unzip.exe v6.00 to extract zip archives from the Windows command line.
past source code releases
on Github, maintained by Evan Nemerson.
WinZPAQ GUI interface (discussion) for benchmark testing, by Sportman.
Zpaq Explorer GUI for browsing archives (list and extract only) by Surfersat.
Github fork of zpaq v6.20 maintained by Yonggang Luo.
PeaZip by Giorgio Tani, supports ZPAQ streaming format, among many others. Does not support journaling format.
Debian packages, very old version.
ZPAQ development began on Feb. 15, 2009 with a series of mutually incompatible experimental programs (v0.01 through v0.09, now obsolete). Archives created with these versions could only be read by the same version. The level 1 standard became fixed on Mar. 12, 2009 with version 1.00. All versions from 1.00 onward are level 1 compliant and can extract older archives. With two exceptions, all programs can also read archives produced by newer versions. This forward compatibility was broken twice:
Other major developments:
The SHA-1 code used in these older versions (prior to libzpaq 1.00) is derived from RFC 3174, which is copyright (C) 2001, The Internet Society. Please see this document for the full license.
Open source (C++) and Win32 executables, Feb. 15, 2009.
zpaq v0.02 adds E8E9 transform. Fully supports post-processing. Not compatible with v0.01. Feb. 19, 2009.
zpaq v0.03 modifies MIX, MIX2, IMIX to fix poor compression on large files. Not compatible with v0.02. Feb. 19, 2009.
zpaq v0.04 modifies train() and squash() for improved compression. Not compatible with v0.03. Feb. 21, 2009.
zpaq v0.05 modifies probability representation and mixer weights to prevent mixer overflow and to improve compression for highly redundant data. Not compatible with v0.04. Feb. 26, 2009.
zpaq v0.06 adds SHA1 checksums, replaces IMIX2 with ISSE. Not compatible with v0.05. Feb. 27, 2009.
zpaq v0.07 improves ISSE and bit-history state table. Not compatible with v0.06. Feb. 28, 2009.
zpaq v0.08 adds LZP transform and minor improvements. Not compatible with v0.07. Mar. 8, 2009.
zpaq v0.09 removes counters from ISSE and ICM to improve speed. Not compatible with v0.09. Mar. 9, 2009.
zpaq v1.00 (first level 1 compliant version) includes unzpaq1 candidate reference decoder. Simplified bit history tables. Not compatible with earlier versions. Mar. 12, 2009.
fast.cfg written Apr. 26, 2010. Now part of libzpaq distribution.
unzpaq 1.01 updates reference decoder comments and help message and fixes some VS2005 compiler issues. Compatible with 1.00. Apr. 27, 2009.
unzpaq 1.02 and zpaq 1.02 closes extracted files immediately after decompression instead of when program exits. Fixes g++ 4.4 warnings. Compatible with 1.00 and 1.01. June 14, 2009.
unzpaq 1.03 and zpaq 1.03 has a default compression mode (mid.cfg), supports compressing files in segments to separate blocks and extracting them as suggested in part 7 of the spec. Does not store paths by default. Does not extract to absolute paths by default. Some minor improvements. Sept. 7, 2009 (added zpaq.exe Sept. 8, 2009).
zpaq103b adds zpaqsfx 1.03, a stub for creating self extracting archives. No changes to zpaq or unzpaq. Sept. 14, 2009.
zpaq104 can list and extract from self extracting archives without running them. Added progress meter. zpaqsfx.exe stub is slightly smaller. unzpaq unchanged. Sept. 18, 2009.
zpaq105 removes built in x and p preprocessors and makes them separate programs called from config files with compile time postprocessor testing. Adds if-else-endif and do-while to ZPAQL. Many small changes. Sept. 28, 2009.
zpaq106 adds "ta" to append locater tags to allow ZPAQ streams to be detected when embedded in arbitrary data. zpaq1.pdf revision 1 adds this recommendation. unzpaq106.cpp implements it. Sept. 29, 2009.
zpaqsfx 1.06 self extracting archive stub is now separate from the ZPAQ distribution. Updated Sept. 29, 2009, posted Oct. 26, 2009. (Replaced by zpsfx in libzpaq 2.01)
zpipe v1.00, a simple streaming compressor, Sept. 30, 2009. Linux patch added Jan. 18, 2010.
zpaq107 adds config file parameters and fixes some bugs. From now on the specification and reference decoder are not included unless they change. Oct. 2, 2009.
bwt_j2 is a config file (by Jan Ondrus) and preprocessor for BWT compression. Posted Oct. 7, 2009.
bwt_j3 is a bug fix for bwt_j2 to accept multiple files. Jan Ondrus, Oct 7, 2009.
exe_j1 is a config file and preprocessor for .exe and .dll files. It extends the E8E9 transform in exe.cfg to conditional jumps. Jan Ondrus, Oct. 7, 2009.
unzpaq108.cpp removes undefined behavior of ZPAQL shifts larger than 31 bits on non x86 hardware. Oct. 14, 2009.
zpaq108 generates optimized code that runs about twice as fast on systems with a C++ compiler installed. Oct. 14, 2009.
bmp_j4, configuration for .bmp files by Jan Ondrus, Oct. 14, 2009.
bwt_slowmode1 BWT compression based on BBB slow mode. Jan Ondrus, Oct. 15, 2009.
jpg_test2 JPEG config by Jan Ondrus, Oct. 20, 2009, posted Oct. 26, 2009.
zpaq109 Linux port and some cosmetic bug fixes. Oct. 21, 2009.
zpaq110 bug fix for Linux/g++ 4.4.1, Dec. 28, 2009.
zp v1.00 simple ZPAQ compatible archiver with 3 optimized compression levels, Apr. 26, 2010.
Added a license file to zpaq 1.10, zpipe 1.00, zpaqsfx 1.06, and zp 1.00 distributions on May 23, 2010. No software changes.
libzpaq 0.01, Sept. 27, 2010.
libzpaq 0.02, Sept. 28, 2010.
zpipe 2.00 updated to use libzpaq 0.02, Sept. 28, 2010.
libzpaq 1.00, Sept. 29, 2010. Package includes libzpaq, ZPAQ specification, reference decoder, zp, zpipe, and fast, mid, max config files.
libzpaq 1.01, Oct. 14, 2010. Updates libzpaq interface to use inheritance instead of templates, requiring changes to zp and zpipe. Now compiles faster.
libzpaq 1.02, Oct. 20, 2010. Adds zpsfx self extracting archive stub. Separates optimized models from libzpaq.cpp to libzpaqo.cpp.
libzpaq 2.00, Oct. 30, 2010. Ports zpaq to libzpaq, replacing zp.
libzpaq 2.01, Nov. 5, 2010. Added optimized self extracting archives. Simplified installation.
libzpaq 2.02, Nov. 13, 2010. zpaq shows compression component statistics. Libzpaq support added.
zpaq 2.03, Dec. 23, 2010, adds Linux support. The remaining code is split into libzpaq 2.02, zpipe 2.01, zpsfx 1.00, and configuration files min, fast, mid, and max.
zpaq 2.04, Dec.29, 2010, adds support for Visual C++, Borland, and Mars compilers in addition to g++. A Windows install script is added.
zpaq 2.05, Jan. 5, 2011. Fixed a bug in which zpaq crashed when decompressing an unnamed file (as created with zpipe or zpaq nc) without renaming. Separated zpaq.1.pod. (Updated corrupted install.sh on Jan. 13, 2011).
libzpaq 2.02a, Jan. 6, 2011. Updates the documentation.
pzpaq 0.01 parallel file compressor, Jan. 21, 2011.
pzpaq 0.02, Jan. 26, 2011, adds large file support (over 2 GB) to Windows.
pzpaq 0.03, Feb. 2, 2011, optimizes decompression for nonstandard compression levels by recompiling itself with g++ (like "zpaq ox").
pzpaq 0.04, Feb. 4, 2011, Windows version uses native threads and no longer requires pthreads-win32.
pzpaq 0.05, Feb. 10, 2011 removes -s option, puts temporary files in $TMPDIR or %TEMP%.
bwt v1, Mar. 16, 2011. 4 BWT based configurations.
unzp 1.00, May 10, 2011, a block level parallel decompresser optimized for fast, mid, max, bwtrle1, bwt2 models with source level JIT for other models.
zp 1.01, May 12, 2011, a block level parallel compressor with 4 levels (bwtrle1, bwt2, mid, max). With unzp replaces pzpaq.
zp 1.02, May 16, 2011. Fixed -t option.
May 18, 2011. Undated zp.102.zip and unzp.100.zip with static x86-64 Linux binaries.
zp 1.03, May 26, 2011. Merges the compressor and decompresser unzp into one program.
wbpe 1.00, June 12, 2011. Dictionary preprocessor for text files.
wbpe 1.10, June 21, 2011.
zpaq 3.00, July 16, 2011. Combines features of zpaq v2.05 and zp v1.03. zp support is discontinued. Windows only.
zpaq 3.01, July 21, 2011. Adds 64 bit Linux support. Includes libzpaq 3.00.
bmp_j4a, July 21, 2011. Updated bmp_j4 .bmp configuration for zpaq v3.01.
libzpaq 3.00, July 28, 2011, from zpaq v3.01 but as a separate download.
libzpaq 4.00, Nov. 13, 2011. libzpaq.cpp, libzpaq.h, libzpaq.3.pod. Replaces source-level JIT with internal JIT for x86-32 and x86-64.
zpaq 4.00, Nov. 13, 2011. zpaq.cpp, zpaq.1.pod for use with libzpaq 4.00. Removes source generation, b and e commands and -j option.
calgarytest.zpaq, Nov. 13, 2011. Test case for ZPAQ compliance.
zpipe v2.01, Nov. 13, 2011. zpipe.exe linked to libzpaq v4.00. Source unchanged.
zpaq v4.01, Nov. 26, 2011. Source code adds incremental update and extraction.
zpaq v4.02, Nov. 28, 2011. Source code adds commands c, x output/, list hcomp/pcomp. Updated pi.cfg for this version.
libzpaq v4.01, Dec. 20, 2011. Fix for Mac OS (MAP_ANONYMOUS -> MAP_ANON).
zpaq v4.03, Dec. 21, 2011. Adds -n, -r, and -f options. Fixed bug in u (did not save filenames with no args).
lz1.zip, Dec. 29, 2011. LZ77 model.
ZPAQ level 2 standard, unzpaq200.cpp reference decoder, libzpaq 5.00 support, and calgarytest2.zpaq test case, Feb. 1, 2012. Level 2 allows the COMP section to be empty to store uncompressed (but possibly preprocessed) data to support faster compression models.
libzpaq 5.01, Feb. 2, 2012. Removed debugging code from libzpaq.cpp.
tiny_unzpaq.cpp v1.0, Mar. 21, 2012.
zpaq v4.04, Mar. 26, 2012. Fixed bug in r command that truncated output file.
zpsfx v1.01, Apr. 4, 2012. Self extractor modified by Klaus Post to create directories as needed.
zpaq v5.00, Aug. 27, 2012. Candidate release, primarily a small development tool. Updates libzpaq to v6.00a to include ZPAQL compiler.
zpaq v6.00, Sept.26, 2012. Candidate release. Adds journaling, incremental update, and deduplication to support large backups. Includes 4 compression levels (fast and slow LZ77, BWT, mid) plus all v5.00 features.
zpaq201.pdf, Sept. 28, 2012. Updated specification to describe streaming and journaling archive formats.
zpaq v6.01, Sept. 28, 2012. Adds -method 0, -list -force, improves -list -detailed, and bug fixes.
zpaq v6.02, Sept. 29, 2012. Speed and compression improvements. Adds -quiet option.
zpaq v6.03, Sept. 30, 2012. Saves and restores file attributes. Cleans up -list.
zpaq v6.04, Oct. 1, 2012. Compression and speed improvements by sorting by filename extension and storing uncompressible data. Adds -list -quiet.
bmp_j4b.zip, Oct. 1, 2012. Updated bmp_j4 .bmp model to work with zpaq v6.xx
zpaq v6.05, Oct. 2, 2012. Adds -list -history -summary, Linux port, bug fixes, and improved docs.
zpaq v6.06, Oct. 4, 2012. Simplifies -list and adds -compare.
zpaq v6.07, Oct. 7, 2012. Fixes porting issues with Mac OS/X and Visual C++.
lazy v1.00, Oct. 10, 2012. A fast LZ77 compressor/preprocessor and and config.
lazy2 v1.00, Oct. 31, 2012. lazy with an E8E9 filter (and 1 GB file size limit).
zpaq v6.16, Nov. 5, 2012. Better compression using lazy (-method 1) + e8e9 (all methods). Adds -test and -post.
zpaq v6.17, Dec. 13, 2012. Fixed display of international characters. libzpaq v6.17 has slightly faster SHA1. Has bugs. Do not use.
zpaq v6.18, Dec. 14, 2012. Bug fix.
zpaq v6.19, Jan. 23, 2013. Splits into zpaq (journaling archiver) and zpaqd (development tool). Adds methods 5-9. libzpaq v6.19 adds single pass compression checksums.
bmp_j4c, Jan. 24, 2013. Updated .bmp config file to work with new zpaq/zpaqd syntax.
zpaq v6.20, Feb. 1, 2013. Improved compression for methods 5 through 9. zpaq64.exe added Feb. 4, 2013.
zpaq v6.21, Feb. 6, 2013. Extract directories restores timestamps and attributes. Adds -until date. Lists alphabetically. Fixed docs. zpaq621-64.exe added Feb. 8, 2013.
zpaq v6.22, Feb. 13, 2013. -method supports custom algorithms. zpaqd and libzpaq fixes for Win64. Command line accepts international characters.
zpaq v6.23, Mar. 25, 2013. -method supports config files without preprocessors. zpaqd 6.23 speed improvements for g++ 4.7.0 and "ds" command. libzpaq 6.23 faster initialization.
zpaq v6.24, Apr. 12, 2013. Adds d (delete) command. Works with wildcards. zpaqd adds built-in configs 1..3.
zpaq v6.24a, Apr. 14, 2013. Recompile zpaq.exe, zpaqd.exe to get around compiler bug in 64 bit version of MinGW causing 32 bit zpaq to crash in WinXP.
zpaq v6.25, Apr. 19, 2013. libzpaq optimizations (3-5% faster) and bug fix for WinXP. No changes to zpaq or zpaqd except version number.
zpaq v6.26, May 9, 2013. Optimizations: zpaq improves grouping of incompressible files into blocks, faster StringBuffer. libzpaq JIT optimizes consecutive ZPAQL increments. zpaqd fixes compiler warning.
zpaq v6.27, May 13, 2013. Adds -all and -test options. Improved recovery of damaged archives. zpaqd updated to verify checksums when listing journaling archives.
zpaq v6.28, May 22, 2013. Changed zpaq -test to a command. Improved handling of damaged archives.
zpaq202.pdf, June 3, 2013. Level 2 revision 2 of spec adds a fragmentation recommendation for deduplication compatibility.
zpaq v6.29, June 6, 2013. Improved compression. Extended method 1 and 2 LZ77 parameters. Test command implements new 2.02 spec.
zpaq v6.30, June 7, 2013. Fixes bug in extracting read-only files. Adds -attr option.
zpaq v6.31, June 7, 2013. Changed -attr default to select all files.
zpaqd v6.27, June 11, 2013. From zpaq 6.27 but now a separate distribution.
zpaqd v6.32, June 19, 2013. Faster I/O when linked with libzpaq v6.32 (included).
zpaqd v6.33, June 20, 2013. libzpaq 6.33 bug fix and recompile to fix list command. No change to zpaqd 6.32 source.
zpaq v6.33, June 21, 2013. Improved compression, supports block sizes, streaming mode, -fragile option and compress to empty archive. Removed -attributes, -above. -method is 0..7
zpaq v6.34, June 25, 2013. Supports long LZ77 offsets. -method is 0..6. Default block size increased to 64 MB for 2..6.
zpaq v6.35, June 28, 2013. LZ77 look-ahead and other improvements. Better handling of nonexistent input files.
zpaq v6.36, July 5, 2013. LZ77 compression improvements. Memory options for special methods.
zpaq v6.37, July 15, 2013. Adds purge command.
zpaq v6.38, July 18, 2013. Fixes extraction bug in v6.28-6.37. Adds compare command.
zpaq v6.39, July 25, 2013. List command shows compression ratios. Fixes -method 0 compression in DEBUG mode.
zpaq v6.40, July 28, 2013. Adds -noattributes option. Windows version does not add reparse points.
zpaq v6.41, Aug. 2, 2013. Adds restore command. Fixed wildcard handling and extract -fragile.
zpaq v6.42, Sept. 26, 2013. Adds list -duplicates, faster updates, minor bug fixes.
Discussion about ZPAQ updates.
ZPAQ is intended to replace PAQ and its variants (PAQ8, PAQ9A, LPAQ, LPQ1, etc) with similar or better compression in a portable, standard format. Current versions of PAQ break archive compatibility with each compression improvement. ZPAQ is intended to fix that. I no longer maintain the older PAQ code.
ZPAQ was written by Matt Mahoney, mattmahoneyfl (at) gmail (dot) com