Zstd: Multi-threading / multi-threaded support for multi-core system
Please consider adding support for multithreading / multithreaded compresion and decompression for faster processing on systems with multiple CPU cores.
Hedda
All 4 comments
pzstd is a command line tool that does multithreaded compression and decompression. The pzstd format uses zstd skippable frames to encode the information it needs for parallel decompression, so pzstd compressed data is compatible with standard zstd decompression.
Adding parallel compression to the zstd binary is planned. Issue #471 also tracks the addition of a parallel library function.
terrelln
on 19 Dec 2016
There is also an zstdmt library. But it's API is not considered stable currently... also the documentation is not finished.
ghost
on 20 Dec 2016
FWIW, the Blosc library also has a multithreaded implementation for a series of codecs (currently BloscLZ, LZ4, LZ4HC, Snappy and Zstandard), plus support for shuffling filters with support for SSE2 and AVX2 SIMD instruction set in Intel-compatible processors.
FrancescAlted
on 20 Dec 2016
Multi-threading support has been added to latest release v1.1.3.
Since it's still experimental, it needs to be explicitly targeted (make zstdmt from root file).
Then it's possible to request nb of threads from the command line, for example :
zstd -T4 file
Cyan4973
on 6 Feb 2017
Related issues
planet36
·
3Comments
itsnotvalid
·
3Comments
terrelln
·
3Comments
animalize
·
3Comments
AbdulrahmanAltabba
·
3Comments
Most helpful comment
pzstd is a command line tool that does multithreaded compression and decompression. The pzstd format uses zstd skippable frames to encode the information it needs for parallel decompression, so pzstd compressed data is compatible with standard zstd decompression.
Adding parallel compression to the
zstdbinary is planned. Issue #471 also tracks the addition of a parallel library function.