HN

Need help?
<- Back
bzip3

bzip3

tosh

Comments (79)

  • altairprime
    Previously:“Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407“bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html(2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713(4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439
  • 8organicbits
    I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...
  • ot
    The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.If we match the window size of zstd to that of bzip3 we get dramatically different results: % gzcat *.gz | time zstd -T8 -16 | wc -c # baseline 2819113884 zstd -T8 -16 2054.50s user 3.47s system 783% cpu 4:22.80 total % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c 196405076 zstd -T8 -16 --long=29 1083.06s user 2.41s system 783% cpu 2:18.55 total Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).(the baseline number is slightly different because I don't have the exact Perl version set used by the author)Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.
  • CodesInChaos
    I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.
  • amelius
    > However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.Sounds like perhaps a nice testcase for formalization + AI?
  • JdeBP
    An interesting unintentional benchmark is to go to https://github.com/iczelia/bzip3/releases and see to what degree bzip3 compresses its own release archives; and go to https://github.com/iczelia/bzip3/blob/master/.github/workflo... to see what options have been chosen for the other compressors here.
  • sehw
    For data recovery of archival stuff it's better to keep it uncompressed, no?
  • ThiraSoft
    I would be interrested in a comparison with openzl
  • charcircuit
    It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.
  • sergiotapia
    "DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE."So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.
  • red_admiral
    Why LGPL when the original license is more permissive? (https://sourceware.org/bzip2/manual/manual.html)
  • self_awareness
    Any relation to Iczelion from the masm32 tutorial?
  • algorithm314
    Bro just decompile agiannis_text
  • roschdal
    Imagine bzip4
  • sedatk
    The latest release is a year ago, the last commit is two months ago, and the build is failing.The claim “stronger than bzip2” is strange. What does it even mean?Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.
  • sylware
    Isn't that XZ?
  • kosolam
    Impressive compression benchmark. Four times smaller than z standard.