DataCpmpression with XZ Utils

[MetalKing]

I did find that there is an OpenSource DataCompression-Software with very significant better Compression. First Tests say it compresses ca. 20%. That means that if bzip2, normally better than gzip, compresses a File to 100kb than xz compresses it even to 80kb. Assuming that would work for whole Wesnoth-Package ca. 300MB the new Package would be just 240MB and that's really a Difference valuable to consider Constraints of Changing to XZ. Changing is a Question of Success of that Format but 20% better Compression would be a clear Answer about Success. However, I did download Source (<1mb) and will check out what it can do.
XZ can get "embedded". If I got it right that means XZ can produce self-extracting (SFX) Archives and XZ-Decompressor takes just 8-20KB from SFX-Archive.


Source from ProjectSite: http://tukaani.org/xz/

[ancestral]

Looks possible.

Data compression of art assets, sound, and really anything has been discussed before. I think people are generally welcome to smaller file and project sizes. If done well, it could make for a decent GSoC proposal, or long-term project.

[Dugi]

If you compress text files, its size will decrease significantly. But music and pictures will not decrease in size. Or they will, but only a very little bit.

The size of the executed file is 10 MiB. The size of Wesnoth core is 231 MiB when uncompressed (on Linux). Music is 145 MiB from it. Images are 77 MiB. Sounds are 7.4 MiB. Total 230 MiB out of 231 MiB. The total size of campaigns is 74 MiB. For UtBS, the size of images is 2.4 MiB, the size of maps, scenarios, units and utils totals at 1.7 MiB. This is the campaign with the most text (as it has only a very few profiles and very complicated scenarios. HttT is 3.8/0.7. LoW is 7.4/0.8. Others are like that. We can conclude that it has about 5 times more pictures than text. That is 14 MiB of text, although it is probably less. In total, that is 25 MiB of text, and if it is compressed with a 20% effectivity and its size will be decreased by 30%, it would be then 1.6 MiB less data. It seems to me that it would not do too much help.

Edit: Translations' size is 74 MiB and they are text files. That might be 3 MiB additional size taken.

Just to try it, decompressed the contents of the installer (I chose the .exe one, the Linux one is not in one file). Then I compressed it into an XZ archive. The size of the recompressed archive was 313.6 MiB. The size of the .exe file is 312.7 MiB. The size of a BZ2 archive was 336 MiB. Not sure if it would be an improvement.

[Iris]

I don’t consider file size to be very relevant anymore (at least below 0.5 GiB), except for software packages up for download. As far as I can tell, slow or quota-restricted Internet connections are far more common than the lack of cheap, large storage media.

If there’s a better, widely supported compression format available for use for our release tarballs then the person you need to convince of this is Ivanovic, not the forums.

The other area where a different compression format could be useful in certain circumstances is the network protocol. Currently, all WML transmitted to and from the add-ons and MP servers is streamed through a gzip filter at both endpoints. This is generally good enough for most purposes, but it may fall short of the players’ expectations when dealing with large add-ons from the add-ons server, or large scenarios with loads of WML/large tile maps from the MP server. On the other hand, as far as I know gzip is a balanced solution in terms of CPU usage, which is very important given how our online infrastructure is currently set-up for practical and financial reasons.

Of course, in the add-ons server case in particular, the bulk of the content of the larger add-ons consists of binary files (mainly PNG and Ogg Vorbis files) that already use their own compression techniques and don’t benefit much from an extra layer of encoding. It’s also not clear at this point whether the future add-ons server will continue to use the WML protocol.

Just my two cents.

[MetalKing]

I did check XZ with wesnoth-1.10.1.tar:

bzip2 (-9 ?) wesnoth-1.10.1.tar -> 326.1 MB
xz -9 wesnoth-1.10.1.tar -> 301.1 MB
Ergo 25 MB safed with xz-Compressor.

[Nafac]

Container format should be used, installing hundreds or even thousands of little images, sound files etc. is a pain.

[MetalKing]

Theory about Compression and Media-Data is interesting but XZ DOES compress significantly at least mp3 and at least one Image on my PC. Can anyone confirm that?

[Iris]

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -Jcf music.tar.xz music/
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music
139M    music
shadowm@reicore:~/src/wesnoth/data/core$

EDIT: since I am fairly sure this will crop up later in some fashion:

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -cf music.tar music/
shadowm@reicore:~/src/wesnoth/data/core$ xz -9 music.tar 
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz 
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$

[MetalKing]

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -Jcf music.tar.xz music/
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music
139M    music
shadowm@reicore:~/src/wesnoth/data/core$

EDIT: since I am fairly sure this will crop up later in some fashion:

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -cf music.tar music/
shadowm@reicore:~/src/wesnoth/data/core$ xz -9 music.tar 
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz 
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$

That's almost nothing but for OverALl-Performance of TarBall of Wesnoth the Numbers are fine.