DataCpmpression with XZ Utils

Brainstorm ideas of possible additions to the game. Read this before posting!

Moderators: Forum Moderators, Developers

Forum rules
Before posting a new idea, you must read the following:
Post Reply
User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

DataCpmpression with XZ Utils

Post by MetalKing » March 22nd, 2012, 5:25 pm

I did find that there is an OpenSource DataCompression-Software with very significant better Compression. First Tests say it compresses ca. 20%. That means that if bzip2, normally better than gzip, compresses a File to 100kb than xz compresses it even to 80kb. Assuming that would work for whole Wesnoth-Package ca. 300MB the new Package would be just 240MB and that's really a Difference valuable to consider Constraints of Changing to XZ. Changing is a Question of Success of that Format but 20% better Compression would be a clear Answer about Success. However, I did download Source (<1mb) and will check out what it can do.
XZ can get "embedded". If I got it right that means XZ can produce self-extracting (SFX) Archives and XZ-Decompressor takes just 8-20KB from SFX-Archive.


Source from ProjectSite: http://tukaani.org/xz/
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

User avatar
ancestral
Developer
Posts: 1108
Joined: August 1st, 2006, 5:29 am
Location: Motion City

Re: DataCpmpression with XZ Utils

Post by ancestral » March 22nd, 2012, 5:31 pm

Looks possible.

Data compression of art assets, sound, and really anything has been discussed before. I think people are generally welcome to smaller file and project sizes. If done well, it could make for a decent GSoC proposal, or long-term project.
Wesnoth BestiaryPREVIEW IT HERE )
Unit tree and stat browser
CanvasPREVIEW IT HERE )
Exp. map viewer

User avatar
Dugi
Posts: 4931
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: DataCpmpression with XZ Utils

Post by Dugi » March 22nd, 2012, 6:04 pm

If you compress text files, its size will decrease significantly. But music and pictures will not decrease in size. Or they will, but only a very little bit.

The size of the executed file is 10 MiB. The size of Wesnoth core is 231 MiB when uncompressed (on Linux). Music is 145 MiB from it. Images are 77 MiB. Sounds are 7.4 MiB. Total 230 MiB out of 231 MiB. The total size of campaigns is 74 MiB. For UtBS, the size of images is 2.4 MiB, the size of maps, scenarios, units and utils totals at 1.7 MiB. This is the campaign with the most text (as it has only a very few profiles and very complicated scenarios. HttT is 3.8/0.7. LoW is 7.4/0.8. Others are like that. We can conclude that it has about 5 times more pictures than text. That is 14 MiB of text, although it is probably less. In total, that is 25 MiB of text, and if it is compressed with a 20% effectivity and its size will be decreased by 30%, it would be then 1.6 MiB less data. It seems to me that it would not do too much help.

Edit: Translations' size is 74 MiB and they are text files. That might be 3 MiB additional size taken.


Just to try it, decompressed the contents of the installer (I chose the .exe one, the Linux one is not in one file). Then I compressed it into an XZ archive. The size of the recompressed archive was 313.6 MiB. The size of the .exe file is 312.7 MiB. The size of a BZ2 archive was 336 MiB. Not sure if it would be an improvement.

User avatar
Iris
Site Administrator
Posts: 6588
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: DataCpmpression with XZ Utils

Post by Iris » March 23rd, 2012, 12:49 am

I don’t consider file size to be very relevant anymore (at least below 0.5 GiB), except for software packages up for download. As far as I can tell, slow or quota-restricted Internet connections are far more common than the lack of cheap, large storage media.

If there’s a better, widely supported compression format available for use for our release tarballs then the person you need to convince of this is Ivanovic, not the forums.

The other area where a different compression format could be useful in certain circumstances is the network protocol. Currently, all WML transmitted to and from the add-ons and MP servers is streamed through a gzip filter at both endpoints. This is generally good enough for most purposes, but it may fall short of the players’ expectations when dealing with large add-ons from the add-ons server, or large scenarios with loads of WML/large tile maps from the MP server. On the other hand, as far as I know gzip is a balanced solution in terms of CPU usage, which is very important given how our online infrastructure is currently set-up for practical and financial reasons.

Of course, in the add-ons server case in particular, the bulk of the content of the larger add-ons consists of binary files (mainly PNG and Ogg Vorbis files) that already use their own compression techniques and don’t benefit much from an extra layer of encoding. It’s also not clear at this point whether the future add-ons server will continue to use the WML protocol.

Just my two cents.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: DataCpmpression with XZ Utils

Post by MetalKing » March 23rd, 2012, 2:41 pm

I did check XZ with wesnoth-1.10.1.tar:

bzip2 (-9 ?) wesnoth-1.10.1.tar -> 326.1 MB
xz -9 wesnoth-1.10.1.tar -> 301.1 MB
Ergo 25 MB safed with xz-Compressor.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

Nafac
Posts: 2
Joined: March 24th, 2012, 11:00 am

Re: DataCpmpression with XZ Utils

Post by Nafac » March 24th, 2012, 11:46 am

Container format should be used, installing hundreds or even thousands of little images, sound files etc. is a pain.

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: DataCpmpression with XZ Utils

Post by MetalKing » March 25th, 2012, 3:14 pm

Theory about Compression and Media-Data is interesting but XZ DOES compress significantly at least mp3 and at least one Image on my PC. Can anyone confirm that?
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

User avatar
Iris
Site Administrator
Posts: 6588
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: DataCpmpression with XZ Utils

Post by Iris » March 25th, 2012, 3:38 pm

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -Jcf music.tar.xz music/
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music
139M    music
shadowm@reicore:~/src/wesnoth/data/core$
EDIT: since I am fairly sure this will crop up later in some fashion:

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -cf music.tar music/
shadowm@reicore:~/src/wesnoth/data/core$ xz -9 music.tar 
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz 
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: DataCpmpression with XZ Utils

Post by MetalKing » March 25th, 2012, 4:39 pm

shadowmaster wrote:

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -Jcf music.tar.xz music/
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music
139M    music
shadowm@reicore:~/src/wesnoth/data/core$
EDIT: since I am fairly sure this will crop up later in some fashion:

Code: Select all

shadowm@reicore:~/src/wesnoth/data/core$ tar -cf music.tar music/
shadowm@reicore:~/src/wesnoth/data/core$ xz -9 music.tar 
shadowm@reicore:~/src/wesnoth/data/core$ du -sh music.tar.xz 
137M    music.tar.xz
shadowm@reicore:~/src/wesnoth/data/core$

That's almost nothing but for OverALl-Performance of TarBall of Wesnoth the Numbers are fine.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

Post Reply