LZMA2-compressed savegames

Brainstorm ideas of possible additions to the game. Read this before posting!

Moderators: Forum Moderators, Developers

Forum rules
Before posting a new idea, you must read the following:
Post Reply
Spixi
Posts: 82
Joined: August 23rd, 2010, 7:22 pm

LZMA2-compressed savegames

Post by Spixi » October 15th, 2012, 6:58 pm

Hi, I have seen that ai0867 has merged the file-compressor bzip2 to trunk. Why didn't you use LZMA2 (.XZ)

LZMA2 has a higher compression rate than bzip2, however it needs a little more memory.

The XZ container format is the standard package format in Slackware GNU/Linux and LZMA2 is also used by 7zip - the most popular free file compressor for Windows users (Linux users use XZ utils instead).

I tried to compress the same file with gz, bzip2 and xz and unfortunately bzip2 has the worst compression rate.

mari[1.10]$ du -b save_index*
415828 save_index
23317 save_index.bz2
23002 save_index.gz
15024 save_index.xz

So I would suggest to use LZMA2 instead of Bzip2 (Note: libzma is supported by many platforms. It is public domain, therefore it is free to use by everyone.)

The liblzma Source is obtainable here: http://tukaani.org/xz/
Attachments
save_index.tar
This tarball compares a savegame.
Contents:
- plain uncompressed
- gzipped file
- bzip2'ed file
- xz'ed file
(470 KiB) Downloaded 152 times
Last edited by Spixi on October 16th, 2012, 7:04 pm, edited 1 time in total.

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing » October 16th, 2012, 2:06 pm

I Pro that Matter and refer to my Arguments written here: DataCpmpression with XZ Utils

Small Note: WinRAR is afaik able to extract Files from XZ as well afaik by treating the Archive as 7z.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max » October 16th, 2012, 8:32 pm

bzip2 came more or less for free as part of boost libraries. i'm pretty sure integrating lzma2 would be quite some effort.

Spixi
Posts: 82
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi » October 17th, 2012, 9:59 pm

An implementation and integration of LZMA2 compression into GIMP only took 238 lines see here I don't think.that there is much efford ...

User avatar
ancestral
Developer
Posts: 1108
Joined: August 1st, 2006, 5:29 am
Location: Motion City

Re: LZMA2-compressed savegames

Post by ancestral » October 18th, 2012, 12:46 am

You can certainly submit a Feature Request.

Meanwhile, patches are always welcome.
Wesnoth BestiaryPREVIEW IT HERE )
Unit tree and stat browser
CanvasPREVIEW IT HERE )
Exp. map viewer

Spixi
Posts: 82
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi » October 19th, 2012, 6:25 pm

I filed a feature request in the ticket system of Boost here. Maybe - if this would be implemented - it could be used in a later release of BfW.

Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max » October 19th, 2012, 8:08 pm

Spixi wrote:An implementation and integration of LZMA2 compression into GIMP only took 238 lines see here I don't think.that there is much efford ...
have a closer look. this commit relies on an external library and a couple of header files.

Spixi
Posts: 82
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi » October 20th, 2012, 12:35 pm

Maybe we could use the Boost Lzma implementation from fusecompress ...

Max
Posts: 1449
Joined: April 13th, 2008, 12:41 am

Re: LZMA2-compressed savegames

Post by Max » October 20th, 2012, 2:32 pm

again - they are just using an external library.

that means this library would have to exist on all available platforms, work for all compilers. additional external dependencies put a heavy burden on maintainers and developers.

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing » October 27th, 2012, 3:42 pm

Spixi wrote:Maybe we could use the Boost Lzma implementation from fusecompress ...
Is their LZMA=LZMA2 ?
Max wrote:again - they are just using an external library.

that means this library would have to exist on all available platforms, work for all compilers. additional external dependencies put a heavy burden on maintainers and developers.
What Service does the external Lib provide? The Compressor should't do a Lot other than read and write a File and do internal Work with other Libs. I wonder ..

Is XZ implemented in C or C++? I prefer the OOP Way but Wrap is sweet like Sugar not Food.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

User avatar
shadowm
Site Administrator
Posts: 6573
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: LZMA2-compressed savegames

Post by shadowm » October 27th, 2012, 6:53 pm

Whether it is using C or C++ is not terribly relevant since C++ code can call C code just fine most of the time, and that is in fact how a lot of functionality is implemented (besides calls to system services). For example, Wesnoth uses a bundled version of the Lua interpreter, which is all done in C although it is built using a C++ compiler for compatibility reasons.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.

Spixi
Posts: 82
Joined: August 23rd, 2010, 7:22 pm

Re: LZMA2-compressed savegames

Post by Spixi » October 27th, 2012, 10:44 pm

liblzma is a small file compression library. You can check for it in a configure script and only compile lzma-relating code if the library is found (e. g. with a define like HAVE_LZMA).

The main difference between LZMA and LZMA2 is, that the latter handles uncompressable files (for example some megabytes from /dev/urandom) better.

The original LZMA2 implementation used in XZ Utils is written in C. The LZMA SDK from 7-Zip also comes with C++ code

User avatar
MetalKing
Posts: 197
Joined: July 8th, 2011, 11:34 am

Re: LZMA2-compressed savegames

Post by MetalKing » November 14th, 2012, 3:53 pm

Are there any significant Constraits for Integration? I don't see any Reason against using the better Compressor and don't expect Problems with Integration. If the necessary Effort is lesser than writing a Posting I won't say anything about that Matter.
"Sir! We are surrounded by our enemies!" - "Excellent ! We can attack in every direction!"
"Make everything as simple as possible, but not simpler." -- Albert Einstein
No Source - No Binary - No Trust!
Map Wesnoth Springs - The great War [200x120],Player=9

Post Reply