Archiving/Packaging Resources (Resource Management)

[governor]

What it is:

Moving resource files to archived files rather than individual files in sub directories. This would include all UMC, all UMC could be its own archived file (like quake 3 and their .pk3 files).

Why it is necessary:

Currently a number of issues arise due to quantity of files. Long copy, move and delete times. Also, none of these files are compressed (except for their native format compression), which means BfW could do better. As a quick proof consider the following:

Version Wesnoth 1.8.0
$WESNOTH_DIR/data: 8688 files, 403 folders, 249MB. ~5 min copy time.
Compressed (normal compression using 7 zip), 1 file, 234MB. ~20 sec copy time.

Also remember that not all users are developers.

Considerations

Expected Opposition (#1): Developers wont be able to easily edit files and develop for wesnoth.
Solution: Design the game to load either the resource packages or use the original directory structure. I.E. content developers could extract the files to the wesnoth root dir and not have to repackage them since wesnoth loads them instead of the packaged resources.

Expected Opposition (#2): Increased time to load resources.
Validity: Since most games use this I don't think this is a valid issue. Expected to perhaps add a few seconds.

Expected Opposition (#3): No real gain in compression.
Validity: True, but additional compression would be small benefit.

Expected Opposition (#4): Might add another dependency.
Validity: This is a turn off.
Solution: Use zlib (a current wesnoth dependency) to accomplish this. Ex. http://gpwiki.org/index.php/SDL:Tutoria ... s_and_zlib

Of course, some utility for packaging the files would also have to be supplied for content publishers.

Regards,
gov

[ancestral]

Actually, I don't think this would be a radical change, and maybe by compressing resources it would save quite a bit of space.

Essentially, just offer people to gzip their add-ons (or do it automatically). Load times would increase, however.

[Atz]

Version Wesnoth 1.8.0
$WESNOTH_DIR/data: 8688 files, 403 folders, 249MB. ~5 min copy time.
Compressed (normal compression using 7 zip), 1 file, 234MB. ~20 sec copy time.

Also remember that not all users are developers.

...

Expected Opposition (#2): Increased time to load resources.
Validity: Since most games use this I don't think this is a valid issue. Expected to perhaps add a few seconds.

How frequently does the average player copy huge slabs of data like that? I would much rather it take five minutes to copy all the data than take an extra five seconds to load every time I want to play a game. I can't even remember the last time I copied anything like that. I think it may have been when I went from an earlier version of 1.6 to the latest version a couple of years ago? And that was just my userdata folder, which wasn't anywhere near as large as the core data.

I'm just not seeing who this is supposed to benefit. Players will be annoyed by the increased loading times. Developers will be annoyed by having to compress and decompress, and by increased loading times every time they need to start up Wesnoth to test something. The only real benefit I can see is that add-ons may be marginally faster to download since filesize will be a little smaller, but most of them aren't that large anyway. So my questions are:

1. Who does decreasing copy and move time actually benefit? Why do we need to do this?
2. Why is this benefit important enough to be worth substantially increasing load times for everyone, inconveniencing developers, and other issues?

[pauxlo]

Depending on the speed of the hard drive, this may even increase the load speed, since we don't need to load a lot of small files, instead only a couple of bigger ones (and at all even less, since compressed).

But this would have to be tested.

[governor]

1. Who does decreasing copy and move time actually benefit? Why do we need to do this?

Players shouldn't notice a change in load time (if there even is one). Reasons for the change are explained in OP, other benefits like decreased install time were omitted - valid for those of us who like to keep up to date but need to use installation binaries, and who like to backup their game before changing values. You read only what you wanted to read. Did you notice a change in load times when save games moved to compressed files? Of course not. In fact, I could argue that times improved since there is less read/write to disk.

2. Why is this benefit important enough to be worth substantially increasing load times for everyone, inconveniencing developers, and other issues?

The inconvenience to the devs depends on implementation. It could be negligible or non-existent. Devs already have the core files on their pcs using svn. Also, UMC content devs archiving their content before publishing will save bandwidth, another potential benefit. Heck, the archiving could be handled by a publishing mechanism thereby removing any UMC content developer inconvenience - they wont even realize that its happening.
Load times wont substantially increase.
Other issues? I have yet to see a valid issue in your response.

Hopefully this post clarifies how the idea and its implementation are separate elements. This is not a radical idea.

[Atz]

1. Who does decreasing copy and move time actually benefit? Why do we need to do this?

Players shouldn't notice a change in load time (if there even is one).

Expected Opposition (#2): Increased time to load resources.
Validity: Since most games use this I don't think this is a valid issue. Expected to perhaps add a few seconds.

Your OP suggested you expected an increase in load times of several seconds, did it not? If that's not an issue that's great, but it seems like you've given conflicting information.

Personally it seems like the influence on compression on loading times would depend on how it was implemented, the specs of the computer you're using, and exactly what is being loaded. It is possible that load times might be decreased for some users and increased for others. However, I am not really an expert on this.

Reasons for the change are explained in OP, other benefits like decreased install time were omitted - valid for those of us who like to keep up to date but need to use installation binaries, and who like to backup their game before changing values.

The reason given was, and I quote: "Currently a number of issues arise due to quantity of files. Long copy, move and delete times. Also, none of these files are compressed (except for their native format compression), which means BfW could do better." You didn't explain why long copy, move and delete times were actually problematic, or what you're doing that long copy times are annoying you. I've never found them a problem so I wasn't really sure where you were coming from on that.

The inconvenience to the devs depends on implementation. It could be negligible or non-existent. Devs already have the core files on their pcs using svn. Also, UMC content devs archiving their content before publishing will save bandwidth, another potential benefit. Heck, the archiving could be handled by a publishing mechanism thereby removing any UMC content developer inconvenience - they wont even realize that its happening.

Well, if UMC devs are borrowing files from elsewhere they'd have to decompress those, but that shouldn't really be a problem since it only has to be done once. But basically we need to be careful of how it's implemented and keep in mind that it could impact UMC devs, is what I'm saying.

[governor]

Found an implementation that covers what I am talking about perfectly:

http://www.kekkai.org/roger/sdl/rwops/rwops.html

Code: Select all

 /* zzip_open does some magic here - if there is a directory called
     "penguin" and there is a "penguin.bmp" there, zzip_read (which
     we'll use later) will act just like stdio's read.  If, however,
     there is a penguins.zip that has a penguin.bmp in it, that will be
     read.  Keep in mind that zzip will prefer real files to files in
     .zip archives if there's a conflict. */

[silene]

Moving resource files to archived files rather than individual files in sub directories.

Note that most (if not all) developers agree that it's the way to go. And it's nothing new; we were already talking about that five years ago. So just posting about it won't help. What would help is that someone actually implements it. It won't magically happen.