Put files into one file

Discussion among members of the development team.

Moderator: Forum Moderators

Eldmannen
Posts: 37
Joined: June 8th, 2005, 2:29 pm

Put files into one file

Post by Eldmannen »

My 0.9.2 Wesnoth folder contains 3922 files and 131 folders!
That is a huge amount of files and folders!

Quake (1, 2 and 3) puts all files into a couple .pak files.
Doom (1 and 2) puts all files into a a .wad file.
Doom 3 puts all files into a couple .pk4 files.
Tibia puts all graphic files into a .spr file.
Command & Conquer also put its files into a .mix file or something.
Half-Life puts all its graphic files into .wad files and some other content in .pakfiles.

Many games do this to avoid having several thousand of files on the disk.
Eldmannen
Posts: 37
Joined: June 8th, 2005, 2:29 pm

Post by Eldmannen »

3041 of those files are under the "images" directory.

I think that put all the .png files into some sort of archive would be a great idea.
User avatar
turin
Lord of the East
Posts: 11662
Joined: January 11th, 2004, 7:17 pm
Location: Texas
Contact:

Post by turin »

Simply put, no.

What purpose would it serve? It would reduce the number of files, yes, but that would make it harder to understand the files. Having them divided make it so you can easily find the data you're looking for and edit it. There are some games I have played with all the files combined, and when I tried to edit of of the configuration files I found it impossible to find the data I wanted to edit.

Combining the images would be even worse than combining the data files, IMHO.
For I am Turin Turambar - Master of Doom, by doom mastered. On permanent Wesbreak. Will not respond to private messages. Sorry!
And I hate stupid people.
The World of Orbivm
scott
Posts: 5248
Joined: May 12th, 2004, 12:35 am
Location: San Pedro, CA

Post by scott »

Yeah, I don't see the value in reducing the number of files or folders. The value is in reducing file sizes. The benefit of the current method is that it allows players to understand, modify, expand, and contribute to the game's inner workings
Hope springs eternal.
Wesnoth acronym guide.
Eldmannen
Posts: 37
Joined: June 8th, 2005, 2:29 pm

Post by Eldmannen »

Players supposed to play the game, not go into thousand of files and directories and snoop and change, thats what is behind the scene is for the developers todo. :arrow:

Every file no matter how small is atleast 4096 bytes because of clusters on the disk. :!:

And the more files on the disk, then the slower the computer goes, the more fragmented the disk become and more slow. :!:

Size: 51,2 MB (53 751 597 bytes)
Size on disk: 100 MB (105 250 816 bytes)

As you can see, it roughly doubles the amount of space needed. :!:

Also games like Quake, Half-Life etc have tons of mods even though they have their files in one file.
Attila
Posts: 243
Joined: January 29th, 2005, 7:17 pm

Post by Attila »

Eldmannen wrote:Players supposed to play the game, not go into thousand of files and directories and snoop and change, thats what is behind the scene is for the developers todo. :arrow:

Every file no matter how small is atleast 4096 bytes because of clusters on the disk. :!:

And the more files on the disk, then the slower the computer goes, the more fragmented the disk become and more slow. :!:

Size: 51,2 MB (53 751 597 bytes)
Size on disk: 100 MB (105 250 816 bytes)

As you can see, it roughly doubles the amount of space needed. :!:

Also games like Quake, Half-Life etc have tons of mods even though they have their files in one file.


There's a difference between games such as Quake, and Half-Life, those games are made to be sold and make a profit. If people could easily tamper with the files inside them and change the game to their advantage, it would quickly defeat the purpose of those games. BfW is open-source allowing everyone to screw with the program and change it to whatever they want to see in it.
scott
Posts: 5248
Joined: May 12th, 2004, 12:35 am
Location: San Pedro, CA

Post by scott »

No one is making players look under the hood, but because they can the project has advanced much quicker. Since open source development sometimes depends on community contributions, this is a good thing.

Some operating systems automatically defrag files as they're used. It's not our fault if you're using inferior software ;)

Could you explain how you came up with the file size calculation?
Hope springs eternal.
Wesnoth acronym guide.
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Post by silene »

This thread is going nowhere. Just to prevent FUD from spreading further: Wesnoth does support zipped files. The only official release is the source tarball, and it contains support for zipped files. It is then up to the maintainers of the unofficial binary packages whether they enable it or not. It is disabled in the Windows binaries, it is enabled in the Debian binaries, for example. If you are not happy with the choice of your respective binary packagers, you are free to compile Wesnoth yourself. You can even contribute your binary package once you have built it.
Eldmannen
Posts: 37
Joined: June 8th, 2005, 2:29 pm

Post by Eldmannen »

scott,
I right-clicked on the "Wesnoth" directory and selected "Properties" in Windows XP.
This summarizes the content of that folder and displays it in the General tab. It tells what type it is, the size of it, the attributes of it (you know read only, etc) and other things.

silene,
Thanks for your informative reply.
Could you please tell that guy who makes the win32 distribution of Wesnoth to use that zip setting next time?
Also, I would like to point out that there is a format called 7-Zip that is open-source and under the GNU LGPL and it provides even better compression than zip. http://www.7-zip.org/
User avatar
turin
Lord of the East
Posts: 11662
Joined: January 11th, 2004, 7:17 pm
Location: Texas
Contact:

Post by turin »

Eldmannen wrote:Could you please tell that guy who makes the win32 distribution of Wesnoth to use that zip setting next time?
Lets make one thing clear: Those who make the binaries are volunteers, under no obligation to the project, and the developers have no hold over them. They can suggest, but not order, as you imply. :)

Still, it might be a good idea to suggest he do that.
Eldmannen wrote:Also, I would like to point out that there is a format called 7-Zip that is open-source and under the GNU LGPL and it provides even better compression than zip. http://www.7-zip.org/
There is also gzip, and b2zip (sp?), etc... there are many formats we could concievably use.
For I am Turin Turambar - Master of Doom, by doom mastered. On permanent Wesbreak. Will not respond to private messages. Sorry!
And I hate stupid people.
The World of Orbivm
Dave
Founding Developer
Posts: 7071
Joined: August 17th, 2003, 5:07 am
Location: Seattle
Contact:

Post by Dave »

Eldmannen wrote: Could you please tell that guy who makes the win32 distribution of Wesnoth to use that zip setting next time?
That would be me. I'm afraid I have no interest in putting everything in one zip file, and as such I won't expend effort on doing so.

If you would like it to happen you can make your own win32 binary.

Also the space each file uses is very filesystem-dependent. Some filesystems also have trouble with very large files.

It's very important to us to keep barrier-to-entry for contributors as low as possible. I've even had second thoughts about making save files smaller and faster by placing them in binary format -- since we did that it has caused substantial confusion for many people.

David
“At Gambling, the deadly sin is to mistake bad play for bad luck.” -- Ian Fleming
torangan
Retired Developer
Posts: 1365
Joined: March 27th, 2004, 12:25 am
Location: Germany

Post by torangan »

While putting everything into a couple of files is surely a bad idea, I got confirmation from graphics contributors like freim that it would actually be usefull to put related images into one .png. E.g. all animations belonging to one unit or all transitions for a terrains. This would not only reduce the number of files but also make them better compressable and artists wouldn't need multiple windows to have those mostly very similiar graphics close together. This surely won't happen pre 1.0 but hopefully soon after.
js138
Posts: 145
Joined: February 23rd, 2005, 7:45 pm
Location: Cambridge
Contact:

Post by js138 »

silene wrote: Wesnoth does support zipped files
It does? I can't see anything obvious in the configure options. How is it enabled?

One of the slow parts of compilation/installation is handling all the small files - as a semi-casual compiler skipping some of that appeals.
silene
Posts: 1109
Joined: August 28th, 2004, 10:02 pm

Post by silene »

js138 wrote:How is it enabled?
It is automatically enabled by the configure script if the zipios library is available on your system. And to get zip files instead of all the small files, there is the "zip-install" Makefile target.
js138 wrote:One of the slow parts of compilation/installation is handling all the small files - as a semi-casual compiler skipping some of that appeals.
You won't avoid it: the files are first installed, then zipped, and finally removed. But I would be glad if somebody submitted a patch to correct this behavior. A patch to add a configure option that disables zip support (although the zipios library is available) would also be gladly accepted.
User avatar
Jetrel
Posts: 7242
Joined: February 23rd, 2004, 3:36 am
Location: Midwest US

Post by Jetrel »

torangan wrote:While putting everything into a couple of files is surely a bad idea, I got confirmation from graphics contributors like freim that it would actually be usefull to put related images into one .png.
Other artists, like me, would hate that. It would completely clash with my workflow - I use the fact of the frames being separate to line them up correctly.

I can use photoshop to show a normal size preview of whatever active file I am working on in a separate window, and can then click between that and several others in the series to preview how the animation looks - OR - more importantly, to align the frames correctly.

I could replicate this same process with other programs like preview, but I could not do this if they were all in the same file.

torangan wrote:E.g. all animations belonging to one unit or all transitions for a terrains. This would not only reduce the number of files but also make them better compressable and artists wouldn't need multiple windows to have those mostly very similiar graphics close together. This surely won't happen pre 1.0 but hopefully soon after.
I hope it never happens, also for a number of technical reasons. I learned myself in coding my own game that making sprite sheets was a very stupid mistake. It created a lot of unnecessary complexity in code that I never should have had in the first place.

The only downside to having separate files in a slight increase in shipping file size, and a much more complex naming tree for the files.
Post Reply