Avoiding storing sprites in 9000 files, how?

Contribute art for mainline Wesnoth.

Moderators: Forum Moderators, Developers

Forum rules
Before posting critique in this forum, you must read the following thread:
User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Avoiding storing sprites in 9000 files, how?

Post by Dugi »

The current game graphics are stored in over 9000 files. So many files is quite annoyingly much, it causes trouble when finding files, diagnosing disk usage,... So, a GSoC (Google Summer of Code) project is planned to resolve this issue.

The way of achieving this preferred by the people I saw on #wesnoth-dev is to make sprite sheets, big pictures containing many sprites. Editing would not be much harder in this case, just identifying the desired sprite between many others. A problem would come when adding new sprites with unusual (not 72x72) resolutions.

I suggested an alternative to put all the sprites into several .zip archives (maybe one archive for one folder) and checking folders with same names if the archive is not found. It would basically need that anybody editing it would have to open the archive in WinRar or something similar (Windows will point to its website if trying to open such an archive without software for that installed; on most Linux distributions, software like this is even preinstalled), use it to unpack it and delete it. This means installing a free program and a few extra clicks before editing. After doing this, all editing would be the same than before.

I wonder which of these would you, artists, prefer.
Last edited by Dugi on March 20th, 2014, 9:20 pm, edited 2 times in total.

SkeletonCrew
Developer
Posts: 787
Joined: March 31st, 2006, 6:55 am

Re: Avoiding storing sprites in 9000 files, how?

Post by SkeletonCrew »

I asked Dugi to post this qustion to the artist to get feedback on the idea.

@Dugi, please explain the workflow for the artist in more detail. Also please make no claims on code
complexity. (We don't even know what will be easier code-wise and this is not the place to discuss that.)

We like to know how the artist feel about using zip archives, before we even start to consider the coding
part. If you like the idea we can look whether it is feasable from a coding point of view.

User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Dugi »

Edited it.

User avatar
zookeeper
WML Wizard
Posts: 9742
Joined: September 11th, 2004, 10:40 pm
Location: Finland

Re: Avoiding storing sprites in 9000 files, how?

Post by zookeeper »

No matter what you're doing with the sprites, having them in an archive would always mean 1) opening the archive (not a big deal, just a double-click on any sane setup), 2) using your archiver to browse the files (likely with no thumbnails, etc), 3) getting the image from the archiver to your image editor/viewer (not trivial at least in 7-Zip unless you just happen to have file associations set up right). And then if you actually edit the file, you need to save it somewhere else and then figure out what it is you want/need to do with it.

Now, if we assume that you for example edited a sprite and want to commit your changes? You have to reconstruct the archive or otherwise replace the file in it with your edited version. You can't view a diff of the image anymore. Everyone else can't see which sprite you changed (except from the commit message) because the whole archive changed, and likewise cannot view a diff to actually see the change conveniently.

I'd call that a usability nightmare.

User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Dugi »

1) opening the archive (not a big deal, just a double-click on any sane setup), 2) using your archiver to browse the files (likely with no thumbnails, etc), 3) getting the image from the archiver to your image editor (not trivial at least in 7-Zip unless you just happen to have file associations set up right) and 4) if you edit the file, you need to save it somewhere else and then figure out what it is you want/need to do with it.
You haven't even read what I wrote. These steps can be easier:
1) using your archiver to unpack the folder
2) delete the archive
3) do whatever you want, without having to search through a big image for a frame that looks almost exactly the same like 5 others
You can't view a diff of the image anymore.
Yes, but imagine a big image consisting of 100 sprites. Look at two big images like this, and try to find what is different. This can be avoided simply by committing them in uncompressed form, and add compressing the sprites into the makefile recipe.

User avatar
Sapient
Inactive Developer
Posts: 4453
Joined: November 26th, 2005, 7:41 am
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Sapient »

For spritesheets: Why not make all the sprites bigger than 72x72?
Also providing utilities to chop up and reassemble the spritesheet?
http://www.wesnoth.org/wiki/User:Sapient... "Looks like your skills saved us again. Uh, well at least, they saved Soarin's apple pie."

User avatar
vultraz
Developer
Posts: 959
Joined: February 7th, 2011, 12:51 pm
Location: Dodging Daleks

Re: Avoiding storing sprites in 9000 files, how?

Post by vultraz »

If I read correctly, your fear is the usability of finding a specific frame in a sheet?

For all the animations would be listed in a specific order, allowing the person editing to see the animation played out in a still line and easily identify the area they need to edit. As a spriter myself, I'd find spritesheets much easier to work with than a zip archive.
Creator of Shadows of Deception (for 1.12) and co-creator of the Era of Chaos (for 1.12/1.13).
SurvivalXtreme rocks!!!
What happens when you get scared half to death...twice?

User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Dugi »

For all the animations would be listed in a specific order, allowing the person editing to see the animation played out in a still line and easily identify the area they need to edit.
You can tell what is attack frame 1, attack frame 2, attack frame 3 etc, but telling a defend animation from an attack animation can be hard in some cases. And I wonder how would you guess that with sprites like Gryphon, Rider that share frames between standing, moving, attacking and defending. Also, units like Master at Arms have multiple attack animations that share some frames. You'll have to check some numbers in the unit's WML and check the coordinates on the picture.

I am under the impression that everybody thinks that the zip archive idea means editing files in an archive all the time. It doesn't, it just means that it will have to be packed before releasing, and unpacked between downloading and editing.

User avatar
GunChleoc
Translator
Posts: 489
Joined: September 28th, 2012, 7:35 am
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by GunChleoc »

How about having a row in a spritesheet for each animation? This will make spritesheets bigger if the same frame is used in different animations, but they would be easier to understand.

I'm not a spriter, just throwing an idea out there.

gfgtdf
Developer
Posts: 1318
Joined: February 10th, 2013, 2:25 pm

Re: Avoiding storing sprites in 9000 files, how?

Post by gfgtdf »

Dugi wrote: The way of achieving this preferred by the people I saw on #wesnoth-dev is to make sprite sheets, big pictures containing many sprites. Editing would not be much harder in this case, just identifying the desired sprite between many others. A problem would come when adding new sprites with unusual (not 72x72) resolutions.

I suggested an alternative to put all the sprites into several .zip archives (maybe one archive for one folder) and checking folders with same names if the archive is not found. It would basically need that anybody editing it would have to open the archive in WinRar or something similar (Windows will point to its website if trying to open such an archive without software for that installed; on most Linux distributions, software like this is even preinstalled), use it to unpack it and delete it. This means installing a free program and a few extra clicks before editing. After doing this, all editing would be the same than before.

I wonder which of these would you, artists, prefer.
first: windows (idk for windows xp or less) can open a zip arcieve without any programm installed.

I like this idea, except i'd prefer using the unzipped version of a file if both exist. I think that's also how other games do it.

If we for exampke make the client pack the images directory or subdirectories in a zip archieve when uploading, the work for artists would be exactly the same as before.
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.

Loci
Posts: 40
Joined: October 28th, 2008, 10:49 pm

Re: Avoiding storing sprites in 9000 files, how?

Post by Loci »

zookeeper wrote:Now, if we assume that you for example edited a sprite and want to commit your changes? You have to reconstruct the archive or otherwise replace the file in it with your edited version. You can't view a diff of the image anymore. Everyone else can't see which sprite you changed (except from the commit message) because the whole archive changed, and likewise cannot view a diff to actually see the change conveniently.
The sprites could easily be stored in the repository in the exact same manner they are now. Just add an extra step to the build process that gathers all the sprites and creates the archives for release. With this setup the archives are a build product, not a versioned resource, and diffs, commits, etc. all function the same as they do now.

Although I don't really edit enough to comment, I prefer having separate files (archived or not), since it makes the process easier to understand for occasional editors. Using spritesheets requires knowledge of conventions (like size limitations, how to shift frames, where to add a new frame, how to index into the spritesheet, etc.). Copying an existing single-frame file and modifying it is, comparatively, quite straightforward, even for someone completely unfamiliar with the project.

One suggestion I would make for an archived approach is to make uncompressed frames override their compressed counterparts. This would allow editors to change one frame without extracting and deleting the entire archive. I believe this is similar to how other mod-able games handle resource archives as well.

User avatar
Coffee
Developer
Posts: 180
Joined: October 12th, 2010, 8:24 pm

Re: Avoiding storing sprites in 9000 files, how?

Post by Coffee »

Just wondering what would be the benefit of zipping all the images?

For the record the thread for the GSoC spritesheets is here and being looked into by Aishiko and mentored by Mordante/SkeletonCrew. I wrote the current [] syntax for 1.11.2+ to make the WML easier for animations that I would expect would be expanded on with spritesheets with a similar end syntax.

Spritesheets done well would make the image files smaller in total size, simplify the WML of complex units, and allow you to work on multiple sprites at once. This would make it easier to work with indexed colored sprites and apply color shifts to whole lines at once. The GPU would load the spritesheet all at once and reference rectangular coordinates to load faster the images to the screen, in the process allowing for (very) minor graphics optimizations. Remember that we are also talking about animated terrain. Multiple spritesheets could be used for a single unit if the dimensions differed or they were completely different units (like if Konrad's sceptre of fire made his spritesheet bigger than 72x72) and mix and matched or overlayed.

These are the advantages I see and don't really see any benefit from zipping multiple unit images into a single file. You would still be able to use individual images as well, as overlays or however you want.

User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Dugi »

@Coffee
The basic benefits are the same as with spritesheets - not too many files thus loading faster, slightly smaller total size. It has further advantages:
1. you don't have to mess up with coordinates (both in unit files and images)
2. you have no particular issues to look up what you want in an archive
3. editing small pictures that are alone in their own files is more comfortable unless recolouring and so on, but that's done rarely
4. porting all core graphics and add-on graphics would be an immense amount of work if done with spritesheets, little work if done with archives
The spritesheets' idea had a thread around here as well, with a poll, and nobody really liked that idea.

If you complain about lack of technical detail, please know I have talked with SkeletonCrew on IRC and he told me to tell as few technical detail as possible because this is an artists' forum, not a coders' forum.

Spixi
Posts: 87
Joined: August 23rd, 2010, 7:22 pm

Re: Avoiding storing sprites in 9000 files, how?

Post by Spixi »

I don't understand your arguments to zip the images. PNG already supports compression and we use wesnoth-optipng to minimize the file size. There is also no performance win if you read an uncopressed zip, tar or cpio archive instead of single files. All modern OSes collect read() calls in a queue and optimize the hard disk access.

Git already compresses files, so a pull won't be faster, but even slower. If you only change a single file in the archive, the whole archive will be changed and get a new version on you next commit.

So I don't think, zip is a good idea.

User avatar
Dugi
Posts: 4955
Joined: July 22nd, 2010, 10:29 am
Location: Carpathian Mountains
Contact:

Re: Avoiding storing sprites in 9000 files, how?

Post by Dugi »

Git compresses files, that's right. Files are loaded quite quickly, that's right. But have you tried copying one file with a size of a megabyte and 100 files with the size of ten kilobytes and compared the speed?

Anyway, whether it's really needed to mash them together or not is irrelevant, it appears that a decision has been taken that the quantity of files will have to be decreased. The problem was the actual way how it would be done. The idea was to make huge images consisting of many smaller images and I found it very impractical compared to the possibility to connect them into archives, keeping the old structure and so on. So the point was connecting them into archives or connecting them into big images, not connecting them or not (that's even more out of my hands and I don't disagree with the main direction there).

Anyway, this topic is already old and irrelevant, Alishiko got that GSoC project accepted and he promised that he'll create scripts that will be able to merge and separate these sprite sheets just like generic archive-managing programs. Maybe a moderator could lock this because this topic.

P.S. So glad that I noticed the typo in the word 'lock' and changing the last sentence from '...moderator could lick this...'.

Locked