Sprite Scaler

[Telcontar]

I was actually thinking about a very similar project for Google Summer of Code 2015. My idea was to save the resulting images for later post-processing by a human, but it seems in some cases the results are so good that this may not be necessary.

Do you think that the results could be improved by minor tweaks to the algorithm? For example, I think that anything with larger areas (such as armor) tends to come out well, while faces are too pixelated in the original art to scale up well.

Maybe special cases can be implemented to recognize faces?

[iceiceice]

I was actually thinking about a very similar project for Google Summer of Code 2015. My idea was to save the resulting images for later post-processing by a human, but it seems in some cases the results are so good that this may not be necessary.

Yeah it seems there may have been some sort of confluence actually, shortly after I merged this into 1.13, fabi mentioned some interest in a different approach by Kopf and Lipchinski: http://research.microsoft.com/en-us/um/ ... /pixelart/

That approach is really much more aggressive -- they are trying to automatically rewrite all the pixel art as scalable vector graphics. In fabi's conception of it, it would be a preprocessing step we would take before distributing the game, then when you zoom in in-game, it would try to render these vector graphics images. Thunderstruck remarkably sent an email on the same topic but with different interested parties to the dev-list recently (like a week ago maybe).

I could easily see some purists calling this "a bridge too far", after all for many of our players the pixel art is one of the big attractions, and straight-up vectorization is pretty antithetical to this. I personally have questions about how practical such an approach would be, because it seems pretty clear to me that rendering svg's is going to be a lot slower. Actually I'm pretty sure that our current basic implementation can be sped up quite a bit, because it is really doing an incredible amount of caching, even caching the scaled up versions of sprites. To me that doesn't make so much sense... the scaled up versions take a lot more space and they are very cheap to compute from the originals, so it seems much better to continually recompute the scalings, rather than exhaust the cache and suffer constant cache misses. If we were rendering scaled up SVG's though I don't think any such improvements would be possible and probably we would cache them, since they would be expensive to recompute. There might also be other difficulties, I'm not sure.

For comparison xBRZ is not "vectorizing" the art, it is indeed a traditional "scanning" image filter -- the difference with a traditional scaler is that it looks at a larger neighborhood than usual, i.e. 4 x 4 instead of 2 x 2, and tries to do some pattern matching in there to detect edges. One of the crucial aspects of the whole thing is coming up with a good "color distance function" -- in the xBRZ implementation, pattern detection is basically based on computing distances between these different pixels in each window, and checking the computed distance against a threshold. If we can see that some section of the pixels are close enough to constitute a "region", then when we scale up, the algorithm attempts to preserve that pattern by extending it in a painterly way. So you can think of it as "locally" vectorizing each little window of the image -- the different windows don't really coordinate, so we just have to hope that at the end it all pretty much works out and fits together. And indeed it doesn't really seem to be distinguishable from if the image had been globally vectorized (at least IMO for our sprites).

The main difficulty right now is how to properly extend the color distance function to take alpha into account. For the most part, xBRZ has mainly been used for emulators, the original being MAME. In that case there is no alpha, and you are not trying to scale individual sprites -- you have some emulator of an old console, maybe making 320x200 output, and now you want to rescale that at 60fps to a 1024-768 display. So you will have xBRZ running continually (actually multiple threads of it in parallel doing each frame buffer -- since it is a totally local algorithm you can handle far away rows in parallel. Probably for SVG's you can't.)

I did message Zenju for his thoughts about alpha, he had some insights and he now has uploaded a version 1.1 with alpha channel support added to his color distance function of choice. However I have a competitor to this one, and I am intending to test it against many of our sprites and especially try to fix the minor artifacts I have found. I haven't made any movement for a couple weeks though, been preoccupied with 1.12 release.

I don't have that much insight into how to tweak xBRZ's rules for finding patterns / extending them. It is definitely one of the primary focuses of the algorithm to try to "preserve small features" when scaling, especially eyes. In the case of our sprites, some of the faces turn out pretty well (IMO drakes), and some of them kind of don't (IMO loyalist bowman). I'm actually not sure if that's best addressed by tweaks to the algorithm, or just by tweaks to the base frames of the sprites. (Such tweaks are admittedly unlikely.)

It might be kind of interesting if you can turn it into a machine learning problem somehow, instead of saying "I'm just going to tweak it until I think it looks good" as most people seem to do. For instance if you somehow had a bunch of example sprites drawn at two sizes, and you view the algorithm as attempting to "recover" the large one after it has been scaled down to the small size, then you could try to fit the parameters so that you get the closest fit in the most cases, and see if that gives a general improvement. Ambitiously, you could try to somehow learn new xBRZ-style scaling rules or something by the same means.

Maybe special cases can be implemented to recognize faces?

To be honest, I'm not an artist and I actually feel I have very little insight into pixel art. If you feel there are common patterns you could exploit, and/or simple techniques to give a more natural enlargement of a face then it might very well be a good direction.

It also depends a bit exactly which sprites you want to work on -- wesnoth sprites look pretty different from many of the sprites in games like MAME and PPSSPP and other places that use xBRZ. But if you think you can see a way to make faces look better in many cases, it might attract interest beyond wesnoth.

In any case, if you want to play around with it, the xBRZ code base is well-written and quite readable IMO (there are many tricks for optimization, and it does make use of macros, but this greatly improves readability rather than hindering it as one might expect). I did not have too much problem disabling various aspects of the pattern matching and checking the output to try to get a feel for what the different parts are doing and how it fits together.

Actually, IMO one of the more obvious limitations of the whole thing is that it is much more biased to "round things off" than to draw sharp points. Pixel art tends to be kind of rough, so a lot of rounding is typically warranted. But at least for our sprites, it means that if the sword is too wide, it ends up getting rounded off and looking like a wet noodle, rather than coming to a sharp point. I have no idea how this could be improved at the source code level right now, but at least it seems conceivable to me that improving this could improve some facial features for certain sprites and improve other examples.

[jbreiden]

This brings back memories. I wrote the bilinear interpolation nearly a decade ago, as my one and only contribution to Wesnoth. Your work looks terrific and you are clearly having fun. You don't need my blessing, but you certainly have it. Also, what a crazy world where I can notice this discussion so many years later.

[Telcontar]

I was actually thinking about a very similar project for Google Summer of Code 2015. My idea was to save the resulting images for later post-processing by a human, but it seems in some cases the results are so good that this may not be necessary.

Yeah it seems there may have been some sort of confluence actually, shortly after I merged this into 1.13, fabi mentioned some interest in a different approach by Kopf and Lipchinski: http://research.microsoft.com/en-us/um/ ... /pixelart/

That approach is really much more aggressive -- they are trying to automatically rewrite all the pixel art as scalable vector graphics. In fabi's conception of it, it would be a preprocessing step we would take before distributing the game, then when you zoom in in-game, it would try to render these vector graphics images. Thunderstruck remarkably sent an email on the same topic but with different interested parties to the dev-list recently (like a week ago maybe).

I also saw this paper when it came out, and discussed this as a project idea for GSoC. In my opinion, it could even be possible to process the result by hand (in Inkscape) so we could later have high-quality vector graphics as "master" sprites.

I could easily see some purists calling this "a bridge too far", after all for many of our players the pixel art is one of the big attractions.

I just see this as an artifact of older computers. It's technically trivial to keep copies of the original (current) sprites, so let's think about all the users who like high-res graphics.

I personally have questions about how practical such an approach would be, because it seems pretty clear to me that rendering svg's is going to be a lot slower.

Again I see no big problem. For mobile platforms, the right resolution(s) can be pre-computed (say, x2 and x3), and PCs are fast enough that I don't think it's an issue.
If it is, the sprites can be scaled up once and then cached; memory/disk usage is not a problem for this kind of use.

I did message Zenju for his thoughts about alpha, he had some insights and he now has uploaded a version 1.1 with alpha channel support added to his color distance function of choice. However I have a competitor to this one, and I am intending to test it against many of our sprites and especially try to fix the minor artifacts I have found. I haven't made any movement for a couple weeks though, been preoccupied with 1.12 release.

I don't have that much insight into how to tweak xBRZ's rules for finding patterns / extending them. It is definitely one of the primary focuses of the algorithm to try to "preserve small features" when scaling, especially eyes. In the case of our sprites, some of the faces turn out pretty well (IMO drakes), and some of them kind of don't (IMO loyalist bowman). I'm actually not sure if that's best addressed by tweaks to the algorithm, or just by tweaks to the base frames of the sprites. (Such tweaks are admittedly unlikely.)

It might be kind of interesting if you can turn it into a machine learning problem somehow, instead of saying "I'm just going to tweak it until I think it looks good" as most people seem to do. For instance if you somehow had a bunch of example sprites drawn at two sizes, and you view the algorithm as attempting to "recover" the large one after it has been scaled down to the small size, then you could try to fit the parameters so that you get the closest fit in the most cases, and see if that gives a general improvement. Ambitiously, you could try to somehow learn new xBRZ-style scaling rules or something by the same means.

...

Actually, IMO one of the more obvious limitations of the whole thing is that it is much more biased to "round things off" than to draw sharp points. Pixel art tends to be kind of rough, so a lot of rounding is typically warranted. But at least for our sprites, it means that if the sword is too wide, it ends up getting rounded off and looking like a wet noodle, rather than coming to a sharp point. I have no idea how this could be improved at the source code level right now, but at least it seems conceivable to me that improving this could improve some facial features for certain sprites and improve other examples.

For our use, some kind of machine learning could indeed help. For example, grey usually means swords and sharp edges (except for Trolls). So if we can generate a couple of variants, then we can let users vote on the best one, and use this feedback to teach the computer which version to use in which case. We could even isolate certain sections of an image to focus on specific features. If we have nine quadrants, then the face is usually in the top center one. So algorithms that tend to round off features are good there.

I think this would make for an interesting GSoC project, if we have some people who know enough about image processing and machine learning.

[iceiceice]

I also saw this paper when it came out, and discussed this as a project idea for GSoC. In my opinion, it could even be possible to process the result by hand (in Inkscape) so we could later have high-quality vector graphics as "master" sprites.

This is something you would have to propose to Jetrel and the other artists.

I personally have questions about how practical such an approach would be, because it seems pretty clear to me that rendering svg's is going to be a lot slower.

Again I see no big problem. For mobile platforms, the right resolution(s) can be pre-computed (say, x2 and x3), and PCs are fast enough that I don't think it's an issue.
If it is, the sprites can be scaled up once and then cached; memory/disk usage is not a problem for this kind of use.

Hmm... that's not currently how the game works. Currently the default (1x) has each hex rendered as 72 x 72. When you press + / - zoom, the number of pixels to a hex is increased / decreased by 4. Whenever the user changes the zoom value, the image cache is flushed completely, and all the scalings are recalculated and recached. If there are many unit types in the game, with many animations, this can be costly. For instance, on my 1 year old laptop, if I start a 4p droid game, zoom in to 4x, and press the zoom key during the droid's turn, I get about a 1 second delay, during which I will drop animation frames.

It's probably just that populating the cache is expensive to begin with, and it isn't about the speed to scale things -- that's why I only ask how fast SVG rendering can be done. If it's 1000 times slower than "nearest neighbor scaling" then perhaps it would become the bottleneck, it's hard for me to guess. Also the point is not that we frequently need to rescale the game during droid turns and such, just that we are sort of "at the cusp" of it not really working very well. Also it's hard to overestimate how much slower it is to render things in software... and we do support running wesnoth on extremely old machines.

If you want to drop support for intermediate zoom values, and only support 2x, 3x, 4x, or something, that's something we could contemplate, it was actually requested recently in a feature request here: https://gna.org/bugs/?22929

Probably the whole system could stand to be significantly improved anyways.