Translating images?

[caslav.ilic]

Was it at any time considered to promote localization of images? By this I mean simply allowing the technical possibility of using a localized image should some translation team produce it, and providing somewhere the "sources" of the images, like .xcf files and such (speaking of which, what's the relation between image sources and GPL?) Good candidates for localization are BfW logo and maps.

From a quick look, supporting this coding-wise seems trivial to the point of non-existence: just underscore image file paths in .cfg files, and use gettext calls in sources. And this only on specific images upon first team coming up with that particular image localized, no need to do any in-front marking.

For example, in data/campaigns/Heir_To_The_Throne/scenarios/07_Crossroads.cfg, there are few lines like:

Code: Select all

background="maps/wesnoth-httt.jpg"

which would become:

Code: Select all

background=_"maps/wesnoth-httt.jpg"

and translators could do in the PO file:

Code: Select all

msgid "maps/wesnoth-httt.jpg"
msgstr "maps/wesnoth-httt-ll.jpg"

i.e. the ll-localized image file wesnoth-httt-ll.jpg residing in the same folder as original wesnoth-httt.jpg.

Regarding the policy of accepting localized images, I guess it would be reasonable that the incumbent art-masters check them for proper quality, give improvement advice, and final thumbs up (as is usual for any art).

I don't think this would be much extra burden on organization, as there are only handfull of images obvious for localization, and I guess the translation teams are not that well staffed with artists :) I only pose this proposal because, on my language side, one person commented that he would like to give it a shot if he had access to corresponding image sources.

[Viliam]

Two problems with localization of images:

1) MBs growing quickly...

2) another technology necessary for translators


If localication is voluntary (as it seems), then (2) is not a problem. It would be nice to have an estimate about the growth in MB, if the logos and maps were provided in e.g. 15 languages... is it a small change, or a dramatic one?

[caslav.ilic]

1) MBs growing quickly...

The images/misc/logo.png plus the files in images/maps are in total just under one meg. So, for 15 languages, eh, heh... :)

This problem didn't cross my mind, but it shouldn't be a showstopper. The localized images can be distributed separate from the game proper on non-Linux platforms once they get too big. On Linux, the distributions are already chopping Wesnoth into smaller pieces, at least Debian and Ubuntu have core, data, music, and one package per each campaign.

If localication is voluntary (as it seems), then (2) is not a problem. [...]

*Chusslove quickly slams the door to his translation slave-pit*

[caslav.ilic]

In the best personal tradition of never letting go, I've prepared another round on this topic :)

First, a bit more on the need of localized images. On the second look, it's not only about the in-game content, but also the Help pages, where there are several screenshots, and the new manual, which likewise needs localized images. Having all these non-localized is a bit annoying, against the usual practice of documentation localization. And these images are trivial to produce, just taking screenshots, unlike perhaps in-game content.

Regarding technical implementation, I've reduced it to +15 lines to one source file, no need to do anything else -- none of the underscore-marking as above, nor the need to request it, etc. (Hurrah for the Wesnoth programmer who centralized image fetching into one function :)

For the installation part, no changes to the build system needed. To the image installed as /.../wesnoth/.../image.png, the translated counterpart should be at /.../wesnoth/.../l10n/lang/image.png, and the code will pick it up automatically if present (that's what those extra lines of code do). This means that the structure of distribution is also the same, with extra l10n subdirs where applicable.

The size is the problem, granted. I've looked up all the images containing English text, and added them to ~6 megs total. This means chopping up the distribution as soon as several teams start providing localized images. But maintaining such split shouldn't be an issue, due to the simple build system requirements: just a parallel tree of the same structure as the original, but with those l10n subdirs. E.g.

Code: Select all

branches/1.4/
    data/
        ...
            images/
                maps/
                    ...

branches/1.4/l10n-extras/
    data/
        ...
            images/
                maps/
                    l10n/
                        lang1/
                            ...
                        lang2/
                            ...
                        ...

This makes it very easy to chop the l10n-extras for packaging, just collecting all */l10n/langx/ subpaths for the langx package.

The mandatory proof-of-concept screenshot follows:

[caslav.ilic]

For the moment I considered lack of feedback to my latest proposal a bit pusling, but I think I know what's going on :)

On one hand side, while l10n packages as proposed above might be simple by themselves, going from a single to more packages for Wesnoth's core distribution is an organizational bump, no doubt there. Therefore, it'd be properly worth it if a number of l10n teams were to commit right now to using the possibility of image localization. On the other hand, given the nature of contributions to free software endeavors, mass blind committing to a novelty is not really to be expected. The possibility needs to exist, and then people would jump on the boat along the way.

A classical drake and egg problem.

So, frankly, I posed myself this question: how to further improve the implementation, so that it is not unreasonable to accept the whole system for the sake of only one language at start. Following the implementation of my previous post, the only obstacle to this that I see is the size of localized data, 6 MB/language, and the ensuing packaging calamity, etc. Please state any other problem that you may see, if for the sake of exercise :)

And I'm writing this, of course, because I think I got the upper hand to the data size problem. Looking more carefully, it turns out that when the maps are excluded, the total of to-localize images amounts to 600 kB. Furthermore, only very small parts of map images (i.e. some of the names) need localization. So here's the trick: aside from the full localized images, make the code also accpet overlay images to the original, and compose full images at runtime.

For example, the main menu map is 480 kB, but the overlay for my language (which has every name translated) is only 30 kB (attached). Using overlays for maps, and full localized images for other stuff, I estimate ~700 kB/language. To stretch it a bit, it would take 15 languages localizing images in earnest to up the monolithic distro size from current 140 MB to 150 MB. I think this is a reasonable proposition.

As for the implementation, albeit of course not so few lines as before, it is still quite simple and compartmentalized to a single file. I'm attaching the patch, just for reference. Quick overview: if the original image is .../foo.png, first there is a check for full localized image at .../l10n/lang/foo.png, and if not found, for the overlay image .../l10n/lang/foo--ovr.png; if an overlay is found, the localized image is composed in user's data folder, and a path to it returned.

More amicable now? :)

[ivanovic]

There is one *huge* problem regarding images and having "translated alternatives":

How to keep them up to date and determine when they are outdated? There is no simple way to mark an existing image as outdated. This is a very valid problem when it comes to the ingame help stuff. Beside this: The single "town name overlays" like they are used in the maps of trow would be no too big problem, the image path would be marked translatable and that's it. The problem is that this scheme is not used everywhere and without an existing overlay that is just replaced, it will be *hell* to have things done.

Regarding packaging:
Since I do create the tarball and such I do not want to split it in parts. The reason is simple: it soon becomes a real maintenance hell. I don't know what you have counted for your numbers, so I won't comment on the exact increase due to added translated images. But I think it will be >=1MB per translation with completely translated images.

The main problem I still see and which would require a really good solution is keeping the images in sync with the main once, trying to sort this out by hand is not really possible. We had something like this in the times of the old manual, where each translated version was *extremely* outdated compared to the original version since it was never updated after the initial translation.

[caslav.ilic]

[...] I don't know what you have counted for your numbers, so I won't comment on the exact increase due to added translated images. But I think it will be >=1MB per translation with completely translated images.

I went through all the images found in branches/1.4 (skipping obvious nothing-to-localize categories, like unit frames, terrain, etc.), eyeballing them for text in English. That summed to grand total of 6 megs full, or reduced to ~700 kB using map overlays, as reported above.

The single "town name overlays" like they are used in the maps of trow would be no too big problem, the image path would be marked translatable and that's it. [...]

This was my thought too in the very first post, but turns out it is not needed, due to file-lookup code being centralized. With the above patch, any localized image in proper position will be automatically picked up.

The main problem I still see and which would require a really good solution is keeping the images in sync with the main once, trying to sort this out by hand is not really possible. [...]

Ah, yes, that one too :)

The system with l10n/lang/ subdirectories is actually exactly like the one I've implemented for KDE4, and naturally there was also the question of maintaining the sync. So I wrote a script that will keep track of localized resources organized in this fashion ("lbundles"), described here; of the "out-of-source" and "in-source" cases mentioned, Wesnoth's is "in-source", the easier.

To make it short, at top level (per branch) we keep a file name l10n-spec, with this content:

Code: Select all

# l10n-spec for Wesnoth (in-source lbundles).
source-vcs = svn
languages = aa bb ... zz

and the designated operator runs from time to time:

Code: Select all

$ lbundle-check.py $WESSVN/branches/1.4/

which will in first run create, and in later runs update the top level (per branch) file called l10n-track, with the following content:

Code: Select all

# Do not edit manually, except to remove complete lines.

# aa
ok        ¦data/.../l10n/aa/mainmenu.png¦  37c4a7...8fe3a  725322
ok        ¦data/.../l10n/aa/logo.png¦      2dc1da...fe2ea  755647
...

# bb
ok        ¦data/.../l10n/bb/mainmenu.png¦  37c4a7...8fe3a  734567
ok        ¦data/.../l10n/bb/logo.png¦      2dc1da...fe2ea  756680
...

...

Each line contains, in order: the state, localized file path, the original file checksum, the original file revision. When an original file changes, next run of lbundle-check.py will see different checksum and set the state of all corresponding localized files in l10n-track to fuzzy; if the original is deleted/renamed, the state will be obsolete. Translators can then compare the new and old versions of the original file, by the supplied revision number for which the cheksum was valid; when an localized file is updated, its line is removed from l10n-track, and next run of lbundle-check.py will readd it as ok.

I only have to update the script to recognize overlay images (*--ovr.*), but that's a trifle.

Another easy addition to the script could be that it not only sets the state in l10n-track, but also rename the localized files themselves, adding e.g. *~fuzzy* and *~obsolete* markers. That way it cannot happen that they appear at runtime unless manually corrected (like fuzzy flag in POs). When the updated files from translators are put into the tree, the script would remove the *~fuzzy*/*~obsolete* versions, and set ok state (this instead of manual removal of the lines from l10n-track as needed right now).

Altogether this would imply that the operator of lbundle-check.py just runs it from time to time and commits any changes made in the tree, nothing more, while translators send their updates along the way. Pretty much the same as for the PO files and merging with templates. It probably goes without saying that I volunteer as the operator :) Both of the lbundle-check.py tracker, and as translators' reception point for images.

[VS]

Well, I am not opposed to the idea per se, but... so far the whole l10n business was about text, which was easy to handle on the "dummy" side using poedit (maintainers aside). Now where am I going to find someone to make graphics? If you manage to ease rendering these overlay labels from text, I would say it's worth trying on my side, at least to see how it looks. I am not talking about some auto-renderer, having the font letters available for copy/paste could be enough.

[tsr]

I think it would be possible to integrate ImageMagick into the release cycle so that the images are actually translated as strings and the release-manager 'just' runs a (few) IM scripts to generate the actual images that are to be shipped.

That would also solve a few problems about outdated translations, just do it like you do with normal strings. 'All' that is needed is a new textdomain (wesnoth-core-images) and integration with IM (of course that would probably invalidate the images for trunk, but I think that's a very small drawback)

/tsr - I have some experience with ImageMagick and can offer some help

[caslav.ilic]

[...] so far the whole l10n business was about text, which was easy to handle on the "dummy" side using poedit (maintainers aside). Now where am I going to find someone to make graphics?

You can look at this from several viewpoints.

The most basic would be, if you can't/won't/whatever localize images, you're no worse off than before. The image localization as proposed is a transparent additional feature, for those who care about it. An anyone is free to jump into the boat at any later point.

But the previous paragraphs is kind of cold thought :) Instead, when the implementation is approved, I intend to start bothering artists for assistance. Take the logo (the shield with "The Battle..." superimposed) for example. We should ask artists for the "source" of the image, which would be far more editing-friendly. They could also list the steps to get the same letterface effect for our texts; or, if the originator of the image needs only minutes to do it (I've no feeling, really), he could possibly make localized logos himself given texts from translators in some weeks -- if we say pretty please :)

Finally, if it goes awry for a particular image (the originating artist took off, no image source to be found), then one able translator -- or another friendly artist for that matter -- could spend some time retouching the image, but make it so that other teams can then much more quickly add their stuff. I did the map on the screenshot above in such a way: spent some time clearing the original text, placing labels by layouts, etc. etc. but now it's a matter of minutes for someone else to grasp what's needed in Gimp and then spend, say, half an hour replacing text and tuning its position and overlay cut.

If you manage to ease rendering these overlay labels from text, I would say it's worth trying on my side [...]

The solutions rendering localized images out of text are not generally possible, in my opinion. If you consider the map from above, other languages will have different lengths of text pieces, and then these would awkwardly overlap map features, so some eyeball repositioning is necessary.

One should also bear in mind that currently there is total of 30 images to localize, about half of which are screenshots. Designing necessarily complex generalized solutions for the sake of 15-20 images would count as major overengineering in my book. Instead, I more have in mind case-by-case wiki articles, e.g. "How to make localized map overlay" with list of images and their nicely tuned sources (making the process almost semi-automatic).

I think it would be possible to integrate ImageMagick into the release cycle so that the images are actually translated as strings [...] That would also solve a few problems about outdated translations, just do it like you do with normal strings. [...]

One thing is that I mentioned above, some hand-tuning should usually be necessary. E.g. text can be of any length almost anywhere in Wesnoth's runtime displays, but the text on images is heavily dependent on other elements of the image. There are also screenshots, which cannot be automatized in this way.

Then, about keeping sync, an automatic rendering system necessarily means that the artists should start thinking about localization too (did I leave enough space here? should this overlap stretch? what's that command so that I can visualize what it looks like? etc.) Kind a furball.

Regardless, of course as the time passes we could come up with some fancy automations, but I'd start small :)

[caslav.ilic]

How to keep them up to date and determine when they are outdated? [...]

I've upped the lbundle-check.py as I had propossed above, to mangle filenames too, avoiding outdated content. Here's a tentative session transcript:

Code: Select all

$ cd $WESSVN/branches/1.4
$
$ # someone sent me the logo.png and mainmenu.png images for sr
$ # let me copy it to proper locations
$ find -iname logo.png | grep -v l10n
./images/misc/logo.png
$ mkdir -p ./images/misc/l10n/sr
$ cp $TMP_UNPACK/logo.png ./images/misc/l10n/sr/
$ find -iname mainmenu.png | grep -v l10n
./data/core/images/maps/mainmenu.png
$ mkdir -p ./data/core/images/maps/l10n/sr
$ cp $TMP_UNPACK/mainmenu.png ./data/core/images/maps/l10n/sr/
$
$ # now I let loose the tracker
$ lbundle-check.py
svn add ./data/core/images/maps/l10n/sr
A         data/core/images/maps/l10n/sr
A  (bin)  data/core/images/maps/l10n/sr/mainmenu.png
svn add ./images/misc/l10n/sr
A         images/misc/l10n/sr
A  (bin)  images/misc/l10n/sr/logo.png
A      l10n-track
--------------------
New 'ok': 2
  ./data/core/images/maps/l10n/sr/mainmenu.png
  ./images/misc/l10n/sr/logo.png
$
$ # lemme check the track file
$ grep l10n/sr l10n-track
ok        ¦data/core/images/maps/l10n/sr/mainmenu.png¦  6ea0bd5fa077bc56bed2384db1efdf83  23122
ok        ¦images/misc/l10n/sr/logo.png¦  53e6ff669af1e6d14e3f157717744026  19993
$
$ svn status
A      images/misc/l10n/sr
A      images/misc/l10n/sr/logo.png
A      data/core/images/maps/l10n/sr
A      data/core/images/maps/l10n/sr/mainmenu.png
M      l10n-track
$
$ svn ci -m "Update l10n images."
$
$ # a day passes...
$ # ...an original image, images/misc/logo.png, changes...
$ # a day passes...
$
$ # stop playing Wesnoth, run the tracker, dammnit
$ lbundle-check.py
svn mv ./images/misc/l10n/sr/logo.png ./images/misc/l10n/sr/logo~fuzzy.png
D      images/misc/l10n/sr/logo.png
A      images/misc/l10n/sr/logo~fuzzy.png
M      l10n-track
--------------------
New 'fuzzy': 1
  ./images/misc/l10n/sr/logo~fuzzy.png
$
$ grep l10n/sr l10n-track
ok        ¦data/core/images/maps/l10n/sr/mainmenu.png¦  6ea0bd5fa077bc56bed2384db1efdf83  23122
fuzzy     ¦images/misc/l10n/sr/logo~fuzzy.png¦  53e6ff669af1e6d14e3f157717744026  19993
$
$ svn status
D      images/misc/l10n/sr/logo.png
A  +   images/misc/l10n/sr/logo~fuzzy.png
M      l10n-track
$
$ svn ci -m "Update l10n images."
$
$ # a day passes...
$ # a day passes...
$ # a day passes...
$
$ # I got updated logo.png from sr translator
$ cp $TMP_UNPACK/logo.png ./images/misc/l10n/sr/
$ ls ./images/misc/l10n/sr/
logo~fuzzy.png  logo.png
$ lbundle-check.py
svn rm ./images/misc/l10n/sr/logo~fuzzy.png
D      images/misc/l10n/sr/logo~fuzzy.png
svn add ./images/misc/l10n/sr/logo.png
A      images/misc/l10n/sr/logo.png
M      l10n-track
--------------------
New 'ok': 1
  ./images/misc/l10n/sr/logo.png
$
$ grep l10n/sr l10n-track
ok        ¦data/core/images/maps/l10n/sr/mainmenu.png¦  6ea0bd5fa077bc56bed2384db1efdf83  23122
ok        ¦images/misc/l10n/sr/logo.png¦  fcdc4697e57768204d639affc8e21b47  25413
$
$ svn status
A  +   images/misc/l10n/sr/logo.png
D      images/misc/l10n/sr/logo~fuzzy.png
M      l10n-track
$
$ svn ci -m "Update l10n images."
$

I.e. the operator is just routing newly received files to proper locations (as with POs), running the tracker (as merging POs), and committing the changes that it makes.

[ivanovic]

Short question:

Why should we not head into the direction that was taken for Trow? There the map itself and the "tag overlays" are seperate images. Doing things this way we would "only" end with "font-images" needed with the text on it and the translator would just enter an own path in place. Still the infrastructure would be needed to keep stuff in sync with mainline, yes, but no "real" work on the code side needed.

[Iris]

Why should we not head into the direction that was taken for Trow? There the map itself and the "tag overlays" are seperate images. Doing things this way we would "only" end with "font-images" needed with the text on it and the translator would just enter an own path in place. Still the infrastructure would be needed to keep stuff in sync with mainline, yes, but no "real" work on the code side needed.

False. Many parts of the GUI don't accept arbitrary image overlays like the Story WML system does.

[ivanovic]

False. Many parts of the GUI don't accept arbitrary image overlays like the Story WML system does.

But the only place where those maps are used anyway is on the story screens. All other images are screenshots which can not easily be "emulated" in a decent way and are used in the ingame help or the manual. We are mainly talking about those drawn map images and I am rather confident that it will be possible to have an image path in a translateable string, an exchangeable image path that is.

[caslav.ilic]

Why should we not head into the direction that was taken for Trow? There the map itself and the "tag overlays" are seperate images. [...] Still the infrastructure would be needed to keep stuff in sync with mainline [...]

Obviously, if the overlays are fitting and the artists/programmers willing to cooperate, no problem. But, sometimes the map aesthetics will just not permit such action (e.g. the main menu map), or it could be hard to herd all the people needed to cooperate on it.

However, even if this approach is pressed, the localized stuff must still be loaded at runtime, and as you note the syncing is still in order; thus, the mechanisms I propose above are still needed. What becomes easier is the creation of localized stuff, but that is independent of the loading/syncing, and can be further pondered about along the way.

And while maps are one resource, there are also the screenshots.

Also, we shouldn't shy from code additions, given that the patch I attached is not that voluminous. The alternative, using _"..." for images, is much harder to later keep in check.