A proposal for replacing the default CJK font

[CloudiDust]

Dear Wesnothians:

Greetings!

I am CloudiDust, maintainer of Simplified Chinese translations.

Here's my proposal for replacing the default CJK font in wesnoth.

--> Objective:

Improve the rendering quality of CJK (Chinese, Japanese, Korean) characters in wesnoth.

--> Current Status:

The default CJK font in wesnoth is now 'WenQuanYi ZenHei' (wqy-zenhei), which has the following shortcomings:

1. It cannot be used to render italic characters when used with other fonts.*
2. It cannot provide proper glyphs for certain Japanese characters.
3. It cannot render Korean properly, even the regular ones.
4. The font file is 12.6 MB, which is large.

* If I remove everything but wqy-zenhei from the two 'order' entries in font.cfg, italics can be rendered. And I have no idea why. As far as I know, this seems to be a windows specific problem, and 'pango under windows' being a host of glitches seems no news (Oops).

As a result, CJK players, especially the windows guys, are required to manually configure the game for a proper display. Usually this means replacing wqy-zenhei with another font, thus completely defeating the reason to include wqy-zenhei in the first place. Moreover, this process is hard for normal players, isn't something they should be forced to do, and they are likely to just abandon the game. Definitely not what we would like to see.

--> Previous Proposals:

It has been proposed by Clearpotion (http://forums.wesnoth.org/viewtopic.php?f=4&t=32115) to include TakaoPGothic as the default Japanese font. Chinese and Korean issues are not addressed.

There are some concerns about Takao fonts, that their license, the IPA Font License, is not GPL compatible. As a result wesnoth may not be able to distribute TakaoPGothic in the source tarball. As far as I know, this is a grey area, like many other GPL related issues.

So,

--> My Proposal:

#1. Replace wqy-zenhei with Google Android's 'Droid Sans Fallback' (DSF) and 'Droid Sans Japanese' (DSJ).
#2. Remove the obsoleted entries in data/hardwired/fonts.cfg.

--> Reasons:

My reasons for #1:

1) They can be used to render italic characters unconditionally.
2) DSJ can provide Japanese-specfic glyphs over Chinese ones offered in DSF.
3) The Korean characters are not 'broken' with these two fonts.
4) The total size of the two is 6.17 MB, only half of wqy-zenhei.

Compared to the other possible combination (Takao + DSF), DSJ + DSF has the following advantages:

1) Droid fonts are under Apache License 2.0, which is GPL compatible, says FSF.
2) Droid fonts provides a unified style for CJK.
3) Takao is 5.94 MB, while DSJ is only 1.1 MB as it can fallback to DSF.

My reasons for #2:

The obsoleted entries are for wqy-zenhei-gb2312 and sazanami-gothic. The former is a stripped version of wqy-zenhei, while the latter is only for Japanese and cannot provide italics when used with other fonts either.

--> Impact If Accepted:

1. Better CJK character rendering overall and support for italics.
2. fonts/wqy-zenhei.ttc would be replaced by DroidSanFallbackFull.ttf and DroidSansJapanese.ttf.*
3. data/hardwired/fonts.cfg would be modified accordingly.
4. The license of the Droid fonts (Apache License 2.0) would be included in wesnoth/fonts/COPYING.
5. Translators would have to 'retranslate' (reorder, actually) the two font related messages in textdomain 'wesnoth'.
6. If a language other than CJK depends on glyphs only available in wqy-zenhei, there might be new display glitches introduced.

* The DroidSansJapanese.ttf to be included in the official distribution would be a modified one if a certain bug doesn't get fixed in the Android git repository, see Known Issues(3).

--> Known Issues:

1. Once again, a seemingly windows specific issue, likely caused by pango. The texts rendered are not properly anti-aliased in some parts of the game, so are less pretty than the others. (Please compare the texts in the loading screen and the ones in the main menu screen.) In this regard, wqy-zenhei and Takao fonts also have their respective strong and weak areas. Until the day pango finally settles things down with windows, we may have to live with this.

2. The DSF + DSJ combination lack (like wqy-zenhei and Takao) some of the proper 'dash' characters introduced in wesnoth 1.9.x. A better solution would be giving Droids priority and fallback to DejaVu Sans only when necessary, but somehow this setup doesn't work. The game won't fallback to DejaVu. So I do it the other way around, preferring Dejavu to Droids for the non-CJK characters. As DejaVu and Droid have different styles, sometimes texts would feel slightly out of place. Nevertheless I would prefer this 'imperfection' to utter-user-unfriendly character-not-found rectangles any day.

Then some may ask, why not introduce Droid Sans that has those characters? Because we only need the dashes, and adding a new font for fewer than four characters seems not economical. Then maybe more importantly, throwing Droid Sans into the combination still means rectangles, no matter how I set the font order. Again, I don't know why, or if this is another windows specific issue.

3. Yet another Windows Specific Issue (TM). The DroidSansJapanese.ttf from the Android git repository is not valid on windows thanks to an incomplete cmap (the font has a format 12 subtable, but no accompanying format 4 subtable as required by windows). And the game will always fallback to DSF. A modified version of DSJ that works on windows is employed here. DSF used to have this issue too, but the Android developers fixed it in some update. DSJ is not so lucky on this one.

4. There seems to be a bug in the Korean translation that prevents the game from being saved. However this bug also shows when I use the original font.cfg and wqy-zenhei.

--> Help Needed:

Wesnothians, please test the new fonts in the game if possible. I sincerely hope you will enjoy them.

The steps to use the new fonts are:

1. Download the data archive from: http://wesnoth-translation-cn.googlecod ... nts.tar.gz

2. Extract it and put the contents of the extracted 'wesnoth' directory into the wesnoth resource directory. That means the game's root directory under windows, or (usually) /usr/share/wesnoth or /usr/share/games/wesnoth under linux.

3. There will be a fonts.cfg.ja and a fonts.cfg.ff in {resource dir}/data/hardwired, rename one of them to fonts.cfg and replace the original one. (Make sure a backup is made.) Japanese players will need fonts.cfg.ja, which gives DSJ priority over DSF, while Chinese and Korean players may prefer fonts.cfg.ff ('Fallback' First). The two files should work equally well for players using other languages. If it's not the case, please let me know.

4. Run the game.

It's worth mentioning that fonts.cfg.{ja,ff} both contain an entry for TakaoPGothic.ttf, and it's priority is only topped by DejaVu Sans. So if you would like to see Takao in action, just move the font file from {resource dir}/fonts/takao to {resource dir}/fonts and its ready to go.

--> Notes:

1. All my tests are performed on Windows 7 Ultimate (zh_CN), with wesnoth 1.9.13. This obviously is far from enough, and any input would be appreciated.

2. Unmodified Droid fonts are available at: http://android.git.linaro.org/gitweb?p= ... data/fonts

Unmodified DroidSansJapanese.ttf should work fine under linux, so there is no need for linux packagers to pack my modified version if this proposal gets accepted. A better solution would be submitting a bug report to Android to finish the work once and for all, which won't be discussed further in this proposal.

3. Takao fonts are available at: https://launchpad.net/takao-fonts

--> The End.

Comments are welcome.

Thanks in advance.

Best regards,
CloudiDust

--------------------------------------------------------

Attached are sample screenshots using the new fonts:

(Windows is used, expect the rendering quality to be better under linux.)

Simplified Chinese

Traditional Chinese

Japanese

Korean

[ancestral]

Have you looked at CJK Unifonts? It also uses a copyleft licesne (Arphic Public License) as recognized by the FSF.

Also, couldn't Wesnoth just rely on the system default? Obviously, stylistically, and for completeness it's nice to bundle a full set of CJK fonts, but wouldn't most users already have compatible fonts in their system? Why not choose to not bundle any?

[Espreon]

Also, couldn't Wesnoth just rely on the system default? Obviously, stylistically, and for completeness it's nice to bundle a full set of CJK fonts, but wouldn't most users already have compatible fonts in their system? Why not choose to not bundle any?

This cannot be done till the transition to GUI2 is complete since the GUI1 dialogs and such cannot use the system fonts (unless the user copies them to the game’s “fonts” directory and changes fonts.cfg so that the game will load them, of course), thus the need to bundle fonts with the game.

[CloudiDust]

Have you looked at CJK Unifonts? It also uses a copyleft licesne (Arphic Public License) as recognized by the FSF.

The CJK-Unifonts project contains two fonts: AR PL UMing (The 'Ming' variant, or 'Serif'.), AR PL UKai (The 'Kai' variant, or 'Regular Script', which is CJK specific.) But wesnoth mainly uses a sans-serif font (DejaVu Sans), so style-wise I would prefer Hei (as in WenQuanYi Zen 'Hei') or Gothic (as in TakaoP 'Gothic'), which can be seen as the Chinese and Japanese names for 'Sans', respectively. On the other hand, there's a reason Droid Sans Fallback and Droid Sans Japanese are named as such.

And the UMing and UKai are too large. (At 21 MB and 17 MB respectively, even bigger than wqy-zenhei!)

(To be fair, Microsoft Ya Hei, which is the default zh_CN font in Windows 7, is 20.7 MB, and doesn't contain any regional variants like UMing or UKai do.)

Even with such big font sizes, the two still lacks Japanese specific glyphs, as they focus mainly on differences between the glyphs of the same Chinese characters used in different Chinese-speaking regions, which is less obvious than differences between Chinese characters and their Japanese conterparts.

In an ideal world, wesnoth can use the system fonts, or bundle one font for each supported language without getting bloated, and we can forget about han unification altogether. But in the real world, compromise is required, at least before GUI 2 dominates.

[luojie-dune]

Go bundle typeface == harder; seamless experience
Use system typeface == easier; may break the User Interface Design

It is good to have some options, however, human resource weight beyond options.

IMO, just go Droid Sans Family - it is easier, for porting to mobile device as well; other benefits including active bug solve for current period, more internationalization support (WQY is purely for Chinese, other options have large bias on their specific C or J or K)

Let Life Be Easier Here Please.

[Dugucloud]

Fine. But why not 文泉驿微米黑?

[mistzone]

I've tested your DSF font with Korean language and it works better than wqy-zenhei.

I like this.

[CloudiDust]

Fine. But why not 文泉驿微米黑?

文泉驿微米黑(WQY Micro Hei) and 文泉驿细米黑(WQY Micro Hei Light) (0.2 beta) still have the italic problem under Windows when I tested them. As those two fonts are derived from Droid Sans Fallback (the original version), I don't know why WQY fonts have this problem while DSF (at least the newest version, ver 2.53) doesn't.

And DSF 2.53 now have more glyphs than WQY Micro Hei series. The Micro Hei series are developed because, back when the MH project started, DSF didn't cover the full CJK range (For those curious, this 'CJK' refers to a specific codepoint region in the unicode spec). Now it does, and also covers CJK ext A (in 2.53). So Micro Hei has no real advantage now.

[CloudiDust]

I've tested your DSF font with Korean language and it works better than wqy-zenhei.

I like this.

Glad that you like it.

Would you please tell me if you experienced the Korean specific bug I mentioned in the proposal? I now suppose that it was because I was using a zh_CN system, and on Korean systems there would be no issues. Am I right?

[pansz]

I like the idea, since a smaller default font is always better. DSF is smaller.

Reason: I will always use system font for my game, which includes manually copy system font to wesnoth directory, but that worth the effort.

Average user may not know how to use system font in wesnoth, and they can use the default, droid sans fallback seems very good for this purpose.

[ivanovic]

Okay, so what is the status with this one? Is the general agreement that this change should be done and what exactly should I commit? I am asking this since I'd like to release rc1 in about 30h and any change this big *not* added in rc1 is too late for 1.10.0 and has to wait for 1.10.1, the first bugfix release for 1.10. From what I heard the situation for all langs is only improved and nothing is worse when using the droid font, correct?

[CloudiDust]

Okay, so what is the status with this one? Is the general agreement that this change should be done and what exactly should I commit? I am asking this since I'd like to release rc1 in about 30h and any change this big *not* added in rc1 is too late for 1.10.0 and has to wait for 1.10.1, the first bugfix release for 1.10. From what I heard the situation for all langs is only improved and nothing is worse when using the droid font, correct?

I haven't been able to contact the Japanese maintainers or any other Japanese translator yet (except yma9, who has replied in the 'Japanese Font' thread and agrees with the proposal), so their 'official' opinion is unclear.

The Traditional Chinese maintainer (Steven2880) and Korean maintainer (mistzone) also support the proposal. (Steven wrote me a mail but didn't post here or to wesnoth-i18n.)

And no translators or players of non-CJK languages have commented.

So to be honest I cannot say there is already a conclusion. However all existing comments are positive.

I will soon send you a package containing the things to be commited if necessary.

[ivanovic]

I will soon send you a package containing the things to be commited if necessary.

I just committed this to trunk so it will be included in the next release. I also sent a note to the i18n mailing list to make sure that all translation maintainers that might be affected are aware of the change.

[CloudiDust]

The testing package for three more Japanese fonts is ready.

http://wesnoth-translation-cn.googlecod ... ts.tar.bz2

This package includes the following fonts:

Sazanami Gothic, Motoya L Cedar, Motoya L Maru

Sazanami is reintroduced because players on 2ch asked for it, and my tests show that Sazanami can be used to render italics. I have no idea why clearpotion wasn't able to do so. Maybe there was a version or config issue. (My Sazanami Gothic font was downloaded from cooltext.com.)

IMHO Motoya is similar to Droid, while Sazanami is similar to Takao, regarding their respective file sizes, and strong/weak areas when rendering.

Japanese players, please test those three fonts (also DSJ and Takao P Gothic in the 'old' test package) and comment.

Thanks in advance.

[AxalaraFlame]

Hey cloudy
@cloudiust
Sorry for disappointing you that I don't know much about how's the Chinese translation works going for a long time, because I have been using English version playing all my games since I have been playing wesnoth. I feel it would be a regrettable waste if Chinese version translation works have nearly finished, while I did not even take a look. Obviously, I did, but that was rather disappointing before(1.6 version, I slammed myself and tell me it was not true that I saw Lord Hamel speaking Ancient Chinese, gee ). So I picked it up again, and I thought I shall provide some feedback.

I made some test games by Chinese version few days before, and I found that there are always some words or signals missing problems after reloading; it somehow get fixed after reloading for several times as well, but generally word displaying issue is a bug. Just check Local MP games and you will find there are lots of missing words in the names of maps. The Chinese typing add-on does not work properly as well, basically, I guess, is induced by same kinds of technical bugs.

If you have some time or the ability to solve this issue, plz fix it.

BR