Idea to remove the limits of the current terrain system

[Rhuvaen]

As far as WML is concerned there are only the usual letters, their meaning is changed but WML doesn't know.

It's the forum php map code that wouldn't like this system.

UTF-8 would have the advantage that the codes actually used in maps correspond to a specific terrain. With your translation system, how would you ensure the map editor wouldn't come up with crazy translations? When you save a map, it would try to find which of the standard translations letters are unused and then select those for the translations, correct? So any terrain other than the standard set could end up as any of the letters, always depending on the composition of the map? That's rather ugly... I think it's important to keep in mind the process of making maps, and that is via the editor (who might not make intelligent or obvious choices).

EDIT: not bashing your idea, just poking it

[jg]

I also have a question, that what happens if you want to use more terrain than there is letters, on your map? This is very unlikely, but still possible.

On the other hand, I like your idea. Seems quite sophisticated and mature.

jg

[SkeletonCrew]

As far as WML is concerned there are only the usual letters, their meaning is changed but WML doesn't know.

It's the forum php map code that wouldn't like this system.

And nobody can modify PHP code I guess

UTF-8 would have the advantage that the codes actually used in maps correspond to a specific terrain.

@Zookeeper I didn't have that much time yesterday so a bit longer reply about UTF-8.
I'm not really familiar with UTF-8 but won't it give at least trouble with entering the text?
So far I haven't figured out how use a sharp S on this system (Windows with American keyboard.) Also some programs don't reconize the normal dead keys and that's aergerlich (annoying). I won't start about all the other languages with their own special characters. Will this be a problem or not, I'm not really working with WML and maps that much. Also I don't know whether the WML parser will like UTF-8 if not the has to rewritten to use UTF-8.

With your translation system, how would you ensure the map editor wouldn't come up with crazy translations? When you save a map, it would try to find which of the standard translations letters are unused and then select those for the translations, correct? So any terrain other than the standard set could end up as any of the letters, always depending on the composition of the map? That's rather ugly... I think it's important to keep in mind the process of making maps, and that is via the editor (who might not make intelligent or obvious choices).

The editor needs a rewrite and since it's a program it will be stupid . The idea is to have menu to configure your translation letters, by default it will use the default translation. If the scenario has translations defined these will be loaded. When writing it will compare which characters have a modified translation and these will be written into the scenario file. You can only use tiles in the editor, that have a valid translation and when changing the translation it will check for tiles without a letter.
AFAIK the current editor can read scenario files, otherwise I know where my keyboard is

The idea is if you see a nice map with villages, grass, hills and some forest and like it but would like to have it in a snow terrain, you only need to change 4 translations and you winter map is there. The disadvantage ot this change will be if you load the map in 1.0 or 1.2 it will still have the grass, so I guess it would be wise to use the default map as often as possible.

I also have a question, that what happens if you want to use more terrain than there is letters, on your map? This is very unlikely, but still possible.

On the other hand, I like your idea. Seems quite sophisticated and mature.

jg

I already thought of that too, I'm thinking about making a system which uses the numbers a proposed before as map_data. eg map_data="1000 4001 8000 ..." but then WML also needs to know about the numbers. Also that would mean the lost of the single letter system. Maybe I'm also writing support for that but one step at a time.
Does it mean that you're going to make new tiles so there will be more than 60 tiles which you can use in one map, without making it look stupid

[jg]

That's true , but I usually like to have all aspects thought through before finishing them.

jg

[SkeletonCrew]

That's true, but I usually like to have all aspects thought through before finishing them.

jg

Me too , I thought about it but decided to work on that in a later stage. 61 characters should be enough for every map (don't quote me on that ).

[zookeeper]

@Zookeeper I didn't have that much time yesterday so a bit longer reply about UTF-8.
I'm not really familiar with UTF-8 but won't it give at least trouble with entering the text?
So far I haven't figured out how use a sharp S on this system (Windows with American keyboard.) Also some programs don't reconize the normal dead keys and that's aergerlich (annoying). I won't start about all the other languages with their own special characters. Will this be a problem or not, I'm not really working with WML and maps that much. Also I don't know whether the WML parser will like UTF-8 if not the has to rewritten to use UTF-8.

UTF-8 or just anything that allows more letters (extended ascii or something? UTF-8/16 sounds like the logical choice though).

I don't know if it'd just take a few lines of code to change the preprocessor to eat UTF-8 (or whatever), I hope someone finds the answer. As for some characters being hard to input...well, for mainline use we can just pick ones that are easy to input (I guess things like £, §, ~, ¨ and ^ are), or put a hint on how to produce each letter via an alt-numpad combo next to each one on the TerrainLettersWML page. And people can still use copypaste to enter odd letters, which at least I've found to be generally a viable way when I don't want to bother learning how to really input them.

[SkeletonCrew]

@Zookeeper I didn't have that much time yesterday so a bit longer reply about UTF-8.
I'm not really familiar with UTF-8 but won't it give at least trouble with entering the text?
So far I haven't figured out how use a sharp S on this system (Windows with American keyboard.) Also some programs don't reconize the normal dead keys and that's aergerlich (annoying). I won't start about all the other languages with their own special characters. Will this be a problem or not, I'm not really working with WML and maps that much. Also I don't know whether the WML parser will like UTF-8 if not the has to rewritten to use UTF-8.

UTF-8 or just anything that allows more letters (extended ascii or something? UTF-8/16 sounds like the logical choice though).

I don't know if it'd just take a few lines of code to change the preprocessor to eat UTF-8 (or whatever), I hope someone finds the answer. As for some characters being hard to input...well, for mainline use we can just pick ones that are easy to input (I guess things like £, §, ~, ¨ and ^ are), or put a hint on how to produce each letter via an alt-numpad combo next to each one on the TerrainLettersWML page. And people can still use copypaste to enter odd letters, which at least I've found to be generally a viable way when I don't want to bother learning how to really input them.

No offence but £, §, ¨ are not on my keyboard. Is alt-numpad multi-platform? I have no idea how hard it is to change the parser to UTF-8, I've to look at the source later today.

What do you think of my idea since it also aims at 1 terrain 1 letter, only the letter can be different between maps?

[khamul]

FWIW: Character encoding standards...

Each glyph (picture of a character) is defined to have a specific value according to the Unicode Specification. This is known as its CODE POINT. There are various ways of recording this code point:
UCS2 = 2 bytes per character (always). Covers most common languages.
UCS4 = 4 bytes per character (always). All languages (I think)
UTF-8 = variable bytes.

UTF-8 is defined such that for all code points matching the ASCII standard (up to 127) a single byte is used, with the same value as the ASCII byte. After 127, different standards did different things with the 8th bit - so UTF-8 uses it to flag the fact that this is a multi-byte character.

I've typed this, so I'll post it, but I've just realised that Wesnoth has rather good internationalisation support, so someone probably understands this rather better than I do... please feel free to correct my mistakes.

[zookeeper]

No offence but £, §, ¨ are not on my keyboard. Is alt-numpad multi-platform? I have no idea how hard it is to change the parser to UTF-8, I've to look at the source later today.

I don't know if alt-numpad is multi-platform, but I'm assuming that all platforms have at least some similar and reasonably simple way of inputting strange characters (not having one would be surprisingly stupid, right?). Personally I don't see inputting odd characters to be a big problem, since you rarely have to input them manually anyway (in which case it doesn't matter if you have to for example copypaste it those few times). And most users will most likely be able to use alt-numpad or something similar (correct me if I'm wrong).

What do you think of my idea since it also aims at 1 terrain 1 letter, only the letter can be different between maps?

It's better than a real multi-letter map format. However, I don't like the idea of breaking the global one letter <-> one terrain thing we now have. We'd have to use the number codes in WML (when plotting terrain with [terrain], in terrain graphics WML, etc), which especially in terrain graphics WML would be rather horrible since it already is almost unreadable. Using the number codes everywhere would just make it insane (although the whole terrain graphics system might be in need of an overhaul anyway eventually).

How would your system handle situations where we can now input a series of letters, for example something like type=kscZ\|/wY ? I guess you could have the number codes separated by commas, but I have a feeling it would become rather annoying to edit those lists since the numbers are much harder to remember.

[Viliam]

UTF-8 is defined such that for all code points matching the ASCII standard (up to 127) a single byte is used, with the same value as the ASCII byte. After 127, different standards did different things with the 8th bit - so UTF-8 uses it to flag the fact that this is a multi-byte character.

Yes, this is correct. Each file using only ASCII characters up to 127 is automatically a valid UTF-8 file. This way, the majority of ASCII files in the world did automatically conform to the new standard. And the tools processing ASCII files were able to process most of UTF-8 files automatically. That is, you can write e.g. a C++ program in Unicode editor, save it in UTF-8 and compile it with an ancient compiler.

The only problem with backward compatibility is that Unicode introduces some new whitespace characters, and the old parsers do not understand them as whitespace characters. By the way, the Wesnoth had the same problem (I do not know if it still has), it was not able to recognize the Unicode whitespace as whitespace in CFG files.

Because all the CFG files are UTF-8 encoded, and the maps are included in CFG file, it is absolutely logical to assume that the maps are UTF-8 encoded too.


Now one question is, how much we care about people who would like to edit their map files in text editors. (Because it is a minor problem to teach the Wesnoth map editor some new characters.) First, those text editors will have to be UTF-8 aware. Second, it will be necessary to have a font supporting all the characters used... otherwise you will just see a few question marks or empty squares or whatever your system uses as a replacement for unknown glyphs.

By the way, I think it would be cool to use Chinese characters for terrains. Just use the character "forest" for forest, "bridge" for bridge, "snow" for snow, etc. I do not know if these are one-character words. But it could be cool... especially if the characters would visually resemble the trees and bridges... almost like a graphical editor. -- Or we could use accented Latin letters for similar terrains. For example village, elven village, desert village, etc, would be the same basic letter with different accent marks.

Another question is, whether we want to have fixed characters for terrains, or assign a "terrain palette" to a map. (But then the Wesnoth map editor would probably need to understand the "terrain palettes" too.) Or do something completely different. These are actually two different questions: How do we want to represent terrains inside of program; and how do we want to encode that representation in WML files?

[zookeeper]

Well I'll be damned.

For example this (the ® should be the proper UTF-8 one) does work perfectly nicely in the editor:

Code: Select all

[terrain]
    symbol_image=desert-mountains
    id=foo
    name= _ "foo"
    char=®
    aliasof=m
[/terrain]

There are some problems when saving and loading maps that use these ®'s as terrain, however, but it looks like it might be enough to just tweak the read/write code a bit (the editor saves the ® hex into the map file as just one byte (?) when it should use two, I think). Or something like that.

[SkeletonCrew]

@Viliam The palette would be my idea and if we do that I want to change the editor to support the palette, see an earlier post.

@Zookeeper tweaking the read/write code a bit is what I'm doing at the moment. A bit is an understatement too much code assumes the data is 1 byte or rather 1 char . So first I've to make sure the code doesn't assume it's 1 byte.

[Viliam]

This is my attempt to describe how terrain system works now.

Currently terrains have three functions. First, they determine movement points and defense of units. Second, some of them have specific use: villages give gold, castles allow to recruit, starting positions are keeps which determine leader position at start of scenario. Third, they have different designs. I will divide the existing mainline terrains into a few groups:


Basic terrains define movement points and defenses for different units types: "Canyon", "Castle", "Cave", "Cave Wall", "Deep Water", "Forest", "Grassland", "Hills", "Mountains", "Sand", "Shallow Water", "Snow", "Swamp", "Village". Each unit (or more exactly: each 'movement type') must specify their MPs and defenses for each of the basic terrains. This is what you see listed in unit description.

Combined terrains provide a combination of two basic terrains. For movement points and defense, the better values for each units are selected. (The idea is that e.g. bridge provides solid ground for ground units, but also water for water units.) These are: "Bridge" and "River Ford" = Grassland + Shallow Water, "Desert Village" = Sand + Village, "Hill Village" = Hills + Village, "Mountain Village" = Mountain + Village, "Snow Village" = Show + Village, "Swamp Ruin" = Castle + Swamp, "Swamp Village" = Swamp + Village, "Sunken Ruin" = Castle + Shallow Water, "Tropical Forrest Village" = Grassland + Village, "Underground Village" = Cave + Village.

Modified terrains are just like basic terrains, but with some modification: "Merman Village" = Shallow water + healing + gold; "Keep" = Castle + put here leader when recruiting, "Oasis" = Desert + healing.

Alias terrains have different bitmaps, but otherwise are completely same as other terrains. "Desert" = Sand, "Desert Hills" and "Snow Hills" = Hills, "Desert Mountains" = Mountains, "Elven Village" = Village, "Encampment" and "Dwarven Castle" and "Ruin" = Castle, "Ice" = Snow, "Road" and "Desert Road" and "Dirt" and "Savanna" = Grassland, "Lava" = Canyon, "Tropical Forrest" and "Snow Forrest" = Forrest. BTW, there are 3 different terrain types for "Bridge", depending on how it is rotated. This is obsolete; the current terrain layout system could choose the right image automatically.


I think we should not expand the number of 'basic' terrains, because of the KISS principle. At the first versions of game, all terrains were 'basic' terrains; now the situation is completely different. (BTW, I suggest merging "Village" and "Castle" to one type.)

Each terrain type can be described as: one or two basic terrains + some flags + unique graphics. I do not know how closely the C code reflects this... I just want to say that all suggestions to modify the C code should take this into account.

[Viliam]

After reading this thread again, and some thinking...

I see no need to assign any special numbers to terrains in C code (as was suggested at the beginning of this thread). Currently all terrains are loaded from the file "terrain.cfg", and the file "terrain-graphics.cfg" specifies how they look. By adding similar code to user campaign, it is possible to create additional terrains, with almost the same functionality. (It is not possible to create additional 'basic' terrains in user campaigns, which is IMHO not necessary.)

In C code, the terrains can just be loaded into one big array, and internally referred to simply as "terrain #1", "terrain #2", etc. With regard to WML, there are two important WML attributes. Identifier "id" is a string which allows this terrain to be referred from "units.cfg" -- this is how unit movement points and defenses are mapped to basic terrains. Attribute "char" is one (Unicode) character which represents the terrain in maps, saved games, and for other purposes e.g. in terrain graphics algorithms. (At least WML language allows to use any Unicode character. Maybe C code does not support this yet... IMHO it should.)


And... the longer I think about it, the more I fail to see where is the problem.
Seriously... with Unicode letters, saved as UTF-8, we have backward compatibility, and almost unlimited supply of available letters for user campaigns. I just hope it is relatively easy to tweak the C code to support Unicode terrain characters.

The other thing is that the current terrain system is somewhat ugly, and deserves to be redesigned. However, "ugly (but functional) terrain system" and "running out of letters for terrains" are not related. We can think about redesigning the terrain system, but we can continue with the existing one without problems, if the C code will support Unicode terrain characters.

[SkeletonCrew]

It's better than a real multi-letter map format. However, I don't like the idea of breaking the global one letter <-> one terrain thing we now have. We'd have to use the number codes in WML (when plotting terrain with [terrain], in terrain graphics WML, etc), which especially in terrain graphics WML would be rather horrible since it already is almost unreadable. Using the number codes everywhere would just make it insane (although the whole terrain graphics system might be in need of an overhaul anyway eventually).

How would your system handle situations where we can now input a series of letters, for example something like type=kscZ\|/wY ? I guess you could have the number codes separated by commas, but I have a feeling it would become rather annoying to edit those lists since the numbers are much harder to remember.

(Ignoring the UTF-8 option for a moment)Calling type=kscZ\|/wY almost unreadable is an understatement isn't it The strings inside the scenario WML keep using the letters the ones in terrain-graphics.cfg will use the numbers comma separated numbers. This due to the fact that there will be more numbers than letters (at least if there are artists who want to make new tiles ). This was the reason to post a list of possible numbers. Since numbers are harder to remember it would be nice to have them in logical ranges. eg everything in the 1000 to 2000 range is a village, everything in the 7000-8000 range is a hill. Of course a nice terrain editor would solve this horror...

@khamul thanks for the info

@viliam thanks for the info, the goal is to allow more tiles still the same base type but with different settings.