questions about the preprocessor

[SlowThinker]

In short:
I edited a .cfg file. Most changes work, but one change is never taken into account, and games behave like the change has never been done.

A detailed history of the problem:

I have duplicated a .cfg file of the "Conquest" add-on in my local folder. The name of the original file is "conquest_surdmark.cfg", the new name is "conquest_surdmark_telep.cfg". Then I edited the new file + restarted Wesnoth, both several times.

The new file "conquest_surdmark_telep.cfg" contains the following part (it differs from the original file in "add 2": there is "add 3" in the original)

Spoiler:

Code: Select all

#Yetrab
[if]
	[have_location]
		x=64
		y=44
		owner_side={SIDE}
	[/have_location]
	[have_location]
		x=66
		y=46
		owner_side={SIDE}
	[/have_location]
	[then]
		{VARIABLE_OP income_bonus add 2}
	[/then]
[/if]

Then I found out any MP game of "conquest_surdmark_telep.cfg" behaves like the code was old (with "add 3"), and also in savefiles this part is represented the following way:

Spoiler:

Code: Select all

[if]
	[have_location]
		owner_side="1"
		x="64"
		y="44"
	[/have_location]
	[have_location]
		owner_side="1"
		x="66"
		y="46"
	[/have_location]
	[then]
		[set_variable]
			add="3"
			name="income_bonus"
		[/set_variable]
	[/then]
[/if] 

Other changes I made in the file work correctly - so it is not like the old file is used in place of the new one: it is like from the old file only the part "add 3" is used.

(both .cfg files are attached])

[zookeeper]

That piece of code is part of the CALCULATE_INCOME_BONUS_SURDMARK macro. First that macro is defined in the original .cfg, and then again in your duplicated .cfg. So yes, it's very possible that in order to re-define an already defined macro you need to #undef it first, so try adding #undef CALCULATE_INCOME_BONUS_SURDMARK right before #define CALCULATE_INCOME_BONUS_SURDMARK in your new .cfg.

[SlowThinker]

Thank you a lot. My fault was I didn't study the PreprocessorRef enough.
I did it now, and I have read also http://wiki.wesnoth.org/ScenarioWML
which says:

id: A unique identifier for this scenario. All scenarios must have an id.

___________________________________________________
I suppose things could work this way, please correct me if I am mistaken:

First data/_main.cfg is preprocessed, and "includes" are added recursively.
I suppose ~add-ons/ is added during this process? I wasn't able to find where it happens.
Anyway I suppose ~add-ons/ is included some way. Then...

...once a player chooses a game from a list of multiplayer scenarios, all [multiplayer] tags with the corresponding id become valid for the game. (This is why the id shall be unique}.
But Conquest has equal id values for all scenarios, and so the code of all scenarios affects any Conquest game.

__________________________________________________
But I am still confused. In order to test the previous hypothesis
1) I made the id unique, and it didn't help
2) I made another addon "Conquest_debug" with only one Surdmark map and a unique id, and everything is ok there.

[zookeeper]

Thank you a lot. My fault was I didn't study the PreprocessorRef enough.
I did it now, and I have read also http://wiki.wesnoth.org/ScenarioWML
which says:

id: A unique identifier for this scenario. All scenarios must have an id.

___________________________________________________
I suppose things could work this way, please correct me if I am mistaken:

First data/_main.cfg is preprocessed, and "includes" are added recursively.
I suppose ~add-ons/ is added during this process? I wasn't able to find where it happens.
Anyway I suppose ~add-ons/ is included some way. Then...

I think ~add-ons/ is hard-coded; it's just included by the engine automatically after data/ has been included.

...once a player chooses a game from a list of multiplayer scenarios, all [multiplayer] tags with the corresponding id become valid for the game. (This is why the id shall be unique}.
But Conquest has equal id values for all scenarios, and so the code of all scenarios affects any Conquest game.

I don't know how the game picks a scenario out of multiple ones with the same id, but it has nothing to do with your problem: even if there's multiple scenarios with the same id, the game still just picks one of them and uses that. It doesn't magically take some code from the other ones which have the same id.

__________________________________________________
But I am still confused. In order to test the previous hypothesis
1) I made the id unique, and it didn't help
2) I made another addon "Conquest_debug" with only one Surdmark map and a unique id, and everything is ok there.

Well, it didn't help because it didn't fix the problem, which is that in one file you're defining a macro and then in another file you're defining a macro of the same name again and just with different contents, which apparently doesn't work. Making another add-on solves the problem because I think the preprocessor automatically forgets about the macro definitions of an add-on before it starts to preprocess the next one.

[SlowThinker]

the unique id

I don't know how the game picks a scenario out of multiple ones with the same id, but it has nothing to do with your problem: even if there's multiple scenarios with the same id, the game still just picks one of them and uses that.

Then the condition that the id must be unique may be superfluous. It looks all the Conquest scenarios always had same id, and it worked. So it looks the game picks just the correct scenario, regardless of the id (?)


scopes of definitions

Well, it didn't help because it didn't fix the problem, which is that in one file you're defining a macro and then in another file you're defining a macro of the same name again and just with different contents, which apparently doesn't work. Making another add-on solves the problem because I think the preprocessor automatically forgets about the macro definitions of an add-on before it starts to preprocess the next one.

I expected WML had some clear rules that define which definition is used in case there are several definitions for one symbol. Or that people was able to derive them from some preprocessing algorithms.
To be more concrete: You said that if I am defining a symbol in a scenario, I can affect / be affected by same definitions in sibling scenarios, but I cannot affect / cannot be affected by any code in sibling addons. The question is which one is primary.
And other relationships are unanswered: core-scenario, addon-scenario, addon-addon....

Intuitively I expected a hierarchy like this:

Code: Select all

core	-campaign {campaigns/}	-
									-
		-multiplayer {multiplayer/}	- add-ons{~add-ons/} 	- addon1
																				- addon2
																				- addon3{~add-ons/addon3/utils/}	- scenario1
																																- scenario2
																																- scenario3

I also expected siblings in this hierarchy wouldn't affect one another, and when editing scenario3 I needn't to take care about the code of other scenarios (I expected I must take care about the ancestors only).

Some hierarchy of the WML clearly exists: it may be defined by both

• inclusions (an included code may be considered as a descendant)
• hierarchic _main.cfg system (dir/a/_main.cfg is a descendant of dir/_main.cfg)

But do you know whether this hierarchy affects scopes of definitions of the symbols?


sibling scenarios affecting one another
From your advice, I see two painful ways how to assure the sibling scenarios don't affect one another:
a) to check for equal symbols anytime a new symbol is added
b) to use "undef" everywhere

Do you think there are better ways?

I want to try a test today:
Now there is no hierarchy among Conquest/utils/ and Conquest/scenarios:

Code: Select all

#ifdef MULTIPLAYER
	{~add-ons/Conquest/utils}
	{~add-ons/Conquest/scenarios}
#endif

I need utils are ancestors of scenarios, and scenarios are siblings: scenarios will get one level deeper if each one will get its own _main.cfg. I will try what will happen then...

[silene]

So it looks the game picks just the correct scenario, regardless of the id (?)

The engine always picks the first scenario it encountered for the given id. (Knowing which scenario was the first one might be a bit complicated though.)

I expected WML had some clear rules that define which definition is used in case there are several definitions for one symbol.

The engine uses the last definition it encountered for the given macro.

But do you know whether this hierarchy affects scopes of definitions of the symbols?

The hierarchy doesn't affect the scope of definitions. However, macros are reset to the ones from core after each addon has been loaded.

[Anonymissimus]

I think you should not mix up the preprocessor and scenario ids. They are different concepts. Scenario ids are more like unit ids, they come into affect during runtime, not during preprocessing time.

There is no hirarchic meaning of folders or subfolders - things get copied (included) at that point where they are called.

Also it seems that the meaning of preprocessor scopes isn't obvious to you. When the game starts (a campaign is started, loaded or a mp scenario is entered) fore each choice only a small part is preprocessed/included into each other. (that campaign's #ifdef, or everything in an #ifdef multiplayer clause for a mp scenario). So it can (should - if the addon author does things right) never happen that a macro from an addon overwrites the one from another addon, for sp campaigns at least since they all have their own preprocessor scope. MP scenarios and campaigns live in the same scope so they'll conflict sooner or later.

As for #undef etc - wmlscope is the tool which warns you about duplicate definitions (an addon could overwrite definitions from core; consider using #ifndef ...#define... or so). Also I'd expect the 1.9 preprocessor to throw an error in this case.

For all in this post you'd need to ask silene who wrote the preprocessor to confirm; I however bet that it's all largely correct.

EDIT
silene beats me - however, the post still makes sense

[SlowThinker]

The engine always picks the first scenario it encountered for the given id. (Knowing which scenario was the first one might be a bit complicated though.)

I meant a situation a player picks a game from the list of MP games:
All Conquest scenarios have an equal id and different names. And it still works (people don't play just one scenario).
So it looks the engine must check both id and name. And that the combo id+name should be unique, and not the id itself.

The preprocessor is applied recursively, so included files will be parsed for macros, and after macro expansion will be parsed for macros again, and so on.

I am not sure I have understood it correctly. I suppose:
The engine doesn't distinguish between {MACROS} and {includes}.
Once first {braces} is reached, it is processed (=replaced), then the resulting code is taken as whole and processed again and again (in other words the tree is parsed by depth-first traversal)

The engine uses the last definition it encountered for the given macro.

'Encountered' mean the engine adds words to the resulting code one by one (just anihilating any encountered {braces}), and keeps a list of macro definitions found in this temporary code?
In other words the macro definition must exist prior any use (either directly in the file, or must be derived from {braces} in time)?

The hierarchy doesn't affect the scope of definitions. However, macros are reset to the ones from core after each addon has been loaded.

What is "Loading of addons"? Is it processing of a virtual file that contains just {~add-ons/}?
I suppose addons are loaded just after data/_main.cfg is processed? Like {~add-ons/} were last line in _main.cfg?

As for #undef etc - wmlscope is the tool which warns you about duplicate definitions (an addon could overwrite definitions from core; consider using #ifndef ...#define... or so).

to use wmlscope is painful too...
I think there should not be a problem with rewriting the core by error, none should expect a macro would be able to do 2 different things...
Oh no, I just realised it is not rewritten just locally in one file, but in all files of that add-on that are processed afterwards...
These global symbols are really somewhat unpleasant...

[silene]

The engine always picks the first scenario it encountered for the given id. (Knowing which scenario was the first one might be a bit complicated though.)

I meant a situation a player picks a game from the list of MP games:
All Conquest scenarios have an equal id and different names. And it still works (people don't play just one scenario).
So it looks the engine must check both id and name. And that the combo id+name should be unique, and not the id itself.

First, the game never uses the name, since it is translated. Second, it works, but only by chance, not by intent. It works locally since the host just uses the scenario number with respect to the selected cell, and it works remotely since the host sends the whole scenario instead of just the id. But if you are writing a multiplayer campaign, it will obviously fail since the host then searches the scenario by id, not by number.

The engine doesn't distinguish between {MACROS} and {includes}.
Once first {braces} is reached, it is processed (=replaced), then the resulting code is taken as whole and processed again and again (in other words the tree is parsed by depth-first traversal)

When a brace is encountered, its content is inserted at the current point and processing continues from the current point (that is, just before the inserted content). Any content is processed only once, at the time it is loaded.

In other words the macro definition must exist prior any use (either directly in the file, or must be derived from {braces} in time)?

The macro definition must exist before prior use, that is, the preprocessor must have encountered a #define before the corresponding brace. Preprocessing is completely linear, it just reads one character after another and never goes back.

What is "Loading of addons"? Is it processing of a virtual file that contains just {~add-ons/}?
I suppose addons are loaded just after data/_main.cfg is processed? Like {~add-ons/} were last line in _main.cfg?

You can see it like that. But it is simplistic since it doesn't explain why macros defined in an addon are discarded when the next addon is read.

Oh no, I just realised it is not rewritten just locally in one file, but in all files of that add-on that are processed afterwards...
These global symbols are really somewhat unpleasant...

That's what a preprocessor is. That's why most (all?) modern programming languages don't provide one.

[SlowThinker]

I suppose all the code is preprocessed at whole, when the player picks an item from the main Wesnoth menu? (at that moment all the flags for conditional directives are set (only a campaign must wait for a campaign difficulty level and a campaign id))

But it is simplistic since it doesn't explain why macros defined in an addon are discarded when the next addon is read.

Are they all discarded, or just core macros are re-read?


global symbols

That's what a preprocessor is. That's why most (all?) modern programming languages don't provide one.

It looks to me the global symbols are not a product of the preprocessor itself but of the fact that you collect all the macro definitions. This is my idea how to get rid of the global symbols:

The {braces} define a hierarchy in the code. The natural goal is when you are programming any part of the code, you are affected only by a code of ancestors (= you can use their definitions), and you can override these definitions.
So during preprocessing you must not simply cumulate all the macro definitions. During the preprocessing you travel the tree and you must pursue multiple definitions of symbols. You need to distinguish

• whether you recede from the tree root (data/_main.cfg) towards leafs: in this case you must disable definitions of symbols from upper layers, but only temporarily
• or whether you return to the tree root: in this case you must fully abandon definitions of symbols from lower layers, and (possibly) activate the temporarily disabled definitions from upper layers

This was just a basic idea - there are some things that should be adjusted: for example the question is whether the hierarchy defined by {braces} is exactly equal to the ideal hierarchy from a programmer's view.

[silene]

I suppose all the code is preprocessed at whole, when the player picks an item from the main Wesnoth menu? (at that moment all the flags for conditional directives are set (only a campaign must wait for a campaign difficulty level and a campaign id))

Yes.

But it is simplistic since it doesn't explain why macros defined in an addon are discarded when the next addon is read.

Are they all discarded, or just core macros are re-read?

Discarded.

The {braces} define a hierarchy in the code. The natural goal is when you are programming any part of the code, you are affected only by a code of ancestors (= you can use their definitions), and you can override these definitions.

I don't understand how that would even work. In particular, all the macros defined in the data/core/macros directory would be discarded as soon as the preprocessor is done reading them, since there is nothing else there.

[SlowThinker]

Of course you must ensure that if a macro definition is supposed to be used on several places of the code, then this definition must be written in a node that is an ancestor of all these places.
So the core macros must be placed on the very top. For example this way:

a content of data/_main.cfg

Code: Select all

{core/macros/} ##only moves the topmost _main.cfg out of data; may be hardcoded

(a note: the "macros" directory should be rather renamed to something like "root")

a content of core/macros/_main.cfg

Code: Select all

{./} 				##all files in core/macros/ except _main.cfg
{./node_root_1} 	##a sub-node with a path root/1
{./node_root_2} 	##a sub-node with a path root/2

a content of core/macros/node_root_1/_main.cfg

Code: Select all

{some_macro_definitions} ##possibly macros local for this scope; these macros would overwrite the core definitions
{core/about.cfg} ##in general node_root_1_1 may be here
{core/help.cfg} ##in general node_root_1_2 may be here
{core/hotkeys.cfg}
{core/terrain.cfg}
{core/terrain-graphics/}
{core/units.cfg}
{core/editor/}

a content of core/macros/node_root_2/_main.cfg

Code: Select all

{some_macro_definitions} ##possibly macros local for this scope; these macros would overwrite the core definitions
{~add-ons/} ##in general node_root_2_1 may be here

I am not sure whether the split to node_root_1 and node_root_2 is really useful, in other words whether different macros for the core and addons are needed. I presented it in order to show the principle.

__________________________________________________________

BTW all the idea is a kind of a generalisation of the procedure you do with the macros from add-ons: you also discard them once you make a backward step towards the tree root.

[silene]

I still don't understand. You said

whether you return to the tree root: in this case you must fully abandon definitions of symbols from lower layers

and

generalisation of the procedure you do with the macros from add-ons: you also discard them once you make a backward step towards the tree root.

So, according to your system, once the macros of data/core/macros have been defined (they are in a lower layer), the preprocessor steps back to the tree root and discards them.

So the core macros must be placed on the very top.

How do you achieve that, except by physically putting thousand lines of #define directly inside data/_main.cfg?

[SlowThinker]

So, according to your system, once the macros of data/core/macros have been defined (they are in a lower layer), the preprocessor steps back to the tree root and discards them.

You are right, my previous construction is a nonsense. It doesn't work with the current syntax.
But the next construction works:

We need {braces} roll out into this local code:

Code: Select all

#define macro1 ... #enddef
#define macro2 ... #enddef
#define macro3 ... #enddef
...
{childcode_1}
{childcode_2}

(it is supposed all wml code is contained in {childcode_x})
This is doable by a directory include. There must be core/macros/_final.cfg with this content:

Code: Select all

{childcode_1}
{childcode_2}

then processing {core/macros/} results in the code above.

But this doesn't look very flexible. So maybe you should allow also this syntax:

Code: Select all

{core/macros/ + childrens.cfg}

(childrens.cfg contains same code as _final.cfg above)

or this syntax:

Code: Select all

{core/macros/ + "{childcode_1} {childcode_2}"}

[silene]

But this doesn't look very flexible. So maybe you should allow also this syntax:

Code: Select all

{core/macros/ + childrens.cfg}

(childrens.cfg contains same code as _final.cfg above)
or this syntax:

Code: Select all

{core/macros/ + "{childcode_1} {childcode_2}"}

If we were to change the concepts and syntax of the preprocessor, rather than the top-down approach you suggest, I would go for a bottom-up one, so as to get modularity. In other words, macros defined in a parent node or in a sibling node are invisible from the current node. Only macros defined in a child node are visible. As a consequence, you no longer need to know the context a file is included from in order to understand what it actually gets preprocessed too; every node becomes self-contained.

In particular, a modern system like import/export would be a sane one. Basically, it comes down to having three directives #import, #export, #include. The first two are used for files containing only macro definitions. The last one is used for files containing WML but no macros (or only macros for local use). When you type "#import file.cfg", all the macros defined or exported by file.cfg becomes visible in the current node. When you "#export file.cfg", it's as if you had typed "#import", but in addition, any imported macro is automatically imported into any node importing the current node. (Export is just there to ease the creation of macro libraries, since it forwards definitions.) When you type "#include file.cfg", all the WML from file.cfg is processed and inserted at the current point, but macros don't leak. Therefore, such a system makes a clear distinction between files providing macros and files providing content; it also solves naturally the addon issue, since all the addons get to live in their very own preprocessing space, by definition.

Your second example becomes

Code: Select all

#import core/macros/
{childcode_1} {childcode_2}

Major downside of this system is that most WML files will need a #import line at their start, but that's the price to pay for modularity.