Emulators for past versions offer a stable environment for add-ons development and backward support

[demario]

If you wonder if it could be made easier to keep compatibility across versions of BfW, if you think it is a pity to have so many add-ons that are dropped at each new major release, if you wish it was more straightforward to make add-ons from previous versions available now with confidence that an undocumented incompatibility will not break the game in a obscure way that ruins the experience, if you are sad that the game can't be played in a localized version of your language when it used to be the case, this thread is dedicated to you.
------

Throughout the years many people wished that compatibility between major releases of wesnoth was improved and the work of porting add-ons would be less overwhelming (compilation of argumentations in this thread). More than 10 years ago, Coffee, a developer at the time, even dreamed about a system "for everything to work exactly as it worked in 1.12,1.10, 1.8, etc" (see full text in this post). It would be implemented so that when you "set the version of wesnoth the addon is playable against and the compatibility layer would load WML, etc. exactly to match".

Ten years later, after a couple of years of slow progress, such a system now exists for wesnoth: emulators. It is based on the definition of different cores where the old parameters of the game (the "WML, etc." from the quote) are isolated from the current ones. They offer to the players the possibility to select the version they wish to use. This is an add-on that is available for download on the server. The selection of the desired emulator is done through the core selection dialog that can be open using the 'k' (or possibly CTRL+SHFT+k) shortkey on the main screen after the add-on is installed.

This has been a one-person effort, and while it proved that the concept works, generalizing it on a large number of add-ons across different emulators would require more people to get involved. So if you are the author of an add-on that you would like to be available on the latest version without going through the trouble of porting it, if you would like to make some content that you appreciated in the past available for others to enjoy, if you like to work on translation on a stable text corpus and based on a past translation that has been recovered, if you would like to help improving the testing for add-ons using emulators, please, come forward and volunteer to help.
------

This post will not go into details over the inner working of the emulators. It will just try to answer the following question:

• what are the use cases for the emulators?

As a consequence of different attempts at tweaking emulators, I have identified different use cases where they can be useful to different people.
Here is the list I compiled from this experience:

Use cases

• restoring old versions of mainline campaigns
  How:

  Quite plainly, this is making an old version of mainline campaigns available in an add-on following consequent updates in the new release. For example, the add-on "Benvingut a Wesnoth!" makes the campaigns "Eastern Invasion" and "Under the Burning Suns" from BfW 1.16 available in emulator 1.16. Their content is now different in version 1.18 of the game.

• playing old scenario and old campaigns
  How:

  With the concept of "Stock Add-on", different add-ons from past versions are available for the current version of the game. As an example, the add-on "Swamplings (stock 1.12)" make the campaign "Swamplings" from Boru [in 2016] available through the emulator 1.12. The "Stock Add-on" tag requires some limitations on the changes that are allowed to qualify (no changes in text, no changes in code, changes only in _main.cfg and text domains, ...)

• making campaigns and scenario available in old translated versions
  How:

  When further changes are made to dialogs after a translation is complete, the new text returns to untranslated state until a new translator shows up (usually years later). Keep using the old text and that problem is fixed. For example, the add-on "Chào mừng đến với Wesnoth!" contains the version of different mainline campaigns with the full translation in Vietnamese using emulator 1.12. The campaigns for version 1.18 of the game are only 51% translated in Vietnamese.

• providing stable runtime environment for campaigns and scenario creators
  How:

  If the idea that your work is available for players for longer than a couple of release of wesnoth is important to you as a content creator, and you don't feel like being dragged by the work of porting your content to a newer version, you can screen off yourself from the future changes in the game engine by developing your content inside an emulator. Most of the emulator contains a map editor that generates map compatible with that version.

• supporting the work of campaigns and scenario porters
  How:

  There have been different strategies for porting add-ons from past versions (WMG, vghetto...) Most rely on wmllint first but more complex incompatibilities are hard to address within such toolkit (including lua changes). By porting an add-on through an emulator, the work on these issues is done once and you benefit from the experience and the fixes from other porters. When a fix solves an issue in an emulator, it is pooled for every content using that emulator.

• supporting the work of campaigns and scenario translators
  How:

  If you focus your translation work on the "Stock Add-ons" described in the second point, you are working on a text that will not change in the future. You are thus making sure that the lifetime of your contribution will be longer than if you were translating a campaign that will keep changing. Furthermore, if you translate the campaign "Swamplings (stock 1.12)" in Vietnamese for instance, it will run on the emulator 1.12 where Vietnamese translation is restored to about 100% (all units, abilities and help are translated).

• starting wesnoth with old graphics
  Wdym:

  At this point of time, the emulators are mostly using the graphics of the current of the game. But old graphics are already embedded for some units as easter eggs. Loading more old graphics could be available as an option later.

• replaying old multiplayer saves (default and ladder era)
  Wdym:

  A lot of work has been done on this. While this feature relies on emulators, changes is required to the game engine to be able to replay old saved (a full compilation of modified source code is required.)

There may be more cases in which using the emulators might be useful that I haven't identified.
If you have some, feel free to contribute.

[hermestrismi]

I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.

[Pentarctagon]

I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.

Moved to the thread.

[hermestrismi]

I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.

Moved to the thread.

Thank you Pentarctagon
It is a great job. Hope you will succeed on it Demario

[demario]

First thoughts about using cores in wesnoth.

The core is a lightweight mechanism for isolating an add-on from the game configuration. In this way we can see it as sandboxing.
The two limits to the independence to the game engine are:

• in the top layer, you still need to meet the requirements that are set by the C++ engine. It means that you need to pass the right amount of settings to the game engine, otherwise the core will behave badly. That includes initializing terrains, game config (ie the ruleset) and lua among others.

• in the bottom layer, you still need to rely on the default parser and scripting (so not calling the current lua API will end with errors).

Basically everything between these two can be configured differently. But anything that is left unchanged must be declared explicitly in the core (see it like there is a wall around the core that prevent default settings from being reached).

There could be 2 ways for wesnoth to implement cores that I can think of, both have advantages and disadvantages.

• the core must be selected at start. The choice of the core leading to different contents shown to the user
  This is how it is currently implemented in wesnoth.
  Advantages:
  • as the core is selected at start, the whole user experience can be dependent on the core.
    A spacenoth core could replace swords by laser sabres in wesnoth logo, an after-the-fall core could replace the map shown in title screen.
  • for two cores linked to the same world, the selection of the core leads to show only a selection of the installed content, making the choice easier
  • two copies of the same scenario could be linked to different cores, and they would not show at the same time to avoid confusion for the user
  • as tips and help can be accessed from title screen, the loaded core can provided all the relevant data.
  Disadvantages:
  • first and foremost: the user need to know some specifics about cores (how to load one, how it is reloading each time)
  • it works best if wesnoth was distributed with different cores, but there is no alternative to the default core in wesnoth
  • by filtering everything that is not listing the selected core, most of the add-ons will not be available outside of the default core
  • it doesn't allow the user to compare and make a choice between scenarios in different cores based on general information (latest version, mainline vs add-on version, version with full translation...)
  • this pretty excludes cores to have any role in multiplayer. You could only connect to the server "related to your core" (relationship that doesn't exist at this time)[edit] Actually the current behavior is that the MP server is neutral towards cores. All cores can connect to it, which has its own benefits, like not splitting the player community.[/edit]
  The benefits of this option are also reduced from limitations in wesnoth cores:
  • while an early selection of the core could allow full different looks of the game, the [gui] configuration cannot be set in a core (both from lack of WML to modify GUI and possible conflicts with C++ GUI code). So the tittle screen will still look the same for all cores (only tips can be changed).
  • the isolation of add-ons in cores is still incomplete: the add-on id, path and scenario id for campaigns still have to be unique across all cores. So putting a copy of an add-on in new core still requires a lot of renaming. That renaming can be easily extended to names and descriptions to let the user know the specifics of each versions if they were listed at the same time.

• the core is loaded as a result of a selection of a scenario: when the selected content requires a core, the core is loaded before the scenario is started
  Advantages:
  • as the core are loaded based on the selected scenario, the user doesn't need to know about it
  • as all add-ons are available for selection, a secondary criteria could be used for listing (mainline/UMC, version number, translation status...)
  • possibility to select modifications linked to default core (or no core) to be active with scenario bound to another core (as they are not filtered-out)
  • the core would behave more like a super-resource
  Disadvantages:
  • if the core is loaded based on the selected add-on, each time a game is ended, the engine would need to load again the default core until a new scenario is started
  • complications for knowing which add-ons are compatible with a core. It would be left to the user to sort it out
  Engine limitations:
  • if all add-ons are listed together, additional information should be listed to help telling them apart (translation status...)
  • [edit] at this time the concept of a core being a super-resource is limited by the fact that core can't load resources. If a core needs to copy events to the era/scenario it needs to define a modification that must be hand-selected when running the core.[/edit]

The core is not checked when:

• joining a server, which makes it possible to run games with past default era on 1.16 server

• joining a game. If the host and the guests are using different cores, they will all use the default era/units but the stats may be different

• observing a game, which could lead to either OOS (if unit stats are different) or scripting error (if an add-on is installed but related to another core)

• loading a save (using the wrong core for a campaign will break on next scenario, as you have suffixed their id, remember?)

------

Code: Select all

    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.

I fixed the typo at core version 0.1.6. You'll need to update.

[demario]

I am looking into the core as a way to load past versions of mainline translated content.

With the target to run an old version of campaigns to restore the past translations, my current knowledge is:

• if the new version of the campaign exists, change id for campaigns and scenario to append a suffix. The "isolation" provided by the core doesn't extend to campaigns declaration.

• change all the paths (to maps, images...) to point to the local copy defined in the add-on (the local path can't be conflicting with current mainline content)

• update the campaign _main.cfg so that it meets the expectations of the current C++ engine (no [tutorial], no difficulties=...)

• the old translations must be stored inside the add-on and be explicitly referenced in each campaign _main.cfg

• add dependency to your core (in _info.cfg or _server.pbl). At this time, it seems you also have to explicitly import _main.cfg from the core. [that would define each trait twice, leading to "strong,strong" units]

• you may again avoid distributing the old images from the campaigns by referencing the current ones from the game (data/campaigns/.../images/)

• you need to keep the AI settings up-to-date to make sure the old scenarios work with the underlying functions from the current engine.

At this time, it appears that it can help replace some of the text domains with old ones: wesnoth-units, wesnoth-help, wesnoth-multiplayer and all domains related to campaigns. The hope to be able to load old wesnoth textdomain is slim (that's the domain for messages from C++, lua, macros).
Both wesnoth-manpages and wesnoth-manual are not loaded into the game IIRC.

The main problem is wesnoth-lib as it contains the strings for the user interface (button labels...). To support a good user experience they need to be loaded in the version that is translated. It seems there are some limitations that prevent that:

• The data/gui wml files can't be loaded into a core. So it doesn't seem possible to change the layout of the screens to "delete" the buttons that add new features having untranslated strings.
  
  It matches my experience that any slight change in a data/gui wml file will generally lead to a C++ exception if the corresponding cpp changes are missing.

• Loading themes in a core is possible. But I didn't see any example where themes are "removing" (aka making them invisible) widgets completely. Themes are usually tweaking font size, adding overlays, changing/adding menus...

• Lastly I am still looking for themes that apply changes beyond the main screen of the game (minimap, status...). For example the title screen or the dialogs for game creation. Beside removing some new widgets (RNG selection in campaign startup screen...), it would be required to change the labels of some buttons (like "Campaign" instead of "Campaigns" in the title screen)

When all that is sorted out, then I would need a system that loads to right core based on the chosen language. I don't remember seeing any code that checks the active language to drive decisions about behaviors (except displaying the translated strings obviously).

[demario]

Here is a summary of my findings and the thoughts I got about using wesnoth cores to provide some backward support to old materials:

The support that can be provided can be split into the following categories:

• Macro
  With each version, the syntax of some macros changes: either the macros gets dropped (MAGENTA_IS_THE_TEAM_COLOR) or they are changed to get more parameters. Using an existing macro with the wrong number of parameters will lead to a parser error.
  
  So at each version existing UMC must be updated to be compatible with new version. It is done by the author, a new maintainer that takes over or be part of crowd effort.
  
  The core can help by offering the old syntax for the macros. As the number of macros is known, this kind of support can be written once for all.
  All add-ons can then just link to that specific core rather than the default one and the unchanged add-on can be used in the new version. The most basic one-faction add-on and simple multiple-faction era can be recovered with this mechanism. There can be one core for each past version of wesnoth and have each core to provide the original default era for compatibility.
  
  We could also consider connecting to the past version add-on server when the matching core is loaded, download these add-ons and append the loaded core to the core="..." inside the _info.cfg.

• WML actions
  The WML actions can be changed to be renamed or the keys can be changed to be made mandatory or renamed to more self-explanatory. Using the wrong key will be ignored or generate a WML parsing error.
  
  The core can help by providing a bridge between old WML actions keys and the new ones. This is done by catching the WML action in lua, rename the keys to the new version or fill the missing ones and fire the WML action with the new data. As the number of WML action is known, this kind of support can be written once for all. However given that it involves firing events, it must be put inside a modification because core can't define events at this time. The modification must be link to the core (to benefit the support for macro too) and be selected when using the add-on coming from a past version.
  
  In the long run, it would be easier to allow wesnoth cores to define their own events and to have them added to the scenario when one core is loaded.

• Other WML and lua
  There is some WML that cannot be caught as WML action. They can be also changed in a way that generates an error when add-ons are loaded in new version. One typical example is the old way of defining difficulty in campaigns:
  The old way (example)

  Code: Select all

      difficulties=EASY,NORMAL,HARD
      difficulty_descriptions={MENU_IMG_TXT2 "units/legion/legionnaire.png~TC(1,magenta)" _"Legionnaire" _"(Easy)"} +
                       ";*" + {MENU_IMG_TXT2 "units/legion/propugnator.png~TC(1,magenta)" _"Decurion" _"(Medium)"} +
                       ";*" + {MENU_IMG_TXT2 "units/legion/centurion_b.png~TC(1,magenta)" _"Centurion" _"(Hard)"}
  

  The new way (another example)

  Code: Select all

      {CAMPAIGN_DIFFICULTY EASY   "units/elves-wood/fighter.png~RC(magenta>red)" ( _ "Fighter") ( _ "(Beginner)")}
      {CAMPAIGN_DIFFICULTY NORMAL "units/elves-wood/lord.png~RC(magenta>red)" ( _ "Lord") ( _ "(Normal)")} {DEFAULT_DIFFICULTY}
      {CAMPAIGN_DIFFICULTY HARD   "units/elves-wood/high-lord.png~RC(magenta>red)" ( _ "High Lord") ( _ "(Challenging)")}
  

  There is no way to "catch" this part of WML in lua and turn the old definition to the new format.
  Since most add-on authors do not care for deprecation warnings. most campaigns in past version still use the old version and as such are broken when running the new version. A manual edit is required. Which limits the "automatic" way to load old add-ons by just appending the matching core id in _info.cfg of the downloaded add-on.
  
  Another example (that is not SP specific) is the replacement of [choice] option definition by [combo] that again was enforced in BFW 1.16. While the [choice] is dropped silently (not an error), if that option is defining a variable used later in the code, that variable will be missing, leading to wrong behavior. That is the only manual change I had to make to the "(unofficial) Half Civ (1.14)" that I put up on the 1.16 add-on server after linking it to 1.14 core from "Support 1.10-1.14 cores".

• Engine
  Beside changes in the WML, some changes to the C++ engine might also lead to different behaviors that lead to different experience to the player. One of the most recent example is how rounding is done when applying traits. While these problems could be considered as regression bugs, this is not how the devs think and they won't be fixed.
  
  These problems must be sorted on case-by-case basis. For that specific bug, the solution is to apply a post-recruit correction. But it is only applied to all unit types known to have the problem and so it will have to be extended whenever an era is defining a unit having the same problem.
  
  The need to have the exact same XP for a recruited spearman when playing the 1.14 version of the default era may seems to be only interesting to purists but is actual is quite significant when we are interested in replaying old saves (see next section).
  
  Another error of this type is that while in 1.14, the prestart events defined in sequence are executed in the same sequence. It seems that in BFW 1.16 the events will be executed in a different sequence. So if one event is using the result of another event prior in the sequence, when the sequence is shuffled, the expected data is not present and there is an error. The solution would be to make the sequence explicit by calling the different events by id.
  That is one problem that I encountered when porting "(Good old) World Conquest II"

• Replays
  Trying to replay old saves in the current version of BFW opens its own can of worms.
  The two critical things to keep in mind when trying to allow replaying old saves are:
  • The engine must resolve the old save data exactly the same way as the original engine did. Otherwise OOS are thrown (and replays often ends badly).
  • The data in the replay is self contained (except references to external lua files) and no event will be added by the engine before the replay is started.
  As the engine must performs exactly as in the old version and as the most common way to fix engine behavior by appending events cannot be used when replaying saves, it is impossible to solve the situation within these constraints. The only way to support this feature currently is to hack in the C+ code and provide the feature as a patch that require people to compile their own version of the game.
  
  
  The selection of an additional modification (as described before) to offer some WML support is also impossible as the only modifications that will be loaded during replay are the ones that were enable when the save was generated (usually none). So offering some support from wesnoth core to replay saves must be through allowing cores to define their own events that will be added to the scenario where the core is loaded.
  
  When replaying a save, the core events must also be injected into the game data. That is a change in the design in the replay controller. The way to limit possible impact of such a change would be to only append core events when loading a save for replay. It will be then up to the developer of the core to make sure the events from the core is not doing anything "funny" with the data.
  We could also enforce that the core only defines preload events (but other events can be defined inside preload events).
  
  I believe that if cores are allowed to define their own events and that they can be injected into save data before replay, there is a good shot at supporting replay of saves from version 1.14.
  
  We could also make sure that the selected item in the directory-selector to the file dialog is defined inside the core (and matching the current directory for the default core).

• Random number generator
  For 1.12 and beyond, there is a need to select different RNG based on the save to replay. Again, the engine could be changed so that is gets the information about the RNG to use from the loaded core (matching current RNG for default core). But at this time, there are too many missing features to entertain such a solution.

[demario]

I added support for running the editor with core 1.14 so it is possible to open maps from BFW1.14 in the current version now.
That is concluding the full list of features for core 1.14 and it is available on add-on server as version 1.0

With this version, you are able to run campaigns from 1.14 in BFW 1.16 if you have the add-on "Support 1.10-1.14 cores" installed.
If your favorite 1.14 campaign is missing on the current add-on server, you can play it locally by following these straight steps:

• download the campaign of your choice from BFW 1.14 add-on server (using the 1.14 client or using a file downloader on addons.wesnoth.org/1.14)

• move the directory to the user directory for the version 1.16 of wesnoth (path ending with data/add-ons/)

• edit the _info.cfg file in the add-on directory to add the line core=core_1_14 inside the [info] section.

• that's it! No script, no further edit.

If the campaign you're trying to run is depending on other add-ons, you will have to apply the same process to the dependencies.
If the add-ons are already on the 1.16 server, you can either have a try (it may very possibly work) or just delete the 1.16 add-on and apply the process to the add-on from version 1.14.
After loading the core "Bienvenue (Wesnoth 1.14 core)", the campaign will be listed in the empty campaign list.

To celebrate the release of version 1.0 of core 1.14, I have uploaded two campaigns "Pax Romana: The Rise of Vespasian" and "Napoleonic Wars" following this process. You can have a try at running them first to get familiar with the concept.

Without further change, only the easy setting is available.

[Yomar]

Bravo !
Good job, I hope you will do the same for the next version of Wesnoth,
for BFW 1.17.

[demario]

Bravo !

Cheers bro. I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on.
It is just a hunch at this time, but I encountered the same error a couple of times intermittently before. I still think there is nothing wrong with the add-on, and this kind of spurious behavior might be linked (it often is) on how the game is handling cache. My guess is that the engine doesn't invalidate the cache when we switch to another core. So when you start the game with a core that was not the one the cache was generated from, some macros might be missing in the cache,

If this problem repeats, you can either start the game with --nocache or delete old caches or make sure you load the core that the cache was generated from before quitting the game (that's the core that was loaded when you last installed/deleted/update/modified an add-on). You can always start the game with the right core by specify at startup --core=default/core_1_14. Downloading/deleting any add-on will trigger a rebuild of the cache too. Good luck!

[egallager]

Bravo !

Cheers bro. I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on.

Well I mean I put in an update to fix it anyways, so, regardless of whether this add-on was involved or not, it works now...

[demario]

I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on.

Well I mean I put in an update to fix it anyways, so, regardless of whether this add-on was involved or not, it works now...

Sorry, I didn't mean to put you on the spot. I should have made clear that the reported issue happened to me without "War of Terrador" installed. It just listed other add-ons with the same error. It shows that the error is not in any third-party add-on, so the authors should not have to fix it (thanks anyway for doing it).

That's why I posted the warning on this thread so the users of "Support 1.10-1.14 cores" add-on are aware of the issue (which again, for me, is an engine issue).

[Hiromasa]

In the _main.cfg file, add
[textdomain]
name="wesnoth-Pax_Romana"
path="data/add-ons/Pax_Romana/translations"
[/textdomain]
but at the beginning of the scenario file, it says
#textdomain wesnoth-Pax_Romana
is not written, so it is useless.
in each .cnf file.
#textdomain wesnoth-Pax_Romana
in each .cnf file, but even if a translation file is created, each person who plays the game has to create their own .cnf file with
#textdomain wesnoth-Pax_Romana
at the beginning of the .cnf file because it will not be reflected unless you add the following to the beginning of the .cnf file.
#textdomain wesnoth-Pax_Romana
to the beginning of the .cnf file as it will not be reflected.

[demario]

As a general comment, both "Pax Romana: The Rise of Vespasian" and "Napoleonic Wars" are coming with no translation as there were none available in the version on the server 1.14.

Hello Hiromasa, here is how I understand your message:

In the _main.cfg file, it says

Code: Select all

[textdomain]
    name="wesnoth-Pax_Romana"
    path="data/add-ons/Pax_Romana/translations"
[/textdomain]

but at the beginning of the scenario files, the #textdomain wesnoth-Pax_Romana is missing, so the definition will not be used.
Each .cfg file using translatable strings should have #textdomain wesnoth-Pax_Romana written at the beginning.
if a person creates a translation file, they will have to edit the .cfg files by adding #textdomain wesnoth-Pax_Romana,
otherwise the translation will not be used when they play the game.

I was a bit confused by your report until I checked your last post -- dating from February 2013, welcome back! -- where it appears that you are generating locally translation for the dialogs in Japanese using automatic translation tools. If you do so for "Pax Romana", it will indeed fail as you describe. But it seems that you have found the method to fix the problem locally. So is your problem solved?

For the general public, we can make a deal: you send me your Japanese translation and I publish a fix for your country fellows to enjoy the campaign in their mother tongue. The catalog is in the attachment if you need it. Just rename the unzipped file into ja.po before editing.

[Hiromasa]

I wrote it because I was asked about it on Discord's # wesnoth Japanese channel.
My name on Discord is "J6K1"
1.16 The add-on failed to work in the environment, so I added "# textdomain wesnoth-Pax _ Romana" to the top of all the .cnf files of the add-on on Linux in the virtual machine environment and machine translated only the first part after creating the POT file to answer the question I asked.

However, with my English skills, which rely on machine translation, it is impossible to prepare enough translations to be included in add-ons officially, so I hoped the add-ons would handle the translation.