Experiment in loading past eras/campaigns in cores

Discussion of all aspects of the game engine, including development of new and existing features.

Moderator: Forum Moderators

demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Experiment in loading past eras/campaigns in cores

Post by demario »

First thoughts about using cores in wesnoth.

The core is a lightweight mechanism for isolating an add-on from the game configuration. In this way we can see it as sandboxing.
The two limits to the independence to the game engine are:
  • in the top layer, you still need to meet the requirements that are set by the C++ engine. It means that you need to pass the right amount of settings to the game engine, otherwise the core will behave badly. That includes initializing terrains, game config (ie the ruleset) and lua among others.
  • in the bottom layer, you still need to rely on the default parser and scripting (so not calling the current lua API will end with errors).
Basically everything between these two can be configured differently. But anything that is left unchanged must be declared explicitly in the core (see it like there is a wall around the core that prevent default settings from being reached).

There could be 2 ways for wesnoth to implement cores that I can think of, both have advantages and disadvantages.
  • the core must be selected at start. The choice of the core leading to different contents shown to the user
    This is how it is currently implemented in wesnoth.
    Advantages:
    • as the core is selected at start, the whole user experience can be dependent on the core.
      A spacenoth core could replace swords by laser sabres in wesnoth logo, an after-the-fall core could replace the map shown in title screen.
    • for two cores linked to the same world, the selection of the core leads to show only a selection of the installed content, making the choice easier
    • two copies of the same scenario could be linked to different cores, and they would not show at the same time to avoid confusion for the user
    • as tips and help can be accessed from title screen, the loaded core can provided all the relevant data.
    Disadvantages:
    • first and foremost: the user need to know some specifics about cores (how to load one, how it is reloading each time)
    • it works best if wesnoth was distributed with different cores, but there is no alternative to the default core in wesnoth
    • by filtering everything that is not listing the selected core, most of the add-ons will not be available outside of the default core
    • it doesn't allow the user to compare and make a choice between scenarios in different cores based on general information (latest version, mainline vs add-on version, version with full translation...)
    • this pretty excludes cores to have any role in multiplayer. You could only connect to the server "related to your core" (relationship that doesn't exist at this time)[edit] Actually the current behavior is that the MP server is neutral towards cores. All cores can connect to it, which has its own benefits, like not splitting the player community.[/edit]
    The benefits of this option are also reduced from limitations in wesnoth cores:
    • while an early selection of the core could allow full different looks of the game, the [gui] configuration cannot be set in a core (both from lack of WML to modify GUI and possible conflicts with C++ GUI code). So the tittle screen will still look the same for all cores (only tips can be changed).
    • the isolation of add-ons in cores is still incomplete: the add-on id, path and scenario id for campaigns still have to be unique across all cores. So putting a copy of an add-on in new core still requires a lot of renaming. That renaming can be easily extended to names and descriptions to let the user know the specifics of each versions if they were listed at the same time.
  • the core is loaded as a result of a selection of a scenario: when the selected content requires a core, the core is loaded before the scenario is started
    Advantages:
    • as the core are loaded based on the selected scenario, the user doesn't need to know about it
    • as all add-ons are available for selection, a secondary criteria could be used for listing (mainline/UMC, version number, translation status...)
    • possibility to select modifications linked to default core (or no core) to be active with scenario bound to another core (as they are not filtered-out)
    • the core would behave more like a super-resource
    Disadvantages:
    • if the core is loaded based on the selected add-on, each time a game is ended, the engine would need to load again the default core until a new scenario is started
    • complications for knowing which add-ons are compatible with a core. It would be left to the user to sort it out
    Engine limitations:
    • if all add-ons are listed together, additional information should be listed to help telling them apart (translation status...)
    • [edit] at this time the concept of a core being a super-resource is limited by the fact that core can't load resources. If a core needs to copy events to the era/scenario it needs to define a modification that must be hand-selected when running the core.[/edit]
The core is not checked when:
  • joining a server, which makes it possible to run games with past default era on 1.16 server ;)
  • joining a game. If the host and the guests are using different cores, they will all use the default era/units but the stats may be different
  • observing a game, which could lead to either OOS (if unit stats are different) or scripting error (if an add-on is installed but related to another core)
  • loading a save (using the wrong core for a campaign will break on next scenario, as you have suffixed their id, remember?)
Last edited by demario on November 22nd, 2022, 12:19 am, edited 16 times in total.
"simply put, it's an old game"T — Cackfiend
User avatar
hermestrismi
Posts: 612
Joined: February 6th, 2016, 11:28 pm
Location: Tunisia
Contact:

Re: Changing user interface based on WML core

Post by hermestrismi »

I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.
User avatar
Pentarctagon
Project Manager
Posts: 5526
Joined: March 22nd, 2009, 10:50 pm
Location: Earth (occasionally)

Re: Changing user interface based on WML core

Post by Pentarctagon »

hermestrismi wrote: May 6th, 2022, 7:22 pm I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.
Moved to the thread.
99 little bugs in the code, 99 little bugs
take one down, patch it around
-2,147,483,648 little bugs in the code
User avatar
hermestrismi
Posts: 612
Joined: February 6th, 2016, 11:28 pm
Location: Tunisia
Contact:

Re: Changing user interface based on WML core

Post by hermestrismi »

Pentarctagon wrote: May 6th, 2022, 7:23 pm
hermestrismi wrote: May 6th, 2022, 7:22 pm I didn't found the thread for the new addon-modification "core 1.12"
after installing it, this message keep appers

Code: Select all

Error validating data core.

Details:

    Core ID: core_1_12
    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
    Skipping the core.
Moved to the thread.
Thank you Pentarctagon
It is a great job. Hope you will succeed on it Demario
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Changing user interface based on WML core

Post by demario »

I am looking into the core as a way to load past versions of mainline translated content.

With the target to run an old version of campaigns to restore the past translations, my current knowledge is:
  • if the new version of the campaign exists, change id for campaigns and scenario to append a suffix. The "isolation" provided by the core doesn't extend to campaigns declaration.
  • change all the paths (to maps, images...) to point to the local copy defined in the add-on (the local path can't be conflicting with current mainline content)
  • update the campaign _main.cfg so that it meets the expectations of the current C++ engine (no [tutorial], no difficulties=...)
  • the old translations must be stored inside the add-on and be explicitly referenced in each campaign _main.cfg
  • add dependency to your core (in _info.cfg or _server.pbl). At this time, it seems you also have to explicitly import _main.cfg from the core. [that would define each trait twice, leading to "strong,strong" units]
  • you may again avoid distributing the old images from the campaigns by referencing the current ones from the game (data/campaigns/.../images/)
  • you need to keep the AI settings up-to-date to make sure the old scenarios work with the underlying functions from the current engine.
hermestrismi wrote: May 6th, 2022, 7:22 pm

Code: Select all

    Core Path: ~add-ons/Campaigns_1_12_Core/core-1.12/
    File not found.
I fixed the typo at core version 0.1.6. You'll need to update.
Last edited by demario on November 17th, 2022, 12:46 am, edited 3 times in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

At this time, it appears that it can help replace some of the text domains with old ones: wesnoth-units, wesnoth-help, wesnoth-multiplayer and all domains related to campaigns. The hope to be able to load old wesnoth textdomain is slim (that's the domain for messages from C++, lua, macros).
Both wesnoth-manpages and wesnoth-manual are not loaded into the game IIRC.

The main problem is wesnoth-lib as it contains the strings for the user interface (button labels...). To support a good user experience they need to be loaded in the version that is translated. It seems there are some limitations that prevent that:
  • The data/gui wml files can't be loaded into a core. So it doesn't seem possible to change the layout of the screens to "delete" the buttons that add new features having untranslated strings.

    It matches my experience that any slight change in a data/gui wml file will generally lead to a C++ exception if the corresponding cpp changes are missing.
  • Loading themes in a core is possible. But I didn't see any example where themes are "removing" (aka making them invisible) widgets completely. Themes are usually tweaking font size, adding overlays, changing/adding menus...
  • Lastly I am still looking for themes that apply changes beyond the main screen of the game (minimap, status...). For example the title screen or the dialogs for game creation. Beside removing some new widgets (RNG selection in campaign startup screen...), it would be required to change the labels of some buttons (like "Campaign" instead of "Campaigns" in the title screen)
When all that is sorted out, then I would need a system that loads to right core based on the chosen language. I don't remember seeing any code that checks the active language to drive decisions about behaviors (except displaying the translated strings obviously).
Last edited by demario on November 17th, 2022, 12:13 am, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

Here is a summary of my findings and the thoughts I got about using wesnoth cores to provide some backward support to old materials:

The support that can be provided can be split into the following categories:
  • Macro
    With each version, the syntax of some macros changes: either the macros gets dropped (MAGENTA_IS_THE_TEAM_COLOR) or they are changed to get more parameters. Using an existing macro with the wrong number of parameters will lead to a parser error.

    So at each version existing UMC must be updated to be compatible with new version. It is done by the author, a new maintainer that takes over or be part of crowd effort.

    The core can help by offering the old syntax for the macros. As the number of macros is known, this kind of support can be written once for all.
    All add-ons can then just link to that specific core rather than the default one and the unchanged add-on can be used in the new version. The most basic one-faction add-on and simple multiple-faction era can be recovered with this mechanism. There can be one core for each past version of wesnoth and have each core to provide the original default era for compatibility.

    We could also consider connecting to the past version add-on server when the matching core is loaded, download these add-ons and append the loaded core to the core="..." inside the _info.cfg.
  • WML actions
    The WML actions can be changed to be renamed or the keys can be changed to be made mandatory or renamed to more self-explanatory. Using the wrong key will be ignored or generate a WML parsing error.

    The core can help by providing a bridge between old WML actions keys and the new ones. This is done by catching the WML action in lua, rename the keys to the new version or fill the missing ones and fire the WML action with the new data. As the number of WML action is known, this kind of support can be written once for all. However given that it involves firing events, it must be put inside a modification because core can't define events at this time. The modification must be link to the core (to benefit the support for macro too) and be selected when using the add-on coming from a past version.

    In the long run, it would be easier to allow wesnoth cores to define their own events and to have them added to the scenario when one core is loaded.
  • Other WML and lua
    There is some WML that cannot be caught as WML action. They can be also changed in a way that generates an error when add-ons are loaded in new version. One typical example is the old way of defining difficulty in campaigns:
    The old way (example)

    Code: Select all

        difficulties=EASY,NORMAL,HARD
        difficulty_descriptions={MENU_IMG_TXT2 "units/legion/legionnaire.png~TC(1,magenta)" _"Legionnaire" _"(Easy)"} +
                         ";*" + {MENU_IMG_TXT2 "units/legion/propugnator.png~TC(1,magenta)" _"Decurion" _"(Medium)"} +
                         ";*" + {MENU_IMG_TXT2 "units/legion/centurion_b.png~TC(1,magenta)" _"Centurion" _"(Hard)"}
    
    The new way (another example)

    Code: Select all

        {CAMPAIGN_DIFFICULTY EASY   "units/elves-wood/fighter.png~RC(magenta>red)" ( _ "Fighter") ( _ "(Beginner)")}
        {CAMPAIGN_DIFFICULTY NORMAL "units/elves-wood/lord.png~RC(magenta>red)" ( _ "Lord") ( _ "(Normal)")} {DEFAULT_DIFFICULTY}
        {CAMPAIGN_DIFFICULTY HARD   "units/elves-wood/high-lord.png~RC(magenta>red)" ( _ "High Lord") ( _ "(Challenging)")}
    
    There is no way to "catch" this part of WML in lua and turn the old definition to the new format.
    Since most add-on authors do not care for deprecation warnings. most campaigns in past version still use the old version and as such are broken when running the new version. A manual edit is required. Which limits the "automatic" way to load old add-ons by just appending the matching core id in _info.cfg of the downloaded add-on.

    Another example (that is not SP specific) is the replacement of [choice] option definition by [combo] that again was enforced in BFW 1.16. While the [choice] is dropped silently (not an error), if that option is defining a variable used later in the code, that variable will be missing, leading to wrong behavior. That is the only manual change I had to make to the "(unofficial) Half Civ (1.14)" that I put up on the 1.16 add-on server after linking it to 1.14 core from "Support 1.10-1.14 cores".
  • Engine
    Beside changes in the WML, some changes to the C++ engine might also lead to different behaviors that lead to different experience to the player. One of the most recent example is how rounding is done when applying traits. While these problems could be considered as regression bugs, this is not how the devs think and they won't be fixed.

    These problems must be sorted on case-by-case basis. For that specific bug, the solution is to apply a post-recruit correction. But it is only applied to all unit types known to have the problem and so it will have to be extended whenever an era is defining a unit having the same problem.

    The need to have the exact same XP for a recruited spearman when playing the 1.14 version of the default era may seems to be only interesting to purists but is actual is quite significant when we are interested in replaying old saves (see next section).

    Another error of this type is that while in 1.14, the prestart events defined in sequence are executed in the same sequence. It seems that in BFW 1.16 the events will be executed in a different sequence. So if one event is using the result of another event prior in the sequence, when the sequence is shuffled, the expected data is not present and there is an error. The solution would be to make the sequence explicit by calling the different events by id.
    That is one problem that I encountered when porting "(Good old) World Conquest II"
  • Replays
    Trying to replay old saves in the current version of BFW opens its own can of worms.
    The two critical things to keep in mind when trying to allow replaying old saves are:
    • The engine must resolve the old save data exactly the same way as the original engine did. Otherwise OOS are thrown (and replays often ends badly).
    • The data in the replay is self contained (except references to external lua files) and no event will be added by the engine before the replay is started.
    As the engine must performs exactly as in the old version and as the most common way to fix engine behavior by appending events cannot be used when replaying saves, it is impossible to solve the situation within these constraints. The only way to support this feature currently is to hack in the C+ code and provide the feature as a patch that require people to compile their own version of the game.


    The selection of an additional modification (as described before) to offer some WML support is also impossible as the only modifications that will be loaded during replay are the ones that were enable when the save was generated (usually none). So offering some support from wesnoth core to replay saves must be through allowing cores to define their own events that will be added to the scenario where the core is loaded.

    When replaying a save, the core events must also be injected into the game data. That is a change in the design in the replay controller. The way to limit possible impact of such a change would be to only append core events when loading a save for replay. It will be then up to the developer of the core to make sure the events from the core is not doing anything "funny" with the data.
    We could also enforce that the core only defines preload events (but other events can be defined inside preload events).

    I believe that if cores are allowed to define their own events and that they can be injected into save data before replay, there is a good shot at supporting replay of saves from version 1.14.

    We could also make sure that the selected item in the directory-selector to the file dialog is defined inside the core (and matching the current directory for the default core).
  • Random number generator
    For 1.12 and beyond, there is a need to select different RNG based on the save to replay. Again, the engine could be changed so that is gets the information about the RNG to use from the loaded core (matching current RNG for default core). But at this time, there are too many missing features to entertain such a solution.
Last edited by demario on November 22nd, 2022, 12:22 am, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

I added support for running the editor with core 1.14 so it is possible to open maps from BFW1.14 in the current version now.
That is concluding the full list of features for core 1.14 and it is available on add-on server as version 1.0

With this version, you are able to run campaigns from 1.14 in BFW 1.16 if you have the add-on "Support 1.10-1.14 cores" installed.
If your favorite 1.14 campaign is missing on the current add-on server, you can play it locally by following these straight steps:
  • download the campaign of your choice from BFW 1.14 add-on server (using the 1.14 client or using a file downloader on addons.wesnoth.org/1.14)
  • move the directory to the user directory for the version 1.16 of wesnoth (path ending with data/add-ons/)
  • edit the _info.cfg file in the add-on directory to add the line core=core_1_14 inside the [info] section.
  • that's it! No script, no further edit.
If the campaign you're trying to run is depending on other add-ons, you will have to apply the same process to the dependencies.
If the add-ons are already on the 1.16 server, you can either have a try (it may very possibly work) or just delete the 1.16 add-on and apply the process to the add-on from version 1.14.
After loading the core "Bienvenue (Wesnoth 1.14 core)", the campaign will be listed in the empty campaign list.

To celebrate the release of version 1.0 of core 1.14, I have uploaded two campaigns "Pax Romana: The Rise of Vespasian" and "Napoleonic Wars" following this process. You can have a try at running them first to get familiar with the concept.

Without further change, only the easy setting is available.
Last edited by demario on November 22nd, 2022, 12:27 am, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
User avatar
Yomar
Posts: 394
Joined: October 27th, 2011, 5:14 am
Contact:

Re: Experiment in loading past eras/campaigns in cores

Post by Yomar »

Bravo !
Good job, I hope you will do the same for the next version of Wesnoth,
for BFW 1.17. 🙂
Beholded Wesnoth's Origins.
Max G on WIF
Rank 🌟🌟🌟🌟🌟
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

Yomar wrote: November 20th, 2022, 4:35 pm Bravo !
Cheers bro. I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on. :oops:
It is just a hunch at this time, but I encountered the same error a couple of times intermittently before. I still think there is nothing wrong with the add-on, and this kind of spurious behavior might be linked (it often is) on how the game is handling cache. My guess is that the engine doesn't invalidate the cache when we switch to another core. So when you start the game with a core that was not the one the cache was generated from, some macros might be missing in the cache,

If this problem repeats, you can either start the game with --nocache or delete old caches or make sure you load the core that the cache was generated from before quitting the game (that's the core that was loaded when you last installed/deleted/update/modified an add-on). You can always start the game with the right core by specify at startup --core=default/core_1_14. Downloading/deleting any add-on will trigger a rebuild of the cache too. Good luck!
Last edited by demario on July 24th, 2023, 8:16 pm, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
User avatar
egallager
Posts: 576
Joined: November 19th, 2020, 7:27 pm
Location: Concord, New Hampshire
Contact:

Re: Experiment in loading past eras/campaigns in cores

Post by egallager »

demario wrote: November 30th, 2022, 4:33 am
Yomar wrote: November 20th, 2022, 4:35 pm Bravo !
Cheers bro. I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on. :oops:
Well I mean I put in an update to fix it anyways, so, regardless of whether this add-on was involved or not, it works now...
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

egallager wrote: November 30th, 2022, 8:40 pm
demario wrote: November 30th, 2022, 4:33 am I am sorry to say that the issue with "War of Terrador" that you reported elsewhere might be related to this add-on. :oops:
Well I mean I put in an update to fix it anyways, so, regardless of whether this add-on was involved or not, it works now...
Sorry, I didn't mean to put you on the spot. I should have made clear that the reported issue happened to me without "War of Terrador" installed. It just listed other add-ons with the same error. It shows that the error is not in any third-party add-on, so the authors should not have to fix it (thanks anyway for doing it).

That's why I posted the warning on this thread so the users of "Support 1.10-1.14 cores" add-on are aware of the issue (which again, for me, is an engine issue).
"simply put, it's an old game"T — Cackfiend
Hiromasa
Posts: 12
Joined: February 3rd, 2013, 1:56 am

Re: Experiment in loading past eras/campaigns in cores

Post by Hiromasa »

In the _main.cfg file, add
[textdomain]
name="wesnoth-Pax_Romana"
path="data/add-ons/Pax_Romana/translations"
[/textdomain]
but at the beginning of the scenario file, it says
#textdomain wesnoth-Pax_Romana
is not written, so it is useless.
in each .cnf file.
#textdomain wesnoth-Pax_Romana
in each .cnf file, but even if a translation file is created, each person who plays the game has to create their own .cnf file with
#textdomain wesnoth-Pax_Romana
at the beginning of the .cnf file because it will not be reflected unless you add the following to the beginning of the .cnf file.
#textdomain wesnoth-Pax_Romana
to the beginning of the .cnf file as it will not be reflected.
demario
Posts: 131
Joined: July 3rd, 2019, 1:05 pm

Re: Experiment in loading past eras/campaigns in cores

Post by demario »

As a general comment, both "Pax Romana: The Rise of Vespasian" and "Napoleonic Wars" are coming with no translation as there were none available in the version on the server 1.14.

Hello Hiromasa, here is how I understand your message:
Hiromasa wrote: December 19th, 2022, 9:12 pm In the _main.cfg file, it says

Code: Select all

[textdomain]
    name="wesnoth-Pax_Romana"
    path="data/add-ons/Pax_Romana/translations"
[/textdomain]
but at the beginning of the scenario files, the #textdomain wesnoth-Pax_Romana is missing, so the definition will not be used.
Each .cfg file using translatable strings should have #textdomain wesnoth-Pax_Romana written at the beginning.
if a person creates a translation file, they will have to edit the .cfg files by adding #textdomain wesnoth-Pax_Romana,
otherwise the translation will not be used when they play the game.
I was a bit confused by your report until I checked your last post -- dating from February 2013, welcome back! :) -- where it appears that you are generating locally translation for the dialogs in Japanese using automatic translation tools. If you do so for "Pax Romana", it will indeed fail as you describe. But it seems that you have found the method to fix the problem locally. So is your problem solved?

For the general public, we can make a deal: you send me your Japanese translation and I publish a fix for your country fellows to enjoy the campaign in their mother tongue. The catalog is in the attachment if you need it. Just rename the unzipped file into ja.po before editing.
Attachments
wesnoth-Pax_Romana.pot.gz
Catalog of translatable strings for Pax Romana (stock 1.14)
(44.57 KiB) Downloaded 69 times
"simply put, it's an old game"T — Cackfiend
Hiromasa
Posts: 12
Joined: February 3rd, 2013, 1:56 am

Re: Experiment in loading past eras/campaigns in cores

Post by Hiromasa »

I wrote it because I was asked about it on Discord's # wesnoth Japanese channel.
My name on Discord is "J6K1"
1.16 The add-on failed to work in the environment, so I added "# textdomain wesnoth-Pax _ Romana" to the top of all the .cnf files of the add-on on Linux in the virtual machine environment and machine translated only the first part after creating the POT file to answer the question I asked.
Image
However, with my English skills, which rely on machine translation, it is impossible to prepare enough translations to be included in add-ons officially, so I hoped the add-ons would handle the translation.
Post Reply