[mod] Adding support to replay old saves (1.4-1.12 1v1) in BFW1.14 and 1.16

[egallager]

First version of the backward support for 1v1 replays in BFW1.16. It is still quite an early release (it has been tested on about 30 saves).

I cut the following corners for this release:

• commented the whole checksum on recruit in the code (src/synced_checkup.cpp) as I didn't find a way to disable it by modifying save content.

• I kept the 1.14 unit types under data/core (instead of a standalone add-on) as the cpp kind of expect the unit definition to be present (maybe there is a way to check the presence of the add-on in the code, but I haven't look at it)

• no support for ladder era

• no support for dunefolk

• known issue: differences between version on intelligent trait

The good news is that the RNG hasn't changed at all (assuming the intelligent trait is not RNG related) and no old RNG is needed. The consequence is that you can replay 1.14 saves and play 1.16 games without limitations.

Hi, could you submit this patch as a pull request on GitHub and then iterate on it there? It would be great to get this merged so that everyone can benefit from it.

[demario]

Third Fifth Sixth version of the backward support for 1v1 replays in BFW1.16.

I cut the following corners for this release:

• I still haven't worked out how to skip the recruit checksum

• no support for dunefolk

I have decided to split the code in different patches, each one digging deeper in the changes required to the engine to support replaying old saves. You will have to apply the changes for the highest version first (working backward from 1.14)

It is split in different bunches of changes:

• support 1.14: [Updated on Dec 12 to unitless rev 3]
  • it could have been pretty straightforward (only unit stat changes) if not for the issue of differences between version on intelligent trait. So it requires in the conversion to know the SPDE add-on era, launching the RNG, generating the random numbers to check for intelligent trait on new recruits to fix the xp rounding.
    The fix for xp rounding is done by catching recruit and advancement events for the impacted units with intelligent trait.
  • Good thing is that the RNG is unchanged from BFW1.16
  • It also starts to rollback of context-free grammar reduce the expected number of random numbers needed for names from gender-less races
  • Support some specific save format from replay server

• support 1.14 Ladder era: [Updated on Dec 12 to unitless ladder rev 1]
  • Support for Ladder era 0.1.1-0.1.17

• Support 1.12: [Removed for now]
  • it uses its own RNG and the memory of the desired engine version needs to be saved in a global variable (cause since BFW1.14, the RNG is not reset but is recreated from scratch for each new seed [and doesn't support any initialization data]).
  • So a config file must be passed the first time using an RNG to select the right version (at this time the global variable is not reset when switching back to RNG 1.16).
  • It also fixes the different numbers of random numbers required for nameless recruits. It starts getting fun in combination with plague attacks to create nameless walking corpses
  • It also completes the rollback of context-free grammar started in support 1.14 for names from gendered races

• Support 1.8-1.10 [Removed for now]
  • (1.10) this is the first version where in some cases, the random numbers cannot be recreated (seed is lost). Replaying them requires the [replay] to be loaded into the RNG and have the RNG read the numbers from a stack populated with this data rather than generating its own numbers (called simulation code)
  • (1.10) the RNG simulation must be activated/disabled at the appropriate time
  • (1.10) turn the seed back and forth from integer to hexa-decimal
  • (1.10) maintain the leader's position to fill it into the recruit commands
  • (1.10) fill the seed for recruit commands from the previous attack, maintain the number of random calls
  • (1.10) also maintain the number of random calls when pre-recruited units are involved (statues, loyal in Hornshark Island)
  • (1.10) address two other issues
  • (1.10) fix the Rush By Yetis (early random map picker) specific settings
  • (1.8 ) this version uses the same RNG as BFW1.10, with the only difference is that the random pool can be a negative number
  • (1.8 ) fix the petrified units (statues in Sulla's Ruins, ...)
  • (1.8 ) fix different terrain codes
  • (1.8 ) fix the hp rounding difference with BFW1.10 when some units are strong/intelligent and quick

• Support 1.4-1.6
  • (1.6) the RNG at this step is much simpler and only one seed is used for the whole game
  • (1.6) at the time, compatibility between RNG versions was already broken and a no-check mode supported dropping the actual numbers for those in the saves
  • (1.6) the WML for the [replay_start] is different from now (eg. the side leaders were not defined as [units]s)
  • (1.6) the random numbers for traits were drawn after those for names
  • (1.6) a random number was drawn even for gender-less units
  • (1.6) one additional number was drawn for each unit
  • (1.6) fix for this issue on maps with controller=ai side (statues)
  • (1.6) different behavior for healthy and feral traits
  • (1.4) identical RNG
  • (1.4) difference in rounding hitpoints for some units with quick trait
  • (1.4) the format of the move command in replay is different
  • (1.4) different behavior for healthy trait
  • (1.4) different behavior for drains ability
  • (1.6 and 1.4) two issues are still unresolved

Most unitless regressions should be fixed by now except known problem with Hornshark Island saves.

[egallager]

First version of the backward support for 1v1 replays in BFW1.16.

Known issue: differences between version on intelligent trait

Hi, could you submit this patch as a pull request on GitHub and then iterate on it there?

Hi, this is a non-official mod to the engine for the wesnoth community so I would rather keep the development progress visible on the forum.

I think it should become official though; the community would be able to benefit from it more if it were merged upstream and then shipped to everyone. Also I find checking out a pull request to be easier than applying patches. And you can always link back to the PR to keep it visible from here, as well.

[demario]

New (and possibly last) iteration on support 1.14 replay in BFW1.16 in attachment.

This version of the code (to apply on clean BFW1.16) doesn't require the SPDE add-on anymore.
Most of the changes are now done by WML in the events added to the era which mean no more additional unit types are created. That is why I label it the unitless era. This does reduce the risk of a slow down on the engine (which I never noticed) due to overhead in processing/storing/accessing more type definitions.
Second change is that the era is defined right inside the patch under a new data/add-ons sub-directory so that is doesn't mix with default macros but is always available when the modified code is used. There is no unit type definition under data/add-ons/Support_Past_Default_Era_Unitless for 1.14 support.

[edit] New patch to replay ladder-1.14 saves in BFW1.16. All units changed in ladder era still have a ladder-1.14 specific type (as identified by their own unit type when saves were recorded)[/edit]
[edit] I removed the 1.8-1.10 and 1.12 unitless patches. The reason is that this design is broken on the long term as I will try to explain below.

The design of unitless support is to add WML [event]s to make corrections to original (ie. new version) unit stats as the units appears on the map.
The issue is that the [event]s are executed after some engine actions are already executed. For example, if you want to restore a couple of hitpoints to some unit type, the [event]s will be applied each time such a unit is recruited but after the unit has got its traits. So the correction will be applied to the hitpoints after traits are taken into account. For rounding reasons, for some trait combination the correction may be wrong and it needs to be corrected again.

An example

Troll Rocklobber changed from 49 to 51hp at version 1.16. So the raw correction would be to remove 2hp to restore 1.14 stats.

If this Rocklobber is quick (-5% hp), applying this correction after the -5%hp reduction leads to 46hp while applying the -5%hp on original 49hp lead to 47hp.
So a final correction is required to add one hitpoint.

These final corrections depend on the hitpoints at the latest version and the amount of the corrections for older versions.
So that at each new version, any correction to support past versions could be possibly broken.
As I believe that the support for oldest versions should be unchanged with a new version (ie. we only need to add support for old stable version), the only solution is to go back to version specific unit types (where the hitpoints are set to an absolute value, not through a correction).[/edit]

You can either apply all the patches in order and compile once after they're applied (remember to run cmake), or you can apply the patch one by one with compilation after each patch (apply cmake+make at each step).

The SPDE add-on is still on removed from the add-on server, for use with the old patches please use backup in attachment.

[demario]

After a year, it is time to put this to an end. Last version of the mod to support replay old saves in BFW1.16 (I used v1.16.1, cmake, gcc). Compilation on windows might be fixed.

I finally decided to break the changes in two parts:

• support 1.14+ladder: it supports replaying default era saves for 1v1 maps with "unitless" era. I am mixing the default-1.14 with ladder-1.14 given that at time of download/load, one might not know with which era the save is generated. Apply this patch first. There is no specific RNG to support this mode. The additional eras are created under data/add-ons/Support_Past_Default_Era_Unitless directory. The creation of this directory is not supported by the development team, so don't use this patch .

• support 1.4-1.14: it supports replaying default era saves for 1v1 maps with 6 version specific unit types (for each unit). There are 3 specific RNG to support this mode. Apply this patch second. The additional eras are created under data/add-ons/Support_Past_Default_Era directory. You don't need to download SPDE add-on to use this.
  Be aware that after applying this patch:
  • 6 versions-specific eras will be listed, which is kind of annoying (sorry about that). So only apply this patch is you're actively looking for replaying 1.4 to 1.12 saves. Saves from 1.14 can be applied with first patch only.
  • 3 version specific RNG will be added and, as described in the opening post, you have to close wesnoth between replay and play/obs

Cheers.

[demario]

Since I started looking into cores, it was quite clear that it had the potential of making the support to replay old saves easier when possible.

The support to replay of saves is based on units stats modification to reinstate the stats from the time the save was generated. The stats could be reverted either by loading old unit definitions (with suffixed id to avoid conflict with current default units) or by applying corrections to unit stats based on current default (UNITLESS model). The core is making it now possible to drop the current default units completely and start from scratch. We can just copy the old unit definitions with original ids into the core and we are good to go.

Beside using the correct stats, support for old replays also requires injection of code to tweak the engine behavior to match the old behavior. This injection of code defined in a core is not done in the mainline replay function and the code in attachment is implementing it. This is an alternative way to support old replay to the current convert_old_saves() defined in mainline.

[demario]

The core is making it now possible to drop the current default units completely and start from scratch. We can just copy the old unit definitions with original ids into the core.

The benefit of using core to support old version is twofold:

• as we don't have suffixes to unit id, there is no need to go through all the WML/lua code from the era or the scenario to look for place where unit ids are used to append a suffix;

• as the engine correction events are defined in the core instead of inside an era (unlike the UNITLESS model), the corrections apply to all eras.

As an illustration, here is a version of the patch to apply to the engine to support replaying 1.14 World Conquest 2 saves in BFW 1.16.
Most of the additional changes done in the core is to support reverting changes to 1.16 Dunefolk units (and liminal alignment).

To use this solution you have to:

• apply the patch in attachment on a pristine version of BFW 1.16;

• compile the code;

• go to the add-on server and download "Support 1.10-1.14 cores" (minimum version 0.10.x). See in the description how to define the hotkey;

• press CTRL+SHIFT+k to select the "Bienvenue" core for 1.14 support;
  
  [NB: The "Reign of the Lords" add-on is known to be incompatible with the core add-on.]

• load a World Conquest 2 save from 1.14 to replay;

• press again CTRL+SHIFT+k to select the "Default" core when you are done with replaying 1.14 saves. You don't need to restart wesnoth.

I put a table of the different issues I got when testing replaying 1.14 World Conquest 2 saves and the current status of each issue. Beside two occurrences of strange side definition problem (wrong starting castle and wrong units), the remaining problem is about lua related to menu events (found dependent command in replay while is_synced=false). These OOS can be safely skipped.

This version is supporting replaying 1.14 1v1 saves with default era altogether.

[demario]

As BFW 1.16's days are coming close to an end, and as I am sitting on an old patch, I might as well publish it.

With this patch come three new features:

• support from 1.10 to 1.14 saves (1v1) using the core model. Saves earlier than 1.10 are not supported because there is no matching core.

• the support mechanism will now automatically load the matching core based on the version recorded in the save. So you don't have to fear to load the replay with the wrong core. This is useful when using the feature to load saves from the default save directory from old versions.
  That's what I am talking about:

  

  show_save_subdirs_1_14.dropdown.png (259.64 KiB) Viewed 1996 times

• finally I have embedded in the patch the code to run the replays in batch as a test (--replay_test=...) on which I worked for #7029.
  So you can basically run something like this bash command on linux:
  for i in `(cd ~/.local/share/wesnoth/1.16/saves/; ls 202201*)`; do (echo $i; wesnoth --replay_test=$i); done
  to run at full speed all the saves in succession. This is useful when looking for regressions or failure cases.
  You've better to put a meaningful filter (eg 202201*) to avoid running it for too long (CTRL+z is your friend otherwise)
  If you want to run on one savefile, put it under the wesnoth saves/ directory and pass it without the path up to saves/ (ie without the .../saves/ part). You can't pass several saves when calling wesnoth with --replay_test.

To use this solution you have to:

• apply the patch in attachment on a pristine version of BFW 1.16;

• run cmake and compile the code;

• go to the add-on server and download "Support 1.10-1.14 cores" (minimum version 1.0.1). See in the description how to define the hotkey;[NB: The "Reign of the Lords" add-on is known to be incompatible with the core add-on.]

• load any 1v1 save from 1.10-1.16 to replay (the right core will be loaded automatically);

• you will need to restart wesnoth when you are done with the replays.