Adding support to replay old saves (1.4-1.12 1v1) in BFW1.14

Discussion of all aspects of the game engine, including development of new and existing features.

Moderator: Forum Moderators

demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Adding support to replay old saves (1.4-1.12 1v1) in BFW1.14

Post by demario »

At version BFW1.14, wesnoth doesn't offer any support for loading old saves to replay.
If you are interested in replaying old saves, additional support must be added to BFW1.14 code.

Here is an attempt to implement this support from version 1.12 to version 1.4. It focuses on 1v1 games (no survival) with default era (excluding dunefolk) without additional modifications. This support has been tested using saved games found on the ladder site. Kudos to them for safeguarding that part of history for wesnoth multiplayer.
  • To activate this support, download the archive in attachment, uncompress it and apply it to a clean BFW1.14 install using "patch -p2" command.
    If you use cmake tool chain, you have to regenerate the Makefiles (-DCMAKE_BUILD_TYPE=Release) as new sources files have been added.
    You can then compile the usual way (only cmake Makefiles have been tested).
  • Failure in support for an old save can materialize in four ways:
    - OOS, error message showing difference before recorded game and replayed game (only some can be skipped safely)
    - WML error, the replay loads, but doesn't start or stops at turn 1 with one side winning
    - corrupt save, the save will not load
    - fatal error, wesnoth crashes
  • Here are the statistics for backward support:
    1.14 saves:
    Let's assume 100% saves can be replayed in BFW1.14. This is probably not true as some saves may be corrupted.
    But if such a problem happens, this is a bug in BFW1.14 save/replay that is better reported to the development team.
    1.12 saves:
    93.7% of saves are replaying OK (283 out of 302 saves from ladder records from 2015/04/12 to 2017/06/28).
    1.10 saves:
    89.6% of saves are replaying OK (467 of 521 saves from ladder records from 2012/01/30 to 2013/04/30).
    1.8 saves:
    86.3% of saves are replaying OK (145 of 168 saves from ladder records from 2011/02/17 to 2012/02/01).
    1.6 saves:
    81.5% of saves are replaying OK (53 of 65 saves from ladder records from 2009/03/26 to 2009/08/04).
    1.4 saves:
    84.7% of saves are replaying OK (61 of 72 saves from ladder records from 2008/07/17 to 2009/03/22).
  • This version supports you to:
    - load replays from different versions in sequence
    - observe and play on MP server from a newly started wesnoth
    This version does not support you to:
    - observe or play on MP server right after you loaded replays from a different version (you will get MP OOS)


    So while you may have only one version of BFW1.14, it should be restarted between backward replay mode and play/obs on MP server.

--- Original opening post as of July 2nd, 2021 ---
Original title: Questions about replays, random seed, recruit and unit checksum

I am trying to load some 1.12 replays in wesnoth 1.14.14.
It raises a couple of OOS that I have trouble to understand. Maybe someone is knowledgeable to help?

First question: how many random calls to recruit a unit?
In BFW1.12, it seems all "drake faction" units are recruited with 14 calls to random.
In BFW1.14, saurians are still recruited with 14 seeds but drakes are taking 22 seeds.
I look at changes in Glider/Clasher cfg, the difference is new alternative portraits but they seem not related to the number of call to randomness (if removed, still 22). I think the unit facing on recruit may be random in BFW 1.14 but when that call is removed from src/units/unit.cpp, the number of calls is still 22 (saurians have new mirror portraits too anyway). They went from Markov chain names to Context-free grammar names.

Second question: does the existence of OOS impact the recruited unit during the replay?
In seems that in BFW1.14 an OOS in replay will lead to all recruited units to have "intelligent, resilient" traits if checksum OOS is raised. They all then have the same checksum which is the one reported in the OOS, which may show that the traits may not originate from OOS (hence the following question). That is unlikely.

Third question: is is there any incompatibility in random seed (new_seed in replays) between the two versions?
A replay from BFW1.12 is giving seed as integer while the seed for BFW1.14 is a string. Yes, generated by different algorithms.

Fourth question: are random choices made in music, map scenery, alternative portraits used in help/dialog... impacting the random seed before first recruit?
I guess it should not, as the result of these random choices are not part of the replay data. It doesn't matter really.

Fifth question: what would be a possible design to embed the different RNG in the current code and select one based on replay version? Here is the idea I got.
Attachments
BFW1.14.load_1.4_replay_backward_support.v3.diff.gz
Apply over clean BFW1.14 for 1.12-1.4 replays, patch -p2, build with cmake
(88.48 KiB) Downloaded 28 times
troubles-1.4.md
Known problems with 1.12-1.4 replays and status
(35.92 KiB) Downloaded 48 times
Last edited by demario on September 18th, 2021, 11:59 pm, edited 10 times in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

/me wrote: Third question: is is there any incompatibility in random seed (new_seed in replays) between the two versions?
A replay from BFW1.12 is giving seed as integer while the seed for BFW1.14 is a string.
So it seems that the incompatibility must be broken down in two parts:
- integer vs string format: BFW 1.14 is still using integer to initialize the random pool and the "string" is just the hexadecimal encoding of that integer.
- if integer seeds from BFW 1.12 replays are encoded in hexadecimal format, there is still OOSes. Beside the seed value, the algorithm using that seed may be incompatible. If different algorithms are provided the same seed, they will generate different numbers.

It seems there are at least four "incompatible" random engines:
- BFW 1.14 (mt_rng, based on boost random?)
- BFW 1.12 (random_new, based on standard C rand()?)
- BFW 1.10 (simple_rng, based on a specific generator with unsigned random pool)
- BFW 1.8 and earlier (simple_rng, based on the same specific generator but with signed random pool)

It appears that both BFW 1.14 and BFW 1.12 are using different seeds through the game. The seeds are stored in replay (the drawn numbers are not).
But BFW 1.10 and BFW 1.8 are using one single seed for the whole game Well the actual logic seems more complicated than that.
BFW 1.10 and BFW 1.8 are using a single seed for all recruit actions. The drawn numbers are stored in saved game but actual values are replaced by values from replay if not matching ("no check" draws), which means no OOS. One the other hand on each attack. the seed is reset to the value stored in replay while the drawn numbers are used without being checked and OOS happen on die event. As one seed is reset, and the other one is kept, it means there are 2 RNG engines: replay contains a rng generator_ and game_state contains another one cailed rng_.

So it seems unlikely to be able to replay an old save using the current random generator and we should use the RNG matching with version of the replay.
Last edited by demario on September 11th, 2021, 9:32 am, edited 5 times in total.
"simply put, it's an old game"T — Cackfiend
gnombat
Posts: 396
Joined: June 10th, 2010, 8:49 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by gnombat »

:? It seems a lot simpler just to keep an old version of the game installed to view old replays...
Soliton
Site Administrator
Posts: 1650
Joined: April 5th, 2005, 3:25 pm
Location: #wesnoth-mp

Re: Questions about replays, random seed, recruit and unit checksum

Post by Soliton »

You're focusing a lot on random number generation as the source of OOS but surely there are countless other reasons. There is generally no effort spent to keep compatibility between different stable versions.
"If gameplay requires it, they can be made to live on Venus." -- scott
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

Soliton wrote: January 17th, 2021, 9:43 pm You're focusing a lot on random number generation as the source of OOS but surely there are countless other reasons.
I will clarify the limits of my endeavor: I am not interested in recovering local campaign replays to keep playing in the next version.
I am only interested in multiplayer game replays. By multiplayer, I don't consider loading the replay on the server which is excluding OOS from networking. I mean the passive replay we are viewing from "Load game" dialog on the main screen.

I only plan to load existing replays which is basically limit the scope to:
- stable versions of wesnoth (with minor version as even number)
- openly available 1v1 replays (which mainly begun at version BFW 1.4 for ladder game and BFW 1.6 for non-ladder)
I also don't consider user add-ons to focus on default scenarios (which I expect to be maintained and updated to newest version).

With these limitations, I believe if the replays can be loaded and recruits got right properties (MP, HP, XP, damage, def/resists, ...), the only cause of OOS left is numbers from RNG to be wrong. But I am probably wrong myself as I might find out in a near future.
"simply put, it's an old game"T — Cackfiend
Soliton
Site Administrator
Posts: 1650
Joined: April 5th, 2005, 3:25 pm
Location: #wesnoth-mp

Re: Questions about replays, random seed, recruit and unit checksum

Post by Soliton »

There is not much difference between network OOS and replay OOS. Multiplayer pretty much consists of sending ReplayWML around.

Simple properties are not everything. Every action needs to work as in the version the replay is from. Occasionally how stuff works is changed and that is not necessarily just contained in WML/lua. How random numbers are used is one thing you already noticed.

To give an ancient example: Drain used to be able to "heal" units above max HP. A replay played with that logic will unlikely work now. A little more recently drain was changed to only drain as much as the victim had HP left. (It always drained half damage regardless before.) Same outcome, old replays will unlikely work anymore since the amount of HP drained does not match. (They might if there are no draining units in the replay or the amount of drained HP never made a significant difference in the game...)

Same thing for trait changes etc etc.
"If gameplay requires it, they can be made to live on Venus." -- scott
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

Soliton wrote: January 18th, 2021, 6:45 pm To give an ancient example: Drain used to be able to "heal" units above max HP. A replay played with that logic will unlikely work now. A little more recently drain was changed to only drain as much as the victim had HP left. (It always drained half damage regardless before.) Same outcome, old replays will unlikely work anymore since the amount of HP drained does not match. (They might if there are no draining units in the replay or the amount of drained HP never made a significant difference in the game...)
Thanks for the details, I was not aware how the drains logic was changed in the past.
Still it might not changed the technique to reach the goal. It is clear that old replays can't be used in new version using the current stats of the units.
It stands for simple changes as damage, MP, number of traits, ... It can only be run using the stats at the time of the replay.
That is where targeting only stable version replay works best because no change in stats for any minor version update.
For example, the orcish archer had his ranged attack changed in version 1.10 (bow boosted at 6-3), so if I want to use a 1.8 replay, I need to use the previous stats:

Code: Select all

#textdomain wesnoth-units
[unit_type]
    id=Orcish Archer-1.8
    name="Orcish Archer-1.8"
    [base_unit]
        id=Orcish Archer-1.10
    [/base_unit]

    advances_to=Orcish Crossbowman-1.8

    hide_help=yes
    do_not_list=yes

    [attack]
        name=dagger
        description=_"dagger"
        icon=attacks/dagger-orcish.png
        type=blade
        range=melee
        damage=3
        number=2
    [/attack]
    [attack]
        name=bow
        description=_"bow"
        icon=attacks/bow-orcish.png
        type=pierce
        range=ranged
        damage=5
        number=3
    [/attack]
[/unit_type]
I believe the example of drains change could be handled the same (id=drains-1.x). Of course, more complex the change, harder is the code to write.
Last edited by demario on January 27th, 2021, 6:17 am, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

Soliton wrote: January 18th, 2021, 6:45 pm Occasionally how stuff works is changed and that is not necessarily just contained in WML/lua.
I see what you mean now. If I look at the code of the healthy trait, there is no parameter that gives any details on its behavior

Code: Select all

#define TRAIT_HEALTHY
    # Units with trait Healthy get 1 more HP plus 1 per level and always rest heal.
    [trait]
        id=healthy
        description= _ "Always rest heals"
        [effect]
            apply_to="healthy"
        [/effect]
    [/trait]
#enddef
It looks just like a marker that is caught in the code to trigger some behavior (+2HP each turn, -2 damage from poison).
If we want another behavior (BFW1.4 healthy is +2HP if no fight last turn and no effect on poison), we need to either change the code in a compatible way or implement the feature fully in WML. For the example of healthy trait, it seems that WML could be doing the job, lucky enough.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

/me wrote:
Soliton wrote: January 18th, 2021, 6:45 pm Occasionally how stuff works is changed and that is not necessarily just contained in WML/lua.
I see what you mean now. If I look at the code of the healthy trait, there is no parameter that gives any details on its behavior
If we want another behavior, we need to either change the code in a compatible way or implement the feature fully in WML.
That is what is making the RNG problem a bit special: you can change parameters of a unit type through WML. For abilities and traits, most can be implemented in WML/lua too, and for the worst ones you may have to write some new C code to support the feature.

For random number generator, it is impossible to write a new algorithm that it is generating the same numbers as an old one when fed with the old generator seeds. So the only thing you can do it to keep the old generator's code alive and integrate it to the current version so it takes over from the new generator to throw the same sequence of random numbers as it did when the game was played (condition for getting no OOS).
/me wrote: Fifth question: what would be a possible design to embed the different RNG in the current code and select one based on replay version?
Here is the idea I got to do this.

Below is a heavily manually edited diff file of the solution to focus on the solution "design" (more changes are needed to run).
The new RNG engine is keeping a reference to the old one and is delegating the requests for random numbers to it when the version in the config file matches the criteria for the switch.

Code: Select all

diff -r a29e9bfbe4c5 -r bf6a20f5e0be wesnoth-1.10.7/src/simple_rng.cpp
--- a/wesnoth-1.10.7/src/simple_rng.cpp	Wed Jan 27 09:29:04 2021 +0800
+++ b/wesnoth-1.10.7/src/simple_rng.cpp	Wed Jan 27 09:29:19 2021 +0800
@@ -29,21 +30,32 @@ int simple_rng::get_next_random()
 simple_rng::simple_rng(const config& cfg) :
 	random_seed_(cfg["random_seed"]),
 	random_pool_(random_seed_),
-	random_calls_(cfg["random_calls"].to_int(0))
+	random_calls_(cfg["random_calls"].to_int(0)),
+	first_version_("1.9.0"),
+	loaded_version_(cfg["version"]),
+	legacy_rng_()
 {
+	if(loaded_version_ < first_version_)
+	{
+		legacy_rng_ = simple_rng_1_8(cfg);
+	}
 	for ( unsigned calls = 0; calls < random_calls_; ++calls )
 		random_next();
 }
 
 int simple_rng::get_next_random()
 {
+	if(loaded_version_ < first_version_) return legacy_rng_.get_next_random();
 	random_next();
 	++random_calls_;
 	DBG_RND << "pulled user random " << random_pool_
@@ -82,15 +96,18 @@ simple_rng_1_8::simple_rng_1_8(const config& cfg)
 simple_rng_1_8::simple_rng_1_8(const config& cfg) :
 	random_seed_(cfg["random_seed"]),
 	random_pool_1_8_(random_seed_),
-	random_calls_(cfg["random_calls"].to_int(0))
+	random_calls_(cfg["random_calls"].to_int(0)),
+	first_version_("1.7.0")
 {
 	for ( unsigned calls = 0; calls < random_calls_; ++calls )
 		random_next();
 }
diff -r a29e9bfbe4c5 -r bf6a20f5e0be wesnoth-1.10.7/src/simple_rng.hpp
--- a/wesnoth-1.10.7/src/simple_rng.hpp	Wed Jan 27 09:29:04 2021 +0800
+++ b/wesnoth-1.10.7/src/simple_rng.hpp	Wed Jan 27 09:29:19 2021 +0800
@@ -112,6 +116,15 @@ class simple_rng
 
 	/** Sets the next random number in the pool. */
 	void random_next();
+
+	/** First version to use this RNG engine. */
+	version_info first_version_ = version_info("1.9.0");
+
+	/** The version for the RNG engine. */
+	version_info loaded_version_;
+
+	/** The legacy RNG engine use for old versions. */
+	simple_rng_1_8 legacy_rng_;
 };
 
 } // ends rand_rng namespace
I am not very happy with this as I think:
- the two RNG engines should inherit from one another
- the reference to the legacy engine legacy_rng_ would be better not to be typed so it could contain any simple_rng_1_10, simple_rng_1_12 ... in the future.

Maybe some C++ expert has some guidance to provide :whistle:

Anyway it is working and in attachment is the full version based on 1.10.7 code to load replays from 1.4 onward. Feel free to try and report any problem if you are lucky enough to be able to compile BFW 1.10 on your platform. Together with the list of issues for different versions of replay that have been fixed already (or not).
Attachments
troubles.txt
List of replay specific issues that have been fixed
(6.54 KiB) Downloaded 160 times
BFW1.10.7.load_replay_backward_support.diff.gz
You can "patch -p2" over version 1.10.7 of BFW. Build with cmake is working.
(45.04 KiB) Downloaded 163 times
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

/me wrote: First question: how many random calls to recruit a unit?
In BFW1.12, it seems all "drake faction" units are recruited with 14 calls to random.
In BFW1.14, saurians are still recruited with 14 seeds but drakes are taking 22 seeds.
So the situation is getting clearer now.
One of the differences between BFW1.12 and 1.14 is the change in names for different races in version 1.13.5. They went from Markov chain names to Context-free grammar names. In terms of random numbers it means it takes 20 random numbers for a name in BFW1.14 when it took 12 before the change. So that is the difference (8) between 14 and 22. Saurians still use markov-chain names in BFW1.14.
The second change in BFW1.14 is that nameless units (bats, undead, ...) are not calling the random number generator for name anymore (as they used to), which leads to some other warnings and side effects.
/me wrote: Second question: does the existence of OOS impact the recruited unit during the replay?
That seems unlikely. What happens is that the random numbers are generated "as expected by the replay" but are allocated by the current engine.
For example, if the engine need 20 numbers for a name and the replay is only having 16, all the replay numbers will be eaten by the name and the traits will not match the ones in the game.
Second situation is that the numbers are sometimes allocated in different sequence. For example in 1.6, the traits are created based on the last 2 generated random numbers while in version 1.8, the traits are generated before the name (which mean using 2 of the first 3 random numbers taking gender into account). So the 1.8 engine loading a 1.6 replay will force the traits to be created from the first numbers instead of the last.
These two things happening independently from which RNG generator is used.
/me wrote: Fourth question: are random choices made in music, map scenery, alternative portraits used in help/dialog... impacting the random seed before first recruit?
There are two situations.

In early versions (BFW1.4 to 1.10), the total of generated random numbers for recruit is higher than what is really needed (needed is: 12 for name and 2 for trait + 1 for gender if multiple genders available). But for those versions there was 14 (1 trait units without gender) to 16 numbers for each recruit. So there are a couple of numbers that are generated for something else (potentially the orientation of the sprite). But it doesn't matter because if the numbers are not matching, the engine will override its own RNG numbers by those in the replay.

In late versions (BFW1.12 to 1.14), the generated numbers matches the number needed for trait (0-2), gender (0-1) and name (0, 12 or 20). So we know where each generated number is going to and the answer is: no.
Which works well because in case of mismatch the engine can't override its own number (and replay doesn't store the original ones anymore). This creates a problem when loading 1.10 replay in BFW1.14, as the engine can't override output from its RNG while early replays kinda rely on that feature (AFAIK).
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

I made some progress on another front.
Here is There are the changes to apply over BFW1.14 (I used wesnoth-1.14.14) to be able to load replays from BFW1.12.

While most of the known replay breaking OOS are fixed, the unit checksum is still breaking and can't be disabled from WML (like removing the original checksum). So either you disable OOS for each load ("ingore all" + "no" in OOS dialog) or you can apply the second diff to disable the checksum check in the code. If disabling the checksum on unit creation, I advice to do it on an alternative wesnoth instance so that it stays enable in the instance you are using to play. If you chose to ignore all OOS, any replay breaking OOS would materialize as units never moving on player's turn (and rest-heal).

You better apply the changes in diff files from inside wesnoth directory using "patch -p2 ...". I got them compiled with cmake only. Let me know if it works for you. Together with the list of issues for different versions of replay that have been fixed already (or not).
Last edited by demario on March 11th, 2021, 7:24 am, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

Before making a pause on this, I am putting in attachment a compilation of information for comparison in the use of random between BFW1.10, 1.12, 1.14. It is likely making no sense at all and you shall open it at your own risk of brain damage. :mrgreen:

I am also going to record the current progress: the focus is on loading 1.10 replays on BFW1.12 as a preliminary step to load them in BFW1.14.
There is no outstanding issue at this time.

(solved) [random_seed] command
In BFW1.10, a recruit command executed after an attack (in the same side turn) is using the same seed as the one set for attack.
In BFW1.10 for an attack followed by a recruit the log from random is:

Code: Select all

20210226 08:53:20 info replay: up to replay action 127/778
20210226 08:53:20 debug random: Seeded random with 963713432 with 0 calls, pool is now at 963713432
20210226 08:53:20 info replay: Replaying attack with seed 963713432
20210226 08:53:20 debug replay: Attacker XP (before attack): 0
20210226 08:53:20 debug random: pulled user random 1317299697 for call 1 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3470067158 for call 2 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3523141719 for call 3 with seed 963713432
20210226 08:53:20 debug random: pulled user random 2888447812 for call 4 with seed 963713432
20210226 08:53:20 debug random: pulled user random 761819181 for call 5 with seed 963713432
20210226 08:53:20 debug random: pulled user random 1467197794 for call 6 with seed 963713432
20210226 08:53:20 debug replay: Attacker XP (after attack): 1
20210226 08:53:20 debug replay: expected_advancements.size: 0

20210226 08:53:20 info replay: up to replay action 129/778
20210226 08:53:20 debug random: pulled user random 1571236595 for call 7 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3124069296 for call 8 with seed 963713432
20210226 08:53:20 debug random: pulled user random 554230313 for call 9 with seed 963713432
20210226 08:53:20 debug random: pulled user random 442956206 for call 10 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3900670799 for call 11 with seed 963713432
20210226 08:53:20 debug random: pulled user random 1986472668 for call 12 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3676600293 for call 13 with seed 963713432
20210226 08:53:20 debug random: pulled user random 1261241018 for call 14 with seed 963713432
20210226 08:53:20 debug random: pulled user random 249234795 for call 15 with seed 963713432
20210226 08:53:20 debug random: pulled user random 1789532360 for call 16 with seed 963713432
20210226 08:53:20 debug random: pulled user random 816426337 for call 17 with seed 963713432
20210226 08:53:20 debug random: pulled user random 622366854 for call 18 with seed 963713432
20210226 08:53:20 debug random: pulled user random 354585927 for call 19 with seed 963713432
20210226 08:53:20 debug random: pulled user random 3534485876 for call 20 with seed 963713432
20210226 08:53:20 error engine: SYNC: In recruit Orcish Archer_1_10: has checksum ...
The logs show clearly that the same seed is used for both with random numbers for recruit being generated by pulling 14 additional random calls from the original attack seed.

For BFW1.12, each recruit command must provide its related random_seed using dependent=yes command. The direct translation of the recruit command in 1.12 syntax is not working as seed for recruit is taken as a new_seed which means that the numbers are pulled by the first 14 calls from that same seed again instead of taking the 14 numbers following the previous attack calls. As a consequence the first 6 numbers for the recruit commands are a repetition of the ones used in last attack.
The fix is to create a new syntax for the dependent random_seed command linked to recruit to add a random_calls attribute. The value of this attribute is set with the number of previous calls on the seed coming from the attack command. It looks like:

Code: Select all

[command]
	sent = 
	[recruit]
		type = Orcish Archer_1_10
		x = 17
		y = 5
		[from]
			x = 14
			y = 4
		[/from]
	[/recruit]
[/command]
[command]
	dependent = yes
	from_side = server
	sent = yes
	[random_seed]
		current_seed = 963713432
		random_calls = 6
	[/random_seed]
[/command]
This solution has some limitation. Sometimes the attack command doesn't contain the list of random numbers that were generated, making it impossible to know how many times the RNG has been called to resolve the attack. The next best guess is to assume the number of calls equals the total of strikes for both attacker and defender weapon of choice (as present in the command).
This backup calculation would lead to the wrong result in case of:
- one of the unit dying during the combat (except at last strike) as the dying output is also stored in the [random] part of command
- one of the unit has an ability to make the combat to repeat (berserk, fury ...)

In these cases, I found no method to predict the state of generator for next recruit which would probably lead later to an OOS (most often when target hex is not reached due to missing quick trait).

(solved) [move] command
In BFW1.10, move commands are always successful and the final target is reached. Here is how BFW1.10 is recording an interrupted move (moving unit sights an enemy when reaching 7,19 in middle of its way to final target 7,18):

Code: Select all

	[command]
		undo=""
		[move]
			x="7,7,7"
			y="20,19,18"
		[/move]
	[/command]
In BFW1.12 however, the move command may be interrupted by sighting an enemy unit and this event is recorded in the savegame. Here is how BFW1.12 is recording a similar situation (moving unit sights an enemy when reaching 7,15 in middle of its way to final target 8,14):

Code: Select all

	[command]
		[move]
			x="7,7,7,8"
			y="17,16,15,14"
		[/move]
		[checkup]
			[result]
				final_hex_x=7
				final_hex_y=15
				stopped_early=yes
			[/result]
		[/checkup]
	[/command]
	[command]
		[move]
			x="7,8"
			y="15,14"
		[/move]
		[checkup]
			[result]
				final_hex_x=8
				final_hex_y=14
				stopped_early=no
			[/result]
		[/checkup]
	[/command]
Here the move interrupt is explicit in the replay and the full move takes 2 commands.

Consequently when BFW1.12 is reading the 1.10 command, it assumes that the move has stopped_early=no and catches an inconsistency between the record (which says that the final target is reached) and its own record (that the movement is interrupted from sighting units). This inconsistency leads to the unit never reaching its target leading to an OOS (missing source for attack/movement) soon after.

The way to fix this issue would be to add to BFW1.12 move command an attribute to tell that all movements should be completed without check is to add skip_sighted="all" attribute to the move command.

So this is where I'll sit for a while, chilling out in my camp on the shores of the WML Rubicon :evil:
Attachments
rng_comparison.md
Attempt at summarizing differences in RNG 1.10-1.14
(19.39 KiB) Downloaded 142 times
Last edited by demario on March 11th, 2021, 12:26 am, edited 3 times in total.
"simply put, it's an old game"T — Cackfiend
User avatar
Celtic_Minstrel
Developer
Posts: 1839
Joined: August 3rd, 2012, 11:26 pm
Location: Canada
Contact:

Re: Questions about replays, random seed, recruit and unit checksum

Post by Celtic_Minstrel »

demario wrote: February 3rd, 2021, 1:29 am So the situation is getting clearer now.
One of the differences between BFW1.12 and 1.14 is the change in names for different races in version 1.13.5. They went from Markov chain names to Context-free grammar names. In terms of random numbers it means it takes 20 random numbers for a name in BFW1.14 when it took 12 before the change. So that is the difference (8) between 14 and 22. Saurians still use markov-chain names in BFW1.14.
This is not accurate as far as I know. Saurians are using a context-free grammar as well (unless there's a bug preventing it from being loaded by the engine). The number of random calls is also not fixed with CFG and depends on the grammar. In fact, looking at the drake and saurian grammars, I suspect 14 calls is possible for both of them, and likewise 22 calls is possible for both of them, though that's speculation based on their similar structure. Naively I would've expected only 3 or 4 calls for each drake or saurian name; maybe the inflated count is a result of the effort to avoid duplicate names?
demario wrote: February 3rd, 2021, 1:29 amneeded is: 12 for name
Where do you get this 12 from?
Author of The Black Cross of Aleron campaign and Default++ era.
Maintainer of Steelhive.
demario
Posts: 55
Joined: July 3rd, 2019, 1:05 pm

Re: Questions about replays, random seed, recruit and unit checksum

Post by demario »

I have updated the status.
Celtic_Minstrel wrote: March 2nd, 2021, 6:39 pm
demario wrote: February 3rd, 2021, 1:29 am Saurians still use markov-chain names in BFW1.14.
This is not accurate as far as I know. Saurians are using a context-free grammar as well (unless there's a bug preventing it from being loaded by the engine).
I have spent enough time on this to say that it is more likely to be a bug than me being wrong. :roll:
If you give old German names to the drake faction units, you can see for yourself which ones are more likely to be named Edwin, Reinhold, Friedrich or Winfried.

The fact is that the name_generator=_ <<...>> does not work while the male_name_generator=_ <<...>> works well.
Celtic_Minstrel wrote:The number of random calls is also not fixed with CFG and depends on the grammar. [...] Naively I would've expected only 3 or 4 calls for each drake or saurian name; maybe the inflated count is a result of the effort to avoid duplicate names?
I am not sure which information you rely on to make these statements. This is not how I read the code (src/utils/markov_generator.cpp, src/utils/context_free_grammar_generator.cpp). In BFW1.14, for both markov and context-free generators the first thing the code does when generating a name is filling a vector with an arbitrary number maxlen/seed_size of random numbers. The number of calls to RNG is constant, only depending on the generator used. That is what the log and the savegame are confirming.
Celtic_Minstrel wrote:
demario wrote: February 3rd, 2021, 1:29 amneeded is: 12 for name
Where do you get this 12 from?
From the code.
Attachments
old_german_drakes.diff
Old German names for drakes for testing
(2.69 KiB) Downloaded 137 times
Last edited by demario on March 5th, 2021, 10:38 pm, edited 1 time in total.
"simply put, it's an old game"T — Cackfiend
User avatar
Celtic_Minstrel
Developer
Posts: 1839
Joined: August 3rd, 2012, 11:26 pm
Location: Canada
Contact:

Re: Questions about replays, random seed, recruit and unit checksum

Post by Celtic_Minstrel »

demario wrote: March 5th, 2021, 1:22 am
Celtic_Minstrel wrote:The number of random calls is also not fixed with CFG and depends on the grammar. [...] Naively I would've expected only 3 or 4 calls for each drake or saurian name; maybe the inflated count is a result of the effort to avoid duplicate names?
I am not sure which information you rely on to make these statements.
That was just based on inspecting the grammar and seeing that you need to make 3 or 4 choices to fully substitute it.
demario wrote: March 5th, 2021, 1:22 amThis is not how I read the code (src/utils/markov_generator.cpp, src/utils/context_free_grammar_generator.cpp). In BFW1.14, for both markov and context-free generators the first thing the code does when generating a name is filling a vector with an arbitrary number maxlen/seed_size of random numbers. The number of calls to RNG is constant, only depending on the generator used.
Ah, I see, so they use more random numbers than actually needed in most cases.

Based on the code you posted, you're still simplifying it slightly - with Markov chains, the number of random calls is equal to the chain length (12 by default, but can be set in WML), while for CFG it's a minimum of 20 but could be more if the grammar is exceptionally complex. I guess that's irrelevant to mainline units though.
Author of The Black Cross of Aleron campaign and Default++ era.
Maintainer of Steelhive.
Post Reply