[Engine feedback and testing] Automatic testing, in order to eliminate warnings between different game engines
Moderator: Forum Moderators
[Engine feedback and testing] Automatic testing, in order to eliminate warnings between different game engines
Right now I am testing the latest stable (I think),
which is wesnoth 1.18.3.
For reference purposes here is the copy/paste from
running wesnoth version feedback via the commandline,
on my Linux box:
Battle for Wesnoth v1.18.3 x86_64
Started on Mon Dec 2 16:09:52 2024
Battle for Wesnoth 1.18.3
Library versions:
Boost: 1.86
Lua: 5.4.6
OpenSSL/libcrypto: 3.4.0-dev (runtime 3.4.0-dev)
libcurl: 8.11.0 (runtime 8.11.0)
Cairo: 1.18.0 (runtime 1.18.0)
Pango: 1.54.0 (runtime 1.54.0)
SDL: 2.30.9 (runtime 2.30.9)
SDL_image: 2.6.3 (runtime 2.6.3)
SDL_mixer: 2.8.0 (runtime 2.8.0)
Optional features:
Lua console completion: yes
D-Bus notifications back end: yes
So far things work mostly fine. I only had one odd freeze
of wesnoth (IceWM had to kill it), and a few minor warnings.
But most things seem to be quite fine, which is nice. I used
to have problems in particular with boost in the past. Anyway,
this is just generic feedback, now more to the thread here.
I am currently re-playing To Lands unknown. Here also things
work fine, except for a small "obsolete unit ID" or a similar
message, appearing on the top left. It seemed to be a harmless
warning or something (so I ignored it mostly; I don't even recall
the exact message content), but for a player this may still be a
little bit of a shock, because ... such a warning could indicate that
something is not working. If it is even important. But if it is important,
the player may not know what to do about ... so it is still a bit of a
shock to some players I think, in particular the very panicky player
type who may freeze and be unable to continue to play (ok ok,
these may not exist, but imagine like the fainting goats, a player
may stare at the warnings open-wide eyed ... for many seconds ...
that thought is scary to me. Me, on the other hand, I gloriously
ignore everything until it really doesn't work, but usually things
actually work, so I just continue).
Now I assume, and please correct me if I am wrong, that something
may have changed in the WML/lua part, so possibly something
in To Lands unknown has to be updated. But ... I was also thinking ...
Could we enable automatic tests, to ensure such an upgrade may
happen, especially WITHOUT players having to go through and
report that?
Here I am thinking something along those lines:
- ALL input given by the player, could also happen via a script,
aka automated program, and the SAME ISSUES that a player
may find, could ALSO be discovered by a script/bot doing the
same actions 1:1. So, all those reported warnings for a player,
could in theory be generated, and reported by the game engine
as it were to play against "itself". So we'd have a list of all those
outdated WML entries or something; and once we have that data,
we could easily report it, and have campaign authors or other
people update the things that should be updated.
I saw in another forum post that someone used recursive grep
to find outdated entries, and while I do that on my local linux
machine a lot too, I feel that wesnoth could make use of more
automated testing as such.
I am not saying everything should be tested necessarily, but in
particular things such as outdated WML and lua data could be
tested more systematically.
This should also make upgrades to major version changes more
easily, e. g. we could run a script, that script does some automatic
tests as if a player were to play through all campaigns multiple
times, and all reported issues could be automatically gathered and
stored into a report-file that hold all the specifices, e. g. "in Campaign
WhackTheNoodle, in scenario KeepTheNoodleAlive, in unit type
NoodleEatingWorm, trait Epicness 2.0 is deprecated. Consider
adjusting to the new trait Shyness". Or something like that. (And if
a trait/ability is removed, this could also be reported, same with unit
IDs and so forth.)
Anyway this is meant mostly as feedback. I did not feel this would be
a good github issue, as this depends a bit more on devs agreeing on
tests being useful, as opposed to indicating errors or warnings to
unsuspecting users who are not quite sure what to do with such
warnings.
which is wesnoth 1.18.3.
For reference purposes here is the copy/paste from
running wesnoth version feedback via the commandline,
on my Linux box:
Battle for Wesnoth v1.18.3 x86_64
Started on Mon Dec 2 16:09:52 2024
Battle for Wesnoth 1.18.3
Library versions:
Boost: 1.86
Lua: 5.4.6
OpenSSL/libcrypto: 3.4.0-dev (runtime 3.4.0-dev)
libcurl: 8.11.0 (runtime 8.11.0)
Cairo: 1.18.0 (runtime 1.18.0)
Pango: 1.54.0 (runtime 1.54.0)
SDL: 2.30.9 (runtime 2.30.9)
SDL_image: 2.6.3 (runtime 2.6.3)
SDL_mixer: 2.8.0 (runtime 2.8.0)
Optional features:
Lua console completion: yes
D-Bus notifications back end: yes
So far things work mostly fine. I only had one odd freeze
of wesnoth (IceWM had to kill it), and a few minor warnings.
But most things seem to be quite fine, which is nice. I used
to have problems in particular with boost in the past. Anyway,
this is just generic feedback, now more to the thread here.
I am currently re-playing To Lands unknown. Here also things
work fine, except for a small "obsolete unit ID" or a similar
message, appearing on the top left. It seemed to be a harmless
warning or something (so I ignored it mostly; I don't even recall
the exact message content), but for a player this may still be a
little bit of a shock, because ... such a warning could indicate that
something is not working. If it is even important. But if it is important,
the player may not know what to do about ... so it is still a bit of a
shock to some players I think, in particular the very panicky player
type who may freeze and be unable to continue to play (ok ok,
these may not exist, but imagine like the fainting goats, a player
may stare at the warnings open-wide eyed ... for many seconds ...
that thought is scary to me. Me, on the other hand, I gloriously
ignore everything until it really doesn't work, but usually things
actually work, so I just continue).
Now I assume, and please correct me if I am wrong, that something
may have changed in the WML/lua part, so possibly something
in To Lands unknown has to be updated. But ... I was also thinking ...
Could we enable automatic tests, to ensure such an upgrade may
happen, especially WITHOUT players having to go through and
report that?
Here I am thinking something along those lines:
- ALL input given by the player, could also happen via a script,
aka automated program, and the SAME ISSUES that a player
may find, could ALSO be discovered by a script/bot doing the
same actions 1:1. So, all those reported warnings for a player,
could in theory be generated, and reported by the game engine
as it were to play against "itself". So we'd have a list of all those
outdated WML entries or something; and once we have that data,
we could easily report it, and have campaign authors or other
people update the things that should be updated.
I saw in another forum post that someone used recursive grep
to find outdated entries, and while I do that on my local linux
machine a lot too, I feel that wesnoth could make use of more
automated testing as such.
I am not saying everything should be tested necessarily, but in
particular things such as outdated WML and lua data could be
tested more systematically.
This should also make upgrades to major version changes more
easily, e. g. we could run a script, that script does some automatic
tests as if a player were to play through all campaigns multiple
times, and all reported issues could be automatically gathered and
stored into a report-file that hold all the specifices, e. g. "in Campaign
WhackTheNoodle, in scenario KeepTheNoodleAlive, in unit type
NoodleEatingWorm, trait Epicness 2.0 is deprecated. Consider
adjusting to the new trait Shyness". Or something like that. (And if
a trait/ability is removed, this could also be reported, same with unit
IDs and so forth.)
Anyway this is meant mostly as feedback. I did not feel this would be
a good github issue, as this depends a bit more on devs agreeing on
tests being useful, as opposed to indicating errors or warnings to
unsuspecting users who are not quite sure what to do with such
warnings.