WML syntax tree analysis

Discussion of all aspects of the game engine, including development of new and existing features.

Moderator: Forum Moderators

Post Reply
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

WML syntax tree analysis

Post by Ravana »

Repository: https://github.com/ProditorMagnus/WML_tree_tools

I am creating a tool, that can read addon folder, and return information on where certain defined WML patterns exist. My first use case is finding events without id.

I use wesnoth -p and wmlparser3.py.

Idea for this is from

Code: Select all

20170528 20:08:35< zookeeper> well, it seems really hard to get a sense of how often death animations have weapon filtering in UMC
..
20170528 21:28:11< DeFender1031> zookeeper, I'm wondering if there's a simple way to grep for that... (Probably not.)
..
20170528 21:29:43< celticminstrel> DeFender1031: It's impossible to grep for.
..
20170528 21:47:56< DeFender1031> what we really need is some kind of tool that is capable of parsing WML that you can basically give parameters to saying "search for anywhere where a WML tag that matches these criteria exists"
20170528 21:48:46< celticminstrel> That sounds doable with the C++ WML API.
20170528 21:48:51< DeFender1031> (That actually shouldn't be that hard when there's already a WML parser... The biggest issue is having some clean way to define the criteria")
20170528 21:48:55< celticminstrel> Well probably even with Lua.
20170528 21:49:25< celticminstrel> Except that Lua can't access the unfiltered game config.
20170528 21:49:39< DeFender1031> I'd suggest a separate tool.
20170528 21:49:46< celticminstrel> (By which I mean the root config loaded from data/_main.cfg)
20170528 21:49:55< DeFender1031> right
20170528 21:50:03< celticminstrel> (I guess the addon configs are also merged into it.)
Nothing works correctly yet, but it has already been useful in finding some real errors in ageless.
gfgtdf
Developer
Posts: 1431
Joined: February 10th, 2013, 2:25 pm

Re: WML syntax tree analysis

Post by gfgtdf »

didn't iceiceice at some point wrote a wml parser that was somhow better then wesnoths internal wml parser? Maybe you coudl use that?

EDIT: this is the thread i think viewtopic.php?f=10&t=44584
Scenario with Robots SP scenario (1.11/1.12), allows you to build your units with components, PYR No preperation turn 1.12 mp-mod that allows you to select your units immideately after the game begins.
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

I dont know how to compile wesnoth, so that does not help me much.

I found that normal core preprocessing skips some macros, therefore I need to use SKIP_CORE, and first collect all core macros.

Code: Select all

call([wesnoth_path, "--preprocess-defines", "MULTIPLAYER,SKIP_CORE", "-p", core_path, core_out_path, "--preprocess-output-macros"])

call([wesnoth_path, "--userdata-dir", userdata_path, "--preprocess-defines", "MULTIPLAYER,SKIP_CORE", "--preprocess-input-macros", join(core_out_path, "_MACROS_.cfg"), "-p", input_path, addon_out_path])
Current progress is support for most basic querys, like finding all unit types with attack of 25 damage.
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

Now supports:
query without full path;
attribute check with comparison operators, implemented as lambda, so could be used with any function;
way to select which attributes to include in output

Some selection examples:

Select id, experience, level from [unit_type]s that are under toplevel [units] and have attribute experience=100

Code: Select all

[units]/[unit_type]/experience==100~id,experience,level
Select all [movetype] names

Code: Select all

//[movetype]/name~name
Select used movetypes

Code: Select all

[units]/[unit_type]/movement_type~movement_type
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

Now supports query with attributes from different tag levels

Code: Select all

parsed_query = [parse_wml_query("[units]/[unit_type]/[attack]/damage>15"),
                parse_wml_query("[units]/[unit_type]/hitpoints==70")]
output_keys = ["id", "cost", "hitpoints"]
Supports some negation. Currently trying to say with multiple conditions that attribute should not exist in non-terminal(from the first query point of view) node passes when there is any node in path where such attribute does not exist.

Code: Select all

parsed_query = [parse_wml_query("[units]/[unit_type]/[event]/id!")]
output_keys = ["id", "name", "first_time_only"]
https://github.com/ProditorMagnus/WML_t ... out_id.cfg
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

Continuing with this to find possible typos. https://github.com/ProditorMagnus/WML_t ... 53c03701R8 reports uncommon damage types.
User avatar
Celtic_Minstrel
Developer
Posts: 2158
Joined: August 3rd, 2012, 11:26 pm
Location: Canada
Contact:

Re: WML syntax tree analysis

Post by Celtic_Minstrel »

I notice you've hardcoded your personal Wesnoth path into those files... and what the heck is "ghype"?

That aside, this definitely seems useful.
Author of The Black Cross of Aleron campaign and Default++ era.
Former maintainer of Steelhive.
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

ghype is developing new era for future inclusion for ageless, using that to refer to his version.

compile_ageless.py is supposed to hardcode paths, it is used to build two command line calls that would be too long normally. find_patterns.py can be changed though.
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

Added some changes to make it easier to use, and with different addons as well.

Now only one file has absolute paths, others use relative paths.
User avatar
josteph
Inactive Developer
Posts: 741
Joined: August 19th, 2017, 6:58 pm

Re: WML syntax tree analysis

Post by josteph »

Nice :)

I usually use wmlparser3 in json output mode and then pipe the output to jq.
Ravana wrote: June 5th, 2017, 8:24 pm Select id, experience, level from [unit_type]s that are under toplevel [units] and have attribute experience=100

Code: Select all

[units]/[unit_type]/experience==100~id,experience,level

Code: Select all

data/tools/wesnoth/wmlparser3.py -w ./wesnoth -i data/_main.cfg -j | jq '.units[0].unit_type | map(select(.experience = 100)) | map({id, experience, level})'
Ravana wrote: June 5th, 2017, 8:24 pm Select all [movetype] names

Code: Select all

//[movetype]/name~name

Code: Select all

data/tools/wesnoth/wmlparser3.py -w ./wesnoth -i data/_main.cfg -j | jq '.units[0].movetype | map(.name)'
Ravana wrote: June 5th, 2017, 8:24 pm Select used movetypes

Code: Select all

[units]/[unit_type]/movement_type~movement_type

Code: Select all

data/tools/wesnoth/wmlparser3.py -w ./wesnoth -i data/_main.cfg -j | jq '.units[0].unit_type | map(.movement_type) | map(select(. != null)) | sort | unique'
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

Interesting. Looks like it would support all normal cases. Technically I support selecting with stateful function too, but it is not what I use in practice. Select all units that have more health than the last match - https://github.com/ProditorMagnus/WML_t ... 6b0c2ecfda gives

Code: Select all

[[['units', 'unit_type'], ['hitpoints', <function inc at 0x053A00C0>], ['id']]]
found match at ['units', 'unit_type'] [{}, {'id': 'AE_arc_ukians_Royal_Constable', 'hitpoints': '70'}]
found match at ['units', 'unit_type'] [{}, {'id': 'AE_arc_ukians_Royal_FireKnight', 'hitpoints': '88'}]
found match at ['units', 'unit_type'] [{}, {'id': 'AE_arc_menagerie_Green_Mage', 'hitpoints': '444'}]
found match at ['units', 'unit_type'] [{}, {'id': 'AE_feu_khaganate_Rigid', 'hitpoints': '555'}]
User avatar
josteph
Inactive Developer
Posts: 741
Joined: August 19th, 2017, 6:58 pm

Re: WML syntax tree analysis

Post by josteph »

You could probably this with with_entries or reduce, but if you don't want to learn a new DSL just for this, you could use Python for the stateful part:

Code: Select all

jq '.units[0].unit_type | map({ id, hitpoints })' | jq -c '.[]' | perl -MJSON::PP -nE '$h = decode_json $_; say $h->{id} if $h->{hitpoints} > $prev; $prev = $h->{hitpoints}'
User avatar
Ravana
Forum Moderator
Posts: 2934
Joined: January 29th, 2012, 12:49 am
Location: Estonia
Contact:

Re: WML syntax tree analysis

Post by Ravana »

I will remain with python. Even just being able to add breakpoints makes it worth it.
Post Reply