Add-ons registration API and UI proposal

Discussion among members of the development team.

Moderators: Forum Moderators, Developers

Post Reply
User avatar
shadowm
Site Administrator
Posts: 6367
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Add-ons registration API and UI proposal

Post by shadowm » August 1st, 2015, 11:00 pm

Note: this is a highly technical post and you are expected to be fully familiarized with general WML concepts and the WML syntax used in .pbl files before reading.


The Problem

How often has it happened to you that you have downloaded an add-on only to see it conflict with yours, be it in-game or in the map editor? After all, even with a vast number of potential terrain string combinations or virtually infinite possible unit type names, the shared namespace design pretty much guarantees that you will run into name (and sometimes functionality) collisions at some point.

Right now, there are two layers to the process of loading add-ons from _main.cfg files:
  • Reading the add-ons themselves, by loading WML from every <user data dir>/data/add-ons/_main.cfg file in its own preprocessor context with certain engine-defined preprocessor symbols (such as MULTIPLAYER).
  • Loading campaigns by repeating the process with add-on-specified (by a [campaign] data node) preprocessor symbols such as CAMPAIGN_AFTER_THE_STORM (define=CAMPAIGN_AFTER_THE_STORM), NORMAL (difficulties=EASY,NORMAL,HARD), and CAMPAIGN_AFTER_THE_STORM_EPISODE_I (extra_defines=CAMPAIGN_AFTER_THE_STORM_EPISODE_I). These symbols can be seen by every add-on installed since they are shared for each individual preproc context.

Code: Select all

#
# Highly simplified _main.cfg. Don't use as reference.
#

[campaign]
    define=CAMPAIGN_AFTER_THE_STORM
    difficulties=EASY,NORMAL,HARD
    extra_defines=CAMPAIGN_AFTER_THE_STORM_EPISODE_I
[/campaign]

#ifdef CAMPAIGN_AFTER_THE_STORM

{./common_campaign_stuff.cfg}

#ifdef MULTIPLAYER
{./mp_stuff.cfg}
#endif

#ifdef EDITOR
{./editor_stuff.cfg}
#endif

#ifdef CAMPAIGN_AFTER_THE_STORM_EPISODE_I
{./episode_1_stuff.cfg}
#endif

#endif
Symbols defined by the engine in either step can be “seen” by every add-on installed and since the parser output for each add-on is combined into a unique namespace, you get to see eras and their unit types inhabit the same space in MP game modes, sometimes resulting in problems beyond the player’s control (and I don’t mean just the preposterous load times). This is but one of the most glaring usability problems plaguing Wesnoth since its very inception, and it does not precisely help players (or content creators like me, when it comes to the map editor) feel confident about installing add-ons.

In an ideal world, every aspect of the game would be containerized so that every individual add-on would get its own unit types namespace, scenarios namespace, terrain definitions and terrain builder rules namespace, resource paths namespace, and so on. This might seem like an attractive idea to some (especially programmers), but let’s face it, the amount of engine work required for this would be tremendous and not pretty at all. For things like browsing the game’s help, selecting an era in MP, or picking terrain palettes in the map editor we would also need additional UI elements to support the concept, making an already crufty interface even cruftier. It doesn’t help that we are still stuck in 2004 and hand-coding most of the lobby and themable UI sidebar layout.

So what do I propose as an intermediate step? Promoting add-on metadata into a first-class citizen in order to facilitate the implementation of selective add-on loading.


The Solution

Add-ons right now only expose data. The game does not require anything from an add-on other than data. The most we have at the moment is a version and title for each add-on downloaded from the add-ons server, which requires the client to generate its own _info.cfg file with the requisite metadata borrowed from the server. If there is no such metadata, Wesnoth will still accept the add-on in all contexts, and fill the gaps (like the add-on names in the WML load error report or the Remove Add-ons dialog) with guesswork. This makes the underlying C++ code crummier, since there are contexts where we want to display the add-on titles, requiring us to look at _server.pbl/Addon_Name.pbl first (if present), _info.cfg second, and guess from the add-on file/dir name third. The same logic applies to version numbers, except that in the third step we just assume the default version value of 0.0.0.

So what we want to do here is have a user interface that lists the installed add-ons with their titles, descriptions, version information and all the other attributes you normally find in a .pbl file, which the server retains but the client does not. What we want to do with it is disable individual add-ons without removing them. We also want add-ons to declare their dependencies in two tiers (required and optional) with version requirements attached to them and have the game enforce these dependencies rather than assuming everything is optional like we do right now.

For this end we want to merge the metadata from _info.cfg and _server.pbl files into a single, mandatory add-on manifest file (say, _manifest.cfg) with all the information that both the add-ons server and client need to know, reducing the _server.pbl file’s role to that of an authentication information provider for uploads and nothing else.


The Syntax

Code: Select all

#
# <addon dir>/_server.pbl file for legacy auth scheme:
#
passphrase=<string>                 # Add-on upload passphrase
email=<string>                      # Author email

#
# <addon dir>/_manifest.cfg file:
#
[addon]
    id=<string>                     # Add-on id (formerly its file/dir name) [protocol attribute, not used by the client]
    title=<tstring>                 # Add-on title
    description=<tstring>           # Add-on description
    icon=<string>                   # Add-on icon

    version=<version>               # Version number.
    author=<string>                 # Authors/maintainers.

    type=<type>                     # Add-on type field.

    dependencies=<string>           # Old-style required dependencies (deprecated) (optional)
    core=<string>                   # Core (part of the undocumented 1.13.0 core mods functionality)
                                    # (deprecated, use requires instead) (optional)

    requires=<depends>              # Required dependencies (optional)
    recommends=<depends>            # Recommended dependencies (optional)

    mode=any/sp/mp/none             # Whether to try to load the add-on in all modes (default),
                                    # single-pĺayer only, multi-player only, or leave it to other
                                    # add-ons to try to include it directly (e.g. for resource packs).
                                    # Intended primarily as a time saver, may not be implemented.

    [feedback]
        topic_id=<string>           # Feedback topic number from forums.w.o (optional)
    [/feedback]

    [license]
        # ??? (TBD) (Also, hey, have a cookie if you read this far)
    [/license]
[/addon]
(Note: attributes marked as deprecated probably won’t be supported in 1.14.x at all.)

You will note that the add-on title and description attributes are suggested to be translatable strings. Local support for this is reasonably trivial for an installed add-on (although it may be needed to move the usual [textdomain] declaration from _main.cfg to _manifest.cfg), but it’s not currently clear how this will be implemented into the add-ons server so the actual implementation might not actually allow translatable strings here.

The <depends> value type is a more complicated syntax detail. For my first draft from years ago (that I originally intended to implement for 1.12) I suggested making every dependency its own WML node with attributes, but perhaps people might not like that. So, this time the proposal is as follows:

Code: Select all

[addon]
    name="After the Storm"
    # ...
    version=0.10.1
    requires="Shadowm_Addons_Framework >= 1.1.4"
    recommends="AtS_Music >= 0.2.0, Invasion_from_the_Unknown"
[/addon]
Dependencies are provided as a comma-separated list of ids, each optionally paired with a comparison operator (one of > >= < <= == !=) and a version number; if no operator and version number are provided for a given dependency, the client assumes all versions of the given add-on are accepted. All dependencies are forcibly AND’d, but at a later time it may become possible to OR them if anyone feels this might be useful. Finally, dependencies are to be verified only on the client side, both before downloading (to make sure they are satisfied by the add-ons server and the locally installed add-on versions) and after downloading (to make sure the player can’t disable add-ons that are required by other enabled add-ons). It may become possible to one or more checks in debug mode or when an Advanced Preferences option is enabled.

Enabling an add-on (the default out-of-the-box state) will result in the engine defining a ADDON:Addon_Id and ADDON_VERSION:Addon_Id preprocessor symbols for every _main.cfg preproc context during the WML load stage, allowing other add-ons to check whether a given add-on is enabled and which version is present using the usual #ifdef and #ifver directives. Whether and how some or all of this information is going to be exposed to Lua will be decided at a later time.

Code: Select all

#
# _main.cfg example.
#

#ifndef ADDON:Shadowm_Addons_Framework
    #error Show and error message or something.
#else
    #ifver ADDON_VERSION:Shadowm_Addons_Framework < 1.1.4
        #warning Maybe just tell the user they've inadvertently voided the warranty by doing something silly.
    #endif
#endif
_manifest.cfg will probably continue to be a parser-only file (meaning no preprocessor directives will be available) like its direct predecessors for the sake of simplicity and compatibility with the Python add-ons client (data/tools/wesnoth_addon_manager). Campaigns will continue to require one or more separate [campaign] declarations in _main.cfg (or a file included by it), but they are no longer guaranteed to be declared when the add-on is installed, only when it’s enabled by the player. Naturally, all add-ons will be enabled by default, and the on/off status of individual add-ons will be saved to an additional file in the user config dir; this has the side-effect of requiring add-on ids (dir names) to be valid WML attribute names, but this should hardly be a problem for anyone.


Who?

Of course, I’m taking full responsibility for the implementation of this proposal. The one aspect of the project that will probably take me the longest to implement is the tiered dependencies part, but I hope I’ll be able to implement it in a year’s time. The UI tasks involved are relatively trivial, and implementing the core functionality of the _manifest.cfg model is mostly just a matter of moving code around.

And in case you are wondering, no, there won’t be any compatibility with the previous _info.cfg/_server.pbl model at all, so add-on authors will need to provide both a _server.pbl and a _manifest.cfg for 1.13.x once the project is complete. Addon_Name.pbl and Addon_Name.cfg files will no longer be supported either, so as to keep the implementation as simple and clean as possible, as well as the accompanying documentation for content authors.

(Also note that, strictly speaking, there is no need to implement the _manifest.cfg model together with the add-on state UI. However, since this change will result in a consolidation of Wesnoth’s internal add-ons support API, it will allow me to implement the UI using cleaner code and without having to readapt it to the new model in a third tedious step.)

But of course, before I start working on this huge undertaking I want to know if people like the general idea or would rather do it in some other way, whether they have any suggestions or complaints about the proposed syntax, et cetera. I am also willing to answer all and any questions you might have. In fact, scratch that: I want to read and answer your questions in order to understand better what I am about to do and also keep myself motivated.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.
Elsewhere: shadowmBlogFollow me on Twitter

User avatar
doofus-01
Art Contributor
Posts: 3658
Joined: January 6th, 2008, 9:27 pm
Location: USA, the civilized part.

Re: Add-ons registration API and UI proposal

Post by doofus-01 » August 2nd, 2015, 1:38 am

Sorry if you specified this and I'm just not seeing it, but this user interface means that the user manually resolves conflicts, in some cases? Maybe I don't fully grasp the problem.

I get the MULTIPLAYER and EDITOR conflicts issue, and stuff that is included outside the #ifdef CAMPAIGN_AFTER_THE_STORM. So then does the proposed system load all data in all_manifest.cfgs, then later just not load anything that isn't specified in the single selected campaign/era/map _manifest.cfg?

I guess my confusion is over what the user interface is for.
BfW 1.12 supported, but active development only for BfW 1.13/1.14: Bad Moon Rising | Trinity | Archaic Era |
| Abandoned: Tales of the Setting Sun
GitHub link for these projects

User avatar
shadowm
Site Administrator
Posts: 6367
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: Add-ons registration API and UI proposal

Post by shadowm » August 2nd, 2015, 2:01 am

doofus-01 wrote:Sorry if you specified this and I'm just not seeing it, but this user interface means that the user manually resolves conflicts, in some cases? Maybe I don't fully grasp the problem.

I get the MULTIPLAYER and EDITOR conflicts issue, and stuff that is included outside the #ifdef CAMPAIGN_AFTER_THE_STORM. So then does the proposed system load all data in all_manifest.cfgs, then later just not load anything that isn't specified in the single selected campaign/era/map _manifest.cfg?

I guess my confusion is over what the user interface is for.
It still loads everything that is currently enabled (again, the default state). This is in part to still allow things like insert RNG tweak mod here to exist and work with existing campaigns, but also to keep the most common use case unchanged: reading [campaign] definitions for the Campaigns menu.

Disabling/enabling add-ons is intended to be used for manual conflict resolution for the editor and MP mods, as well as keeping load times or memory usage under control if you feel so inclined.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.
Elsewhere: shadowmBlogFollow me on Twitter

User avatar
pauxlo
Posts: 1017
Joined: September 19th, 2006, 8:54 pm

Re: Add-ons registration API and UI proposal

Post by pauxlo » August 2nd, 2015, 8:02 pm

I guess a

Code: Select all

conflicts=<depends>
key might be useful, to declare already known conflicts between add-ons, which mean that the user can only enable one of them if the other one is disabled.

User avatar
Celtic_Minstrel
Developer
Posts: 1116
Joined: August 3rd, 2012, 11:26 pm
Contact:

Re: Add-ons registration API and UI proposal

Post by Celtic_Minstrel » August 14th, 2015, 7:50 am

I read the entire post and quite like the proposal. One additional thing I thought of when I saw the feedback tag is a way to specify a URL to a version control repository, though that's not really something most people would need to know.

As things are right now, the license tag seems a little bit pointless since all addons are forced to be GPL.

The best part, of course, is that the passphrase is not in the same file as the metadata.
Author of The Black Cross of Aleron campaign and Default++ era.
Maintainer of Steelhive.

Post Reply