[engine] Using GitHub to host addons

[hagabaka]

Recently I played To Lands Unknown, which was a good campaign, but a headache associated with it was the 70+ MB download. And when the addon was updated, I had to download 70+ MB again. So at first I thought: why not make the addon server serve XDeltas to save bandwidth? However, soon I had a second idea which I thought was better: let a distributed version control service like GitHub do all the work.

On GitHub, users create git repositories for their projects, which others can view, download, or fork. GitHub also makes it easy for a developer to ask "forkers" to pull their changes, or for a forker to review new changes from other developers and selectively pull them.

Here's how I think it could work in Wesnoth's addon system. Addon developers will create repositories for their addons, and use most of the current file structure, except files currently used for authentication which will be unnecessary. They can invite collaborators to work on the same repository, or others can fork the repository and ask the official maintainer to pull the changes. If the official maintainer becomes inactive, others can ask Wesnoth addon moderators to make their fork the official one. Maybe not many current addon developers know how to use git, but it's not very hard, especially with GitHub which even allows you to make changes to files in the browser. I think the convenience it provides for collaborators and moderators will be worth the effort. It should be also possible to use GitHub's web services to write an uploader which works just like the current one.

The Wesnoth addon moderators will maintain a "mirror" repository for each addon. When an addon is first approved, the mirror will start out the same as the addon developer's repository. Whenever the developer's repository updates, the moderator can review the changes and decide whether to pull them. These mirror repositories will all be placed under a single GitHub user, say wesnoth_addons, so that GitHub can provide an index for approved addons. Moderators will be collaborators to the repository. Maybe there isn't much "moderation" going on with the current addon server, but I have seen posts where moderators spot problems with addons and ask them to be fixed or for the addon to be deleted. GitHub shows which files are changed in a commit, so moderators can easily spot copyright issues or simple errors, and contact the maintainers to fix them. If automatic approval is still preferred, a web service could be written to allow addon developers to announce their addon, then a script could inspect the addon and approve it if it looks OK. Later a moderator can revert commits or delete the mirror repository.

The Wesnoth program will use git to download addons, which will provide differential downloads which don't get huge when small changes are made to a huge addon. It will use GitHub's web services to list addons.

I think the benefits of this idea are:

• All bandwidth related to addons are shifted to GitHub.
• Users download differentials, and addon developers also upload differentials, when updating an addon.
• Most of the programming is already done by GitHub and git, except optional tools to make the process easier for addon developers or to automatically approve addons.
• Addon developers, contributors and moderators will be able to work together in a more organized way.

The disadvantages are that git will be required for using and creating addons, and addon developers and moderators will need to learn how to use it.

What do you think?

[Gambit]

I like the idea. It sounds a lot like Wesnoth UMC Dev. But why is this here instead of ideas?

[AI]

Yes, this forum is for stuff that is *not* intended to go mainline.

The problem with using git for this, is that the users will clone the *entire* repository, rather than just most recent release, unless you use some sort of special protocol for only pushing tags to the addon-download-branch, but git won't like that much.
The result will probably still be be smaller than getting the releases seperately, owing to git's compression, but a specialized diff system will produce much better results. Perhaps rsync?

[hagabaka]

Yes, this forum is for stuff that is *not* intended to go mainline.

Ah, sorry that I posted in the wrong section. Please feel free to move it.

The problem with using git for this, is that the users will clone the *entire* repository, rather than just most recent release, unless you use some sort of special protocol for only pushing tags to the addon-download-branch, but git won't like that much.
The result will probably still be be smaller than getting the releases seperately, owing to git's compression, but a specialized diff system will produce much better results. Perhaps rsync?

Git actually allows "shallow" clones where you just download the latest revision(s). It looks like tests with a bunch of software (which probably have more changes than most addons) show shallow clones don't decrease the repository size or download time very much, or full clones don't increase them very much. But shallow clones might be better for addons which make many changes to binary files throughout their history. Either way shallow clones are pretty easy to do, just "git clone --depth 1" instead of "git clone".

I don't really know much about how git handles binary files. I guess to fully optimize for that it's best to let the addon server generate XDeltas and it could even just allow authors to upload XDeltas. However someone will need to code that.

[Gambit]

Moved to ideas.