Heavy repository

[Freem]

Hmm, most of the talk until now revolved around the use of "git submodule".

I never mentioned it, I know how annoying it is to use submodules.
Plus, it would not have actually split the repo, it would have implied to remove history by starting a completely new repo, which is obviously not a good solution. The submodules would be a part of a solution for a unified repo which would contain all current stuff, while the subtree would be the part of the solution to split the repo into smaller entities.

To be honest, I thought that the process was doable because, in the past, flare-rpg had everything in a single repo, that they split, but I had no idea about how they did, so I first asked what you would think about that, and since everyone which replied here says that they see the problem but not the solution, I did some researches (which would have been useless if the reply would have been: "we like it how it is").
Their old repo is still around, but not maintained. When using the 2 sub-repos (engine and game data in their case) it seems (I have only take a quick look) you can still see the parts of history concerning the period before the split.
Now, flare-rpg is still a young project, compared to wesnoth, and the intent was from the start to write an engine independent from the game, unlike wesnoth. The transition have probably been easier for them that it will be for you, if you do it.
Also, they use separated repo for the actual game and for the code, which imply that if you want the full development version, you need to do 2 clones and then use cmake on both to compile (for binary) and install (for both). You can also give flare the path of the game content on the command-line, IIRC.

I just took a look at the "git subtree" man-page, and it does look like that might work better than a submodule. The question though is, does it actually solve the problem? Whether it's a submodule or a subtree, the full history is still there, right? Will the subtree history be automatically downloaded when you clone? Do you need extra steps to download the subtree when you clone?

In short: you will have the full history of the branch you cloned. If you clone all branches, you will have full history of the project, if you only clone a branch with source code, you will only have history related to source code.
The history of the generated branch seems (I discovered the command recently, only played a bit to check if it is feasible or not, I should probably do some experiments on a repo made for that with a small history, which would make things easier to understand...) to retain only things related to the subtree. So, if a commit changed files in both the subtree and another folder, you will have the full commit description but only the changes which concerns the subtree in the history.
Also, using the "git subtree split" command will not recreate the directory structure with empty folders, it will use the given folder as the git root. In my example, if I take the wesnoth-musics repo, I only have a bunch of .ogg files and the .git folder.
The original branch is never modified.

Splitting this into a new repo is easy: just create a repo (git init) and pull only the branch made by the subtree (git pull original-repo new-branch). But you will obviously not have stuff from all repos, which is what I am trying you to convince to do, so that the repos would be lighter.

To build the full game with full data would still need to download the whole 3G monster repo (or an archive) and would imply to install all those which interest you separately since they would become different projects.
Now, do regular users use the git repo?
Keeping around 3.2G on a disk, pulling and compiling and each new version (not to speak about installing dependencies) seems not likely to me. Only maintainers or people playing with internals probably use the repo, and except maintainers, who need the full tree? I mean, someone able to work on all the aspects without damaging wesnoth quality is probably very rare (I may not agree with all choices, but I have to say that the quality was greatly improved since 1.6, and even at that time it was not ugly at all).

I do not say that it should be difficult to download an archive or a snapshot of the full game, of course.
I only think that history is only useful when you want to change or understand something (having access to commit diff really helps me understanding what does what), and that most (I only see people doing the packages for installing to be interested by having everything, and still, some of them split stuff into separate packages for space/bandwith reasons[1]) people are not interested in full history, but only by the history of the part they want to play with.
For people preparing in need of the full repo, I am not sure, but git subtree or pull might give alternatives to submodules by using different merging strategies, but I have to admit that, at the moment, I have no idea about how it could be done. The best way to do that would probably depends on how wesnoth's stuff would be split, if it is.

Now, it's up to you wesnoth's contributors to decide if and how to split the repo to provide easier/faster clone with the most balanced burden for all sides (project maintainers, distro maintainers and outsiders).

1: I think it's why debian does that anyway. It might also be related to the fact that some data are not arch-dependent.

PS: sorry for long posts... I hope I am not too hard to read.

[Celtic_Minstrel]

So, considering the case of a dev who wants to work on the campaigns but also take advantage of the latest master... would they have to clone twice (on two separate branches) in order to get everything they need? Or three times if they also want the music? It doesn't seem like it's much more convenient for new devs compared to the submodule approach...

For people building the full game, downloading all the history actually isn't necessary – they only need the latest commit, without any history. A shallow clone or github's source download option would suffice, then. Thus any potential split would be mainly targeted at new developers.

In any case, the subtree possibility will certainly be considered.

[Freem]

If the single clone without more steps or uncommon clone options is mandatory (as history conservation) then I doubt that there is a solution. As I said, "git subtree --split" moves the content of the subtree at the root of the git repo of it's branch, so I do not see how using several branches of subtrees could be used to have an easy to work full repo.

If adding some steps or modifying the "git clone" usual command-line (I just discovered that adding --recursive to it will do all the submodule stuff needed to initialize, but I'm not really fond of submodules. Maybe I just don't work with them often enough.) is an option, then I think that first using subtree to split the repository into several ones, and then creating a "master" repo with submodules could be acceptable.

About new developers which would need the freshest stuff on several parts, I wonder how many people will accept to spend more than 4 hours (with a fast and reliable connection) just to clone. Well, some could accept, especially if the project was a tool to work with, but wesnoth is a game. A great open source game, sure, but still a game.
If those parts were split, one could start by cloning the part he intends to work on, then the others, and while the others are cloning, do what he want. In my example (moving campaigns and musics outside, which is not an optimal example, it was more meant as a demo), if that person want to work on campaigns, that person would only need to clone a ~500M repo to be able to start working. Admittedly it would have taken 4 hours to download the full repo of ~3G, the time would be reduce to 40min (here, campaigns repo is smaller, and full repo bigger than numbers I gave, plus I spent a lot more than 4 hours to clone, so the gain would be way bigger for me and other people with a download bandwidth of around 250kio). Of course, that person would still need to spend the remaining time to have the full version and test in optimal conditions. Or use shallow clones for the other ones. So, being able to have full history of the subtree (maybe not very useful for arts, I do not know, but for code and wml I think it can) you want to work on, plus the most recent stuff from others with shallow clones.
Split repositories may add steps, but by adding steps they also allow to reduce the problems made by the fact git commands can not be resumed (if the connection have a problem, or if the bandwidth is needed for something else in those hours, one can only resume the part that was interrupted).
But this is only my opinion, and I'm not an actual contributor, so I have no problem if you disagree, I'll just accept it. At least I would have tried to be convincing

[nuorc]

So, when I read that: ...

there are 10 people with 20+ commits YTD and another 3 with 10+ out of 18 total committers.

Check out IRC before making any conclusions: irc://irc.wesnoth.org/#wesnoth-dev

That irc link might be interesting for you if you don't like waiting for replies on something.

Regarding the possible split: maybe BfW 2.0 would be a good point in time for that?

[SigurdFireDragon]

I tested the github .zip and shallow clone options.

the zip is only 420MB in size, but it's only the current state of each file in the repository, without any history.

I tried two different shallow clone commands (depths of 1 & 100). The results are listed below.
Note the final receiving object size of approx 400 & 450 mb respectively for data download
the repositories unpacked to a size of 1.04 & 1.10 GB.

Also, push & pull can be done on shallow repos since git 1.9 (https://www.perforce.com/blog/141218/gi ... low-clones)


As such, I think repo size is not an adequate reason to split the repo. Furthermore, it seems like it could be disruptive, so if it was to be done, I think it would be better to wait for the next dev cycle.

Code: Select all

$ git clone --depth 1 "https://github.com/wesnoth/wesnoth.git" wesnoth-shallow-test
Cloning into 'wesnoth-shallow-test'...
remote: Counting objects: 19401, done.
remote: Compressing objects: 100% (18743/18743), done.
remote: Total 19401 (delta 948), reused 10371 (delta 424), pack-reused 0
Receiving objects: 100% (19401/19401), 397.56 MiB | 5.29 MiB/s, done.
Resolving deltas: 100% (948/948), done.
Checking connectivity... done.
Checking out files: 100% (19050/19050), done.


$ git clone --depth 100 "https://github.com/wesnoth/wesnoth.git" wesnoth-shallow-test-2
Cloning into 'wesnoth-shallow-test-2'...
remote: Counting objects: 34812, done.
remote: Compressing objects: 100% (28001/28001), done.
remote: Total 34812 (delta 13290), reused 19020 (delta 6522), pack-reused 0
Receiving objects: 100% (34812/34812), 449.03 MiB | 6.57 MiB/s, done.
Resolving deltas: 100% (13290/13290), done.
Checking connectivity... done.
Checking out files: 100% (19050/19050), done.

[Celtic_Minstrel]

As such, I think repo size is not an adequate reason to split the repo. Furthermore, it seems like it could be disruptive, so if it was to be done, I think it would be better to wait for the next dev cycle.

I actually think that, if it is going to be done, the best time would be just before a 1.14 release; that way, the 1.14 stable branch would get the benefit of the split as well.

I'm still not convinced that it's worth it in the first place, though.

[Iris]

I actually think that, if it is going to be done, the best time would be just before a 1.14 release; that way, the 1.14 stable branch would get the benefit of the split as well.

git log, git blame and git bisect, anyone?

[gfgtdf]

Also, push & pull can be done on shallow repos since git 1.9 (https://www.perforce.com/blog/141218/gi ... low-clones)

But you usually need more that git push/pull. do you know whether 'git rebase' and 'git push -f' are also supported?

git log, git blame and git bisect, anyone?

I dont use 'git blame' direcoty put i use those things on the github page like in https://github.com/wesnoth/wesnoth/blam ... c/team.cpp

I actually assumedd using git blame woudl still be possiblw when the respsity is split it just woldn't show the parts related to those subdirectired anymore. unless you explicitly do the git blame there.

wesnoth-translations: 880M
wesnoth-campaigns: 487M
wesnoth-musics: 297M

This sounds to me like it'd be indeed worthy to move those parts to submodules/subtrees. I also agree that if we do that we should do that during 1.13 if we relly intend to do this.

[SigurdFireDragon]

Well, how about the following to download the complete repository on a slow connection?
the initial clone shows a dl of around 400mb, with each subsequent fetch depth increase to be approx 150-300mb.
fixing the implied --single-branch at the end takes about 217mb.

First:
git clone --depth 1 "https://github.com/wesnoth/wesnoth.git" wesnoth-shallow-test

Then do:
git fetch --depth=2000
git fetch --depth=4000
git fetch --depth=6000
git fetch --depth=8000
git fetch --depth=10000
git fetch --depth=12000
git fetch --depth=15000
git fetch --depth=20000
git fetch --depth=25000
git fetch --depth=30000
git fetch --depth=35000
git fetch --depth=40000
git fetch --depth=50000
git fetch --depth=60000
git fetch --depth=70000

when an increase in depth no longer gets additional data, check with
git fetch --unshallow
if the depth is maxed out, it will say unshallowing a complete repo doesn't make sense.

There's still the implied single-branch with the initial clone checkout with --depth=1, but that's fixed by doing:
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
git fetch origin
(which is only about 217mb at this point)

here's my log of doing this. Note that the most recent 10K commits seem rather heavy, which is why I suggest going by 2000 for the first 10-15K

Code: Select all

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=8000
remote: Counting objects: 209600, done.
remote: Compressing objects: 100% (56864/56864), done.
remote: Total 209600 (delta 164917), reused 193744 (delta 151032), pack-reused
Receiving objects: 100% (209600/209600), 749.34 MiB | 6.56 MiB/s, done.
Resolving deltas: 100% (164917/164917), completed with 4011 local objects.
From https://github.com/wesnoth/wesnoth
   2ac59d2..69b0fdd  master     -> origin/master
remote: Counting objects: 26, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 26 (delta 0), reused 26 (delta 0), pack-reused 0
Unpacking objects: 100% (26/26), done.
 * [new tag]         1.10.0     -> 1.10.0
 * [new tag]         1.11.0     -> 1.11.0
 * [new tag]         1.11.1     -> 1.11.1
 * [new tag]         1.11.10    -> 1.11.10
 * [new tag]         1.11.10-retag -> 1.11.10-retag
 * [new tag]         1.11.11    -> 1.11.11
 * [new tag]         1.11.2     -> 1.11.2
 * [new tag]         1.11.3     -> 1.11.3
 * [new tag]         1.11.4     -> 1.11.4
 * [new tag]         1.11.5     -> 1.11.5
 * [new tag]         1.11.6     -> 1.11.6
 * [new tag]         1.11.7     -> 1.11.7
 * [new tag]         1.11.8     -> 1.11.8
 * [new tag]         1.11.9     -> 1.11.9
 * [new tag]         1.13.0     -> 1.13.0
 * [new tag]         1.13.1     -> 1.13.1
 * [new tag]         1.13.2     -> 1.13.2
 * [new tag]         1.13.3     -> 1.13.3
 * [new tag]         1.13.4     -> 1.13.4
 * [new tag]         1.9.10     -> 1.9.10
 * [new tag]         1.9.11     -> 1.9.11
 * [new tag]         1.9.12     -> 1.9.12
 * [new tag]         1.9.13     -> 1.9.13
 * [new tag]         1.9.14     -> 1.9.14
 * [new tag]         1.9.7      -> 1.9.7
 * [new tag]         1.9.8      -> 1.9.8
 * [new tag]         1.9.9      -> 1.9.9
 * [new tag]         original_master -> original_master

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=2000
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=10000
remote: Counting objects: 41594, done.
remote: Compressing objects: 100% (21282/21282), done.
remote: Total 41594 (delta 30393), reused 29690 (delta 20077), pack-reused 0
Receiving objects: 100% (41594/41594), 109.30 MiB | 6.56 MiB/s, done.
Resolving deltas: 100% (30393/30393), completed with 6991 local objects.
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 0), reused 5 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.9.2      -> 1.9.2
 * [new tag]         1.9.3      -> 1.9.3
 * [new tag]         1.9.4      -> 1.9.4
 * [new tag]         1.9.5      -> 1.9.5
 * [new tag]         1.9.6      -> 1.9.6

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=13000
remote: Counting objects: 48548, done.
remote: Compressing objects: 100% (18640/18640), done.
remote: Total 48548 (delta 36817), reused 40017 (delta 29308), pack-reused 0
Receiving objects: 100% (48548/48548), 118.17 MiB | 6.57 MiB/s, done.
Resolving deltas: 100% (36817/36817), completed with 3484 local objects.
remote: Counting objects: 2, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 2 (delta 0), reused 2 (delta 0), pack-reused 0
Unpacking objects: 100% (2/2), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.9.0      -> 1.9.0
 * [new tag]         1.9.1      -> 1.9.1

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=18000
remote: Counting objects: 89928, done.
remote: Compressing objects: 100% (25294/25294), done.
remote: Total 89928 (delta 72056), reused 80893 (delta 64286), pack-reused 0
Receiving objects: 100% (89928/89928), 159.54 MiB | 5.35 MiB/s, done.
Resolving deltas: 100% (72056/72056), completed with 2369 local objects.
remote: Counting objects: 14, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 14 (delta 0), reused 14 (delta 0), pack-reused 0
Unpacking objects: 100% (14/14), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.7.10     -> 1.7.10
 * [new tag]         1.7.11     -> 1.7.11
 * [new tag]         1.7.12     -> 1.7.12
 * [new tag]         1.7.13     -> 1.7.13
 * [new tag]         1.7.14     -> 1.7.14
 * [new tag]         1.7.15     -> 1.7.15
 * [new tag]         1.7.3      -> 1.7.3
 * [new tag]         1.7.4      -> 1.7.4
 * [new tag]         1.7.5      -> 1.7.5
 * [new tag]         1.7.6      -> 1.7.6
 * [new tag]         1.7.7      -> 1.7.7
 * [new tag]         1.7.8      -> 1.7.8
 * [new tag]         1.7.9      -> 1.7.9
 * [new tag]         1.8.0      -> 1.8.0

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=25000
remote: Counting objects: 102525, done.
remote: Compressing objects: 100% (27635/27635), done.
remote: Total 102525 (delta 81269), reused 94021 (delta 74531), pack-reused 0
Receiving objects: 100% (102525/102525), 231.05 MiB | 5.22 MiB/s, done.
Resolving deltas: 100% (81269/81269), completed with 2262 local objects.
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 15 (delta 0), reused 15 (delta 0), pack-reused 0
Unpacking objects: 100% (15/15), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.5.10     -> 1.5.10
 * [new tag]         1.5.11     -> 1.5.11
 * [new tag]         1.5.12     -> 1.5.12
 * [new tag]         1.5.13     -> 1.5.13
 * [new tag]         1.5.14     -> 1.5.14
 * [new tag]         1.5.5      -> 1.5.5
 * [new tag]         1.5.6      -> 1.5.6
 * [new tag]         1.5.7      -> 1.5.7
 * [new tag]         1.5.8      -> 1.5.8
 * [new tag]         1.5.9      -> 1.5.9
 * [new tag]         1.6        -> 1.6
 * [new tag]         1.6a       -> 1.6a
 * [new tag]         1.7.0      -> 1.7.0
 * [new tag]         1.7.1      -> 1.7.1
 * [new tag]         1.7.2      -> 1.7.2

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=32000
remote: Counting objects: 91300, done.
remote: Compressing objects: 100% (21253/21253), done.
remote: Total 91300 (delta 76118), reused 83739 (delta 69593), pack-reused 0
Receiving objects: 100% (91300/91300), 108.92 MiB | 6.57 MiB/s, done.
Resolving deltas: 100% (76118/76118), completed with 1698 local objects.
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 17 (delta 0), reused 17 (delta 0), pack-reused 0
Unpacking objects: 100% (17/17), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.3.10     -> 1.3.10
 * [new tag]         1.3.11     -> 1.3.11
 * [new tag]         1.3.12     -> 1.3.12
 * [new tag]         1.3.13     -> 1.3.13
 * [new tag]         1.3.14     -> 1.3.14
 * [new tag]         1.3.15     -> 1.3.15
 * [new tag]         1.3.16     -> 1.3.16
 * [new tag]         1.3.17     -> 1.3.17
 * [new tag]         1.3.18     -> 1.3.18
 * [new tag]         1.3.19     -> 1.3.19
 * [new tag]         1.3.9      -> 1.3.9
 * [new tag]         1.4        -> 1.4
 * [new tag]         1.5.0      -> 1.5.0
 * [new tag]         1.5.1      -> 1.5.1
 * [new tag]         1.5.2      -> 1.5.2
 * [new tag]         1.5.3      -> 1.5.3
 * [new tag]         1.5.4      -> 1.5.4

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=45000
remote: Counting objects: 194312, done.
remote: Compressing objects: 100% (52003/52003), done.
remote: Total 194312 (delta 144998), reused 189901 (delta 141733), pack-reused
Receiving objects: 100% (194312/194312), 321.28 MiB | 5.69 MiB/s, done.
Resolving deltas: 100% (144998/144998), completed with 1086 local objects.
remote: Counting objects: 75, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 75 (delta 8), reused 75 (delta 8), pack-reused 0
Unpacking objects: 100% (75/75), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         1.0.1      -> 1.0.1
 * [new tag]         1.1        -> 1.1
 * [new tag]         1.1.1      -> 1.1.1
 * [new tag]         1.1.1-rc1  -> 1.1.1-rc1
 * [new tag]         1.1.1-rc2  -> 1.1.1-rc2
 * [new tag]         1.1.1-rc3  -> 1.1.1-rc3
 * [new tag]         1.1.10     -> 1.1.10
 * [new tag]         1.1.11     -> 1.1.11
 * [new tag]         1.1.12     -> 1.1.12
 * [new tag]         1.1.13     -> 1.1.13
 * [new tag]         1.1.14     -> 1.1.14
 * [new tag]         1.1.2      -> 1.1.2
 * [new tag]         1.1.2a     -> 1.1.2a
 * [new tag]         1.1.3      -> 1.1.3
 * [new tag]         1.1.4      -> 1.1.4
 * [new tag]         1.1.5      -> 1.1.5
 * [new tag]         1.1.6      -> 1.1.6
 * [new tag]         1.1.8      -> 1.1.8
 * [new tag]         1.1.9      -> 1.1.9
 * [new tag]         1.2        -> 1.2
 * [new tag]         1.2.1      -> 1.2.1
 * [new tag]         1.3.1      -> 1.3.1
 * [new tag]         1.3.2      -> 1.3.2
 * [new tag]         1.3.3      -> 1.3.3
 * [new tag]         1.3.4      -> 1.3.4
 * [new tag]         1.3.5      -> 1.3.5
 * [new tag]         1.3.6      -> 1.3.6
 * [new tag]         1.3.7      -> 1.3.7
 * [new tag]         1.3.8      -> 1.3.8
 * [new tag]         v0_8_10    -> v0_8_10
 * [new tag]         v0_8_10rc2 -> v0_8_10rc2
 * [new tag]         v0_8_10rc3 -> v0_8_10rc3
 * [new tag]         v0_8_10rc4 -> v0_8_10rc4
 * [new tag]         v0_8_10rc5 -> v0_8_10rc5
 * [new tag]         v0_8_11    -> v0_8_11
 * [new tag]         v0_8_11rc1 -> v0_8_11rc1
 * [new tag]         v0_8_11rc2 -> v0_8_11rc2
 * [new tag]         v0_8_11rc3 -> v0_8_11rc3
 * [new tag]         v0_8_11rc4 -> v0_8_11rc4
 * [new tag]         v0_8_11rc5 -> v0_8_11rc5
 * [new tag]         v0_8_12rc1 -> v0_8_12rc1
 * [new tag]         v0_8_5     -> v0_8_5
 * [new tag]         v0_8_5miyo1 -> v0_8_5miyo1
 * [new tag]         v0_8_5miyo2 -> v0_8_5miyo2
 * [new tag]         v0_8_5miyo3 -> v0_8_5miyo3
 * [new tag]         v0_8_5miyo4 -> v0_8_5miyo4
 * [new tag]         v0_8_5miyo5 -> v0_8_5miyo5
 * [new tag]         v0_8_5miyo6 -> v0_8_5miyo6
 * [new tag]         v0_8_5miyo7 -> v0_8_5miyo7
 * [new tag]         v0_8_5miyo8 -> v0_8_5miyo8
 * [new tag]         v0_8_5rc1  -> v0_8_5rc1
 * [new tag]         v0_8_5rc2  -> v0_8_5rc2
 * [new tag]         v0_8_5rc3  -> v0_8_5rc3
 * [new tag]         v0_8_5rc4  -> v0_8_5rc4
 * [new tag]         v0_8_5rc5  -> v0_8_5rc5
 * [new tag]         v0_8_6miyo1 -> v0_8_6miyo1
 * [new tag]         v0_8_6miyo2 -> v0_8_6miyo2
 * [new tag]         v0_8_6rc1  -> v0_8_6rc1
 * [new tag]         v0_8_6rc2  -> v0_8_6rc2
 * [new tag]         v0_8_6rc3  -> v0_8_6rc3
 * [new tag]         v0_8_6rc4  -> v0_8_6rc4
 * [new tag]         v0_8_7     -> v0_8_7
 * [new tag]         v0_8_7rc1  -> v0_8_7rc1
 * [new tag]         v0_8_7rc2  -> v0_8_7rc2
 * [new tag]         v0_8_7rc3  -> v0_8_7rc3
 * [new tag]         v0_8_7rc4  -> v0_8_7rc4
 * [new tag]         v0_8_8     -> v0_8_8
 * [new tag]         v0_8_8rc1  -> v0_8_8rc1
 * [new tag]         v0_8_9     -> v0_8_9
 * [new tag]         v0_8_9rc1  -> v0_8_9rc1
 * [new tag]         v0_8_9rc2  -> v0_8_9rc2
 * [new tag]         v0_9_0     -> v0_9_0
 * [new tag]         v0_9_0rc1  -> v0_9_0rc1
 * [new tag]         v0_9_0rc2  -> v0_9_0rc2
 * [new tag]         v0_9_0rc3  -> v0_9_0rc3
 * [new tag]         v0_9_0rc4  -> v0_9_0rc4
 * [new tag]         v0_9_1     -> v0_9_1
 * [new tag]         v0_9_1rc1  -> v0_9_1rc1
 * [new tag]         v0_9_2     -> v0_9_2
 * [new tag]         v0_9_2rc1  -> v0_9_2rc1
 * [new tag]         v0_9_3     -> v0_9_3
 * [new tag]         v0_9_3fixed -> v0_9_3fixed
 * [new tag]         v0_9_3rc1  -> v0_9_3rc1
 * [new tag]         v0_9_4     -> v0_9_4
 * [new tag]         v0_9_4rc1  -> v0_9_4rc1
 * [new tag]         v0_9_4rc2  -> v0_9_4rc2
 * [new tag]         v0_9_5     -> v0_9_5
 * [new tag]         v0_9_5rc1  -> v0_9_5rc1
 * [new tag]         v0_9_5rc2  -> v0_9_5rc2
 * [new tag]         v0_9_5rc3  -> v0_9_5rc3
 * [new tag]         v0_9_6     -> v0_9_6
 * [new tag]         v0_9_6rc1  -> v0_9_6rc1
 * [new tag]         v0_9_6rc2  -> v0_9_6rc2
 * [new tag]         v0_9_7     -> v0_9_7
 * [new tag]         v0_9_7rc1  -> v0_9_7rc1
 * [new tag]         v0_9rc1    -> v0_9rc1
 * [new tag]         v1_0_rc1rc1 -> v1_0_rc1rc1
 * [new tag]         v1_0rc1    -> v1_0rc1

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=52000
remote: Counting objects: 30809, done.
remote: Compressing objects: 100% (10518/10518), done.
remote: Total 30809 (delta 20949), reused 29470 (delta 20208), pack-reused 0
Receiving objects: 100% (30809/30809), 48.63 MiB | 5.56 MiB/s, done.
Resolving deltas: 100% (20949/20949), completed with 56 local objects.
remote: Counting objects: 33, done.
remote: Total 33 (delta 0), reused 0 (delta 0), pack-reused 33
Unpacking objects: 100% (33/33), done.
From https://github.com/wesnoth/wesnoth
 * [new tag]         v0_4_8     -> v0_4_8
 * [new tag]         v0_4_8rc1  -> v0_4_8rc1
 * [new tag]         v0_4_8rc3  -> v0_4_8rc3
 * [new tag]         v0_4_8rc4  -> v0_4_8rc4
 * [new tag]         v0_5       -> v0_5
 * [new tag]         v0_5_1     -> v0_5_1
 * [new tag]         v0_5rc5    -> v0_5rc5
 * [new tag]         v0_6       -> v0_6
 * [new tag]         v0_6_1     -> v0_6_1
 * [new tag]         v0_6_1rc1  -> v0_6_1rc1
 * [new tag]         v0_6_99_1  -> v0_6_99_1
 * [new tag]         v0_6_99_2  -> v0_6_99_2
 * [new tag]         v0_6_99_3  -> v0_6_99_3
 * [new tag]         v0_6_99_4  -> v0_6_99_4
 * [new tag]         v0_6_99_5  -> v0_6_99_5
 * [new tag]         v0_6rc1    -> v0_6rc1
 * [new tag]         v0_6rc2    -> v0_6rc2
 * [new tag]         v0_6rc3    -> v0_6rc3
 * [new tag]         v0_7       -> v0_7
 * [new tag]         v0_7_1     -> v0_7_1
 * [new tag]         v0_7_10    -> v0_7_10
 * [new tag]         v0_7_11    -> v0_7_11
 * [new tag]         v0_7_2     -> v0_7_2
 * [new tag]         v0_7_3     -> v0_7_3
 * [new tag]         v0_7_4     -> v0_7_4
 * [new tag]         v0_7_5     -> v0_7_5
 * [new tag]         v0_7_6     -> v0_7_6
 * [new tag]         v0_7_7     -> v0_7_7
 * [new tag]         v0_7_8     -> v0_7_8
 * [new tag]         v0_7_9     -> v0_7_9
 * [new tag]         v0_8       -> v0_8
 * [new tag]         v0_8_1     -> v0_8_1
 * [new tag]         v0_8_2     -> v0_8_2
 * [new tag]         v0_8_3     -> v0_8_3
 * [new tag]         v0_8_4     -> v0_8_4
 * [new tag]         v0_8_4rc1  -> v0_8_4rc1
 * [new tag]         v0_8_4rc2  -> v0_8_4rc2
 * [new tag]         v0_8_4rc3  -> v0_8_4rc3

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=65000
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --depth=100000
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch origin
remote: Counting objects: 84831, done.
remote: Compressing objects: 100% (71/71), done.
remote: Total 84831 (delta 51283), reused 51326 (delta 51277), pack-reused 3348
Receiving objects: 100% (84831/84831), 214.57 MiB | 6.57 MiB/s, done.
Resolving deltas: 100% (73078/73078), completed with 10253 local objects.
From https://github.com/wesnoth/wesnoth
 * [new branch]      1.0        -> origin/1.0
 * [new branch]      1.10       -> origin/1.10
 * [new branch]      1.12       -> origin/1.12
 * [new branch]      1.2        -> origin/1.2
 * [new branch]      1.4        -> origin/1.4
 * [new branch]      1.4-lint   -> origin/1.4-lint
 * [new branch]      1.6        -> origin/1.6
 * [new branch]      1.8        -> origin/1.8
 * [new branch]      BRANCH_vendor_gettext -> origin/BRANCH_vendor_gettext
 * [new branch]      SDL_gpu    -> origin/SDL_gpu
 * [new branch]      alep_android_port -> origin/alep_android_port
 * [new branch]      ancestral-lato -> origin/ancestral-lato
 * [new branch]      asio_umcd  -> origin/asio_umcd
 * [new branch]      asio_wesnothd -> origin/asio_wesnothd
 * [new branch]      branch_gettext1 -> origin/branch_gettext1
 * [new branch]      formula-ai -> origin/formula-ai
 * [new branch]      gameplay_refactoring -> origin/gameplay_refactoring
 * [new branch]      gp2x       -> origin/gp2x
 * [new branch]      help_gui2  -> origin/help_gui2
 * [new branch]      mordante_terrain -> origin/mordante_terrain
 * [new branch]      mp_registration -> origin/mp_registration
 * [new branch]      new_addon_server -> origin/new_addon_server
 * [new branch]      new_mp_create_ui -> origin/new_mp_create_ui
 * [new branch]      ogl        -> origin/ogl
 * [new branch]      qt         -> origin/qt
 * [new branch]      renderpath_redo -> origin/renderpath_redo
 * [new branch]      spritesheet -> origin/spritesheet
 * [new branch]      terrain-layer -> origin/terrain-layer
 * [new branch]      water_animation -> origin/water_animation
 * [new branch]      wesnoth-gl -> origin/wesnoth-gl
 * [new branch]      wesnoth-python -> origin/wesnoth-python
 * [new branch]      xan        -> origin/xan
 * [new tag]         gnu-gettext-0_14_4 -> gnu-gettext-0_14_4
remote: Counting objects: 50, done.
remote: Total 50 (delta 0), reused 0 (delta 0), pack-reused 50
Unpacking objects: 100% (50/50), done.
 * [new tag]         1.0        -> 1.0
 * [new tag]         1.0.2      -> 1.0.2
 * [new tag]         1.0rc2     -> 1.0rc2
 * [new tag]         1.10.1     -> 1.10.1
 * [new tag]         1.10.2     -> 1.10.2
 * [new tag]         1.10.3     -> 1.10.3
 * [new tag]         1.10.4     -> 1.10.4
 * [new tag]         1.10.5     -> 1.10.5
 * [new tag]         1.10.6     -> 1.10.6
 * [new tag]         1.10.7     -> 1.10.7
 * [new tag]         1.11.12    -> 1.11.12
 * [new tag]         1.11.13    -> 1.11.13
 * [new tag]         1.11.14    -> 1.11.14
 * [new tag]         1.11.15    -> 1.11.15
 * [new tag]         1.11.16    -> 1.11.16
 * [new tag]         1.11.17    -> 1.11.17
 * [new tag]         1.11.18    -> 1.11.18
 * [new tag]         1.11.19    -> 1.11.19
 * [new tag]         1.12.0     -> 1.12.0
 * [new tag]         1.12.1     -> 1.12.1
 * [new tag]         1.12.2     -> 1.12.2
 * [new tag]         1.12.3     -> 1.12.3
 * [new tag]         1.12.4     -> 1.12.4
 * [new tag]         1.12.5     -> 1.12.5
 * [new tag]         1.2.2      -> 1.2.2
 * [new tag]         1.2.3      -> 1.2.3
 * [new tag]         1.2.4      -> 1.2.4
 * [new tag]         1.2.5      -> 1.2.5
 * [new tag]         1.2.6      -> 1.2.6
 * [new tag]         1.2.7      -> 1.2.7
 * [new tag]         1.2.8      -> 1.2.8
 * [new tag]         1.4.1      -> 1.4.1
 * [new tag]         1.4.2      -> 1.4.2
 * [new tag]         1.4.3      -> 1.4.3
 * [new tag]         1.4.4      -> 1.4.4
 * [new tag]         1.4.5      -> 1.4.5
 * [new tag]         1.4.6      -> 1.4.6
 * [new tag]         1.4.7      -> 1.4.7
 * [new tag]         1.6.1      -> 1.6.1
 * [new tag]         1.6.2      -> 1.6.2
 * [new tag]         1.6.3      -> 1.6.3
 * [new tag]         1.6.4      -> 1.6.4
 * [new tag]         1.6.5      -> 1.6.5
 * [new tag]         1.8.1      -> 1.8.1
 * [new tag]         1.8.2      -> 1.8.2
 * [new tag]         1.8.3      -> 1.8.3
 * [new tag]         1.8.4      -> 1.8.4
 * [new tag]         1.8.5      -> 1.8.5
 * [new tag]         1.8.6      -> 1.8.6
 * [new tag]         original_1.10 -> original_1.10

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense

User@Computer MINGW64 /c/Wesnoth/Test/wesnoth-shallow-test (master)
$

[Freem]

actually think that, if it is going to be done, the best time would be just before a 1.14 release; that way, the 1.14 stable branch would get the benefit of the split as well.

I don't even speak about when, I just try to convince your team to do it. Time point is a question which is not worth thinking about when you do not know if it will be a project at all.

I'm still not convinced that it's worth it in the first place, though.

Well, I did my best to convince, I have no more arguments to give you, and I won't insult people by repeating my arguments. I try to not waste time of people, especially when they give that time for everyone. I hope I didn't waste yours (if I did, my apologies, and thank you for being so polite. The second point is still valid if I didn't ).

I actually think that, if it is going to be done, the best time would be just before a 1.14 release; that way, the 1.14 stable branch would get the benefit of the split as well.

git log, git blame and git bisect, anyone?

I don't really see the relation of those commands on this subject?
git log allows to read log. blame can allow to know who changed a line, and bisect is really useful to find bugs.
All those commands are just to know what happened, if I am not wrong. Now, I must admit that I never used git blame, and that I only used bisect once, I might miss something important (if so, feel free to enlighten me, I would be grateful for that, still have a lot to learn about git).

Well, how about the following to download the complete repository on a slow connection?

Good idea. You just have to say what you name a slow connection, since slow is a word which depends on context.

the initial clone shows a dl of around 400mb, with each subsequent fetch depth increase to be approx 150-300mb.
fixing the implied --single-branch at the end takes about 217mb.

Allow me to do the same, with 2 changes.
1st change, I'll use the "time" function to show how many time it uses to clone.
2nd thing, I will only do the cloning with 1 commit and a download speed around 350Kio/s (this number is from regular looks to xosview, if someone knows a tool to measure the bandwidth an application uses, I would be happy to hear about it):

Code: Select all

/tmp% time LANG=C git clone --depth 1 "https://github.com/wesnoth/wesnoth.git"                             
Cloning into 'wesnoth'...
remote: Counting objects: 19404, done.
remote: Compressing objects: 100% (18750/18750), done.
remote: Total 19404 (delta 950), reused 10368 (delta 420), pack-reused 0
Receiving objects: 100% (19404/19404), 397.67 MiB | 340.00 KiB/s, done.
Resolving deltas: 100% (950/950), done.
Checking connectivity... done.
Checking out files: 100% (19041/19041), done.
LANG=C git clone --depth 1 "https://github.com/wesnoth/wesnoth.git"  40,88s user 17,05s system 3% cpu 26:44,22 total

So, event a clone with the most recent commit is long. Almost half an hour. My current download speed does not seems that slow for me. In fact, I had a lot worse in non-urban areas. Now, maybe we

While I'm thinking about that, the wiki gives a pretty introduction to git (I already said that I read wiki before forums/IRC, and that I believe it's a usual behavior) so it might be a good idea to say there that the repo is actually heavy, and that it would be better for many things to use the "--depth" parameter.
Fun (or not) fact: I had no idea what shallow meant before I started this topic (and when I first seen that word, I checked some translation tool to understand). I guess it does not serves me to say that, but since I am trying to be honest...

[pauxlo]

git log, git blame and git bisect, anyone?

I don't really see the relation of those commands on this subject?
git log allows to read log. blame can allow to know who changed a line, and bisect is really useful to find bugs.
All those commands are just to know what happened, if I am not wrong. Now, I must admit that I never used git blame, and that I only used bisect once, I might miss something important (if so, feel free to enlighten me, I would be grateful for that, still have a lot to learn about git).

The point is that all those tools cease to work properly when the history doesn't continue, but starts from scratch.

[Celtic_Minstrel]

I don't think anyone was suggesting truncating the history.

[SigurdFireDragon]

Good idea. You just have to say what you name a slow connection, since slow is a word which depends on context.

An example of a slow connection would be the connection you used when you made your original post in the thread.

Hello.

I was curious about some parts of wesnoth's codebase and wanted to take a look. So I did as usual: git clone. And it took me around 2 days.

It took me that many time for those reasons:

* wesnoth codebase... hum, no, in fact, wesnoth's repository is huge. More than 3GiB if I can trust the "du" command.
* git clone can not be interrupted. If it is, you need to restart.
* my connection had problems, or I needed it for something else at some point.

The method I listed allows all three of those problems to be dealt with, by breaking the download into smaller pieces. In a way, it's like splitting the repository, except by splitting history, instead of directories. If the user doesn't need the history, they can just do a depth of 1 on the checkout, if they do, they can run additional commands to get as much as needed.

So, event a clone with the most recent commit is long. Almost half an hour.

It's around 400MB, approx the same size (give or take 10-20MB) as the Source, Windows & OS X package downloads on sourceforge. Since it's expected for the userbase to be able to handle downloading one of those, there's no reason to make it any smaller.

Since the method I provided solves the stated download issues, I'm convinced that splitting the repository is unnecessary.
As such, the method should be added to the wiki, and I have nothing further to add.