Ideas for handling disconnects/lag in Multiplayer

[Pentarctagon]

the terraindata is normaly only a small part of a game snapshot. a repletive complicatede scenario has a mapfile of uncompressed ~50kb and the snapshot part of the savefile is uncompressed 2.5 mb. I think you'll get less network traffic if the other uses just send you their actions like it is surrently done "unit at (1,3) attacks unit at (4,5) with attack 2" than sending snapshots. a normal turn is has uncomressed data ~5kb from which >50% is additional checkup data to detect oos errors.

I don't understand. How is it less data to send all movements and attacks that have occurred throughout the entire game plus OOS detection data than to just send where the units are at the current turn?

[iceiceice]

Related question: Is wml that is sent over the network compressed or uncompressed?

[Iris]

Compressed, using the gzip algorithm.

[gfgtdf]

I tried playing Wesnoth multiplayer again the other night from a location with 330K/second down / 37K / second up bandwidth, and again suffered lag on the 2nd turn and had to quit a multiplayer game, after notifying the other players of the problem. I have had to quit about 20 - 50 multiplayer games from this location.

did i understand you correct that you normaly play 20-50 mp games at the same time?

In the beginning, I was not aware that the problem was local, and thought it might be the server or another player. But I have come to a conclusion recently after comparing my multiplayer experience between one location and another. Sometimes I waited for half an hour for the lag to resolve. Of course, the other players had to wait as well in those cases. It is only natural that other players may conclude someone stepped away from the keyboard, when in reality the problem is a technical, network issue. I am not sure of the reason why, and the documentation does not offer any insight. I have researched this problem extensively on Google and read many forum messages and blog posts and change-logs written by the developers. If there were already information available on troubleshooting the problem, then I would have pursued that angle. That is why I was speculating that Wesnoth is transmitting a massive amount of data during gameplay.

you can enable log for network data which prtints all sended/recieved game data in the stderr file.

the terraindata is normaly only a small part of a game snapshot. a repletive complicatede scenario has a mapfile of uncompressed ~50kb and the snapshot part of the savefile is uncompressed 2.5 mb. I think you'll get less network traffic if the other uses just send you their actions like it is surrently done "unit at (1,3) attacks unit at (4,5) with attack 2" than sending snapshots. a normal turn is has uncomressed data ~5kb from which >50% is additional checkup data to detect oos errors.

I don't understand. How is it less data to send all movements and attacks that have occurred throughout the entire game plus OOS detection data than to just send where the units are at the current turn?

because the info the clients need is far more info than "where the units stands". At the latest when it is the local players turn it needs the complete gamestate (=snapshot) which contains which attacks the untis have, the wml variables, which events are currently active.... .

[Pentarctagon]

I don't understand. How is it less data to send all movements and attacks that have occurred throughout the entire game plus OOS detection data than to just send where the units are at the current turn?

because the info the clients need is far more info than "where the units stands". At the latest when it is the local players turn it needs the complete gamestate (=snapshot) which contains which attacks the untis have, the wml variables, which events are currently active.... .

I know it needs the gamestate, but why does it need the game history as well?

[GunChleoc]

I know it needs the gamestate, but why does it need the game history as well?

If I understand what has been posted correctly, it gets the history instead of the game state. The game state is then calculated from the history. There is no game state transmitted at all.

[gfgtdf]

I know it needs the gamestate, but why does it need the game history as well?

If I understand what has been posted correctly, it gets the history instead of the game state. The game state is then calculated from the history. There is no game state transmitted at all.

yes exactly, the gamestate is calculated on all clients synchronously using the history. So we don't need to transfer all the gamestate data.

[iceiceice]

I know it needs the gamestate, but why does it need the game history as well?

I'm just guessing here, but what I am taking away from gfgtfd's comments is that even though the game history determines the game state, it still may take more bits to write down the game state than to write down the game history.

One explanation for how this might happen is that, in a game state (snapshot), every unit is written down with ALL its details. That means, all its max hp, resistances, all its attacks, all the strings which correspond to the images corresponding to those attacks, all the wml for its animations and strings for this images, the entire help file description string, etc. And this is repeated for every instance of that unit. This guarantees that future versions of wesnoth will have at least a chance to open saves from past versions, even if they are really crazy custom scenarios, and that crazy custom scenarios can be reloaded from a save. But it may also contribute dramatically to snapshot size.

The alternative, where you write down just the [replay_start], which is a snapshot of turn 1, and the [replay] which is all the orders and rng roles in order made by players, doesn't do this. You will just store the era, which will describe every unit once, and reconstruct everything from that.

gfgtdf also writes that all wml variables and scripted events must be written down in any snapshot as well. Actually I think these must appear in the [replay_start] also and its not clear that this will contribute more information to [replay_start] than to [snap_shot] or vice versa.

*However*, if the info is transmitted using gzip compression, then I would think that the repetition of unit data would not matter. In fact, we discussed this in an earlier thread, where shadowm suggested, and my experiments seemed to confirm, that in a compressed state [replay] is usually the most significant part of any file, be it a vanilla wesnoth game or a modded scenario.

The reason suggested there is that while [snapshot] is large when uncompressed, it is mostly because these unit descriptions keep getting repeated, so it is easily compressible. On the other hand large sequences of orders are rarely repeated, and of course rng sequences should basically never compress. So the [replay] part of the file should not be expected to compress much at all.

(It will compress somewhat because strings like "[command]", "[random]" will each be compressed to like 1 or 2 256-bit symbols. But that's all you will get, and of course that and much more will happen in the [snapshot].)

So this suggests to me that Pentarctagon would be right and sending the snapshot could be better. It would not be hard to do some experiments.

Edit: The experiments from the previous thread don't actually address our question here -- the question there was, is there savings if you archive a folder full of replays instead of zipping them separately. It's relevant however, because [replay_start] sections of replays may often be similar if the same era is used, because of repeated unit descriptions, while [replay] sections will always be pretty different. The fact that there was no savings suggests that [replay] doesn't compress as well as [replay_start], and this is a pretty believable theory. So we can make educated guesses, but a better experiment would be to gzip [replay], [replay_start], and [snapshot] sections from various replays and look at their compression ratios / compressed sizes.

[gfgtdf]

even if the replay in a savefile is be bigger than the snapshot in a savefile, remember that the replay in the savegame contains the whole replay, of the game, if for example the savegame is from turn 20, it contains the replay of 20 turns and the replay start, to estimate which data is less to send durign an actual game in for one turn, you have to compare the size of replay of one turn to the size of the snapshot.

and at last you shouldn't forget that some people WANT to see exactly what other player did, and also note, that if the other clients just send the snapshot the it no possibilty to determine wether the other players cheated.

[iceiceice]

even if the replay in a savefile is be bigger than the snapshot in a savefile, remember that the replay in the savegame contains the whole replay, of the game, if for example the savegame is from turn 20, it contains the replay of 20 turns and the replay start, to estimate which data is less to send durign an actual game in for one turn, you have to compare the size of replay of one turn to the size of the snapshot.

Good point. So the test should be done using save files at some point in the middle of the game. I assume that the uncompressed savefile wml is pretty much the same as the uncompressed network wml? I guess I remember reading something about simple_wml at some point but I think that is only relevant to the server implementation...?

and at last you shouldn't forget that some people WANT to see exactly what other player did, and also note, that if the other clients just send the snapshot the it no possibilty to determine wether the other players cheated.

Very good point. So the server should in any case continue to save replays exactly as it currently does, so that this is still possible


Edit: Relevant to the first point, it's possible that we could set it up so that in the first 20 turns, the server will just send the game history, but after some cutoff it switches to requesting the game state from the host.

[gfgtdf]

just to be sure, are you talking about the situaltion during a normal game or when one player disconnects and rejoins? i am talking about the first.

ahm no my point was that it no use to compare the size of a 20 turn replay because during one turn only the replay data from one turn is send, which is then the (uncompressed) 1/20 of the replay data of a 20 turn savefile.

and dont dont think people want to watch the rapleys just to notice that someone cheated, currently cheating is detected in form of oos erros.

talking about the situation that someone dissconnects and rejoins: maybe we could make a solution that the rejoining player can just use the data form his last autosave file, and then gets replay data form that turn until the current state.

[iceiceice]

Ok, I thought everyone was talking about the situation when a player joins the game. It is obvious that for just a turn between connected players you should just send the moves of the turn.

I think that its possible that sending [snapshot] from the host could, in long games, be always better than sending [replay_start] and [replay] from the server. It might be better also to allow *observers* to choose this option.

But your solution of loading from the last autosave and then sending some turns of [replay] to catchup a dc player is clearly the best for that case. You would have to code some new features into the server though I think.

[gfgtdf]

Ok, I thought everyone was talking about the situation when a player joins the game. It is obvious that for just a turn between connected players you should just send the moves of the turn.

puh i already thought the majority of people in this theads wants to replace the current sync system with sending snapshots :s.

I think that its possible that sending [snapshot] from the host could, in long games, be always better than sending [replay_start] and [replay] from the server. It might be better also to allow *observers* to choose this option.

ye i think you are right even, even if you execute the usercommands silent, meaning without showing anything to the user, the time to calcualte the gamestate could be quite long.

But your solution of loading from the last autosave and then sending some turns of [replay] to catchup a dc player is clearly the best for that case. You would have to code some new features into the server though I think.

here the code that sends replays: https://github.com/wesnoth/wesnoth-old/ ... e.cpp#L991
line 991 sends the [replay_start] and line 1002 the [replay]
for example if we just alow the client to specify wether the replay_start should be ommitted, we would save something, and could extract the important part of the [replay] on clientside if we don't want to do it serverside.

i think the server side code would be less hard than the client side code.

[linuxminter]

did i understand you correct that you normaly play 20-50 mp games at the same time?

I have had to quit about 20 - 50 multiplayer games from this location on separate occasions.

I had a very nice multiplayer experience tonight in my other location, however. I played Wesnoth for about 7 - 10 hours. Had absolutely no problems. I watched as a couple of other players disconnected and suffered lag. I'm a lot more sympathetic due to my own experiences. Most players don't seem to have problems. I'd guess that the percentage of players that encounter network issues is around 15-20%, tops. The numbers may be skewed because those that encounter problems all the time probably quit trying at some point.

If a developer is interested in troubleshooting lag and disconnects, I am willing to help with the testing/QA. Feel free to shoot me an email. Email may be best, because I don't visit the forum on a daily basis.

[iceiceice]

for example if we just alow the client to specify wether the replay_start should be ommitted, we would save something, and could extract the important part of the [replay] on clientside if we don't want to do it serverside.

i think the server side code would be less hard than the client side code.

I think this is a good idea. On client side I guess you would just count through [endturn] commands until you have caught up to the autosave?

I decided to make good on what I said and do an actual test. I have 8 saves from 1 v 1 wesnoth games which are stopped in the middle, I was using these to test my AI project. I made them by downloading a bunch of Cackfiend's and Amikrop1's ladder replays (in v1.10), loading in 1.11 from replay start and playing through, stopping them just before the climactic battle, and saving. It took me about 15 minutes to write scripts which grab the [replay], [replay_start], and [snapshot] sections and compress them. (It should have been <5 min but alas I am not a black belt in sed-fu, merely a yellow belt stripe ). Here's my results:

Code: Select all

drwxr-xr-x 2 chris chris   4096 Nov 30 22:11 G
drwxr-xr-x 2 chris chris   4096 Nov 30 21:57 O
-rw-r--r-- 1 chris chris 204068 Nov 30 22:30 replay1
-rw-r--r-- 1 chris chris  20144 Nov 30 22:30 replay1.gz
-rw-r--r-- 1 chris chris 177636 Nov 30 22:30 replay2
-rw-r--r-- 1 chris chris  18504 Nov 30 22:30 replay2.gz
-rw-r--r-- 1 chris chris 185000 Nov 30 22:30 replay3
-rw-r--r-- 1 chris chris  19794 Nov 30 22:30 replay3.gz
-rw-r--r-- 1 chris chris 339977 Nov 30 22:30 replay4
-rw-r--r-- 1 chris chris  32119 Nov 30 22:30 replay4.gz
-rw-r--r-- 1 chris chris 334727 Nov 30 22:30 replay5
-rw-r--r-- 1 chris chris  33233 Nov 30 22:30 replay5.gz
-rw-r--r-- 1 chris chris 113644 Nov 30 22:30 replay6
-rw-r--r-- 1 chris chris  12582 Nov 30 22:30 replay6.gz
-rw-r--r-- 1 chris chris 113555 Nov 30 22:30 replay7
-rw-r--r-- 1 chris chris  13581 Nov 30 22:30 replay7.gz
-rw-r--r-- 1 chris chris 339977 Nov 30 22:30 replay8
-rw-r--r-- 1 chris chris  32119 Nov 30 22:30 replay8.gz
-rwxr-xr-x 1 chris chris     44 Nov 30 22:07 replay.sed
-rw-r--r-- 1 chris chris      0 Nov 30 22:05 replay_start
-rw-r--r-- 1 chris chris  30046 Nov 30 22:30 replay_start1
-rw-r--r-- 1 chris chris   5199 Nov 30 22:30 replay_start1.gz
-rw-r--r-- 1 chris chris  26609 Nov 30 22:30 replay_start2
-rw-r--r-- 1 chris chris   4816 Nov 30 22:30 replay_start2.gz
-rw-r--r-- 1 chris chris  27254 Nov 30 22:30 replay_start3
-rw-r--r-- 1 chris chris   5465 Nov 30 22:30 replay_start3.gz
-rw-r--r-- 1 chris chris  35983 Nov 30 22:30 replay_start4
-rw-r--r-- 1 chris chris   5479 Nov 30 22:30 replay_start4.gz
-rw-r--r-- 1 chris chris  42401 Nov 30 22:30 replay_start5
-rw-r--r-- 1 chris chris   6621 Nov 30 22:30 replay_start5.gz
-rw-r--r-- 1 chris chris  35725 Nov 30 22:30 replay_start6
-rw-r--r-- 1 chris chris   5721 Nov 30 22:30 replay_start6.gz
-rw-r--r-- 1 chris chris  35019 Nov 30 22:30 replay_start7
-rw-r--r-- 1 chris chris   5574 Nov 30 22:30 replay_start7.gz
-rw-r--r-- 1 chris chris  35983 Nov 30 22:30 replay_start8
-rw-r--r-- 1 chris chris   5479 Nov 30 22:30 replay_start8.gz
-rwxr-xr-x 1 chris chris     56 Nov 30 22:08 replay_start.sed
-rwxr-xr-x 1 chris chris    340 Nov 30 22:11 script
-rw-r--r-- 1 chris chris      0 Nov 30 22:05 snapshot
-rw-r--r-- 1 chris chris 197182 Nov 30 22:30 snapshot1
-rw-r--r-- 1 chris chris  22199 Nov 30 22:30 snapshot1.gz
-rw-r--r-- 1 chris chris 190534 Nov 30 22:30 snapshot2
-rw-r--r-- 1 chris chris  19390 Nov 30 22:30 snapshot2.gz
-rw-r--r-- 1 chris chris 206360 Nov 30 22:30 snapshot3
-rw-r--r-- 1 chris chris  20819 Nov 30 22:30 snapshot3.gz
-rw-r--r-- 1 chris chris 244213 Nov 30 22:30 snapshot4
-rw-r--r-- 1 chris chris  20501 Nov 30 22:30 snapshot4.gz
-rw-r--r-- 1 chris chris 266072 Nov 30 22:30 snapshot5
-rw-r--r-- 1 chris chris  21488 Nov 30 22:30 snapshot5.gz
-rw-r--r-- 1 chris chris 193543 Nov 30 22:30 snapshot6
-rw-r--r-- 1 chris chris  20553 Nov 30 22:30 snapshot6.gz
-rw-r--r-- 1 chris chris 259765 Nov 30 22:30 snapshot7
-rw-r--r-- 1 chris chris  24132 Nov 30 22:30 snapshot7.gz
-rw-r--r-- 1 chris chris 244583 Nov 30 22:30 snapshot8
-rw-r--r-- 1 chris chris  20496 Nov 30 22:30 snapshot8.gz
-rwxr-xr-x 1 chris chris     48 Nov 30 22:08 snapshot.sed
-rw-r--r-- 1 chris chris 741110 Nov 30 22:24 test.tar.gz

Attached find the scripts to generate, 1.10 original gzips in /O, and 1.11 modified in /G.

The saves correspond to games at various points in time. Replay 1 is earliest, I think it is at like turn 6 or 7. Replay 4 is probably the latest, it is at like turn 22 or turn 24 or something. (Actually, it looks like 8 is the same as 4... oops.) The rest are somewhere in between.

These results don't confirm what I said, instead it looks like [replay] and [snapshot] both usually compress at about a 10 - 1 ratio. [replay_start] compresses a bit less, like 7:1. It looks like [snapshot] in these games is always about 20k-25k bytes compressed. [replay_start] is always about 5k bytes compressed. And [replay] varies alot but even in long games with a lot of units may only be about 30k bytes compressed.

So I'm not sure if a patch to send [snapshot] instead of [replay] and [replay_start] is really worth it. Probably you would need the game to go like 40 turns before it will make even a factor of two difference in transmission size. (Edit: Also my test is inherently biased against [snapshot] since I picked the turns which have the most units alive and in position to attack. So there's that also.)

The players for which it would make a difference are those who play wesnoth on slow machines, which actually struggle to compute the [snapshot] from the [replay]. I know that my old netbook definitely struggled compared to my new laptop. The new laptop is probably 10-100 times faster to play through a replay with quick saves (no animations option). On the old netbook it would take about a second if I clicked "play full turn" in the replay viewer with animations off. On the new machine I can actually rapid click that button without missing.

Anyways that's some food for thought. It might be a good idea to also test some savegames with lots of scripted events / going into many more turns. The patch you suggest, to try to load from an autosave instead of replay_start, is probably a good idea no matter what.

Edit: If it doesn't really make a big difference anyway, it might not even be necessary to modify the server code, just read have the client know to skip replay_start as well.

Edit: I repeated the test with bz2 compression since we are switching to that in 1.12 anyways, just to see if it changed anything.

It looks like shorter end replays are still about 10:1, but longer replays are more like 12:1 or 15:1. replay_start is about 7:1 still. And snapshot is about 12:1 or 15:1, similar to replay. So I think this gives less reason to make a patch that sends [snapshot], unless for the reason of helping out slow machines.

Code: Select all

drwxr-xr-x 2 chris chris   4096 Nov 30 23:01 G
drwxr-xr-x 2 chris chris   4096 Nov 30 23:01 O
-rw-r--r-- 1 chris chris 204068 Nov 30 23:03 replay1
-rw-r--r-- 1 chris chris  13387 Nov 30 23:03 replay1.bz2
-rw-r--r-- 1 chris chris 177636 Nov 30 23:03 replay2
-rw-r--r-- 1 chris chris  13135 Nov 30 23:03 replay2.bz2
-rw-r--r-- 1 chris chris 185000 Nov 30 23:03 replay3
-rw-r--r-- 1 chris chris  13940 Nov 30 23:03 replay3.bz2
-rw-r--r-- 1 chris chris 339977 Nov 30 23:03 replay4
-rw-r--r-- 1 chris chris  20199 Nov 30 23:03 replay4.bz2
-rw-r--r-- 1 chris chris 334727 Nov 30 23:03 replay5
-rw-r--r-- 1 chris chris  19938 Nov 30 23:03 replay5.bz2
-rw-r--r-- 1 chris chris 113644 Nov 30 23:03 replay6
-rw-r--r-- 1 chris chris   9836 Nov 30 23:03 replay6.bz2
-rw-r--r-- 1 chris chris 113555 Nov 30 23:03 replay7
-rw-r--r-- 1 chris chris  10706 Nov 30 23:03 replay7.bz2
-rw-r--r-- 1 chris chris 339977 Nov 30 23:03 replay8
-rw-r--r-- 1 chris chris  20199 Nov 30 23:03 replay8.bz2
-rwxr-xr-x 1 chris chris     44 Nov 30 23:01 replay.sed
-rw-r--r-- 1 chris chris  30046 Nov 30 23:03 replay_start1
-rw-r--r-- 1 chris chris   4949 Nov 30 23:03 replay_start1.bz2
-rw-r--r-- 1 chris chris  26609 Nov 30 23:03 replay_start2
-rw-r--r-- 1 chris chris   4425 Nov 30 23:03 replay_start2.bz2
-rw-r--r-- 1 chris chris  27254 Nov 30 23:03 replay_start3
-rw-r--r-- 1 chris chris   5127 Nov 30 23:03 replay_start3.bz2
-rw-r--r-- 1 chris chris  35983 Nov 30 23:03 replay_start4
-rw-r--r-- 1 chris chris   5480 Nov 30 23:03 replay_start4.bz2
-rw-r--r-- 1 chris chris  42401 Nov 30 23:03 replay_start5
-rw-r--r-- 1 chris chris   6516 Nov 30 23:03 replay_start5.bz2
-rw-r--r-- 1 chris chris  35725 Nov 30 23:03 replay_start6
-rw-r--r-- 1 chris chris   5439 Nov 30 23:03 replay_start6.bz2
-rw-r--r-- 1 chris chris  35019 Nov 30 23:03 replay_start7
-rw-r--r-- 1 chris chris   5248 Nov 30 23:03 replay_start7.bz2
-rw-r--r-- 1 chris chris  35983 Nov 30 23:03 replay_start8
-rw-r--r-- 1 chris chris   5480 Nov 30 23:03 replay_start8.bz2
-rwxr-xr-x 1 chris chris     56 Nov 30 23:01 replay_start.sed
-rwxr-xr-x 1 chris chris    346 Nov 30 23:03 script
-rw-r--r-- 1 chris chris 197182 Nov 30 23:03 snapshot1
-rw-r--r-- 1 chris chris  17086 Nov 30 23:03 snapshot1.bz2
-rw-r--r-- 1 chris chris 190534 Nov 30 23:03 snapshot2
-rw-r--r-- 1 chris chris  15674 Nov 30 23:03 snapshot2.bz2
-rw-r--r-- 1 chris chris 206360 Nov 30 23:03 snapshot3
-rw-r--r-- 1 chris chris  16775 Nov 30 23:03 snapshot3.bz2
-rw-r--r-- 1 chris chris 244213 Nov 30 23:03 snapshot4
-rw-r--r-- 1 chris chris  16470 Nov 30 23:03 snapshot4.bz2
-rw-r--r-- 1 chris chris 266072 Nov 30 23:03 snapshot5
-rw-r--r-- 1 chris chris  16868 Nov 30 23:03 snapshot5.bz2
-rw-r--r-- 1 chris chris 193543 Nov 30 23:03 snapshot6
-rw-r--r-- 1 chris chris  15848 Nov 30 23:03 snapshot6.bz2
-rw-r--r-- 1 chris chris 259765 Nov 30 23:03 snapshot7
-rw-r--r-- 1 chris chris  17104 Nov 30 23:03 snapshot7.bz2
-rw-r--r-- 1 chris chris 244583 Nov 30 23:03 snapshot8
-rw-r--r-- 1 chris chris  16497 Nov 30 23:03 snapshot8.bz2
-rwxr-xr-x 1 chris chris     48 Nov 30 23:01 snapshot.sed