Now that we have an online multiplayer system and framework in Pokitto+, we have got to know some limitations of multiplaying. One of the most notable is the presence of the network lag. That is something that always will exist in the multiplayer game where the players are in different physical locations. The lag is a direct result from the law of physics, but there are some ways to overcome or at least hide it. The lag is especially disturbing in fast action games, not so much in e.g. turn based games.
Here is my multiplayer action game example which I will use in this experiment
For the gameplay mechanics I was thinking about a kind of tag game but in reverse. You are a fish and you mission is to steal the worm (or the crown or the golden coin?) from the fish which currently carries it. Then you should swim away with the worm and try to prevent other fishes stealing the worm by colliding to you.
To bring some variation and randomness to the game there are also static plants and a moving medusa on the water which you must go around. By eating a randomly appearing shrimps you can get a speed boost for few seconds.
For the winning condition I am not yet sure which one wins: the player who has kept the worm the longest time or the player who has the worm when the time runs out.
A bit more about the lag compensation system I am going to experiment. That is called the “rollback netcode”. That is the system developed for GGPO for playing one-to-one fighting games online. The “rollback netcode” system is used in variety of games, like Street Fighter, Nickelodeon All-Star Brawl, and in RetroArch for making online multiplay possible with various retro-system emulators.
I short, Rollback Netcode solves one tricky problem in multiplayer (MP) games. As there exists a lag, the input that comes from the remote player comes always too late, and so it is a response to an incorrect situation. It is a response to the situation in the past. To solve this Rollback Netcode does the following (a two player online game is used as an example):
It tries to guess what the remote response (key input) might be even if it has not come yet so that there is no pause in the gameplay.
When the message comes, it rolls back the game engine to the situation where the remote input is actually related to. It means that if the game has drawn 1000 frames from the start of the game, and the first remote input is related to the frame 995:
The game engine is rolled back to the frame 995. All the frames that we had run after that are now “wiped out”.
Then we replay the game engine (without drawing) to from the frame 995 to the frame 1000 using the actual input of both players, the remote and local player.
That way we do not miss any interactions between players due the lag and the events come in the "correct " time and correct order. Note that after the replay the end result at the frame 1000 might be different if we guessed wrong in the phase 1, in what case there might be some visual glitches. In reality the guesses are correct in most cases, and the rollback length is so small (typically max 8 frames which in 60 fps game is about 130 ms) that we hardly see any glitches. Of course, this all depends very much on how big the actual lag is.
Note that sometimes “rollback” is called “rewind” and “replay” is called “fast forward”.
The rollback netcode technique is a rather interesting technique. I’m mostly curious how well it performs with limited memory as you have to track the state of everything at each frame with a max lag compensation (ie. max number of frames you can store). One-on-one fighters shouldn’t be too bad as you really just have a few variables per frame if you condense everything well, but for more complex games it might get tricky.
Can’t wait to see how you implement it and how well it turns out.
A short update, as I have been building the framework for Rollback Netcode system. Not much to show yet but the clients can see each other and there is the lag between server and the local client visible after the local player name.
Not surprisingly, the lag in the simulator in windows is low, mostly caused by the FPS limit. The server is running in the same computer as the sim clients.
In the HW the lag variates from 60 to 280 ms. (There is not any averaging).
#5
The beauty of the Rollback Netcode is that basically only key events are being sent, i.e. up,down, left, right, etc. So basically it acts as a remote game control. Any game that has a local multiplay could be, in theory, easily modified to use Rollback Netcode, no matter what kind of game it is.
As you might have guessed there is a price to pay for that kind of generic solution. There are some requirements for the game:
The game should save its full state for last N frames, and be able load any saved state. The “state” can be as simple as the position and velocity of each object, and player scores.
The game engine must be deterministic. It means that if, in the state S1 (e.g. game objects are in certain positions), the game is given the certain input (i.e the keyboard events of each player) it always produces the same end state, S2. So we can rewind and reply the frames as much as we want and end up to the correct state always.
One more requirement (derived from the above requirement) is that it should be possible to run the game engine only, i.e. without drawing anything.
Just a quick observation I’d like to throw out there: because it is deterministic, the system state doesn’t need to be saved every frame; it can be saved in a “coarse-grained” fashion. That is, you can choose keyframes, and when you receive an input, rewind to the next older keyframe before the divergence in the input. Then run the replay including the divergence. Obviously that may result in doing a few more cycles, so there’s a memory/performance tradeoff.
But there’s also nothing that says the keyframes need to be evenly spaced. So I would also suggest that the longer you go without getting a packet from the other user, the more you can start spacing them out, since it will become increasingly likely that the system state is invalid and you’ll be discarding the most recent frames anyway.
All that to say, you might consider a keyframing “schedule” based on the time since the last packet. For example: every frame for the first 100 ms, every other frame for the next 100, and then every third beyond. Maybe take a look at Human Benchmark for inspiration.
A good point! Saving the past states can quickly become a limiting factor in a game what comes to number of game objects and the size of a single game object. I am hoping to use the 2+2 kb additional memory banks (sram1 & sram2) in Pokitto for that but lets see.
There are numerous ways to help actually - for instance, some game elements can be designed to be completely predictable (e.g. a platform at the left side at 0s, at the right side at 5s on so on), so you won’t need to store their state, and just use a formula to recalculate them. it’ll be highly dependent on the gameplay of course
True! Not to talk about “decorative” objects, like bubbles in my test game, which do not need to be in sync.
However, as bubbles still use rand() they affect indirectly to other objects as well (as rand() is deterministic only if there are same number of rand calls). Hmm… it would be worthwhile to have a separate rand-instance for non-deterministic objects.
#6
Before going to the rollback based netcode, lets try something easier. As in this experiment we are seeking for a generic solution we want to send only the input events to the other clients (not e.g. the object coordinates). For that to succeed we need to keep the frames of the remote clients synchronized, so that the event go to the same frames and so the game is acting and looks the same way in each client.
The easiest way to synch is just wait that both clients have drawn the frame 1, then we can proceed to the frame 2, … etc. That is what happens in the video below.
You can see immediately the problem: it is dead slow. That is because there is total 240 ms network delay: 120 ms (from the client 1 to the server) + 120 ms (the server to the client 2). The game performance is limited to 30 fps, but because of the lock-step system and the network delay the real perf is only 3-4 fps.
We can try to ease frame stalls (locking) by delaying the input so that when you press a key the input is read by the game engine only after 3 frames later. That makes possible to send these input events in advance to the remote client and so it does not need to lock the engine as there are already necessary input events waiting in the buffer. That is the video below. The performance is now seemingly better.
However, there are still some stalls as we only delay the input events for 3 frames, but the total net lag is 240 ms, which means about 7 frames. Lets try to put a delay of 7 frames to the input events. As shown in the video below, the performance is now nearly the ideal.
It cannot be seen in the video, but the downside is that as we are now delaying the local and remote input events by 7 frames i.e. about 240 ms, that might affect to the perceived gameplay.
This is how the delay-based multiplay works. It was used a lot in online games before the rollback netcode became popular.
Disclaimer: As usual the anim-gif is much more choppy than the real game.
#7
No updates for some time but I have been slowly coding the delay based system.
I noticed that sending each players input events over UDP needs some mechanism to resend and queue old input events in case there occurs packet losses. The “Reliable UDP” I have implemented for Pokitto already has this kind of system. It also has less lag in this case as it does re-sending handshaking purely between the server-client, not client1-server-client2 which should be done for application specific solutions.
But is RelUDP fast enough for 30 fps communication? When I tested it was lagging at first, at about 20 fps. Then I made some changes to the resending algorithm both in the client and server code (that is the benefit of having an own reliable udp implementation instead of TCP ). Now it sends all “ready-to-send” packets right away instead of sending only one packet per frame. I could get near 30 fps which is very good The remaining problem in the HW is that it occasionally has slowdowns (in a 5-10 seconds interval). During the slowdown the fps drops below 20 fps. I also noticed slowdowns during using the plain UDP so it is not due the RelUDP queuing. Maybe it is some congestion in serial comms between LPC and ESP or something (?). Need to study it more.
After a lot of debugging, it seems like time to time ESP networking is giving UDP packets in bursts to the ESP program. Meaning at first, the UDP packets come in a stedy rate fo about 6-10 seconds. Then comes a “burst” phase where multiple packets come at once. There can be 4-5 bursts in a second. I cannot quite understand the reason for bursts. Maybe ESP is at regular intervals doing some internal work, and during that time the incoming UDP packets are getting queued by ESP networking, and that is causing bursts after our ESP program gets its turn.
Random thoughts
Another thing. Now that we are sending data from the client at 30 fps rate, it means the receiving client gets packets at 60 fps rate when using relUDP protocol (that is received data packets + ack packets from the server related to the sent packtes). Having e.g. 4 players that means 3*60 = 180 fps per client. That is a lot of small packets! Sending each packet from ESP to LPC via the serial requires an interrupt to run in LPC to receive that packet to the ring buffer. While 180 Hz might seem to be a lot it is nothing compared to e.g. the audio interrupt, which can be e.g. 22 kHz. So we should be ok.
Maybe the more critical part is using ESP networking at 180 Hz (or more) frequency? If that becomes a problem, we could always try to pack several data packets in one bigger packet and send that from the server to the client.
Conclusion: Maybe it is better not to pay attention to bursts at this point. It will be seen if they really affect to the gameplay. I need to do more testing with 4 clients, now I have used only 2.
After the game jam I have continued with this experiment. It has been quite difficult and I have fixed many bugs in the server and client side.
Now finally I managed to run the test at 30 fps without errors for a long time (10 minutes or so). I am using ReliableUDP for sending remote keypresses to the other client.
Collect points by eating food bits. Avoid the medusa as it will hit you and take your points. By eating a special food you can morph in to a torpedo for a short time and gain even more speed.
I will soon organize an online multiplay test session to test this. You can attend either with the Pokitto+ HW or with the Simulator (Linux/Win).
Sorry about the choppy gif. The frame rate is about 30 in real life.
There are still few bugs that prevent starting the multiplay test.
Occasional hang which stops all clients. Might be some kind of race condition/deadlock where every client waits a frame from the others and do not send further frames because of that.
The points are not always synchronised, even if the positions of the fishes seem to be in sync. Might be because the food looks to be raffled in different places in each client. The gameplay (and random numbers) should be deterministic but looks like it isn’t…
