Networking rewrite, need help

Networking rewrite, need help

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

genixpro
Posts: 4
Joined: 25 Jun 2007, 01:44

Networking rewrite, need help

Post by genixpro »

Hello, I posted about a new networking model on mantis id=585 (i can't post hyper link, new forum members not allowed apparrently) from which I think Spring could greatly benefit. I have talked with many, people about this new system on chat, including talking to tvo on the bugreport.

The system is a departure from the current system, and the big changes that will result in better performance are as follows.

A) Spring doesn't schedule orders, as in they aren't given a set frame to execute at by the client who gave the order
B) Spring relies on heavy network communication, 30 fps of new_frame and messages, plus when i send an order it has to be sent back to me from the host, causing delays, and the adjustments in case my unit was destroyed in that time.

Both of these contribute to the poor performance we get. The new system builds up from the stability of A, basically, we can have a system where we know whether stuff is going to happen this frame. Also, stuff doesn't need to be sent every frame (for B), every 4 or 5 frames is more than sufficient. no humans can issue orders 30 times per second even if they hammer the mouse with both hands, but network traffic must go at that speed. More traffic means slower overall network performance.

The other game that I have designed networking for, Globulation 2. runs comfortably at about 160 b/s per player, which is next to nothing as far as online games go, even a person with a modem could play with 16 players, as long as he wasn't downloading at the same time. I think spring could achieve the same thing using my new system to its fullest extent.

I'm am willing to help implement the system, but I have a couple requests:
I hate putting together patches. They are always so difficult, especially of something with this many changes. Its even more difficult to keep up to date with SVN (although globulation uses the improved Mercurial), and difficult to track changes even with automated systems. I would prefer having SVN access and coding on another branch, not to affect anything, to make it easier to merge upstream and commit in succession, and test with multiple people. If thats not possible ok.

Also, I'm not really sure how to get started. Perhaps someone with more experience with the dev process. Should I use forums, mailing lists that I should join? Code wise, is there particular classes someone could direct me to to help me get started?
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Post by Masure »

I'm not aware of the Spring engine but as a player, I never experienced network overload issues. The only thing that make game slow down is the CPU needs.

When you have a big game with hundreds units moving, the pathfinding needs a lot of client-side cpu power. When a client can't compute fast enough, the whole game slow down to stay synced between players capabilites. But that's not a network bandwith limit.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Networking rewrite, need help

Post by lurker »

http://spring.clan-sy.com/mantis/view.php?id=585
genixpro wrote:A) Spring doesn't schedule orders, as in they aren't given a set frame to execute at by the client who gave the order
After reading your mantis entry and talking to you I still fail to see why it matters whether it is scheduled by the client or the server.
Client-side has the advantage of allowing p2p, but it also makes everyone's commands lag as much as the slowest player's ping.
genixpro wrote:B) Spring relies on heavy network communication, 30 fps of new_frame and messages, plus when i send an order it has to be sent back to me from the host, causing delays, and the adjustments in case my unit was destroyed in that time.
Wtf? If the command is sceduled client-side, you also have to make adjustments in case your unit is destroyed in that time. And commands only don't have to be sent back from the host if you have pure p2p and everyone has proper port forwarding set up. Think about it. Commands may not have to be sent back to you, but they have to have enough buffer time to be sent to everyone else, so speed isn't improved, and no sending the data back to you will only save a fraction of net data.
genixpro wrote:The new system builds up from the stability of A, basically, we can have a system where we know whether stuff is going to happen this frame.
:evil: we already know whether stuff is going to happen this frame. You wait the ~30ms from frame start until the server sends the message indicating the start of the next frame, and voila, you have the exact list of commands for the frame.
genixpro wrote:I hate putting together patches. They are always so difficult, especially of something with this many changes. Its even more difficult to keep up to date with SVN (although globulation uses the improved Mercurial), and difficult to track changes even with automated systems. I would prefer having SVN access and coding on another branch, not to affect anything, to make it easier to merge upstream and commit in succession, and test with multiple people. If thats not possible ok.
If you want, I'll help with any conflicts in making a patch.
genixpro wrote:Also, I'm not really sure how to get started. Perhaps someone with more experience with the dev process. Should I use forums, mailing lists that I should join? Code wise, is there particular classes someone could direct me to to help me get started?
You can do what I did and start coding at random and just try again when your head explodes. Oh wait, you already know the language... bah you'll be fine.
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Post by Tobi »

Here is the link to mantis, for anyone interested:
http://spring.clan-sy.com/mantis/view.php?id=585

Wrt commit access, drop me a PM with a username & password. I think it is fine if you want to develop in a branch given that you already have experience :-)

As for the process, these forums and IRC (#taspring on FreeNode or #sy on QuakeNet) are preferred, our mailing list has been mostly dead for a while so there's not much point using that.

The struct Command in rts/Game/command.h is the one you want for adding a frameNumber in I think. Basically this class directly reflects what's sent over the net for AI/user commands.

Then there's CCommandAI (rts/Sim/Units/CommandAI.{cpp,h}) which is the base class for the different unit-level AI's, it has a commandQue member of type CCommandQue to hold, well, the command queue ;-) . Every CUnit has a separate CCommandAI (or derived) object.

Further, the lowlevel netcode lives in rts/System/Net mostly, though I'm not sure you need to modify it. And rts/Game.cpp and main.cpp are the things to look at for changes in when e.g. CGame::SimFrame() is executed.
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Networking rewrite, need help

Post by Tobi »

lurker wrote:http://spring.clan-sy.com/mantis/view.php?id=585
genixpro wrote:A) Spring doesn't schedule orders, as in they aren't given a set frame to execute at by the client who gave the order
After reading your mantis entry and talking to you I still fail to see why it matters whether it is scheduled by the client or the server.
Client-side has the advantage of allowing p2p, but it also makes everyone's commands lag as much as the slowest player's ping.
Scheduling orders to execute a fixed time in future instead of executing them as fast as possible (which is the point here, not really whether it's scheduled client side or server side) proves for a smoother user experience, even if the actual delay is larger. Read the articles about the Age Of Empires network model, they have performed a user test that showed this IIRC.

An advantage of scheduling the orders client side is that the client can already render the orders in the GUI: the user gets immediate feedback when he gives an order. (Contrary to the current situation where the GUI only shows the orders after they have roundtripped to the server.)
It has been shown (possibly by the AOE developers too) that immediate feedback makes a game feel less laggy, even when the actual lag is actually larger.
genixpro wrote:B) Spring relies on heavy network communication, 30 fps of new_frame and messages, plus when i send an order it has to be sent back to me from the host, causing delays, and the adjustments in case my unit was destroyed in that time.
Wtf? If the command is sceduled client-side, you also have to make adjustments in case your unit is destroyed in that time. And commands only don't have to be sent back from the host if you have pure p2p and everyone has proper port forwarding set up. Think about it. Commands may not have to be sent back to you, but they have to have enough buffer time to be sent to everyone else, so speed isn't improved, and no sending the data back to you will only save a fraction of net data.
In Spring, commands will have to roundtrip anyway for proper demo recording I think.
genixpro wrote:The new system builds up from the stability of A, basically, we can have a system where we know whether stuff is going to happen this frame.
:evil: we already know whether stuff is going to happen this frame. You wait the ~30ms from frame start until the server sends the message indicating the start of the next frame, and voila, you have the exact list of commands for the frame.
The bad side of our system though, is that the experienced lag varies with the actual lag, because the server just inserts commands in the broadcasted network stream as soon as they arrive. A fluctuating lag feels worse than a stable lag, even if the stable lag is larger than the average fluctuating lag.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Re: Networking rewrite, need help

Post by lurker »

Tobi wrote:An advantage of scheduling the orders client side is that the client can already render the orders in the GUI: the user gets immediate feedback when he gives an order. (Contrary to the current situation where the GUI only shows the orders after they have roundtripped to the server.)
It has been shown (possibly by the AOE developers too) that immediate feedback makes a game feel less laggy, even when the actual lag is actually larger.
Okay. But showing the orders before actually passing them to the units adds a layer of complexity, and I don't think genixpro has mentioned doing anything of the sort yet.
Tobi wrote:The bad side of our system though, is that the experienced lag varies with the actual lag, because the server just inserts commands in the broadcasted network stream as soon as they arrive. A fluctuating lag feels worse than a stable lag, even if the stable lag is larger than the average fluctuating lag.
But does that apply to keeping the same fluctuating lag except for setting the minimum to X? You can't make a randomly slow packet travel faster. Doesn't adding a minimum delay to micro hurt a lot more? You can show the commands instantly without changing the networking at all.
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Post by Masure »

I never feeled the network laggy cause of spring needs. Tobi & lurker, you have the knowledge of the networking system, can you say if the code really needs improvement on network ?

As pathfinding and shoot trajectory computing require a lot of CPU, I believe that the CPU slowdowns issues are the worst thing for a good game experience.

I would rather like someone try to improve that point instead of the netcode.

I'm not a dev and never looked into the sources so that it's just player feeling.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Post by lurker »

This is more about instability. Right now the networking can cause twitchy lag and even implode even when conditions aren't that bad.
Masure
Posts: 581
Joined: 30 Jan 2007, 15:23

Post by Masure »

How to experience those instabilities ? What does it look like when happening ?
Tobi
Spring Developer
Posts: 4598
Joined: 01 Jun 2005, 11:36

Re: Networking rewrite, need help

Post by Tobi »

lurker wrote:
Tobi wrote:The bad side of our system though, is that the experienced lag varies with the actual lag, because the server just inserts commands in the broadcasted network stream as soon as they arrive. A fluctuating lag feels worse than a stable lag, even if the stable lag is larger than the average fluctuating lag.
But does that apply to keeping the same fluctuating lag except for setting the minimum to X? You can't make a randomly slow packet travel faster. Doesn't adding a minimum delay to micro hurt a lot more? You can show the commands instantly without changing the networking at all.
I know that micro is impossible if the game speed varies heavily due to bad network conditions (sometimes the game almost pauses for a (sub)second and then it runs multiple simulation frames too fast to catch up).

I also know micro is hard or impossible if the delay between giving an order and it being executed is larger.

The first thing could be solved by defining a game wide command delay so commands are always executed a fixed time in the future, combined with removal of the NETMSG_NEWFRAME message and executing CGame::SimFrame() at current game speed as long as the commands for the frames to simulate have arrived.

This solution is at the cost of slightly more lag, but I do not know what proves to be worse in reality.
Masure wrote:Tobi & lurker, you have the knowledge of the networking system, can you say if the code really needs improvement on network ?
It is definitely not perfect. As lurker said, even on reasonably normal conditions it sometimes causes really high lag. Though this is more about the high level design then the low level netcode.
Masure wrote:As pathfinding and shoot trajectory computing require a lot of CPU, I believe that the CPU slowdowns issues are the worst thing for a good game experience.

I would rather like someone try to improve that point instead of the netcode.
I think this would be a waste of time because CPUs get faster faster then we can make the code faster.

Personally I am not going to do any micro optimizations before the code design is good enough and the game is 100% stable and never desyncs.
User avatar
smoth
Posts: 22306
Joined: 13 Jan 2005, 00:46

Post by smoth »

Image

Seriously, make one.
genixpro
Posts: 4
Joined: 25 Jun 2007, 01:44

Post by genixpro »

Yes its true that units can die between when an order is requested and when its executed. But at least this time period is known, rather than being inside the round-trip of the server.

User feedback means stuff like placing the building with half-alpha, so the user knows the building is there and will be constructed, so they can continue clicking away placing more buildings even though the order for the first one hasn't even been executed. Or if you right click to destroy the enemy, the red "attack" line can be added immediately so the player knows that the engine knows it will happen, and they can continue with other business.

Also, I've never got local games to work. Presumably because of UDP? Will I need two computers on the network? If so I'll get to work compiling Spring on my windows computer. I suppose I'll need to test windows-linux anyhow while I'm at it.

Despite my big changes, there are little tweaks, like the drawing of units at location + velocity*time, I will turn time into std::min(100ms, time). That way, game is still smooth for lag high speed games, but it appears to freeze for low speed games, which is more predictable and in practice looks nicer than the stuttering thats present.

Another tweak I've described is that the player knows who is being waited on in the game if it is lagging out, rather than being left in the dark. Perhaps that could be done with LuaUI.

At first, I intend to schedule orders at a fixed, 480 ms ahead of time for testing. Afterwards I'll do tests to see good numbers for varied latency, which is far more common. Like that Age of Empires article reads (I read it too), latency changes should be slow, a stable latency is much more predictable to the user.

And yes I fully understand that all orders run adjusted for the player of the highest ping. You don't want anyone with high pings, kill em!
genixpro
Posts: 4
Joined: 25 Jun 2007, 01:44

Post by genixpro »

Also, about recording, we can make recordings much more stable (as in they don't represent the lag of the game that was recorded). Basically, this is because orders where given a set frame to execute at. The recording has all orders, it can execute them normally, no lag. Rather than being a log of net traffic, its a log of game orders.

This has one flaw, and its due to things like text messages. The solution is that text messages and drawing on the map etc.. are considered a kind of "order" and scheduled like anything else is. The user probably wouldn't notice, and if they did they'd appreciate that everything is more consistent. And recordings would be perfect as far as recordings go
imbaczek
Posts: 3629
Joined: 22 Aug 2006, 16:19

Post by imbaczek »

genixpro wrote:Also, I've never got local games to work. Presumably because of UDP? Will I need two computers on the network? If so I'll get to work compiling Spring on my windows computer. I suppose I'll need to test windows-linux anyhow while I'm at it.
You need TASServer and a lobby that understands the protocol version it uses. I had to hack out sending NAT traversal port, because TASClient didn't like it.

Then you simply connect two TASClients, preferably from different directories, host a battle and have two springs running. Setting up a battle without a lobby is fail, don't do that. You can reuse the generated script afterwards.
User avatar
lurker
Posts: 3842
Joined: 08 Jan 2007, 06:13

Post by lurker »

genixpro wrote:Also, about recording, we can make recordings much more stable (as in they don't represent the lag of the game that was recorded). Basically, this is because orders where given a set frame to execute at. The recording has all orders, it can execute them normally, no lag. Rather than being a log of net traffic, its a log of game orders.
Couldn't that also be done with minimal editing of the network stream that is saved? Since they're scheduled by the server, that sounds more like a bug than an actual problem with the way networking runs.
genixpro wrote:And yes I fully understand that all orders run adjusted for the player of the highest ping. You don't want anyone with high pings, kill em!
I know at least with my current connection I can join a game and it hurts no one else, though I can't micro very well. You really think it's better to kick people for something that only currently affects them?
genixpro wrote:Yes its true that units can die between when an order is requested and when its executed. But at least this time period is known, rather than being inside the round-trip of the server.
That helps how? The current system seems to work fine to me, with the order filtered to nothing when recieved as the unit is lost.
genixpro wrote:User feedback means stuff like placing the building with half-alpha, so the user knows the building is there and will be constructed, so they can continue clicking away placing more buildings even though the order for the first one hasn't even been executed. Or if you right click to destroy the enemy, the red "attack" line can be added immediately so the player knows that the engine knows it will happen, and they can continue with other business.
Don't forgot that that can just as easily be done with the current system.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Post by Argh »

Not to be an incredible pain, but if, for example, some of the stuff that is currently synched that doesn't need to be, and should in fact run on COB-style millisecond timing models, instead of synch timing (FX is the biggest area here) then I suspect that we'd lose a lot of lag anyhow.

And there's always the option of dynamically dropping the frames per second of simulation, by automated agreement, based on historical performance.

In that model, the sim checks the clock, sees the slowdown occuring, and asks all parties to wait 60, 90, 120ms before registering new command decisions, for X seconds.

Command registration slows down, as in the AOE model, but the sim does not, so that the explosions and whatnot are still occuring, but new sim decisions aren't, until the next agreed-upon update. This would preserve the illusion of continual motion even when one player's machine is overwhelmed for a few seconds.

What happens now, where a single nuke going off in a packed base causes large delays for everyone (even in single-player, with AIs, when it happens where a real player cannot even see the nuke going off, I might add) and noticable stutter seems counterproductive and maybe preventable.

I think that an AOE model, however beneficial in terms of lag prevention, would be somewhat counterproductive in terms of gameplay, because we're not ordering large flocks around, we're ordering swarms of individuals, and sometimes, the speed at which you can order a particular unit around is vital to gameplay. It'd be a very self-limiting decision.
SJ
Posts: 618
Joined: 13 Aug 2004, 17:13

Post by SJ »

The first thing could be solved by defining a game wide command delay so commands are always executed a fixed time in the future, combined with removal of the NETMSG_NEWFRAME message and executing CGame::SimFrame() at current game speed as long as the commands for the frames to simulate have arrived.
This is basically a contradiction, since we must know that all orders for a frame have arrived we must send something each frame and a human will as noted not send new commands 30 times a second. Also this will make the game more prone to micro pauses since it will be enough that one player get a lag spike and then everyone must wait for his commands from that frame (+delay) to arrive before continuing simulation.

The only way around that is to allow the server to delay his commands if they arrive too late but then you lose the fixed lag on commands thing and you will have to listen for your own commands from the server since they might have been delayed.

Besides we dont really send 30 udp packets over the network each second since the CNet class will buffer them to about 10 per second or so (if no one has changed it). And then the client keeps a small buffer of incoming packets and eats through them slowly to even out the frame rate.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Post by Argh »

<snipped>

Soz, was very tired.
Last edited by Argh on 21 Aug 2007, 22:33, edited 1 time in total.
User avatar
jcnossen
Former Engine Dev
Posts: 2440
Joined: 05 Jun 2005, 19:13

Post by jcnossen »

Argh please be concise and not OT. This thread is about discussing genixpro's proposal.
Auswaschbar
Spring Developer
Posts: 1254
Joined: 24 Jun 2007, 08:34

Post by Auswaschbar »

B) Spring relies on heavy network communication, 30 fps of new_frame and messages, plus when i send an order it has to be sent back to me from the host, causing delays, and the adjustments in case my unit was destroyed in that time.
This is not realy true. In comparison to other games spring generates less traffic. Im not sure but you will have ~ 5MB traffic for a 1 hour game with 12 players.
Also like already mentioned there is some delay which causes some newframe messages to be sent together in one packet. I guess there are ~10 messages per socend.
Both of these contribute to the poor performance we get. The new system builds up from the stability of A, basically, we can have a system where we know whether stuff is going to happen this frame. Also, stuff doesn't need to be sent every frame (for B), every 4 or 5 frames is more than sufficient. no humans can issue orders 30 times per second even if they hammer the mouse with both hands, but network traffic must go at that speed. More traffic means slower overall network performance.
The current networking code take the time since the last message and the size of the buffer into account before decieding if a new packet has to be sent. This means that stuff is sent every 3th or 4th frame when there are no orders and almost instantly when there are some.

my to cents about a p2p system:
Currently commands are executed when they arrive. If you have a fast connction you will hopefully not experience lot of lag. In a p2p system I think you will experience the same lag as the slowest player. If a commad is executed a half second later than its given, this will make stuff like controling units in FPS view somewhat useles.
An other downside is that every client has to upload n-1 times the commands it has given (currently it has to upload it once to he server and then download it again). In addition the protocol overhead will increase about n-1 times since you have to keep track of every paket sent.
[/code]
Post Reply

Return to “Engine”

cron