Definitely the most technical talk I attended today (in a good way). We had arrived at some similar conclusions about an approach for some skunkworks tech research inside Ensemble. I was hoping for a little more Mercs 2 demo goodness, to be honest, but the talk was quite engaging on a difficult subject.
Mercenaries 2: Networked Physics in a Large Streaming World
Glenn Fiedler
Large world 8k x 8k. Havok physics. Every object destructible. Emergent gameplay, not scripting. Drop in co-op at any time (2 players). Which has been quite a challenge!
Most network games today are static worlds. Not a derogatory term, just analysis – static BSP with characters running around. Mercs 2 is a step toward a dynamic world. This requires a different networking model.
Trend towards this – more dynamic today than it was back in the 90s.
Static world to establish context:
Input, state. Determinism: given same state + inputs, same simulation results. Not necessarily hard determinism like in an RTS though.
Pure client / server: Client is dumb terminal, but input has to wait for latency.
Client side prediction: Hide latency on the client – move fwd immediately without waiting for latency. Server needs to be able to correct the client (accumulated errors, anti-cheat tech). Correction is in the past, so how do we apply it? Rewind and replay is one solution.
In a dynamic world – why do we have to change? Player interacts with physically simulated objects. It is a two-way interaction! Player > objects > player. This make things complicated!
Blocks… not too bad. But then you have stacks! Joints, linked bodies. Not going to work if you apply those one at a time, need an integrated simulation.
Client side prediction: We still want this, but can’t afford to rewind & replay. If you just did one object you would get horribly divergent, something way incorrect. But what if we just didn’t apply corrections? Limited client side authority.
Authority management: Client takes authority over objects he interacts with. These objects are client side predicted along with client player object. Client no longer accepts server corrections for these objects. Authority propagates. Works really really well when two people aren’t both working on the same islands.
Client can turn white objects blue. Client authority trumps default authority. Server can turn white and blue objects red – server authority trumps all. Objects turn white again when they come to rest. Player cubes are always colored.
Fixed delta time. Variable delta gives no good way to explain exactly when to apply things. Client runs ahead of server, delivers input/state early. Apply state at a known time on a given machine. Only stepping forward a multiple of integer frames (v-sync). Everyone using 60hz sim internally. Havok doesn’t let you move things independently, you have to sim the whole world at once. Jitter is a little trickier, but you can filter to provide estimates. Input should always extrapolate from known state on the other machine. All clients ahead in time, but each can be differently ahead. So when input arrives early it just leaves it in the input buffer.
Speed up / slow down time on clients relative to server. Don’t actually scale time with fixed delta timestep. Very subtle client effect. Dynamically adjust to network conditions.
To set physics state, snap it hard (to support physics stacks). If you interpolate, the stack will fall. Correct one meter to the left, you’d topple the stack unless you do this with a hard state change. Visual smoothing on top.
How to compress physics state? Compress quaternion at reasonable precision. Low precision vector, known radius, quantization. Linear velocity / angular velocity, bounded in Havok and sent as compressed data. Quantize on both sides though! So clients have to apply the compression locally in their sim, not just in transmission.
How do we extend this to a large streaming world? “Object scoping”. Bubble around each player in Mercs 2. Actually on each object. Objects within this bubble wake up an simulate. This is different than creating and deleting objects, just waking them up. Separate treatment in Mercs 2 of dynamic vs pre-placed objects to help with this.
Even with this scoping, you can’t fit all objects in your packet. What are the N most important? Standard object prioritization ( by distance, distributive priority – longer without update more important).
Authority based priority? Blue objects need to be sent from client to server, and the red ones need to go from server to client.
Conclusion: client side predict with authority management. Using scoping and authority management to determine objects to send.
Corrections arrive in latent time, then transmitted into predictive time on the client. If there is a conflict (a blue item that turned red), you see a latency-time pop / warp on the client.
Q: How do you handle stacks of objects that are larger than the packet-limitation on objects? In the future this scale won’t be a problem. For now the mitigation is that the state updates go into predictive time.
Q: Security issues? It’s a co-op game so we don’t care.
Q: How do you handle non-determinsm in Havok? As long as it is roughly close enough, we don’t care. A second of approximate determinism is all we care about. Running whole simulation, just slamming in a subset of state for things we receive updates on.
Q. Havok’s islands, or roll your own? Demo just blasting all the state, can’t talk about M2 since still in development.
Objects that are rolling stay in authority because they don’t come to rest. Weak authority where you aren’t directly touching objects. These are some complications, but generally nothing smoothing can’t deal with. Worst case you wind up snapping back.
Some things aren’t distributed – like scripting execution.
No explicit state update when a client joins – the “washing machine” model, eventually objects cycle around. In an initial streaming state wouldn’t let the client grab authority yet.