I find it fascinating when different people independently arrive at the same architecture when working on a hard problem like this. In my company we built our offline-first apps with PowerSync, which has the same idea of optimistic local changes while waiting for the central server to acknowledge the definitive changes. In PowerSync's case, the sync engine reads Postgres replication logs directly.
Yeah, I'm surprised this pattern doesn't have a more general name.
I've also used it to synchronize content for local multi-window editing applications: each window is its own process but routes through one "coordination" process to ensure consistent ordering.
Easily solves some cross-process concurrency issues.
It's not quite the same since we aren't doing any sort of interpolation. I would acknowledge it's a similar idea though. Perhaps the similarity is one reason we came to this solution as a game studio.
For those interested in a keep-everything-hot approach like this, it's worth checking out the 25-year-old library Prevayler. Full ACID guarantees, and radically faster than a database. I happily used it for a project forever ago and was disappointed to see it so thoroughly ignored.
Yeah, I glossed over a bunch of the things we tried near the bottom of the "In-Memory Database" section. One of the things was CRDTs.
We did a few days of CRDT investigation along with the other techniques and decided the juice wasn't worth the squeeze when we found a simpler solution to our problem.
That being said, we've actually retained a CRDT-like implementation in our codebase and have been considering moving shapes (which have a slightly different set of tradeoffs) to that model.
I don’t quite understand the “funnel” section. Users see some change local immediately (S1->S2). And all users send all commands to the host. The host then effectively chooses the sort order and broadcasts back.
So does the initial user effectively rewind and replay state? If action applied to S1 was actually from another player they rewind to S1 and apply the sequence?
How many state snapshots do they need to persist? There was no mention of invertible steps.
Whether users see the action locally or not is decided on a per-action basis. For more visual actions like movement, we use a local imposter that gets overwritten on completion of the action (success or failure). These revert to the most recent “committed” state which already passed through the action funnel.
We have some actions which happen when saving config UI which don’t really need realtime sync when “save” is pressed and we treat them more like a normal POST request (e.g. we wait for full commit before rerendering). The form fields are kinds a built-in imposter.
For state snapshots we only do big ones. The “full save” that happens every 25 actions or so. These aren’t generally used for reverting people.
So it kinda works like:
1. Do I have an outstanding imposter? If so read from it.
I find it fascinating when different people independently arrive at the same architecture when working on a hard problem like this. In my company we built our offline-first apps with PowerSync, which has the same idea of optimistic local changes while waiting for the central server to acknowledge the definitive changes. In PowerSync's case, the sync engine reads Postgres replication logs directly.
Yeah, I'm surprised this pattern doesn't have a more general name.
I've also used it to synchronize content for local multi-window editing applications: each window is its own process but routes through one "coordination" process to ensure consistent ordering.
Easily solves some cross-process concurrency issues.
It does have a name:
https://en.wikipedia.org/wiki/Client-side_prediction
It's not quite the same since we aren't doing any sort of interpolation. I would acknowledge it's a similar idea though. Perhaps the similarity is one reason we came to this solution as a game studio.
In this case it's just a (transaction/commit) log. Very common in distributed systems, databases, and filesystems.
For those interested in a keep-everything-hot approach like this, it's worth checking out the 25-year-old library Prevayler. Full ACID guarantees, and radically faster than a database. I happily used it for a project forever ago and was disappointed to see it so thoroughly ignored.
Oh, this is interesting. What sort of project did you use it for?
I’m surprised there was no mention of operational CRDT’s, or CRDT’s generally.
Yeah, I glossed over a bunch of the things we tried near the bottom of the "In-Memory Database" section. One of the things was CRDTs.
We did a few days of CRDT investigation along with the other techniques and decided the juice wasn't worth the squeeze when we found a simpler solution to our problem.
That being said, we've actually retained a CRDT-like implementation in our codebase and have been considering moving shapes (which have a slightly different set of tradeoffs) to that model.
There's no need for CRDTs here because there is always a central authority.
Cool post!
I don’t quite understand the “funnel” section. Users see some change local immediately (S1->S2). And all users send all commands to the host. The host then effectively chooses the sort order and broadcasts back.
So does the initial user effectively rewind and replay state? If action applied to S1 was actually from another player they rewind to S1 and apply the sequence?
How many state snapshots do they need to persist? There was no mention of invertible steps.
I feel like I’m missing a step.
Whether users see the action locally or not is decided on a per-action basis. For more visual actions like movement, we use a local imposter that gets overwritten on completion of the action (success or failure). These revert to the most recent “committed” state which already passed through the action funnel.
We have some actions which happen when saving config UI which don’t really need realtime sync when “save” is pressed and we treat them more like a normal POST request (e.g. we wait for full commit before rerendering). The form fields are kinds a built-in imposter.
For state snapshots we only do big ones. The “full save” that happens every 25 actions or so. These aren’t generally used for reverting people.
So it kinda works like:
1. Do I have an outstanding imposter? If so read from it.
2. Otherwise read from the committed state.