Syncing servers (horizontal scaling)?

1

Yjs is great, thanks!
I have a single server implementation of yjs that works great. I store the state encoding (encodeStateAsUpdate) in the db and do a Y.applyUpdate with it on the server for the first user who connects to that room.
To allow horizon scaling with multiple server instance, I added redis and redis-adapter. I use socket.io namespaces, so all broadcasts are sent to clients on all server instances. Out of the box, awarenesses worked great. I could see the cursor from users on all server instances. But updates would only work in one direction from the last server instance.

Is this happening because the server instances are not sharing vectors of the document?
How can I share the vectors between to server instances (e.g. use encodeStateVectorFromUpdate as input to applyUpdate on secondary server instances)?

2

I’m not sure, but maybe this reference would be useful for you if you’re not already familiar with it: GitHub - kapv89/yjs-scalable-ws-backend

3

Thanks @raine. Yeah, I saw that. I think that solution is similar to mine in regards to yjs. They don’t share vectors. I’ll have to dive deeper into to see if they are doing something different than me. If their solution is working, does that mean vector’s don’t need to be shared, each Y.Doc on a separate server just needs a common encodeStateAsUpdate?

4

Unfortunately I couldn’t say. I don’t know how horizontal scaling works in general, much less with Yjs :grin:. Good luck!

5

lol, thanks :slight_smile:

I think :crossed_fingers: I figured it out. Looks like the shared vector needs to be the first applyUpdate for the docs to be shared.

Server sending vector to client:

  private readonly startSynchronization = (socket: Socket, doc: YDocument): void => {
    socket.emit('sync-step-1', Y.encodeStateVector(doc), (update: Uint8Array) => {
      Y.applyUpdate(doc, new Uint8Array(update), this)
    })
    socket.emit('awareness-update', AwarenessProtocol.encodeAwarenessUpdate(doc.awareness, Array.from(doc.awareness.getStates().keys())))
  }

Client adds the vector to it’s Y.Doc and returns to the server to be applyUpdate

private readonly initSyncListeners = (): void => {
    this.socket.on('sync-step-1', (stateVector: ArrayBuffer, syncStep2: (update: Uint8Array) => void) => {
      syncStep2(Y.encodeStateAsUpdate(this.doc, new Uint8Array(stateVector)))
      this.synced = true
    })

    this.socket.on('sync-update', this.onSocketSyncUpdate)
  }
1 Like
6

@kapv89 @dmonad Can someone please explain conceptually what is needed for server syncing? Is server syncing the exact same three step process as client-server syncing (syncStep1, syncStep2, syncUpdate)?

Given I have two servers (A and B) each with their own clients, is this the correct process?:

  1. Server A sends ServerB it’s Y.encodeStateVector(doc) (assuming ServerA was running first)

  2. ServerB responds to ServerA by sending its Y.encodeStateAsUpdate(this.doc, serverAStateVector) to ServerA

  3. ServerA does a Y.applyUpdate(doc, serverBstateUpdate)

  4. ServerA sends its Y.encodeStateAsUpdate(ydoc) to ServerB

When I do this, I’m seeing all the awarenesses and deletes moving in both directions (between servers), but the inserts are only moving in one direction (e.g. ServerA to ServerB). The problem seems to be that updates from one of the servers is being ignored by the other server.

Is my conceptual understanding correct for server syncing?

Any ideas what would cause my unusual issue?

7

Yes, from a client-server perspective, server syncs are identical to client syncs in Yjs. They’re both Yjs clients and they exchange updates the same way.

I’m sure you have seen this before, so I’m not sure it’s very helpful, but when I look at the documentation for syncing…

const stateVector1 = Y.encodeStateVector(ydoc1)
const stateVector2 = Y.encodeStateVector(ydoc2)
const diff1 = Y.encodeStateAsUpdate(ydoc1, stateVector2)
const diff2 = Y.encodeStateAsUpdate(ydoc2, stateVector1)
Y.applyUpdate(ydoc1, diff2)
Y.applyUpdate(ydoc2, diff1)

…and compare that to your steps, it seems that you’re missing Server B’s applyUpdate. I could be missing something though.

8

Thanks @raine. Yes, I actually do the last applyUpdate but didn’t show it as step 5 where ServerB does an applyUpdate from the output of ServerA on step 4.

I find the Document Updates documentation to be inconsistent with the code. The documentation implies that two clients (I assume this means any two devices) can be in sync fairly easily (like your example) but all the example code has a complex three step process with syncStep1, syncStep2, and syncUpdate. Why the difference?

9

What do you mean by syncUpdate?

10

I think syncUpdate is just the first update after the sync process.

11

I see. Yeah, getting in sync and staying in sync are slightly different things.

I’m not sure why you are not getting updates on one of your servers though.

1 Like