Multiple room sync & subdocument

4

I believe @dmonad is working on a significant update to y-websocket. If his update doesn’t include multiplexing support, I’d be happy to adapt my solution and submit a PR once the update goes out. As it stands now, my fork of y-websocket has diverged greatly from the current version of y-websocket and would be made irrelevant once the new y-websocket goes out anyways. That being said, in the meantime I can outline a loose solution:

Add new message types such that the full y-websocket protocol is:

message formats:
[messageSync][messageType][data]
[messageSyncSub][subdocId][messageType][data]

Extend websocket provider on the client and WSSharedDoc on the server to have a data structure that tracks loaded subdocs.

class WSSharedDoc extends Y.Doc {
   ...
   subdocs = {}
   ...
   loadSubdoc(subdocId) {
      let subdoc = getYDoc(subdocId)
      this.subdocs[subdocId] = subdoc
      return subdoc
      // or return an already created subdoc
   }
}

const messageHandler = (conn, doc, message) => {
   const encoder = encoding.createEncoder()
   const decoder = decoding.createDecoder(message)
   const messageType = decoding.readVarUint(decoder)
   
   switch(messageType) {
      case messageSync:
         // same as default y-websocket implementation
      case messageSubSync:
        let subdocId = decoding.readVarString(decoder)
        let subdoc = doc.loadSubdoc(subdocId)
        // now that we have the subdoc, the sync process becomes identical to the standard case.
        // be sure to broadcast any changes uses your subsync message type
        syncProtocol.readSyncMessage(decoder, encoder, subdoc, null)
        send(doc, conn, encoding.toUIntArray(encoder))
   }
}

When we detect a subdoc sync message, we just extract the subdocID, and then from there, the sync process becomes identical to a non-subdoc. There’s probably a clever way of reducing code re-use here, but I haven’t refactored much. Regardless, it works very well and can sync an enormous amount of subdocs very quickly.

The client does the exact same thing. Add a subdocs collection to WebsocketProvider, when we detect a subdoc message, use the subdoc ID to load the new Y.doc into memory, and then we use the contents of the message to apply updates to the doc, just like a non-subdoc Y.Doc. A lot of these details change when you introduce YJS’ newest features, mergeUpdate and diffUpdate, but I’m still getting acquainted with the api.

4 Likes