Extend y-websocket provider to support sub docs synchronization in one websocket connection

Show
1

The native implementation of y-websocket only supports one doc in a room, when nests sub documents in one doc, you have to start a websocket connection for each sub document, this can be not acceptable if you have a server, there can be too many connections in the same time. So, I extend y-websocket, adding doc guid (main document or sub document) into the sync message, then other clients or servers can fetch the right doc to update when receive an message.

here is my implementation (see multidoc branch):
y-multidoc-websocket

client use as follows:

const mainDoc = new Y.Doc()
const provider = new WebsocketProvider(host, roomname, mainDoc)

mainDoc.on('subdocs', ({added, removed, loaded}) => {
  loaded.forEach(subdoc => {
    provider.addSubdoc(subdoc) 
  })
})

const subDoc = mainDoc.get('sub')
subDoc.load()

// after sync finished, you can fetch data out of the sub document
5 Likes
2

I am very interested by this :+1: For now I have one connection per document so it can grow to huge numbers very fast.

3

Yeah, my application had the same problem before, I fixed it by this way.

4

Hi! This is very interesting for my use case too. The documentation mentions

Providers (e.g. y-websocket, y-indexeddb) are responsible for syncing subdocuments. Not all providers support subdocuments yet.

This seems to imply they would receive native support for subdocuments in the future. Do you think your implementation is similar to how this would be implemented by the package officially?

Since I am integrating with synced storage and Vue, I wonder how explicitly loading subdocuments would work with the reactivity mechanism (especially since the subdocument is effectively missing/empty until it’s loaded).

Since my application involves users collaborating on potentially multiple documents, I would also like to use awareness to display cursor positions and participants. Do you think splitting awareness up such that each user only gets the relevant awareness data for the documents they are participating in is possible? For example, if I have 100 documents but one particular user is only using 1 or 2 of them, I don’t want all the cursor and user activity of the users of all the other documents to be synchronized with everybody.

5

This seems to imply they would receive native support for subdocuments in the future. Do you think your implementation is similar to how this would be implemented by the package officially?

Sorry, I haven’t seen the official implementation, but the basic idea is same I think. Yjs provides most basic api already, you can implement your own functions with these basic parts.

Do you think splitting awareness up such that each user only gets the relevant awareness data for the documents they are participating in is possible?

sure, you can do that, awareness is also an independent module, you can customize it as your needs !

6

@LeeSanity - Appreciate your efforts on this, that is what I am exactly looking for(i.e. one connection supports multiple docs), I will do some experiments on your code/bran.

Regarding the official network provider as @douira mentioned, I think they don’t support multiple docs for now after reading the code.

8

@LeeSanity - I think in your code has a bug, the if condition seems wrong, if the second user access that subdoc, then it won’t trigger the sync step 1

if (subm && subm.has(targetDoc.name)) {
            // sync step 1 done before.
          }
1 Like
9

Yeah, you are right !!! It was a quick POC of one day work. Can you just request an issue in the repo? , I will fix it later. Thanks!

1 Like
10

@LeeSanity Do you have any suggestions how to add and remove subdocs directly from the backend and send event to the user?

11

based on my understanding, you can add/remove subdoc in the same way with the client side, to be simple, you can treat the backend as client !

12

@LeeSanity I have implemented the solution, but once the browser has been refreshed, only the root document has been synced, subdocs are existing in the document as empty structure without content. Any suggestions how to solve this.

13

In the root document, subdocs are only references, you must initialize subdocs by yourself