I’ve been digging into various threads on efficient document storage but haven’t landed on a solid approach yet. I’m reaching out to see if anyone can provide some clarity or suggestions.
My setup is as follows:
- Frontend: prosemirror + yjs (+ custom provider using websockets and offline cache)
- Backend: golang (websockets)
At the moment, a new document is initialized when created by a user, with updates stored individually. When a page is loaded, a refresh request fetches the document along with all updates via the websocket server. The document gets loaded, updates are applied and everything runs smoothly.
However, the hiccup arises when, say, 10 users are collaborating on a lengthy document, generating a multitude of updates rapidly. If a new user joins in, the initial load can get quite cumbersome as it involves loading all updates made since the document’s creation.
I’m considering two main solutions:
1- On the client side, have regular calls to
Y.encodeStateAsUpdate(doc) to save it as the initial document entity, while removing all preceding updates. But, I suspect this might disrupt the collaboration amongst users editing the document concurrently. How and who should trigger this action is also a concern.
2- On the server side, load the document with updates, run
Y.encodeStateAsUpdate(doc) to update the document entity, and clear all previous updates. This seems to only affect new users, keeping the experience intact for current users, unless there are unsynced offline updates. This solution, though, necessitates a new Node service, veering from our current all-Golang backend which isn’t ideal.
I’m very open to any insights or suggestions on these methods or any other efficient way to tackle this issue.
I’ve also looked into these threads:
- Handling slow mergeUpdates on server
- How to implement data persistence on the server side
- Guidance on persistence storage and working with databases
- Best ways to compress size of an Document/Update in the DB
Appreciate your help!