Does anybody have opinions about or data on how much load (however quantified, on hardware of your choosing) a single y-websocket server can support?
1 Like
The performance bottleneck of y-websocket server is the amount of memory that is used by each Y.Doc instance (every Yjs document that is currently being edited is loaded to memory). A single document (a small note, depending on your application), will use about 100kb of memory. That means you should be able to open about 10k connections on a 100mb instance.
Depending on your type of application, your users might create larger documents and you might need more or less memory.
I have plans to rework y-websocket to not load the document to memory at all. Until then you need to plan to scale y-websocket horizontally.
1 Like
Hi @dmonad, can you write a tutorial on scaling y-websocket horizontally? I was initially planning to use sharedb, but the delay in subscribe-queries acknowledging new records and the 16MB mongo-db doc limit have made me look elsewhere for the collaborative diagramming tool I am working on.
I like y-js, but not having any clear guide on how to scale y-websocket horizontally is holding me back. It’d be very helpful if you can finish this page: https://docs.yjs.dev/tutorials/untitled-3
16MB is pretty big for a document, even with Yjs’ ~50% overhead for text documents. Though if you’re talking diagramming, perhaps you are generating tons of updates from user dragging actions or something. Even still, you might do some experiments to confirm the 16MB limit in MongoDB is a bottleneck. Most databases don’t like documents this big; Mongo I think has one of the higher limits. (This can also be increased manually, iirc.) If you use Yjs’s differential update feature, you could technically ensure you never store updates larger than a certain size using a naive bin-packing algorithm, but you’d need to implement this.
One method of scaling is using a Pubsub service. Let’s say you have two instances of your server, A and B. User kapv89 connects to “Cool Document” on server A, and User braden connects to “Cool Document” on server B. When you update “Cool Document”, it gets rebroadcast on any websocket connections so other clients can see the update. Since I’m on Server B, I don’t receive the update and we go out of sync.
Instead of broadcasting messages directly via the websocket connections, you send the update to a Pubsub service, like Redis Pubsub for example. (Again, need to fork y-websocket to implement this). Each server listens to the corresponding Pubsub channel, and when it receives a message over the pubsub channel, it then rebroadcasts it along any active websocket connections.
kapv89 —ws—> Server A ------> Pubsub -----> ServerB —ws–> braden
This would work for any number of instances. The caveat here is each server would be holding the doc in memory. It might also be beneficial to rig up some kind of server affinity, so that connections to the same document prefer being routed to the same server instance. Another thing to watch out for is persistence; if every instance is persisting updates for the documents to a database, it can get tricky depending on how you are persisting state.
1 Like
Hi @kapv89,
@braden put it nicely. You can scale your application horizontally using any pubsub server. y-redis is just one approach. I’m planning to rework the current y-redis implementation which is why the section is still unfinished.
Ueberdosis is working on a really nice alternative backend for Yjs that scales using y-redis. You should follow their progress. https://next.tiptap.dev/guide/collaborative-editing#our-plug--play-collaboration-backend
Unfortunately, there is no plug-n-play indefinitely scalable backend for Yjs yet. But all the tools are there to build it yourself.
Cheers,
Kevin
The example ws-server available in y-websocket is a bit far off from something that can be made horizontally scalable. For example an update that is received on a socket connection somewhere over here https://github.com/yjs/y-websocket/blob/master/bin/utils.js#L169 will be the one that will be persisted to the database, because ydoc can be updated by redis pub-sub too.
I had to dig in the code y-protocols/sync and copy relevant parts till I had the update which would be applied to the y-doc, and then persisted that update to the db. This seems to be working fine.
Now comes the part of making the socket server horizontally scalable by applying patches from y-redis into my makeshift socket server connection
@dmonad @braden Can any of you help me with one specific thing - https://github.com/yjs/y-redis/blob/master/src/y-redis.js#L72 - why does y-redis use a queue to synchronize updates between processes? If I am able to isolate the update that a client has sent to a websocket server, I can just publish that update and all subscribers with the same docname can receive that update and apply that update to the doc they hold in memory, thus sending it forward to all the clients connected to them.
If a standard pubsub works, then I think I have a horizontally scalable implementation of a websocket-server for y-js with persistence on postgres using knex.
If pubsub has some issues, then I’d like to know them, and will implement a queue-system similar to y-redis on my server.
Either way, will like to open-source the horizontally scalable websocket-backend for yjs this weekend.
The messages are stored in a queue to retrieve the initial content after a server subscribed to the room. But you can also solve that differently (e.g. by asking other peers for the initial content directly).
1 Like
Will there be more than 16M if multiple subdocuments are nested?
Check out https://docs.mongodb.com/manual/reference/limits for more information on this.
Subdocuments are just references to other YDocs; nesting subdocuments shouldn’t majorly impact document sizes in your storage provider.
1 Like
I have a mostly working horizontally scalable websocket backend for my project, implemented in this manner - GitHub - kapv89/yjs-scalable-ws-backend … and I have also completed my collaborative diagramming tool which a friend is setting up on kubernetes and eks.
However, there is 1 big issue which was bought to light by this issue on my repo Sync awareness · Issue #3 · kapv89/yjs-scalable-ws-backend · GitHub … how do I deal with awareness when multiple users collaborating on the same doc can be connected to different websocket-servers?
If you can point me in the right direction as to how to horizontally scale the awareness side of things, that’ll be great.
1 Like
I use a similar awareness implementation to that in y-websocket, except every Awareness message is also sent to a Redis PubSub channel. The Awareness instances accordingly subscribe to that Redis PubSub, so that every instance in a cluster of servers receive the same awareness messages.
1 Like
@braden thanks for the reply. I have one question - what happens when there is 1 websocket server that 2 clients are connected to and then a 3rd client joins and it connects to a 2nd websocket server … do the awareness states of clients 1 & 2 get merged with awareness states of client 3 only when awareness information is exchanged between the two websocket servers?
Every user basically broadcasts their own awareness state. The servers need to make sure that all clients receive all individual states. It’s not really a big connected state (like the Yjs document). Each user simply shares their own state. Connected clients will represent the states in the Awareness object.
Hey @dmonad -
I have plans to rework y-websocket to not load the document to memory at all.
Id like to pick your brain on this comment. Could you explain how this could be implemented?
Hey,
I was also investigating possibilities how to scale effectively yjs websockets servers.
Here are few ideas:
- Using sticky sessions (Sticky sessions for your Application Load Balancer - Elastic Load Balancing). If we use documentId as a param for sticky sesions all users will be pointed to the same server. This will reduce number of messages which needs to be processed.
- Using redis pub-sub to synchronize messages across instances. This solution is implemented here Extensions – Tiptap . The downside of this approach is that all messages are distributed across all services - this is quite big limitation.
Does anyone try the approach with session stickiness?
I’d love to hear feedback on that.
I just released y-redis. It uses redis streams. Each server only listens to the streams that their clients subscribe to.