Memory issue with yjs

1

I am using ReactFlow to build an interactive diagram with yjs and y-Websocket. One current issue is the Yjs doc memory that the server needs to load.

This is the log of process.memoryUsage() on a Node.js server. I have converted it to MB for readability.

Before I load a big document

{
  rss: '84.45 MB',
  heapTotal: '19.45 MB',
  heapUsed: '17.89 MB',
  external: '38.66 MB',
  arrayBuffers: '36.48 MB'
}

After I load the document

{
  rss: '190.22 MB',
  heapTotal: '109.45 MB',
  heapUsed: '95.40 MB',
  external: '41.32 MB',
  arrayBuffers: '39.14 MB'
}

The size increased significantly, but I’m unsure how to horizontally scale y-websocket. It crashes the server.

Can you help me reduce its memory usage?

There are several solutions that I am thinking of

  • Using YKeyValue from y-utility. It will reduce the document size that we load a lot

  • Using y-redis on top of y-websocket

I would greatly appreciate any suggestions or ideas you may have.

Thank you!

2

It is running wild if I keep refreshing the page. It is just for two websocket connections and two yjs documents

{
  rss: '1424.52 MB',
  heapTotal: '1292.91 MB',
  heapUsed: '874.64 MB',
  external: '62.97 MB',
  arrayBuffers: '60.79 MB'
}

Why and how it could happens if the document store in MongoDB is only 15.7 MB

3

Hi @toantd90 ,

I’m not exactly sure how to interpret your heap sizes.

Nodejs allocates much more memory from the system than it needs. That’s a chrome-inspired feature. I’m also not a big fan of it.

Loading a huge document in Yjs will consume a lot of memory. You can optimize memory usage by reducing the number of writes you perform on a Yjs document. Try to make as few changes as possible. I’ve seen in the past that users make Yjs modifications for every single mouse movement, which will blow up the size of a Yjs document over time.

In y-websocket we keep the Yjs document in-memory until all clients disconnected. If you didn’t specify persistence, then the document will be kept in-memory forever.

y-redis is an alternative backend to y-websocket that doesn’t keep documents in-memory. However, it’s new and slightly harder to setup.

There are also other providers like y-sweet or hocuspoces that give you scaling options. With liveblocks as a backend for Yjs, you don’t have to think about memory usage on the server.

1 Like
4

Hi @dmonad Thank you for your reply.

You can optimize memory usage by reducing the number of writes you perform on a Yjs document. Try to make as few changes as possible. I’ve seen in the past that users make Yjs modifications for every single mouse movement, which will blow up the size of a Yjs document over time.

I believe this would be a good solution. Based on what I’ve observed in my application, yMap tracks numerous changes when a node is just moved from its current position to another position. ( This is because I am creating an interactive diagram with nodes ). When building a collaborative diagram application, it’s necessary for other users to see nodes move when I move them.

In y-websocket we keep the Yjs document in-memory until all clients disconnected. If you didn’t specify persistence , then the document will be kept in-memory forever.

Yes, I do specify persistence and have a similar code to unload the doc y-websocket/bin at master · yjs/y-websocket · GitHub