I have a document that is persisted using a Y.encodeStateAsUpdate and restored using a Y.applyUpdate (also tried with V2).
The document has a map that receives key updates every second or so and it’s rapidly becoming a 6mb document and gc isn’t making it get any smaller.
I’m trying to figure out a way to throw out the old state and save just the current state without the 5k known clientids and associated state vectors.
const stateVector1 = Y.encodeStateVector(document) doesn’t seem to let me make a new document.
Alternatively, I think I can’t set a clientid, right?
Any help is appreciated, I just want to update keys on a map every second and have the doc size remain reasonable.
I myself don’t know any levers to pull to boost perf if you have such hotly edited doc. 5k known clientIds sounds like you could use mapping of userIds to clientIds whenever you open the doc. If I recall the issue then can become if users are online from multiple machines so the clientIds start to conflict.
Regardless, to fix it now sounds like it’s going to be pain. One not so great solution would be to determine absolute snapshot points where you create an empty doc and apply the current state. Then, every client connecting would have to discard their docs and restart from the snapshot point. You could check also whether they had edits after the snapshot by comparing the last edit timestamps but you’d need some mechanism to extract the changes and reapply them (basically some JSON diff and apply it to the map).
Or, alternatively, redirect the old doc to new doc with the freshly initialized state and allow users to transfer their changes manually.
As you said you are using a map, you could also try just creating a last-write wins protocol outside yjs where every last change to a key would replace the value. If you like tinkering.
I found this other discussion here: Clear document history and reject old updates - #10 by ViktorQvarfordt and it sounds like Map is pretty expensive for random writes and it looks like YKeyValue would be more efficient, but also might be difficult for us to transition to.
I did end up doing a last write wins algo and not using YJS for this particular section of data and that’s working well. Otherwise, my doc was growing like 30MB per day (without even constant writing)
1 Like