Split ydoc update

1

Hey,
I am the developer of the y-mongodb-provider, which is mostly a fork of y-leveldb, and I am currently facing a very special problem. I have a use case where it can happen that I want to write a single update that is bigger than 16MB, which is the maximum allowed document size for MongoDB. In my opinion, there are two ways to reduce this size:

  1. I can call my MongoDB provider multiple times with smaller updates.
  2. I can detect these big updates in my provider and split the update there. I would definitely prefer this one; however, I don’t actually know how to split an update.

Lets get you some code context.
This is how I call the provider to write an update:

const persistedYdoc = await mdb.getYDoc(docName);
const persistedStateVector = Y.encodeStateVector(persistedYdoc);
const diff = Y.encodeStateAsUpdate(updatedDoc, persistedStateVector);
await mdb.storeUpdate(docName, diff);

This is the important part of what the provider does in storeUpdate:

await db.put(createDocumentUpdateKey(docName, clock + 1), { value:  buffer.Buffer.from(update) });

Does anyone know how to split document updates?

Thanks for the help!

2

I just found this post. Presumably, his recommendation has not changed. But since it has already been a year, I have not deleted this post, still holding onto the small hope that maybe it did.

But presumably, I will split MongoDB documents into multiple frames to solve this issue.

1 Like