I’m looking for an advice in troubleshooting data loss in yjs documents. We are using tiptap editor with hocuspocus on server side, hocuspocus and y-indexeddb providers on client. On server side yjs document is persisted into s3 compatible storage.
With this setup we are experiencing rare issues when document content is reverted to some old state when a client connects. I thought that the issue might happen when document sync happens before content is loaded from storage, but this happens even for documents that are already in memory.
Never has happened to me. I suppose you might have some logic error.
Perhaps the version in IndexedDB is applied as a new update which then is—as it should—the latest version? Or something like that. I have been planning to use IndexedDB myself as well as the immediate rendered version but there can be subtle bugs where doc is not loaded properly
We use it exactly for the same purpose - as an immediate rendered version while the version from the server is loading.
Perhaps the version in IndexedDB is applied as a new update which then is—as it should—the latest version?
I suppose this may happen if remote provider synced before the IndexedDB version has been loaded? I thought about this scenario, but decided that yjs will manage to figure out order of updates.
Are there any known issues with y-indexeddb? I looked through issues in repository and posts here, but did not notice anything critical.
Haven’t used y-indexeddb myself and I’ll probably write a custom IndexedDB adapter. One common bug that causes dramatic issues is when you render a blank document first, without waiting for yjs to sync. This blank doc then replaces the original server version, as it then has the latest timestamp.
But, uhh. In this case, I’m not sure what could be the cause. You can try adding artificial delay to y-indexeddb loading the doc as well as to the server. And also try inputting some changes while the server version loads.
You definitely should be able see what version is loaded and whether the server version is used or not.
EDIT: Oh yeah, now that I recall, my reasoning for writing a custom IndexedDB adapter was that I thought the way y-indexeddb stores the updates is too cute–they are stored as individual binary updates instead of single blob. I don’t really think the overhead of constantly writing the whole doc is that big of a problem.
I have a relatively similar setup and have experienced data loss cases several times.
One source of issues is the fact that indexed db is not reliable. Sometimes the connection will be closed (for whatever reason, different browsers have different behaviors) and you have to manually handle such cases and reopen the connection. I ended up writing my own storage provider to better handle errors.
There’s also a lot of opportunity for failure on the server side as well. In general, server failure is complemented by the client storage redundancy, but since the client storage is not reliable, a problem on the server could lead to data loss.
Problems I have experienced in the hocuspocus server:
out of order persistence using the hocuspocus redis locks strategy (I highly discourage using it and we decided to move away from it)
temporary failure to save the document and then the client disconnects, in which case there’s no attempt to save again
I’m using Tiptap and Hocuspocus. y-indexeddb in the browser and SQLite on the server. In the Browser I check which providers are being used and then wait for all of them to sync before I render any content.
This process is somewhat convoluted, however I’ve never seen any data sync/loss issues. Prior to doing this I would see strange issues.