pendingStructs accumulate and the document is broken

1

Hello.
I am creating a whiteboard tool using a combination of tldraw and yjs.
I am using postgres for persistence on the server, but on rare occasions I am unable to restore the document. (It worked as expected during client synchronization, but once everyone disconnected and tried to connect again, it only restored halfway through.)

Upon closer inspection, I’m beginning to think that the contents of the pendingStructs are bloated and that data may be missing.

Is it possible to forcefully compensate for the corrupted data?
Also, is the possibility of corruption more likely due to an insert failure?
Is there anything that could go wrong if the data is too large?

Thank you.

2

You probably need to identify steps to reproduce before anything useful can be said about it. Unfortunately that’s easier said than done if it is intermittent.

Any logging you can add to help get more clues when it does go wrong could be helpful.

1 Like
3

Thank you for your reply.

I have been looking for a procedure to reproduce this for about 2 weeks now and cannot reproduce it.
There are just two huge broken histories.

I am at a loss, so I am asking this question.
I am sorry I have no clue what to do.

4

What provider do you use to communicate updates to the server? y-websocket?

Do you save the state vector on the server after updates are successfully persisted? Maybe the state vector is being saved without a full confirmation that the update is saved.

5

This is the expected behavior of yjs. You must lose one update when syncing the server and client data. This will cause ALL following updates to be ignored and put into pendingStructs.

I made utils to inspect if an update will cause pending struct a few months ago.

function willMissingUpdateImpl (
  doc: Doc,
  update: Uint8Array,
  decode: typeof decodeUpdateV2 | typeof decodeUpdate
): false | Map<number, number> {
  const { structs } = decode(update)
  // clientId -> clock
  const missingMap = new Map<number, number>()

  // find if missing update in the structs
  for (let i = 0; i < structs.length; i++) {
    const struct = structs[i]
    const client = struct.id.client
    const items = doc.store.clients.get(client) ?? []
    const lastItem = items.at(-1)
    if (!lastItem) {
      if (struct.id.clock !== 0) {
        missingMap.set(client, struct.id.clock)
      }
      continue
    }
    const nextClock = lastItem.id.clock + lastItem.length
    if (nextClock < struct.id.clock) {
      missingMap.set(client, struct.id.clock)
    }
  }
  return missingMap.size > 0 ? missingMap : false
}

export function willMissingUpdate (
  doc: Doc, update: Uint8Array): false | Map<number, number> {
  return willMissingUpdateImpl(doc, update, decodeUpdate)
}

export function willMissingUpdateV2 (
  doc: Doc, update: Uint8Array): false | Map<number, number> {
  return willMissingUpdateImpl(doc, update, decodeUpdateV2)
}
2 Likes