Why Am I Losing Data?

1

I have a YSupabase provider that sends updates to Supabase (PostgreSQL). This is the way it works:

  async pushUpdates(origin?: string) {
    const currentState = Y.encodeStateAsUpdate(this.doc);

    const update = this.currentSyncPoint
      ? Y.diffUpdate(currentState, this.currentSyncPoint)
      : currentState;

    const updatePayload = { update: Array.from(update) };

    const { error } = await this.supabase
      .from(this.config.table)
      .insert(updatePayload);

    if (!error) {
      this.updateSyncPoint();
    }
  }

  updateSyncPoint() {
    this._currentSyncPoint = Y.encodeStateVector(this.doc);
    this.persistSyncPoint();
  }

If I make a lot of subsequent updates like this:

ydoc.getMap('items').get('someId').set('someStringField', 'someValue');

When I apply all of the updates on a fresh client, all the fields of the someId YMap will be undefined.

If I give a long enough debounce to the updateSyncPoint function, I cannot reproduce the issue.

What is the mechanism that causes this?

I assume there’s a race condition in which an update with an older syncPoint is saved to the database after an update with a newer one. Is that it? If so, how is it different from updates coming from clients with different states?

Thank you

2

I’m afraid it’s not possible to tell just based on this information, without seeing the larger whole of your solution. However, one thing that caused my headache at first was that if the item “someId” is created in multiple different updates, only the latest one will apply.

So if you apply

Update 1 = ydoc.getMap(‘items’).set(‘someId’, new Y.Map()) and set someStringField = 1
Update 2 = ydoc.getMap(‘items’).set(‘someId’, new Y.Map())

The resulting “someId” nested map will be empty because Y.js will not merge the two conflicting values for “someId”.

If on the other hand, you create someId only once, this problem does not occur.

Also, I hope you have appropriate debouncing in place in the provider so that there will never be 2 concurrent pushUpdates calls running at the same time. Otherwise, due to unpredictable execution time of asynchronous I/O, your currentSyncPoint may end up in an inconsistent state.

3

Thanks for looking at this. I am quite confident that it’s not a matter of how the document gets updated as the local copy is sound and the only thing required to fix the issue is changing the logic of syncPoint.

I do think it’s caused by an inconsistent state, but I’m not quite sure about what that state looks like and why it causes this issue. This is important for me to understand because a slow enough network can cause concurrent pushUpdates even with a reasonably long debounce

4

You might actually want to change the logic a little to ensure that the your currentSyncPoint represents that point that was actually stored, not the one that was present after storing (changes may have occurred between!). Something like this…

async pushUpdates(origin?: string) {
  const currentState = Y.encodeStateAsUpdate(this.doc);
  
  const update = this.currentSyncPoint
    ? Y.diffUpdate(currentState, this.currentSyncPoint)
    : currentState;

  const newSyncPoint = Y.encodeStateVector(this.doc);
  
  const updatePayload = { update: Array.from(update) };
  
  const { error } = await this.supabase
    .from(this.config.table)
    .insert(updatePayload);
  
  if (!error) {
    this.updateSyncPoint(newSyncPoint);
  }
}
  
updateSyncPoint(newSyncPoint) {
  this._currentSyncPoint = newSyncPoint;
  this.persistSyncPoint();
}
5

Good catch! Thank you.