How to implement a custom Yjs provider?

1

Hello,

I have an application featuring a rich content editor based on TipTap. Currently, the application supports real-time updates for a single editor, using websockets for data synchronization. Now, I’m interested in extending this to allow multi-user editing. For this, I’ve started integrating @tiptap/extension-collaboration along with Yjs.

Objective:

I wish to create a custom provider to plug into my existing websocket service for Yjs. Specifically, the provider must adhere to a custom save format that my application uses.

Issue:

I have initiated the development of a custom Yjs provider, but it is incomplete and not functional. I need guidance on the required logic and structure.

Code Sample:

Here is a skeleton of the custom provider I started building:

import * as Y from 'yjs';
import { Observable } from 'lib0/observable';

export class MyCustomProvider extends Observable {
  constructor(doc, updateHandler) {
    super();

    this.doc = doc;
    this._synced = false;
    this._updateCallback = null;

    this._updateHandler = (update, origin) => {
      if (origin !== this) {
        if (updateHandler) {
          // This is working, I get the updates every time the doc is updated
          updateHandler(update, origin);
        }
      }
    }
    this.doc.on('update', this._updateHandler);
  }

  // This is called by the editor to apply updates
  applyUpdates(updates) {
    this.doc.transact(() => {
      updates.forEach(update => 
        // This is not working, the updates are not applied to the doc
        Y.applyUpdate(this.doc, update)
      )
    });
  }

  destroy() {
    this.doc.off('update', this._updateHandler);
    super.destroy();
  }
}

I instantiate and use it like so:

import { Editor } from '@tiptap/core';
import Collaboration from '@tiptap/extension-collaboration';
import * as Y from 'yjs';

const doc = new Y.Doc();
const documentProvider = new NapkinCustomProvider(doc, onUpdate);

const editor = new Editor({
  extensions: [
    Collaboration.configure({
      document: documentProvider.doc,
    }),
  ],
});

function onUpdate(update, origin) {
  // Send the update through the network
}

// Called when the second user receives updates from the network
function onReceiveUpdates(updates) {
  // Apply the update to the document
  this.documentProvider.applyUpdates(updates);
}

Problems:

  1. Although the onUpdate function receives updates, applying these updates on the second user’s document using Uint8Array does not yield any changes.

  2. There is a gap in my understanding of how to correctly structure a custom provider for Yjs to use with TipTap.

Questions:

  1. Is there a guide, documentation, or example detailing how to correctly structure a custom Yjs provider?

  2. Are there specific steps to ensure that Uint8Array updates received through websockets are correctly applied to the Yjs document?

I looked at the documentation and didn’t find anything.

I look forward to your assistance on these issues.

Thank you!

2

Hi @erwan, welcome.

Could you say a little more about this? Are you creating a brand-new encoding protocol or update format, or are you using the default lib0 protocol?

What is the custom save format, and why? That would help determine how deep the change is and what infrastructure changes would be required.

If you haven’t seen it already, I recommend this thread that offers some solid clues:

A custom provider needs to handle two main cases:

  1. Initial sync - Load from DB/network + sync with local Doc.
  2. Ongoing sync - Save Doc changes back to the DB/network. Also, changes from the network need to be applied to the local Doc.

A network provider is much more complicated than a storage provider since it needs to manage connections and client-server messaging. y-websocket is less than 500 lines but it is not trivial by any means.

I would recommend using an existing network provider if at all possible. (On the other hand, maybe you know what you’re doing and you just need to figure out how to apply that damn update! :sweat_smile:)

3

Hi @raine, thank you for your reply and your pointers!

Could you say a little more about this? Are you creating a brand-new encoding protocol or update format, or are you using the default lib0 protocol?

As we have a graph layer and the TipTap editor working together, we save the content of each block associated with graph layer ids for example (everything is saved as JSON).

Unfortunately, I cannot use an existing provider as it would be very hard to integrate it with my existing application (we are using websockets for a lot of things and we have our own messages mechanisms and backend).

I’ll look deeper at the cases that you mentioned, I believe I’m missing the initial sync, which causes issues later on.

1 Like
4

Update: I solved most of my issues thanks to your points @raine!

I described how I fixed my issue in this post.

1 Like