Stateless server broadcasting implementation (in Go)

1

First of all, I’m very excited to work with Y.js. Great project.

Some context: our ‘PubSub’ backend is written in Go. Frontend React/JS. Clients connect to the server using WebSockets. This is also handling the authentication. However, we would like to start using Y.js CRDTs in our app without porting all of Y.js to Go (for now). So the server would not be able to load the Y.doc.
It looks like we have these options:

  1. Run Y.js in a new node instance for operating on Y.doc structures, persistence etc. Make our backend communicate with this.
  2. Y.js only on clients, the server would only do broadcasting of messages to clients. Almost like P2P but with a central authority? Server does not need to understand Y.js protocol, and would only handle broadcasting of messages.
  3. Like 2) but we could we persist all update messages (e.g. in Redis)? I’m sure this has downsides (no compression!), but it might work?

I would love to get started with option 2), because it means we can keep our current stack and don’t need to operate a new server. I’ve looked at y-websocket and prepared a new version of it that would connect with our existing Go backend.

A few questions remaining: what would the broadcasting logic look like to make option 2 work? How would we initialize the document state if the server cannot load a Y.doc in memory? For new clients, should we request the document from other clients by “signalling” them with a special message, or do we store each update (as data blobs) in the server and “replay” each of the update events? Trying to wrap my head around the limitations of these approaches.

2

Hi @sander-m,

These are all valid options. You need Yjs only for computing the differences and merging updates together. So you could start a nodejs instance to help you to do that or find language bindings to the C# port of Yjs.

Once you have a method to merge updates and compute the differences (see the Yjs documentation on updates) you can implement your custom persistence layer and also sync with other clients.

Coincidentally I just implemented approach 2) in python: https://github.com/QuantStack/jupyterlab/pull/1/files#diff-9d5bdfaf022b365d5ddfc99193c11d8f96dace8a6f53da64d19cb27b1769d863

You are right, you simply need an “echoing” server, but you also want to make sure that you receive the initial state from the connected clients (note how I manually send SyncStep1). The python implementation uses y-websocket on the client, so it will be easy to switch to a fully-fledged python-yrs server at some point.

1 Like
3

Thanks! I think I will start with an “echoing” server for our first prototype. The Python example will be useful for me. Like you said, it should be relatively easy to switch to a more capable server at some point.

4

@sander-m We’re you able to get Yjs to work with golang?