How would you model a complex diagram page?

#1

Hello together,

I have created a wireframe solution: https://mydraft.cc. It is my playground project where I test new stuff. Today I discovered yjs because Github suggested my tldraw (GitHub - tldraw/tldraw: a very good whiteboard) and I would like to give it a dry for learning purposes.

For my understanding I use a good shared type and then I subscribe to changes and update my local model. I am trying to figure out what a good type could be for my application. Right now the structure is as followed:

// Simplified
root: (UndoRedo)
   current: (Editor)
      page: (Document)
         shape: (Shape)

So it is very nested. For undo / redo I leverage the immutability data structure, so my actions do not need to implement an undo operation. When I store a a document to the server I use the snapshot of the event a few changes ago and the last 20 applied actions (to make undo possible after loading a document). So it is very similar to snapshotting in event sourcing. I was hoping to implement yjs on top of my current structure to leverage immutability and redux features with the redux developer tools and everything that comes with that.

For my understanding the tldraw model is basically a simple map of records: (see TLRecord.ts)

So it maps naturally to Y.Map. But I have no idea how to map a deep data yjs. I would also like to implement this on top of my existing solution and not integrate yjs to deep into the system.

I have 2 ideas:

1. State as sequence of actions

I could store the state as sequence of actions and use that for synchronization. Not all actions are idempotent, through. For example undo is not and need to be refactored to something like switchTo(snapshotId) internally.

Furthermore the data structure would be very big because it would contain all changes from the past because I am not sure if we can implement something like snapshots.

2. Represent everything as map

This is basically the same idea as tlsdraw. I could represent my document as simple map. Some of the keys would probably be fixed like cursor or the current document, some others not. We could store documents, shapes, user settings and everything into this map and use Ids to reference other items from the list.

Advantages

  • This would then map easily to Y.Map.
  • The undo / redo operation can also be modelled, by calculating all changes records and transfer them. But for some operation this can be very big, e.g. when a big document has been deleted.

Disadvantages

  • Because the actual structure is not flat, so we probably need complex resolvers to create some hierarchical structures again if they are needed later.
  • Some operations are much more complex. For example: When you delete an document, you have to loop over all the shapes and also deleted them. Then you synchronize all changes.
  • We can only edit whole objects. This could loose in data loss, because when multiple users edit user properties at the same time, this change would be lost.

Perhaps someone can guide a me a little bit with his or her experience.

3. Integrate yjs deeply into the system and build an object graph

According to the documentation, yjs primitives can be nested and therefore it should be possible to refactor the current object graph based on yjs. I am not sure if I understand how this would work with redux because the object graph would be mutable and many features in react and redux are based on immutability.

#2

Hi, welcome.

You can nest Y.Maps. However, descendants can get overwritten if two different clients create the nested Map at the same time at the same key.

e.g.

Client 1:

const current = doc.getMap()
const page = new Y.Map()
const shape1 = new Y.Map()
current.set('home', page)
page.set(shape1)

Client 2:

const current = doc.getMap()
const page = new Y.Map()
const shape2 = new Y.Map()
current.set('home', page)
page.set(shape2)

In this case, two clients add a shape to the home page, but instead of ending up with a page with two shapes, only one will survive (because one new Y.Map will overwrite the other). This can occur even if you wait for the provider to sync, as it is always possible that another client is offline and the update comes later.

The point is, nested Maps with non-unique keys are susceptible to conflicts, so use guid keys if possible. Or prefix keys like page-41b1, page-f7ca, page-2caa, etc.

The simplest and most efficient approach is to use the shared types directly, without any intermediary models. However, for existing applications that might be too much work to change the infrastructure. I had a Redux application that added YJS to. I added middleware to push changes to the shared types, and I subscribe to changes where I dispatch an action to update the Redux state. It works great.

YJS already stores the entire history of updates. Append-only logs don’t make a lot of sense in YJS for this reason. So you might as well store the state in a structured way.

This should work pretty well, with the aforementioned caveat about nested Y.Maps.

That shouldn’t be a problem. Just do it all in a single transaction for atomicity.

Yeah, using the middleware + subscription approach will be better for you. Just treat YJS as the source of truth.

#3

This sounds easy for a simple data model, but how do you do that for a complex model? Do you always make a full sync of everything? Sounds very expensive.

#4

I capture changes before they are applied to the Redux state. I push them into a state.pushQueue array which gets cleared after sync. Then the middleware can push just the changed data to the YJS model instead of the entire state.

#5

So for each action you need:

  1. A reducer method (case)
  2. Some code to create the diff
  3. Some code to convert the diff back to redux state

Sounds like a lot of code.

#6

It depends how complex/heterogeneous the state is that you want to persist to YJS. In my use case, I just need to persist a tree which is pretty uniform. I create a list of node updates in the reducer and push them to a queue which the middleware picks up and pushes to YJS. That way I don’t have to do a full diff on the state. (Though that is entirely possible; I take that approach with a custom undo/redo manager.)

const updates = { 
  [node1]: { ... }, 
  [node2]: null, 
  [node3]: null, 
  ... 
}

It’s an unfortunate drawback that YJS does not easily plug in to existing applications with well-established model infrastructure. That is true of any CRDT though. In order to handle merges without conflict, they typically work with raw updates rather than snapshots. Perhaps that is something that could be abstracted in the future though.

#7

Is your code public? I would like to get an understanding of the complexity. Thb it sounds easier to rebuild my models using the yjs types

#8

Yes, here are the relevant files:

1 Like
#9

I have created a demo application to test synchronization: GitHub - SebastianStehle/yjs-test

It seems to work fine. If you are interested your feedback is welcome. Perhaps I am creating a library from that and improve that further. What is left are tests and I am not sure yet, how to write them.

1 Like
#10

In case you want to have a look: I fixed the folder structure. So everything that is the actually “lib” (it is not a lib yet) would be in this folder: https://github.com/SebastianStehle/yjs-test/tree/main/src/sync

I also added tests to solve a lot of small issues.

the other stuff is a trello like test project and some classes that I use in my main project: https://github.com/SebastianStehle/yjs-test/tree/main/src/immutability

I formulated some requirements in the Readme.md:

  1. Make only the necessary changes.
  2. Keep redux instances intact if nothing has been changed.
  3. Support custom domain object classes.
  4. Support custom collections.
  5. Support custom value types (e.g. Color, Vector and so on).

These are important for me and the existing redux binding does not support them. It just replaces the whole state:

#11

Nice. That seems like a good basis for a yjs-redux integration library.