Unifying Awareness & Y.Doc

Has anyone given any thought to unifying the Awareness and Y.Doc events & types?

One of the things I’m trying to do in Relm is create game objects that can be either awareness-based (no change history) OR document-based (with change history). Abstracting this away is proving slightly difficult.

For example:

  • if a Player’s state (name, rotation, position, etc.) is communicated as an Awareness state, the Awareness events send everything (not just changed state, such as position). This is because Awareness uses setLocalStateField, which accepts a key and a value.

  • if a Player’s state is communicated as Document state, it can be more granular about which updates it listens for–for example, if position is a Y.Map inside a parent Y.Map, I need only listen to the child Y.Map to get position updates.

There are differences in event signatures, too:

  • awareness.on(‘change’) gets an object with { added, updated, removed }

  • Y.Array observe gets an event object with .changes.added and .changes.removed

Just curious if anyone has tried to unify any of this in the past. If not, I will try to continue to noodle on ways to abstract the differences away.

So here’s another idea: can a (separate) Y.Doc be used for awareness? This would “unify” my two APIs (because it would be a single API), but it seems I would run into difficulty because there is no way to limit the history.

That makes me wonder: Is there a way to limit the history? It would be wonderful if I could say “only keep the last 10 modifications” on any given Y.Map. Has this been attempted?

The alternative I’m considering is resetting the Y.Doc that keeps track of awareness every once in a while.

I’m sure you already understand the reason why there are separate implementations for awareness and for persisted state data.

The Yjs document needs to be persisted, and it needs to be able to resolve sync conflicts. Hence the log of all operations are kept (even with ydoc.gc = true all operations are kept in a minified form). Awareness information is not persisted and can be shared without having sync conflicts (every client can only update its own data field). Therefore, we don’t need to keep the operation log.

That makes me wonder: Is there a way to limit the history? It would be wonderful if I could say “only keep the last 10 modifications” on any given Y.Map. Has this been attempted?

The log of all operations (even in a minified form) is kept to resolve conflicts. So in Yjs, you definitely need to keep all operations.

if a Player’s state is communicated as Document state, it can be more granular about which updates it listens for–for example, if position is a Y.Map inside a parent Y.Map, I need only listen to the child Y.Map to get position updates.

It would be nice to have Yjs types for awareness data. One approach would be that every client shares it’s own Yjs document until it is not available anymore. But that would require to implement an additional protocol. Furthermore, the size of the Yjs document will increase over time until the client goes offline.

Usually, clients only sync very little awareness information. With ipv6 it really doesn’t make a difference if you send 20 bytes of data or if you send 65kb of data (the size of an ipv6 package is ~65kb). So it is usually fine to share the whole awareness state. The advantage is a very easy protocol that doesn’t grow in size over time, and that doesn’t rely on two-way-handshakes (asking another client for the current state and sending the differences).

if a Player’s state is communicated as Document state, it can be more granular about which updates it listens for–for example, if position is a Y.Map inside a parent Y.Map, I need only listen to the child Y.Map to get position updates.

It probably makes sense for some applications to be more granular on what they update. If updating, for example, the position is really expensive, it makes sense to do that only when absolutely necessary. I suggest that you store the previous value somewhere, and then check beforehand if there is a difference before you update the position. I usually update all information from that client if anything changes (e.g. in the editor bindings).

The Awareness state is influenced by the idea of having immutable state that you can compare. I feel more comfortable using immutable state whenever possible, instead of mutable state like Y.Array. Although mutable state makes more sense when working with large datasets / long arrays.