Awareness update causes N^2 messages, which leads CPU issue on server

a3626a · March 10, 2024, 4:18am

Issue

When an awareness update is sent to server, the server broadcasts to all connected users (N).
For each users, after they recieve new updates, they send awareness updates to the server again,
which make the server broadcast again (N x N)

In general, the websocket server seems like being loaded as O(N^3), because awareness updates usually occurs more linearly as the number of user grows.

This makes high cpu load on the server with medium sized rooms. (20 users in the same room)

Background

I recently deployed a Yrb-actioncable server and Yjs based client system. This service is targeted to school, so one collaborative room is about 20 users.

I have found that the actioncable (websocket) server has unexpected high cpu load, just for a few classes (3~5)

I checked the log, It seems like same message is repeated N times

Each “receive” log makes a single broadcasting, so it is actually doing N*N single message transmissions.

I think this is not a optimal behavior, and started to investigate this issue.
(Still I am not sure that this repeated message transmissions are the main cause of cpu load.)

Details

I actually deployed Yrb-actioncable, but it is just a different version of Yrb-websocket. So I will cover only y-websocket here.

When an awareness update is occured by setLocalState, it causes ‘update’ event of the Awareness object.

github.com

yjs/y-protocols/blob/master/awareness.js#L134


      
              }
            } else {
              updated.push(clientID)
              if (!f.equalityDeep(prevState, state)) {
                filteredUpdated.push(clientID)
              }
            }
            if (added.length > 0 || filteredUpdated.length > 0 || removed.length > 0) {
              this.emit('change', [{ added, updated: filteredUpdated, removed }, 'local'])
            }
            this.emit('update', [{ added, updated, removed }, 'local'])
          }
          
          
/**
           * @param {string} field
           * @param {any} value
           */
          setLocalStateField (field, value) {
            const state = this.getLocalState()
            if (state !== null) {
              this.setLocalState({

Websocket server listens this ‘update’ event, and send a awareness update message to the websocket server.

github.com

yjs/y-websocket/blob/1638374702d694e5b58ea89d291fdf9d41faa09a/src/y-websocket.js#L364


      
          this._exitHandler = () => {
            awarenessProtocol.removeAwarenessStates(
              this.awareness,
              [doc.clientID],
              'app closed'
            )
          }
          if (env.isNode && typeof process !== 'undefined') {
            process.on('exit', this._exitHandler)
          }
          awareness.on('update', this._awarenessUpdateHandler)
          this._checkInterval = /** @type {any} */ (setInterval(() => {
            if (
              this.wsconnected &&
              messageReconnectTimeout <
                time.getUnixTime() - this.wsLastMessageReceived
            ) {
              // no message received in a long time - not even your own awareness
              // updates (which are updated every 15 seconds)
              /** @type {WebSocket} */ (this.ws).close()
            }

github.com

yjs/y-websocket/blob/1638374702d694e5b58ea89d291fdf9d41faa09a/src/y-websocket.js#L344-L353


      
          this._awarenessUpdateHandler = ({ added, updated, removed }, _origin) => {
            const changedClients = added.concat(updated).concat(removed)
            const encoder = encoding.createEncoder()
            encoding.writeVarUint(encoder, messageAwareness)
            encoding.writeVarUint8Array(
              encoder,
              awarenessProtocol.encodeAwarenessUpdate(awareness, changedClients)
            )
            broadcastMessage(this, encoding.toUint8Array(encoder))
          }

The server recieves this message, and broadcasts to all connected users (in the same room)

Each clients recieves the message from the server.

github.com

yjs/y-websocket/blob/1638374702d694e5b58ea89d291fdf9d41faa09a/src/y-websocket.js#L139-L145


      
          websocket.onmessage = (event) => {
            provider.wsLastMessageReceived = time.getUnixTime()
            const encoder = readMessage(provider, new Uint8Array(event.data), true)
            if (encoding.length(encoder) > 1) {
              websocket.send(encoding.toUint8Array(encoder))
            }
          }

And applies update to its Awareness object.

github.com

yjs/y-websocket/blob/1638374702d694e5b58ea89d291fdf9d41faa09a/src/y-websocket.js#L70-L82


      
          messageHandlers[messageAwareness] = (
            _encoder,
            decoder,
            provider,
            _emitSynced,
            _messageType
          ) => {
            awarenessProtocol.applyAwarenessUpdate(
              provider.awareness,
              decoding.readVarUint8Array(decoder),
              provider
            )
          }

And this causes update event again.

github.com

yjs/y-protocols/blob/master/awareness.js#L291


      
                  updated.push(clientID)
                }
              }
            }
            if (added.length > 0 || filteredUpdated.length > 0 || removed.length > 0) {
              awareness.emit('change', [{
                added, updated: filteredUpdated, removed
              }, origin])
            }
            if (added.length > 0 || updated.length > 0 || removed.length > 0) {
              awareness.emit('update', [{
                added, updated, removed
              }, origin])
            }
          }

Now each clients sends a new awareness update message to the server, overall N awareness updates, and these mesages are broadcasted again.

And this time, there’re no changes, so it is finished here.

Solution?

I think awareness update messages shoud not send when the awareness object is updated by external message.