How is `encodeStateAsUpdate` output meant to be used remotely?

I’d like to do the following:

  • Load a document
  • Encode the document as a single update
  • Send the update over http to a client, when requested

What’s not working is loading the update at the other end. I originally thought this might be the browser, fetch API or http causing the trouble but after writing a simple unit test (see below) I realise it’s unclear how to encode it for transport.

I’ve tried the following:

  • Using to/from Uint8 function to convert to base64 before sending (when simply recieving the data as a ArrayBuffer didn’t work)
  • Encoding according to the below example code
  • Only decoding (no encoding) on the assumption the encodeStateAsUpdate already does some encoding
  • Using the example sync protocol from the ws server code on github
  • Combinations of all 1, 2 and 3

I get either of the following everytime:

  • TypeError: contentRefs[(info & binary__namespace.BITS5)] is not a function.
  • Error: Integer out of range.
  • Works without exception (in the case of base64 conversion) but applyUpdate does not work for nested maps on the document I’m trying to send.

So, what I’m wondering is - how am I meant to use encodeStateAsUpdate when I would like to apply the update remotely? The examples mention, applying it remotely but there doesn’t appear to be any actual examples of how to achieve this over websockets or http (apologies if I’ve missed it).

Can anyone tell me where I’ve gone wrong here?

Here’s some example code of how I was testing encoding/decoding before trying over a network (it did not work):

import * as Y from 'yjs';
import fetch from 'node-fetch';
import * as decoding from 'lib0/decoding';
import { encoding } from 'lib0';
import { writeUpdate } from 'y-protocols/sync';

const encodeDocumentState = (documentToSend: Y.Doc) => {
  const update = Y.encodeStateAsUpdate(documentToSend);

  const encoder = encoding.createEncoder();
  writeUpdate(encoder, update);
  return encoding.toUint8Array(encoder);
};

const decodeDocumentState = (response: ArrayBuffer) => {
  const decoder = decoding.createDecoder(new Uint8Array(response));
  return decoding.readVarUint8Array(decoder);
};

// build doc one
const doc = new Y.Doc();
const board = doc.getMap('board');
board.set('foo', 'bar');

const encodedState = encodeDocumentState(doc);
const decodeState = decodeDocumentState(encodedState);
const doc2 = new Y.Doc();

Y.applyUpdate(doc2, decodeState); // this breaks. If I remove the encoding/decoding it doesn't. It doesn't work when I send data over http, regardless of whether I encode it or not.

console.log('applied doc output:', doc2.toJSON());

First Y.encodeStateAsUpdate(doc) already contains the update encoded in a UInt8Array you just need to encode it in base64 in NodeJS Buffer.from(update).toString('base64')
The encoder/decoder you are using take any data and encode it into UInt8Array so you doing it twice.

Same for Y.applyUpdate(doc, update) it takes the UInt8Array generated by the previous one and if you want to parse properly base64 in browser you need to look at Document Updates - Yjs Docs

Then for getting the minimum update I would recommend to do a snapshot of the doc after filled it of the data to just send back the delta.

const prevSnapshot = Y.snapshot(doc)
const prevStateVector = Y.encodeStateVector(prevSnapshot.sv)
const update = Y.encodeStateAsUpdate(doc, prevStateVector)

I also advise you to look at GitHub - yjs/y-leveldb: LevelDB database adapter for Yjs for the easy storage solution and then use any levelup binding you need.

Hope this helps

2 Likes

How can i create Buffer in React(frontend)?

npm install buffer
const Buffer = require('buffer/').Buffer
1 Like

Why would you use Buffer? is it the question how to have Uint8Array <> Base64? Then I advise using the native version of either browser or nodejs and here following a small helper.

const error = () => {
  throw new Error('Base64 operation not supported, neither window nor Buffer is available');
};

export const toBase64 = (arr: Uint8Array): string => {
  if (typeof window !== 'undefined') {
    return window.btoa(String.fromCharCode.apply(null, arr as any));
  }
  if (typeof Buffer !== 'undefined') {
    return Buffer.from(arr).toString('base64');
  }
  return error();
};

export const fromBase64 = (str: string): Uint8Array => {
  if (typeof window !== 'undefined') {
    return new Uint8Array(Array.from(window.atob(str)).map((char) => char.charCodeAt(0)));
  }
  if (typeof Buffer !== 'undefined') {
    return new Uint8Array(Buffer.from(str, 'base64'));
  }
  return error();
};
1 Like

I prefer to use UInt8Array directly instead because it’s js standard.