Posting encodeStateAsUpdate to server

#1

Hey,

I have a small problem (probably related to encoding) trying to save the doc to database by posting it from front-end (Quill). I use Y.encodeStateAsUpdate(doc) and then POST the result to DB.

For some reason it creates an object ({“0”:1,“1”:1,“2”:159,“3”:245,"…) which does not work when reading data back from DB (TypeError: contentRefs[(info & binary.BITS5)] is not a function).

When doing the same exact save directly on server side (saving result of Y.encodeStateAsUpdate(doc) in the websocket server’s writeState) it works fine.

#2

Welcome to the discussion board @dev ,

Document updates are Uint8Array. They are not JSON-encodable. But there is help: https://docs.yjs.dev/api/document-updates#example-base64-encoding

#3

Thanks!

I’ve never felt so dumb, but … now that I use fromUint8Array on front-end and toUint8Array right before saving it to DB (postgres) … for some reason it gets saved malformed: compared to the same data saved by WS (to same db, same table, same bytea field) it’s 861 bytes blob vs 15 bytes blob - most of it is gone.

Yet if I look at the result after using toUint8Array on server side, it looks just fine (seems very similar to what I see if I look at Y.encodeStateAsUpdate(doc) in WS); state instanceof Uint8Array = true on both cases.

:confused:

#4

Funny enough, if I just save the base64 and use toUint8Array later when reading the state back from database for WS (Y.applyUpdate) … it works. But I’d prefer keeping it in a bytea field as Uint8Array tho.

#5

Not all databases support binary data. Most node libraries actually require a Buffer instead of Uint8Array (Although Buffer inherits from Uint8Array).

In your case, it might be helpful to use typescript. Because it checks for these types of type mismatches.

#6

Switched from “js-base64” to “rfc4648” (https://www.npmjs.com/package/rfc4648) and at least now the PG encode/decode functions from/to base64 work fine. Guess the base64 string generated by js-base64 had something in it that PG did not like.

#7

That is a bit disturbing because the encoding/decoding should be standardized. Is there any way that you could show the difference for a given document update?

Ideally, you can use any base64 decoder to decode update messages. If js-base64 is not compatible, I don’t want to recommend it (it just happened to be the most popular package that also exports esm).

#8

The “rfc4648” module definitely did not like the base64 strings I had created with “js-base64” and was not able to return them to Uint8Arrays – I couldn’t just switch the modules but had to delete and re-save all of the test data as well (that was saved to DB as base64).

In “rfc4648” there’s no special case exposed for Uint8Arrays, I just use their “stringify” to get the base64. Maybe it’s just not compatible with whatever “js-base64” does in their “fromUint8Array”.

Error I got from PG “decode” was “Error: invalid symbol “-” while decoding base64 sequence”.

I’ll try to recreate the situation and share.