Posting encodeStateAsUpdate to server

Hey,

I have a small problem (probably related to encoding) trying to save the doc to database by posting it from front-end (Quill). I use Y.encodeStateAsUpdate(doc) and then POST the result to DB.

For some reason it creates an object ({“0”:1,“1”:1,“2”:159,“3”:245,"…) which does not work when reading data back from DB (TypeError: contentRefs[(info & binary.BITS5)] is not a function).

When doing the same exact save directly on server side (saving result of Y.encodeStateAsUpdate(doc) in the websocket server’s writeState) it works fine.

Welcome to the discussion board @dev ,

Document updates are Uint8Array. They are not JSON-encodable. But there is help: https://docs.yjs.dev/api/document-updates#example-base64-encoding

Thanks!

I’ve never felt so dumb, but … now that I use fromUint8Array on front-end and toUint8Array right before saving it to DB (postgres) … for some reason it gets saved malformed: compared to the same data saved by WS (to same db, same table, same bytea field) it’s 861 bytes blob vs 15 bytes blob - most of it is gone.

Yet if I look at the result after using toUint8Array on server side, it looks just fine (seems very similar to what I see if I look at Y.encodeStateAsUpdate(doc) in WS); state instanceof Uint8Array = true on both cases.

:confused:

Funny enough, if I just save the base64 and use toUint8Array later when reading the state back from database for WS (Y.applyUpdate) … it works. But I’d prefer keeping it in a bytea field as Uint8Array tho.

Not all databases support binary data. Most node libraries actually require a Buffer instead of Uint8Array (Although Buffer inherits from Uint8Array).

In your case, it might be helpful to use typescript. Because it checks for these types of type mismatches.

Switched from “js-base64” to “rfc4648” (https://www.npmjs.com/package/rfc4648) and at least now the PG encode/decode functions from/to base64 work fine. Guess the base64 string generated by js-base64 had something in it that PG did not like.

That is a bit disturbing because the encoding/decoding should be standardized. Is there any way that you could show the difference for a given document update?

Ideally, you can use any base64 decoder to decode update messages. If js-base64 is not compatible, I don’t want to recommend it (it just happened to be the most popular package that also exports esm).

The “rfc4648” module definitely did not like the base64 strings I had created with “js-base64” and was not able to return them to Uint8Arrays – I couldn’t just switch the modules but had to delete and re-save all of the test data as well (that was saved to DB as base64).

In “rfc4648” there’s no special case exposed for Uint8Arrays, I just use their “stringify” to get the base64. Maybe it’s just not compatible with whatever “js-base64” does in their “fromUint8Array”.

Error I got from PG “decode” was “Error: invalid symbol “-” while decoding base64 sequence”.

I’ll try to recreate the situation and share.