Y-websocket breaks when using webpack

1

When I run y-websocket/bin/utils with node 14 using a level-db persistence layer there are no errors. However… as soon as I run y-websocket/bin/utils using webpack the code initially runs okay, until you restart and read the document from the store. This results in the following error…

Error during saving transaction Error: Integer out of range!
    at readVarUint (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:16008:13)
    at readClientsStructRefs (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:5775:97)
    at /Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:6046:16
    at transact (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:7708:5)
    at readUpdateV2 (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:6041:3)
    at applyUpdateV2 (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:6134:3)
    at Module.applyUpdate (/Users/johnstleger/code/trint-backend/trint-servers/yjs-collaborative/dist/index.js:6148:58)

This error points to:

/**
 * Read unsigned integer (32bit) with variable length.
 * 1/8th of the storage is used as encoding overhead.
 *  * numbers < 2^7 is stored in one bytlength
 *  * numbers < 2^14 is stored in two bylength
 *
 * @function
 * @param {Decoder} decoder
 * @return {number} An unsigned integer.length
 */
const readVarUint = decoder => {
  let num = 0
  let len = 0
  while (true) {
    const r = decoder.arr[decoder.pos++]
    num = num | ((r & _binary_js__WEBPACK_IMPORTED_MODULE_1__["BITS7"]) << len)
    len += 7
    if (r < _binary_js__WEBPACK_IMPORTED_MODULE_1__["BIT8"]) {
      return num >>> 0 // return unsigned number!
    }
    /* istanbul ignore if */
    if (len > 35) {
      throw new Error('Integer out of range!')
    }
  }
}

Does anyone have any ideas around this. I’m at a bit of a loss…

We use webpoack to build packages in our monorepo.

2

Hi @jstleger0,

you can use webpack like that, but you need to be careful that no packages are polyfilled.

Usually, webpack builds a browser bundle that doesn’t work in nodejs. The problem is probably that you are using an incompatible buffer polyfill, or that you don’t bundle the nodejs version of isomorphic.js.

It may make sense to keep lib0 or at least isomorphic.js as an external dependency.

3

An error like this indicates that you don’t get a nodejs buffer / Uint8Array from the leveldb database. You might be able figure out the problem when you debug what is inserted into Y.applyUpdate when loading the initial content from leveldb. As I said, it must be a proper Uint8Array (or a object that inherits from Uint8Array like nodejs’ Buffer).