I got something wrong using y-mongodb-provider example

1

Due to the existence of a lot of more than 16M docs in my project, when I use the code of example 1, the code runtime memory consumption is too high, which makes me have to try other methods to solve this problem, I use example 2, the code runtime memory dropped significantly, but in the first step to get the doc from the database often encountered the error report, so much so, that the fetched " persistedYdoc" object value is empty. I suspect that the process of depositing the object into mongo is somehow wrong when the object is too large, but havenā€™t found concrete evidence of this.
Since I copied the code into the project without modifying it, Iā€™m not sure if ā€œAn other exampleā€ doesnā€™t apply to large documents, but here are some of the errors Iā€™ve encountered more often:

Error during saving transaction Error: Unexpected end of array
at Object.error.create (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/error.js:12:28)
at Object. (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:36:35)
at Module._compile (node:internal/modules/cjs/loader:1368:14)
at Object.Module._extensionsā€¦js (node:internal/modules/cjs/loader:1426:10)
at Module.load (node:internal/modules/cjs/loader:1205:32)
at Function.Module._load (node:internal/modules/cjs/loader:1021:12)
at Module.require (node:internal/modules/cjs/loader:1230:19)
at require (node:internal/modules/helpers:179:18)
at Object. (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/dist/decoding.cjs:11:16)
at Module._compile (node:internal/modules/cjs/loader:1368:14)

Error during saving transaction TypeError: The encoded data was not valid for encoding utf-8
at TextDecoder.decode (node:internal/encoding:443:16)
at Array._readVarStringNative (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:374:45)
at Object.readAny (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:497:79)
at UpdateDecoderV1.readAny (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/UpdateDecoder.js:98:21)
at Array.readContentAny (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/structs/ContentAny.js:105:21)
at readItemContent (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/structs/Item.js:703:83)
at readClientsStructRefs (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:164:13)
at /app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:391:16
at transact (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/Transaction.js:425:14)
at readUpdateV2 (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:384:3) {
code: ā€˜ERR_ENCODING_INVALID_ENCODED_DATAā€™
}

Error during saving transaction TypeError: readAnyLookupTable[(127 - readUint8(ā€¦))] is not a function
at readAny (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:497:79)
at Array. (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:479:18)
at Object.readAny (/app/node_modules/.pnpm/lib0@0.2.88/node_modules/lib0/decoding.js:497:79)
at UpdateDecoderV1.readAny (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/UpdateDecoder.js:98:21)
at Array.readContentAny (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/structs/ContentAny.js:105:21)
at readItemContent (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/structs/Item.js:703:83)
at readClientsStructRefs (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:164:13)
at /app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:391:16
at transact (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/Transaction.js:425:14)
at readUpdateV2 (/app/node_modules/.pnpm/yjs@13.6.10/node_modules/yjs/src/utils/encoding.js:384:3)

node version: 21.7.2
mongo version: 7.0

2

You already opened a ticket in the y-mongodb-provider. This forum is not meant for bug reports. Also, one ticket is enoughā€¦ Please donā€™t overload the (voluntary) maintainers by asking questions more than once.

3

Ok, I thought the two places might maintained by different people, sorry for the trouble

4

Even then, you donā€™t need to distract multiple people to solve a single issue. Please keep in mind that no one here gets paid for answering your question. Please be more considerate in the future.

6

You make a fair point. Distracting multiple volunteers unnecessarily was inconsiderate of me. Thank you for the feedback, I will learn from this experience.

1 Like