OOM error on y-websocket/y-leveldb

#1

I recently saw this out-of-memory error on NodeJS on our production server while using y-websocket in conjunction with y-leveldb. We’re using a slightly modified version of the source code for our server, so it’s possible I’ve caused this on my own, but because it’s in the lib0/dist/buffer code that it failed, it may also be due to yjs:

Moving file to '/home/relmprod/relm/server/assets/fc9618e5fa0800bd1c26835408a27f78-16746.png'
Moving file to '/home/relmprod/relm/server/assets/8f998301a5aa8b1ce88523d438ffa537-10218.webp'
[66.249.75.230] 400 (0f0a2c6c): can't authenticate
Error: can't authenticate
    at /home/relmprod/relm/server/middleware.js:65:29
    at process._tickCallback (internal/process/next_tick.js:68:7)
Caused By:
Error: invalid signature
    at Object.findOrCreateVerifiedPubKey (/home/relmprod/relm/server/db/player.js:74:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)

<--- Last few GCs --->

[2168:0x3f5c790] 909158418 ms: Mark-sweep 1369.8 (1456.2) -> 1369.7 (1456.2) MB, 1905.0 / 0.0 ms  (average mu = 0.133, current mu = 0.000) allocation failure GC in old space requested
[2168:0x3f5c790] 909160235 ms: Mark-sweep 1369.7 (1456.2) -> 1369.8 (1437.7) MB, 1815.7 / 0.0 ms  (average mu = 0.073, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xf9e402dbe1d]
    1: StubFrame [pc: 0xf9e412840b1]
Security context: 0x209e7bb1e6c1 <JSObject>
    2: writeVarUint [0x2b8880461ed9] [/home/relmprod/relm/server/node_modules/yjs/node_modules/lib0/dist/buffer-69e778d4.cjs:246] [bytecode=0x16a574b77831 offset=0](this=0x2b888043fb89 <Object map = 0xcacff75d851>,encoder=0x18dbbbf9e679 <Encoder map = 0xcacff77bca9>,num=137)
    3: writeLeftID [0x2b888045db39] [/home/relmp...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8fb090 node::Abort() [node]
 2: 0x8fb0dc  [node]
 3: 0xb0322e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb03464 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xef74c2  [node]
 6: 0xef75c8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf036a2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf03fd4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf054e9 v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [node]
10: 0xf06cae v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
11: 0xecef25  [node]
12: 0xed3be4 v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [node]
13: 0xe5d11a v8::internal::TranslationBuffer::CreateByteArray(v8::internal::Factory*) [node]
14: 0xc739af v8::internal::compiler::CodeGenerator::GenerateDeoptimizationData() [node]
15: 0xc74142 v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
16: 0xd82cab v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
17: 0xd830bd v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl(v8::internal::Isolate*) [node]
18: 0xc470cd v8::internal::Compiler::FinalizeCompilationJob(v8::internal::OptimizedCompilationJob*, v8::internal::Isolate*) [node]
19: 0xc3d65b v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
20: 0xea494b v8::internal::StackGuard::HandleInterrupts() [node]
21: 0x1170185 v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
22: 0xf9e402dbe1d
./run-prod.sh: line 5:  2168 Aborted                 (core dumped) PORT=1235 YPERSISTENCE=./relm-data DATABASE_NAME=relm-prod node server.js
#2

See also https://github.com/yjs/y-websocket/issues/23

1 Like
#3

For those following along, looks like the underlying memory leak that was causing this has been fixed.

1 Like
#4

Excellent! Thanks to all those who helped track it down (@micrology, @ellisonbg)!

#5

At https://github.com/micrology/y-websocket/tree/wss there is a fork of y-websocket that includes both this fix and the code for using wss://, as well as a README with additional suggestions about how to implement y-webserver on a remote server. Hopefully @dmonad will merge some or all of this into the master sometime.

1 Like