I have a Map that needs to be able to support 1 million+ entries (representing nodes in a graph) synced with IndexedDB or with a server through WebsocketProvider. There is no deterministic way to partition the map, and random access by the guid key is required.
I obviously don’t want to load them all into memory. Only a few dozen nodes will be rendered at a time, and I can manually load adjacent nodes when the user clicks to expand connections.
The usual recommendation for lazy loading is to use subdocuments. However, YJS still loads all the keys into memory. I tested this using Chrome’s memory profiler. There is a separate Y.Doc instance in memory for each Map entry.
Is there a way to lazy load extra large Maps, i.e. only sync specific keys?