How to lazy load an extra large Map?

1

I have a Map that needs to be able to support 1 million+ entries (representing nodes in a graph) synced with IndexedDB or with a server through WebsocketProvider. There is no deterministic way to partition the map, and random access by the guid key is required.

I obviously don’t want to load them all into memory. Only a few dozen nodes will be rendered at a time, and I can manually load adjacent nodes when the user clicks to expand connections.

The usual recommendation for lazy loading is to use subdocuments. However, YJS still loads all the keys into memory. I tested this using Chrome’s memory profiler. There is a separate Y.Doc instance in memory for each Map entry.

Is there a way to lazy load extra large Maps, i.e. only sync specific keys?

2

I gave it some thought and came to the conclusion that using a single, large Map is a poor approach in this case. It would be better, I believe, to take advantage of the inherent structure of the graph and daisy-chain subdocuments which can be lazy loaded. The user will only be viewing a small subgraph at a time. Full text search will require a separate indexing process anyway.

I’ll have to read up on subdocuments which appear to have limited/less well-documented provider support.