Transactions + Nested Subdocuments

[raine]

@chrysalis

Subdocuments are being touted as a game-changer, but they are not quite turn-key out-of-the-box solution one might expect after reading about them.

Yes, that’s been my experience. Seems like they have a lot of potential, but a bit undercooked.

@chrysalis

• You can’t really deepObserve subdocuments. Listening for changes requires doing everything that you would do for an ordinary y document (e.g. observe the changes to shared types or update events on the ydoc)

There is some good starter code here: How to sync thousands of documents and have local persistent store? - #6 by nokola. (Not sure why this hasn’t made it into a PR in two years. I know, we’re all busy )

@chrysalis

I find the lazy loading as a default behavior a bit annoying

Fair enough. Hopefully better support for autoLoad: true will make this easier.

@chrysalis

I am curious about the use case where you need atomicity of updates across multiple subdocuments. Can you tell me a bit about what exactly you are trying to do?

I have a tree with 1 million+ nodes per user (personal knowledge management app). It’s far too much to load into memory at once, so I need lazy loading, hence I need subdocuments.

A delete operation needs to delete all descendants atomically across nodes (i.e. subdocuments).

• a
  • b
    • c
    • d

Deleting b must also delete c and d. If this were a nested Y.Map I could use transact, but I can’t do this when they are separate subdocuments.

Now for this example you might suggest using “tombstoning” or another technique to clean up orphaned nodes, but this is just one example of many multi-node operations that require atomicity, so I don’t believe any kind of solution involving post hoc clean up is sufficient.

For example, consider a command called collapse which deletes a node and moves all its children up a level. If activated on b above, it should result in:

• a
  • c
  • d

Moving multiple nodes at once should be atomic.

I’m sure I will learn a lot more once I start writing some code, but hoping to avoid putting the effort into a custom provider if there are known (or implied) limitations to consistency when building trees or graphs with subdocuments.

Which goes back to @dmonad’s quote in the OP. What exactly are the limitations of subdocuments re: atomicity? What is the alternative without losing lazy loading and granular access control?

Thanks for reading this far