I’m building an end-to-end encrypted note taking app using Yjs and I’ve been wondering about the quality of the choices I’ve made and how to improve them. Any help is appreciated.
End-to-end encryption algorithm: I chose to use classical end-to-end encryption (not nearly as advanced as Serenity Notes) based on the algorithms of Bitwarden and LastPass, mainly because I lack time and knowledge about double-ratchet-based end-to-end encryption. I’m wondering if the current level of end-to-end encryption is acceptable or if I should invest time to bring double-ratchet into the app. I’m also wondering about the performance aspects of using double-ratchet in the realtime collaborative environment.
Update syncing challenges: I followed @dmonad’s suggestion to store a linear history of all the document updates. The issue I’m facing is that my app generates a relatively high amount of updates, so sometimes the decryptions start to impact the document loading performance. I’m using libsodium for encryption/decryption, by the way. To alleviate this issue I made the client always send the full merged update back to the server after the initial sync. This doesn’t seem very scalable so I’m wondering how I can further improve this. The next step is probably to somehow throttle this act of merging all updates, but I’m wondering what would be the optimal heuristic in this case. There is also the issue that the act of replacing the small updates with the full updates together with the decryption performance impact affects the client’s ability to ask for a range of document updates, so I’m wondering how to deal with that too.
Database performance challenges: At the beginning of development I decided to use Postgres as the main database. All encrypted updates are stored in a table called
page_updatesand I’m implementing a Redis wrapper to cache the reads of the updates and to buffer the writes, but I don’t think that’s very sustainable. Each page can easily take around 300kb of Redis’ memory. That size may lower too much the max capacity of concurrent users and the effectiveness of update caching. I saw @dmonad’s suggestion to use leveldb for caching, but I’m wondering how to do it and how much can leveldb help. I saw it can only be used by a single process. Should I write a server around it? I guess I just don’t understand the strengths of leveldb very well.