Hey @Gaman It is great to hear that you had a good experience with Yjs!
ProseMirror and Quill are very adaptable. So it should be no problem at all to modify the toolbar, the design, or adding custom nodes / formatting options. For that I refer to the respective projects.
Yjs integrates over an editor-plugin to the editors. Since the editor data-models are strictly defined (Quill uses the delta format, ProseMirror has a schema definition). Yjs will just sync your data, even if you add custom nodes / formatting options to the editor.
For example, the Atlaskit editor (open-sourced by Atlassian) is based on ProseMirror. It adds a lot of custom nodes to the editor (e.g. tables, mentions, jira issues, …). Syncing still works as you can see in this demo: https://yjs-demos.now.sh/prosemirror-atlaskit/
On the other hand, if you want to write your own editor (not based on any of the editors that Yjs supports). Then you would need to write your own editor plugin.