we have an implementation with QuillJS and ShareDB for a collaborative editor (based on OT). Now we are searching for a solution that supports a track changes functionality (which is not really working with OT) and we luckily found Yjs (in combo with Prosemirror) which seems to be able to do this In the first place we only want to assign a user id to the text a user has written, formatted or deleted (which should then be colored or colored and stroked). Accepting/rejecting changes either removes the user id and makes the text therefore neutral or removes the text. Versioning is not a necessary feature at the moment.
The prosemirror-versions example (https://demos.yjs.dev/prosemirror-versions/prosemirror-versions.html) seems already to be very close to what we search for. But when activating the “Live Tracking” checkbox, the user can’t edit any text anymore (which is a bit strange since “live” suggests that the tracking should work live?). We played a bit around with the ySyncPlugin to solve this, but did not find a simple solution so far. Before we dig deeper, I just wanted to ask if there is maybe a simpler solution which we have overseen? Or is it even a bug? Or is it just a complicated problem and we need to go deeper?
The “live-tracking” feature tracks any changes live other users are editing. In the versioning demo, we are interested in the differences to the last version. This is achieved by walking through Yjs’ CRDT model and transforming it into a ProseMirror document (while reading deleted changes as if they were still part of the document). It is currently not supported to edit while tracking deletions because we need to account for position changes.
The versioning approach really is just about tracking and rendering the differences between versions. For a google-docs-like “suggestion feature” you don’t need the versioning approach. I would implement “suggestions” using decorations and relative positions. Deletions and Insertions would be represented differently:
Deletions
Deletions would simply be ProseMirror-decorations on the document. You would basically mark a range as “suggested to delete” and use ProseMirrors decoration approach to highlight them in a specific color. You could easily allow a dialog that allows you to render the associated user and a button to accept the change. Quill has a similar concept that would allow you to do this.
Insertions
Insertions would be attributed changes to the document. For an added paragraph/text-node you would assign it an attribute that associated it to the user that created it. You could maintain suggested changes even in a separate Yjs document to handle permissions more strictly.
That said, implementing this feature is definitely possible in ShareDB as well by performing OT transformations. The OT approach is not foolproof either. This is just a really complicated feature. Unfortunately, there are no Open-Source solutions for this yet.
I would love to create a suggestions-feature for Quill or ProseMirror in exchange for funding. This is how I finance the Yjs project. Let me know if this is something that you & your company are interested in.
thanks a lot for your detailed reply! Unfortunately we are no company, but a small group of 3 people developing an open source software in our free time (https://github.com/openevocracy/openevocracy). But that has also an advantage since we do not depend on any external goals, can freely choose our programming focus and can be open source with everything.
I think the information from you gives us a good starting point. About a year ago I was investing some time and tried solving this with Quill and ShareDB and came quite far, but I had problems with hierarchic (html) structures at some point. As far as I understood, OT was originally developed for linear structures (like plain text). What I read about the (proprietary) OT-implementation from the CKeditor developer team, it seems that “upgrading” OT to fully support html is a heavy problem (which we probably cannnot solve with our small group and very limited time). But when I understood this correctly, CRDT should be able to do this and my hope is that we “only” need to do similar things I did before with OT. Using the suggestions from you as a rough roadmap and my earlier experiences from Quill/ShareDB may bring us to a solution.
As soon as we have something usable, we will directly share it here in the forum to keep you and the community updated. This here seems to be a nice place
Decentralized communities (OpenEvocracy) are definitely welcome here as well!
The real value of Yjs & CRDTs is that you can build decentralized applications (no central server required). Another nice side-effect - and you are right about that - is that Yjs natively supports structured documents. If you are just interested in solving the suggestion feature, then the OT solution might be easier to understand. Still, I want to encourage you to try out the described approach above. As the author of ShareDB said, CRDTs are the future https://josephg.com/blog/crdts-are-the-future/
In Stackoverflow I have just written a post about my current state of knowledge about track changes. Just wanted to backlink it here for people who are also interested in this topic:
Thanks for putting this together and sharing what you learned @carlo!
I would say that you are actually looking for a “suggestions” approach. Track-changes usually refers to something like an editing-history. Something you can use to compare different time-stamped versions with each other. In GDocs you have a linear history of changes that you could describe as track changes. Yjs & y-prosemirror are able to track-changes using the versioning approach (https://demos.yjs.dev/prosemirror-versions/prosemirror-versions.html) that can be used to implement something very similar to Google Docs’ Track Changes.
“Suggestions” is another feature of GDocs that allows you to suggest text-changes. This is what you described.
Of course, there is no strict definition of what these terms mean. But Google was the first company that implemented them like this.
Yes, I think your right. But my experience is that most people call it “track changes”. In addition suggestions can be confusing, since there exist prosemirror suggestions plugins, which are a totally different thing. It’s not so easy to find the correct wording in this young field. Anyway, I also think now that suggestions is the more precise term and I updated my Stackoverflow post in a away that both names are covered. At some point one probably has to start the right way
as I am on the same quest to implement a track-changes functionality with ProseMirror, I wondered if you @carlo or anyone else, had any progress or ideas on this?
What I am interested to learn is how possible it is to retain the information to revert a certain change with Yjs? Say we have one “original version” for which track changes is applied. Then further down the edit history we have some change that isn’t what we would have liked to have applied to the document. To revert the change you can’t really just pick a snapshot before the said change occurred, as that would revert all the other changes that happened after that snapshot too (unless it was the last one).
So instead, you’d have to map the document from the reverted doc before the change to the current doc but I am not sure if that is possible with Yjs? Basically what I am asking can you time-walk the snapshots, drop a change, then re-apply the changes after that very much like you do in Git history. Hmm could you abuse UndoManager for this purpose?
And well I think change should be in plural as there most often is a series of changes you want to revert. To make a snapshot after each change could be kinda burdensome but I’m not sure about that?
just quickly wanted to inform you that there is no considerable progress from our side. Too limited time at the moment. But the topic is still hot. We started to look deeper into the wax-prosemirror code as it seems to be the most active and advanced project. It seems that they copied and improved a lot of code from Fidus Writer
Regarding your questions: to my current understanding this should be doable. But maybe @dmonad has some more details about it?
Yes this definitely is one of the most advanced features one can implement and it seems like it will take a very considerable amount of time to get it right. About that Fidus/wax editor thing, I’m not sure I would say wax improved the code that much as to me it seems their implementation does not work as well as Fidus’ does. Sure they reworked it a little but yeah, dunno. It’s a lot of code though.
@dmonad did reply to me in Gitter in which he said it should be doable:
I think you are asking whether it is possible to rewind specific operations - that also should be possible. You need to find the right operation, move it to the top of the stack and then undo that.
The yjs.dev website contains a Google-Docs like track-changes feature for the Prosemirror editor. It is incomplete and I want to do a rewrite of the functionality. But even in the current state, it works really well. The downside is that it uses more memory than it should, which is the reason for the rewrite. But unless you are writing a book, you should be fine.
However, the proper approach to me is still unclear; do I have to retrofit the UndoManager somehow or do some crazy stuff with the documents directly? One important feature I want is to revert a change(s) without hassle. PM has an invert operation with its transaction steps which is quite handy in this case but not sure how to do this in Yjs (or can I get the original PM transaction from the Yjs change somehow).
In a conclusion it was deemed that Yjs might be just a bit too much to add in with PM already being a quite handful as it is. If advancements are made to make this easier with Yjs we might take a look into it but for now, PM-only seems more doable.
Basically, you can create snapshots (a small binary string that describes the previos state of a document) and then render them using y-prosemirror. You can also render the differences and highlight the changes by the user who created them.
It is possible to revert to an old version. Simply revert to an old state (using snapshots and y-prosemirror), capture the current prosemirror state, and then overwrite the current Yjs state. In the future we could offer an API for this feature.
I’m not sure if you should use it now in production. I know that a few companies already work on this. Just to be clear, I will eventually rework the current API and build something different. So I don’t offer the same level of support for this.
Since a picture tells more than thousand words and so on, here’s what I have come up with PM only logic React App It still needs some additional work and for example, accepting a change doesn’t work.
Do you think there is some easy way to accomplish the same with Yjs? Basically what prosemirror-changeset is doing is keeping track of the changed content as Spans with user metadata and so on. It doesn’t however show changed mark-up and I also have to implement some special logic to revert a wrapping of nodes with other nodes (eg toggling a blockquote).
If Yjs isn’t directly able to emulate this I guess you could at least wrap the ChangeSet with a Yjs data structure to make broadcasting the changes a lot more convenient? At the moment I’m not sure exactly how to do it since you’d only want to send a delta of the changes instead of the whole set. Certainly you need a live PM document instance somewhere to compute the changes, similar to normal prosemirror-collab implementation, and send the changeset with the whatever else steps to any new users starting to edit the document.
If you can assume that all users receive the same steps and, therefore, get the same document eventually I guess the client-side prosemirror-changeset should result in the same changeset as in the server even without syncing the changes. But hmm you would still need to persist them to DB with ids and all to attach say comments to them so maybe that won’t work.
It would be cool if there was an easy solution to this with Yjs but I understand it’s a difficult problem to solve (and if you look at the source prosemirror-changeset it’s obvious there is some hairy computation involved (although I think diffing changes might be overkill)). I have no idea how flexible using prosemirror-changeset in the end is but I don’t see any other as approachable solutions at the moment.
Sure, I’ve gotten pretty far with the Fiduswriter approach of marks and attributes. Decided to not to couple this with Yjs as that would have required understanding the internals of Yjs as the API isn’t suited at the moment for live-tracking. Also there’s that bug with rendering node attributes from snapshots.
But yeah, it’s quite a task . Very difficult. I guess what I’m doing will be open-sourced at some point so people can witness the insanity for themselves. Just making proper tests will be some undertaking. I already found bugs in both Fiduswriter and CKEditor as I tried out various copy-paste combinations of partial slices.
Oh sure. A roadblock hah? It seems there are only roadblocks. The amount of edge cases is really staggering. But they all seem solvable although really time-consuming.
From how I see it now the problem boils down to being able to generate diffs from transactions as semantic changes/operations that merge seamlessly into the existing ones, reverting back to original content when they cancel each other out. And that sounds about as difficult as it is.
Hey @TeemuKoivisto@redon any progress in your solution? I am also trying to implement the suggestion feature from fidusWriter. However, I am stuck with their implementation.
Well, it can be done. I’m though happy I moved on from that project. And decided it’s not worth my time to maintain such library, no matter how useful that would be. Got better things to do.
In the first place we only want to assign a user id to the text a user has written, formatted or deleted (which should then be colored or colored and stroked). Accepting/rejecting changes either removes the user id and makes the text therefore neutral or removes the text. Versioning is not a necessary feature at the moment.
. Very difficult. I guess what I’m doing will be open-sourced at some point so people can witness the insanity for themselves. Just making proper tests will be some undertaking. I already found bugs in both Fiduswriter and CKEditor as I tried out various copy-paste combinations of partial slices.