Gc is off but still losing delection history

Hey!
First of all thank you for this awesome library.

I am trying to build something similar to Google Docs version history and came across the prosemirror-versions demo. It works great but only if page is not refreshed. As soon as page is refreshed and data is loaded from the DB (indexeddb for example), all the deletion history is lost. Even though no additional changes were applied to the document in the mean time. It also reproduces on official demo of y-prosemirror, but I have also prepared my own demo for Tiptap here

I am using latest version of YJS and y-prosemirror. Any lead would be really beneficial.

Hereā€™s a demo video as well

Thank you :raised_hands:t2:

You should disable garbage-collection on the backend as well.

In Hocus Pocus we donā€™t initiate the document, so wonder where we need to do that? When we initiate the provider like new HocuspocusProvider({ document: yDoc }) we have to provide yDoc. And itā€™s gc is false. Backend just applies the updates using Y.applyUpdate if itā€™s in DB.

Maybe ask the hocuspocus folks directly.

@dmonad And why is the issue reproducible on the official prosemirror versions demo as well?

@Maneet Because I garbage-collect on the backend. There are too many users & changes for the same document - I had to enable garbage-collection. But it should work fine locally, in the same browser.

1 Like

And yes, this is quite confusing. Iā€™m sorry for that. Unfortunately, I havenā€™t been able to work more on the versions implementation to make it more accessible. It is being used by a couple of companies in production though.

1 Like

Here is what worked for me

Method #1 If you are using the Hocupocusā€™s database extension

const { Server } = require(ā€œ@hocuspocus/serverā€);
const { Database } = require(ā€œ@hocuspocus/extension-databaseā€);
const { TiptapTransformer } = require(ā€œ@hocuspocus/transformerā€);
const Y = require(ā€˜yjsā€™);

// Express endpoint for websocket

app.ws(ā€œ/:usernameā€, (websocket, request) => {
const server = Server.configure({
port,
debounce: 500,

    extensions: [
        new Database({
            fetch: async ({ documentName,document }) => {
                document.gc = false   //   Disabling Garbage Collection on Server
                const noteId = documentName
                const note = await Notes.findById(noteId)
                if (note.document) return note.document


                const ydoc = TiptapTransformer.toYdoc(
                    // the actual JSON
                    note.description,
                    // the `field` youā€™re using in Tiptap. If you donā€™t know what that is, use 'default'.
                    "default",
                    // The Tiptap extensions youā€™re using. Those are important to create a valid schema.
                    tipTapExtensions
                )

                const state = Y.encodeStateAsUpdate(ydoc)
                note.document = Buffer.from(state)
                await note.save()
                return state
            },
            store: async ({ documentName, state }) => {
                const noteId = documentName;
                const note = await Notes.findById(noteId)
                note.document = state;
                await note.save();
            }
        })
    ]
});
  
server.handleConnection(websocket, request);

});

Method #2 Without the database extension

const { Server } = require(ā€œ@hocuspocus/serverā€);
const { TiptapTransformer } = require(ā€œ@hocuspocus/transformerā€);
const Y = require(ā€˜yjsā€™);

app.ws(ā€œ/:usernameā€, (websocket, request) => {
const server = Server.configure({
port,
debounce: 500,
async onStoreDocument({ documentName, document }) {
const currentState = Y.encodeStateAsUpdate(document);
const noteId = documentName;
const note = await Notes.findById(noteId)
note.document = Buffer.from(currentState);
await note.save(); // save current document state to db
},
async onLoadDocument({ documentName, document }) {
document.gc = false;
const noteId = documentName;
const note = await Notes.findById(noteId)
//If the document state is present in DB
if (note.document) {
const stateInDb = note.document
const newDoc = new Y.Doc({ gc: false });
Y.applyUpdate(newDoc, stateInDb)
return newDoc
}

        // Else Create Yjs document from Tiptap JSON
        const docFromTiptapJSON = TiptapTransformer.toYdoc(
            note.description,
            // the `field` youā€™re using in Tiptap. If you donā€™t know what that is, use 'default'.
            "default",
            // The Tiptap extensions youā€™re using. Those are important to create a valid schema.
            tipTapExtensions
        )
        const freshState = Y.encodeStateAsUpdate(docFromTiptapJSON)
        note.document = Buffer.from(freshState)
        await note.save()  // save the state to db
        const docFromTiptapJSONwithGCDisabled = new Y.Doc({ gc: false });
        Y.applyUpdate(docFromTiptapJSONwithGCDisabled, freshState)
        return docFromTiptapJSONwithGCDisabled

    },
    
});

server.handleConnection(websocket, request);

});

This can seem confusing so let me know If you need any help.

Actually, I took some help from hocuspocus discord server and they have yDocOptions property available, so no need to do complex stuff use following:

Server.configure({
    yDocOptions: {
        gc: false,
        gcFilter: () => false
      }
})

It works but user name somehow gets lost and shows Unknown on each change and donā€™t know the reason.

Have you been able to show the diffs b/w snapshots?

@Maneet yes, please check the example sandbox in my original post. Replace IndexeddbPersistence with HocuspocusProvider and it works fine for me, except the username thing.

@dmonad is there any reason on why username (for example ā€œU1ā€) in permanentUserData.setUserMapping(yDoc, yDoc.clientID, "U1") doesnā€™t persist on refresh? If you try to snapshot couple of changes it shows correct username, but as soon as page is refreshed the username goes away and it shows Unknown. I agree that Y.PermanentUserData is an experimental feature, but it shouldnā€™t be showing on the first place, so trying to figure out if itā€™s Y.PermanentUserData or my backend? I donā€™t think itā€™s related to Hocuspocus because it happens with indexeddb as well. Check this sandbox for instance.

1 Like

@usman-web-dev @dmonad

While switching between snapshots I am getting following error

Applying a mismatched transaction
RangeError: Applying a mismatched transaction
    at EditorState.applyInner (http://localhost:3000/static/js/bundle.js:187157:40)
    at EditorState.applyTransaction (http://localhost:3000/static/js/bundle.js:187112:23)
    at EditorState.apply (http://localhost:3000/static/js/bundle.js:187087:17)
    at Editor.dispatchTransaction (http://localhost:3000/static/js/bundle.js:138294:30)
    at EditorView.dispatch (http://localhost:3000/static/js/bundle.js:194754:50)
    at renderSnapshot (http://localhost:3000/static/js/bundle.js:3929:5)
    at onClick (http://localhost:3000/static/js/bundle.js:3942:18)
    at HTMLUnknownElement.callCallback (http://localhost:3000/static/js/bundle.js:56730:18)
    at Object.invokeGuardedCallbackDev (http://localhost:3000/static/js/bundle.js:56774:20)
    at invokeGuardedCallback (http://localhost:3000/static/js/bundle.js:56831:35)

Any ideas what I am doing wrong?

Cannot say anything, until there is some code. You can check my sample and it works there on a minimal reproduction. Please put your code on sandbox, then I can have a look.

@usman-web-dev Actually I am using various other extensions like Tiptap Image and Youtube Extension with the editor.

Maybe these are causing issues. Any idea how to track additions / modifications of images and youtube embeds in history