Create Y.Map is empty

1

Hey,
I want to create a Y.Map and then add it to another Y.Map. However, it seems to be impossible to create a Y.Map with data?
I created a simple example project here for you to check out: create ymap empty - CodeSandbox
You just have to click on the button and check the console logs. You will see that their is no data stored in the Y.Map even though I just called .set() before.

For everyone who doesnt want to go into the code sandbox I just copy the code in here as well:

const newMap = new Y.Map();
newMap.set("name", "name");

console.log(newMap, newMap.toJSON()); // Y.Map, { }

What exactly is the problem here?

2

Shared types only work if they have been inserted into a Y.Doc:

const doc = new Y.Doc();
const rootMap = doc.getMap();
const newMap = new Y.Map();
rootMap.set('map', newMap)
newMap.set("name", "name");
3

Thank you, I thought so too. However, this does not always seem to be the case.

I just updated the example codesandbox (which makes it much more complicated unfortunately).
The important code part is in src/TodoList/TodoInput.tsx. Here I do it in the “wrong” order but it still works. How is this possible?

And even more confusing: If you add the y-websocket provider to sync the ydoc, it will only work for the first added “todo” and not for any other. This is actually not completely true. It is correct if I use my websocket, but not if I use the demo server. With the demo-server it just doesnt sync at all.

So I guess my problem is different than I thought and my y-websocket is the problem? And this despite the fact that I built the simplest version of it.

4

newTodoMap is added to todoMap which is set to activeCategory which is in categoriesMap which is attached to your ydoc :slight_smile:. That’s why it works in the second case. In the first case the new Y.Map was not attached to a Y.Doc.

I wouldn’t trust the demo server. Run your own y-websocket server from localhost to confirm the functionality before trying it over a domain. That will rule out various network issues that can be the case when connecting to a public websocket server.

You just said that your y-websocket worked and the demo websocket was the problem, so I’m not really following you here.

5

I guess I worded it badly. With the demo ws all added todos are shown. with my own (seen on the linked github) it only works until the connection is established. With both no content is synced.

6

Can you describe the steps to reproduce? The one | two | three is being synced properly over the websocket for me when I have the codesandbox open in two tabs.

7

Man. I really did bad debugging before this forum post. I am sorry to waste your time.
It doesnt sync with the demo server for me because it doesnt connect with it in the first place (Error 503: Unavailable).

With my own local websocket server, it does not work at all.

const WebSocket = require("ws");
const http = require("http");
const Y = require("yjs");

const server = http.createServer((request, response) => {
  response.writeHead(200, { "Content-Type": "text/plain" });
  response.end("okay");
});

const port = process.env.PORT || 1234;

// y-websocket
const wss = new WebSocket.Server({ server });
wss.on("connection", yUtils.setupWSConnection);

server.listen(port, () => {
  console.log(`listening on port: ${port}`);
});

As soon as it is connected, I have the original problem again. The set I try to add is empty again. I cant really check if it is a problem with my websocket or with my frontend as I cant use the demo server right now.

But I guess, my local frontend is identical to the codesandbox and it does work for you. Furthermore, it does kind of work as long as I dont use my websocket server. So the problem should be with my websocket implementation, but I tried to built the most simple version of it and cant find an error there. Maybe you can see the problem?

8

No worries! I’m just glad it’s something I can actually help with. There are a lot of questions on here about quill and prosemirror that I don’t know anything about.

We know it’s a problem with your websocket, because the data is set properly on the client when not connected.

The simplest server would be to use the built-in server:

HOST=localhost PORT=1234 npx y-websocket

The next simplest server is:

const server = require('y-websocket/server')

server().listen(port, host, () => {
  console.info(`running at '${host}' on port ${port}`)
})

A more advanced setup is only needed if you want to customize the underlying server code, such as adding authentication. This is closest to your code above. I think you’re just not upgrading the HTTP connection to WS. Websocket connections always start with an HTTP request that gets upgraded.

const WebSocket = require('ws')
const http = require('http')
const wss = new WebSocket.Server({ noServer: true })
const setupWSConnection = require('./utils.js').setupWSConnection

const host = process.env.HOST || 'localhost'
const port = process.env.PORT || 1234

const server = http.createServer((request, response) => {
  response.writeHead(200, { 'Content-Type': 'text/plain' })
  response.end('okay')
})

wss.on('connection', setupWSConnection)

server.on('upgrade', (request, socket, head) => {
  wss.handleUpgrade(request, socket, head, ws => {
    wss.emit('connection', ws, request)
  })
})

server.listen(port, host, () => {
  console.log(`running at '${host}' on port ${port}`)
})
1 Like
9

Thank you very much. I really thought I understood y-websocket by now, but I totally forgot the http upgrade. :smiley:
However, it still does not work. And it confuses me even more I guess.

So first, I added the upgrade code to my version of the websocket server and I was able to add entries to my list again, even after the connection was established. BUT the data did not sync at all.

I always overlooked this option in the documentation, so thanks for that. However, similar problem still exists. As soon as it is connected, I cant add new data to the list.
But now it gets interesting. I can add new categories to top level menu. And as soon as I switched there once I can add data AND it gets synced. Unfortunately I wont have time to investigate it this further now. But I thought you may have another good idea so I post this anyways. Ill try to figure more out tomorrow.

Thank you again!

10

I would start with the smallest example that syncs over a local websocket (using the built-in server) and then build up iteratively until you you have your Todo list. Start with a single Y.Doc and Y.Map with no user interaction. If you get stuck, post another Code Sandbox and maybe someone can help.

11

Now this is actually the problem I originally wanted to solve with this codesandbox, because it happened to me in my actual project. I’ll try to explain it, I’ve adapted the CodeSandbox so that you can test it very easily. On line 10 I added the constant variable:

const USE_CATEGORIES = false;

In case the demo server doesnt work, you can run a websocket locally with:

HOST=localhost PORT=1234 npx y-websocket

and change the URL in line 11.

Lets now get to the problem description:
Top level SharedTypes are synced without problem. If you set USE_CATEGORIES to false the Todo Y.Map is synced correctly. If you set USE_CATEGORIES to true you can see that the categories (in the gray menu list) are synced correctly.
But if USE_CATEGORIES is true, the Todos are not synced automatically. You first have to manually switch the category in the menu to reload the Todo Y.Map.

Codewise the problem seems to be here (line 95-105):

const categoriesMap: YMap<YMap<any>> = ydoc.getMap("categories");

// get activeCategory Map
let activeCategory = categoriesMap.get(activeCategoryName);
// activeCategory is undefined on first load, but the if block here does add a new map with the correct key to the top level category Map
if (!activeCategory) {
  activeCategory = new Y.Map();
  categoriesMap.set(activeCategoryName, activeCategory);
}
setTodoMap(activeCategory);

If the activeCategory is loaded before the y-websocket connection is established it doesnt sync, if it is loaded after the connection it works as expected.

12

categoriesMap.set will override any existing value, even if it has not yet synced. Last write wins.

So a synchronous !activeCategories check might be executing too early, making the app think that it hasn’t been initialized.

I would recommend waiting until the provider has synced before setting the categoriesMap.

wsProvider.on('sync', function(isSynced: boolean))

Also you may want to setTodoMap in the categoriesMap observer rather than once on load. It is a reactive type, so you want to subscribe to all changes. Sometimes I set a loaded or synced flag after the first sync if other parts of my app need to know about it.

13

Thanks again for your great reply!

Oh. I thought this is not the case for Yjs Datatypes and I dont know why I thought so, I guess I assumed that it works this way since it works this way with top level yjs datatypes.
ydoc.getMap("name") does get merged automatically if two clients define it in the same way, right? Is there a way to get the same functionality with nested SharedTypes?
Like:

const map1 = ydoc.getMap("map1");
const map11 = map1.getMap("map1");

I don’t want to have to wait for the ws provider to sync, as I want to build an app that works completely offline as well.

This seems to be a good idea, ill try it out to see what changes.

14

A map can only have one value per key, so if two clients set the same key, one will win.

getMap is a getter, so it it will never overwrite data.

You should be able to do offline first. I’m not sure how to explain this process with nested types unfortunately.

The CRDT ensures that updates on different clients received in different orders will converge deterministically to the same value in the end. It is conflict-free, but not collision-free, so to speak. If two clients set the same key, one of them has to win. If you want to keep both, use a Y.Array or Y.Map with unique keys. Otherwise, you have to have a strategy that works for your app. What exactly do you want to have happen when two clients set the same categoriesMap? Thinking through that might make it easier to determine the technical solution.

15

That’s true about yjs, I agree. I just thought that since ydoc.getMap() is somehow also a setter (it creates a new map if there was none), that a ymap.set(“key”, ymap2) would also merge instead of overwrite. I admit that this doesn’t really make sense, but I just assumed it.

I do not have a use case for the thing I wanted to figure out with this forum post. I just wanted to know tbh :smiley:

So, thank you again for the wonderful help here.

1 Like