Towards the Fibers-free Meteor

If you’re working with Meteor, or at least know that i

t exists, you are most likely aware that it has one surprising and uncommon “superpower” – it can synchronously wait for a promise (typings, source). Of course, there’s nothing magical about it, just a ton of hacks some library called node-fibers.

As the name suggests, it’s a Node.js library, therefore, it won’t work in the browser. (And there’s no way around that, sorry!) It was created way back in 2011 – long before promises were even a thing; let alone async / await. If you ask me, it had a great run – it made mixing synchronous with asynchronous code possible with the same API. For example, it allowed Meteor to have the entire Collection API stay synchronous in both the browser and the server. (Just to make it clear – the browser has something called MiniMongo; it doesn’t use an actual database instance.) This made sharing the code between the front- and backend not only possible but also seamless. That’s also why optimistic UI works out-of-the-box in Meteor.

Now, let’s fast forward about ten years. Node.js v16 is soon to be released, introducing a few performance improvements and awaited (pun intended) features. It also broke node-fibers entirely, making it obvious that making Meteor work with v16 is not going to be easy. Soon after, the Meteor community raised a valid question: what now? Of course, it took a lot of (GitHub) discussions, issues, forum posts, and thinking, but for a relatively long time, nothing really happened.

Finally, a couple of months later, the Meteor team created meteor/meteor#11505 – an actionable roadmap to Fibers-less Meteor. Again, it made a lot of noise, as the plan was (and still is) complex and will definitely be the challenge the Meteor community has to face. I was following it closely but didn’t really participate, as I saw enough movement already. I wanted to wait and see if this would go as planned.

About a month later, Filipe Névola (Meteor CTO at that time) started working on probably the most tricky part of it all – Mongo API (meteor/meteor#11605). You see, now, when the “synchronous magic” is off the table, we have to make Collection API asynchronous (huh, again). Of course, we can’t do it in one step, as it’ll either break tens of existing projects or make the migration nearly impossible for bigger ones (or both, actually).

Luckily, it was already taken care of. The plan was to add *Async counterparts to all of the Collection API methods in the first step, let people use both for some time (a.k.a., the migration period), at some point remove the synchronous ones, and finally remove the unnecessary *Async suffix. Of course, every step has to be reflected in the released versions, i.e., there will be at least one version at each stage.

The first thing that I didn’t like about this pull request, was the fact that the .find() method also received an asynchronous counterpart. It was surprising but also deemed unnecessary to me, as it was already synchronous in the official MongoDB API. After a brief discussion, it got changed and matched the driver’s API. The second thing were the asynchronous constructors. There’s no way of making it work in TypeScript, and as the entire Meteor codebase (actually, most of the whole JavaScript ecosystem) is using it more and more, I found it a rather significant problem. However, it affected only the very internal APIs, and we all agreed that it could stay like that for now.

Time has passed, the pull request got stale and only got refreshed once in a while. I thought there was some more significant problem in one of the projects that Filipe tested it on, but no – that was not the case. The reason was that Filipe left Meteor. (It caused a lot of fuss in the community, but the air cleared in the end.)

That’s the moment I got more involved. We, Vazco, reached the Meteor team (thank you, Fred!) and agreed that I could actually continue Filipe’s work. It took us some time to finish all of the paperwork finally meet (timezones are hard), and it went public by the end of April. I reviewed the existing work, added more tests, cleaned up a little, and filed meteor/meteor#12028 about two weeks later.

Honestly, I was pretty sure we’ll be done with it in a week or two. But then Michael Newman got involved, and it soon became clear that it'd need more work and thus get delayed. It was actually for the best – the more we iron it out before the first release, the better. Long story short, one of Filipe’s and my assumptions was not really on point with the reality. We both assumed that the synchronous and asynchronous APIs should have separate collection objects. The idea was simple – newly created ones would follow the new API and throw when the old API would be used (the same goes for the other way round).

It was great on paper and in tests but caused a significant problem – what do we do with the Atmosphere packages? For example, if there’s a package X and it requires a collection instance, we have to always know which one it needs. Furthermore, the migration in the app is more complicated – there are more objects to pass around and take care of. If we’d just add the *Async counterparts, then the entire migration is as easy as adding an await and the suffix now, and removing the suffix later (i.e., once the synchronous API gets removed). Go and read these two long comments for a detailed discussion: 1, 2. (There were also some other, rather minor things, that I won’t mention here.)

Anyway, I nagged contacted Filipe for one last thing and… We were ready to go! A few small commits and reviews later the pull request was finally done. Two weeks later, it got released as a part of the v2.7.4 beta (2.7.4-beta.2, to be exact). Yes, you can give it a go today!

Overall, it was a wild ride. And that’s literally the first pull request from the list – there are many more to go, towards the Fibers-free Meteor. Don’t worry – we’re just starting. And it’s not only us – go and check out the list of companies and individuals that the Meteor team has on board to make it happen! I won’t tell you what Vazco will work on next (yet), but as soon as it’ll be publicly available, we’ll most likely use our Open Source Group’s GitHub board.