The Developer who blocked the Event Loop

Recently, our team started working on using Node.js v12 in our v10 app. Luckily, we have unit tests that cover most of the core features. In Node.js 12, one unit test failed. 😱

The test assert that the event loop isn't blocked, even when a large amount of data is processed. Data is processed using async patterns with promises. The unit test add a separate task, triggered with setTimeout, and should be executed during the processing of the large amount of data. But now it is processed after. What? 

Something has changed.

Everything is fine.

Oh no, the code is blocking the event loop.

Here's example code I've written for this post that will behave differently in Node.js 10 and 12.
setTimeout(() => log('first timeout: done!'));

setTimeout(() => {
  processData()
    .then(processData)
    .then(processData)
    .then(processData);
});

// this one should run before all the "processData" calls are done.
setTimeout(() => log('second timeout: done!'));

[full example here]

Breaking changes? 💔
There's a lot of good things in the blog and change log about the Node.js 12 release, but I can't find any breaking API changes. What about Node.js 11? I couldn't find anything about changed event loop behavior there either. 

Promises can block the Event Loop 💤
I found a blog post (linked below) and a GitHub conversation about harmonizing how tasks are prioritized in the browser and in Node.js. Promises and features like setTimeout are since Node.js 11 grouped into micro and macro tasks, with different prioritization in the event loop.

A simple hack 
Here's one way to make the example code behave like it does in Node.js 10. This will make the example code non blocking.

processData()
  .then(() => {
     setTimeout(processData);
});

I hope this post will give you some ideas and help when upgrading the Node.js version. 

Merry Christmas & Happy Coding!
🎅


References
New Changes to the Timers and Microtasks in Node v11.0.0
MacroTask and MicroTask execution order

Pythons & Windows

For a couple of years now, I have been developing software on machines with Mac OS X or Linux. Installing dev environments and tools for coding in languages like Node.js or Python is usually without any issues. Or is it just luck? At work, our team maintain packages, apps, tools and frameworks written in different languages (JavaScript, C#, Python, Ruby). Everything we develop should be platform independent.

Recently, we built a tool written in Python 3 and the tool is intended to be used by many teams within the organization. So, I asked a fellow developer with a Windows machine to try it out. As it turned out, installing Python on Windows was ... a bit different. 

Why use more than one version of Python?
It's important for us to be able to run both Python 2.7 and 3 on the same machine, because of peer projects using build tools like node-gyp (that still has dependencies to Python 2).

This is what we learned about setting up and run both the Pythons on a Windows machine.

First step: Download and install Python
Download the latest version of Python from https://www.python.org/downloads/
Install Python 3.7.2 or later using the downloaded installer.

What about Python 2.7?
Ok, Python 2.7 was already installed on the Windows machines we used, so we skipped that step. But you can download and install it from the same python.org site, or install the windows-build-tools.

Use PowerShell
PowerShell is cool. To make PowerShell as smooth as Bash, I think you should create aliases for the different Python versions. An alias can be added to the PowerShell profile. Check out the guide from Microsoft about verifying existing profiles and creating new ones.

Adding Python aliases to your PowerShell profile will make it possible to run both Python 2.7 and Python 3 in the same way as in other systems.

But where did the Windows installers put the Pythons? To find out, you can run this command in PowerShell:
where.exe python

Now you're ready to edit the PowerShell profile. Open it by typing notepad $profile, and add these rows:
New-Alias python C:\<my path to Python 2.7>\python.exe
New-Alias python3 C:\<my path to Python 3.7.2 or later>\python.exe 

Note: you may need an additional step, enabling profile scripts to run when a PowerShell window is opened. To do that, open a new PowerShell window in Admin mode (important) and write:
Set-ExecutionPolicy RemoteSigned

When finished, close the admin window, and open a new regular one. 

Almost done 
Verify the aliases, by typing:

python --version
(should output something like 2.7.*)

python3 --version 
(should output something like 3.7.*)

The Python virtual environment on Windows
Creating a virtual environment is the same as in Mac OS X or Linux: Install it with pip install virtualenv (only once), and create a virtual environment in your Python project folder with virtualenv venv.

To activate in bash, I'm used to write source venv/bin/activate

But that doesn't work on Windows. Instead, VirtualEnv for Windows have custom PowerShell scripts prepared for activation. You'll find them in the venv\Scripts folder that virtualenv has created with the virtualenv venv command.

.\venv\Scripts\activate.ps1 will activate the virtual environment in your Python folder.


With this setup, running Python apps will be close to identical to the setups in Mac OS X and Linux.

Updates to the Apache ZooKeeper Node.js client

At last, Node.js is platform independent again!

Eh, what?

Apache Zookeeper?
From the docs: "... ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C. ..."
(the docs)

A Node.js Zookeeper Client
You will find several Node.js implementations when browsing on npmjs.org, and one of them is simply called: zookeeper. Recently, I was added to the list of maintainers of the project, hosted on GitHub.

Background
The project got my attention when we at work were looking for a new ZooKeeper client, because the API of the current one we were using had gone out of date. We were all developing on Mac OS X, and pushed changes to a CI using a Linux docker container. Implementing the new client seemed like a breeze.

Until a developer in a peer team came by our spot at the office:
- Hey, I cloned the app repo and followed your setup guide. But I get errors when installing packages. I think it's the "zookeeper" dependency that fails. Do you know what's wrong?

We quickly learned it was because Windows. We realized that the entire build process - compiling C code to a native Node.js add-on on the fly - used non-Windows compliant features like make, bash and gcc.

Oh sh*t. How can this be solved?
What about bypassing it with Docker? I've learned that Docker on Windows can be painful, because of access right issues between a Windows host and Linux in a container. Windows Subsystem for Linux (WSL) would probably work after some tweaking, but adding WSL as a requirement to run (or develop) an app is ... well ... a bit depressing.

How about fixing the root cause of the problem, the source code? I did choose that path and have spent a lot of time learning about Node.js add-ons, (a little bit of) C and C++, Windows equivalents to bash commands, node-gyp, and how to compile the Apache Zookeeper C Client from source.

Months later, I hit the "Create Pull Request" button. Shortly after, the changes were merged and an updated npm package was published.

Besides supporting Windows, the zookeeper npm package also builds on Node.js 10. Previous versions failed, because the use of deprecated V8 features.

Details of recent changes can be found in the Changelog.

Contribute to the Open Source project
If you are interested in contributing, check out the issues tab at the GitHub repo. Currently there are some issues labeled with "good first issue", go ahead and grab one!