Friday, March 15, 2019

Pythons & Windows

For a couple of years now, I have been developing software on machines with Mac OS X or Linux. Installing dev environments and tools for coding in languages like Node.js or Python is usually without any issues. Or is it just luck? At work, our team maintain packages, apps, tools and frameworks written in different languages (JavaScript, C#, Python, Ruby). Everything we develop should be platform independent.

Recently, we built a tool written in Python 3 and the tool is intended to be used by many teams within the organization. So, I asked a fellow developer with a Windows machine to try it out. As it turned out, installing Python on Windows was ... a bit different. 

Why use more than one version of Python?
It's important for us to be able to run both Python 2.7 and 3 on the same machine, because of peer projects using build tools like node-gyp (that still has dependencies to Python 2).

This is what we learned about setting up and run both the Pythons on a Windows machine.

First step: Download and install Python
Download the latest version of Python from
Install Python 3.7.2 or later using the downloaded installer.

What about Python 2.7?
Ok, Python 2.7 was already installed on the Windows machines we used, so we skipped that step. But you can download and install it from the same site, or install the windows-build-tools.

Use PowerShell
PowerShell is cool. To make PowerShell as smooth as Bash, I think you should create aliases for the different Python versions. An alias can be added to the PowerShell profile. Check out the guide from Microsoft about verifying existing profiles and creating new ones.

Adding Python aliases to your PowerShell profile will make it possible to run both Python 2.7 and Python 3 in the same way as in other systems.

But where did the Windows installers put the Pythons? To find out, you can run this command in PowerShell:
where.exe python

Now you're ready to edit the PowerShell profile. Open it by typing notepad $profile, and add these rows:
New-Alias python C:\<my path to Python 2.7>\python.exe
New-Alias python3 C:\<my path to Python 3.7.2 or later>\python.exe 

Note: you may need an additional step, enabling profile scripts to run when a PowerShell window is opened. To do that, open a new PowerShell window in Admin mode (important) and write:
Set-ExecutionPolicy RemoteSigned

When finished, close the admin window, and open a new regular one. 

Almost done 
Verify the aliases, by typing:

python --version
(should output something like 2.7.*)

python3 --version 
(should output something like 3.7.*)

The Python virtual environment on Windows
Creating a virtual environment is the same as in Mac OS X or Linux: Install it with pip install virtualenv (only once), and create a virtual environment in your Python project folder with virtualenv venv.

To activate in bash, I'm used to write source venv/bin/activate

But that doesn't work on Windows. Instead, VirtualEnv for Windows have custom PowerShell scripts prepared for activation. You'll find them in the venv\Scripts folder that virtualenv has created with the virtualenv venv command.

.\venv\Scripts\activate.ps1 will activate the virtual environment in your Python folder.

With this setup, running Python apps will be close to identical to the setups in Mac OS X and Linux.

Sunday, March 10, 2019

Updates to the Apache ZooKeeper Node.js client

At last, Node.js is platform independent again!

Eh, what?

Apache Zookeeper?
From the docs: "... ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C. ..."
(the docs)

A Node.js Zookeeper Client
You will find several Node.js implementations when browsing on, and one of them is simply called: zookeeper. Recently, I was added to the list of maintainers of the project, hosted on GitHub.

The project got my attention when we at work were looking for a new ZooKeeper client, because the API of the current one we were using had gone out of date. We were all developing on Mac OS X, and pushed changes to a CI using a Linux docker container. Implementing the new client seemed like a breeze.

Until a developer in a peer team came by our spot at the office:
- Hey, I cloned the app repo and followed your setup guide. But I get errors when installing packages. I think it's the "zookeeper" dependency that fails. Do you know what's wrong?

We quickly learned it was because Windows. We realized that the entire build process - compiling C code to a native Node.js add-on on the fly - used non-Windows compliant features like make, bash and gcc.

Oh sh*t. How can this be solved?
What about bypassing it with Docker? I've learned that Docker on Windows can be painful, because of access right issues between a Windows host and Linux in a container. Windows Subsystem for Linux (WSL) would probably work after some tweaking, but adding WSL as a requirement to run (or develop) an app is ... well ... a bit depressing.

How about fixing the root cause of the problem, the source code? I did choose that path and have spent a lot of time learning about Node.js add-ons, (a little bit of) C and C++, Windows equivalents to bash commands, node-gyp, and how to compile the Apache Zookeeper C Client from source.

Months later, I hit the "Create Pull Request" button. Shortly after, the changes were merged and an updated npm package was published.

Besides supporting Windows, the zookeeper npm package also builds on Node.js 10. Previous versions failed, because the use of deprecated V8 features.

Details of recent changes can be found in the Changelog.

Contribute to the Open Source project
If you are interested in contributing, check out the issues tab at the GitHub repo. Currently there are some issues labeled with "good first issue", go ahead and grab one!