fredag 10 januari 2020

Documentation: The Good Parts

Writing documentation is probably the most boring part of software development. As developers, many of us avoid it. We write docs only when someone else force us, twist our arms or assign us JIRA tickets.
- Was it the project manager, or maybe that senior tech lead guy with the massive beard that made you write the docs?
What I want to describe in this post isn't complex Word documents explaining some software. It isn't a guide on how to write something clever about a public FirstName property of a Person class.

It's about JavaScript and data types. Keeping it simple, and fun.

 

https://commons.wikimedia.org/wiki/File:Unofficial_JavaScript_logo_2.svg
JavaScript is dynamic
JavaScript won't stop you from passing in whatever you want to a function, even when the code is expecting data of a specific type. It will crash during runtime instead, yes. When parsing or iterating something that cannot be parsed or iterated.

- No type checks for you!
This is also what makes the language very flexible. Simplistic syntax, functions as first class citizens and with a little less noise. There is no need for explicit data type declarations, generics or interfaces, making it easy to write code in less than 100 characters width. In fact, you will find yourself writing code in less than 80 chars quite often. In addition, having only one thread to worry about is a good thing (for most of us). And I believe the V8 engine is fast enough too.
- It would be nice if there was something that could let us have a little bit of both.

Did you say TypeScript?
TypeScript is cool. But I'm not sure it is the way to go for back-end code written for Node.js. Node.js itself is probably a bad choice if you are in great need of strongly typed language features on the server. Why not go for C# instead? 


If your code lives in the browser, well, that is something different. You are basically stuck with JavaScript for web development - whether you like it or not. TypeScript, or WASM, can probably help you - after some initial setup, headache & build scripts.

 

Have you tried JSDoc?
During my work on the Open Source project node-zookeeper, I have learned how JSDoc can help us make the JavaScript Developer Experience a lot better - without the headache. Your editor will give you hints and warnings about parameters and data types.

Here's an example:

/** @returns {number} */
function getRandom () {
    ...
}


With parameters:

/**
 * @param {number} minValue
 * @param {number} maxValue
 * @returns {number}
 */

function getRandom (minValue, maxValue) {
    ...
}

 
In the example, documentation is used only for describing the data types of input parameters and the return type. Nothing else. I would recommend you to try hard and refactor the actual code, making additional comments not needed. Is the function name getRandom good enough?

- Maybe I should add some comments to the code here?
(don't do it)

Your editor is smart
Your IDE or editor will figure out the JSDoc syntax, even for custom type definitions. Custom type definitions are for objects - custom data types - that cannot be described as built in strings, booleans, numbers, objects or arrays.

Another example, this one is from the typedefs.js of the node-zookeeper project, describing a ZooKeeper node changed callback:


/**
 * Watch callback
 * @typedef {callback} watchCb
 * @param {number} type
 * @param {number} state
 * @param {string} path
 */
 


The type definition for the watcher can be referenced in the code like this:
/** @param {watchCb} callback */

The node-zookeeper project has a typedefs.js file in the root folder. A smart IDE, like WebStorm, will parse it and use the type definitions to give you and your coder friends a nice developer experience.



When dynamic isn't good enough, use JSDoc. But be careful, use it only when needed. Or else, you will be stuck in the Documentation Swamp.

söndag 5 januari 2020

Experimental Node Mode

The latest Long-Term Support (LTS) version of Node.js has support for writing and consuming modules in ECMAScript style - without any tools or frameworks. That is good news! I have mostly been writing back-end code using the CommonJS standard, but have always preferred the ECMAScript way, commonly used in front-end development with tools like Babel.

Now, with Node.js 12, there is native support for ECMAScript modules. But it has to be manually enabled, by using the '--experimental-modules' flag.

The current version - Node.js 13 - doesn't need a flag to enable ECMAScript modules. Great! It is on by default. But the runtime will log a message on startup, a warning that the feature is not ready for production yet. Maybe we can expect it to be a default setting, without experimental warnings, in the next LTS later this year?

I have tried out ECMAScript modules in Node.js, using the experimental flag and the type: module in package.json.


WebStorm IDE with ES Module support
Import tooling support, not perfect, but good enough.

Cool, but not perfect yet
I ran into issues when writing & running unit tests. My favorite unit testing library is tape, but it doesn't support ECMAScript modules natively:
Must use import to load ES Module

What about the AVA test runner ? No native support yet. Mocha? I couldn't find any info about it, but tests won't run.

Tape is just JavaScript 👍
It is convenient, but tape unit tests doesn't require the built in runner. Both AVA and Mocha need a runner. Using tape, you can run the unit tests just like any other JavaScript file with the Node.js CLI.

Don't try this at ... work?
Besides that using a default unit testing runner usually is a good thing, there's also a huge drawback with the experimental & non-standard approach that I describe in this post. The node command will run a single file only. 👀 

But, hey, we're in Experimental Mode right! A workaround would be to execute a custom runner containing all our unit tests. Just don't forget to add every new file to it.

"scripts": {
    "test": "node --experimental-modules test/run.js | tap-spec"
}

(from the script section of package.json)

The test/run.js file:
import 'path-to-unit-test'
import 'path-to-another-unit-test'


Try this at home, folks. Don't try it at work (yet). 🏡

torsdag 26 december 2019

The Developer who blocked the Event Loop

Recently, our team started working on using Node.js v12 in our v10 app. Luckily, we have unit tests that cover most of the core features. In Node.js 12, one unit test failed. 😱

The test assert that the event loop isn't blocked, even when a large amount of data is processed. Data is processed using async patterns with promises. The unit test add a separate task, triggered with setTimeout, and should be executed during the processing of the large amount of data. But now it is processed after. What? 

Something has changed.


Everything is fine.

Oh no, the code is blocking the event loop.


Here's example code I've written for this post that will behave differently in Node.js 10 and 12.
setTimeout(() => log('first timeout: done!'));

setTimeout(() => {
  processData()
    .then(processData)
    .then(processData)
    .then(processData);
});

// this one should run before all the "processData" calls are done.
setTimeout(() => log('second timeout: done!'));

[full example here]

Breaking changes? 💔
There's a lot of good things in the blog and change log about the Node.js 12 release, but I can't find any breaking API changes. What about Node.js 11? I couldn't find anything about changed event loop behavior there either. 

Promises can block the Event Loop 💤
I found a blog post (linked below) and a GitHub conversation about harmonizing how tasks are prioritized in the browser and in Node.js. Promises and features like setTimeout are since Node.js 11 grouped into micro and macro tasks, with different prioritization in the event loop.

A simple hack 
Here's one way to make the example code behave like it does in Node.js 10. This will make the example code non blocking.

processData()
  .then(() => {
     setTimeout(processData);
});

I hope this post will give you some ideas and help when upgrading the Node.js version. 

Merry Christmas & Happy Coding!
🎅


References
New Changes to the Timers and Microtasks in Node v11.0.0
MacroTask and MicroTask execution order

fredag 15 mars 2019

Pythons & Windows

For a couple of years now, I have been developing software on machines with Mac OS X or Linux. Installing dev environments and tools for coding in languages like Node.js or Python is usually without any issues. Or is it just luck? At work, our team maintain packages, apps, tools and frameworks written in different languages (JavaScript, C#, Python, Ruby). Everything we develop should be platform independent.

Recently, we built a tool written in Python 3 and the tool is intended to be used by many teams within the organization. So, I asked a fellow developer with a Windows machine to try it out. As it turned out, installing Python on Windows was ... a bit different. 

Why use more than one version of Python?
It's important for us to be able to run both Python 2.7 and 3 on the same machine, because of peer projects using build tools like node-gyp (that still has dependencies to Python 2).

This is what we learned about setting up and run both the Pythons on a Windows machine.

First step: Download and install Python
Download the latest version of Python from https://www.python.org/downloads/
Install Python 3.7.2 or later using the downloaded installer.

What about Python 2.7?
Ok, Python 2.7 was already installed on the Windows machines we used, so we skipped that step. But you can download and install it from the same python.org site, or install the windows-build-tools.

Use PowerShell
PowerShell is cool. To make PowerShell as smooth as Bash, I think you should create aliases for the different Python versions. An alias can be added to the PowerShell profile. Check out the guide from Microsoft about verifying existing profiles and creating new ones.

Adding Python aliases to your PowerShell profile will make it possible to run both Python 2.7 and Python 3 in the same way as in other systems.

But where did the Windows installers put the Pythons? To find out, you can run this command in PowerShell:
where.exe python

Now you're ready to edit the PowerShell profile. Open it by typing notepad $profile, and add these rows:
New-Alias python C:\<my path to Python 2.7>\python.exe
New-Alias python3 C:\<my path to Python 3.7.2 or later>\python.exe 

Note: you may need an additional step, enabling profile scripts to run when a PowerShell window is opened. To do that, open a new PowerShell window in Admin mode (important) and write:
Set-ExecutionPolicy RemoteSigned

When finished, close the admin window, and open a new regular one. 

Almost done 
Verify the aliases, by typing:

python --version
(should output something like 2.7.*)

python3 --version 
(should output something like 3.7.*)



The Python virtual environment on Windows
Creating a virtual environment is the same as in Mac OS X or Linux: Install it with pip install virtualenv (only once), and create a virtual environment in your Python project folder with virtualenv venv.

To activate in bash, I'm used to write source venv/bin/activate

But that doesn't work on Windows. Instead, VirtualEnv for Windows have custom PowerShell scripts prepared for activation. You'll find them in the venv\Scripts folder that virtualenv has created with the virtualenv venv command.

.\venv\Scripts\activate.ps1 will activate the virtual environment in your Python folder.


With this setup, running Python apps will be close to identical to the setups in Mac OS X and Linux.



söndag 10 mars 2019

Updates to the Apache ZooKeeper Node.js client

At last, Node.js is platform independent again!

Eh, what?

Apache Zookeeper?
From the docs: "... ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C. ..."
(the docs)

A Node.js Zookeeper Client
You will find several Node.js implementations when browsing on npmjs.org, and one of them is simply called: zookeeper. Recently, I was added to the list of maintainers of the project, hosted on GitHub.

Background
The project got my attention when we at work were looking for a new ZooKeeper client, because the API of the current one we were using had gone out of date. We were all developing on Mac OS X, and pushed changes to a CI using a Linux docker container. Implementing the new client seemed like a breeze.

Until a developer in a peer team came by our spot at the office:
- Hey, I cloned the app repo and followed your setup guide. But I get errors when installing packages. I think it's the "zookeeper" dependency that fails. Do you know what's wrong?

We quickly learned it was because Windows. We realized that the entire build process - compiling C code to a native Node.js add-on on the fly - used non-Windows compliant features like make, bash and gcc.

Oh sh*t. How can this be solved?
What about bypassing it with Docker? I've learned that Docker on Windows can be painful, because of access right issues between a Windows host and Linux in a container. Windows Subsystem for Linux (WSL) would probably work after some tweaking, but adding WSL as a requirement to run (or develop) an app is ... well ... a bit depressing.

How about fixing the root cause of the problem, the source code? I did choose that path and have spent a lot of time learning about Node.js add-ons, (a little bit of) C and C++, Windows equivalents to bash commands, node-gyp, and how to compile the Apache Zookeeper C Client from source.

Months later, I hit the "Create Pull Request" button. Shortly after, the changes were merged and an updated npm package was published.

Besides supporting Windows, the zookeeper npm package also builds on Node.js 10. Previous versions failed, because the use of deprecated V8 features.

Details of recent changes can be found in the Changelog.

Contribute to the Open Source project
If you are interested in contributing, check out the issues tab at the GitHub repo. Currently there are some issues labeled with "good first issue", go ahead and grab one!