Showing posts with label Apache ZooKeeper. Show all posts
Showing posts with label Apache ZooKeeper. Show all posts

Sunday, April 18, 2021

Confessions of an Open Source Contributor

I am one of the contributors to an Open Source project called zookeeper. It's an Apache ZooKeeper client made for Node.js. You can find the package on npm and the source code at GitHub.

What’s ZooKeeper?

"Apache ZooKeeper ... providing configuration information, naming, synchronization and group services over large clusters in distributed systems ..."

In short, Apache ZooKeeper is made of two parts: a service, and clients that communicate with it. Included in the packaging from Apache, there is a client made for the Java platform and also a C based client. You will most likely find one for your favorite language out there as a third party Open Source project.

Why joining Open Source?

A couple of years ago, I was part of a team in an organisation that use Apache ZooKeeper heavily. The team maintains a Node.js based app that communicates with a ZooKeeper service.

One day, the app began to raise strange warnings and unexpected errors. We looked into it and found out that the reason was probably caused by the client no longer being compatible with the API of the service.

There wasn't much happening in the client's repository:
unanswered questions from users, issues not taken care of and pull requests not being reviewed.

The project was abandoned.

So, we used a different client that seemed fairly updated and stable. The strange behavior in our app was now gone. Great! It worked on our machines and in production. All good!


๐Ÿ™‹"Hey, team. I can't install your app on my Windows machine."

Sometimes other teams need to run our app locally for testing and such things. It turned out that our new client didn't support Windows at all.

Wait a minute, isn't Node.js supported in all major platforms? Well, yes. But this one is building a Native Node.js AddOn using bash scripts. Windows don't have that.

๐Ÿคฆ‍♂️

Even though it wasn't my fault only, all I could think of is that it was my fault. I have ruined it for the teams at this workplace.

What else to do than try fix it on my spare time? So, I grabbed the source code to try to understand what's going on behind the scenes.

๐Ÿ˜ญ



And that's how my Open Source journey began: triggered by guilt and feeling ashamed.

Coding in the Dark

Late nights and weekends were spent trying to figure out how to solve this huge problem, including C/C++ code, CMake, bash and node-gyp.

One day, I figured it out. I had succeeded in making it work on Mac OS X, Linux and on Windows! The next day - a sunny morning in beautiful Stockholm - I went to the office smiling. Exhausted by the lack of sleep, but happy.

๐Ÿ˜Ž


Do I have to fix everything? ๐Ÿ˜Ÿ

While struggling with the source code, I realized there were more things that should be fixed. By then, I also realized that this Open Source project suffered from similar issues as the one we had replaced earlier: there wasn't really that much activity in the repo.

This is probably a common pattern: you work with something on a daily basis and contribute to Open Source projects. The years go by, you drift way from the project, simply because you now work with other things unrelated to the project.

How to maintain an Open Source project

Instead of grabbing the next task to spend late nights and weekends with, I decided to learn more about how to maintain Open Source projects.

I found a very useful guideline at GitHub, and I began to add notes about all kinds of things I thought should be done. Other Open Source projects were using labels in a way that I liked, so I added labels like help-wanted and good-first-issue to most of the tasks.

I think labels can help people getting an overview of the tasks and to understand the degree of difficulty.

Every time a question comes up, or someone is filing a bug report, I have replied to most of them within a day or two. I think this is important. The person reporting will know there is someone that has actually received and read the message.

As a result, really nice things have happened. The project has received Pull Requests from people located all over the world. Solving a broad variation of things from typos in the docs, to rewrites of the source code to modern JavaScript.

What's happening today?

Today, I think the project is stable and working well. I don't work nights or weekends. Even though I have moved on and no longer work with ZooKeeper as before, the feedback from users is my main inspiration to keep on working. Also, the amount of users of the client looks pretty good.

It inspires me to do the best I can with the project.



Photo by Justin Lim on Unsplash

Monday, December 28, 2020

The ZooKeeper's New Year's Resolution

(... will probably not be lose weight. Because it has already been fulfilled)




What happened during 2020?


The Node.js ZooKeeper Client - simply called zookeeper - have been released 5 times and the current version is now 4.7.1.

๐ŸŽ‰ Packaging


The source code is packaged differently, making the footprint a lot smaller and the install process simplified. To summarize: the ZooKeeper lost weight.

๐ŸŽ‰ Prebuilds make the install process really, really fast


The install process is faster than ever. If you are a Mac OS X or Windows user, there is no longer any need to build an AddOn during the install process. Everything is already included in the package, thanks to a cool tool called prebuildify. Linux user? Don't worry, the installer will quickly build a Native Node.js AddOn for the Linux flavor you are running. During 2021 we will probably include prebuilds for the most common Linux flavors.

๐ŸŽ‰ TypeScript


TypeScript declarations makes the Developer Experience very nice. With type hints, function parameter docs and autocomplete.



๐ŸŽ‰ async zookeeper


Promise based functions are the default.
await client.create(path, data, constants.ZOO_EPHEMERAL);

๐ŸŽ‰ Same client for all platforms


The same version of the C Client source code is now used in Linux, Mac OS X and Windows 10 to build a Node.js native AddOn. Before that, there was an issue with Windows clients connecting to a ZooKeeper server v3.4.8 that made us add a workaround - reverting to a really old client version for Windows users. But that is now solved and all platforms share the same set of modern features. Yay!

๐Ÿš€ What's next?


Adding support for the new node types introduced in Apache ZooKeeper server v3.5.5:
Container and TTL. The features will probably be released during January 2021.

๐ŸŽ‰ Happy New Year! ๐ŸŽ‰

Sunday, March 10, 2019

Updates to the Apache ZooKeeper Node.js client

At last, Node.js is platform independent again!

Eh, what?

Apache Zookeeper?
From the docs: "... ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C. ..."
(the docs)

A Node.js Zookeeper Client
You will find several Node.js implementations when browsing on npmjs.org, and one of them is simply called: zookeeper. Recently, I was added to the list of maintainers of the project, hosted on GitHub.

Background
The project got my attention when we at work were looking for a new ZooKeeper client, because the API of the current one we were using had gone out of date. We were all developing on Mac OS X, and pushed changes to a CI using a Linux docker container. Implementing the new client seemed like a breeze.

Until a developer in a peer team came by our spot at the office:
- Hey, I cloned the app repo and followed your setup guide. But I get errors when installing packages. I think it's the "zookeeper" dependency that fails. Do you know what's wrong?

We quickly learned it was because Windows. We realized that the entire build process - compiling C code to a native Node.js add-on on the fly - used non-Windows compliant features like make, bash and gcc.

Oh sh*t. How can this be solved?
What about bypassing it with Docker? I've learned that Docker on Windows can be painful, because of access right issues between a Windows host and Linux in a container. Windows Subsystem for Linux (WSL) would probably work after some tweaking, but adding WSL as a requirement to run (or develop) an app is ... well ... a bit depressing.

How about fixing the root cause of the problem, the source code? I did choose that path and have spent a lot of time learning about Node.js add-ons, (a little bit of) C and C++, Windows equivalents to bash commands, node-gyp, and how to compile the Apache Zookeeper C Client from source.

Months later, I hit the "Create Pull Request" button. Shortly after, the changes were merged and an updated npm package was published.

Besides supporting Windows, the zookeeper npm package also builds on Node.js 10. Previous versions failed, because the use of deprecated V8 features.

Details of recent changes can be found in the Changelog.

Contribute to the Open Source project
If you are interested in contributing, check out the issues tab at the GitHub repo. Currently there are some issues labeled with "good first issue", go ahead and grab one!