Saturday, July 9, 2022

Just use Dictionaries

A Python dictionary has a simple & well-known API. It is possible to merge data using a nice & minimalistic syntax, without mutating or worrying about state. You're probably not gonna need classes.

Hey, what's wrong with classes? 🤔

From what I've seen in Python, classes often add unnecessary complexity to code. Remember, the Python language is all about keeping it simple.

My impression is that in general, class and instance-based code feels like the proper way of coding: encapsulating data, inheriting features, exposing public methods & writing smart objects. The result is very often a lot of code, weird APIs (each one with its own) and not smart-enough objects. That kind of code quickly tend to be an obstacle. I guess that's when workarounds & hacks usually are added to the app.

Two ways of solving a problem: class-based vs data-oriented.
Less code, less problems.

What about Dataclasses?

Python dataclasses might be a good tradeoff between a heavy object with methods and the simple dictionary. You get typings and autocomplete. You can also create immutable data classes, and that's great! But you might miss the flexibility: the simplicity of merging, picking or omitting data from a dictionary. Letting data flow smoothly through your app.

Hey, what about Pydantic?

That's a really good choice for things like defining FastAPI endpoints. You'll get the typed data as OpenAPI docs for free.

I would as early as possible convert the Pydantic model to a dictionary (using the model.dict() function), or just pick the individual keys and pass those on to the rest of the app. By doing that, the rest of the app is not required to be aware of a domain specific type or some base class, created as workaround for the new set of problems introduced with custom types.

Just data. Keeping it simple.

What about the basic types? 🤔

That is certainly a tradeoff when using dictionaries, the data can be of any type and you will potentially get runtime errors. On the other hand, is that a real problem when using basic data types like dict, bool, str or int? For me, I can't remember that ever has been an issue.

But shouldn't data be private?

Classes are often used to separate public and private functionality. From my experience, explicitly making data and functionality private rarely adds value to a product. I think Python agrees with me about this. By default, all things in a Python module is public. I remember when learning about it, and the authors saying that’s okay because we’re all adults here. I very much liked that perspective!

Do you like Interfaces? 🤩

Yes! Especially when structuring code in modules and packages (more about that in the next section). Using __init__.py is a great way to make the intention of a small package clearer and easier to grasp. Maybe there's only one function that makes sense to use from the outside? That's where the package interface (aka the __init__.py file) feature fits in well.

Python files, modules, packages?

In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into an app. Using a package interface makes sense when structuring code in this way.

Keeping it simple. 😊

I'm finishing off this post with a quote from the past:

“Data is formless, shapeless, like water.
If you put data in a Clojure map, it becomes the map.
You put data in a Python list and it becomes the list.
Now data in a program can flow or it can crash.

Be data, my friend.

Bruce Lee, 1940 - 1973

ps.

If neither Bruce or I convinced you about the great thing with simple data structures, maybe Rich Hickey will? Don't miss his "just use maps" talk!

ds.



Top photo by Syd Wachs on Unsplash

Saturday, May 28, 2022

Hi podman, Bye docker

A 🌧️ Saturday.

This feels like a great day to uninstall Docker Desktop on my Mac, and try to go all-in Podman.

What is Podman?

" ... a daemonless container engine for developing, managing, and running OCI Containers ... "

Podman is an open source project and you can run containers as root, or as a non-privileged user. It supports the Docker API and can be used with tools like docker-compose.

What's wrong with docker?

Nothing, docker is a great tool! Free to use for Open Source. The Desktop app for Mac and Windows will continue to be free for small businesses, but require a license for bigger companies. For some time now, I have wanted to find out what the alternatives are to using docker.

Uninstalling docker

I ran docker system prune to delete all existing containers and images.

Then I uninstalled the desktop client. By selecting the "Troubleshoot" menu option I found an uninstall button and pressed it. It required some research (aka googling) to figure out how and where to find an uninstall option.

As you can see in the video, the docker app is still there even if the Desktop client was uninstalled. I just deleted that one, and then deleted the docker-related files and folders that were still in my file system:

Installing podman

I followed the official install guideline to install podman. In addition to that, also installed the Mac OS X specific helper tool to be able to run docker-compose. I learned that from the print-out displayed when the brew install command was completed.

I can now run containers with podman! Here I'm starting a ZooKeeper server running in a container:

podman run --rm -p 2181:2181 zookeeper

Where's my commands?

But wait, my 🧠 is already wired to use the docker command. Do I have to unlearn things? Podman is compatible with the docker commands already, but you can also create an alias:

alias docker=podman

It is now possible to use the docker command as before!

Installing docker-compose

By installing the podman-mac-helper tool as described before in this post, it is possible to use podman with docker-compose.

You might wonder: didn't I uninstall all docker related things? 🤔
Yes, but the docker-compose tool is available to install separately & you'll find it on Homebrew. 🍻

brew install docker-compose

Quirks & workarounds

When trying all of this out, the podman command worked well for building and running containers. However, I got an error when running the docker-compose up command. That's kind of a dealbreaker, isn't it?

The error: listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 404"

As I understand it (from this conversation on Github), there is something not working as expected when using the latest version(s) of docker-compose and podman.

Luckily, there is a simple workaround to use until the bug is fixed. Prefixing the docker-compose command to disable the buildkit API that is causing the bug. Doing this will make things work very well.


That's it! 👋



Top photo by frank mckenna on Unsplash

Sunday, February 20, 2022

What would slackbot do?

I recently found out about Should I deploy today? and laughed. A humorous way of guidning us developers when we hesitate about deploying code to production. This one is extra funny on a Friday or Saturday. Every now and then, memes about (not) releasing code to production on Fridays appear on my Twitter timeline. There's even T-shirts for sale, with No Deploy Fridays printed on them.

Funny because it's true?

I totally get the idea of minimizing the chance of ruining your or someone else's weekend. There's also a potential risk with postponing & stacking up features. The things we develop are there to add value, why make users wait for it?

Ship it!

Fridays shouldn't be different than any of the other working days, at least when it comes to deploying code to production. Just like on a Tuesday, you should probably think twice about deploying stuff just before you'll call it a day. When a fresh release has started causing errors for users, you might have already checked out from the office with loud music in the headphones, the mind focused on something else and unaware of the mess you've left to friends on call.

Deploy with some slack

It's a good idea to monitor the live environment for a while, making sure the deploy has gone well. Keep an eye on it, while working on the next thing. Prepare to act on alerts, errors appearing in your logs or messages in the company Slack.

Have a browser tab with graphs monitoring memory consumption, CPU usage and response times. For how long and what metrics to monitor is depending on the product. From my experience, being attentive to things like this is mostly good enough. When I call it a day a couple of hours later, I can go ahead and play that loud music in my headphones and relax.

When things go 💥

I haven't fact checked this - but unless a deploy is about database schema changes or to an offline machine, it should be relatively easy to roll back. Most deploys are probably easy to undo. If not, find out why and try making it easier.

I would suggest teams to practice: commit, push, merge, release to production and rollback. You'll probably find infrastructure related things to fix or tweak, maybe there's some automation missing? The team will know what to do when things go boom for real.

I've been fortunate working in environments with the basic automation already in place. We, the autonomous dev team, are the ones responsible for pushing features to production. We do that on a daily basis. Before, we hesitated about the deploy-on-a-Friday thing. Now we have better trust in our ways of working, and this Friday we deployed new features, dev experience improvements and bug fixes. Several small deployments went live, the last one about 3-4 hours before we called it a weekend 😀

Automate it

A good deployment process will stop and let the team know about when things break. The obvious code related issues will be caught this way. So, automation will get you far, but not all the way. That's why we'll need the monitoring. More importantly, releasing small portions of code changes is much safer than the ones with lot of code changes in it. Release small and release often.

What would slackbot do?

These days, when I hesitate if a new feature should be deployed right now or not, I'll just ask slackbot what to do.



Top photo by Ochir-Erdene Oyunmedeg on Unsplash

Wednesday, February 9, 2022

A fresh take on Monorepos in Python

"... What if we had Polylith in Python ..."

For software development in general, it seems to be an ongoing trend towards using Monorepos. You know, organizing code in one big single repo. On the other hand, it seems to be a general trend in going the opposite way too. Developing features into many small repos.

I think that the JavaScript and Python communities in particular, are in favor of the latter approach. Developing features in isolation - in separate repos - and publish versioned libraries to package repositories (such as npm and pypi). That has potential to introduce some headache: versioning, backwards compatibility and keeping dependencies up to date. Probably also duplication of common source code and repeating the deployment infrastructure. You can solve those kind of issues, by using a Monorepo.

🔥 Monorepos

There are already solutions out there on how to develop software in Monorepos. It is my impression that they are mainly about solving the deployment experience. Which is good, because it is a huge problem in a lot of code bases out there. What about the Developer experience?

In the Clojure community, we have a thing called Polylith. It is an architecture, (including a tool) that is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk. The Polylith architecture is focusing on making both the Developer & Deployment experience great. There is also a very nice tool to help you get started with Polylith. Helping you create & keeping track on components, projects and much more.

🤔 So, what is Polylith?

From the official docs:

"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."

A Polylith code-base is structured in a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services. The components live in the same repository; a Polylith monorepo. The Polylith architecture is becoming popular in the Clojure community.

🐍 What about Python?

Again, I find myself copying some existing Python code - a small module creating a "logger" - into a new repo. I think it’s the third time I do this. The logger code is too tiny to even bother packaging as a library, but still very useful in many code projects.

My mind is wandering. I'm daydreaming:

"What if we had Polylith in Python. I would just simply reuse an existing component."

We have Polylith in Python now

Porting Polylith to Python

A couple of weeks ago, I decided to give porting Polylith to Python a try. I quickly realized that there are some fundamental differences between Python and Clojure to be aware of when implementing an architecture (and developing a tool) like this.

Especially when it comes to namespacing and packaging. Python code is based on the concept of modules. Also, Python code is not compiled into a binary.

Short note on modules, packages, namespaces & libraries

In Python, a file is a module. One or more modules in a folder is a package. If you put a package in a folder, it is now namespaced. One or more packages can also be combined and built into a library to be used as a third party dependency. From my perspective, a Polylith component in Python should be a namespaced package, and not a library.

Almost like Poetry

When trying to figure out how to solve modules, paths & packaging problems I found Poetry. It is a tool that makes Python packaging and dependency management really easy. Out of the box, Poetry supports project based dependencies and references only. But the latest version of Poetry - currently in preview - has support for third party plugins. So I developed a plugin that enables the concept of workspaces, making it possible to reference dependencies outside of a project boundary.

This was the first step. A very important step. Now it became possible to build tooling support for a Python implementation of the Polylith architecture. Earlier this week I released a first version of such a tool. It is a new Poetry plugin, containing the very basic tooling support for the Polylith architecture.

poetry-polylith-plugin

This brand new Poetry plugin will help you create a Polylith workspace and add components, bases & projects to it. Poetry itself will handle the building & packaging. It is still in early development, but you should be able to develop apps, serverless functions and services with it.

As of today, I wouldn’t (at least not yet) recommend building libraries with it, mostly because of how the packaging is done in Poetry & the nature of Python modules. I’ll try to figure out a way to solve this in the future.

There’s also a lot missing & left to do compared to the original poly tool. But hey, you gotta start somewhere 😀

Check out the 10 minute video for a quick overview.

Learning about Polylith

I recommend to read the official docs to learn more about what Polylith is (and what it isn’t) - even though the code examples are about Clojure. You can easily bring the ideas into a Python context, now that we have tooling support for it.

My intentions with this is to introduce the Polylith architecture to the Python community, and I very much hope that it will be useful in your daily work as a developer.

Direct link to the Porting Polylith to Python intro video.



Top photo by Katarzyna Kos on Unsplash

Wednesday, December 22, 2021

Simple Component Driven ClojureScript

That Component Driven thing

Component Driven is about developing user interfaces and apps in a certain way. In short, it is about grouping relevant code into components, combining compontents into features, combining features into views (or pages) - and combining the views into an app. I don't think it should be that much about the tools.

Just like Test Driven Development, Component Driven Development encourages us to start small. In a way, it also reminds of LEGO when combining building blocks into something useful.

As your app grows and the coding is done deeper into the app structure, developing user interfaces might be more of a challenge. Finding the spot where a new feature is to be added, might require some clicks and navigation. You know, stuff like logging in, a specific app state - or even waiting for data fetched from a third party service.

This is when the idea of Component Driven Development is really useful. To simplify and speed up the development, we need something simple to host our components. A UI that is isolated from any app specific state or dependencies. A place where a component can be rendered without the need to set things up. Having that, it is possible to develop a component in isolation and test it with different input.

Do I need a tool?

Storybook is a tool that will solve all of this, with a bunch of other really nice things like docs, previews, example usage, variants, code snippets and viewports. But it is possible to try out the Component Driven style of creating user interfaces, without going all-in with a new tool.

Start small

I’m suggesting a very simplistic approach, that will enable rendering components in a short-lived isolated view. You throw it away when done.

Here, I’m using shadow-cljs and have defined a new alias that is targeting a browser. I have also set entry points. A setup almost identical to the web app alias. This one will run in a separate port and the output is compiled to a separate public folder. In that one, I have copied the HTML template from the public folder used by the web app.

Click to enlarge the image, or browse an example gist here.

Keep it Simple

The entry point here is a simplistic and simplified version of the one used in the actual web app. But this entry point does only care about the component you are developing right now. Here is where you render and try out a component during development. This is a temporary place to work Component Driven. It’s almost like when evaluating code in a REPL, but with HTML and compiled JavaScript in a browser.

Click to enlarge the image, or browse an example gist here.

Run both aliases. The App in one browser folder, the component in a different one.

The browser is a powerful tool, with nice developer features.

Take it to the next level

At some point, you will probably want something more long-lived than this very simplistic scratch-style approach. That’s where Storybook fits in!

Component Driven ClojureScript with Storybook

With Storybook, you can focus on styling, different viewports and UI events when developing a single component. You write "stories" (i.e. running components in isolation), including documentation and lot of in-browser interactivity. The stories & docs can also be shared, by deploying it as a separate web app.

Recently, I talked about getting started with Storybook at the re:Clojure 2021 conference.

In this less than 25 minutes talk, I demo how you can use Storybook in your ClojureScript app and work Component Driven.

I hope you’ll like it!

Direct link to the Component Driven ClojureScript with Storybook video.



Top photo by Brett Jordan on Unsplash