FOSDEM 25

FOSDEM, a Conference different from any other I've been to before. I'm very happy for the opportunity to talk, share knowledge and chat with the fellow devs there. I also met old and new friends in Brussels, and learned a little bit more about the nice Belgian beer culture. 😀

The FOSDEM event is free and you don't even register to attend (only speakers do that in a call for speakers process). Just come by the University Campus, located not far from the center of the Inner City.

I travelled towards Belgium with the Night Train from Stockholm, Sweden, and arrived the next morning in Hamburg, Germany. From there, I took the DB ICE Train to Köln/Cologne. At some point, the top speed was about 250 km/h. That's fast! From there, another fast train to Brussels.

The variety of topics and the amount of tracks at FOSDEM was mind blowing, with thousands of participants. The overall vibe was very friendly & laid back. I joined Rust focused talks, JavaScript and UX/Design talks. And, of course, the Python room talks.

My talk was in the Python room and was about Python Monorepos and the Polylith Developer Experience. Everything Python related was on FOSDEM Day 2 and I think my presentation went really well! There was a lot of questions afterwards and I got great feedback from the people attending.

The next day I went back the same route as before. I got a couple extra of hours to spend in Hamburg before onboarding the Night Train back home to Stockholm. A great Weekend Trip!

Here's the recording of my talk from FOSDEM 25:

^{The video was downloaded from fosdem.org.
Licensed under the Creative Commons Attribution 2.0 Belgium Licence.}

Resources

• The FOSDEM 2025 website
• The Python tools for the Polylith Architecture at GitHub

Simple Kubernetes in Python Monorepos

"Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications."
^{(from kubernetes.io)}

Setting up Kubenetes for a set of Microservices can be overwhelming at first sight.

I'm currently learning about K8s and the Ecosystem of tooling around it. One thing that I've found difficult is the actual K8s configuration and how the different parts relate to each other. The YAML syntax is readable, but I also find it hard to understand how to structure it - impossible to edit without having the documentation close at hand. When I first got the opportunity to work with Python code running in Kubernetes, I realized that I have to put extra effort in understanding what's going on in there.

I was a bit overwhelmed by what looked like a lot of repetitive and duplicated configuration. I don't like that. But there's a tool called Kustomize that can solve this, by incrementally constructing configuration objects with snippets of YAML.

Kustomize is about managing Kubernetes objects in a declarative way, by transforming a basic setup with environment-specific transformations. You can replace, merge or add parts of the configuration that is specific for the current environment. It reminds me of how we write reusable Python code in general. The latest version of the Kubernetes CLI - Kubectl - already includes Kustomize. It used to be a separate install.

Microservices

From my experience, the most common way of developing Microservices is to isolate the source code of each service in a separate git repository. Sometimes, shared code is extracted into libraries and put in separate repos. This way of working comes with tradeoffs. With the source code spread out in several repositories, there's a risk of having duplicated source code. Did I mention I don't like duplicated code?

Over time, it is likely that the services will run different versions of tools and dependencies, potentially also different Python versions. From a maintainability and code quality perspective, this can be a challenge.

YAML Duplication

In addition to having the Python code spread out in many repos, a common setup for Kubernetes is to do the same thing: having the service-specific configuration in the same repo as the service source code. I think it makes a lot of sense to have the K8s configuration close to the source code. But with the K8s configuration in separate repos, the tradeoffs are very much the same as for the Python source code.

For the YAML part in specific, it is even likely that the configuration will be duplicated many times across the repos. A lot of boilerplate configuration. This can lead to unnecessary extra work when needing to update something that affects many Microservices.

One solution to the tradeoffs with the source code and the Kubernetes configuration is: Monorepos.

K8s configuration in a Monorepo

A Monorepo is a repository containing source code and multiple deployable artifacts (or projects), i.e. a place where you would have all your Python code and where you would build & package several Microservices from. The purpose of a Monorepo is to simplify code reuse, and to use the same developer tooling setup for all code.

The Polylith Architecture is designed for this kind of workflow (I am the maintainer of the Python tools for the Polylith Architecture).

While learning, struggling and trying out K8s, I wanted to find ways to improve the Configuration Experience by applying the good ideas from the Developer Experience of Polylith. The goal is to make K8s configuration simple and joyful!

Local Development

You can try things out locally with developer tools like Minikube. With Minikube, you will have a local Kubernetes to experiment with, test configurations and to run your containerized microservices. It is possible to dry-run the commands or apply the setup into a local cluster, by using the K8s CLI with the Kustomize configs.

I have added examples of a reusable K8s configuration in the The Python Polylith Example repo. This Monorepo contains different types of projects, such as APIs and event handlers.

The K8s configuration is structured in three sections:

• A basic setup for all types of deployments, i.e. the config that is common for all services.
• Service-type specific setup (API and event handler specific)
• Project-specific setup

All of theses sections have overlays for different environments (such as development, staging and production).

As an alternative, the project-specific configuration could also be placed in the top /kubernets folder.

I can run kubectl apply -k to deploy a project into the Minikube cluster, using the Kustomize configuration. Each section adds things to the configuration that is specific for the actual environment, the service type and the project.

The base, overlays and services are the parts that aren't aware of the project. Those Project-specific things are defined in the project section.

Using a structure like this will make the Kubernetes configuration reusable and with almost no duplications. Changing any common configuration only needs to be done in one place, just as with the Python code - the bricks - in a Polylith Monorepo.

That’s all. I hope you will find the ideas and examples in this post useful.

Resources

• The Python Polylith Example repo
• The Python tools for the Polylith Architecture

^{Top photo by Justus Menke on Unsplash}

Python Monorepo Visualization

What's in a code repository? Usually you'll find the source code, some configuration and the deployment infrastructure - basically the things needed to develop & deploy something. It might be a service, an app or a library. A Monorepo contains the same things, but for more than one artifact. In there, you will find the code, configurations and infrastructure for several services, apps or libraries.

The main use case for a Monorepo is to share code and configuration between the artifacts (let's call them projects).

These things have to be simple

Sharing code can be difficult. Repos can be out of date. A Monorepo can be overwhelming. With or without a Monorepo, the most common way of sharing code is to package them as libraries that the projects can add as external dependencies. But managing different versions and keeping the projects up-to-date could lead to unexpected and unwanted extra work. Some Monorepos solve this by using symlinks to share code, or custom scripts for copying things into the individual projects during deployment.

Doing that can be messy, I've seen it myself. I was once part of a team that migrated away from a horrible Monorepo, into several smaller single-repo microservices. The tradeoffs: source code spread out in repos with an almost identical structure. Almost is the key here. Also, code and config duplications.

These tradeoffs have a negative impact on the Developer Experience.

The Polylith Architecture has a different take on organizing and sharing code, with a nice developer experience. These things have to be simple. Writing code should be fun. Polylith is Open Source, by the way.

The most basic type of visualization

In a Polylith workspace, the source code lives in two folders named bases and components. The entry points are put in the bases folder, all other code in the components folder. At first look, this might seem very different from a mainstream Python single-project structure. But it isn't really that different. Many Python projects are using a src layout, or have a root folder with the same name as the app itself. At the top, there's probably an entry point named something like app.py or maybe main.py? In Polylith, that one would be put in the bases folder. The rest of the code would be placed in the components folder.

  components/
     .../
       auth
       db
       kafka
       logging
       reporting
       ...
  

You are encouraged to keep the components folder simple, and rather put logically grouped modules (i.e. namespace packages) in separate components than nested structures. This will make code sharing more straightforward than having a folder structure with packages and sub-packages. It is also less risk of code duplication with this kind of structure, because the code isn't hidden in a complex folder structure. As a side effect, you will have a nice overview over the available features: the names of the folders will tell what they do and what's available for reuse. A folder view like this is surprisingly useful.

Visualize with the poly tool

Yes, looking at a folder structure is useful, but you would need to navigate the actual source code to figure out where it is used and which dependencies that are used in there. Along with the Polylith Architecture there is tooling support. For Python, you can use the tooling together with Poetry, Hatch, PDM or Rye.

The poly info command, an overview of code and projects in the Monorepo.

Besides providing commands for creating bases, components and projects there are useful visualization features. The most straightforward visualization is probably poly info. Here, you will get an overview of all the bricks (the logically grouped Python modules, living in the bases and components folders), the different projects in the Workspace and also in which projects the bricks are added.

Third-party libraries & usages

There's a command called poly libs that will display the third-party dependencies that are used in the Workspace (yes, that's what the contents of the Monorepo is called in Polylith). It will display libraries and the usages on a brick-level. In Polylith, a brick is the thing that you share across projects. Bricks are the building blocks of this architecture.

The poly libs command, displaying the third-party dependencies and where they are used.

The building blocks and how they depend on each other

A new thing in the Python tooling is the command called poly deps. It displays the bricks and how they depend on each other. You can choose to display an overview of the entire Workspace, or for an individual project. This kind of view can be helpful when reasoning about code and how to combine bricks into features. Or inspire a team to simplify things and refactor: should we extract code from this brick into a new one here maybe?

A closer look at the bricks used in a project with poly deps.

You can inspect a single brick to visualize the dependencies: where it is used, and what other bricks it uses.

A zoomed-in view, to inspect the usages of a specific brick.

Export the visualizations

The output from these commands is very easy to copy-and-paste into Documentation, a Pull Request or even Slack messages.

poly deps | pbcopy

📚 Docs, examples and videos

Have a look at the the Polylith documentation for more information about getting started. You will also find examples, articles and videos there for a quick start.

^{Top image made with AI (DALL-E) and manual additions by a Human (me)}

A fresh take on Monorepos in Python

"... What if we had Polylith in Python ..."

For software development in general, it seems to be an ongoing trend towards using Monorepos. You know, organizing code in one big single repo. On the other hand, it seems to be a general trend in going the opposite way too. Developing features into many small repos.

I think that the JavaScript and Python communities in particular, are in favor of the latter approach. Developing features in isolation - in separate repos - and publish versioned libraries to package repositories (such as npm and pypi). That has potential to introduce some headache: versioning, backwards compatibility and keeping dependencies up to date. Probably also duplication of common source code and repeating the deployment infrastructure. You can solve those kind of issues, by using a Monorepo.

🔥 Monorepos

There are already solutions out there on how to develop software in Monorepos. It is my impression that they are mainly about solving the deployment experience. Which is good, because it is a huge problem in a lot of code bases out there. What about the Developer experience?

In the Clojure community, we have a thing called Polylith. It is an architecture, (including a tool) that is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk. The Polylith architecture is focusing on making both the Developer & Deployment experience great. There is also a very nice tool to help you get started with Polylith. Helping you create & keeping track on components, projects and much more.

🤔 So, what is Polylith?

From the official docs:

"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."

A Polylith code-base is structured in a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services. The components live in the same repository; a Polylith monorepo. The Polylith architecture is becoming popular in the Clojure community.

🐍 What about Python?

Again, I find myself copying some existing Python code - a small module creating a "logger" - into a new repo. I think it’s the third time I do this. The logger code is too tiny to even bother packaging as a library, but still very useful in many code projects.

My mind is wandering. I'm daydreaming:

"What if we had Polylith in Python. I would just simply reuse an existing component."

✨We have Polylith in Python now✨

Porting Polylith to Python

A couple of weeks ago, I decided to give porting Polylith to Python a try. I quickly realized that there are some fundamental differences between Python and Clojure to be aware of when implementing an architecture (and developing a tool) like this.

Especially when it comes to namespacing and packaging. Python code is based on the concept of modules. Also, Python code is not compiled into a binary.

Short note on modules, packages, namespaces & libraries

In Python, a file is a module. One or more modules in a folder is a package. If you put a package in a folder, it is now namespaced. One or more packages can also be combined and built into a library to be used as a third party dependency. From my perspective, a Polylith component in Python should be a namespaced package, and not a library.

Almost like Poetry

When trying to figure out how to solve modules, paths & packaging problems I found Poetry. It is a tool that makes Python packaging and dependency management really easy. Out of the box, Poetry supports project based dependencies and references only. But the latest version of Poetry - currently in preview - has support for third party plugins. So I developed a plugin that enables the concept of workspaces, making it possible to reference dependencies outside of a project boundary.

This was the first step. A very important step. Now it became possible to build tooling support for a Python implementation of the Polylith architecture. Earlier this week I released a first version of such a tool. It is a new Poetry plugin, containing the very basic tooling support for the Polylith architecture.

poetry-polylith-plugin

This brand new Poetry plugin will help you create a Polylith workspace and add components, bases & projects to it. Poetry itself will handle the building & packaging. It is still in early development, but you should be able to develop apps, serverless functions and services with it.

As of today, I wouldn’t (at least not yet) recommend building libraries with it, mostly because of how the packaging is done in Poetry & the nature of Python modules. I’ll try to figure out a way to solve this in the future.

There’s also a lot missing & left to do compared to the original poly tool. But hey, you gotta start somewhere 😀

Check out the 10 minute video for a quick overview.

Learning about Polylith

I recommend to read the official docs to learn more about what Polylith is (and what it isn’t) - even though the code examples are about Clojure. You can easily bring the ideas into a Python context, now that we have tooling support for it.

My intentions with this is to introduce the Polylith architecture to the Python community, and I very much hope that it will be useful in your daily work as a developer.

Direct link to the Porting Polylith to Python intro video.

^{Top photo by Katarzyna Kos on Unsplash}