A fresh take on Monorepos in Python

"... What if we had Polylith in Python ..."

For software development in general, it seems to be an ongoing trend towards using Monorepos. You know, organizing code in one big single repo. On the other hand, it seems to be a general trend in going the opposite way too. Developing features into many small repos.

I think that the JavaScript and Python communities in particular, are in favor of the latter approach. Developing features in isolation - in separate repos - and publish versioned libraries to package repositories (such as npm and pypi). That has potential to introduce some headache: versioning, backwards compatibility and keeping dependencies up to date. Probably also duplication of common source code and repeating the deployment infrastructure. You can solve those kind of issues, by using a Monorepo.

🔥 Monorepos

There are already solutions out there on how to develop software in Monorepos. It is my impression that they are mainly about solving the deployment experience. Which is good, because it is a huge problem in a lot of code bases out there. What about the Developer experience?

In the Clojure community, we have a thing called Polylith. It is an architecture, (including a tool) that is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk. The Polylith architecture is focusing on making both the Developer & Deployment experience great. There is also a very nice tool to help you get started with Polylith. Helping you create & keeping track on components, projects and much more.

🤔 So, what is Polylith?

From the official docs:

"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."

A Polylith code-base is structured in a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services. The components live in the same repository; a Polylith monorepo. The Polylith architecture is becoming popular in the Clojure community.

🐍 What about Python?

Again, I find myself copying some existing Python code - a small module creating a "logger" - into a new repo. I think it’s the third time I do this. The logger code is too tiny to even bother packaging as a library, but still very useful in many code projects.

My mind is wandering. I'm daydreaming:

"What if we had Polylith in Python. I would just simply reuse an existing component."

✨We have Polylith in Python now✨

Porting Polylith to Python

A couple of weeks ago, I decided to give porting Polylith to Python a try. I quickly realized that there are some fundamental differences between Python and Clojure to be aware of when implementing an architecture (and developing a tool) like this.

Especially when it comes to namespacing and packaging. Python code is based on the concept of modules. Also, Python code is not compiled into a binary.

Short note on modules, packages, namespaces & libraries

In Python, a file is a module. One or more modules in a folder is a package. If you put a package in a folder, it is now namespaced. One or more packages can also be combined and built into a library to be used as a third party dependency. From my perspective, a Polylith component in Python should be a namespaced package, and not a library.

Almost like Poetry

When trying to figure out how to solve modules, paths & packaging problems I found Poetry. It is a tool that makes Python packaging and dependency management really easy. Out of the box, Poetry supports project based dependencies and references only. But the latest version of Poetry - currently in preview - has support for third party plugins. So I developed a plugin that enables the concept of workspaces, making it possible to reference dependencies outside of a project boundary.

This was the first step. A very important step. Now it became possible to build tooling support for a Python implementation of the Polylith architecture. Earlier this week I released a first version of such a tool. It is a new Poetry plugin, containing the very basic tooling support for the Polylith architecture.

poetry-polylith-plugin

This brand new Poetry plugin will help you create a Polylith workspace and add components, bases & projects to it. Poetry itself will handle the building & packaging. It is still in early development, but you should be able to develop apps, serverless functions and services with it.

As of today, I wouldn’t (at least not yet) recommend building libraries with it, mostly because of how the packaging is done in Poetry & the nature of Python modules. I’ll try to figure out a way to solve this in the future.

There’s also a lot missing & left to do compared to the original poly tool. But hey, you gotta start somewhere 😀

Check out the 10 minute video for a quick overview.

Learning about Polylith

I recommend to read the official docs to learn more about what Polylith is (and what it isn’t) - even though the code examples are about Clojure. You can easily bring the ideas into a Python context, now that we have tooling support for it.

My intentions with this is to introduce the Polylith architecture to the Python community, and I very much hope that it will be useful in your daily work as a developer.

Direct link to the Porting Polylith to Python intro video.

^{Top photo by Katarzyna Kos on Unsplash}

Functional Python

All in Functional

In a previous post, I had some concerns about doing too much functional programming in Python and that it might not be Pythonic. But I'm not going worry about that here. I'll be writing Functional Python, with some help from a very nice library called toolz.

I have added example code in this GitHub repo and have also recorded a short video (about 8 minutes long). You'll find it embedded in this post, just scroll down a bit.

Functional in a nutshell

The video is very much inspired by one of my favorite recordings about functional programming in general, and about Clojure in specific: Clojure in a nutshell (Func Prog Sweden), by James Trunk. I have learned a lot from that talk and figured it would be fun to copy the workflow and examples that James uses in his excellent talk.

So, I've been translating Clojure code to Python. You'll also notice in the short video that I'm practicing a thing called REPL Driven Development.

Pipe functions with the toolz.thread_last function.

Composing, piping, chaining, threading

What I've learned so far is that it's a smooth ride to apply functional concepts into a language like Python. After all, functions are first class citizens here. Composing functions in a minimalistic - and readable - way is joyful. Yeah, these things also encourage to aim for simplicity in functions. Keeping functions stateless, single purposed and testable. All of that is really good stuff.

Intuitive, Pythonic & Effective?

Are those one-liners, like that thread_last thing that I use heavily in the video, Pythonic? Well, I guess it depends. As with most things, the features of cool toolz should be used wisely.

Yes, Python should be used wisely.

Remember to run import this in a Python shell now and then to remind yourself about and reflect on the Zen of Python 🧘. Does it say anything at all about OOP, or functional?

Here's an improvised 8 minute recording with live coding functional Python, using toolz:

Direct link to the Functional Python video.

^{Top photo by Charlein Gracia on Unsplash}

A Pythonic Railway?

What is Railway Oriented Programming, and how it can be implemented in Python? Is it even a good fit for Python?

🐍 Python on Rails

I’ve experimented with this Railway thing to find out if it is a good fit. If you haven't already, make sure to watch this excellent talk about Railway Oriented Programming from NDC London by Scott Wlaschin.

As I understand Railway Oriented Programming, it is about adding error handling to code without cluttering the program flow.

The source code in my GitHub repo is an attempt to implement a lightweight, and probably a sloppy, variant of Railway Oriented Programming in Python.

What I'm trying to achieve is to add the functional concepts of Railway Oriented Programming, and try keeping a Pythonic mindset. I believe that means not going all-in functional, but I could be wrong here. 🤔

🚂 Why Railways?

As I understand it, this way of writing code is about adding error handling and still keeping a happy path style in the code. This is done by wrapping - or as I’ve been doing, decorating - functions.

The functions are wrapped (or decorated, as in the examples in my GitHub repo) to catch failures. The output of a failed function call will be the input to the next one. The next function will choose a track based on the input: taking the success track, or the fail track. In general, taking the fail track means bypassing the function and skip to the next one.

By using a two-tracked approach in functions, the error handling will be separated from the program. Functions will be less cluttered with try except error handling and probably also with less of if else flow control clauses. As result, In many cases, this will mean less code in functions.

Less code in a function will probably be easier to understand and reason about. Less branches in a program flow will keep the functions simple. I think less is nice. Yeah, less is more. 😎

Here’s an improvised 5 minute video I’ve made to demo the concepts Railway Oriented Development in Python:

Direct link to the A Pythonic Railway? video.

I would love to hear your feedback about this!

^{Top photo by Alex Paganelli on Unsplash}

NDC Oslo Video: JavaScript in 2017 - You might (not) need a framework

The sessions from NDC Oslo 2017 are online!


I had a good time in Oslo, got new insights and left the Norwegian capital with inspiration. There were many interesting talks and lots of fellow geeks to chat with. I am also glad that so many people joined my talk. Thank you! I hope you have learned something useful.


Here's the recording of my NDC Oslo 2017 talk:

(and here's the code used in the talk)

Click the Youtube "Like" button if you like it! Also, I would very much appreciate your feedback. Please write your feedback at the comments section of the Youtube page (or this blog post) if you come up with ideas, questions, suggestions or things that should be improved.

Thank you NDC Oslo!