Practice what you Preach

I'm the maintainer of the Python tools for the Polylith Architecture, that adds useful tooling support for teams having their code organized according to the Polylith Architecture. So far in 2026, I have made seven Feature releases (i.e. minor versions) and a bunch of smaller ones (i.e. patches).

One of the features that was added this year is the possibility to inspect the coupling between individual parts of the code. In Polylith, code is logically grouped into something called bricks. For Python, these are what we know as namespace packages within the same repo.

Each package has an __init__.py, and you can define the interface of the package there. This is optional, and it is perfectly valid Python to access the modules within a package directly. In Polylith, a stricter approach is recommended: explicitly define the things that should be exposed outside of the package (aka brick).

      from my_package import my_function

      __all__ = ["my_function"]
    

The new feature is exactly about this: visualize and validate the usage of brick interfaces across the repository. With a simple command, you will find out if code is bypassing the explicit interface of a brick. This is still optional, and as long as it is valid Python you can safely ignore this. There's benefits of having code that interacts via explicit interfaces, such as refactoring - it's very easy to move code - and low-risk of breaking things when changing the code within the brick.

      # the existing command visualizing coupling between bricks
      poly deps

      # the addition, that will inspect the brick interfaces
      poly deps --interface

      # or inspect a single brick
      poly deps --interface --brick the_brick
    

Vizualize coupling, bypassed interfaces or circular dependencies.

What about circular dependencies between bricks?

Remember, a brick is a namespace package and that means that it is on a level above the individual Python modules. It can be valid Python to have two packages depend on each other, if the imports are for different modules in the packages. But this is an indication that something might be wrong with the Design. I would even go so far calling it a Bug. Two packages that import code from each other could be refactored into three, where the shared things are in a separate package. This is how the Polylith tool identifies circular dependencies between the bricks. The same command as for validating interfaces is used for this feature (poly deps) and it will inspect the code in the entire repo, to look for any circular brick dependencies. The interface lookups and the circular dependency checks are both internally using the builtin AST parsing.

Control your Agents

Many teams use agents today during development and knowing that agents can produce a lot of code in a fast pace, these two features are very useful if you want to avoid any unexpected side-effects of importing Python code without clear boundaries. Agents can use the poly tool too, just like when doing linting and testing. Static analysis is a great thing to have in the Agentic Engineering toolbox. The Polylith tool includes agent skills, so your agent will understand how to use it.

Practice what I Preach

I knew from before that the source code of Polylith tool bypassed interfaces in a couple of places, but I thought this is fine since it is valid Python. But I realized that my future self will thank me if these risky parts are fixed now and not later. Also, it is good if I actually practice what I preach being the maintainer of the Polylith tool for Python. So I fixed that. In addition, I also learned that two bricks in the repo had circular dependencies caused by a recently added feature. In this case, it was a circle of several bricks that initially made me worried. But the solution was simple: move the parts that caused the circularity into a different brick. I already had an existing brick that made sense to host the couple of Python functions that caused the circular dependency.

After refactoring: no interface violations, no circular dependencies.

Related posts and resources

• post: An Agent- and Human-friendly Architecture
• repo: the Python tools for the Polylith Architecture
• docs: the Python tools for the Polylith Architecture

^{Top Image generated by Mistral AI with modifications by me.}

An Agent- and Human-Friendly Architecture

"Software Architecture in the agentic era?"

What's needed for an architecture to fit well in the agentic era? Probably many things, but I would say at least simplicity and available context as two very important things to consider. Even if it for sure was not the purpose from the beginning (it was born many years ago), Polylith is about that and is a good choice for Agentic Engineering. That's a nice side-effect of keeping things simple and having all necessary context at your fingertips, which is the guiding star for this software architecture.

A year ago, agent-friendly architectures was not something I had thought about at all. Today, a lot of things has changed. But many ideas about good software remains valid.

If you haven't heard about Polylith, here's an elevator pitch: the main use case is having Microservices in a Monorepo, and share code between the services.

It's an architecture with useful tooling support and great development experience for both humans and agents. All of it Open Source. I'm the maintainer of the tooling support for Python, and teams can use it with their favorite existing tools (such as uv, poetry, pdm, pixi, maturin, hatch). The Polylith tool also includes a set of agent skills, that will add knowledge about Polylith in general and the available commands to your agent. Recently, I also added skills that are specific for migrating a single-repo Python project into a monorepo. I hope that this will help teams to more easily start using this way of structuring code.

The code in a Polylith monorepo is basically building blocks, just like LEGO bricks. Some bricks are small, some are bigger and some are a combination of other bricks. All code lives in a well-structured Monorepo (without symlinks or complex custom setups). The code is organized into smaller reusable parts with clear boundaries between them. The tooling will notify if these boundaries are bypassed or if references are circular.

LEGO bricks and code, what's the connection? In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into a feature. In Polylith, this is called bricks. There are two types of them: components and bases. Components are the main building blocks in Polylith. This is where the business logic lives, the actual features and functionality. Bases are the entry points to your apps or services. A base should ideally be thin, and delegate the business logic to components.

Microservices are great, but the standard setup can introduce new problems: code in many places, duplications, shared code as libraries (that means even more repositories) and different versions of everything. That can be a lot of things to maintain, for agents too.

With a monorepo structured as a Polylith, the agents will have all the necessary context in one single place. It's right there! Agents perform better when things are simple, very much like us humans do. Besides having all needed context, an agent can also use the Polylith tool just like a human would. This will save tokens, and likely speed up the development process even more. The agent skills of the tool will tell the agent how and when to use it. Having all code there at your fingertips is also great for Test & REPL Driven Development, making coding both joyful and interactive.

Link to the Python tools for the Polylith Architecture repo.

^{Top Photo by Mitchell Hartley on Unsplash}

Please don't break things

"Does this need to be a breaking change?"

During the years as a Software Developer, I have been part of many teams making a lot of task force style efforts to upgrade third-party dependencies or tools. Far too often it is work that add zero value for us. It adds significant cost, though. As a user of third-party tools, you don't have much choice. Even if you might feel productive as a developer when implementing these forced changes, think of all the other stuff you could do instead to improve your product or platform.

The great tools from the community, most of it Open Source, is something we should be thankful for, and appreciate the efforts made by the people out there. This is a plea to have extra attention when making changes that will affect your users. I am also an Open Source maintainer, and try hard to avoid changes that doesn't add value for the users.

An example from Python: uv

warning: The `tool.uv.dev-dependencies` field (used in `pyproject.toml`) is deprecated and will be removed in a future release; use `dependency-groups.dev` instead

The change itself makes a lot of sense. There's a new PEP standard for how to declare the dependencies only needed for development. Most Package & Dependency management tools out there already had their own implementation of this feature and it is probably a good thing to use the standard. From a user perspective, this only means that we need to make changes in all our Python projects. My suggestion is: why not support both options?

Maintainability vs Value

From a tooling developer perspective it is understandable that you don't want to maintain several ways of solving a problem. What about the Developer Experience of all the users of the tool out there? Imagine the teams maintaining many projects with 10, 40 or even 100 different Microservices and libraries. Each one in its own git repo.

Another Python example: Pydantic

The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0.

I like Pydantic, it is a very useful tool with great features. The 2.0 release also came with significant performance improvements. But this change doesn't make sense to me. I understand that it probably fits better within the domain of Pydantic itself.

Does this have to be a breaking change? I would suggest to have the both alternatives there. Yes, it might be a little bit more of maintenance for you as a library developer. More importantly: your users can focus on adding value into their products instead of this mostly zero-added-value work.

Example three: SQLAlchemy

MovedIn20Warning: The ``declarative_base()`` function is now available as sqlalchemy.orm.declarative_base(). (deprecated since: 2.0)

This is probably also correct, from an internal SQLAchemy domain perspective. From a user perspective, this only means that we need to change a lot of existing code. The cost of having the code overlapping in two namespaces is probably low.

I am also a maintainer of tools, and I've also made mistakes, or design choices that turned out to be not that great later on. But I also actively have made the decision to not force users to make the kind of changes that are described in this post. I don't want to break things for users of the tool because of design choices made in the past.

Should the users change their workflows?

The main thing I work on today is Python tooling for Polylith, that is a Monorepo architecture. The changes introduced by uv, Pydantic and SQLAlchemy actually isn't that much work for the developer teams using Polylith today. You'll have only one place in the source code where these changes are needed. This setup is robust, and is ready for any unexpected breaking changes in the tools that are used. Sounds nice, doesn't it?

^{Top Photo by generated by Dall-E, prompted and modified afterwards by me.}

Pants & Polylith

But who is Luke and who is R2?

"Pants is a fast, scalable, user-friendly build system for codebases of all sizes"

"Polylith helps us build simple, maintainable, testable, and scalable backend systems"

Can we use both? I have tried that out, and here's my notes.

Why?

Because The Developer Experience.

Developer Experience is important, but what does that mean? For me, it is about keeping things simple. The ability to write, try out and reuse code without any context switching. By using one single setup for the REPL and the IDE, you will have everything at your fingertips.

The Polylith Architecture solves this by organizing code into smaller building blocks, or bricks, and separating code from the project-specific configurations. You have all the bricks and configs available in a Monorepo. For Python development, you create one single virtual environment for all your code and dependencies.

There is also tooling support for Polylith that is useful for visualizing the contents of the Monorepo, and for validating the setup. If you already are into Pantsbuild, the Polylith Architecture might be the missing Lego bricks you want to add for a great Developer Experience.

Powerful builds with Pants

Pantsbuild is a powerful build system. The pants tool resolves all the dependencies (by inspecting the source code itself), runs the tests and creates distributions in isolation. The tool also support the common Python tasks such as linting, type checking and formatting. It also has support for creating virtual environments.

Dude, where's my virtual environment?

In the Python Community, there is a convention to name the virtual environment in a certain way, usually .venv, and creating it at the Project root (this will also likely work well with the defaults of your IDE).

The virtual environment created by Pants is placed in a dists folder, and further in a Pants-specific folder structure. I found that the created virtual environment doesn't seem to include custom source paths (I guess that would be what Pants call roots).

Custom source paths is important for an IDE to locate the Python source code. Maybe there are built-in ways in Pantsbuild to solve that already? Package management tools like Poetry, Hatch and PDM have support for configuring custom source paths in the pyproject.toml and also creating virtual environments according to the Python Community conventions.

^{Note: If you are a PyCharm user, you can mark a folder as a source root manually and it will keep that information in a cache (probably a .pth file).}

Example code and custom scripts

I have created an example repository, a monorepo using Pantsbuild and Polylith. You will find Python code and configurations according to the Polylith Architecture and the Pantsbuild configurations making it possible to use both tools. In the example repo I have added a script that adds source paths, based on the output from the pants roots command, to the virtual environment created by Pantsbuild. This is accomplished by adding a .pth file to the site_packages folder. For convenience, the script will also create a symlink to a .venv folder at the root of the repo.

Having the virtual environment properly setup, you can use the REPL (my favorite is IPython) with full access to the entire code base:

source .venv/bin/activate
ipython

With an activated virtual environment, you can also use all of the Polylith commands:

poly create
poly info
poly libs
poly deps
poly diff
poly check
poly sync

Pants & Polylith

Pantsbuild has a different take on building and packaging artifacts compared to other tools I've used. It has support for several languages and setups. Some features overlap with what's available in the well-known tooling in the Python community, such as Poetry. Some parts diverge from the common conventions.

Polylith has a different take on sharing code, and also have some overlapping features. Polylith is a Monorepo Architecture, with tooling support for visualizing the Monorepo. From what I've learned so far, the checks and code inspection features are the things you will find in both Pants and Polylith.

Pants operate on a file level. Polylith on the bricks level.

My gut feeling after learning about it and by experimenting, is that Pantsbuild and Polylith shares the same basic vision of software development in general and I have found them working really well together. There are some things I would like to have been a better fit, such as when selecting contents of the Pants-specific BUILD files vs the content in the project-specific pyproject.toml files.

Maybe I should develop a Pants Polylith plugin to fix that part. 🤔
How does that sound to you?

Resources

• The Python tools for the Polylith Architecture
• The Pantsbuild website

^{Top Photo by Studbee on Unsplash}

Write Less Code, You Must

An aspect of Python Software Development that is often overlooked, is Architecture (or Design) at the namespace, modules & functions level. My thoughts on Software Development in general is that it is important to try hard writing code that is Simple, and Easy to move from one place to another.

When having code written like this, it becomes less important if a feature was added in Service X, but a better fit would be Service Y when looking at it from a high-level Architectural perspective. All you need to do is move the code to the proper place, and you're all good. However, this will require that the actual code is moveable: i.e. having the features logically separated into functions, modules and namespace packages.

Less Problems

There's a lot of different opinions about this, naturally. I've seen it in in several public Python forums, and been surprised about the reactions about Python with (too) few lines of code in it. How is it even possible having too little of code?

My take on this in general is Less code is Less Problems.

An example

def my_something_function():
    # Validation
    
    # if valid 
    # else do something
    ... python code here

    # Checking

    # if this 
    # elif that
    # elif not this or not that
    # else do_something
    ... python code here

    # Data transformation

    # for each thing in the things
    #    do a network call and append to a list
    ... python code here

    # Yay, done
    return the_result

This type of function - when all of those things are processed within the function body - is not very testable. A unit test would likely need a bunch of mocking, patching and additional boilerplate test data code. Especially when there are network calls involved.

My approach on refactoring the code above would be to first identify the different tasks within this controller type of function, and begin by extracting each task into separate functions. Ideally these would be pure functions, accepting input and returning output.

At first, I would put the functions within the same module, close to at hand. Quite quickly, the original function has become a whole lot more testable, because the extracted functions can now easily be patched (my preference is using pytest monkeypatch). This approach would be my interpretation of developing software towards a clean code ideal. There is no need for a Dependency Injection framework or any unnecessary complex OOP-style hierarchy to accomplish it.

In addition to testability, the Python code becomes runnable and REPL-friendly. You can now refactor, develop and test-run the individual functions in the REPL. This is a very fast workflow for a developer. Read more about REPL Driven Development in Python here.

With the features living in separate isolated functions, you will likely begin to identify patterns:

"- Hey, this part does this specific thing & could be put in that namespace"

When moving code into a namespace package, the functions become reusable. Other parts of the application - or, if you have a Monorepo containing several services - can now use one and the same source code. The same rows of code, located in a single place of the repo. You will likely structure the repo with many namespace packages, each one containing one or a couple of modules with functions that ideally do one thing. It kind of sounds like the Unix philosophy, doesn't it?

This is how I try to write code on a daily basis, at work and when developing Open Source things. I use tools like SonarCloud and CodeScene to help me keep going in this direction. I've written about that before. The Open source code that I focus on these days (Polylith) has 0% Code Duplications, 0% Code Smells and about a 9.96 long-term Quality Code Scoring. The 0.04 that is left has been an active decision by me and is because of endpoints having 5+ input arguments. It makes sense for me to keep it like that there, but not in functions within the app itself where an options object is a better choice.

This aspect of Software Development is, from my point of view, very important. Even more important than the common Microservices/Events/REST/CQRS debates when Architecture is the topic of discussion. This was my Saturday afternoon reflections, and I thank you for reading this post. ☀️

^{Top Photo by Remy Gieling on Unsplash}

Kafka messaging with Python & Polylith

This article isn't about Franz Kafka or his great novella The Metamorphosis, where the main character one day realizes that he has transformed into a Human-size Bug.

It is about a different kind of Kafka: Apache Kafka, with examples on how to get started producing & consuming messages with Python. All this in a Polylith Monorepo (hopefully without any of the bugs from that Franz Kafka novella).

This article can be seen as Part III of a series of posts about Python & Polylith. Previous ones are:

1. GCP Cloud Functions with Python and Polylith
2. Python FastAPI Microservices with Polylith

If you haven't heard about Polylith before, it's about Developer Experience, sharing code and keeping things simple. You will have all your Python code in a Monorepo, and develop things without the common Microservice trade offs. Have a look at the docs: Python tools for the Polylith Architecture

Edit: don't know what Kafka is? Have a look at the official Apache Kafka quickstart.

I will use the confluent-kafka library and have read up on the Confluent Getting Started Guide about writing message Producers & Consumers.

The Polylith Architecture encourages you to build features step-by-step, and you can choose from where to begin. I have an idea about producing events with Kafka when items have been stored or updated in a database, but how to actually solve it is a bit vague at the moment. What I do know is that I need a function that will produce a message based on input. So I'll begin there.

All code in a Polylith Workspace is referred to as bricks (just like when building with LEGO). I'll go ahead and create a Kafka brick. I am going to use the Python tooling for Polylith to create the brick.

Note: I already have a Workspace prepared, have a look at the docs for how to set up a Polylith Workspace. Full example at: python-polylith-example

The poly tool has created a kafka Python package, and placed it in the components folder. It lives in a top namespace that is used for all the bricks in this Polylith Workspace. I have set it to example here, but you would probably want an organizational name or similar as your top namespace.

bases/
components/
  example/ 
    kafka/
	  __init__.py
	  core.py

There's two types of bricks in Polylith: components and bases. A component is where you write the actual implementation of something. A base is the entry point of an app or service, such as the entry point(s) of a FastAPI microservice or the main function of a CLI. In short: a base is a thin layer between the outside world and the components (containing the features). I will develop the base for my new Kafka feature in a moment.

A Producer and a Consumer

For this example kafka component, I will use code from the Confluent Python guide (with a little bit of refactoring).

def produce(topic: str, key: str, value: str):
    producer = get_producer()

    producer.produce(topic, value, key, callback=_acked)

    producer.poll(10000)
    producer.flush()

Full example at: python-polylith-example

I'll go ahead and write a message consumer while I'm at it, and decide to also put the Consumer within the kafka component.

def consume(topic: str, callback: Callable):
    consumer = get_consumer()

    consumer.subscribe([topic])

    try:
        while True:
            msg = consumer.poll(1.0)

            if msg is None:
                continue

            if msg.error():
                logger.error(msg.error())
            else:
                topic, key, value = parse(msg)
                callback(topic, key, value)
    except KeyboardInterrupt:
        pass
    finally:
        consumer.close()

The kafka component now looks like this after some additional coding & refactoring:

kafka/
  __init__.py
  consumer.py
  core.py
  parser.py
  producer.py

Running a Kafka server locally

I continue following along with the Confluent guide to run Kafka locally, and have added a Docker Compose file. I am storing that one in the development folder of the Polylith workspace.

development/
  kafka/
    docker-compose.yml

I can now try out the Producer and Consumer in a REPL, making sure messages are correctly sent & received without any Kafkaesque situations (👴 🥁).

Producing a message

I already have a messaging API in my python-polylith-example repo, with endpoints for creating, reading, updating and deleting data. This is the acual service that I want to extend with Kafka messaging abilities. The service is built with FastAPI and the endpoints are found in a base.

base/
  example/
    message_api/
       __init__.py
       core.py

@app.post("/message/", response_model=int)
def create(content: str = Body()):
    return message.create(content)

I'll continue the development, by adding the newly created kafka component. While developing, I realize that I need to transform the data into simple data structures - and remember that I already have a component that can be used here. This is where Polylith really shines: developing these kind of smaller bricks makes it easy to re-use them in other places - just by importing them.

Consuming messages

I have the kafka component with a consumer in it, and now is the time when I create a new base: the entry point for my kafka consumer.

This is the code I add to the base. Note the re-use of another already existing Polylith component (the log).

from example import kafka, log

logger = log.get_logger("Consumer-app-logger")


def parse_message(topic: str, key: str, value: str):
    logger.info(f"Consumed message with topic={topic}, key={key}, value={value}")


def main():
    topic = "message"
    kafka.consumer.consume(topic, parse_message)

Adding a project

I now have all code needed for this feature. What is left is to add the infrastructure for it, the actual deployable artifact. This command will create a new entry in the projects folder.

projects/
  consumer_project/
    pyproject.toml

I'm adding the dependencies and needed packages to the project-specific pyproject.toml file. But I am lazy, and will only add the base in the packages section - and then run poetry poly sync. It will add all needed bricks for this project. The poly tool has some magic in it, yes.

When deploying, just use the build-project command to package it properly without any relative paths, and use the built wheel to deploy it where you want it running. That's all!

In this article, I have written about the usage of Polylith when developing features and services, and Kafka messaging in specific. Adding new features & re-using existing code is a simple thing when working in a Polylith workspace. Don't hesitate to reach out if you have feedback or questions.

Additional info

Full example at: python-polylith-example
Docs about Polylith: Python tools for the Polylith Architecture

^{Top photo by Thomas Peham on Unsplash}

Dad Jokes & Python Developer Experience

"What do you call a Frenchman wearing sandals?"

"Philippe Flop." 🥁 👴 😆

Developer Experience, what does that even mean?

For me, a huge part is to quickly being able to write, try out and experiment with code. In Python, that could be about having the dependencies and paths already set up, to easily run snippets in a REPL or in the IDE, and without it crashing because of some missing dependencies or configs.

Testable code

When I develop a new feature, I usually want to run & evaluate the code very early in the process. Maybe I'm working on parsing something, transforming or filtering out data: that's when I want to evaluate the code, while writing and figuring out how to solve the actual problem. I do this to verify the output and get ideas about how to refactor the code. I practice a thing called REPL Driven Development, and have written about it before. The workflow is a form of TDD and the refactoring part comes natural with this kind of workflow.

Reusable & Composable code

Another important thing for me is to have already existing code nearby and available to use again. I usually think of Python code as building blocks, like LEGO, and try to have that in mind when writing new functions and adding features. I have learned that from functional programming and it is helping me to solve problems using a step-by-step approach.

With building blocks, code is composable and is ready for re-use from start.

"What did Yoda say when he saw himself in 4K?"

"HDMI." 🥁 👴 😆

Delay Design Decisions

I don't think it is that important to put code in the right place from start (such as what kind of service or app, domain, repo or a folder). At least it isn't as important as writing code that can easily be moved from one place to another. Very related to the building blocks & LEGO way, mentioned above. This approach will likely lead to less wasted time & less design-upfront planning. It is easier to figure it out the proper place to put the code eventually, while iterating and developing. In my opinion, it is perfectly alright to rename or move around code while developing. It shouldn't be a difficult thing to do that.

I usually like to dive into a feature and start coding, rather than starting off by deciding name of a repo, or what type of service the feature should exist in. These things can be left to decide later, when learning more about what to actually build.

"Where do dads store their dad jokes?"

"In the Dadabase." 🥁 👴 😆

A Developer Friendly Architecture

The Polylith Architecture targets all of these things. During the development you have all code available for experimentation and you compose already existing code with new when building features. All code in a Polylith workspace is referenced as bricks, and you use them just as when building something with LEGO. Pick the ones you need along with writing new ones, and combine them into features. A brick can be used in several apps, services or projects as it is called in Polylith.

During development, you have one single virtual environment, with all paths and dependencies already set up. You also have a separate section especially made for trying out and experimenting. It is very similar to the scratch files in PyCharm, but they are versioned and included in the repo. These ones are very useful for evaluating code.

You can immediately dive in to writing code and push the decision about deployment into the future if you like. Add the project to the Polylith workspace, using a simple command, when you are ready for it.

A Dad Joke microservice

To get some insights in how Polylith and the Python tooling changes the way you develop and improves the Developer Experience, I have recorded an improvised video with live coding. Phew, it is difficult to speak & code at the same time. 😅

In this video, I'm taking the first steps into developing some kind of Dad Joke Service and use the Development Environment of Polylith to figure out how to build it.

^{Top photo by Toa Heftiba on Unsplash}

The last Python Architecture you will ever need?

Should features live in several independent Microservices, or is a Monolith the way to go? What kind of setup is the best one for the code in a repo? Organizing is difficult.

"... I'll just put it in a utils folder for now ..."

There's a thing called Polylith. I've written about different aspects of it before. Polylith is an architecture (and a tool) focusing on the Developer and Deployment experience. It is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk.

The Polylith Architecture is a fresh take on how to share code, and offers a nice and simple solution to the Monolith vs Microservice tradeoffs. In addition to that, it is a good fit for functional programming. Some time ago, I got an idea: how about bringing a couple of the good things from there to here? So I developed something that I believe could be useful for many Python teams out there.

I already released a preview in early 2022, it was missing some essential features and the great developer experience wasn't really there yet. Also, I had little knowledge about how to package Python apps & libraries, but have learned a lot since then.

Today, I believe the Python tools for the Polylith Architecture is ready to use in the Real World. You will find version 1 on PyPI.

A couple of Poetry plugins

I have developed it as two different Poetry plugins. One of them - the Multiproject plugin - adds support for workspaces to the popular Poetry tool, by adding a new command called build-project.

The second one - Polylith plugin - adds useful tooling support for the Polylith Architecture itself. With the tooling, you can add components & projects in a simple way and also keep track of what's happening in the workspace.

Have a look at the repo and the docs.

An Architecture well suited for Monorepos

Polylith is using a components-first architecture. The components are building blocks, very much like LEGO. The code is separated from the infrastructure and the building of artifacts. This may sound complicated, but it isn't.

In a way, it is about:

1. thinking about code as LEGO bricks, that can be combined into features.
2. making it easy to reuse code across apps, tools, libraries, serverless functions and services.
3. Keeping it simple.

Have a look at these introductory videos, describing Polylith in Python and the tooling support:

Python with the Polylith Architecture

The Poetry Polylith Plugin

Give it a try! I would love to hear you feedback.

^{Top photo by frank mckenna on Unsplash}

A simple & scalable Python project structure

File & folder structures - there's almost as many different variations as there are code repositories.

One common thing though, is that you'll probably find the utils folder in many of the code repos out there, regardless of programming language. That's the one containing the files that don't fit anywhere in the current project structure. It is also known as the helpers folder.

Organizing, sorting and structuring things is difficult. There's framework specific CLIs and tools that will create a nice setup for you, specialized for the needs of the current framework.

"There should be one-- and preferably only one --obvious way to do it."
^{The Zen of Python}

Is there one folder structure to rule them all? Probably not, but I have tried out a way to organize code that is very simple, framework agnostic and scalable as projects grow.

Structure for simplicity

A good folder structure is one that makes it simple to reuse existing code and makes it easy to add new code. You shouldn't have to worry about these things. The thing I've tried out with success is very much inspired by the Polylith architecture. Polylith is a monorepo thing, but don't worry. This post isn't about monorepos at all, however this one is if you are interested in Python specific ones.

An entry point and a components folder. You won't need much more. Use your favorite dependencies tool, mine is currently Poetry.

It's all about the components

The main takeaway here is to view code as small, reusable components, that ideally does one thing only. A component is not the same thing as a library. So, what's the difference?

A library is a full blown feature. A component can be a single function, or a parser. It can also be a thin wrapper around a third party tool.

"Simple is better than complex."
^{The Zen of Python}

I think the idea of writing components is about changing mindset. It is about how to approach a problem and how to organize the code that solves a problem.

It shouldn't be too difficult to grasp for Python developers, though. For us Python devs, it's an everyday thing to write functions and have them in modules. Another useful thing, probably more common in library development, is to group the modules into packages.

Modules, packages, namespaces ... and components?

In Python, a file is a module. One or more modules in a folder becomes a package. A good thing with this is that the code will be namespaced when importing it. Where does the idea of components fit in here? Well, a component is a package. Simple as that.

"Namespaces are one honking great idea -- let's do more of those!"
^{The Zen of Python}

To make a package easier to understand, you can add an interface. Interfaces are well supported in Python. Specifying the interface of a package in an __init__.py file is a great way to make the intention of the code clearer and easier to grasp. Maybe there's only one function that makes sense to use from the "outside"? That's when to use an interface for your component.

Only the functions that makes sense should be exposed from a component.

Make code reuse easy

When organizing code into simple components, you will quickly discover how easy it is to reuse it. Code is no longer hidden in some utils folder and you no longer need to duplicate existing private helper functions (because the refactoring might break things), if they already are organized as reusable components with clear and simple APIs. I usually think of components as LEGO bricks to select from when building features. You will most likely produce new LEGO bricks of various shapes along the way.

This is code in a "dictionaries" component. The interface (previous picture) will handle the access to it.

Well suited for large apps

At work, we have a couple of Python projects using this kind of structure. One of them is a FastAPI app with an entry point (we named it app.py) containing the public endpoints. The entry point is importing a bunch of components that does the actual work.

The repo contains about 80 Python files. Most of them are grouped into components (in total about 30 components). This particular project is about 3K lines of Python code, but other repos are much smaller with only a handful of components.

Perfect for functional programming

Even though it is not a requirement, organizing code into components fits very well with functional programming. Separating code into data, calculations and actions are well suited for the component thing described here in this post.

Don't forget to keep the components simple, and try to view them as LEGO bricks to be used from anywhere in the app. You'll have fun while doing it too.

^{Top photo by Maureen Sgro on Unsplash}

Just use Dictionaries

A Python dictionary has a simple & well-known API. It is possible to merge data using a nice & minimalistic syntax, without mutating or worrying about state. You're probably not gonna need classes.

Hey, what's wrong with classes? 🤔

From what I've seen in Python, classes often add unnecessary complexity to code. Remember, the Python language is all about keeping it simple.

My impression is that in general, class and instance-based code feels like the proper way of coding: encapsulating data, inheriting features, exposing public methods & writing smart objects. The result is very often a lot of code, weird APIs (each one with its own) and not smart-enough objects. That kind of code quickly tend to be an obstacle. I guess that's when workarounds & hacks usually are added to the app.

Two ways of solving a problem: class-based vs data-oriented.
Less code, less problems.

What about Dataclasses?

Python dataclasses might be a good tradeoff between a heavy object with methods and the simple dictionary. You get typings and autocomplete. You can also create immutable data classes, and that's great! But you might miss the flexibility: the simplicity of merging, picking or omitting data from a dictionary. Letting data flow smoothly through your app.

Hey, what about Pydantic?

That's a really good choice for things like defining FastAPI endpoints. You'll get the typed data as OpenAPI docs for free.

I would as early as possible convert the Pydantic model to a dictionary (using the model.dict() function), or just pick the individual keys and pass those on to the rest of the app. By doing that, the rest of the app is not required to be aware of a domain specific type or some base class, created as workaround for the new set of problems introduced with custom types.

Just data. Keeping it simple.

What about the basic types? 🤔

That is certainly a tradeoff when using dictionaries, the data can be of any type and you will potentially get runtime errors. On the other hand, is that a real problem when using basic data types like dict, bool, str or int? For me, I can't remember that ever has been an issue.

But shouldn't data be private?

Classes are often used to separate public and private functionality. From my experience, explicitly making data and functionality private rarely adds value to a product. I think Python agrees with me about this. By default, all things in a Python module is public. I remember when learning about it, and the authors saying that’s okay because we’re all adults here. I very much liked that perspective!

Do you like Interfaces? 🤩

Yes! Especially when structuring code in modules and packages (more about that in the next section). Using __init__.py is a great way to make the intention of a small package clearer and easier to grasp. Maybe there's only one function that makes sense to use from the outside? That's where the package interface (aka the __init__.py file) feature fits in well.

Python files, modules, packages?

In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into an app. Using a package interface makes sense when structuring code in this way.

Keeping it simple. 😊

I'm finishing off this post with a quote from the past:

“Data is formless, shapeless, like water.
If you put data in a Clojure map, it becomes the map.
You put data in a Python list and it becomes the list.
Now data in a program can flow or it can crash.

Be data, my friend.”

Bruce Lee, 1940 - 1973

ps.

If neither Bruce or I convinced you about the great thing with simple data structures, maybe Rich Hickey will? Don't miss his "just use maps" talk!

ds.

^{Top photo by Syd Wachs on Unsplash}

Software as building blocks

There seems to be an ongoing trend in software development towards using monorepos. This trend is something I have seen especially in the Clojure community.

Polylith - a monorepo architecture

I like the way Polylith solves how to work with code using a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services.

Read more here: Polylith gitbook

The last architecture you will ever need *

From the Polylith docs:

"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."

Okay, backend systems. What about frontend systems? 🤔

I want to Polylith all the things

Is it possible to use the Polylith architecture for a code base that includes web apps? This is something that I have wanted to find out.

Here's my example repo.

In this repo, I’ve added backend Clojure code, frontend ClojureScript and also some glue in between in the form of cljc files. Cljc is Clojure code that can be consumed by both frontend and backend code. This makes it possible to share code across Clojure and ClojureScript, building things just like with LEGO bricks and baseplates.

All the things?

I'll leave the question if ClojureScript and Clojure really should live in the same ecosystem unanswered and hope to get feedback from you. In my example repo, I have put all components in the same place. Should the building blocks be separated somehow, or is it good enough to have both LEGO and DUPLO in the same box? What are your thoughts about it?

Do we still have the Polylith one-REPL experience?

Well, we can have a two-REPLs experience. One for Clojure, that run on top of the JVM, and one for ClojureScript on top of JavaScript. Running both will make REPL driven backend development and Interactive Web Development possible.

You can add and use new ClojureScript components while the REPL is running. Create the namespace and evaluate the function.

There is one thing that I have no solution for (yet). When creating a new ClojureScript component and evaluating the entire namespace at once, I get a compilation error in the ClojureScript REPL: file not on classpath. The ClojureScript REPL have to be restarted to reload new source paths.

But don't worry, you can still evaluate the individual functions in the namespace and they will be loaded as expected in the ClojureScript REPL.

Tooling support

Polylith has a very nice and useful tool to support creating building blocks and to verify the setup. You can create components, bases and projects - as long as it is Clojure. For ClojureScript, you will have to create components manually.

If you are lazy, like me, just create a component with the poly tool as you would for Clojure, and simply rename the file extension to cljs or cljc afterwards.

Editor support?

Your editor most likely has support for running both Clojure and ClojureScript simultaneously. Emacs is my favourite editor. Start the REPLs with the cider-jack-in-clj&cljs command and you're ready to go!

<sup>Two REPLS running (even though the Cider splash message is confusing)

* The quote is from Joakim Tengstrands and Furkan Bayraktars talk about Polylith at the FuncProg Sweden 2020 meetup.

Photo by Amélie Mourichon on Unsplash</sup>