Sunday, February 18, 2024

Python Monorepo Visualization

What's in a code repository? Usually you'll find the source code, some configuration and the deployment infrastructure - basically the things needed to develop & deploy something. It might be a service, an app or a library. A Monorepo contains the same things, but for more than one artifact. In there, you will find the code, configurations and infrastructure for several services, apps or libraries.

The main use case for a Monorepo is to share code and configuration between the artifacts (let's call them projects).

These things have to be simple

Sharing code can be difficult. Repos can be out of date. A Monorepo can be overwhelming. With or without a Monorepo, the most common way of sharing code is to package them as libraries that the projects can add as external dependencies. But managing different versions and keeping the projects up-to-date could lead to unexpected and unwanted extra work. Some Monorepos solve this by using symlinks to share code, or custom scripts for copying things into the individual projects during deployment.

Doing that can be messy, I've seen it myself. I was once part of a team that migrated away from a horrible Monorepo, into several smaller single-repo microservices. The tradeoffs: source code spread out in repos with an almost identical structure. Almost is the key here. Also, code and config duplications.

These tradeoffs have a negative impact on the Developer Experience.

The Polylith Architecture has a different take on organizing and sharing code, with a nice developer experience. These things have to be simple. Writing code should be fun. Polylith is Open Source, by the way.

The most basic type of visualization

In a Polylith workspace, the source code lives in two folders named bases and components. The entry points are put in the bases folder, all other code in the components folder. At first look, this might seem very different from a mainstream Python single-project structure. But it isn't really that different. Many Python projects are using a src layout, or have a root folder with the same name as the app itself. At the top, there's probably an entry point named something like app.py or maybe main.py? In Polylith, that one would be put in the bases folder. The rest of the code would be placed in the components folder.

  components/
     .../
       auth
       db
       kafka
       logging
       reporting
       ...
  

You are encouraged to keep the components folder simple, and rather put logically grouped modules (i.e. namespace packages) in separate components than nested structures. This will make code sharing more straightforward than having a folder structure with packages and sub-packages. It is also less risk of code duplication with this kind of structure, because the code isn't hidden in a complex folder structure. As a side effect, you will have a nice overview over the available features: the names of the folders will tell what they do and what's available for reuse. A folder view like this is surprisingly useful.

Visualize with the poly tool

Yes, looking at a folder structure is useful, but you would need to navigate the actual source code to figure out where it is used and which dependencies that are used in there. Along with the Polylith Architecture there is tooling support. For Python, you can use the tooling together with Poetry, Hatch, PDM or Rye.

The poly info command, an overview of code and projects in the Monorepo.

Besides providing commands for creating bases, components and projects there are useful visualization features. The most straightforward visualization is probably poly info. Here, you will get an overview of all the bricks (the logically grouped Python modules, living in the bases and components folders), the different projects in the Workspace and also in which projects the bricks are added.

Third-party libraries & usages

There's a command called poly libs that will display the third-party dependencies that are used in the Workspace (yes, that's what the contents of the Monorepo is called in Polylith). It will display libraries and the usages on a brick-level. In Polylith, a brick is the thing that you share across projects. Bricks are the building blocks of this architecture.

The poly libs command, displaying the third-party dependencies and where they are used.

The building blocks and how they depend on each other

A new thing in the Python tooling is the command called poly deps. It displays the bricks and how they depend on each other. You can choose to display an overview of the entire Workspace, or for an individual project. This kind of view can be helpful when reasoning about code and how to combine bricks into features. Or inspire a team to simplify things and refactor: should we extract code from this brick into a new one here maybe?

A closer look at the bricks used in a project with poly deps.

You can inspect a single brick to visualize the dependencies: where it is used, and what other bricks it uses.

A zoomed-in view, to inspect the usages of a specific brick.

Export the visualizations

The output from these commands is very easy to copy-and-paste into Documentation, a Pull Request or even Slack messages.

poly deps | pbcopy

📚 Docs, examples and videos

Have a look at the the Polylith documentation for more information about getting started. You will also find examples, articles and videos there for a quick start.



Top image made with AI (DALL-E) and manual additions by a Human (me)

Thursday, January 25, 2024

Simple & Developer-friendly Python Monorepos

🎉 Announcing new features 🎉

Polylith is about keeping Monorepos Simple & Developer-friendly. Today, the Python tools for the Polylith Architecture has support for Poetry, Hatch and PDM - three popular Packaging & Dependency Management tools in the Python community.

In addition to the already existing Poetry plugin that adds tooling support for Polylith, there is also a brand new command line tool available. The CLI has support for both Hatch and PDM. You can also use it for Poetry setups (but the recommended way is to use the dedicated Poetry plugin as before).

To summarize: it is now possible to use the Simple & Developer-friendly Monorepo Architecture of Polylith with many different kinds of Python projects out there.

"Hatch is a modern, extensible Python project manager."
From the Hatch website

🐍 Hatch

To make the tooling fully support Hatch, there is a Hatch build hook plugin to use - hatch-polylith-bricks - that will make Hatch aware of a Polylith Workspace. Hatch has a nice and well thought-through Plugin system. Just add the hook to the build configuration. Nice and simple! The Polylith tool will add the config for you when creating new projects:

[build-system]
requires = ["hatchling", "hatch-polylith-bricks"]
build-backend = "hatchling.build"
"PDM, as described, is a modern Python package and dependency manager supporting the latest PEP standards. But it is more than a package manager. It boosts your development workflow in various aspects."
From the PDM website

🐍 PDM

Just as with Hatch, there are build hooks available to make PDM aware of the Polylith Workspace. Writing hooks for PDM was really simple and I really like the way it is developed. Great job, PDM developers! There is a workspace build hook - pdm-polylith-workspace - and a projects build hook - pdm-polylith-bricks - to make PDM and the Polylith tooling work well together.

This is added to the workspace build-system section pyproject.toml:

[build-system]
requires = ["pdm-backend", "pdm-polylith-workspace"]
build-backend = "pdm.backend"

And the plugin for projects.
This will be added by the poly create project command for you.

[build-system]
requires = ["pdm-backend", "pdm-polylith-bricks”]
build-backend = "pdm.backend"
"Python packaging and dependency management made easy."
From the Poetry website

🐍 Poetry

For Poetry, just as before, add or update these two plugins and you're ready to go!

poetry self add poetry-multiproject-plugin
poetry self add poetry-polylith-plugin

📚 Docs, examples and videos

Have a look at the the Polylith documentation for more information about getting started. You will also find examples, articles and videos there for a quick start. I'm really excited about the new capabilities of the tooling and hope it will be useful for Teams in the Python Community!



Top photo made with AI (DALL-E) and manual additions by a Human (me)

Tuesday, December 19, 2023

The Lost Balkan Tapes: a Christmas story

A couple of days ago, a nice thing happened that has made me very happy. It happened almost by accident. At work, we have a #music channel on Slack and some of us frequently share music that we like and recommend to the folks in the company. I really like those kind of work-but-not-related-to-the-actual-work kind of things. It helps developing a friendly Organizational culture, and it is also a simple way to get to know each other better.

Background

I was born and raised in the Stockholm area of Sweden, but my dad and his parents (my grandparents) came to Sweden in the late 1960s from former Yugoslavia. They, like many from the Balkans, Finland, Italy and Greece, got job offerings from Swedish companies and they decided to give it a try. They began working with manufacturing chocolate here in Stockholm. I don't know that much about their life over there before Sweden, other than the stories I've heard many times when growing up.

One of my favorite stories is about my Grandfather and when he escaped from some kind of prison camp, set up by the World War II Occupation of former Yugoslavia. He was only a teenager back then, but managed to run away from the guards into a forest - and then catch a passing train, on its way away from the camp. In my imagination, the train was moving fast and he jumped on it in the same way Indiana Jones would do. I remember him telling me and my brother about going undercover, calling himself Ivan Something-something when the train conductor asked who that kid from nowhere was. I loved that story and wanted to hear it over and over again.

Another favorite story was about my Grandmother. She used to be a singer in the 1950s, early 1960s and performed all over the country. I remember the day she told me about the famous people she used to sing for, such as the Ethiopian Emperor Haile Selassie! When I grew up, Reggae and the Rastafari culture was a big thing among us kids in the suburbs, and I certainly knew about Haile Sellasie. My grandma has not only met him, but also has sung for him! Wow. As I understood it, she was popular and during a period of time she was often hired to sing when the officials of the Country expected a visit from abroad.

I have heard only a few songs on cassette when we hang out in the apartment in Fisksätra (a concrete suburb in the Stockholm area that I have spent a lot of time in). I remember the music sounding quite good. But honestly, as a kid and later a young adult, I wasn't really that much of a fan of Balkan Folk Music. Still very cool to hear her sing. She occasionally sang some songs for us too, but time & smoking cigarettes had made her voice different than from the recording of those cassette tapes.

Years later, with both grandma and grandpa no longer around, I thought that the tapes were lost & gone forever. If I recall it correctly, some of it was even accidentally overwritten. Oh no. I made some attempts to find information (and possibly music) online, but failed. I gave up hope of finding anything. This was many years ago.

Today

Back to work. I decide to share some nice Reggae music in the Slack channel. I found the Gratitude Riddim when browsing my online music app, good stuff! The friends at work got inspired and also shared some Jamaican vibes and we had a fun conversation going on there. Then, one person added a picture of Haile Selassie that took me right back to those childhood memories. So, naturally, I told them the Story about my Grandmother and that she has sung for him. But sadly the music is gone, you know. My friend probably got curious, and I believe he just googled her name.

”Wow, very cool to hear about your grandma. Is this her?”
(with a link containing music)

What?

What? Huh? No, wait. That can't be her. Or. Is it? Naw. Gotta be someone else. Then again, how many Folk Music singers from the Balkans named Ikonija Vujic can there be? After a while, I realized that it is in fact my Grandmother! He actually found about 15 songs. All of them beautiful & melancholic Love Songs. Old school Balkan Folk Music with the Tamburitza instrument. Is it the great Janika Balaž playing? This is just me guessing, but according to Wikipedia he lived in the exact same area where my grandparents (and my dad) used to hang out back in the days. They must have met sometime!

I can't think of a better Christmas gift than this and I am forever grateful about these findings. It happened almost randomly, by accident. If I hadn't shared those Reggae songs in Slack, I would still be thinking that the music of my Grandmother was gone. But the Lost Balkan Tapes of Ikonija Vujic have been found again. Thanks to my friend at work and the enthusiast person that has published a huge amount of Balkan Music from the past. Thank you! 🙏

Wednesday, November 15, 2023

A Coding Copilot

When developing software, I spend most of the time thinking & talking about the problems to solve: the what, the why and the how. The thinking-part works best for me when I'm actually away from the desktop, such as walking outdoors or even when grabbing a cup of coffee & a banana by the office kitchen. 🍌

Talking is a really nice thing, we should try that more often in combination with listening to what others have say. Even when not having any human around to chat with, there is always the rubber duck. Speak to it!

Problem Solving

Mob programming takes the Thinking & Talking to the next level. I am, strangely enough, surprised almost every time how great mob and pair programming is for software development. I tend to forget that. It's probably easy to fall back into old familiar behavior.

When figuring out what to try out, the actual typing is done pretty fast for me. A big part of the productivity boost is inspiration and the joy of solving problems with programming. I think this is valid for most creative and problem-solving work.

Everyday practicing

I try hard every day to write clear & concise code. Practicing, learning ... and walking. I really like the Functional approach to programming: separating data from behavior, actions from calculations. I should probably use GitHub Copilot or ChatGPT more, but haven't yet found the need to go all-in (aren't the AI suggestions a bit verbose, or am I doing things wrong?). Those types of in-editor copilots are cool, definitely, and are for sure here to stay. Currently, I find more value in a different kind of automated guidance.

CodeScene

When working on my Open Source Project, the Python tools for the Polylith Architecture, I am guided by tools like SonarCloud and CodeScene. I guess there are some AI involved quite heavily in there too. I especially appreciate the code reviews & even more the CodeScene long-term analysis about the effects of Tech Debt. It's like having a co-pilot, or even a Teacher, guiding you during the development to make you a better software developer than before.

This is different from the kind of review a team of humans usually do when sharing feedback on Pull Requests or even during mob programming sessions.

Even if I sometimes disagree with things the tools report on being a problem, I go ahead and refactor the code anyway. Then, when doing the refactoring, I quickly realize that the code is actually transforming into something simpler and clearer than before. SonarCloud is more into the low level details, and lets me know about code duplications or code smells. We don't want that, no. Better fix that too.

Coding Style

The feedback from these kind of tools is helping me to get closer to writing the type of code I want to write: minimalistic, readable and functional. A coding style that also is a very good fit for the Polylith Architecture, even if that thing isn't about how the code is written. You are encouraged to write in a readable & functional style by the structuring of code and by having all code at your fingertips, ready to experiment with in the REPL or in a Notebook.

CodeScene, SonarCloud and Polylith: all of them helping me in my journey to be a better Developer. Maybe I'll throw in Copilot in there too eventually.

Top Photo by Jamie Templeton on Unsplash

Saturday, September 30, 2023

Software and Projects

How can we simplify things, to add value as early as possible?

Software Projects is a misconception within Software Development. When organizing work into Projects, one might think that there also is a need to estimate the things to be done. Because the Project itself is by definition something that will end, or more likely be delayed.

From my experience, estimation is only adding anxiety and sometimes actually delaying value. I have experienced the same thing with Projects and fixed release dates.

Instead, we could flip it: how can we simplify, to add value for our users as early as possible?

Flip it!

In a team that I was part of some time ago, the Product Managers challenged us by asking those kind of questions. We (the Developers) were at the beginning intimidated, because we had the idea of building the perfect kind of feature and usually we spent time on figuring out & cover as many scenarios as we could imagine. The not-great thing with that approach is that it will likely increase the cost, because it would take longer time and more effort to deliver value for the users of the product.

Think Different

The Product Management folks kept challenging us to think different, and we begun to find ways of delivering smaller parts of the product within days (instead of months). We wouldn't be able to release an entire feature, of course. Far from it. Instead, it would be something - sometimes just a tiny addition - that people could use to simplify their daily work life. They actually did that, we saw it ourselves. There was value added, within a couple of days of development!

Smaller parts

There was no need for us (the Dev Team) to put any Fibonacci numbers or estimating hours to the tasks we identified should be developed. We stopped doing those stressful Big Bang Releases of huge chunks of code. Instead, we found ways to simplify tasks - splitting them out into several smaller parts - to make it possible to push the work to production within days. In my opinion, Projects, Estimation and Deadlines are rarely adding value. Why are there so many Organizations out there still doing that?

Top photo by Patrick Fore on Unsplash