Dad Jokes & Python Developer Experience

"What do you call a Frenchman wearing sandals?"

"Philippe Flop." 🥁 👴 😆

Developer Experience, what does that even mean?

For me, a huge part is to quickly being able to write, try out and experiment with code. In Python, that could be about having the dependencies and paths already set up, to easily run snippets in a REPL or in the IDE, and without it crashing because of some missing dependencies or configs.

Testable code

When I develop a new feature, I usually want to run & evaluate the code very early in the process. Maybe I'm working on parsing something, transforming or filtering out data: that's when I want to evaluate the code, while writing and figuring out how to solve the actual problem. I do this to verify the output and get ideas about how to refactor the code. I practice a thing called REPL Driven Development, and have written about it before. The workflow is a form of TDD and the refactoring part comes natural with this kind of workflow.

Reusable & Composable code

Another important thing for me is to have already existing code nearby and available to use again. I usually think of Python code as building blocks, like LEGO, and try to have that in mind when writing new functions and adding features. I have learned that from functional programming and it is helping me to solve problems using a step-by-step approach.

With building blocks, code is composable and is ready for re-use from start.

"What did Yoda say when he saw himself in 4K?"

"HDMI." 🥁 👴 😆

Delay Design Decisions

I don't think it is that important to put code in the right place from start (such as what kind of service or app, domain, repo or a folder). At least it isn't as important as writing code that can easily be moved from one place to another. Very related to the building blocks & LEGO way, mentioned above. This approach will likely lead to less wasted time & less design-upfront planning. It is easier to figure it out the proper place to put the code eventually, while iterating and developing. In my opinion, it is perfectly alright to rename or move around code while developing. It shouldn't be a difficult thing to do that.

I usually like to dive into a feature and start coding, rather than starting off by deciding name of a repo, or what type of service the feature should exist in. These things can be left to decide later, when learning more about what to actually build.

"Where do dads store their dad jokes?"

"In the Dadabase." 🥁 👴 😆

A Developer Friendly Architecture

The Polylith Architecture targets all of these things. During the development you have all code available for experimentation and you compose already existing code with new when building features. All code in a Polylith workspace is referenced as bricks, and you use them just as when building something with LEGO. Pick the ones you need along with writing new ones, and combine them into features. A brick can be used in several apps, services or projects as it is called in Polylith.

During development, you have one single virtual environment, with all paths and dependencies already set up. You also have a separate section especially made for trying out and experimenting. It is very similar to the scratch files in PyCharm, but they are versioned and included in the repo. These ones are very useful for evaluating code.

You can immediately dive in to writing code and push the decision about deployment into the future if you like. Add the project to the Polylith workspace, using a simple command, when you are ready for it.

A Dad Joke microservice

To get some insights in how Polylith and the Python tooling changes the way you develop and improves the Developer Experience, I have recorded an improvised video with live coding. Phew, it is difficult to speak & code at the same time. 😅

In this video, I'm taking the first steps into developing some kind of Dad Joke Service and use the Development Environment of Polylith to figure out how to build it.

^{Top photo by Toa Heftiba on Unsplash}

The last Python Architecture you will ever need?

Should features live in several independent Microservices, or is a Monolith the way to go? What kind of setup is the best one for the code in a repo? Organizing is difficult.

"... I'll just put it in a utils folder for now ..."

There's a thing called Polylith. I've written about different aspects of it before. Polylith is an architecture (and a tool) focusing on the Developer and Deployment experience. It is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk.

The Polylith Architecture is a fresh take on how to share code, and offers a nice and simple solution to the Monolith vs Microservice tradeoffs. In addition to that, it is a good fit for functional programming. Some time ago, I got an idea: how about bringing a couple of the good things from there to here? So I developed something that I believe could be useful for many Python teams out there.

I already released a preview in early 2022, it was missing some essential features and the great developer experience wasn't really there yet. Also, I had little knowledge about how to package Python apps & libraries, but have learned a lot since then.

Today, I believe the Python tools for the Polylith Architecture is ready to use in the Real World. You will find version 1 on PyPI.

A couple of Poetry plugins

I have developed it as two different Poetry plugins. One of them - the Multiproject plugin - adds support for workspaces to the popular Poetry tool, by adding a new command called build-project.

The second one - Polylith plugin - adds useful tooling support for the Polylith Architecture itself. With the tooling, you can add components & projects in a simple way and also keep track of what's happening in the workspace.

Have a look at the repo and the docs.

An Architecture well suited for Monorepos

Polylith is using a components-first architecture. The components are building blocks, very much like LEGO. The code is separated from the infrastructure and the building of artifacts. This may sound complicated, but it isn't.

In a way, it is about:

1. thinking about code as LEGO bricks, that can be combined into features.
2. making it easy to reuse code across apps, tools, libraries, serverless functions and services.
3. Keeping it simple.

Have a look at these introductory videos, describing Polylith in Python and the tooling support:

Python with the Polylith Architecture

The Poetry Polylith Plugin

Give it a try! I would love to hear you feedback.

^{Top photo by frank mckenna on Unsplash}

Joyful Python with the REPL

REPL Driven Development is a workflow that makes coding both joyful and interactive. The feedback loop from the REPL is a great thing to have at your fingertips.

"If you can improve just one thing in your software development, make it getting faster feedback."
^{Dave Farley}

Just like Test Driven Development (TDD), it will help you write testable code. I have also noticed a nice side effect from this workflow: REPL Driven Development encourages a functional programming style.

REPL Driven Development is an everyday thing among Clojure developers and doable in Python, but far less known here. I'm working on making it an everyday thing in Python development too.

But what is REPL Driven Development?

What is it?

You evaluate variables, code blocks, functions - or an entire module - and get instant feedback, just by a hitting a key combination in your favorite code editor. There's no reason to leave the IDE for a less featured shell to accomplish all of that. You already have autocomplete, syntax highlighting and the color theme set up in your editor. Why not use that, instead of a shell?

Evaluate code and get feedback, without leaving the code editor.

Ideally, the result of an evaluation pops up right next to the cursor, so you don't have to do any context switches or lose focus. It can also be printed out in a separate frame right next to the code. This means that testing the code you currently write is at your fingertips.

Easy setup

With some help from IPython, it is possible to write, modify & evaluate Python code in a REPL Driven way. I would recommend to install IPython globally, to make it accessible from anywhere on your machine.

pip install ipython

Configure IPython to make it ready for REPL Driven Development:

c.InteractiveShellApp.exec_lines = ["%autoreload 2"] c.InteractiveShellApp.extensions = ["autoreload"] c.TerminalInteractiveShell.confirm_exit = False

You will probably find the configuration file here: ~/.ipython/profile_default/ipython_config.py

You are almost all set.

Emacs setup

Emacs is my favorite editor. I'm using a couple of Python specific packages to make life as a Python developer in general better, such as elpy. The auto-virtualenv package will also help out making REPL Driven Developer easier. It will find local virtual environments automatically and you can start coding without any python-path quirks.

Most importantly, set IPython as the default shell in Emacs. Have a look at my Emacs setup for the details.

VS Code setup

I am not a VS Code user. But I wanted to learn how well supported REPL Driven Development is in VS Code, so I added these extensions:

You would probably want to add keyboard shortcuts to get the true interactive feel of it. Here, I'm just trying things out by selecting code, right clicking and running it in an interactive window. It seems to work pretty well! I haven't figured out if the interactive window is picking up the global IPython config yet, or if it already refreshes a submodule when updated.

Evaluating code in the editor with fast feedback loops.
It would be great to have keyboard commands here, though.

Current limitations

In Clojure, you connect to & modify an actually running program by re-evaluating the source code. That is a wonderful thing for the developer experience in general. I haven't been able to do that with Python, and believe Python would need something equivalent to NRepl to get that kind of magic powers.

Better than TDD

I practice REPL Driven Development in my daily Python work. For me, it has become a way to quickly verify if the code I currently write is working as expected. I usually think of this REPL driven thing as Test Driven Development Deluxe. Besides just evaluating the code, I often write short-lived code snippets to test out some functionality. By doing that, I can write code and test it interactively. Sometimes, these code snippets are converted permanent unit tests.

For a live demo, have a look at my five minute lightning talk from PyCon Sweden about REPL Driven Development in Python.

Never too late to learn

I remember it took me almost a year learning & developing Clojure before I actually "got it". Before that, I sometimes copied some code and pasted it into a REPL and then ran it. But that didn't give me a nice developer experience at all. Copy-pasting code is cumbersome and will often fail because of missing variables, functions or imports. Don't do that.

I remember the feeling when figuring out the REPL Driven Development workflow, I finally had understood how software development should be done. It took me about 20 years to get there. It is never too late to learn new things. 😁

^{Top photo by ckturistando on Unsplash}

A simple & scalable Python project structure

File & folder structures - there's almost as many different variations as there are code repositories.

One common thing though, is that you'll probably find the utils folder in many of the code repos out there, regardless of programming language. That's the one containing the files that don't fit anywhere in the current project structure. It is also known as the helpers folder.

Organizing, sorting and structuring things is difficult. There's framework specific CLIs and tools that will create a nice setup for you, specialized for the needs of the current framework.

"There should be one-- and preferably only one --obvious way to do it."
^{The Zen of Python}

Is there one folder structure to rule them all? Probably not, but I have tried out a way to organize code that is very simple, framework agnostic and scalable as projects grow.

Structure for simplicity

A good folder structure is one that makes it simple to reuse existing code and makes it easy to add new code. You shouldn't have to worry about these things. The thing I've tried out with success is very much inspired by the Polylith architecture. Polylith is a monorepo thing, but don't worry. This post isn't about monorepos at all, however this one is if you are interested in Python specific ones.

An entry point and a components folder. You won't need much more. Use your favorite dependencies tool, mine is currently Poetry.

It's all about the components

The main takeaway here is to view code as small, reusable components, that ideally does one thing only. A component is not the same thing as a library. So, what's the difference?

A library is a full blown feature. A component can be a single function, or a parser. It can also be a thin wrapper around a third party tool.

"Simple is better than complex."
^{The Zen of Python}

I think the idea of writing components is about changing mindset. It is about how to approach a problem and how to organize the code that solves a problem.

It shouldn't be too difficult to grasp for Python developers, though. For us Python devs, it's an everyday thing to write functions and have them in modules. Another useful thing, probably more common in library development, is to group the modules into packages.

Modules, packages, namespaces ... and components?

In Python, a file is a module. One or more modules in a folder becomes a package. A good thing with this is that the code will be namespaced when importing it. Where does the idea of components fit in here? Well, a component is a package. Simple as that.

"Namespaces are one honking great idea -- let's do more of those!"
^{The Zen of Python}

To make a package easier to understand, you can add an interface. Interfaces are well supported in Python. Specifying the interface of a package in an __init__.py file is a great way to make the intention of the code clearer and easier to grasp. Maybe there's only one function that makes sense to use from the "outside"? That's when to use an interface for your component.

Only the functions that makes sense should be exposed from a component.

Make code reuse easy

When organizing code into simple components, you will quickly discover how easy it is to reuse it. Code is no longer hidden in some utils folder and you no longer need to duplicate existing private helper functions (because the refactoring might break things), if they already are organized as reusable components with clear and simple APIs. I usually think of components as LEGO bricks to select from when building features. You will most likely produce new LEGO bricks of various shapes along the way.

This is code in a "dictionaries" component. The interface (previous picture) will handle the access to it.

Well suited for large apps

At work, we have a couple of Python projects using this kind of structure. One of them is a FastAPI app with an entry point (we named it app.py) containing the public endpoints. The entry point is importing a bunch of components that does the actual work.

The repo contains about 80 Python files. Most of them are grouped into components (in total about 30 components). This particular project is about 3K lines of Python code, but other repos are much smaller with only a handful of components.

Perfect for functional programming

Even though it is not a requirement, organizing code into components fits very well with functional programming. Separating code into data, calculations and actions are well suited for the component thing described here in this post.

Don't forget to keep the components simple, and try to view them as LEGO bricks to be used from anywhere in the app. You'll have fun while doing it too.

^{Top photo by Maureen Sgro on Unsplash}

Just use Dictionaries

A Python dictionary has a simple & well-known API. It is possible to merge data using a nice & minimalistic syntax, without mutating or worrying about state. You're probably not gonna need classes.

Hey, what's wrong with classes? 🤔

From what I've seen in Python, classes often add unnecessary complexity to code. Remember, the Python language is all about keeping it simple.

My impression is that in general, class and instance-based code feels like the proper way of coding: encapsulating data, inheriting features, exposing public methods & writing smart objects. The result is very often a lot of code, weird APIs (each one with its own) and not smart-enough objects. That kind of code quickly tend to be an obstacle. I guess that's when workarounds & hacks usually are added to the app.

Two ways of solving a problem: class-based vs data-oriented.
Less code, less problems.

What about Dataclasses?

Python dataclasses might be a good tradeoff between a heavy object with methods and the simple dictionary. You get typings and autocomplete. You can also create immutable data classes, and that's great! But you might miss the flexibility: the simplicity of merging, picking or omitting data from a dictionary. Letting data flow smoothly through your app.

Hey, what about Pydantic?

That's a really good choice for things like defining FastAPI endpoints. You'll get the typed data as OpenAPI docs for free.

I would as early as possible convert the Pydantic model to a dictionary (using the model.dict() function), or just pick the individual keys and pass those on to the rest of the app. By doing that, the rest of the app is not required to be aware of a domain specific type or some base class, created as workaround for the new set of problems introduced with custom types.

Just data. Keeping it simple.

What about the basic types? 🤔

That is certainly a tradeoff when using dictionaries, the data can be of any type and you will potentially get runtime errors. On the other hand, is that a real problem when using basic data types like dict, bool, str or int? For me, I can't remember that ever has been an issue.

But shouldn't data be private?

Classes are often used to separate public and private functionality. From my experience, explicitly making data and functionality private rarely adds value to a product. I think Python agrees with me about this. By default, all things in a Python module is public. I remember when learning about it, and the authors saying that’s okay because we’re all adults here. I very much liked that perspective!

Do you like Interfaces? 🤩

Yes! Especially when structuring code in modules and packages (more about that in the next section). Using __init__.py is a great way to make the intention of a small package clearer and easier to grasp. Maybe there's only one function that makes sense to use from the outside? That's where the package interface (aka the __init__.py file) feature fits in well.

Python files, modules, packages?

In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into an app. Using a package interface makes sense when structuring code in this way.

Keeping it simple. 😊

I'm finishing off this post with a quote from the past:

“Data is formless, shapeless, like water.
If you put data in a Clojure map, it becomes the map.
You put data in a Python list and it becomes the list.
Now data in a program can flow or it can crash.

Be data, my friend.”

Bruce Lee, 1940 - 1973

ps.

If neither Bruce or I convinced you about the great thing with simple data structures, maybe Rich Hickey will? Don't miss his "just use maps" talk!

ds.

^{Top photo by Syd Wachs on Unsplash}