Are we there yet?

Continuing with the work on tooling support for interactive & fun development with Python.

A while ago, I wrote an article about my current attempts to make development in Python more interactive, more "test" driven and more fun.

My North Star is the developer experience in Clojure, where you have everything at your fingertips. Evaluating expressions (i.e. code) is a common thing for Lisp languages in general. I've found that it is sometimes difficult to explain this thing to fellow developers with no experience from Clojure development. The REPL is in most languages an external thing you do in a terminal window, detached from the work you usually do in your IDE. But that's not the case with REPL Driven Development (RDD).

Along the way, I have learned how to write and evaluate Python code within the code editor by using already existing tools and how to configure them. Here's my first post and second post (with setup guides) about it. You'll find information about the basic idea with RDD and guides on how to setup your IDE or code editor for Python development.

Can it be improved?

This has been the way I have done Python development most of the time, described in the posts above. But I have always wanted to find ways to improve this workflow, such as actually see the evaluated result in an overlay right next to the code. Not too long ago, I developed support for it in Emacs and it has worked really well! You can read about it and see examples of it here.

What about AI?

While writing Python code, and doing the RDD workflow, there's often a manual step to add some test or example data to variables and function input parameters. Recently, I got the idea to automate it using an LLM. I wrote a simple code editor command that prompts an AI to generate random (but relevant) values and then populate the variables with those. Nowadays, when I want to test Python code, I can prepare it with example data in a simple way by using a key combination. And then evaluate the code as before.

Are we there yet?

One important thing with RDD, that I haven't been able to figure out until now, is how to modify and evaluate code from an actual running program. This is how things are done in Clojure, you write code and have the running app, service, software constantly being changed while you develop it. Without any restarts. It is modified while it is running. Tools like NRepl does this in the background, with a client-server kind of architecture. I haven't dig that deep into how NRepl works, but believe it is similar to LSP (Language Server Protocol).

The workflow of changing a running program is really cool and something I've only seen before as a Clojure developer. So far I have used IPython as the underlying tool for REPL Driven Development (as described in the posts above).

A solution: the Kernel

In Python, we have something similar to NRepl: Jupyter. Many developers use Notebooks for interactive programming, and that is closely related to what I am trying to achieve. With a standard REPL session, you can add, remove and modify the Python code that lives in the session. That's great. But a sesssion is not the same as an actual running program.

A cool thing with Jupyter is that you can start a Python Kernel that clients can connect to. A client can be a shell, a Notebook - or a Code Editor.

jupyter kernel

By running the jupyter kernel command, you have a running Python program and can interactively add things to it, such as initiating and starting up a Python backend REST service. Being able to connect to the Jupyter kernel is very useful. While connected to the kernel, you can add, remove, modify the Python code - and the kernel will keep on running. This means that you can modify and test your REST service, without any restarts. With this, we are doing truly interactive Python programming. You will get instant feedback on the code you write, by evaluating the code in your editor or when testing the endpoint from a browser or shell.

"Dude, ever heard about debugging?"

Yes, of course. Debugging is closly related to this workflow, but it is not the same thing. Debugging is usually a one way flow. A timeline: you run code, pause the execution with breakpoints where you can inspect things, and then continue until the request is finalized. The RDD workflow, with modifying and evaluating a running program, doesn't have a one-way timeline. It's timeless. And you don't add breakpoints.

REPL Driven tooling support

I have developed tooling support for this new thing with connecting to a Jupyter Kernel, and so far it works well! Python is different from languages like Clojure: namespaces are relative to where the actual module is placed in the folder structure of a repo. This means that those connected to a Kernel need to have the full namespace (i.e. the full python path) for the items to inspect. This is what I have added in the tooling, so I can keep the RDD flow with the nice overlays and all.

I am an Emacs user, and the tooling I've written is for Emacs. But it shouldn't be too difficult to add it to your favorite code editor. Have a look at the code. You might even learn some Lisp along the way.

Resources

• The Emacs configuration, with RDD tooling support.

^{Top Photo by David Vujic}

FOSDEM 25

FOSDEM, a Conference different from any other I've been to before. I'm very happy for the opportunity to talk, share knowledge and chat with the fellow devs there. I also met old and new friends in Brussels, and learned a little bit more about the nice Belgian beer culture. 😀

The FOSDEM event is free and you don't even register to attend (only speakers do that in a call for speakers process). Just come by the University Campus, located not far from the center of the Inner City.

I travelled towards Belgium with the Night Train from Stockholm, Sweden, and arrived the next morning in Hamburg, Germany. From there, I took the DB ICE Train to Köln/Cologne. At some point, the top speed was about 250 km/h. That's fast! From there, another fast train to Brussels.

The variety of topics and the amount of tracks at FOSDEM was mind blowing, with thousands of participants. The overall vibe was very friendly & laid back. I joined Rust focused talks, JavaScript and UX/Design talks. And, of course, the Python room talks.

My talk was in the Python room and was about Python Monorepos and the Polylith Developer Experience. Everything Python related was on FOSDEM Day 2 and I think my presentation went really well! There was a lot of questions afterwards and I got great feedback from the people attending.

The next day I went back the same route as before. I got a couple extra of hours to spend in Hamburg before onboarding the Night Train back home to Stockholm. A great Weekend Trip!

Here's the recording of my talk from FOSDEM 25:

^{The video was downloaded from fosdem.org.
Licensed under the Creative Commons Attribution 2.0 Belgium Licence.}

Resources

• The FOSDEM 2025 website
• The Python tools for the Polylith Architecture at GitHub

Better Python Developer Productivity with RDD

"REPL Driven Development is an interactive development experience with fast feedback loops”

I have written about REPL Driven Development (RDD) before, and I use it in my daily workflow. You will find setups for Python development in Emacs, VS Code and PyCharm in this post from 2022. That's the setup I have used since then.

But there's one particular thing that I missed ever since I begun with RDD in Python. I learned this interactive REPL kind of writing code when developing services and apps with Clojure. When you evaluate code in Clojure - such as function calls or vars - the result of the evaluation nicely pops up as an overlay in your code editor, right next to the actual code. This is great, because that's also where the eyes are and what the brain is currently focused on.

The setup from my previous post will output the evaluated result in a separate window (or buffer, as it is called in Emacs). Still within the code editor, but in a separate view. That works well and has improved my Python Developing Productivity a lot, but I don't like the context switching. Can this Python workflow be improved?

Can I improve this somehow?

I've had that question in the back of my mind for quite some time. My Elisp skills are unfortunately not that great, and I think that has been a blocker for me. I've managed to write my own Emacs config, but that's about it. Until now. During the Christmas Holidays I decided to try learning some Emacs Lisp, to be able to develop something that reminds of the great experience from Clojure development.

I used an LLM find out how to do this, and along the way learned more about what's in the language. I'm not a heavy LLM/GPT/AI user at all. In fact, I rarely use it. But here it made sense to me, and I have used it to learn how to write and understand code in this particular language and environment.

I have done rewrites and a lot of refactoring of the result on my own (my future self will thank me). The code is refactored from a few, quite large and nested Lisp functions into several smaller and logically separated ones. Using an LLM to just get stuff done and move on without learning would be depressing. The same goes for copy-pasting code from StackOverflow, without reflecting on what the code actually does. Don't do that.

Ok, back to the RDD improvements. I can now do this, with new feature that is added to my code editor:

Selecting a variable, and inspecting what it contains by evaluating and displaying the result in an overlay.

Selecting a function, execute it and evaluate the result.

The overlay content is syntax highlighted as Python, and will be rendered into several rows when it's a lot of data.

The actual code evaluation is still performed in the in-editor IPython shell. But the result is extracted from the shell output, formatted and rendered as an overlay. I've chosen to also truncate the result if it's too large. The full result will still be printed in the Python shell anyway.
The Emacs Lisp code does this in steps:

1. Adding a hook for a specific command (it's the elpy-shell-send-buffer-or-region command). The Emacs shortcut is C-c C-c.
2. Capture the contents of the Python shell.
3. Create an overlay with the evaluated result, based on the current cursor position.
4. Remove the overlay when the cursor moves.

This is very much adapted to my current configuration, and I guess the real world testing of this new addition will be done after the holidays, starting next week. So far, so good!

Future improvements?

I'm currently looking into the possibilities of starting an external IPython or Jupyter/kernel session, and how to connect to it from within the code editor. I think that could enable even more REPL Driven Development productivity improvements.

You'll find the Emacs Lisp code at GitHub, in this repo, where I store my current Emacs setup.

^{Top Photo by Nicolai Berntsen on Unsplash}

Introducing python-hiccup

"All you need is list, set and dict"

Write HTML with Python

Python Hiccup is a library for representing HTML using plain Python data structures. It's a Python implementation of the Hiccup syntax.

You create HTML with Python, using list or tuple to represent HTML elements, and dict to represent the element attributes. The work on this library started out as a fun coding challenge, and now evolving into something useful for Python Dev teams.

Basic syntax

The first item in the list is the element. The rest is attributes, inner text or children. You can define nested structures or siblings by adding lists (or tuples if you prefer).

["div", "Hello world!"]

Using the html.render function of the library, the output will be HTML as a string: <div>Hello world!</div>

Adding id and classes:["div#foo.bar", "Hello world!"]
The HTML equivalent is: <div id="foo" class="bar">Hello world!</div>

If you prefer, you can define the attributes using a Python dict as an alternative to the compact syntax above: {"id": "foo", "class": "bar"}

Writing a nested HTML structure, using Python Hiccup:
["div", ["span", ["strong", "Hello world!"]]]
The HTML equivalent is:
<div><span><strong>Hello world!</strong></span></div>

Example usage

Server side rendering with FastAPI

Using the example from the FastAPI docs , but without the inline HTML. Instead, using the more compact and programmatic approach with the Python-friendly hiccup syntax.

from python_hiccup.html import render

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()


def generate_html_response():
    data = ["html",
           ["head", ["title", "Some HTML in here"]],
           ["body", ["h1", "Look ma! HTML!"]]]

    return HTMLResponse(content=render(data), status_code=200)


@app.get("/items/", response_class=HTMLResponse)
async def read_items():
    return generate_html_response()

PyScript

Add python-hiccup as any other third-party dependency in the package.toml file: packages = ["python-hiccup"]

Write the HTML rendering in your PyScript files:

from pyweb import pydom
from python_hiccup.html import render

pydom["div#main"].html = render(["h1", "Look ma! HTML!"])

That's it!

python-hiccup aims to make HTML rendering programmatic, simple and readable. I hope you will find it useful. The HTML in this blog post was written using python-hiccup.

Resources

• python-hiccup at GitHub
• PyScript and python-hiccup example

^{Top photo by Susan Holt Simpson on Unsplash}

Simple Kubernetes in Python Monorepos

"Kubernetes, also known as K8s, is an open source system for automating deployment, scaling, and management of containerized applications."
^{(from kubernetes.io)}

Setting up Kubenetes for a set of Microservices can be overwhelming at first sight.

I'm currently learning about K8s and the Ecosystem of tooling around it. One thing that I've found difficult is the actual K8s configuration and how the different parts relate to each other. The YAML syntax is readable, but I also find it hard to understand how to structure it - impossible to edit without having the documentation close at hand. When I first got the opportunity to work with Python code running in Kubernetes, I realized that I have to put extra effort in understanding what's going on in there.

I was a bit overwhelmed by what looked like a lot of repetitive and duplicated configuration. I don't like that. But there's a tool called Kustomize that can solve this, by incrementally constructing configuration objects with snippets of YAML.

Kustomize is about managing Kubernetes objects in a declarative way, by transforming a basic setup with environment-specific transformations. You can replace, merge or add parts of the configuration that is specific for the current environment. It reminds me of how we write reusable Python code in general. The latest version of the Kubernetes CLI - Kubectl - already includes Kustomize. It used to be a separate install.

Microservices

From my experience, the most common way of developing Microservices is to isolate the source code of each service in a separate git repository. Sometimes, shared code is extracted into libraries and put in separate repos. This way of working comes with tradeoffs. With the source code spread out in several repositories, there's a risk of having duplicated source code. Did I mention I don't like duplicated code?

Over time, it is likely that the services will run different versions of tools and dependencies, potentially also different Python versions. From a maintainability and code quality perspective, this can be a challenge.

YAML Duplication

In addition to having the Python code spread out in many repos, a common setup for Kubernetes is to do the same thing: having the service-specific configuration in the same repo as the service source code. I think it makes a lot of sense to have the K8s configuration close to the source code. But with the K8s configuration in separate repos, the tradeoffs are very much the same as for the Python source code.

For the YAML part in specific, it is even likely that the configuration will be duplicated many times across the repos. A lot of boilerplate configuration. This can lead to unnecessary extra work when needing to update something that affects many Microservices.

One solution to the tradeoffs with the source code and the Kubernetes configuration is: Monorepos.

K8s configuration in a Monorepo

A Monorepo is a repository containing source code and multiple deployable artifacts (or projects), i.e. a place where you would have all your Python code and where you would build & package several Microservices from. The purpose of a Monorepo is to simplify code reuse, and to use the same developer tooling setup for all code.

The Polylith Architecture is designed for this kind of workflow (I am the maintainer of the Python tools for the Polylith Architecture).

While learning, struggling and trying out K8s, I wanted to find ways to improve the Configuration Experience by applying the good ideas from the Developer Experience of Polylith. The goal is to make K8s configuration simple and joyful!

Local Development

You can try things out locally with developer tools like Minikube. With Minikube, you will have a local Kubernetes to experiment with, test configurations and to run your containerized microservices. It is possible to dry-run the commands or apply the setup into a local cluster, by using the K8s CLI with the Kustomize configs.

I have added examples of a reusable K8s configuration in the The Python Polylith Example repo. This Monorepo contains different types of projects, such as APIs and event handlers.

The K8s configuration is structured in three sections:

• A basic setup for all types of deployments, i.e. the config that is common for all services.
• Service-type specific setup (API and event handler specific)
• Project-specific setup

All of theses sections have overlays for different environments (such as development, staging and production).

As an alternative, the project-specific configuration could also be placed in the top /kubernets folder.

I can run kubectl apply -k to deploy a project into the Minikube cluster, using the Kustomize configuration. Each section adds things to the configuration that is specific for the actual environment, the service type and the project.

The base, overlays and services are the parts that aren't aware of the project. Those Project-specific things are defined in the project section.

Using a structure like this will make the Kubernetes configuration reusable and with almost no duplications. Changing any common configuration only needs to be done in one place, just as with the Python code - the bricks - in a Polylith Monorepo.

That’s all. I hope you will find the ideas and examples in this post useful.

Resources

• The Python Polylith Example repo
• The Python tools for the Polylith Architecture

^{Top photo by Justus Menke on Unsplash}