Better Python Developer Productivity with RDD

"REPL Driven Development is an interactive development experience with fast feedback loops”

I have written about REPL Driven Development (RDD) before, and I use it in my daily workflow. You will find setups for Python development in Emacs, VS Code and PyCharm in this post from 2022. That's the setup I have used since then.

But there's one particular thing that I missed ever since I begun with RDD in Python. I learned this interactive REPL kind of writing code when developing services and apps with Clojure. When you evaluate code in Clojure - such as function calls or vars - the result of the evaluation nicely pops up as an overlay in your code editor, right next to the actual code. This is great, because that's also where the eyes are and what the brain is currently focused on.

The setup from my previous post will output the evaluated result in a separate window (or buffer, as it is called in Emacs). Still within the code editor, but in a separate view. That works well and has improved my Python Developing Productivity a lot, but I don't like the context switching. Can this Python workflow be improved?

Can I improve this somehow?

I've had that question in the back of my mind for quite some time. My Elisp skills are unfortunately not that great, and I think that has been a blocker for me. I've managed to write my own Emacs config, but that's about it. Until now. During the Christmas Holidays I decided to try learning some Emacs Lisp, to be able to develop something that reminds of the great experience from Clojure development.

I used an LLM find out how to do this, and along the way learned more about what's in the language. I'm not a heavy LLM/GPT/AI user at all. In fact, I rarely use it. But here it made sense to me, and I have used it to learn how to write and understand code in this particular language and environment.

I have done rewrites and a lot of refactoring of the result on my own (my future self will thank me). The code is refactored from a few, quite large and nested Lisp functions into several smaller and logically separated ones. Using an LLM to just get stuff done and move on without learning would be depressing. The same goes for copy-pasting code from StackOverflow, without reflecting on what the code actually does. Don't do that.

Ok, back to the RDD improvements. I can now do this, with new feature that is added to my code editor:

Selecting a variable, and inspecting what it contains by evaluating and displaying the result in an overlay.

Selecting a function, execute it and evaluate the result.

The overlay content is syntax highlighted as Python, and will be rendered into several rows when it's a lot of data.

The actual code evaluation is still performed in the in-editor IPython shell. But the result is extracted from the shell output, formatted and rendered as an overlay. I've chosen to also truncate the result if it's too large. The full result will still be printed in the Python shell anyway.
The Emacs Lisp code does this in steps:

1. Adding a hook for a specific command (it's the elpy-shell-send-buffer-or-region command). The Emacs shortcut is C-c C-c.
2. Capture the contents of the Python shell.
3. Create an overlay with the evaluated result, based on the current cursor position.
4. Remove the overlay when the cursor moves.

This is very much adapted to my current configuration, and I guess the real world testing of this new addition will be done after the holidays, starting next week. So far, so good!

Future improvements?

I'm currently looking into the possibilities of starting an external IPython or Jupyter/kernel session, and how to connect to it from within the code editor. I think that could enable even more REPL Driven Development productivity improvements.

You'll find the Emacs Lisp code at GitHub, in this repo, where I store my current Emacs setup.

^{Top Photo by Nicolai Berntsen on Unsplash}