söndag 11 april 2021

Almost like Clojure

Practicing different programming languages has given me new insights in programming concepts, code styles and tools. My current crush is Clojure. I like the functional and data oriented style of it, the simplicity and minimalism. I think my favorite thing with Clojure, though, is how the REPL is used.

“What’s so cool about that repple thing?” 🧐

Testing at your fingertips 🚦

When using a code editor powered by a REPL, you can instantly evaluate variables, code blocks, functions or the entire file, just by a hitting a key combination. The result pops up right next to the cursor. This means that testing the code you currently write is at your fingertips. You are practicing a Deluxe version of Test Driven Development.

No copy-paste here 👮

It is not necessary to copy-paste code snippets into the actual REPL window or in a shell. In fact, you shouldn’t touch the REPL window at all. Because you don’t want to miss out on all the good stuff like autocomplete, hints and smart navigation that you have in your favorite editor.

For Clojurians only? 💭

I really would like to have a similar workflow in other languages too. It turns out there are some pretty good tools out there, the ones I’ve tried out so far are for JavaScript and ✨Emacs ✨.

Interactive JavaScript development 😍

Here’s me playing around with the node-zookeeper code. The cursor never leaves the editor window, the evaluation result is printed out in the Node.js REPL window to the right. More verbose than I’m used too in Clojure, but I think it works well.

Trying out code and ideas like this is a very nice workflow. I think you will find less need for debugging or console.log statements.

Keep on learning 📚

Learning a new programming language or tool can be really difficult, but also fun when the pieces finally start coming together. The knowledge gathered is something you can bring with you to other areas.

Have a look at my post about Emacs and my current configuration. The Node.js REPL feature is made possible using the js-comint package.

Photo by Marc-Olivier Jodoin on Unsplash

söndag 4 april 2021

Software as building blocks

There seems to be an ongoing trend in software development towards using monorepos. This trend is something I have seen especially in the Clojure community.

Polylith - a monorepo architecture

I like the way Polylith solves how to work with code using a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services.

Read more here: Polylith gitbook

The last architecture you will ever need *

From the Polylith docs:

"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."

Okay, backend systems. What about frontend systems? 🤔

I want to Polylith all the things

Is it possible to use the Polylith architecture for a code base that includes web apps? This is something that I have wanted to find out.

Here's my example repo.

In this repo, I’ve added backend Clojure code, frontend ClojureScript and also some glue in between in the form of cljc files. Cljc is Clojure code that can be consumed by both frontend and backend code. This makes it possible to share code across Clojure and ClojureScript, building things just like with LEGO bricks and baseplates.

All the things?

I'll leave the question if ClojureScript and Clojure really should live in the same ecosystem unanswered and hope to get feedback from you. In my example repo, I have put all components in the same place. Should the building blocks be separated somehow, or is it good enough to have both LEGO and DUPLO in the same box? What are your thoughts about it?

Do we still have the Polylith one-REPL experience?

Well, we can have a two-REPLs experience. One for Clojure, that run on top of the JVM, and one for ClojureScript on top of JavaScript. Running both will make REPL driven backend development and Interactive Web Development possible.

You can add and use new ClojureScript components while the REPL is running. Create the namespace and evaluate the function.

There is one thing that I have no solution for (yet). When creating a new ClojureScript component and evaluating the entire namespace at once, I get a compilation error in the ClojureScript REPL: file not on classpath. The ClojureScript REPL have to be restarted to reload new source paths.

But don't worry, you can still evaluate the individual functions in the namespace and they will be loaded as expected in the ClojureScript REPL.

Tooling support

Polylith has a very nice and useful tool to support creating building blocks and to verify the setup. You can create components, bases and projects - as long as it is Clojure. For ClojureScript, you will have to create components manually.

If you are lazy, like me, just create a component with the poly tool as you would for Clojure, and simply rename the file extension to cljs or cljc afterwards.

Editor support?

Your editor most likely has support for running both Clojure and ClojureScript simultaneously. Emacs is my favourite editor. Start the REPLs with the cider-jack-in-clj&cljs command and you're ready to go!

Two REPLS running (even though the Cider splash message is confusing)

* The quote is from Joakim Tengstrands and Furkan Bayraktars talk about Polylith at the FuncProg Sweden 2020 meetup.

Photo by Amélie Mourichon on Unsplash

söndag 14 mars 2021

Wake up, sleepy lambda

AWS Lambda functions with painfully slow start up times is a problem. But there's hope.

Clojure anywhere

After going through some of the 12 Stages of learning Clojure, I have found this Lisp style language to be very nice and it has become my favorite programming language.

Some of the nice things with Clojure are that data is immutable, the REPL is like magic and the code you write looks minimalistic. Also, you can run Clojure code almost anywhere: as backend services, web frontends and even in shell scripts.

Even in Lambda functions?

I found out that it is not that difficult to make Clojure code to also run in AWS Lambda. The code can live in a Java runtime. Lambda events will be routed to a handler function written in Clojure, when the namespace implements a Java Request handler class (here's an example). Yes, there is some interop needed at the entry point of the Lambda code to make it work. But don't worry.

In addition to the Java interop, the source code should also be ahead-of-time compiled and packaged. I've used the uberdeps library that will make the process smooth when using tools.deps.

If you are not familiar with the Java lingo (like me, I have a background in Node.js, frontend and .NET), words like jar, AOT and even Java can be intimidating. I guess it is possible to sidestep all of this by writing the Lambda function in ClojureScript and run it in a Node.js runtime. But I don't want to opt out of the rich ecosystem of Clojure libraries built for the server side.

Sleepy lambda, slow cold starts

It seems that a Lambda running in a Java or .NET runtime often has painfully slow start up times. Setting a timeout of 3 seconds is probably not even enough. A simplistic solution to this problem is to use the AWS provisioned concurrency with a Lambda alias. No code changes required, only configuration and money. I wanted to find out if there are other ways to solve this problem.

What about GraalVM?

I found this great post about Clojure and Lambda written by Esko Luntola. Code that is compiled with GraalVM and running in Docker, solving the issues with slow cold starts and even makes requests in general super fast. Wow!

However, I haven't succeeded in going through the steps described in the post and am stuck in build failures that I don't know how to solve - yet. But I will try this approach some more. Even though it requires some initial setup with configs and Docker containers, this seems to be the way to go.

What about GraalVM in a custom runtime?

When digging deeper in how to run code in AWS Lambda, I found out that you can create your own runtime. To me, this approach looks simplistic and straight forward.

In this repo, I have written a Hello World example with:

  • a custom runtime (the file called bootstrap) written in bash (grabbed from the AWS Official docs with some additional error handling).
  • Clojure code, with a main function as the entry point. It is not neccessary to implement the Java Request handler class, and the main function returns data via standard output.

Input args as a JSON string

The Clojure code is compiled, and built with GraalVM by using the Native Image feature. Have a look at the Makefile for details.

compile and build with GraalVM Native Image
I've tried to keep things simple and followed the guides at clj-graal-docs and watched Michiel Borkents excellent beginners guide to GraalVM on YouTube.

The custom runtime and the function can be deployed separately. You can reuse the same runtime for several Lambda functions, by creating a Lambda Layer in AWS. The function code can be deployed directly, just upload the zipped file to AWS Lambda.

I like this approach, it works fine in my simplistic hello world example. But I haven't tried it in a real-world scenario. When going beyond an experiment, I think there might be additional resource configuration flags to Native Image required. Probably the setting ReflectionConfigurationFiles.

The tradeoffs?

From what I can see in my totally non-scientific weekend experimental testing is that the code running in my Custom Runtime has a cold start of somewhere between 100 and 300 milliseconds. Good enough. When warmed up, requests are processed between 15-30 milliseconds. Not bad.

When comparing with my other example lambda that is running in a Java runtime (with cold starts usually taking over 2000 milliseconds), the Custom Runtime with GraalVM is way faster.

But once warmed up, the Java runtime is actually super fast, with duration times between 1 - 30 milliseconds. Also, I haven't yet solved how to build all of this in a CI/CD setup. The code was built with GraalVM locally on my machine.

I would very much appreciate your input on the experiments shared in this blog post.

Photo by Abdelrahman Hassanein on Unsplash

lördag 27 februari 2021

Interactive Web Development

The workflow for developing web apps has been very much the same for quite some time, even if the tools have evolved and the programming languages have changed.

Write some code, save the file, switch to the browser window, hit the refresh button and wait.

Sometimes there has even been a couple of extra steps in between: write, save, compile, restart, switch to the browser, refresh, wait and repeat. The problem with this is that - at least for me - the focus on the actual problem to solve is paused or even worse: lost.

"Write code,
Switch to the browser window,
Hit the refresh button,
Wait ...
Now, where was I?"

But there’s hope!

🔥 Hot Reloading

With a modern development environment, you can use automation to get rid of some of the noice. Your code editor can probably be configured to detect when a file has changed and automatically save it.

With tools like Webpack for JavaScript and Shadow-cljs for ClojureScript, you can reload a running development web server - even replace code while the web server is running without loosing the current state. This is often called Hot Reloading and makes the development workflow a lot smoother than before. Very cool.

🧱 Building blocks

I really like the way React has solved problems with web development in general. React simplifies how you can separate features into components. At the same time, it promotes grouping relevant markup and code, by combining JavaScript & HTML into the JSX syntax. I think this helps us keeping focus on the actual problem solving, with less navigating and flipping between files in the code editor. Good stuff.

♻️ The Virtual DOM

The Virtual DOM will give instant feedback when data has changed. The DOM is changed on the fly, without the need for page reloads or server side rendering. This is done by calculating the current state of the browser DOM and an in-memory copy of it. When there is a change in the copy, the relevant parts of the browser DOM will change. Really nice.

🦄 The Developer Superpower

Recently, I’ve learned about a really cool workflow called REPL Driven Development. I have used a REPL before, but have always thought of it as something that is used in a shell for some quick testing, not in the actual code editor. But with Clojure, this is the way we work. While you are in the editor, thinking, writing and looking at the code, you can evaluate it and get instant feedback. You can trigger events, browse state and even query the browser DOM! This is possible thanks to the Interactive REPL.

It’s almost like Magic. 🤩

I think this is unique to Clojure and ClojureScript. I’m not sure why this superpower doesn’t seem to exist elsewhere, but it probably has something to do with the structure of the language itself: code and data share structure and syntax.

✨This is Interactive Web Development

By combining hot reloading, building blocks & the virtual DOM with the Interactive REPL - we have something that I would like to call Interactive Web Development.

Arrange your code editor side-to-side with a browser window, to get instant feedback from both the user interface and the functionality when evaluating and experimenting with the code during development.

Photo at the top by Nong Vang on Unsplash

Have a look at my Func Prog Sweden second 2021 talk at YouTube that is covering this topic too. Direct link: ClojureScript: React with a Hiccup

söndag 14 februari 2021

Test Driven Development Deluxe

You have probably already heard about the red-green-refactor workflow when practicing Test Driven Development: start with a failing test, make the test pass, refactor and repeat.

I don’t think I have ever been very strict about the TDD workflow. I usually jump back and forth between the code and the unit test, often begin with trying out naming of functions and less-important things like that. I also find refactoring difficult at that stage, probably because the goal is to write code that does something and that does it well enough. I find it much easier to reflect and come up with refactoring ideas once I see a Pull Request diff on GitHub.

Too much context switch

Until recently, I have failed to find a workflow that is seamless when writing, running and testing code. There is often context switching involved: executing commands that run tests in a terminal or navigating the test runner of an IDE. Sometimes that means a pause with a delay of seconds, or even minutes. The eyes and the brain have lost attention to the code. The focus and flow is gone. Now where was I?

What would Clojure do?

These days, the programming language that makes me 🤩 is Clojure. It has been quite a journey learning how to use the language and how to use the code editor properly. I guess most of us don’t pick up all the good parts of a language or a tool immediately. It takes a while to get there.

An example: my brain hasn’t had the bandwidth enough to get how to use the interactive REPL until recently. There has been a lot of other things to unpack: language syntax, functional programming concepts and keyboard shortcuts. On a positive note: there are opportunities for daily learning and that’s where I am today.

- Wow, I can evaluate an expression in the code editor and the result pops up right next to it!

Star struck. 🤩

The other day I was frustrated with the slow feedback from a CLI test command running in a terminal window. I guess it has something to do with the Java Virtual Machine starting up every time a command is executed. I’ve tried out tools with file watchers, reacting to changes, that speed things up - but there’s still context switching involved, moving your eyes off the editor & the code.

What would a Clojurist do? I think the answer would be:
- Use the REPL.

The REPL, you must use

With Clojure, using the REPL means writing code in the editor, with autocomplete and everything. Also, evaluating the code while in the editor. There is no need to write a statement in a separate terminal window. Your editor has a connection to the interactive REPL (a running server), that will evaluate your code and return the result back to you. With the interactive REPL, you can quickly try things out, without switching to a different tool or view.

This is what some people call REPL Driven Development. Before I learned about it, I have always thought of a REPL as something that is used in a terminal window, away from the editor and away from the code.

the interactive REPL in action

Development Deluxe

By combining REPL Driven & Test Driven Development we have a setup that solves the issues with context switching and slow test runs. From what I’ve learned and tried out so far, I think it is a very nice workflow. 🤩

A very nice workflow

With REPL Driven Development, you write code, evaluate it and get instant feedback while keeping your focus on the code. Maybe the quick visual feedback in the editor is good enough to continue working? Great. Or, you write one or two commands wrapped in a runnable Clojure (comment) block just below the functions, to simplify the evaluation.

Everyone needs a coffee break. Commit the code (yes, the commands too) and continue with it later. ☕

Take a break

Your brain has been charged and you feel refreshed again. When returning to the code, you realize that the commands are probably evolving into something similar to a unit test. Should it be a proper unit test? If so, wrap it in a (deftest) function with some assertions and put it in a separate file.

Here’s a tip: give the unit test the same namespace as the actual code, but add a -test suffix to it. When doing so, your code editor (if it is smart) can run the corresponding tests when in a namespace. If you are an Emacs user, simply press the keys C-c C-t n to run the tests.

Even though you have put the unit tests in a separate file, in a separate folder, there is no need to switch context. Your tests run from where you are and you get instant feedback on the result. Very cool.

I think this is Test Driven Development Deluxe. 🤩

Photo by Felipe Giacometti on Unsplash.

The screenshots are from email-attachments - A Clojure library that makes extracting attachments from email simple.