A simple & scalable Python project structure

File & folder structures - there's almost as many different variations as there are code repositories.

One common thing though, is that you'll probably find the utils folder in many of the code repos out there, regardless of programming language. That's the one containing the files that don't fit anywhere in the current project structure. It is also known as the helpers folder.

Organizing, sorting and structuring things is difficult. There's framework specific CLIs and tools that will create a nice setup for you, specialized for the needs of the current framework.

"There should be one-- and preferably only one --obvious way to do it."
^{The Zen of Python}

Is there one folder structure to rule them all? Probably not, but I have tried out a way to organize code that is very simple, framework agnostic and scalable as projects grow.

Structure for simplicity

A good folder structure is one that makes it simple to reuse existing code and makes it easy to add new code. You shouldn't have to worry about these things. The thing I've tried out with success is very much inspired by the Polylith architecture. Polylith is a monorepo thing, but don't worry. This post isn't about monorepos at all, however this one is if you are interested in Python specific ones.

An entry point and a components folder. You won't need much more. Use your favorite dependencies tool, mine is currently Poetry.

It's all about the components

The main takeaway here is to view code as small, reusable components, that ideally does one thing only. A component is not the same thing as a library. So, what's the difference?

A library is a full blown feature. A component can be a single function, or a parser. It can also be a thin wrapper around a third party tool.

"Simple is better than complex."
^{The Zen of Python}

I think the idea of writing components is about changing mindset. It is about how to approach a problem and how to organize the code that solves a problem.

It shouldn't be too difficult to grasp for Python developers, though. For us Python devs, it's an everyday thing to write functions and have them in modules. Another useful thing, probably more common in library development, is to group the modules into packages.

Modules, packages, namespaces ... and components?

In Python, a file is a module. One or more modules in a folder becomes a package. A good thing with this is that the code will be namespaced when importing it. Where does the idea of components fit in here? Well, a component is a package. Simple as that.

"Namespaces are one honking great idea -- let's do more of those!"
^{The Zen of Python}

To make a package easier to understand, you can add an interface. Interfaces are well supported in Python. Specifying the interface of a package in an __init__.py file is a great way to make the intention of the code clearer and easier to grasp. Maybe there's only one function that makes sense to use from the "outside"? That's when to use an interface for your component.

Only the functions that makes sense should be exposed from a component.

Make code reuse easy

When organizing code into simple components, you will quickly discover how easy it is to reuse it. Code is no longer hidden in some utils folder and you no longer need to duplicate existing private helper functions (because the refactoring might break things), if they already are organized as reusable components with clear and simple APIs. I usually think of components as LEGO bricks to select from when building features. You will most likely produce new LEGO bricks of various shapes along the way.

This is code in a "dictionaries" component. The interface (previous picture) will handle the access to it.

Well suited for large apps

At work, we have a couple of Python projects using this kind of structure. One of them is a FastAPI app with an entry point (we named it app.py) containing the public endpoints. The entry point is importing a bunch of components that does the actual work.

The repo contains about 80 Python files. Most of them are grouped into components (in total about 30 components). This particular project is about 3K lines of Python code, but other repos are much smaller with only a handful of components.

Perfect for functional programming

Even though it is not a requirement, organizing code into components fits very well with functional programming. Separating code into data, calculations and actions are well suited for the component thing described here in this post.

Don't forget to keep the components simple, and try to view them as LEGO bricks to be used from anywhere in the app. You'll have fun while doing it too.

^{Top photo by Maureen Sgro on Unsplash}

Just use Dictionaries

A Python dictionary has a simple & well-known API. It is possible to merge data using a nice & minimalistic syntax, without mutating or worrying about state. You're probably not gonna need classes.

Hey, what's wrong with classes? 🤔

From what I've seen in Python, classes often add unnecessary complexity to code. Remember, the Python language is all about keeping it simple.

My impression is that in general, class and instance-based code feels like the proper way of coding: encapsulating data, inheriting features, exposing public methods & writing smart objects. The result is very often a lot of code, weird APIs (each one with its own) and not smart-enough objects. That kind of code quickly tend to be an obstacle. I guess that's when workarounds & hacks usually are added to the app.

Two ways of solving a problem: class-based vs data-oriented.
Less code, less problems.

What about Dataclasses?

Python dataclasses might be a good tradeoff between a heavy object with methods and the simple dictionary. You get typings and autocomplete. You can also create immutable data classes, and that's great! But you might miss the flexibility: the simplicity of merging, picking or omitting data from a dictionary. Letting data flow smoothly through your app.

Hey, what about Pydantic?

That's a really good choice for things like defining FastAPI endpoints. You'll get the typed data as OpenAPI docs for free.

I would as early as possible convert the Pydantic model to a dictionary (using the model.dict() function), or just pick the individual keys and pass those on to the rest of the app. By doing that, the rest of the app is not required to be aware of a domain specific type or some base class, created as workaround for the new set of problems introduced with custom types.

Just data. Keeping it simple.

What about the basic types? 🤔

That is certainly a tradeoff when using dictionaries, the data can be of any type and you will potentially get runtime errors. On the other hand, is that a real problem when using basic data types like dict, bool, str or int? For me, I can't remember that ever has been an issue.

But shouldn't data be private?

Classes are often used to separate public and private functionality. From my experience, explicitly making data and functionality private rarely adds value to a product. I think Python agrees with me about this. By default, all things in a Python module is public. I remember when learning about it, and the authors saying that’s okay because we’re all adults here. I very much liked that perspective!

Do you like Interfaces? 🤩

Yes! Especially when structuring code in modules and packages (more about that in the next section). Using __init__.py is a great way to make the intention of a small package clearer and easier to grasp. Maybe there's only one function that makes sense to use from the outside? That's where the package interface (aka the __init__.py file) feature fits in well.

Python files, modules, packages?

In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into an app. Using a package interface makes sense when structuring code in this way.

Keeping it simple. 😊

I'm finishing off this post with a quote from the past:

“Data is formless, shapeless, like water.
If you put data in a Clojure map, it becomes the map.
You put data in a Python list and it becomes the list.
Now data in a program can flow or it can crash.

Be data, my friend.”

Bruce Lee, 1940 - 1973

ps.

If neither Bruce or I convinced you about the great thing with simple data structures, maybe Rich Hickey will? Don't miss his "just use maps" talk!

ds.

^{Top photo by Syd Wachs on Unsplash}

Building a bank with EPiServer?

Can it be done?

Yes. And this blog post is about how it can be done. I want to share my thoughts and experiences about writing EPiServer sites, connected to third party services and/or external APIs, with code examples and design suggestions.

 

Should it be done?

Yeah, why not? Long answer: if you already have a nice and smooth communication between product owners and developers - a daily constant flow of feature and content production releases - you probably wouldn't need a Content Management System (CMS) at all. But if you release code a couple of times a month, maybe even longer in between the releases (like most companies out there), a CMS is great to have in place. EPiServer is a good choice for that. Let's build a bank with it!

Actually, I have built one already. Well, no. Not really. But I wrote an example project with some of the building blocks I think an EPiServer site connected to external back end systems should have. 

Here's the code (at Github)
 

I know, I know ... it's a very simplistic example. I wanted to get rid of all the noise, and focus on the "core" stuff that I think is important. I think most of the features in the example code are ready for you to use today, a few things are for demo purposes only (like the implementation of the logger interface). 
 

NoScript First

The example code is an EPiServer MVC site: one page instance with a form that is posted to a controller on the server. The server code is sending the data to an external web service and the service returns user data to be displayed in the user interface. 

Let's zoom in to the form and the posting. All user input should be sanitized and validated. The data sent should be restricted to http POST, along with an anti forgery token. Use the built in model binding and data annotations for the validation of data.

I would recommend you to start developing with JavaScript disabled in the browser. That is a development style I like to call NoScript First. I think it will help you to focus on the security and the basic flow. When you are done with the basics, enable JavaScript and write client side code for a better user experience. You can hijack the form submit action and post the data with ajax, instead of a full page reload.

Here's some more info (with code examples) about NoScript First:
You might not need JavaScript

 

The basics with Dependency Injection

Try keep the controllers as light weight as possible. Extract methods to specialized helper files and inject all dependencies by using a tool like StructureMap. For testability, use c# interfaces instead of concrete classes when injecting the dependencies.

There are trade offs: your source code may seem complex at first look, especially for developers not used to these kind of patterns and abstractions. By hiding the implementations behind APIs (interfaces), you can switch between different implementations, even in runtime. But why would anyone want to do that? I'll will answer that shortly.

 

Isn't unit testing dead?

Dependency injection will simplify testing. A unit test can focus on a specific part of the source code, by providing it with fake implementations of the dependencies and control the data that is passed between them. But why unit test stuff at all? For me, test driven development is a style. A tool, helping me to write code that is specialized, minimalistic and (hopefully) readable. For me, the actual testing is secondary. That's why I think some parts of the source code can be just fine without unit tests. It is alright! However, be strict with unit testing your validation features, like the custom data annotations in the example code. Write unit tests until you are sure that your validation methods does the right thing. Try different scenarios, not only the happy path. Be evil. If you wake up in the middle of the night, because you had dreamed about ways to bypass the validation, write a unit test that prove it. Solve the issue and run the tests again.

To get the dependency injection stuff in place in your controllers, start with a simple unit test "newing up" a controller, develop and refactor the code from there. I think unit tests eventually will help you find the flow, especially when it's done test driven style.

If your controllers handles EPiServer content, such as traversing an EPiServer page tree to create menus or lists, I recommend using FakeMaker - a tool that simplifies unit testing for scenarios like that.

 

Something between the web and the services: Providers

A third party web service or a back end API is probably developed for general purpose, to be used by many different channels. A web site is specialized and customized. The data passed from a service will most likely not fit perfectly with the view of a web user interface. To create a nice user experience, the data probably has to be rearranged in some way (formatted, combined, simplified, renamed). Do that in a separate class library.

Map the data objects from the service to local domain objects that are customized for your user interface. Map the service class properties you need to a local domain object. Let AutoMapper handle the mapping, it's a great tool. Also, make sure you unit test the mappings! Included in AutoMapper, there is a unit testing feature called "AssertConfigurationValid". It has saved my life many times.

Let the library take care of the connection to the service, and keep the user interface unaware of the service API. The library should have only the features needed by the web user interface, no more than that.

I would call a library like that a Provider. Consume the providers by injecting them into the web project code (as described earlier with Dependency Injection). The web communicates with the API (the interface) of a Provider, that means that controllers also are unaware of the actual provider implementation (the concrete class). In the example code, the mappings between interfaces and concrete classes is handled by the StructureMap configuration.

With providers, you can switch service implementations without the need to alter the code in the web project. The same goes for the providers themselves. One provider implementation can be switched to an other one. But why would anyone want to do that? Okay, maybe it's time to give some answers.

 

Fake it!

During development, the back end services may also be developed in parallel and could go offline from time to time. Being dependent on an unstable external service is risky. To be able to write code, browse and test the features of a site, using fake implementations of providers returning fake data is a way out of the problem. Faking it will make it possible for the team to be "offline" and still be productive, i.e. no direct dependencies to a web service or third party system.

There are other practical uses of fake Provider implementations. The site pages that are heavily based on presenting data from back end systems (like an account transaction list) will not look any good at all when logged in as an EPiServer editor (that most likely isn't identified as an existing customer in the back end system). To create a good user experience for editors (and avoiding embarrassing page exceptions in edit mode) fakes can be very useful. Instead of cluttering the controllers with if-else statements all over the place, determining if the current user is an editor or not, you could use your already existing fake providers! That means fake providers actually will be used in production.

To be able to switch between fake and live providers, the injected dependencies need some modification. That is because the StructureMap configuration will run on application start, and checking the logged in user role there won't work. It has to be done per request, triggered from the controllers.

 

Be water, my friend (inject the Factories)

You probably already have noticed in the example code that some sort of factory is injected into the controller, and not the actual provider. The provider is retrieved later in the constructor, by calling a factory method. This occurs on every request. The factory method is where the if-else checking takes place. Is the current user an editor or not? The StructureMap configuration also has two implementations added - one fake, one live - for a provider interface. This makes it possible for the factory to pull the desired implementation from the dependency resolver.

Note: Today I met Jeremy D. Miller (the creator of StructureMap) at the DevSum conference here in Stockholm, and he showed me some examples on how to configure StructureMap without the need of factories. I like it a lot I will try this out (and update the GitHub code repository).

(Okay, maybe this video hasn't that much to do with injecting Factories, but I like Bruce Lee and think this video is cool)

 
What's Pippi Longstocking got to do with it?
Finally, let's talk about the fake data itself. Using this setup, the development team have the opportunity to stress test the UI every day, by providing data that goes beyond the perfect and well balanced placeholder text areas in the design templates delivered by the UX team. 

One obvious thing to test is the name of the logged in user. Here's an idea: add a fake data user with the name Pippilotta Viktualia Rullgardina Krusmynta Efraimsdotter Långstrump (that's the swedish full name of Pippi Longstocking)! How will the user info section of the site behave with a name like that? Well, maybe "Pippilotta Viktualia Långstrump" would be enough, let's not exaggerate. But be aware, in the real world, there are people out there with some really messed up names ... being prepared for it might be a good idea.

What do you think about all of this? Please share your thoughts and ideas about building a bank with EPiServer.

Here's the code (at Github)

Yeah, we got classes now (ES6). You happy, pappy?

Since the beginning of time, JavaScript has been the class less society. Everyone had a function there. With ECMAScript 6, classes have now been added. Ah, man.

Wait, wait, wait! I actually like it. I believe the class keyword is good thing to JavaScript, mmkay?

There is quite a few frameworks out there that already have introduced classes (or class like solutions) to the language: Dojo, Prototype, CoffeScript, TypeScript ... They all have their own way of doing it. ECMAScript introduces a common way of how to write a class, and that itself is great. How can classes be useful in JavaScript?

I think it can simplify the structure of code, we can write functionality that is only accessible within a class. But then again, we have modules now. A module can export an object and keep code private. Why use classes?

Well, I think the benefit of writing a JavaScript class is inheritance (using the extends keyword). Inheritance can of course also be accomplished by creating literal objects extending other objects, simply by adding properties and methods to it. But the functional inheritance style looks kind of awkward sometimes. Writing a class that extends another class will probably make more sense. Especially for programmers used to write code in c#, java, Ruby or any other object oriented language.

However, I don't think consuming a class the traditional way is a good idea in JavaScript. I think it is an anti-pattern. Creating instances of classes using the new keyword spread out all over the code base? Don't do it! 

In sucks in c#, it sucks doing it in JavaScript too. Dependency Injection and IoC containers solve the "newing up classes problem" in c#. How about JavaScript?

Here's a suggestion.

Create your classes in ES6 modules. Export an already created instance (if you want a singleton like functionality), or export a "make" method that returns a new instance of the class.

By doing that, the consumer doesn't have to worry about if the code within the module is a class or an object literal.

An example:

// the foo module, exported as a singleton
class foo {
     constructor() {
          // constructor code here
     }

     myMethod () {
          // method body code here
     }
}

export default new foo();

// the foo module, exporting a "make instance" method
class foo {
     constructor() {
          // constructor code here
     }

     myMethod () {
          // method body code here
     }
}

let makeFoo = () => new foo();

export default makeFoo;


Using the first example, the consumer will get a baked and ready instance of the foo class. The second example will give the consumer the ability to get different instances, by calling the make function. The actual structure (currently a class) of the module is hidden, and not a part of the API.

What do you think about this? Let me know!





" - Happy, Pappy?"

Is the ES6 import feature an anti-pattern?

The new version of JavaScript is currently occupying my mind. I really like the ECMAScript 6 features, and the fact that we can write code using new stuff like arrow functions, scoped variables and native modules today (by using transpilers) is quite awesome.

There is especially one of the ES6 features that I've been thinking about: importing of modules. Modules will change the way we think about JavaScript for the web. Importing a module is so easy with ES6 and the syntax is very nice, readable and clean. How do I unit test such a module?

-----------
Update: I have experimented with an alternative to the examples in this blog. Have a look at it and please share your thoughts and feedback on the different approaches to unit testing modules:
Test friendly JavaScript modules - without Dependency Injection
-----------

The import statement uses a relative url to perform the module lookup. So, can I somehow configure a "base test suite url" to load fake modules that my module under test is dependent of? Would that require me to write a lot of stubs (files)? Is it even possible, without writing a custom module loader? That's the things I have had on my mind, for a couple of days now.

See, these days I mostly think about code and don't actually write code that much. How come? I am currently enjoying life as a swedish dad on paternity leave. The days with the one year old are great. We read books, build with blocks and laugh a lot. I love it.

When he is sleeping, I catch up on coding by reading blogs (mostly one of the great posts on ES6 by Dr. Axel Rauschmayer) or watching a video tutorial (I recommend the ES6 course on Pluralsight, by Scott Allen and Joe Eames). I am learning new things, slowly, but scheduled and at a steady pace.

Tonight, just when I had finished reading a Pippi Longstocking bedtime story to our four year old son (the big brother), the background process of the brain suddenly sent me a message: keep it simple. Use dependency injection. Dave, you know this already. I think the new syntax and the new ways of writing has blinded me, made me forget about concepts I normally use when writing c# or old school JavaScript (pre ES6).

How about writing modules with dependencies injected, by exposing something like an "init" function, instead of directly importing them? That would make them testable. Something like this:

// foo.js
let message = 'My name is foo';

var foo = {
    getFoo() {
        return message;
    }
};

export default foo;

// bar.js - dependent on foo.js 
let message = '';

var bar = {
    init(obj) {
        let foo = obj.getFoo();
        message = `${foo}bar, and I am foonky`;
    },
    getBar() {
        return message;
    }
};

export default bar;

//app.js
import foo from 'modules/foo';
import bar from 'modules/bar';

// inject dependencies
bar.init(foo);

console.log(bar.getBar());

 
Please let me know what you think about it!

-----------
Update: I have experimented with an alternative to the examples in this blog. Have a look at it and please share your thoughts and feedback on the different approaches to unit testing modules:
Test friendly JavaScript modules - without Dependency Injection

-----------

Twitter Design Pattern

På Twitter har man 140 tecken på sig att säga något, sedan tar det stopp. Texten ska helst vara läsbar, så frktngr får man vara sparsam med.

[raderna ovan innehåller 140 tecken]

Det jag speciellt gillar är att man som twittrare måste ta bort all onödig information för att kunna leverera korta meddelanden utan brus. Finns det här designmönstret på flera ställen?

ONE: #business
Ibland pratar man om att berätta hissversionen av ett förslag eller en affärsidé:

Vi levererar det stora företagets IT-kompetens med det lilla företagets själ och den enskilde konsultens engagemang.
[116 tecken]

Här i Sverige pratar vi ju inte med varandra i hissen, men vad sägs om twitterversionen?

TWO: #requirements
I den agila världen används redan Twitter Design Pattern, där man gärna skriver krav i form av User Stories. Många Scrum-team och beställare gillar kortfattade och tydliga beskrivningar av det som ska göras:

Som en redaktör vill jag kunna lägga till och ta bort nyheter på internetbanken, så att bankens kunder kan ta del av aktuell information.
[137 tecken]

TWEET: #programming?
Jag har inte sett några tydliga tecken på Twitter Design Pattern i utveckling av mjukvara ännu. Men tänk om vi bara fick skriva 140 tecken när vi programmerar fram en metod? Enligt designmönstret behöver jag (trots begränsningarna) skriva koden både läsbar och meningsfull, helst utan krångliga frKrtNgr. Vilken utmaning!

Jag tror att resultatet skulle kunna bli lika enkelt och tydligt som på Twitter:
man behöver ju sällan läsa tidigare tweets för att hänga med. Meddelanden är isolerade och externa beroenden tar man med som länkar, taggar och re:tweets (Dependency Injection, Interface och arv?).

Vilka fler områden finns det som skulle kunna följa Twitter Design Pattern, har du några exempel?

http://twitter.com/davidvujic