tag:blogger.com,1999:blog-14093522047997839922024-03-19T10:27:13.655+01:00Agile & CodingDavid Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.comBlogger112125tag:blogger.com,1999:blog-1409352204799783992.post-2308171671834685122024-02-18T21:55:00.015+01:002024-02-19T16:50:44.897+01:00Python Monorepo Visualization<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
blockquote.large sup {
font-size: 0.7em;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvHBUHHd1dWn7lZpS0fmces8lVG0MmxsJUNxHC8iwmivX6DHxTB1JTCTDVoid9scM9CLmbLCUE_NswTxjaqJ8iPkcV3LwL3gJQtTp97qa9zVCCuxvLeRVYwRYS3CxovGFqiE0ijrflZJJ6v6bm-817_Ro0RWOmA0K9vTNf99f5Duu05zRf2KtMBajfWGQ/s600/young-architects-small.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="344" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvHBUHHd1dWn7lZpS0fmces8lVG0MmxsJUNxHC8iwmivX6DHxTB1JTCTDVoid9scM9CLmbLCUE_NswTxjaqJ8iPkcV3LwL3gJQtTp97qa9zVCCuxvLeRVYwRYS3CxovGFqiE0ijrflZJJ6v6bm-817_Ro0RWOmA0K9vTNf99f5Duu05zRf2KtMBajfWGQ/s400/young-architects-small.png"/></a></div>
<p>
What's in a code repository? Usually you'll find the source code, some configuration and the deployment infrastructure - basically the things needed to develop & deploy something. It might be a service, an app or a library. A Monorepo contains the same things, but for more than one artifact. In there, you will find the code, configurations and infrastructure for <strong>several</strong> services, apps or libraries.
<br/><br/>
The main use case for a Monorepo is to share code and configuration between the artifacts (let's call them projects).
</p>
<h3>These things have to be simple</h3>
<p>
Sharing code can be difficult. Repos can be out of date. A Monorepo can be overwhelming. With or without a Monorepo, the most common way of sharing code is to package them as libraries that the projects can add as external dependencies. But managing different versions and keeping the projects up-to-date could lead to unexpected and unwanted extra work. Some Monorepos solve this by using symlinks to share code, or custom scripts for copying things into the individual projects during deployment.
<br/><br/>
Doing that can be messy, I've seen it myself. I was once part of a team that migrated away from a horrible Monorepo, into several smaller single-repo microservices. The tradeoffs:
source code spread out in repos with an <i>almost</i> identical structure. <i>Almost</i> is the key here. Also, code and config duplications.
</p>
<p>These tradeoffs have a negative impact on the Developer Experience.</p>
<p>
The Polylith Architecture has a different take on organizing and sharing code, with a nice developer experience. These things have to be simple. Writing code should be fun. Polylith is Open Source, by the way.
</p>
<h3>The most basic type of visualization</h3>
<p>
In a Polylith workspace, the source code lives in two folders named <i>bases</i> and <i>components</i>. The entry points are put in the bases folder, all other code in the components folder. At first look, this might seem very different from a mainstream Python single-project structure. But it isn't really that different. Many Python projects are using a <i>src</i> layout, or have a root folder with the same name as the app itself. At the top, there's probably an entry point named something like <i>app.py</i> or maybe <i>main.py</i>? In Polylith, that one would be put in the <i>bases</i> folder. The rest of the code would be placed in the <i>components</i> folder.
</p>
<pre>
components/
.../
auth
db
kafka
logging
reporting
...
</pre>
<p>
You are encouraged to keep the components folder simple, and rather put logically grouped modules (i.e. namespace packages) in separate components than nested structures. This will make code sharing more straightforward than having a folder structure with packages and sub-packages. It is also less risk of code duplication with this kind of structure, because the code isn't hidden in a complex folder structure. As a side effect, you will have a nice overview over the available features: the names of the folders will tell what they do and what's available for reuse. A folder view like this is surprisingly useful.
</p>
<h3>Visualize with the poly tool</h3>
<p>
Yes, looking at a folder structure is useful, but you would need to navigate the actual source code to figure out where it is used and which dependencies that are used in there. Along with the Polylith Architecture there is tooling support. For Python, you can use the tooling together with Poetry, Hatch, PDM or Rye.
</p>
<figure>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg_hr5Rq6Ir8yQr9C3mBePqG3boFWDMBlbLoCzz3W-WQWkt56wB4Pjmcu0iKD57fTEZyHg_-GjOODtWHtbkpwMac_-_c0r7hOx617b0I11TFH3k6l6aWsRleHcLpOzlSNfMX8thuisFuf5UIZT7hNWxo4UfBXGy5VnzGxT6tiBnk5yZQv3KpfDoTahVuc/s1463/poly-info.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1020" data-original-width="1463" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg_hr5Rq6Ir8yQr9C3mBePqG3boFWDMBlbLoCzz3W-WQWkt56wB4Pjmcu0iKD57fTEZyHg_-GjOODtWHtbkpwMac_-_c0r7hOx617b0I11TFH3k6l6aWsRleHcLpOzlSNfMX8thuisFuf5UIZT7hNWxo4UfBXGy5VnzGxT6tiBnk5yZQv3KpfDoTahVuc/s400/poly-info.png"/></a></div>
<figcaption>The poly info command, an overview of code and projects in the Monorepo.</figcaption>
</figure>
<p>
Besides providing commands for creating bases, components and projects there are useful visualization features. The most straightforward visualization is probably <code>poly info</code>. Here, you will get an overview of all the bricks (the logically grouped Python modules, living in the bases and components folders), the different projects in the Workspace and also in which projects the bricks are added.
</p>
<h3>Third-party libraries & usages</h3>
<p>
There's a command called <code>poly libs</code> that will display the third-party dependencies that are used in the Workspace (yes, that's what the contents of the Monorepo is called in Polylith). It will display libraries and the usages on a brick-level. In Polylith, a brick is the thing that you share across projects. Bricks are the building blocks of this architecture.
</p>
<figure>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Q82mxW1IVz3afySw-0DBzdd4K2UxvvrhhXkVQG64VSze_bS9-2rX_r_AeRw1kmSJB7C39JHAvkIlUeozUqFMnAfuANWkAUoo988xhzGWEKlompK4OLE4NBUx6MM6nUPxDHaJBkgTNTqzwWiZTPKvf8DteyO8WXksR4Xr61ZXBFd4d5L119tEKMLnPPs/s914/poly-libs-2.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="602" data-original-width="914" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Q82mxW1IVz3afySw-0DBzdd4K2UxvvrhhXkVQG64VSze_bS9-2rX_r_AeRw1kmSJB7C39JHAvkIlUeozUqFMnAfuANWkAUoo988xhzGWEKlompK4OLE4NBUx6MM6nUPxDHaJBkgTNTqzwWiZTPKvf8DteyO8WXksR4Xr61ZXBFd4d5L119tEKMLnPPs/s400/poly-libs-2.png"/></a></div>
<figcaption>The poly libs command, displaying the third-party dependencies and where they are used.</figcaption>
</figure>
<h3>The building blocks and how they depend on each other</h3>
<p>
A new thing in the Python tooling is the command called <code>poly deps</code>. It displays the bricks and how they depend on each other. You can choose to display an overview of the entire Workspace, or for an individual project. This kind of view can be helpful when reasoning about code and how to combine bricks into features. Or inspire a team to simplify things and refactor: should we extract code from this brick into a new one here maybe?
</p>
<figure>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi1ZDSuowLA4rnR3IZ-r3-B3RJ4RdlQd5rwKLRJAuxsfW9b8cP5OuTvfMZfVy-CJr0ccCfrVqI8GOYm9uhMf5XXtnS6uK4IL4VmgDt2bwACte8FDsj6WbmbruTUEr_MY6PEym4TJ1q7gUiiGgmM2MQa1GxvuNlqrrzpf8RgKIR-ZFGjCd-aLYepjmXlBE/s914/poly-deps.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="602" data-original-width="914" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhi1ZDSuowLA4rnR3IZ-r3-B3RJ4RdlQd5rwKLRJAuxsfW9b8cP5OuTvfMZfVy-CJr0ccCfrVqI8GOYm9uhMf5XXtnS6uK4IL4VmgDt2bwACte8FDsj6WbmbruTUEr_MY6PEym4TJ1q7gUiiGgmM2MQa1GxvuNlqrrzpf8RgKIR-ZFGjCd-aLYepjmXlBE/s400/poly-deps.png"/></a></div>
<figcaption>A closer look at the bricks used in a project with poly deps.</figcaption>
</figure>
<p>You can inspect a single brick to visualize the dependencies: where it is used, and what other bricks it uses.</p>
<figure>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtMvvc5h6Y6rc4aXaRXheDzXm_lDRGVOT7K02kgj8qH1RKcLPiPa3C1HrB46y73yntpXjj88NuSaIipXHk88XCgQ3miv2_ghGMTH2tW7wviOZDvLXEUYiEsu0Mhd1NfjxNe_gCuE8UtEyM7WQJIVrBpuzJ6EjrosFH-AGXXg2Aa_Ls_Z3ehtQYmgDujo4/s914/poly-deps-bricks.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="602" data-original-width="914" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtMvvc5h6Y6rc4aXaRXheDzXm_lDRGVOT7K02kgj8qH1RKcLPiPa3C1HrB46y73yntpXjj88NuSaIipXHk88XCgQ3miv2_ghGMTH2tW7wviOZDvLXEUYiEsu0Mhd1NfjxNe_gCuE8UtEyM7WQJIVrBpuzJ6EjrosFH-AGXXg2Aa_Ls_Z3ehtQYmgDujo4/s400/poly-deps-bricks.png"/></a></div>
<figcaption>A zoomed-in view, to inspect the usages of a specific brick.</figcaption>
</figure>
<h3>Export the visualizations</h3>
<p>
The output from these commands is very easy to copy-and-paste into Documentation, a Pull Request or even Slack messages.
<br/><br/>
<code>poly deps | pbcopy</code>
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikeOhFsd6cal8ioHbl2NZNNFtTc0lgJbyMeKaFAtaHBeNRO-4t1FzxoUGCyOTys9RDe7kZP25slbTTtHi84PziYMd5nr0FINESNbPrOtxkDgKdhunzIIHDeRzHzzPo9ncPIb87FLSEWIyID3vfpg51s-4sT1nXOjGB17Sh-Tnz4wbaPFbNN8Z9nf_3j_0/s1030/slack-poly-deps.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="698" data-original-width="1030" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikeOhFsd6cal8ioHbl2NZNNFtTc0lgJbyMeKaFAtaHBeNRO-4t1FzxoUGCyOTys9RDe7kZP25slbTTtHi84PziYMd5nr0FINESNbPrOtxkDgKdhunzIIHDeRzHzzPo9ncPIb87FLSEWIyID3vfpg51s-4sT1nXOjGB17Sh-Tnz4wbaPFbNN8Z9nf_3j_0/s400/slack-poly-deps.png"/></a></div>
<h3>📚 Docs, examples and videos</h3>
<p>
Have a look at the <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank" title="To the Python tools for the Polylith Architecture docs">the Polylith documentation</a> for more information about getting started. You will also find examples, articles and videos there for a quick start.
</p>
<br/><br/>
<sup>
<span>
Top image made with AI (DALL-E) and manual additions by a Human (me)
</span>
</sup>
</div>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-53312262839577769292024-01-25T11:52:00.003+01:002024-01-25T11:59:09.584+01:00Simple & Developer-friendly Python Monorepos<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
blockquote.large sup {
font-size: 0.7em;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdMyEmiI4p1dDKfveKUUxPSeBDD8U9JrQRSv8n9BxLAKRX1Y338IP8GncHC5FBu4sq9vHJ2mvvgzXsBw1XYEDtj1_f-_BeQYhKa20S3vsWwxpZ_oV36E8KrlnycV2k5WYo-b23iZrAlX2fr4MlMY6bdi4YRSANTjDOP6-w2up-W4G_p1tJeuRQxeex8KQ/s388/poetry-hatch-pdm-snakes-small.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="368" data-original-width="388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdMyEmiI4p1dDKfveKUUxPSeBDD8U9JrQRSv8n9BxLAKRX1Y338IP8GncHC5FBu4sq9vHJ2mvvgzXsBw1XYEDtj1_f-_BeQYhKa20S3vsWwxpZ_oV36E8KrlnycV2k5WYo-b23iZrAlX2fr4MlMY6bdi4YRSANTjDOP6-w2up-W4G_p1tJeuRQxeex8KQ/s400/poetry-hatch-pdm-snakes-small.png"/></a></div>
<p>🎉 Announcing new features 🎉</p>
<p>
Polylith is about keeping Monorepos Simple & Developer-friendly. Today, <a href="https://github.com/DavidVujic/python-polylith" target="_blank" title="to the Polylith GitHub page">the Python tools for the Polylith Architecture</a> has support for <strong>Poetry</strong>, <strong>Hatch</strong> and <strong>PDM</strong> - three popular Packaging & Dependency Management tools in the Python community.
</p>
<p>
In addition to the already existing Poetry plugin that adds tooling support for Polylith, there is also a brand new command line tool available. The CLI has support for both Hatch and PDM. You can also use it for Poetry setups (but the recommended way is to use the dedicated Poetry plugin as before).
<br/><br/>
<strong>To summarize</strong>: it is now possible to use the Simple & Developer-friendly Monorepo Architecture of Polylith with many different kinds of Python projects out there.
</p>
<blockquote class="large">
"Hatch is a modern, extensible Python project manager."<br/>
<sup><a href="https://hatch.pypa.io/" target="_blank" title="To the Hatch website">From the Hatch website</a></sup>
</blockquote>
<h3>🐍 Hatch</h3>
<p>
To make the tooling fully support Hatch, there is a Hatch build hook plugin to use - <i>hatch-polylith-bricks</i> - that will make Hatch aware of a Polylith Workspace. Hatch has a nice and well thought-through Plugin system. Just add the hook to the build configuration. Nice and simple! The Polylith tool will add the config for you when creating new projects:
</p>
<pre>
[build-system]
requires = ["hatchling", "hatch-polylith-bricks"]
build-backend = "hatchling.build"
</pre>
<blockquote class="large">
"PDM, as described, is a modern Python package and dependency manager supporting the latest PEP standards. But it is more than a package manager. It boosts your development workflow in various aspects."<br/>
<sup><a href="https://pdm-project.org/" target="_blank" title="To the PDM website">From the PDM website</a></sup>
</blockquote>
<h3>🐍 PDM</h3>
<p>
Just as with Hatch, there are build hooks available to make PDM aware of the Polylith Workspace. Writing hooks for PDM was really simple and I really like the way it is developed. Great job, PDM developers! There is a workspace build hook - <i>pdm-polylith-workspace</i> - and a projects build hook - <i>pdm-polylith-bricks</i> - to make PDM and the Polylith tooling work well together.
<br/><br/>
This is added to the workspace build-system section pyproject.toml:
</p>
<pre>
[build-system]
requires = ["pdm-backend", "pdm-polylith-workspace"]
build-backend = "pdm.backend"
</pre>
<p>And the plugin for projects.
<br/>
This will be added by the <code>poly create project</code> command for you.
</p>
<pre>
[build-system]
requires = ["pdm-backend", "pdm-polylith-bricks”]
build-backend = "pdm.backend"
</pre>
<blockquote class="large">
"Python packaging and dependency management made easy."<br/>
<sup><a href="https://python-poetry.org/" target="_blank" title="To the Poetry website">From the Poetry website</a></sup>
</blockquote>
<h3>🐍 Poetry</h3>
<p>For Poetry, just as before, add or update these two plugins and you're ready to go!</p>
<pre>
poetry self add poetry-multiproject-plugin
poetry self add poetry-polylith-plugin
</pre>
<h3>📚 Docs, examples and videos</h3>
<p>
Have a look at the <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank" title="To the Python tools for the Polylith Architecture docs">the Polylith documentation</a> for more information about getting started. You will also find examples, articles and videos there for a quick start. I'm really excited about the new capabilities of the tooling and hope it will be useful for Teams in the Python Community!
</p>
<br/><br/>
<sup>
<span>
Top photo made with AI (DALL-E) and manual additions by a Human (me)
</span>
</sup>
</div>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-22736065917627281362023-12-19T23:57:00.022+01:002023-12-22T11:11:57.030+01:00The Lost Balkan Tapes: a Christmas story<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaowIPQc8kJKc79seSankUb7G_lklK4faoCiNXPwG7xYypQLFcJyLJIqJKfw2Ervvzysngb5Je99NJiG5Kd_fY0HCmluWLcpMJZB_6VKJx84j3IUDZ8ZsaajvnSmkinBlpY4ZkYcjmYqModC0eubLxLfFvqC9yMkoASLIU10HZGNKlnV-bG6h8970KVY8/s679/Ikonija-Vujic.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="400" data-original-height="679" data-original-width="668" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaowIPQc8kJKc79seSankUb7G_lklK4faoCiNXPwG7xYypQLFcJyLJIqJKfw2Ervvzysngb5Je99NJiG5Kd_fY0HCmluWLcpMJZB_6VKJx84j3IUDZ8ZsaajvnSmkinBlpY4ZkYcjmYqModC0eubLxLfFvqC9yMkoASLIU10HZGNKlnV-bG6h8970KVY8/s400/Ikonija-Vujic.png"/>
</a>
</div>
<p>
A couple of days ago, a nice thing happened that has made me very happy. It happened almost by accident. At work, we have a <i>#music</i> channel on Slack and some of us frequently share music that we like and recommend to the folks in the company. I really like those kind of <i>work-but-not-related-to-the-actual-work</i> kind of things. It helps developing a friendly Organizational culture, and it is also a simple way to get to know each other better.
</p>
<h3>Background</h3>
<p>
I was born and raised in the Stockholm area of Sweden, but my dad and his parents (my grandparents) came to Sweden in the late 1960s from former Yugoslavia. They, like many from the Balkans, Finland, Italy and Greece, got job offerings from Swedish companies and they decided to give it a try. They began working with manufacturing chocolate here in Stockholm. I don't know that much about their life over there before Sweden, other than the stories I've heard many times when growing up.
</p>
<p>
One of my favorite stories is about my Grandfather and when he escaped from some kind of prison camp, set up by the World War II Occupation of former Yugoslavia. He was only a teenager back then, but managed to run away from the guards into a forest - and then catch a passing train, on its way away from the camp. In my imagination, the train was moving fast and he jumped on it in the same way <i>Indiana Jones</i> would do. I remember him telling me and my brother about going <i>undercover</i>, calling himself Ivan Something-something when the train conductor asked who that kid from nowhere was. I loved that story and wanted to hear it over and over again.
</p>
<p>
Another favorite story was about my Grandmother. She used to be a singer in the 1950s, early 1960s and performed all over the country. I remember the day she told me about the famous people she used to sing for, such as the <a href="https://en.wikipedia.org/wiki/Haile_Selassie" target="_blank">Ethiopian Emperor Haile Selassie</a>! When I grew up, <i>Reggae</i> and the <i>Rastafari</i> culture was a big thing among us kids in the suburbs, and I certainly knew about Haile Sellasie. My grandma has not only met him, but also has sung for him! Wow. As I understood it, she was popular and during a period of time she was often hired to sing when the officials of the Country expected a visit from abroad.
</p>
<p>
I have heard only a few songs on cassette when we hang out in the apartment in Fisksätra (a concrete suburb in the Stockholm area that I have spent a lot of time in). I remember the music sounding quite good. But honestly, as a kid and later a young adult, I wasn't really that much of a fan of Balkan Folk Music. Still very cool to hear her sing. She occasionally sang some songs for us too, but time & smoking cigarettes had made her voice different than from the recording of those cassette tapes.
</p>
<p>
Years later, with both grandma and grandpa no longer around, I thought that the tapes were lost & gone forever. If I recall it correctly, some of it was even accidentally overwritten. Oh no. I made some attempts to find information (and possibly music) online, but failed. I gave up hope of finding anything. This was many years ago.
</p>
<h3>Today</h3>
<p>
Back to work. I decide to share some nice Reggae music in the Slack channel. I found the <i>Gratitude Riddim</i> when browsing my online music app, good stuff! The friends at work got inspired and also shared some Jamaican vibes and we had a fun conversation going on there. Then, one person added a picture of Haile Selassie that took me right back to those childhood memories. So, naturally, I told them the Story about my Grandmother and that she has sung for him. But sadly the music is gone, you know. My friend probably got curious, and I believe he just googled her name.
</p>
<p>
<i>”Wow, very cool to hear about your grandma. Is this her?”</i>
<br/>(with a link containing music)
</p>
<h3>What?</h3>
<p>
What? Huh? No, wait. That can't be her. Or. Is it? Naw. Gotta be someone else. Then again, how many Folk Music singers from the Balkans named <strong>Ikonija Vujic</strong> can there be? After a while, I realized that it is in fact my Grandmother! He actually found about 15 songs. All of them beautiful & melancholic Love Songs. Old school Balkan Folk Music with the <i>Tamburitza</i> instrument. Is it the great <a href="https://en.wikipedia.org/wiki/Janika_Bala%C5%BE" target="_blank">Janika Balaž</a> playing? This is just me guessing, but according to Wikipedia he lived in the exact same area where my grandparents (and my dad) used to hang out back in the days. They must have met sometime!
</p>
<p>
I can't think of a better Christmas gift than this and I am forever grateful about these findings. It happened almost randomly, by accident. If I hadn't shared those Reggae songs in Slack, I would still be thinking that the music of my Grandmother was gone. But the <strong>Lost Balkan Tapes of Ikonija Vujic</strong> have been found again. Thanks to my friend at work and the enthusiast person that has published a huge amount of Balkan Music from the past. Thank you! 🙏
</p>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-28187081161013663712023-11-15T20:50:00.001+01:002023-11-15T21:10:25.169+01:00A Coding Copilot<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLYskoOdxzjrGPFxFzv5Q89VrkLbifbKSndsCld3yjKMyJHYvsweuKo9obJS9SVXztyCEKSRfkbwcHmOyv5CBIT_hccgPFOQHvYBDg1gxR2VWnWhXBCpjEZYkPjktf0uCS8MCfeRJ6BzJy1gYC4Ydvvtuc-kE9LxiImgCqAGEUO9rHSWG8o5wqLv9x3Mc/s1920/jamie-templeton-6gQjPGx1uQw-unsplash-2.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLYskoOdxzjrGPFxFzv5Q89VrkLbifbKSndsCld3yjKMyJHYvsweuKo9obJS9SVXztyCEKSRfkbwcHmOyv5CBIT_hccgPFOQHvYBDg1gxR2VWnWhXBCpjEZYkPjktf0uCS8MCfeRJ6BzJy1gYC4Ydvvtuc-kE9LxiImgCqAGEUO9rHSWG8o5wqLv9x3Mc/s400/jamie-templeton-6gQjPGx1uQw-unsplash-2.jpg"/></a></div>
<p>
When developing software, I spend most of the time thinking & talking about the problems to solve: the <i>what</i>, the <i>why</i> and the <i>how</i>. The thinking-part works best for me when I'm actually away from the desktop, such as walking outdoors or even when grabbing a cup of coffee & a banana by the office kitchen. 🍌
</p>
<p>
Talking is a really nice thing, we should try that more often in combination with listening to what others have say. Even when not having any human around to chat with, there is always the rubber duck. Speak to it!
</p>
<h3>Problem Solving</h3>
<p>
Mob programming takes the Thinking & Talking to the next level. I am, strangely enough, surprised almost every time how great mob and pair programming is for software development. I tend to forget that. It's probably easy to fall back into old familiar behavior.
</p>
<p>
When figuring out what to try out, the actual typing is done pretty fast for me. A big part of the productivity boost is inspiration and the joy of solving problems with programming. I think this is valid for most creative and problem-solving work.
</p>
<h3>Everyday practicing</h3>
<p>
I try hard every day to write clear & concise code. Practicing, learning ... and walking. I really like the Functional approach to programming: separating data from behavior, actions from calculations. I should probably use GitHub Copilot or ChatGPT more, but haven't yet found the need to go all-in (aren't the AI suggestions a bit verbose, or am I doing things wrong?). Those types of in-editor copilots are cool, definitely, and are for sure here to stay. Currently, I find more value in a different kind of automated guidance.
</p>
<h3>CodeScene</h3>
<p>
When working on my Open Source Project, <i>the Python tools for the Polylith Architecture</i>, I am guided by tools like SonarCloud and CodeScene. I guess there are some AI involved quite heavily in there too. I especially appreciate the code reviews & even more the CodeScene long-term analysis about the effects of Tech Debt. It's like having a co-pilot, or even a Teacher, guiding you during the development to make you a better software developer than before.
<br/><br/>
This is different from the kind of review a team of humans usually do when sharing feedback on Pull Requests or even during mob programming sessions.
<br/><br/>
Even if I sometimes disagree with things the tools report on being a problem, I go ahead and refactor the code anyway. Then, when doing the refactoring, I quickly realize that the code is actually transforming into something simpler and clearer than before. SonarCloud is more into the low level details, and lets me know about code duplications or code smells. We don't want that, no. Better fix that too.
</p>
<h3>Coding Style</h3>
<p>
The feedback from these kind of tools is helping me to get closer to writing the type of code I want to write: <strong>minimalistic</strong>, <strong>readable</strong> and <strong>functional</strong>. A coding style that also is a very good fit for the Polylith Architecture, even if that thing isn't about how the code is written. You are encouraged to write in a readable & functional style by the structuring of code and by having all code at your fingertips, ready to experiment with in the REPL or in a Notebook.
<br/><br/>
CodeScene, SonarCloud and Polylith: all of them helping me in my journey to be a better Developer. Maybe I'll throw in Copilot in there too eventually.
</p>
<sup>
<span>
Top Photo by <a href="https://unsplash.com/@jamietempleton?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Jamie Templeton</a> on <a href="https://unsplash.com/photos/brown-wooden-plank-fence-with-this-way-signboard-6gQjPGx1uQw?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Unsplash</a>
</span>
</sup>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-11119706570172460982023-09-30T12:29:00.001+02:002023-09-30T12:51:15.741+02:00Software and Projects<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2SofhTKTxCf29TNT2NTCT9AmBT8UEp7h11_7IVAE_fIU-iUqTHk429ZxTlXblgZke0xWf_r72znEMMMFVGmgrjsTdgtlTifWMsQUhaeWoUX0Jyh-lWpaZDdpS7QElmorRW5kKV8WeU2c1-5EZp1N1GWVtJLh7W4t4ful4kZaDIu9EMV_pCGOGIkhia3g/s1920/patrick-fore-JBghIzjbuLs-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2SofhTKTxCf29TNT2NTCT9AmBT8UEp7h11_7IVAE_fIU-iUqTHk429ZxTlXblgZke0xWf_r72znEMMMFVGmgrjsTdgtlTifWMsQUhaeWoUX0Jyh-lWpaZDdpS7QElmorRW5kKV8WeU2c1-5EZp1N1GWVtJLh7W4t4ful4kZaDIu9EMV_pCGOGIkhia3g/s400/patrick-fore-JBghIzjbuLs-unsplash.jpg"/></a></div>
<blockquote class="large">How can we simplify things, to add value as early as possible?</blockquote>
<p>
Software Projects is a misconception within Software Development. When organizing work into Projects, one might think that there also is a need to estimate the things to be done. Because the Project itself is by definition something that will end, <strong>or more likely be delayed</strong>.
<br/><br/>
From my experience, estimation is only adding anxiety and sometimes actually delaying value. I have experienced the same thing with Projects and fixed release dates.
</p>
<p>Instead, we could flip it: how can we simplify, to add value for our users as early as possible?</p>
<h3>Flip it!</h3>
<p>
In a team that I was part of some time ago, the Product Managers challenged us by asking those kind of questions. We (the Developers) were at the beginning intimidated, because we had the idea of building the <i>perfect</i> kind of feature and usually we spent time on figuring out & cover as many scenarios as we could imagine. The not-great thing with that approach is that it will likely increase the cost, because it would take longer time and more effort to deliver value for the users of the product.
</p>
<h3>Think Different</h3>
<p>
The Product Management folks kept challenging us to think different, and we begun to find ways of delivering smaller parts of the product within days (instead of months). We wouldn't be able to release an entire feature, of course. Far from it. Instead, it would be something - sometimes just a tiny addition - that people could use to simplify their daily work life. They actually did that, we saw it ourselves. There was value added, within a couple of days of development!
</p>
<h3>Smaller parts</h3>
<p>
There was no need for us (the Dev Team) to put any <i>Fibonacci</i> numbers or estimating hours to the tasks we identified should be developed. We stopped doing those stressful <i>Big Bang Releases</i> of huge chunks of code. Instead, we found ways to simplify tasks - splitting them out into several smaller parts - to make it possible to push the work to production within days. In my opinion, <i>Projects</i>, <i>Estimation</i> and <i>Deadlines</i> are rarely adding value. Why are there so many Organizations out there still doing that?
</p>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@patrickian4?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Patrick Fore</a> on <a href="https://unsplash.com/photos/JBghIzjbuLs?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-10866136891236115572023-08-03T16:11:00.002+02:002023-08-09T11:07:31.196+02:00Kafka messaging with Python & Polylith<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNhGoKMppCEh18cOxzHE2Mx3XZlNkuBvp94pP1Bfra8i00sS452jkrqL-b14HEcZ4M_I_VEiOCE1xcnklVOQfxTgK289Ug_lu97cvpOHDj03C6CakHe9DQSxON9ZXf45AZ96tfh4eGumNRmL2oTObBaCb9uD0c5xlXsbZQTggYu4WPFBWAK2ZSbEKGl7M/s1920/thomas-peham-bo48lBVP-Xs-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNhGoKMppCEh18cOxzHE2Mx3XZlNkuBvp94pP1Bfra8i00sS452jkrqL-b14HEcZ4M_I_VEiOCE1xcnklVOQfxTgK289Ug_lu97cvpOHDj03C6CakHe9DQSxON9ZXf45AZ96tfh4eGumNRmL2oTObBaCb9uD0c5xlXsbZQTggYu4WPFBWAK2ZSbEKGl7M/s400/thomas-peham-bo48lBVP-Xs-unsplash.jpg"/></a></div>
<p>
This article isn't about Franz Kafka or his great novella <i>The Metamorphosis</i>, where the main character one day realizes that he has transformed into a Human-size Bug.
<br/><br/>
It is about a different kind of Kafka: <strong>Apache Kafka</strong>, with examples on how to get started producing & consuming messages with <strong>Python</strong>. All this in a <strong>Polylith Monorepo</strong> (hopefully without any of the bugs from that Franz Kafka novella).
</p>
<p>
This article can be seen as Part III of a series of posts about Python & Polylith. Previous ones are:
</p>
<ol>
<li><a href="https://davidvujic.blogspot.com/2023/07/gcp-cloud-functions-with-python-and-polylith.html">GCP Cloud Functions with Python and Polylith</a></li>
<li><a href="https://davidvujic.blogspot.com/2023/07/python-fastapi-microservices-with-polylith.html">Python FastAPI Microservices with Polylith</a></li>
</ol>
<p>
If you haven't heard about Polylith before, it's about Developer Experience, sharing code and keeping things simple. You will have all your Python code in a Monorepo, and develop things without the common Microservice trade offs. Have a look at the docs: <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank">Python tools for the Polylith Architecture</a>
</p>
<p>
<i>Edit: don't know what Kafka is? Have a look at <a href="https://kafka.apache.org/quickstart" target="_blank">the official Apache Kafka quickstart</a>.</i>
</p>
<p>
I will use the <i>confluent-kafka</i> library and have read up on the <a href="https://developer.confluent.io/get-started/python/" target="_blank">Confluent Getting Started Guide</a> about writing message Producers & Consumers.
<br/><br/>
The Polylith Architecture encourages you to build features step-by-step, and you can choose from where to begin. I have an idea about producing <strong>events</strong> with Kafka when items have been stored or updated in a database, but how to actually solve it is a bit vague at the moment. What I do know is that I need a function that will produce a message based on input. So I'll begin there.
<br/><br/>
All code in a Polylith Workspace is referred to as bricks (just like when building with LEGO). I'll go ahead and create a <i>Kafka brick</i>. I am going to use the Python tooling for Polylith to create the brick.
<br/><br/>
Note: I already have a Workspace prepared, have a <a href="https://davidvujic.github.io/python-polylith-docs/setup/" target="_blank">look at the docs</a> for how to set up a Polylith Workspace. Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8eCUJmQeuSsHsYAH25o3LC-juzpsb9icpBgtk0ai4wpzbE7b-WEEo_lfdfqHQ5NA7FRzAknHVSAJN4QP0N_AfWRMGwsvghF-fDIoCMPhlexGaP_KnsKgVdBZ_R9VFkd3YlebxRsbIeHNZJzzqnwXXH0zgn1bkPZ0tB1b3gD8mV_DcvV1PyNkpUp-erUI/s1360/poly%20create%20component%20kafka.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="252" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8eCUJmQeuSsHsYAH25o3LC-juzpsb9icpBgtk0ai4wpzbE7b-WEEo_lfdfqHQ5NA7FRzAknHVSAJN4QP0N_AfWRMGwsvghF-fDIoCMPhlexGaP_KnsKgVdBZ_R9VFkd3YlebxRsbIeHNZJzzqnwXXH0zgn1bkPZ0tB1b3gD8mV_DcvV1PyNkpUp-erUI/s400/poly%20create%20component%20kafka.png"/></a></div>
<p>
The poly tool has created a <i>kafka</i> Python package, and placed it in the components folder. It lives in a top namespace that is used for all the bricks in this Polylith Workspace. I have set it to <code>example</code> here, but you would probably want an organizational name or similar as your top namespace.
</p>
<pre>
bases/
components/
example/
kafka/
__init__.py
core.py
</pre>
<p>
There's two types of bricks in Polylith: <strong>components</strong> and <strong>bases</strong>. A component is where you write the actual implementation of something. A base is the entry point of an app or service, such as the entry point(s) of a <i>FastAPI</i> microservice or the main function of a CLI. In short: a base is a thin layer between the outside world and the components (containing the features). I will develop the base for my new Kafka feature in a moment.
</p>
<h3>A Producer and a Consumer</h3>
<p>
For this example kafka component, I will use code from the Confluent Python guide (with a little bit of refactoring).
</p>
<pre>
def produce(topic: str, key: str, value: str):
producer = get_producer()
producer.produce(topic, value, key, callback=_acked)
producer.poll(10000)
producer.flush()
</pre>
<p>
Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
<br/><br/>
I'll go ahead and write a message consumer while I'm at it, and decide to also put the Consumer within the kafka component.
</p>
<pre>
def consume(topic: str, callback: Callable):
consumer = get_consumer()
consumer.subscribe([topic])
try:
while True:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
logger.error(msg.error())
else:
topic, key, value = parse(msg)
callback(topic, key, value)
except KeyboardInterrupt:
pass
finally:
consumer.close()
</pre>
<p>
The kafka component now looks like this after some additional coding & refactoring:
</p>
<pre>
kafka/
__init__.py
consumer.py
core.py
parser.py
producer.py
</pre>
<h3>Running a Kafka server locally</h3>
<p>
I continue following along with the Confluent guide to run Kafka locally, and have added a Docker Compose file. I am storing that one in the <strong>development</strong> folder of the Polylith workspace.
</p>
<pre>
development/
kafka/
docker-compose.yml
</pre>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTEYr_HblKAeDO6EMFz9EphaOFmIvK-az8IGA-InyXC5nHGNzaUyuEtp8KWjGtsJoHtr18jkUh9oZiuVoF_3h0-Benn8HrLkFKY1tk7TOXKb3SEaHaQX2IXY8HlM8a_S9dw4EpBtTiiu2BmOuuYESOT_Al1xA0-AM0kEvrOjxTEvnnVI1X_bpDBi9Bg6A/s1360/docker%20compose%20up.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="234" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTEYr_HblKAeDO6EMFz9EphaOFmIvK-az8IGA-InyXC5nHGNzaUyuEtp8KWjGtsJoHtr18jkUh9oZiuVoF_3h0-Benn8HrLkFKY1tk7TOXKb3SEaHaQX2IXY8HlM8a_S9dw4EpBtTiiu2BmOuuYESOT_Al1xA0-AM0kEvrOjxTEvnnVI1X_bpDBi9Bg6A/s400/docker%20compose%20up.png"/></a></div>
<p>
I can now try out the Producer and Consumer in a <strong>REPL</strong>, making sure messages are correctly sent & received without any <i>Kafkaesque</i> situations (👴 🥁).
</p>
<h3>Producing a message</h3>
<p>
I already have a messaging API in my <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a> repo, with endpoints for creating, reading, updating and deleting data. This is the acual service that I want to extend with Kafka messaging abilities. The service is built with FastAPI and the endpoints are found in a base.
</p>
<pre>
base/
example/
message_api/
__init__.py
core.py
</pre>
<pre>
@app.post("/message/", response_model=int)
def create(content: str = Body()):
return message.create(content)
</pre>
<p>
I'll continue the development, by adding the newly created kafka component. While developing, I realize that I need to transform the data into simple data structures - and remember that I already have a component that can be used here. This is where Polylith really shines: developing these kind of smaller bricks makes it easy to re-use them in other places - just by importing them.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDSKzY6o-fXrlCgjMPqnbaSALpXVAxg428SzTtlss3MLfA1uDvtf24U-CKlAeFg97C3WBXxmEDZ4sgjrcjYn9safkkLco3dWidwNoXGpnXr46h9fcx5HMPjbmQtGMihJBb9HK8MxqHrcwYpRsGZxMDGAl8c-KGWpEawig6NB31c26-DmSb4dLaBSFvMtc/s1360/%20message-crud-create.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="430" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDSKzY6o-fXrlCgjMPqnbaSALpXVAxg428SzTtlss3MLfA1uDvtf24U-CKlAeFg97C3WBXxmEDZ4sgjrcjYn9safkkLco3dWidwNoXGpnXr46h9fcx5HMPjbmQtGMihJBb9HK8MxqHrcwYpRsGZxMDGAl8c-KGWpEawig6NB31c26-DmSb4dLaBSFvMtc/s400/%20message-crud-create.png"/></a></div>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDVyOzou2ZHkIhc3LTDna-pJNDdQkRu1Xtt87v4nijHo4vR2ilSJtXlsQw4jR-o7eIzyPhfpfhFTQoiuqrIveAyceeWrVCoZg_q5MMGu0MiMvKdYEZgZohDDKQHpVVYqROUT-W6z0A4PIE6iOaVq0P4bU-_nJwlQ32wAv9SAJdWCsm-HkwcHj10O-0gT4/s1360/notify-kafka.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="642" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDVyOzou2ZHkIhc3LTDna-pJNDdQkRu1Xtt87v4nijHo4vR2ilSJtXlsQw4jR-o7eIzyPhfpfhFTQoiuqrIveAyceeWrVCoZg_q5MMGu0MiMvKdYEZgZohDDKQHpVVYqROUT-W6z0A4PIE6iOaVq0P4bU-_nJwlQ32wAv9SAJdWCsm-HkwcHj10O-0gT4/s400/notify-kafka.png"/></a></div>
<h3>Consuming messages</h3>
<p>
I have the kafka component with a consumer in it, and now is the time when I create a new <strong>base</strong>: the entry point for my kafka consumer.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiemCAlTcPAicZCLfI8nz7-QrtPRSyX7n9Gxuat8XlGg2LDnT4bIKgCK7r0UD9pBfMFHl4uL08tUTERtBeKQ6khh827Zu_0uS2cpHej-fXeowYkXjubCSEddxv6juFufPprn-vq_opaDwA4Gl7s5n0YNU4qJ_9a6Vd5SG_qFDkSglN1RArp4WS-I0EpvEE/s1360/poly%20create%20base%20consumer.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="240" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiemCAlTcPAicZCLfI8nz7-QrtPRSyX7n9Gxuat8XlGg2LDnT4bIKgCK7r0UD9pBfMFHl4uL08tUTERtBeKQ6khh827Zu_0uS2cpHej-fXeowYkXjubCSEddxv6juFufPprn-vq_opaDwA4Gl7s5n0YNU4qJ_9a6Vd5SG_qFDkSglN1RArp4WS-I0EpvEE/s400/poly%20create%20base%20consumer.png"/></a></div>
<p>This is the code I add to the base. Note the re-use of another already existing Polylith component (the log).</p>
<pre>
from example import kafka, log
logger = log.get_logger("Consumer-app-logger")
def parse_message(topic: str, key: str, value: str):
logger.info(f"Consumed message with topic={topic}, key={key}, value={value}")
def main():
topic = "message"
kafka.consumer.consume(topic, parse_message)
</pre>
<h3>Adding a project</h3>
<p>
I now have all code needed for this feature. What is left is to add the infrastructure for it, the actual deployable artifact. This command will create a new entry in the projects folder.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_RZN4CrQuBRIRCZDENUm0RV6FmNARnDBRvG4p_rPCoZfHiuxDZY6OOtbm7RRBi5tyWmk8EXO9JKsxsv258yOYVA77zD307PLJ3kkfhhBj5pMD2XfsUpIGImzE93frA-yBBv0WOLQ5cK2QxPYHfMlJtj6gtT-IMQdX2jdnoDOK-kaBfgCk9d10EOh68gM/s1360/poly%20create%20project%20consumer.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="234" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_RZN4CrQuBRIRCZDENUm0RV6FmNARnDBRvG4p_rPCoZfHiuxDZY6OOtbm7RRBi5tyWmk8EXO9JKsxsv258yOYVA77zD307PLJ3kkfhhBj5pMD2XfsUpIGImzE93frA-yBBv0WOLQ5cK2QxPYHfMlJtj6gtT-IMQdX2jdnoDOK-kaBfgCk9d10EOh68gM/s400/poly%20create%20project%20consumer.png"/></a></div>
<pre>
projects/
consumer_project/
pyproject.toml
</pre>
<p>
I'm adding the dependencies and needed packages to the project-specific pyproject.toml file. But I am lazy, and will only add the base in the packages section - and then run <code>poetry poly sync</code>. It will add all needed bricks for this project. The poly tool has some magic in it, yes.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1T_Sc3ZIRl_UP9vkqrs3k5bIiOGzUrVZiHz_tttO3-8Ixddh8tbide_gqAdXpqXJ_jxE8TbiZi5b1xZRpe0Lr1klcsOqa90Q8mHPGuij5g_Lg7mEw-vcqfK29GZptjmN9yZkznkwcdRLi90j5I_qiAsU_4GxjQAe5uFQaCp5qSirYGHQWsFNM_S2-47s/s1360/consumer-pyproject-toml.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="642" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1T_Sc3ZIRl_UP9vkqrs3k5bIiOGzUrVZiHz_tttO3-8Ixddh8tbide_gqAdXpqXJ_jxE8TbiZi5b1xZRpe0Lr1klcsOqa90Q8mHPGuij5g_Lg7mEw-vcqfK29GZptjmN9yZkznkwcdRLi90j5I_qiAsU_4GxjQAe5uFQaCp5qSirYGHQWsFNM_S2-47s/s400/consumer-pyproject-toml.png"/></a></div>
<p>
When deploying, just use the <a href="https://davidvujic.github.io/python-polylith-docs/deployment/" target="_blank">build-project</a> command to package it properly without any relative paths, and use the built <strong>wheel</strong> to deploy it where you want it running. That's all!
</p>
<p>
In this article, I have written about the usage of Polylith when developing features and services, and Kafka messaging in specific. Adding new features & re-using existing code is a simple thing when working in a Polylith workspace. Don't hesitate to reach out if you have feedback or questions.
</p>
<h3>Additional info</h3>
<p>
Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
<br/>
Docs about Polylith: <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank">Python tools for the Polylith Architecture</a>
</p>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@tompeham?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Thomas Peham</a> on <a href="https://unsplash.com/photos/bo48lBVP-Xs?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-8937596447699485642023-07-21T14:59:00.018+02:002023-07-24T11:07:51.794+02:00Python FastAPI Microservices with Polylith<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjjnqbZSREYj1P7vj2T5x1FrIsia_JFHx2IKybRNmAMKtccRGMMJ1aAKLp5l9Bb0pPT8hXqjfVUXswG1kVLNhwHiZpcbd6wi8L0yvVSH5WDfDo4jroAF11sG-hEXyikdLRLMT4rn6lA_HPDIpJVQD2IE2V_hN2kYR9xRdFkWBJmWoEhp0zyF0fyDPC4EE/s1920/yulia-matvienko-kgz9vsP5JCU-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjjnqbZSREYj1P7vj2T5x1FrIsia_JFHx2IKybRNmAMKtccRGMMJ1aAKLp5l9Bb0pPT8hXqjfVUXswG1kVLNhwHiZpcbd6wi8L0yvVSH5WDfDo4jroAF11sG-hEXyikdLRLMT4rn6lA_HPDIpJVQD2IE2V_hN2kYR9xRdFkWBJmWoEhp0zyF0fyDPC4EE/s400/yulia-matvienko-kgz9vsP5JCU-unsplash.jpg"/></a></div>
<p>
<strong>I like FastAPI.</strong> It's is a good choice for developing Python APIs. FastAPI has a modern feel to it, and is also easy to learn. When I think about it, I could say the same about <strong>Polylith</strong>.
</p>
<h3>Modern Python Microservices</h3>
<p>
Building Python services with FastAPI and Polylith is a pretty good combination. With FastAPI, you'll have a modern framework including tools like <strong>Pydantic</strong>. The simplicity of the Polylith Architecture - using bases & components - helps us build simple, maintainable, testable, and scalable services.
<br/><br/>
You will have a developer experience without the common Microservice struggles, such as code duplication or maintaining several different (possibly diverging) repositories.
</p>
<h3>Getting started</h3>
<p>
Okay, let's build something. I will write a simple CRUD service that will handle messages. Since I'm starting from a clean sheet, I can choose where to begin the actual coding. I'd like to start with writing a function in a <i>message</i> module, and will figure out the solution while coding. The Polylith tooling support (totally optional) will help me add new code according to the structure of architecture. I already have a Workspace prepared, have a <a href="https://davidvujic.github.io/python-polylith-docs/setup/" target="_blank">look at the docs</a> for how to set up a Polylith Workspace. Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJoHkPSMN0qJGXEO6PaOmKNAg_ZbH4EP7CFAJq-mFx_7nSGJrYnUful0N9iy4b6By1N0gGjcYFSGbewYXcz7lCEy_giNPGWLM3jcOXNoBhNzuBXcTrE0UPLRfE1ensLOWdfLlZ6NtB0hTDMOy1yO48ofyUbWTjnAhZqO2wR6sE_ksQlkR9Fn-cSaGfD-M/s1360/poly-create-component-message-fixed.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJoHkPSMN0qJGXEO6PaOmKNAg_ZbH4EP7CFAJq-mFx_7nSGJrYnUful0N9iy4b6By1N0gGjcYFSGbewYXcz7lCEy_giNPGWLM3jcOXNoBhNzuBXcTrE0UPLRfE1ensLOWdfLlZ6NtB0hTDMOy1yO48ofyUbWTjnAhZqO2wR6sE_ksQlkR9Fn-cSaGfD-M/s400/poly-create-component-message-fixed.png"/></a></div>
<p>
Great! Now I have a namespace package that is ready for coding. I'll continue with writing a <i>create</i> function. While working on it, I realize I need a persistent storage. I decide to try out <strong>SQLAlchemy</strong> and the Session thing.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiDJHIyg8mT4Y_YHMLnUQn38UasqbqjbKFxm_6RoG4UeuwIGao8kXcAW9ntYimHtw-XPafX2usIKRq76Ht-uVg2inoZBf9aV55Nmyj2NKn4wt3vNG5Ym1-0-gAu3OHKs6OKY5ytzZTUs3osW_kXJSHRa9ymnqLyRWvPOFbtUfAAsy0ZhmSa7TWvfk3mpw/s1360/poetry-add-sqlalchemy-fixed.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiDJHIyg8mT4Y_YHMLnUQn38UasqbqjbKFxm_6RoG4UeuwIGao8kXcAW9ntYimHtw-XPafX2usIKRq76Ht-uVg2inoZBf9aV55Nmyj2NKn4wt3vNG5Ym1-0-gAu3OHKs6OKY5ytzZTUs3osW_kXJSHRa9ymnqLyRWvPOFbtUfAAsy0ZhmSa7TWvfk3mpw/s400/poetry-add-sqlalchemy-fixed.png"/></a></div>
<pre>
def create(content: str) -> int:
with Session.begin() as session:
data = crud.create(session, content)
return data.id
</pre>
<p>
Again, I will use the poly tool to create a <i>database</i> component, where I will put all the SQLAlchemy setup along with a model and the function where data is added to the DB.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9jPZVPqPjkHZY09GgZw6xpdPEI9QTRK1k8DZ35The7MsOaj7kyOQvbDAd1Nu8k5PeO5DF9jRJCY2I-iKnD7z2_mndPiTa9WuT3HE96JZAUMM-SFzNX_iibdTxMK0iOPj_xif2TCyeRFP-qYFI5-IDypSDY9s0SYcERTjuaeMKRtX1Sgo0IGM9d2YD5_k/s1360/poetry-poly-create-component-database-fixed.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9jPZVPqPjkHZY09GgZw6xpdPEI9QTRK1k8DZ35The7MsOaj7kyOQvbDAd1Nu8k5PeO5DF9jRJCY2I-iKnD7z2_mndPiTa9WuT3HE96JZAUMM-SFzNX_iibdTxMK0iOPj_xif2TCyeRFP-qYFI5-IDypSDY9s0SYcERTjuaeMKRtX1Sgo0IGM9d2YD5_k/s400/poetry-poly-create-component-database-fixed.png"/></a></div>
<pre>
def create(session: Session, content: str) -> Message:
data = Message(content=content)
session.add(data)
session.flush()
return data
</pre>
<p>In this example, I will go for <i>SQLite</i> to keep things simple. Use your favorite data storage here.</p>
<pre>
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
DATABASE_URL = "sqlite:///./sql_app.db"
DATABASE_ARGS = {"check_same_thread": False}
engine = create_engine(DATABASE_URL, connect_args=DATABASE_ARGS)
Session = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()
</pre>
<p>
I now have the basics done and am ready to write the endpoint.
<br/><br/>
<i>I could (of course) also have started from the endpoint, if I wanted to. Polylith allows you to postpone design decisions until you are ready for it.</i>
<br/><br/>
Before continuing, I will add the components to the packages section of the workspace pyproject.toml:
</p>
<pre>
packages = [
{include = "my_namespace/database", from = "components"},
{include = "my_namespace/message", from = "components"},
]
</pre>
<h3>The Base Plate</h3>
<p>
API endpoints are recommended to have in what is called a <strong>base</strong> in Polylith. The idea is very much like <strong>LEGO</strong>, when building things with bricks and place them on a base plate. Yes, the FastAPI endpoint is the base plate. I will create a new <strong>base</strong> with the poly tool, and add the FastAPI code in there. It will import the message component I wrote before, and consume the <i>create</i> function.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKfvwq_80YvgBZIy1nqGxZUkYPLqMECzetLsIsvRXE3Wxcs1JE0glGSoTQedNbDv_gJq0mYonvBdyqz83sv5Xvv0GsnJGek96EMhcYoeflkPbs5uv4lTJsJ7u5sdb9RJSivsJPognhE3K7ZZgP9CH0WQf-jb2vuc6TMahQwdzwbOaOUW1ImLTFie4nNKg/s1360/poetry-poly-create-baase-fixed.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKfvwq_80YvgBZIy1nqGxZUkYPLqMECzetLsIsvRXE3Wxcs1JE0glGSoTQedNbDv_gJq0mYonvBdyqz83sv5Xvv0GsnJGek96EMhcYoeflkPbs5uv4lTJsJ7u5sdb9RJSivsJPognhE3K7ZZgP9CH0WQf-jb2vuc6TMahQwdzwbOaOUW1ImLTFie4nNKg/s400/poetry-poly-create-baase-fixed.png"/></a></div>
<pre>
@app.post("/message/", response_model=int)
def create(content: str = Body()):
return message.create(content)
</pre>
<p>Just as with the components, I will add the base to the packages section of the pyproject.toml:</p>
<pre>{include = "my_namespace/message_api", from = "bases"},</pre>
<p>
I can now try out the API in the browser (I have already added the FastAPI dependency). Developing & trying out code is normally done from the development area (i.e. the root of the workspace). This is where you have all the things set up, <strong>all code</strong> and <strong>all dependencies</strong> in one single virtual environment. This is a really good thing for a nice Developer Experience.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghTPb_1yNOBPMiN0XzxOQRUZFFjPFHaAER3ku1wF3hhD_yIEC8OKjDJKc1iu_irKimlLRjExPg3IL92vnhmsbM3aFJNm8CK-ATS3Q0LE_iflNNBobq6gZ32CzQcB9J-tQUQaMf9xheJ-aDb10c5JDuSlUPR7rqk9YNG09fAwjfBlYx9506-eA7jX1LNhU/s1196/run-uvicorn.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghTPb_1yNOBPMiN0XzxOQRUZFFjPFHaAER3ku1wF3hhD_yIEC8OKjDJKc1iu_irKimlLRjExPg3IL92vnhmsbM3aFJNm8CK-ATS3Q0LE_iflNNBobq6gZ32CzQcB9J-tQUQaMf9xheJ-aDb10c5JDuSlUPR7rqk9YNG09fAwjfBlYx9506-eA7jX1LNhU/s400/run-uvicorn.png"/></a></div>
<h3>Reusing existing code</h3>
<p>
I realize that I need something that will log what is happening in the service. Luckily, there's already a log component there that was added before, for a different service. I'll go ahead and import the logger into my endpoint and start logging.
<br/><br/>
✨ This is where Polylith really shines. ✨ The components that you develop are ready to be re-used already from start. There's no need to extract code into libraries or doing something additional. All previosly written components are already there, and you just add them to your project. You might think that the logger is a silly little thing? Yes, it is. But this is only a simple example of code that is shared between projects in a Polylith Workspace.
</p>
<h3>The Project</h3>
<p>
There's one important part missing, and that is the project. A project is the artifact that will be deployed. It is a package of all project-specific code, along with project-specific configuration. Using the poly tool and adding the base to the project-specific pyproject.toml. Note the <strong>relative path</strong> to the bases folder.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1m1chX7R_zIXumgKdX07liu7DXx1sitPw-6WrFfIlX2kqqMVhcO3lwhn6XV0S5Z9bh9IUZho2t2ooEa4DTNBj4uYD4y0dzCVe6MFHZdrx0PBnZYqQpmHa86p2Br_34zMDmCK0lvELABUXuOgiKnE35n4c3rlJ_TMgN8LBr125s0o6UkSalbvKV_5cSAc/s1360/poetry-poly-create-project-fixed.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1m1chX7R_zIXumgKdX07liu7DXx1sitPw-6WrFfIlX2kqqMVhcO3lwhn6XV0S5Z9bh9IUZho2t2ooEa4DTNBj4uYD4y0dzCVe6MFHZdrx0PBnZYqQpmHa86p2Br_34zMDmCK0lvELABUXuOgiKnE35n4c3rlJ_TMgN8LBr125s0o6UkSalbvKV_5cSAc/s400/poetry-poly-create-project-fixed.png"/></a></div>
<pre>
packages = [
{include = "my_namespace/message_api", from = "../../bases"},
]
</pre>
<p>
I'm feeling lazy and will use the poly tool to add the other needed bricks for this specific project (components and bases are often referred to as bricks). The poly tool will analyze the imports and add only the ones needed for this project when running the <code>poetry poly sync</code> command. There's also a <code>check</code> command available that will report any missing bricks.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnpGVyRW5Yx5q5A0SUDjXuL5bhh6p-EGj8ZsOkABExpdZg7AbQlEURTpQ0SMgao0IgEOYPmqY9phnX0R4EjELWXP6xkL4ZnoukQB74RH4-KWtiS-Xr1PMDQS9V2Hy0eW7nfbYopKIvHV7mHT0QpPEgGDJFJ2XIBSSU2S-CVOKlGFJacvmT8sboVJh2mkw/s1360/poetry-poly-sync-and-check-fixed-updated.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="716" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnpGVyRW5Yx5q5A0SUDjXuL5bhh6p-EGj8ZsOkABExpdZg7AbQlEURTpQ0SMgao0IgEOYPmqY9phnX0R4EjELWXP6xkL4ZnoukQB74RH4-KWtiS-Xr1PMDQS9V2Hy0eW7nfbYopKIvHV7mHT0QpPEgGDJFJ2XIBSSU2S-CVOKlGFJacvmT8sboVJh2mkw/s400/poetry-poly-sync-and-check-fixed-updated.png"/></a></div>
<p>
You still need to add the third party dependencies, though. There's limits to <i>the magic of the poly tool</i> 🪄 🧙. To keep track on what is needed in the bricks, you can actually run <code>poetry poly check</code> again. It will notify about the usage of SQLAlchemy in the bricks. You can use the <code>poly check</code> command during development to guide you in the dependencies actually needed for this specific project.
</p>
<p>
Add sqlalchemy to the <code>[tool.poetry.dependencies]</code> section manually. You can update the lock-file for the project by running the builtin Poetry command:
<br/><br/>
<code>poetry lock --directory projects/message_fastapi_project</code>
<br/><br/>
All project-specific dependencies should now be set up properly! When packaging the project, simply run the <code>poetry build-project</code> (with <code>--directory projects/my_message_fastapi_project</code> if you run the command from the workspace root). You can use the built wheel to deploy the FastAPI service. It contains all the needed code, packaged and re-arranged without any relative paths.
</p>
<h3>What's in my Workspace?</h3>
<p>
Check the state of the workspace with <code>poetry poly info</code> command. You will get a nice overview of the added projects, bricks and the usage of bricks in each project.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0CrYwI1Z-5qy2YZxmdkijMjQd9mequh-VH4T-ECB0bTib6nFkH4WYSlEA0C_RDa9BV12PlPhEHAtiBiAfbkTwdGQGbDZwLYVOOb4EOvSe5eyP2bbWeEYwpS3KC3bEG6oLKN3I74xCM1kPkqgfpjrvSz5Vm4mbAWEMuLDQsd2uQUJ5FbIU_sUrbIsgYTM/s2944/poetry-poly-info-example-workspace.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1622" data-original-width="2944" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0CrYwI1Z-5qy2YZxmdkijMjQd9mequh-VH4T-ECB0bTib6nFkH4WYSlEA0C_RDa9BV12PlPhEHAtiBiAfbkTwdGQGbDZwLYVOOb4EOvSe5eyP2bbWeEYwpS3KC3bEG6oLKN3I74xCM1kPkqgfpjrvSz5Vm4mbAWEMuLDQsd2uQUJ5FbIU_sUrbIsgYTM/s320/poetry-poly-info-example-workspace.png"/></a></div>
<p>
In this article, I have written about the usage of Polylith when developing services. Adding new services is a simple thing when working in a Polylith workspace, and the tooling is there for a nice Developer Experience. Even if it sometimes might feel like a superpower, it's basically only about keeping things simple. Don't hesitate to reach out if you have feedback or questions.
</p>
<h3>Additional info</h3>
<p>
Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
<br/>
Docs about Polylith: <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank">Python tools for the Polylith Architecture</a>
</p>
<sup>
<span>
Top photo by <a href="https://unsplash.com/es/@yuliamatvienko?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Yulia Matvienko</a> on <a href="https://unsplash.com/photos/kgz9vsP5JCU?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-65628545583236402922023-07-17T14:43:00.015+02:002023-07-17T20:53:57.853+02:00GCP Cloud Functions with Python and Polylith<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
pre {
white-space: pre-wrap;
white-space: -moz-pre-wrap;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd9NqytZGnDDJ6pmYgZmYHAvz5QWxh1-_nTd_Ltxmtpsc8Uk4mqV1Pg2v4pWdCJ0kaEplKb--nxsUWbvjUBRUaHs9hjsATjT63GPugUP53OxnbZR-MF2tNqOVByELRR38MDekDlJRaE_81CNGvaDXeXeIvD141s9b7FHuYf67VyRxNeVpfgtsILNZsneg/s1920/rodion-kutsaiev-PEm_sLmJT-w-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjd9NqytZGnDDJ6pmYgZmYHAvz5QWxh1-_nTd_Ltxmtpsc8Uk4mqV1Pg2v4pWdCJ0kaEplKb--nxsUWbvjUBRUaHs9hjsATjT63GPugUP53OxnbZR-MF2tNqOVByELRR38MDekDlJRaE_81CNGvaDXeXeIvD141s9b7FHuYf67VyRxNeVpfgtsILNZsneg/s320/rodion-kutsaiev-PEm_sLmJT-w-unsplash.jpg"/></a></div>
<p>
Running a full-blown 24/7 running service is sometimes not a good fit for a feature. Serverless functions (also known as Lambdas or Cloud functions) is an alternative for those single-purpose kind of features. The major cloud providers have their own flavors of Serverless, and <i>Cloud Functions</i> is the one on <i>Google Cloud Platform</i> (GCP).
</p>
<h3>GCP Cloud Functions & Polylith</h3>
<p>
Polylith is good choice for this. The main use case for Polylith is to support having one or more Microservices, Serverless functions or apps, and being able to share code & deploy in a simple and developer friendly way.
<br/><br/>
One thing to notice is that GCP Cloud Functions for Python expects a certain file structure when packaging the thing to deploy. There should be a <strong>main.py</strong> in the top directory and it should contain the <strong>entry point</strong> for the function. Here's an example, using the <i>functions-framework</i> from GCP to define the entry point.
</p>
<pre>
import functions_framework
@functions_framework.http
def handler(request):
return "hello world"
</pre>
<p>You'll find more about Python Cloud Functions on the <a href="https://cloud.google.com/functions/docs/writing/http#http_frameworks" target="_blank">official GCP docs page</a>.</p>
<p>
In addition to the <i>main.py</i>, there should also be a <i>requirements.txt</i> in the top folder, defining the dependencies of the Cloud function. Other than that, it's up to you how to structure the code.
</p>
<pre>
/
main.py
requirements.txt
</pre>
<h3>How does this fit in a Polylith workspace?</h3>
<p>I would recommend to create a base, containing the entry point.</p>
<pre>
poetry poly create base --name my_gcp_function
</pre>
<i>Never heard about Polylith? <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank">Have a look a the documentation.</a></i>
<p>
Polylith will create a namespace package in the bases directory. Put the handler code in the already created <i>core.py</i> file.
<br/><br/>
Make sure to export the handler function in the <i>__init__.py</i> of the base. We will use this one later, when packaging the cloud function.
</p>
<pre>
from my_namespace.my_gcp_function.core import handler
__all__ = ["handler"]
</pre>
<p>If you haven’t already, create a project:</p>
<pre>
poetry poly create project --name my_gcp_project
</pre>
<p>Just as with any other Polylith project, add the needed bricks to the packages section.</p>
<pre>
[tool.poetry]
packages = [{include=”my_namespace/my_gcp_function”,from="../../bases"}]
# this is specific for GCP Cloud Functions:
include = ["main.py", "requirements.txt"]
</pre>
<h3>Packaging a Cloud Function</h3>
<p>
To package the project as a GCP Cloud Function, we will use the <strong>include</strong> property to make sure the needed <i>main.py</i> and <i>requirements.txt</i> files are included. These ones will be generated by a <strong>deploy script</strong> (see further down in this article).
<br/><br/>
Just as we normally would do to package a Polylith project, we will use the poetry build-project command. Before running that command, we need to make some additional tasks.
<br/><br/>
Polylith lives in a Poetry context, using a <i>pyproject.toml</i> to define dependencies, but we can export it to the <i>requirements.txt</i> format. In addition to that, we are going to copy the "interface" for the base (<i>__init__.py</i>) into a <i>main.py</i> file. This file will be put in the root of the built package. GCP will then find the entry point handler function.
</p>
<p>
To summarize:
<ul>
<li>Export the dependencies into a requirements.txt format</li>
<li>Copy the interface into a main.py</li>
<li>run build-project</li>
</ul>
Depending on how you actually deploy the function to GCP, it is probably a good idea to also rename the wheel that has been built to "function.zip". A wheel is already a zipped folder, but with a specific file extension.
</p>
<pre>
# export the dependencies
poetry export -f requirements.txt --output requirements.txt
# copy the interface into a new main.py file
cp ../../bases/<my-namespace>/messages_gcp_function/__init__.py ./main.py
# build the project
poetry build-project
# rename the built wheel
mv dist/my_gcp_function_project-0.1.0-py3-none-any.whl dist/function.zip
</pre>
<p>That's all you need. You are now ready to write & deploy GCP Cloud Functions from your Polylith workspace. 😄</p>
<h3>Additional info</h3>
<p>
Full example at: <a href="https://github.com/DavidVujic/python-polylith-example" target="_blank">python-polylith-example</a>
<br/>
Docs about Polylith: <a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank">Python tools for the Polylith Architecture</a>
</p>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@frostroomhead?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Rodion Kutsaiev</a> on <a href="https://unsplash.com/images/nature/cloud?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-68371804483826729712023-04-30T16:26:00.004+02:002023-05-01T11:11:53.901+02:00Dad Jokes & Python Developer Experience<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGFTJuMvxNsCXEAZJM9cq5AWVrrpJt3nLoAQzZoT2F5bk4sdrfXbrVbWtB5KIj5OIIBgaZMnxuoqVzGbL2n1JPdVjGf7J9gWP_Wg25WiQQxw0MFr_VMhr-MhLJlAL1OlOqZ8-Y7t9oxhezDZS-ZSoCO_Wv5DvDlCU_Q01hVksrYb4mQnKXVnpFg7z2/s2880/toa-heftiba-l_ExpFwwOEg-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="400" data-original-height="2880" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGFTJuMvxNsCXEAZJM9cq5AWVrrpJt3nLoAQzZoT2F5bk4sdrfXbrVbWtB5KIj5OIIBgaZMnxuoqVzGbL2n1JPdVjGf7J9gWP_Wg25WiQQxw0MFr_VMhr-MhLJlAL1OlOqZ8-Y7t9oxhezDZS-ZSoCO_Wv5DvDlCU_Q01hVksrYb4mQnKXVnpFg7z2/s400/toa-heftiba-l_ExpFwwOEg-unsplash.jpg"/>
</a>
</div>
<blockquote class="large">
"What do you call a Frenchman wearing sandals?"
<br/><br/>
"Philippe Flop." 🥁 👴 😆
<br/>
</blockquote>
<strong>Developer Experience, what does that even mean?</strong>
<p>
For me, a huge part is to quickly being able to write, try out and experiment with code. In Python, that could be about having the dependencies and paths already set up, to easily run snippets in a REPL or in the IDE, and without it crashing because of some missing dependencies or configs.
</p>
<h3>Testable code</h3>
<p>
When I develop a new feature, I usually want to run & evaluate the code very early in the process. Maybe I'm working on parsing something, transforming or filtering out data: that's when I want to evaluate the code, while writing and figuring out how to solve the actual problem. I do this to verify the output and get ideas about how to refactor the code. I practice a thing called <strong>REPL Driven Development</strong>, and <a href="https://davidvujic.blogspot.com/2022/08/joyful-python-with-repl.html">have written about it</a> before. The workflow is a form of <strong>TDD</strong> and the refactoring part comes natural with this kind of workflow.
</p>
<h3>Reusable & Composable code</h3>
<p>
Another important thing for me is to have already existing code nearby and available to use again. I usually think of Python code as building blocks, like <strong>LEGO</strong>, and try to have that in mind when writing new functions and adding features. I have learned that from <strong>functional programming</strong> and it is helping me to solve problems using a <strong>step-by-step</strong> approach.
<br/><br/>
With building blocks, code is composable and is ready for re-use from start.
</p>
<blockquote class="large">
"What did Yoda say when he saw himself in 4K?"
<br/><br/>
"HDMI." 🥁 👴 😆
<br/>
</blockquote>
<h3>Delay Design Decisions</h3>
<p>
I don't think it is that important to put code in the <i>right place</i> from start (such as what kind of service or app, domain, repo or a folder). At least it isn't as important as writing code that can easily be moved from one place to another. Very related to the building blocks & LEGO way, mentioned above. This approach will likely lead to less wasted time & less design-upfront planning. It is easier to figure it out the proper place to put the code eventually, while iterating and developing. In my opinion, it is perfectly alright to rename or move around code while developing. It shouldn't be a difficult thing to do that.
<br/><br/>
I usually like to dive into a feature and start coding, rather than starting off by deciding name of a repo, or what type of service the feature should exist in. These things can be left to decide later, when learning more about what to actually build.
</p>
<blockquote class="large">
"Where do dads store their dad jokes?"
<br/><br/>
"In the Dadabase." 🥁 👴 😆
<br/>
</blockquote>
<h3>A Developer Friendly Architecture</h3>
<p>
<a href="https://davidvujic.github.io/python-polylith-docs/" target="_blank" title="To the Python tools for the Polylith Architecture">The Polylith Architecture</a> targets all of these things. During the development you have <strong>all code available</strong> for experimentation and you compose already existing code with new when building features. All code in a Polylith workspace is referenced as <strong>bricks</strong>, and you use them just as when building something with <strong>LEGO</strong>. Pick the ones you need along with writing new ones, and combine them into features. A brick can be used in several apps, services or <i>projects</i> as it is called in Polylith.
<br/><br/>
During development, you have <strong>one single virtual environment</strong>, with all paths and dependencies already set up. You also have a separate section especially made for trying out and experimenting. It is very similar to the <i>scratch files</i> in PyCharm, but they are versioned and included in the repo. These ones are very useful for evaluating code.
<br/><br/>
You can immediately dive in to <strong>writing code</strong> and push the decision about deployment into the future if you like. Add the project to the Polylith workspace, using a simple command, when you are ready for it.
</p>
<h3>A Dad Joke microservice</h3>
<p>
To get some insights in how Polylith and the Python tooling changes the way you develop and improves the Developer Experience, I have recorded an improvised video with live coding. Phew, it is difficult to speak & code at the same time. 😅
<br/><br/>
In this video, I'm taking the first steps into developing some kind of <i>Dad Joke Service</i> and use the Development Environment of Polylith to figure out how to build it.
</p>
<div class="separator" style="clear: both; text-align: center;">
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="oG4OFEer3Tk" width="400" height="322" src="https://www.youtube.com/embed/oG4OFEer3Tk"></iframe></div>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@heftiba?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Toa Heftiba</a> on <a href="https://unsplash.com/photos/l_ExpFwwOEg?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-28920689017930480792022-11-22T08:12:00.002+01:002022-11-22T09:45:41.962+01:00The last Python Architecture you will ever need?<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div style="font-family: 'Open Sans', sans-serif;">
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgrCh31MRZXeyYebt94UkJEnUsnMBJlo-2bUlPy4lu-KPF1J3HUo4DSdWJYk9iZ2mKJt-hkstVsctr_NurTuDNf8pJ2Zq2iYDBr9G_Ve7-P46FCb1xNrYjC8j5ri_JPW5q1JqlD5SAZlI57sLapbyJPGf6tXP0lP_e0Fyi3BFB3iaxotA6aO6AK8EY/s1920/frank-mckenna-CSeCKeW4TM0-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1424" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgrCh31MRZXeyYebt94UkJEnUsnMBJlo-2bUlPy4lu-KPF1J3HUo4DSdWJYk9iZ2mKJt-hkstVsctr_NurTuDNf8pJ2Zq2iYDBr9G_Ve7-P46FCb1xNrYjC8j5ri_JPW5q1JqlD5SAZlI57sLapbyJPGf6tXP0lP_e0Fyi3BFB3iaxotA6aO6AK8EY/s400/frank-mckenna-CSeCKeW4TM0-unsplash.jpg"/></a></div>
<p>
Should features live in several independent Microservices, or is a Monolith the way to go? What kind of setup is the best one for the code in a repo? Organizing is difficult.
</p>
<blockquote class="large">
"... I'll just put it in a utils folder for now ..."
<br/>
</blockquote>
<p>
There's a thing called Polylith. I've written about <a href="https://davidvujic.blogspot.com/2022/08/a-simple-scalable-python-project.html">different aspects</a> of it before. Polylith is an architecture (and a tool) focusing on the Developer and Deployment experience. It is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk.
</p>
<p>
The Polylith Architecture is a fresh take on how to share code, and offers a nice and simple solution to the <strong>Monolith vs Microservice</strong> tradeoffs. In addition to that, it is a good fit for functional programming. Some time ago, I got an idea: how about bringing a couple of the good things from there to here? So I developed something that I believe could be useful for many Python teams out there.
</p>
<p>
I already released a preview in early 2022, it was missing some essential features and the great developer experience wasn't really there yet. Also, I had little knowledge about how to package Python apps & libraries, but have learned a lot since then.
<br/><br/>
Today, I believe <strong>the Python tools for the Polylith Architecture</strong> is ready to use in the Real World. You will find <a href="https://pypi.org/project/poetry-polylith-plugin/" target="_blank">version 1 on PyPI</a>.
</p>
<h3>A couple of Poetry plugins</h3>
<p>
I have developed it as two different Poetry plugins. One of them - the <strong>Multiproject</strong> plugin - adds support for workspaces to the popular Poetry tool, by adding a new command called <code>build-project</code>.
<br/><br/>
The second one - <strong>Polylith</strong> plugin - adds useful tooling support for the Polylith Architecture itself. With the tooling, you can add components & projects in a simple way and also keep track of what's happening in the workspace.
<br/><br/>
Have a look at <a href="https://github.com/DavidVujic/python-polylith" target="_blank">the repo and the docs</a>.
</p>
<h3>An Architecture well suited for Monorepos</h3>
<p>
Polylith is using a components-first architecture. The components are building blocks, very much like LEGO. The code is separated from the infrastructure and the building of artifacts. This may sound complicated, but it isn't.
</p>
In a way, it is about:
<ol>
<li>thinking about code as LEGO bricks, that can be combined into features.</li>
<li>making it easy to reuse code across apps, tools, libraries, serverless functions and services.</li>
<li>Keeping it simple.</li>
</ol>
Have a look at these introductory videos, describing Polylith in Python and the tooling support:
<br/><br/><br/>
<h4>Python with the Polylith Architecture</h4>
<br/>
<div class="separator" style="clear: both;">
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="3w2ffHZb6gc" width="400" height="322" src="https://www.youtube.com/embed/3w2ffHZb6gc"></iframe>
</div>
<br/>
<h4>The Poetry Polylith Plugin</h4>
<br/>
<div class="separator" style="clear: both;">
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="AdKpTP9pjHI" width="400" height="322" src="https://www.youtube.com/embed/AdKpTP9pjHI"></iframe>
</div>
<p>
Give it a try! I would love to hear you feedback.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@frankiefoto?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">frank mckenna</a> on <a href="https://unsplash.com/s/photos/beach?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com1tag:blogger.com,1999:blog-1409352204799783992.post-11980416354149918582022-08-22T08:27:00.003+02:002022-08-23T08:06:10.183+02:00Joyful Python with the REPL<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div style="font-family: 'Open Sans', sans-serif;">
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPCwIrIRTiutdtPe8Hduv7_LhQABFobj-IoKlY1PgoBM1LFdCjg6oc_EG_0msLbMRo1KOEQ6swpNLeZwvMtZPZmojpGcwhN0gVEqo6NUV5zZR6mtfKS0VUMhQnikd_sxn1cnQZJJ9K8dkOfM9yRjmJw5FEY_7_A2YqDCQ_7xYsFMY-sbsCf4aC6AHz/s1920/ckturistando-HTCQCvwV9XY-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPCwIrIRTiutdtPe8Hduv7_LhQABFobj-IoKlY1PgoBM1LFdCjg6oc_EG_0msLbMRo1KOEQ6swpNLeZwvMtZPZmojpGcwhN0gVEqo6NUV5zZR6mtfKS0VUMhQnikd_sxn1cnQZJJ9K8dkOfM9yRjmJw5FEY_7_A2YqDCQ_7xYsFMY-sbsCf4aC6AHz/s400/ckturistando-HTCQCvwV9XY-unsplash.jpg"/></a></div>
<p>
REPL Driven Development is a workflow that makes coding both joyful and interactive. The feedback loop from the REPL is a great thing to have at your fingertips.
</p>
<blockquote class="large">
"If you can improve just one thing in your software development, make it getting faster feedback."
<br/>
<sup><a href="https://twitter.com/davefarley77/status/1560724029924786177?s=12&t=KxEN15qtnJODJUzkmclzmw" target="_blank">Dave Farley</a></sup>
</blockquote>
<p>
Just like Test Driven Development (TDD), it will help you write testable code. I have also noticed a nice side effect from this workflow: REPL Driven Development encourages a functional programming style.
<br/><br/>
REPL Driven Development is an everyday thing among Clojure developers and doable in Python, but far less known here. I'm working on making it an everyday thing in Python development too.
<br/><br/>
But what is REPL Driven Development?
</p>
<h3>What is it?</h3>
<p>
You evaluate variables, code blocks, functions - or an entire module - and get instant feedback, just by a hitting a key combination in your favorite code editor. There's no reason to leave the IDE for a less featured shell to accomplish all of that. You already have autocomplete, syntax highlighting and the color theme set up in your editor. Why not use that, instead of a shell?
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUOHP9F6KyzUE56VrfyX74GJZQu8ZdGdq9sheI_H_ooPTDSzzV1hKaKvf0Fc1OD0PiVhSSN81rVWjsvNNAaTUanzylRyAZ7S-HkWZ5OAhss2Sue-CxgdX3riBL6y9S3_iB8uQXJH3ltP7EDdnFVbNPmTkqTVM7Mh7qoHtdtN-AcHsjbz0YUQtbV5U5/s919/repl-example.gif" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="558" data-original-width="919" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUOHP9F6KyzUE56VrfyX74GJZQu8ZdGdq9sheI_H_ooPTDSzzV1hKaKvf0Fc1OD0PiVhSSN81rVWjsvNNAaTUanzylRyAZ7S-HkWZ5OAhss2Sue-CxgdX3riBL6y9S3_iB8uQXJH3ltP7EDdnFVbNPmTkqTVM7Mh7qoHtdtN-AcHsjbz0YUQtbV5U5/s400/repl-example.gif"/>
</a>
</div>
<figcaption>Evaluate code and get feedback, without leaving the code editor.</figcaption>
</figure>
<p>
Ideally, the result of an evaluation pops up right next to the cursor, so you don't have to do any context switches or lose focus. It can also be printed out in a separate frame right next to the code. This means that testing the code you currently write is at your fingertips.
</p>
<h3>Easy setup</h3>
<p>
With some help from <strong>IPython</strong>, it is possible to write, modify & evaluate Python code in a REPL Driven way. I would recommend to install IPython globally, to make it accessible from anywhere on your machine.
</p>
<code>pip install ipython</code>
<p>Configure IPython to make it ready for REPL Driven Development:</p>
<code>
c.InteractiveShellApp.exec_lines = ["%autoreload 2"]
c.InteractiveShellApp.extensions = ["autoreload"]
c.TerminalInteractiveShell.confirm_exit = False
</code>
<p>You will probably find the configuration file here: <i>~/.ipython/profile_default/ipython_config.py</i>
<br/><br/>
You are almost all set.
</p>
<h3>Emacs setup</h3>
<p>
Emacs is my favorite editor. I'm using a couple of Python specific packages to make life as a Python developer in general better, such as <i>elpy</i>. The <i>auto-virtualenv</i> package will also help out making REPL Driven Developer easier. It will find local virtual environments automatically and you can start coding without any python-path quirks.
<br/><br/>
Most importantly, set IPython as the default shell in Emacs. Have a look at <a href="https://github.com/DavidVujic/my-emacs-config" target="_blank">my Emacs setup</a> for the details.
</p>
<h3>VS Code setup</h3>
<p>
I am not a VS Code user. But I wanted to learn how well supported REPL Driven Development is in VS Code, so I added these extensions:
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTtfSeI2n-PnjH5Pj4vY1B4nBnvH-X8VWBmlrj9bNwfdZk5-MhHd4boBEv_WPIj713XDyK0TxiV7zZG7kQ9N9wiC1E3st6XGsh1LD89EJxBAROa7jVmE9H2a4dvI3nPgBCFeo2pogpakqP2LkEPQEvNaScrAX98xqL-1rMlFEfs2K8oLzGpN2eEF7x/s1318/vscode-extensions.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="320" data-original-height="912" data-original-width="1318" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTtfSeI2n-PnjH5Pj4vY1B4nBnvH-X8VWBmlrj9bNwfdZk5-MhHd4boBEv_WPIj713XDyK0TxiV7zZG7kQ9N9wiC1E3st6XGsh1LD89EJxBAROa7jVmE9H2a4dvI3nPgBCFeo2pogpakqP2LkEPQEvNaScrAX98xqL-1rMlFEfs2K8oLzGpN2eEF7x/s320/vscode-extensions.png"/>
</a>
</div>
<p>
You would probably want to add keyboard shortcuts to get the true interactive feel of it. Here, I'm just trying things out by selecting code, right clicking and running it in an interactive window. It seems to work pretty well! I haven't figured out if the interactive window is picking up the global IPython config yet, or if it already refreshes a submodule when updated.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7BapBZn93IzhFhGVrl488WzVP8_ds3DntVNgJ7eMU6YLsJknyOWazGHDSssrVBywdKhj2iyr-vWzhUUcpHlLXwrnQ6_TTjQCrMd65ps2zyN4YnZwF_w2rYeAeXfXrYhvO-WepHPjH2oQ93Nf9Mf408l80OyfB-q7xQoWpGdNHfj0VVQPVrOTKqEHm/s865/vscode-repl.gif" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="657" data-original-width="865" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7BapBZn93IzhFhGVrl488WzVP8_ds3DntVNgJ7eMU6YLsJknyOWazGHDSssrVBywdKhj2iyr-vWzhUUcpHlLXwrnQ6_TTjQCrMd65ps2zyN4YnZwF_w2rYeAeXfXrYhvO-WepHPjH2oQ93Nf9Mf408l80OyfB-q7xQoWpGdNHfj0VVQPVrOTKqEHm/s400/vscode-repl.gif"/>
</a>
</div>
<figcaption>Evaluating code in the editor with fast feedback loops.<br/>It would be great to have keyboard commands here, though.</figcaption>
</figure>
<h3>Current limitations</h3>
<p>
In Clojure, you connect to & modify an actually running program by re-evaluating the source code. That is a wonderful thing for the developer experience in general. I haven't been able to do that with Python, and believe Python would need something equivalent to <strong>NRepl</strong> to get that kind of magic powers.
</p>
<h3>Better than TDD</h3>
<p>
I practice REPL Driven Development in my daily Python work. For me, it has become a way to quickly verify if the code I currently write is working as expected. I usually think of this REPL driven thing as <i>Test Driven Development Deluxe</i>. Besides just evaluating the code, I often write short-lived code snippets to test out some functionality. By doing that, I can write code and test it interactively. Sometimes, these code snippets are converted permanent unit tests.
<br/><br/>
For a live demo, have a look at my five minute lightning talk from PyCon Sweden about <strong>REPL Driven Development in Python</strong>.
</p>
<div class="separator" style="clear: both; text-align: center;">
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="0HaIYpxTzX8" width="400" height="322" src="https://www.youtube.com/embed/0HaIYpxTzX8"></iframe>
</div>
<h3>Never too late to learn</h3>
<p>
I remember it took me almost a year learning & developing Clojure before I actually "got it". Before that, I sometimes copied some code and pasted it into a REPL and then ran it. But that didn't give me a nice developer experience at all. Copy-pasting code is cumbersome and will often fail because of missing variables, functions or imports. Don't do that.
<br/><br/>
I remember the feeling when figuring out the REPL Driven Development workflow, I finally had understood how software development should be done. It took me about 20 years to get there. It is never too late to learn new things. 😁
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/es/@ckturistando?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">ckturistando</a> on <a href="https://unsplash.com/s/photos/joy?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com4tag:blogger.com,1999:blog-1409352204799783992.post-79372335488006616722022-08-02T21:46:00.003+02:002022-08-02T21:46:44.165+02:00A simple & scalable Python project structure<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div style="font-family: 'Open Sans', sans-serif;">
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6gcNvqsVHP8Lt0inUDwrQ-woeU2IWkDnhualoZkJ18ztUMRDKI23WPYQaxmWOGPgl8zEEVjItEVCvkLZVBRadQo2hvhsDYgmRk-UEjJVrtwK06xP-ppsj6t4fEM_szqCa1fngpHWJ9OkFw2uMqGblLNbL5ehN_-YJ20MElRW1y5HArbfpCmPIkVP/s2560/maureen-sgro-BpFAG6JSugE-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="400" data-original-height="2560" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6gcNvqsVHP8Lt0inUDwrQ-woeU2IWkDnhualoZkJ18ztUMRDKI23WPYQaxmWOGPgl8zEEVjItEVCvkLZVBRadQo2hvhsDYgmRk-UEjJVrtwK06xP-ppsj6t4fEM_szqCa1fngpHWJ9OkFw2uMqGblLNbL5ehN_-YJ20MElRW1y5HArbfpCmPIkVP/s400/maureen-sgro-BpFAG6JSugE-unsplash.jpg"/>
</a>
</div>
<p>
File & folder structures - there's almost as many different variations as there are code repositories.
</p>
<p>
One common thing though, is that you'll probably find the <i>utils</i> folder in many of the code repos out there, regardless of programming language. That's the one containing the files that don't fit anywhere in the current project structure. It is also known as the <i>helpers</i> folder.
<br/><br/>
Organizing, sorting and structuring things is difficult. There's framework specific CLIs and tools that will create a nice setup for you, specialized for the needs of the current framework.
</p>
<blockquote>
"There should be one-- and preferably only one --obvious way to do it."
<br/>
<sup>The Zen of Python</sup>
</blockquote>
<p>
Is there one folder structure to rule them all? Probably not, but I have tried out a way to organize code that is very simple, framework agnostic and scalable as projects grow.
</p>
<h3>Structure for simplicity</h3>
<p>
A good folder structure is one that makes it simple to <strong>reuse existing code</strong> and makes it easy to <strong>add new code</strong>. You shouldn't have to worry about these things. The thing I've tried out with success is very much inspired by the <strong>Polylith architecture</strong>. Polylith is a monorepo thing, but don't worry. This post isn't about monorepos at all, however <a href="https://davidvujic.blogspot.com/2022/02/a-fresh-take-on-monorepos-in-python.html">this one</a> is if you are interested in Python specific ones.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSw53GhHmWSDjl-IFImhXWlbfCnuNwmImK4qdt9r3eNYZZlT3PxzGGRi8rA9H-Paw-vUnJ7nntD7OnKSp7V9LFRNIL1FQdyXNqtKklsi2a6XQrMYfjX586UPlJczl93ZGZV9O6cz4lBiHUePOiGZeCMsNUP5JZmJCkOPdxgW00XhqbbDznrX8dTh2g/s1360/components_file_structure.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="722" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSw53GhHmWSDjl-IFImhXWlbfCnuNwmImK4qdt9r3eNYZZlT3PxzGGRi8rA9H-Paw-vUnJ7nntD7OnKSp7V9LFRNIL1FQdyXNqtKklsi2a6XQrMYfjX586UPlJczl93ZGZV9O6cz4lBiHUePOiGZeCMsNUP5JZmJCkOPdxgW00XhqbbDznrX8dTh2g/s400/components_file_structure.png"/>
</a>
</div>
<figcaption>
An entry point and a components folder. You won't need much more. Use your favorite dependencies tool, mine is currently Poetry.
</figcaption>
</figure>
<h3>It's all about the components</h3>
<p>
The main takeaway here is to view code as small, reusable components, that ideally does one thing only. A component is not the same thing as a library. So, what's the difference?
<br/><br/>
A library is a full blown feature. A component can be a single function, or a parser. It can also be a thin wrapper around a third party tool.
</p>
<blockquote>
"Simple is better than complex."
<br/>
<sup>The Zen of Python</sup>
</blockquote>
<p>
I think the idea of writing components is about changing mindset. It is about how to approach a problem and how to organize the code that solves a problem.
<br/><br/>
It shouldn't be too difficult to grasp for Python developers, though. For us Python devs, it's an everyday thing to write functions and have them in modules. Another useful thing, probably more common in library development, is to group the modules into packages.
</p>
<h3>Modules, packages, namespaces ... and components?</h3>
<p>
In Python, a file is a module. One or more modules in a folder becomes a package. A good thing with this is that the code will be namespaced when importing it. Where does the idea of components fit in here? Well, a component is a package. Simple as that.
</p>
<blockquote>
"Namespaces are one honking great idea -- let's do more of those!"
<br/>
<sup>The Zen of Python</sup>
</blockquote>
<p>
To make a package easier to understand, you can add an interface. Interfaces are well supported in Python. Specifying the interface of a package in an <code>__init__.py</code> file is a great way to make the intention of the code clearer and easier to grasp. Maybe there's only one function that makes sense to use from the "outside"? That's when to use an interface for your component.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkcC1FirT23C0m9h2CKfuaeBS1KY7fdXMiveR6_K9yji-WHUhTU6RSkKpAHS_DOSI7f4L33tHraqF8CU8qV4WGbEzawHRHB_lTSCK6Dtof8vx1zFGM-OdD1q5fUrSgOSeAG3v67G5ALXK0LqjUF6DNTVsjh8awWAMYnZ2BKP0NwBKE3SWljnbdILCs/s1360/pick-omit-interface.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="280" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkcC1FirT23C0m9h2CKfuaeBS1KY7fdXMiveR6_K9yji-WHUhTU6RSkKpAHS_DOSI7f4L33tHraqF8CU8qV4WGbEzawHRHB_lTSCK6Dtof8vx1zFGM-OdD1q5fUrSgOSeAG3v67G5ALXK0LqjUF6DNTVsjh8awWAMYnZ2BKP0NwBKE3SWljnbdILCs/s400/pick-omit-interface.png"/>
</a>
</div>
<figcaption>Only the functions that makes sense should be exposed from a component.</figcaption>
</figure>
<h3>Make code reuse easy</h3>
<p>
When organizing code into simple components, you will quickly discover how easy it is to reuse it. Code is no longer hidden in some <i>utils</i> folder and you no longer need to duplicate existing private helper functions (because the refactoring might break things), if they already are organized as reusable components with clear and simple APIs. I usually think of components as <strong>LEGO</strong> bricks to select from when building features. You will most likely produce new LEGO bricks of various shapes along the way.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihjDznsHD20MBEFB9u0JySrACAn4vCIHy8ouxnfg592LrfzIFQqnBXiMQL2Stj9e_ixKmibdVGbABOmK9Bjg0kR3koI4rYqv4kwJ9dTdoVj5PHEEOLaRz54T3ek09MurJALD7SfFlIIA6rUPRVMPvEryArybvcBVkCOyyvtKLnM0nmj3tWQ_01p5iM/s1360/pick-omit.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="832" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihjDznsHD20MBEFB9u0JySrACAn4vCIHy8ouxnfg592LrfzIFQqnBXiMQL2Stj9e_ixKmibdVGbABOmK9Bjg0kR3koI4rYqv4kwJ9dTdoVj5PHEEOLaRz54T3ek09MurJALD7SfFlIIA6rUPRVMPvEryArybvcBVkCOyyvtKLnM0nmj3tWQ_01p5iM/s400/pick-omit.png"/>
</a>
</div>
<figcaption>This is code in a "dictionaries" component. The interface (previous picture) will handle the access to it.</figcaption>
</figure>
<h3>Well suited for large apps</h3>
<p>
At work, we have a couple of Python projects using this kind of structure. One of them is a <strong>FastAPI</strong> app with an entry point (we named it <code>app.py</code>) containing the public endpoints. The entry point is importing a bunch of components that does the actual work.
<br/><br/>
The repo contains about 80 Python files. Most of them are grouped into components (in total about 30 components). This particular project is about 3K lines of Python code, but other repos are much smaller with only a handful of components.
</p>
<h3>Perfect for functional programming</h3>
<p>
Even though it is not a requirement, organizing code into components fits very well with functional programming. Separating code into data, calculations and actions are well suited for the component thing described here in this post.
<br/><br/>
Don't forget to keep the components simple, and try to view them as LEGO bricks to be used from anywhere in the app. You'll have fun while doing it too.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@msullivan4172?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Maureen Sgro</a> on <a href="https://unsplash.com/s/photos/palette?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-27356144407245700912022-07-09T16:37:00.004+02:002022-08-19T12:42:34.257+02:00Just use Dictionaries<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
.image-columns {
display: flex;
justify-content: space-evenly;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiweqTbney5zUYGtGkBAqNKdRF77zUz8Xo0C0IBy2oWZ6-mYGbvnv1CiS4ChhmYO33FsgdQgyRu5qO0nfAh2aEiCIp3tIqqEpoSVcXOdtnfu4B8-IFXWsJzq1ivqIQOcpQjEUB_Q35OWfJMfJxwQGhV3FEIJxZVhlMTJxouBELHjLLJRNJUNbFS7313/s1920/syd-wachs-slItfWbhijc-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiweqTbney5zUYGtGkBAqNKdRF77zUz8Xo0C0IBy2oWZ6-mYGbvnv1CiS4ChhmYO33FsgdQgyRu5qO0nfAh2aEiCIp3tIqqEpoSVcXOdtnfu4B8-IFXWsJzq1ivqIQOcpQjEUB_Q35OWfJMfJxwQGhV3FEIJxZVhlMTJxouBELHjLLJRNJUNbFS7313/s400/syd-wachs-slItfWbhijc-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<p>
A Python dictionary has a simple & well-known API. It is possible to merge data using a nice & minimalistic syntax, without mutating or worrying about state. You're probably not gonna need classes.
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFggAb4XFEcQ1EI-PPVhj2nbKBXEXEtLBPbcJ-ntABECNWsAnxu8plhy2-Ds9Z7ObOugslscp3Z7VHXQRR17K24w54CftlmCTzDvbN_gRnwng1MqVV_usA1K-HOUGPa7BhvZzNt9XpITjYcwg5zWW_y9GbF_21pQhliBKw9tZip_Y2F7ZSjG-jiasq/s1360/data-merging.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="522" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFggAb4XFEcQ1EI-PPVhj2nbKBXEXEtLBPbcJ-ntABECNWsAnxu8plhy2-Ds9Z7ObOugslscp3Z7VHXQRR17K24w54CftlmCTzDvbN_gRnwng1MqVV_usA1K-HOUGPa7BhvZzNt9XpITjYcwg5zWW_y9GbF_21pQhliBKw9tZip_Y2F7ZSjG-jiasq/s320/data-merging.png"/>
</a>
</div>
<h3>Hey, what's wrong with classes? 🤔</h3>
<p>
From what I've seen in Python, classes often add unnecessary complexity to code. Remember, the Python language is all about keeping it simple.
<br/><br/>
My impression is that in general, class and instance-based code feels like <i>the proper way</i> of coding: encapsulating data, inheriting features, exposing public methods & writing smart objects. The result is very often a lot of code, weird APIs (each one with its own) and not smart-enough objects. That kind of code quickly tend to be an obstacle. I guess that's when workarounds & hacks usually are added to the app.
</p>
<figure>
<div class="image-columns">
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisVb0SD7_8_YMMBp09uhP8zASqH9hGHA9IYMbyGL-6Bkj1ZO62I794dK5RaQpPSPXjoFQdWha92cXf9o54QnUn3pw6agsZSlNl6Cb7l-MNsUU34RhlnFM1QZNZVkOnVU7HvE7gsWN7Rnvh8kwh27U35Tlqb-KwtTfomoWzdfcZ87J7Tq7qVl-TGqFK/s1670/office-oop.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="200" data-original-height="1670" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisVb0SD7_8_YMMBp09uhP8zASqH9hGHA9IYMbyGL-6Bkj1ZO62I794dK5RaQpPSPXjoFQdWha92cXf9o54QnUn3pw6agsZSlNl6Cb7l-MNsUU34RhlnFM1QZNZVkOnVU7HvE7gsWN7Rnvh8kwh27U35Tlqb-KwtTfomoWzdfcZ87J7Tq7qVl-TGqFK/s320/office-oop.png"/>
</a>
</div>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXPQUnmj9CcRlf1mYG-16_pTEQiMIMtET63DReo1CjiFlQo_IOVci3QJ27jltXD8HuxcECwrk1na8EnBXplrza_aXq9DwYxH9LuPNiNS4XwLOzdt4TTBUG8mshhQ4fj0IrM8fRca25J3mhir8douWEKXjRFbmCojfME6Cxpn0fmqZsB5sEIfEnanYj/s1360/office-functional.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="200" data-original-height="726" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXPQUnmj9CcRlf1mYG-16_pTEQiMIMtET63DReo1CjiFlQo_IOVci3QJ27jltXD8HuxcECwrk1na8EnBXplrza_aXq9DwYxH9LuPNiNS4XwLOzdt4TTBUG8mshhQ4fj0IrM8fRca25J3mhir8douWEKXjRFbmCojfME6Cxpn0fmqZsB5sEIfEnanYj/s320/office-functional.png"/>
</a>
</div>
</div>
<figcaption>
Two ways of solving a problem: class-based vs data-oriented.
<br/> Less code, less problems.
</figcaption>
</figure>
<h3>What about Dataclasses?</h3>
<p>
Python dataclasses might be a good tradeoff between a heavy object with methods and the simple dictionary. You get typings and autocomplete. You can also create <strong>immutable</strong> data classes, and that's great! But you might miss the flexibility: the simplicity of merging, picking or omitting data from a dictionary. Letting data flow smoothly through your app.
</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ5CQo43to2fiH0FnlB2d9jicqXDlROA6HvepAnPc0_G-D9utnItBFIK12yMd3U4kjZcm2PhA3leCg0FYr-b51LsTl3iaxlisiv96IlhfMjJfE6aUb1OsktoJYSYGeMPHihcUl0srpYRHHKL61Rrqid7toKpf6pNbvyTohqrpRsMxtZKGmuRmotQgb/s1360/pick-omit.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="398" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ5CQo43to2fiH0FnlB2d9jicqXDlROA6HvepAnPc0_G-D9utnItBFIK12yMd3U4kjZcm2PhA3leCg0FYr-b51LsTl3iaxlisiv96IlhfMjJfE6aUb1OsktoJYSYGeMPHihcUl0srpYRHHKL61Rrqid7toKpf6pNbvyTohqrpRsMxtZKGmuRmotQgb/s320/pick-omit.png"/></a></div>
<h3>Hey, what about Pydantic?</h3>
<p>
That's a really good choice for things like defining FastAPI endpoints. You'll get the typed data as OpenAPI docs for free.
<br/><br/>
I would as early as possible convert the Pydantic model to a dictionary (using the <code>model.dict()</code> function), or just pick the individual keys and pass those on to the rest of the app. By doing that, the rest of the app is not required to be aware of a domain specific type or some base class, created as workaround for the new set of problems introduced with custom types.
<br/><br/>
Just data. Keeping it simple.
</p>
<h3>What about the basic types? 🤔</h3>
<p>
That is certainly a tradeoff when using dictionaries, the data can be of any type and you will potentially get runtime errors. On the other hand, is that a real problem when using basic data types like dict, bool, str or int? For me, I can't remember that ever has been an issue.
</p>
<h3>But shouldn't data be private?</h3>
<p>
Classes are often used to separate public and private functionality. From my experience, explicitly making data and functionality private rarely adds value to a product. I think Python agrees with me about this. By default, all things in a Python module is public. I remember when learning about it, and the authors saying that’s okay because <i>we’re all adults here</i>. I very much liked that perspective!
</p>
<h3>Do you like Interfaces? 🤩</h3>
<p>
Yes! Especially when structuring code in modules and packages (more about that in the next section). Using <code>__init__.py</code> is a great way to make the intention of a small package clearer and easier to grasp. Maybe there's only one function that makes sense to use from the <i>outside</i>? That's where the <i>package interface</i> (aka the <code>__init__.py</code> file) feature fits in well.
</p>
<h3>Python files, modules, packages?</h3>
<p>
In Python, a file is a module. One or more modules in a folder is a package. One or more packages can be combined into an app. Using a package interface makes sense when structuring code in this way.
<br/><br/>
Keeping it simple. 😊
</p>
<p>
I'm finishing off this post with a quote from the past:
</p>
<blockquote>
<span>
“Data is formless, shapeless, like water.
<br/>If you put data in a Clojure map, it becomes the map.
<br/>You put data in a Python list and it becomes the list.
<br/>Now data in a program can flow or it can crash.
<br/><br/>
<strong>Be data, my friend.</strong>”
<br/><br/>
Bruce Lee, 1940 - 1973
</span>
</blockquote>
<p>
ps.
<br/><br/>
If neither Bruce or I convinced you about the great thing with simple data structures, maybe Rich Hickey will? Don't miss his <a href="https://www.youtube.com/watch?v=VSdnJDO-xdg&t=2955s" target="_blank">"just use maps" talk</a>!
<br/><br/>
ds.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@videmusart?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Syd Wachs</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-81973726101056315482022-05-28T19:07:00.024+02:002022-05-28T23:13:34.534+02:00Hi podman, Bye docker<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbvF-16Hw8Xt3YuOvWe7geRi_B_RHpFSLbpsAkMQi8MiGr_M9XFMJ7J0NRz-DM056gXGRhSOgFxPtmxjpxK-BJeZ-4IecWpYupp6mk4GAK6Yk8MiU_b6e65_5AhhLZfIR3u1_x57d8QEoJaYd6Zeth4xNgWM1XCN7FpdxaNJJqsyj7YTE0mTjWlAIz/s1920/frank-mckenna-tjX_sniNzgQ-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1252" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbvF-16Hw8Xt3YuOvWe7geRi_B_RHpFSLbpsAkMQi8MiGr_M9XFMJ7J0NRz-DM056gXGRhSOgFxPtmxjpxK-BJeZ-4IecWpYupp6mk4GAK6Yk8MiU_b6e65_5AhhLZfIR3u1_x57d8QEoJaYd6Zeth4xNgWM1XCN7FpdxaNJJqsyj7YTE0mTjWlAIz/s320/frank-mckenna-tjX_sniNzgQ-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<p>
A 🌧️ Saturday.
<br/><br/>
This feels like a great day to uninstall <strong>Docker Desktop</strong> on my Mac, and try to go all-in <strong>Podman</strong>.
</p>
<h3>What is Podman?</h3>
<blockquote class="large">
<span>" ... a daemonless container engine for developing, managing, and running OCI Containers ... "</span>
</blockquote>
<p>
Podman is an open source project and you can run containers as root, or as a non-privileged user. It supports the Docker API and can be used with tools like docker-compose.
</p>
<h3>What's wrong with docker?</h3>
<p>
Nothing, docker is a great tool! Free to use for Open Source. The Desktop app for Mac and Windows will continue to be free for small businesses, but require a license for bigger companies. For some time now, I have wanted to find out what the alternatives are to using docker.
</p>
<h3>Uninstalling docker</h3>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmd41vPI86KSoFfp1C_YmT6Ux_CdDkT0U_en2_r75Ljqr9tU0S1gs5_MZYJA_wK2BkPR-hqqRX4Xq-VrVQnssX1cDw9eBl69If9RI-n1QHncttdhQpPWdikIrb1N7kPYPkjOGemwuXPtcUL1uxdlp5P6UwgM-EbDzNi524odpuw4RPkTuMKlVjZiep/s640/deletedockershort.gif" style="display: block; padding: 1em 0; text-align: center;">
<img alt="" border="0" width="400" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmd41vPI86KSoFfp1C_YmT6Ux_CdDkT0U_en2_r75Ljqr9tU0S1gs5_MZYJA_wK2BkPR-hqqRX4Xq-VrVQnssX1cDw9eBl69If9RI-n1QHncttdhQpPWdikIrb1N7kPYPkjOGemwuXPtcUL1uxdlp5P6UwgM-EbDzNi524odpuw4RPkTuMKlVjZiep/s320/deletedockershort.gif"/>
</a>
</div>
<p>
I ran <code>docker system prune</code> to delete all existing containers and images.
</p>
<p>
Then I uninstalled the desktop client. By selecting the "Troubleshoot" menu option I found an uninstall button and pressed it.
It required some research (aka googling) to figure out how and where to find an uninstall option.
<br/><br/>
As you can see in the video, the docker app is still there even if the Desktop client was uninstalled. I just deleted that one, and then deleted the docker-related files and folders that were still in my file system:
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFNW1-yoJLRshbhbeOxP6qAXrsINLAHYpJzQRi8jNnX3uo9N1QfzDWHsUNgjniCzo9hTzUP0ytFRM9PTZX-KQi8gdenZb2QyH9pd3pOOuHwFj6AQYRqi0HO-3gvFw6tD5l3_rKVbwO1kKd6dqSF39mAe_E8mJRWceS3O5X3CAOitTnaSVd485OLDh0/s1360/rm-docker.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="604" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFNW1-yoJLRshbhbeOxP6qAXrsINLAHYpJzQRi8jNnX3uo9N1QfzDWHsUNgjniCzo9hTzUP0ytFRM9PTZX-KQi8gdenZb2QyH9pd3pOOuHwFj6AQYRqi0HO-3gvFw6tD5l3_rKVbwO1kKd6dqSF39mAe_E8mJRWceS3O5X3CAOitTnaSVd485OLDh0/s400/rm-docker.png"/>
</a>
</div>
<h3>Installing podman</h3>
<p>
I followed the official <a href="https://podman.io/getting-started/installation" target="_blank">install guideline</a> to install podman. In addition to that, also installed the Mac OS X specific helper tool to be able to run <strong>docker-compose</strong>. I learned that from the print-out displayed when the brew install command was completed.
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguiJXxKD83hnemDFL1eMKfyoqqFykZB2JonrXfpQYA-myjsM2Sjrj0T5uARz-cpqeOr63oYKfSDkLd3MihRxdtfpK8flTgPBq0DnG3PuEnrLBiZeEQZXm3vg0cnD7DB_f4COINiQFWBLwC7RQ5gGRZu7AmHLZtb08cN8WMxlCJ4aEG5AIy7u2QboQx/s1360/install-podman.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="604" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguiJXxKD83hnemDFL1eMKfyoqqFykZB2JonrXfpQYA-myjsM2Sjrj0T5uARz-cpqeOr63oYKfSDkLd3MihRxdtfpK8flTgPBq0DnG3PuEnrLBiZeEQZXm3vg0cnD7DB_f4COINiQFWBLwC7RQ5gGRZu7AmHLZtb08cN8WMxlCJ4aEG5AIy7u2QboQx/s400/install-podman.png"/>
</a>
</div>
<p>
I can now run containers with podman! Here I'm starting a ZooKeeper server running in a container:
<br/><br/>
<code>podman run --rm -p 2181:2181 zookeeper</code>
</p>
<h3>Where's my commands?</h3>
<p>
But wait, my 🧠 is already wired to use the <i>docker</i> command. Do I have to unlearn things? Podman is compatible with the docker commands already, but you can also create an alias:
<br/><br/>
<code>alias docker=podman</code>
<br/><br/>
It is now possible to use the <strong>docker</strong> command as before!
</p>
<h3>Installing docker-compose</h3>
<p>
By installing the <strong>podman-mac-helper</strong> tool as described before in this post, it is possible to use podman with docker-compose.
<br/><br/>
You might wonder: didn't I uninstall all docker related things? 🤔
<br/>
Yes, but the docker-compose tool is available to install separately & you'll find it on Homebrew. 🍻
<br/><br/>
<code>brew install docker-compose</code>
</p>
<h3>Quirks & workarounds</h3>
<p>
When trying all of this out, the podman command worked well for building and running containers. However, I got an error when running the <code>docker-compose up</code> command. That's kind of a dealbreaker, isn't it?
<br/><br/>
The error:
<code>listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 404"</code>
<br/><br/>
As I understand it (from <a href="https://github.com/containers/podman/issues/13889" target="_blank">this conversation on Github</a>), there is something not working as expected when using the latest version(s) of docker-compose and podman.
</p>
<p>
<strong>Luckily, there is a simple workaround</strong> to use until the bug is fixed. Prefixing the docker-compose command to disable the <strong>buildkit API</strong> that is causing the bug. Doing this will make things work very well.
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8xZJ4g_kQw1ytxg2Ymom87hh6VgfQD62OjxEezmL_WT40yV9oEI0IjZyT-iPLH9STnlB1qhPhdaLYubnoPbMPTV0rG8kX71kNttWxw64lHZehlvDBMAQ3hNplxX-SYqkgQWICCn3OmpQ7IyPMFbv1fbe9CMcoEd38uk3rH8RcOg58t3VkAbIm3G2p/s1360/docker-compose%20up.png" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="234" data-original-width="1360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8xZJ4g_kQw1ytxg2Ymom87hh6VgfQD62OjxEezmL_WT40yV9oEI0IjZyT-iPLH9STnlB1qhPhdaLYubnoPbMPTV0rG8kX71kNttWxw64lHZehlvDBMAQ3hNplxX-SYqkgQWICCn3OmpQ7IyPMFbv1fbe9CMcoEd38uk3rH8RcOg58t3VkAbIm3G2p/s400/docker-compose%20up.png"/>
</a>
</div>
<br/>
<p>
That's it! 👋
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@frankiefoto?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">frank mckenna</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-63753211686972213252022-02-20T11:04:00.002+01:002022-02-21T13:16:00.419+01:00What would slackbot do?<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEjfbNnnlftglQbod-yAv9e_1SF7GrVBE0iKCttONs9WD-pxr25QMBK_JRWs3CArUsOm46JsaCj2VbpaOCj5Y8LcQqchikITBazhPkTBd6KPTjcygcCPEFlEzNJkw5lPIesP1Izom-b5Ea-XAd_VLhOb5nT6MKCQ3u_kvqUMZqCL2V5Xpzw5EGC__A_y=s1920" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1920" data-original-width="1920" src="https://blogger.googleusercontent.com/img/a/AVvXsEjfbNnnlftglQbod-yAv9e_1SF7GrVBE0iKCttONs9WD-pxr25QMBK_JRWs3CArUsOm46JsaCj2VbpaOCj5Y8LcQqchikITBazhPkTBd6KPTjcygcCPEFlEzNJkw5lPIesP1Izom-b5Ea-XAd_VLhOb5nT6MKCQ3u_kvqUMZqCL2V5Xpzw5EGC__A_y=s400"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<p>
I recently found out about <a href="https://shouldideploy.today/" target="_blank">Should I deploy today?</a> and laughed. A humorous way of guidning us developers when we hesitate about deploying code to production. This one is extra funny on a Friday or Saturday. Every now and then, <a href="https://twitter.com/iamdevloper/status/450905958139834368" target="_blank">memes</a> about (not) releasing code to production on Fridays appear on my Twitter timeline. There's even T-shirts for sale, with <i>No Deploy Fridays</i> printed on them.
</p>
<h3>Funny because it's true?</h3>
<p>
I totally get the idea of minimizing the chance of ruining your or someone else's weekend. There's also a potential risk with postponing & stacking up features. The things we develop are there to add value, why make users wait for it?
</p>
<h3>Ship it!</h3>
<p>
Fridays shouldn't be different than any of the other working days, at least when it comes to deploying code to production. Just like on a Tuesday, you should probably think twice about deploying stuff just before you'll call it a day. When a fresh release has started causing errors for users, you might have already checked out from the office with loud music in the headphones, the mind focused on something else and unaware of the mess you've left to friends on call.
</p>
<h3>Deploy with some slack</h3>
<p>
It's a good idea to monitor the live environment for a while, making sure the deploy has gone well. Keep an eye on it, while working on the next thing. Prepare to act on alerts, errors appearing in your logs or messages in the company Slack.
<br/><br/>
Have a browser tab with graphs monitoring memory consumption, CPU usage and response times. For how long and what metrics to monitor is depending on the product. From my experience, being attentive to things like this is mostly good enough. When I call it a day a couple of hours later, I can go ahead and play that loud music in my headphones and relax.
</p>
<h3>When things go 💥</h3>
<p>
I haven't fact checked this - but unless a deploy is about database schema changes or to an offline machine, it should be relatively easy to roll back. Most deploys are probably easy to undo. If not, find out why and try making it easier.
<br/><br/>
I would suggest teams to practice: <strong>commit, push, merge, release to production and rollback</strong>. You'll probably find infrastructure related things to fix or tweak, maybe there's some automation missing? The team will know what to do when things go boom for real.
<br/><br/>
I've been fortunate working in environments with the basic automation already in place. We, the autonomous dev team, are the ones responsible for pushing features to production. We do that on a daily basis. Before, we hesitated about the deploy-on-a-Friday thing. Now we have better trust in our ways of working, and this Friday we deployed new features, dev experience improvements and bug fixes. Several small deployments went live, the last one about 3-4 hours before we called it a weekend 😀
</p>
<h3>Automate it</h3>
<p>
A good deployment process will stop and let the team know about when things break. The obvious code related issues will be caught this way. So, automation will get you far, but not all the way. That's why we'll need the monitoring. More importantly, releasing small portions of code changes is much safer than the ones with lot of code changes in it. <strong>Release small and release often</strong>.
</p>
<h3>What would slackbot do?</h3>
<p>
These days, when I hesitate if a new feature should be deployed right now or not, I'll just ask slackbot what to do.
</p>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEiVkXhD7Xw-dLHLOR6bmYG1Hgees5qGTYvsfW0MehBR5Ojgbu-vJNDG8fkOB4X1VoF9aRkm9gyHjfjL-C6dia2Winte6UB94zMMTo37c5FykOhg561xwwt2TUW0CjZw32Oc233mlmOkS5Nh9rJE39kjGx7e0JvGkbZiPZuZc8cC5QIxUUhNN9snupEz=s640" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="353" data-original-width="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEiVkXhD7Xw-dLHLOR6bmYG1Hgees5qGTYvsfW0MehBR5Ojgbu-vJNDG8fkOB4X1VoF9aRkm9gyHjfjL-C6dia2Winte6UB94zMMTo37c5FykOhg561xwwt2TUW0CjZw32Oc233mlmOkS5Nh9rJE39kjGx7e0JvGkbZiPZuZc8cC5QIxUUhNN9snupEz=s400"/>
</a>
</div>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@chiklad?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Ochir-Erdene Oyunmedeg</a> on <a href="https://unsplash.com/s/photos/toy-robot?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-47604475143477063532022-02-09T10:17:00.005+01:002022-02-13T21:31:40.758+01:00A fresh take on Monorepos in Python<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEjAYSYOWiAYUi-MjZ92QfnEMIsjwvaOhMXVd1N__VxTU93segknzR4y45ZdAEwXB2mLQtUg8STHpVjiYAi8OhTZ7IPewD7Ex-djXedVzI1-cIM22mPdzw9t_YH29r8sBb3mUTAE5LCEUG0IQ1JyeoQONiTIrO2aV9SHb7b34PB4Pkoz3gWz7Y_4bRLx=s1920" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1267" data-original-width="1920" src="https://blogger.googleusercontent.com/img/a/AVvXsEjAYSYOWiAYUi-MjZ92QfnEMIsjwvaOhMXVd1N__VxTU93segknzR4y45ZdAEwXB2mLQtUg8STHpVjiYAi8OhTZ7IPewD7Ex-djXedVzI1-cIM22mPdzw9t_YH29r8sBb3mUTAE5LCEUG0IQ1JyeoQONiTIrO2aV9SHb7b34PB4Pkoz3gWz7Y_4bRLx=s400"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<blockquote class="large">"... What if we had Polylith in Python ..."</blockquote>
<p>
For software development in general, it seems to be an ongoing trend towards using Monorepos. You know, organizing code in one big single repo. On the other hand, it seems to be a general trend in going the opposite way too. Developing features into many small repos.
<br/><br/>
I think that the JavaScript and Python communities in particular, are in favor of the latter approach. Developing features in isolation - in separate repos - and publish versioned libraries to package repositories (such as <i>npm</i> and <i>pypi</i>). That has potential to introduce some headache: versioning, backwards compatibility and keeping dependencies up to date. Probably also duplication of common source code and repeating the deployment infrastructure. You can solve those kind of issues, by using a Monorepo.
</p>
<h3>🔥 Monorepos</h3>
<p>
There are already solutions out there on how to develop software in Monorepos. It is my impression that they are mainly about solving the deployment experience. Which is good, because it is a huge problem in a lot of code bases out there. What about the Developer experience?
<br/><br/>
In the Clojure community, we have a thing called Polylith. It is an architecture, (including a tool) that is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk. The Polylith architecture is focusing on making both the Developer & Deployment experience great. There is also a very nice tool to help you get started with Polylith. Helping you create & keeping track on components, projects and much more.
</p>
<h3>🤔 So, what is Polylith?</h3>
<p>
From the <a href="https://polylith.gitbook.io/polylith/" target="_blank">official docs</a>:
<br/><br/>
"... Polylith is a software architecture that applies functional thinking at the system scale. It helps us build simple, maintainable, testable, and scalable backend systems. ..."
<br/><br/>
A Polylith code-base is structured in a components-first architecture. Similar to LEGO, components are building blocks. A component can be shared across apps, tools, libraries, serverless functions and services. The components live in the same repository; a Polylith monorepo. The Polylith architecture is becoming popular in the Clojure community.
</p>
<h3>🐍 What about Python?</h3>
<p>
Again, I find myself copying some existing Python code - a small module creating a "logger" - into a new repo. I think it’s the third time I do this. The logger code is too tiny to even bother packaging as a library, but still very useful in many code projects.
<br/><br/>
My mind is wandering. I'm daydreaming:
</p>
<blockquote>"What if we had Polylith in Python. I would just simply reuse an existing component."</blockquote>
<p>✨<strong>We have Polylith in Python now</strong>✨ </p>
<h3>Porting Polylith to Python</h3>
<p>
A couple of weeks ago, I decided to give porting Polylith to Python a try. I quickly realized that there are some fundamental differences between Python and Clojure to be aware of when implementing an architecture (and developing a tool) like this.
<br/><br/>
Especially when it comes to namespacing and packaging. Python code is based on the concept of modules. Also, Python code is not compiled into a binary.
</p>
<h3>Short note on modules, packages, namespaces & libraries</h3>
<p>
<i>In Python, a file is a module. One or more modules in a folder is a package. If you put a package in a folder, it is now namespaced. One or more packages can also be combined and built into a library to be used as a third party dependency. From my perspective, a Polylith component in Python should be a namespaced package, and not a library.</i>
</p>
<h3>Almost like Poetry</h3>
<p>When trying to figure out how to solve modules, paths & packaging problems I found Poetry. It is a tool that makes Python packaging and dependency management really easy. Out of the box, Poetry supports project based dependencies and references only. But the latest version of Poetry - currently in preview - has support for third party plugins. So I developed a plugin that enables the concept of workspaces, making it possible to reference dependencies outside of a project boundary.
<br/><br/>
This was the first step. A very important step. Now it became possible to build tooling support for a Python implementation of the Polylith architecture. Earlier this week I released a <a href="https://github.com/DavidVujic/poetry-polylith-plugin" target="_blank">first version</a> of such a tool. It is a new Poetry plugin, containing the very basic tooling support for the Polylith architecture.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiJdb3yrhB4Wg3LYGP6mcXQZFpLxUISH0vMhFVwXBI3jjKXcHVM2yzCv5Uooh7VhGiAsiKcVzpsTGyzhPD-AASJBfkKaBfQEjKP__KQHuzqmZETefYaZOKK0J2fAZYzUM9Otv4vzLzhLhReDaI2We3xOtzTErrdmU7R1DXvwxbr2oZbk_YRo1EDjRjc=s1108" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="254" data-original-width="1108" src="https://blogger.googleusercontent.com/img/a/AVvXsEiJdb3yrhB4Wg3LYGP6mcXQZFpLxUISH0vMhFVwXBI3jjKXcHVM2yzCv5Uooh7VhGiAsiKcVzpsTGyzhPD-AASJBfkKaBfQEjKP__KQHuzqmZETefYaZOKK0J2fAZYzUM9Otv4vzLzhLhReDaI2We3xOtzTErrdmU7R1DXvwxbr2oZbk_YRo1EDjRjc=s320"/></a></div>
<h3>poetry-polylith-plugin</h3>
<p>This <a href="https://github.com/DavidVujic/poetry-polylith-plugin" target="_blank">brand new Poetry plugin</a> will help you create a Polylith workspace and add components, bases & projects to it. Poetry itself will handle the building & packaging. It is still in early development, but you should be able to develop apps, serverless functions and services with it.
<br/><br/>
As of today, I wouldn’t (at least not yet) recommend building libraries with it, mostly because of how the packaging is done in Poetry & the nature of Python modules. I’ll try to figure out a way to solve this in the future.
<br/><br/>
There’s also a lot missing & left to do compared to the original poly tool. But hey, you gotta start somewhere 😀
<br/><br/>
Check out the 10 minute video for a quick overview.</p>
<h3>Learning about Polylith</h3>
<p>I recommend to <a href="https://polylith.gitbook.io/polylith/" target="_blank">read the official docs</a> to learn more about what Polylith is (and what it isn’t) - even though the code examples are about Clojure. You can easily bring the ideas into a Python context, now that we have tooling support for it.
<br/><br/>
My intentions with this is to introduce the Polylith architecture to the Python community, and I very much hope that it will be useful in your daily work as a developer.</p>
<div class="separator" style="clear: both; text-align: center;"><iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="yfUYIRcuDSo" width="400" height="322" src="https://www.youtube.com/embed/yfUYIRcuDSo"></iframe></div>
<p>
Direct link to the <a href="https://youtu.be/yfUYIRcuDSo">Porting Polylith to Python intro</a> video.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@kkos?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Katarzyna Kos</a> on <a href="https://unsplash.com/s/photos/breeze?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-61466410647498647612021-12-22T22:20:00.006+01:002021-12-23T09:57:30.781+01:00Simple Component Driven ClojureScript<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
text-align: center;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEjf482QHi0DWl8-DSejY8fBnljid9at8URxRbgnBi-hw2tdxMO9oFtLnANvqFgZ0QHfr1VpQ9yYUZUkk6-P3Pq5FH-vy-bflGIAQdBQ5PRoQdU5478622i01R4vndoHn71ammaWSdHZpz5_dvBpy3t9ziPv-3DhzER20PJ5mwj3uSL-sdlhrMkTsgHa=s1920" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1440" data-original-width="1920" src="https://blogger.googleusercontent.com/img/a/AVvXsEjf482QHi0DWl8-DSejY8fBnljid9at8URxRbgnBi-hw2tdxMO9oFtLnANvqFgZ0QHfr1VpQ9yYUZUkk6-P3Pq5FH-vy-bflGIAQdBQ5PRoQdU5478622i01R4vndoHn71ammaWSdHZpz5_dvBpy3t9ziPv-3DhzER20PJ5mwj3uSL-sdlhrMkTsgHa=s320"/>
</a></div>
<div style="font-family: 'Open Sans', sans-serif;">
<h3>That Component Driven thing</h3>
<p>
Component Driven is about developing user interfaces and apps in a certain way. In short, it is about grouping relevant code into <strong>components</strong>, combining compontents into <strong>features</strong>, combining features into <strong>views</strong> (or pages) - and combining the views into an <strong>app</strong>. I don't think it should be that much about the tools.
<br/><br/>
Just like Test Driven Development, Component Driven Development encourages us to start small. In a way, it also reminds of <strong>LEGO</strong> when combining building blocks into something useful.
<br/><br/>
As your app grows and the coding is done deeper into the app structure, developing user interfaces might be more of a challenge. Finding the spot where a new feature is to be added, might require some clicks and navigation. You know, stuff like logging in, a specific app state - or even waiting for data fetched from a third party service.
<br/><br/>
This is when the idea of Component Driven Development is really useful. To simplify and speed up the development, we need something simple to host our components. A UI that is isolated from any app specific state or dependencies. A place where a component can be rendered without the need to set things up. Having that, it is possible to develop a component in isolation and test it with different input.
</p>
<h3>Do I need a tool?</h3>
<p>
Storybook is a tool that will solve all of this, with a bunch of other really nice things like docs, previews, example usage, variants, code snippets and viewports. But it is possible to try out the Component Driven style of creating user interfaces, without going all-in with a new tool.
</p>
<h3>Start small</h3>
<p>
I’m suggesting a very simplistic approach, that will enable rendering components in a short-lived isolated view. You throw it away when done.
<br/><br/>
Here, I’m using shadow-cljs and have defined a new alias that is targeting a browser. I have also set entry points. A setup almost identical to the web app alias. This one will run in a separate port and the output is compiled to a separate public folder. In that one, I have copied the HTML template from the public folder used by the web app.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEi-07XCKd7P3_tSSfRICJYtRJtWWz2k9Yp4FOixBG7Kx3JxIkY3F4eB9MYX7twlEtwX6BwlasHdydUCi3mBBNjGhNYMW9kOUdChRkTW17mcHXl71BSe7SvSp1UuODKuxwhud2Yc8A3WyOyfZUJjHxLAMKzD6NQMF0K1hXsg9kB9qZAubkpycCO5Qys6=s1206" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="816" data-original-width="1206" src="https://blogger.googleusercontent.com/img/a/AVvXsEi-07XCKd7P3_tSSfRICJYtRJtWWz2k9Yp4FOixBG7Kx3JxIkY3F4eB9MYX7twlEtwX6BwlasHdydUCi3mBBNjGhNYMW9kOUdChRkTW17mcHXl71BSe7SvSp1UuODKuxwhud2Yc8A3WyOyfZUJjHxLAMKzD6NQMF0K1hXsg9kB9qZAubkpycCO5Qys6=s400"/>
</a>
</div>
<figcaption>Click to enlarge the image, or browse an example <a href="https://gist.github.com/DavidVujic/533d528bb4bb3719416458dfea52c8f2" target="_blank">gist here</a>.</figcaption>
</figure>
<h3>Keep it Simple</h3>
<p>
The entry point here is a simplistic and simplified version of the one used in the actual web app. But this entry point does only care about the component you are developing right now. Here is where you render and try out a component during development. This is a temporary place to work Component Driven. It’s almost like when evaluating code in a REPL, but with HTML and compiled JavaScript in a browser.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEjG1CKUNXpCyX4xntD2LQqwHKisLmjX2YsS_2IWZ5DM-ORnjuDSQL0lacaPzHAjgTpdpDnhtUbmT7TeZqHZGC8UGZ8GOym4r-rV12jQz5fwNbwGOFcRxQVh8qRlcT_5bgVmVxgo_fs4TuBuH-a-H_NxCHAUnG4CbNKv6M8-XT6y3dyVjUySWDgBwdyj=s1242" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1242" data-original-width="912" src="https://blogger.googleusercontent.com/img/a/AVvXsEjG1CKUNXpCyX4xntD2LQqwHKisLmjX2YsS_2IWZ5DM-ORnjuDSQL0lacaPzHAjgTpdpDnhtUbmT7TeZqHZGC8UGZ8GOym4r-rV12jQz5fwNbwGOFcRxQVh8qRlcT_5bgVmVxgo_fs4TuBuH-a-H_NxCHAUnG4CbNKv6M8-XT6y3dyVjUySWDgBwdyj=s400"/>
</a>
</div>
<figcaption>Click to enlarge the image, or browse an example <a href="https://gist.github.com/DavidVujic/533d528bb4bb3719416458dfea52c8f2" target="_blank">gist here</a>.</figcaption>
</figure>
<p>
Run both aliases. The App in one browser folder, the component in a different one.
</p>
<figure>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEgdg7BRJikJsM_q9tZ97cbTK1DzUNBk3nGP5EMfov_vUNMDCqEJbN0Z2wBfHCXyl_wzK3N6hWdFVS8Bs-Ux0L2IohBBlqdwyembNDLXzf4yd_bq--3WNZE_u3IaW0yJjK9dIv2Q9HVTKWJG2BTBm7MGc0OO7ug8tYLUv5EhTyatJEQzf1O8b2wH1SdJ=s640" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="306" data-original-width="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEgdg7BRJikJsM_q9tZ97cbTK1DzUNBk3nGP5EMfov_vUNMDCqEJbN0Z2wBfHCXyl_wzK3N6hWdFVS8Bs-Ux0L2IohBBlqdwyembNDLXzf4yd_bq--3WNZE_u3IaW0yJjK9dIv2Q9HVTKWJG2BTBm7MGc0OO7ug8tYLUv5EhTyatJEQzf1O8b2wH1SdJ=s400"/>
</a>
</div>
<figcaption>The browser is a powerful tool, with nice developer features.</figcaption>
</figure>
<h3>Take it to the next level</h3>
<p>
At some point, you will probably want something more long-lived than this very simplistic scratch-style approach. That’s where Storybook fits in!
</p>
<h3>Component Driven ClojureScript with Storybook</h3>
<p>
With Storybook, you can focus on styling, different viewports and UI events when developing a single component. You write "stories" (i.e. running components in isolation), including documentation and lot of in-browser interactivity. The stories & docs can also be shared, by deploying it as a separate web app.
<br/><br/>
Recently, I talked about getting started with Storybook at the <strong>re:Clojure 2021</strong> conference.
<br/><br/>
In this less than 25 minutes talk, I demo how you can use Storybook in your ClojureScript app and work Component Driven.
<br/><br/>
I hope you’ll like it!
<div class="separator" style="clear: both; text-align: center;"><iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="beMFh99EE7w" width="400" height="322" src="https://www.youtube.com/embed/beMFh99EE7w"></iframe></div>
</p>
<p>
Direct link to the <a href="https://youtu.be/beMFh99EE7w">Component Driven ClojureScript with Storybook</a> video.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@brett_jordan?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Brett Jordan</a> on <a href="https://unsplash.com/s/photos/keep-it-simple?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-50658455723363497032021-10-10T17:50:00.001+02:002021-10-10T17:50:24.068+02:00Functional Python<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsj1wxtxRO8UR2_zTxpgMyMZajQbEVBtFkfBlO0U0hQQKzaiiWvvbDZgOOe4imoUeD4jGEkRZ5mObSBwXE5HgAyqB57DQawNCwbZJC-7YnSPVfkpOhMeI42sgdIrNntZ5q-VYh8-Vl62E/s1920/charlein-gracia-mqKKCsZgzXQ-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1285" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsj1wxtxRO8UR2_zTxpgMyMZajQbEVBtFkfBlO0U0hQQKzaiiWvvbDZgOOe4imoUeD4jGEkRZ5mObSBwXE5HgAyqB57DQawNCwbZJC-7YnSPVfkpOhMeI42sgdIrNntZ5q-VYh8-Vl62E/s320/charlein-gracia-mqKKCsZgzXQ-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<h3>All in Functional</h3>
<p>
In a <a href="https://davidvujic.blogspot.com/2021/10/a-pythonic-railway.html">previous post</a>, I had some concerns about doing too much functional programming in Python and that it might not be <i>Pythonic</i>. But I'm not going worry about that here. I'll be writing Functional Python, with some help from a very nice library called <a href="https://toolz.readthedocs.io/en/latest/index.html" target="_blank">toolz</a>.
</p>
<p>
I have added example code in <a href="https://github.com/DavidVujic/functional-python" target="_blank">this GitHub repo</a> and have also recorded a <a href="https://youtu.be/VHS9wtNcnE4">short video</a> (about 8 minutes long). You'll find it embedded in this post, just scroll down a bit.
</p>
<h3>Functional in a nutshell</h3>
<p>
The video is very much inspired by one of my favorite recordings about functional programming in general, and about Clojure in specific: <a href="https://youtu.be/C-kF25fWTO8" target="_blank">Clojure in a nutshell</a> (Func Prog Sweden), by <strong>James Trunk</strong>. I have learned a lot from that talk and figured it would be fun to copy the workflow and examples that James uses in his excellent talk.
<br/><br/>
So, I've been translating Clojure code to Python. You'll also notice in the short video that I'm practicing a thing called <a href="https://davidvujic.blogspot.com/2021/09/can-we-have-that-in-python-too.html">REPL Driven Development</a>.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="417" data-original-width="529" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYPlQ7SJERCHmwEsQcdtnsiTIpA4xd9i_Q53N1yF2y-mhciGhYeFwi9BPyYIj6A2OJJdAjTlN4VMqsKKH6JhxC4ae61MHz4EsqaTWpQ-u8ctA4zfUzpb-6cURLvQvJqeoZIieF2A1m9-Y/s320/longest-words-python.png"/>
</div>
<figcaption>Pipe functions with the toolz.thread_last function.</figcaption>
</figure>
<h3>Composing, piping, chaining, threading</h3>
<p>
What I've learned so far is that it's a smooth ride to apply functional concepts into a language like Python. After all, functions are first class citizens here. Composing functions in a minimalistic - and readable - way is joyful. Yeah, these things also encourage to aim for simplicity in functions. Keeping functions <strong>stateless</strong>, <strong>single purposed</strong> and <strong>testable</strong>. All of that is really good stuff.
</p>
<h3>Intuitive, Pythonic & Effective?</h3>
<p>
Are those one-liners, like that <code>thread_last</code> thing that I use heavily in the video, <i>Pythonic</i>? Well, I guess it depends. As with most things, the features of cool toolz should be used wisely.
<br/><br/>
<strong>Yes, Python should be used wisely.</strong>
</p>
<p>
Remember to run <code>import this</code> in a Python shell now and then to remind yourself about and reflect on the <strong>Zen of Python</strong> 🧘. Does it say anything at all about OOP, or functional?
</p>
<p>Here's an improvised 8 minute recording with live coding functional Python, using toolz:</p>
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="VHS9wtNcnE4" width="400" height="322" src="https://www.youtube.com/embed/VHS9wtNcnE4"></iframe>
<p>
Direct link to the <a href="https://youtu.be/VHS9wtNcnE4">Functional Python</a> video.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@charleingracia?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Charlein Gracia</a> on <a href="https://unsplash.com/s/photos/kung-fu?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-57666343788372170992021-10-08T07:29:00.004+02:002021-10-08T08:51:03.422+02:00A Pythonic Railway?<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlxfUBDb0bS4kIWoKHPZKc6gPmqufW7DHdiZELhPHYzBLifri_XRbRpUEiikXvjjCbmhEkwSGM9uLK_AhTv9Z-U9XPlzObOXsjMGPpB4W-673GtEhSIKfuR58T2dcC_sFNV4bhi_9QbEg/s1920/alex-paganelli-Bm2fpXUnt9c-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1920" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlxfUBDb0bS4kIWoKHPZKc6gPmqufW7DHdiZELhPHYzBLifri_XRbRpUEiikXvjjCbmhEkwSGM9uLK_AhTv9Z-U9XPlzObOXsjMGPpB4W-673GtEhSIKfuR58T2dcC_sFNV4bhi_9QbEg/s320/alex-paganelli-Bm2fpXUnt9c-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<blockquote class="large">
<span>What is Railway Oriented Programming, and how it can be implemented in Python? Is it even a good fit for Python?</span>
</blockquote>
<h3>🐍 Python on Rails</h3>
<p>
I’ve experimented with this Railway thing to find out if it is a good fit. If you haven't already, make sure to watch <a href="https://vimeo.com/113707214" target="_blank">this excellent talk</a> about Railway Oriented Programming from NDC London by Scott Wlaschin.
<br/><br/>
As I understand Railway Oriented Programming, it is about adding error handling to code without cluttering the program flow.
<br/><br/>
The source code <a href="https://github.com/DavidVujic/pythonic-railway" target="_blank">in my GitHub</a> repo is an attempt to implement a lightweight, and probably a sloppy, variant of Railway Oriented Programming in Python.
<br/><br/>
What I'm trying to achieve is to add the functional concepts of Railway Oriented Programming, and try keeping a Pythonic mindset. I believe that means not going <i>all-in functional</i>, but I could be wrong here. 🤔
</p>
<h3>🚂 Why Railways?</h3>
<p>
As I understand it, this way of writing code is about adding error handling and still keeping a <strong>happy path</strong> style in the code. This is done by wrapping - or as I’ve been doing, decorating - functions.
<br/><br/>
The functions are wrapped (or decorated, as in the examples <a href="https://github.com/DavidVujic/pythonic-railway" target="_blank">in my GitHub</a> repo) to catch failures. The <strong>output</strong> of a failed function call will be the <strong>input</strong> to the next one. The next function will choose a track based on the input: taking the success track, or the fail track. In general, taking the fail track means bypassing the function and skip to the next one.
<br/><br/>
By using a two-tracked approach in functions, the error handling will be separated from the program. Functions will be less cluttered with <code>try except</code> error handling and probably also with less of <code>if else</code> flow control clauses. As result, In many cases, this will mean <strong>less code</strong> in functions.
<br/><br/>
Less code in a function will probably be easier to understand and reason about. Less branches in a program flow will <strong>keep the functions simple</strong>. I think less is nice. Yeah, less is more. 😎
</p>
<p>Here’s an improvised 5 minute video I’ve made to demo the concepts Railway Oriented Development in Python:</p>
<iframe class="BLOG_video_class" allowfullscreen="" youtube-src-id="357sRh9OVEg" width="400" height="322" src="https://www.youtube.com/embed/357sRh9OVEg"></iframe>
<p>
Direct link to the <a href="https://youtu.be/357sRh9OVEg">A Pythonic Railway?</a> video.
</p>
<p>I would love to hear your feedback about this!</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@alexpaganelli?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Alex Paganelli</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com1tag:blogger.com,1999:blog-1409352204799783992.post-13254012678476895932021-09-29T23:23:00.002+02:002022-11-29T21:12:39.884+01:00Can we have that in Python too?<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqqAihqoyTMAM51Bs2xjahv98CCFapGr0uYHBXi9YI5LO9AFGS8bi4Bq5qiKKhLknjemcr1f6stMB384aLspKVtIGghsmGIxH7CeRDvHfat37Ahs1WWZwq4Xq9mvfr0I5x-4rEkjzNhZY/s1920/hitesh-choudhary-D9Zow2REm8U-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1080" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqqAihqoyTMAM51Bs2xjahv98CCFapGr0uYHBXi9YI5LO9AFGS8bi4Bq5qiKKhLknjemcr1f6stMB384aLspKVtIGghsmGIxH7CeRDvHfat37Ahs1WWZwq4Xq9mvfr0I5x-4rEkjzNhZY/s320/hitesh-choudhary-D9Zow2REm8U-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<blockquote class="large">
<span>(REPL Driven Development)</span>
</blockquote>
<h3>REPL Driven what? 😐</h3>
<p>
<strong>REPL Driven Development</strong> is about fast feedback loops during development. It is not about typing code into a terminal window. Don’t do that. A <i>Read Eval Print Loop</i> (REPL) is often described as a shell tool that you use to try out a programming language. That’s not what I mean with REPL Driven Development.
</p>
<figure>
<div class="separator" style="clear: both; text-align: center">
<img alt="" border="0" width="280" data-original-height="656" data-original-width="1202" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBIiwCFZAljfNHmg-fTNj23VNEWV6i14qtsIdRexGFgLKFhMpnc3kkPRwFae65bKHN76JP72obIBa_Y8WU4J7eP-zcoTMGyS2AdYOwVzrCVA26kZH-MopSTBijI-yWLxM0boG4qnRMBOQ/s320/Ska%25CC%2588rmavbild+2021-09-29+kl.+18.51.45.png"/>
</div>
</figure>
<p>
<strong>What is it then?</strong>
</p>
<p>
With this workflow, the REPL is acting behind the scenes. You don’t have to type code in a shell. Yes, a Read Eval Print Loop process is running, but in an interactive mode. You do all the work in your code editor. That’s where you have <strong>autocomplete</strong>, <strong>syntax highlighting</strong>, your favorite <strong>color theme</strong> and <strong>everything</strong>.
</p>
<h3>⚡ The Workflow ⚡</h3>
<p>
The feedback loop is fast in REPL Driven Development. You evaluate code blocks and functions as soon as you have typed them. The result of an evaluation is printed out on your screen. A common thing with REPL Driven Development is to write some short lived code snippets next to the actual code that you are developing. With this, you are able to quickly run a function with some test data as input and verify the output.
<br/><br/>
This workflow is similar to Test Driven Development (TDD), but the feedback loop is much faster. After a while, you might discover that those code snippets should actually be a proper unit test. Well, wrap the code in a test function, move it to a separate file and your’e all good.
</p>
<blockquote class="large">
<span>So, REPL Driven Development is lazy TDD?</span>
</blockquote>
<p>
Yeah, that’s probably right. Another way of looking at it might be that it is <a href="https://davidvujic.blogspot.com/2021/02/test-driven-development-deluxe.html">Test Driven Development Deluxe</a>.
<br/><br/>
The Interactive REPL is an essential tool in the Clojure & ClojureScript ecosystems. It is a superpower I’ve not seen before anywhere else. Once learned, it is something you’ll want in other languages too. But is that even possible, outside of a Lisp environment?
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="270" data-original-width="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtEm9Q46lswTJ8oiccD0cAPrEWd7MIRQ6t0-uDfm2w8xhjXoQ0BAq_F_cYmXzmCXqQ0miQIO_8bQefnmfpf0Min3j4jpPXt80IsvFbufA5d5wYHOYZ0roEJJAcJ2-BolgWYsWcyA5XZeE/s320/Interactive-Web-Devlopment.gif"/>
</div>
<figcaption>Evaluating vars, state and functions in real time with ClojureScript</figcaption>
</figure>
<h3>Can we have that in Python too?</h3>
<p>
I think I have found something similar to the Clojure way of writing software, that works really well. You can pass an entire buffer, a selected region or a function from the code editor into a running <a href="https://ipython.readthedocs.io/en/stable/" target="_blank">IPython</a> process, that quickly evaluates the code and prints the result. I have replaced the standard Python shell with IPython. I’ll explain why.
<br/><br/>
When evaluating code, IPython will output the result while the cursor remains in the code editor. That is what I mean with fast feedback loops. Evaluated functions, variables and imports can also be redefined - without having to restart anything. I’ve <a href="https://github.com/DavidVujic/my-emacs-config#python-shell" target="_blank">configured IPython to auto-reload</a> sub modules when changed, to avoid having to restart the process. That feature is essential for this kind of workflow and the main reason why I’ve switched out the standard Python shell.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmmW5YZiP34t24C5Xa9aYf_u4wr-A-Vhy8zG0hpOMbUfyJ-xvEc67PrScEYsRjuyev-Adzl54rh-3EbHzet17aT3jBGEYxYG2g82s7048XdwVK2bl6NXDn3ppxyDpa8JGrcsWdBQcuqdg/s320/REPL+Driven+Python+blogg+-+SD+480p.gif"/>
</div>
<figcaption>REPL Driven development in Python</figcaption>
</figure>
<p>
It would be really cool to have the output pop up in the editor, right next to the actual code (as when evaluating Clojure or ClojureScript). I think that’s mainly a tooling thing, currently not available.
<br/><br/>
Also, I haven’t yet figured out how to connect to an actual running Python program and redefine functions or variables while the program is running (as you do in Clojure).
<br/><br/>
However, IPython is closely related to <a href="https://jupyter.org/" target="_blank">Jupyter</a>, often referred to as an interactive notebook. There’s a thing called <strong>the kernel</strong>, handling code evaluation and returning result to connected clients. Sounds a bit like NRepl & Cider! I should probably dig deeper into and learn more about the possibilities with Jupyter.
</p>
<h3>Joyful Python</h3>
<p>
To summarize, even though there are some limitations, I think the REPL Driven Development workflow makes writing Python code joyful. The fast feedback loop is a great thing to have at your fingertips. Just like TDD, REPL Driven Development can be very helpful when writing <strong>testable, simplistic</strong> and <strong>functional</strong> code.
</p>
<p>
And it is fun too. 😁
</p>
<p>
<strong>Update:</strong> <i>have a look at <a href="https://davidvujic.blogspot.com/2022/08/joyful-python-with-repl.html">this post of mine from 2022</a>, for more info about how to setup your code editor. You'll also find a 5-minute video in there about this style of development.</i>
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@hiteshchoudhary?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Hitesh Choudhary </a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-78376222537601826422021-09-13T09:46:00.011+02:002021-09-13T09:52:53.911+02:00ClojureScript. Amplified.<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
figcaption {
font-size: 0.8em;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibVwi8tgW_aFP2kioI4RYxUQwpkVMB2fvaFPOlNhSjkKrWfjgWpowrWi8LPBdq0Lii4IpzFdpFFDuRpHvz-whWc4w1FJZmvP_9dV9LgIWCRl6sl_wcpvbxTBEB2hZmn_LfsAG_sJu3Yso/s1920/drew-patrick-miller-_o6AAx9dl_Y-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1277" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibVwi8tgW_aFP2kioI4RYxUQwpkVMB2fvaFPOlNhSjkKrWfjgWpowrWi8LPBdq0Lii4IpzFdpFFDuRpHvz-whWc4w1FJZmvP_9dV9LgIWCRl6sl_wcpvbxTBEB2hZmn_LfsAG_sJu3Yso/s320/drew-patrick-miller-_o6AAx9dl_Y-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<blockquote>
"… we're going to build a web app with storage, authentication and everything …"
</blockquote>
<p>But how?</p>
<h3>My favorite building blocks 🤩</h3>
<p>
I have tried out some tools that I find very useful. As you may have guessed, the code is developed with <strong>ClojureScript</strong>. A language I appreciate more each day. Developing an app, without <a href="https://davidvujic.blogspot.com/2021/02/interactive-web-development.html">the Interactive REPL</a>? That would be taking a huge step backwards.
</p>
<p>
I have found that <strong>Material-UI</strong> works really well in the development of the app I'm working on. The components library cover many usage scenarios, it is well documented and there's a lot of code examples to learn from.
</p>
<p>
Recently, I added <strong>Storybook</strong> and really like the idea of developing user interfaces according to Component Driven Design. It reminds me of how I experiment with code using Test Driven Development. Just like TDD, I think tools like Storybook and Devcards changes the way we develop, to the better. <strong>Start small & build quality in.</strong>
</p>
<p>
Let's begin with a building block from my favorites list: <strong>AWS Amplify</strong>.
</p>
<h3>A basic setup for AWS Amplify & ClojureScript</h3>
<p>
The official AWS Amplify documentation is all about JavaScript. In this post, and in <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">this example code repo</a>, you will find a ClojureScript setup that I think will get you up & running quickly. First, set up AWS Amplify by following the first parts of <a href="https://docs.amplify.aws/start/getting-started/installation/q/integration/react/" target="_blank">the Official getting started guide</a>. You will be creating an AWS account and installing a CLI to initialize your setup.
</p>
<p>If want more details, have a look at <a href="https://davidvujic.blogspot.com/2021/08/hey-webpack-hey-clojurescript.html">Hey Webpack, Hey ClojureScript</a>. It's a post describing a ClojureScript & Amplify specific setup that is used in the example repo.</p>
<h3>Start coding, but where? 🤔</h3>
<p>
A pitfall might be where to place the auto generated client code from Amplify. I usually put files like the <i>aws-exports.js</i> at the same level as the root namespace.
<br/><br/>
In <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">the example code repo</a>, the ClojureScript source code lives in the <code>src/main</code> folder. That's also where I've added the <i>aws-exports.js</i> file.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="131" data-original-width="550" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEju7QMP-wsuVQTHdRY9H442sdeKOIcJ5FJ3XAAbaEyk5nWGkWKKpuNABVb6i1B9EPoFFKAu0kA7XoBT4b5Kc42RNONzhWardhWPZyRjWvgtWAyw_XiK8lIC06beCMOpuEJN2TJw9kMdFdk/s320/core-ns.png"/>
</div>
</figure>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="110" data-original-width="550" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6_QP4stGdfv9yjoAnCGXOF3UjzrtkNirRk8Oq2VlTEAjA5K9vJRhoPBj2c_MSsIDaaw35RamtnHg3J0BfN8EKyXFkIobjEz2Wk3x6EHOSEQ7oX39wa70yhOYtGeL3K-j4EoRklom84wY/s550/core-init-fn.png"/>
</div>
<figcaption>Configure Amplify in the init function of your app.</figcaption>
</figure>
<h3>Login & Logout with ClojureScript & the Amplify CLI</h3>
<p>
Adding authentication can easily be done by using the Amplify CLI.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="89" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNhSUwb0mbOHeKIEwbfhzfVpAJqMuazjChrc6QOfjZT1F3q37PP-IKTMni6z3SMLdabqG70fB_kyh4U1xPcnBqqLRDCOwqo6yX0pOXc5EDHsEtfumohr-immfFd-SANeS53Ein3x6bsnw/s320/amplify-add-auth.png"/>
</div>
</figure>
<p>From the docs:</p>
<blockquote>
"... The Amplify CLI supports configuring many different Authentication and Authorization workflows, including simple and advanced configurations of the login options, triggering Lambda functions during different lifecycle events, and administrative actions which you can optionally expose to your applications ..."
</blockquote>
<h3>Use the Auth features in Amplify UI</h3>
<p>
There's a very useful Higher-Order React Component (aka <a href="https://reactjs.org/docs/higher-order-components.html" target="_blank">HOC</a>) in the AWS Amplify UI Library, that will render login and signup views.
<br/><br/>
I've chosen this less-lines-of-code approach by using the <i>withAuthenticator</i> HOC. To customize things like signup fields or styling, there is also a fully customizable Authorization component available in the Amplify UI library. Or you can use your own components.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="131" data-original-width="550" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3Xx0baDa34xiP-3McqnP-wiPY672AIByzKDfdvgFYrpM5vY9-knBHeb2OlKoN8PxNfsjxO4brVlou9GbQzorrJVbHSChnHLhWN0NpRP9kTPYf-M2z2ixXJNbgjZm89ZsTuUsK-WmPUkg/s320/core-app-fn.png"/>
</div>
<figcaption>The views entry point.</figcaption>
</figure>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="215" data-original-width="650" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh691bq6b5cZ55dltdeQvCdUIabviG27F6B2-UJmumGJTaywWhczUcypgxSzZ07DuGamk04Cm7neaqVLCwSZP1Dp8cQATX6UKnaJZfaKABiBx_GL5bDgSbjnS3VAj03tTDm0e6_ymRjouU/s320/with-auth.png"/>
</div>
<figcaption>The passed in component is wrapped in the <i>withAuthenticator</i> HOC.</figcaption>
</figure>
<p>
If you already have browsed the code in my example repo, you may have noticed the Reagent <i>reactify-component</i> and <i>as-element</i> functions. These functions are needed when passing components between ClojureScript and JavaScript.
<br/><br/>
Read more about React <a href="https://reactjs.org/blog/2015/12/18/react-components-elements-and-instances.html" target="_blank">Components, elements and instances</a> and how the interop is <a href="https://github.com/reagent-project/reagent/blob/master/doc/InteropWithReact.md" target="_blank">done with Reagent</a>.
</p>
<h3>Create, read, update and delete</h3>
<p>
So far, we have used built in features that will render UI views and handle all auth communication with a backend service. To store, read and update data, we can use the <strong>Amplify API</strong> feature. Use the CLI to create a backend:
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="89" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh51h3qTWi2Zo95JmgPRrEiSSqQbU5UmeV3njSF6x6LX9qz0OnMdXfO1O1oBXZoEfEI0VVsR62oCU8Nt4-jiim3lxDBZOzng8vM19oajEhimJ_tGsh8xZiLt108M71KcqL7oI4glR4IueU/s320/amplify-add-api.png"/>
</div>
</figure>
<p>
I have chosen <i>GraphQL</i> in my example code, but there's also a REST API option. When using <i>GraphQL</i>, all you need to do is to define data models. Amplify will create the backend infrastructure according to your models. In a matter of minutes, there are <strong>AWS AppSync</strong> endpoints & <strong>DynamoDB</strong> tables created for your app.
</p>
<h3>GraphQL</h3>
<p>
Here's a GraphQL model. An authenticated user should be able to add data to the profile, like a summary or an "About me" text. The user will be able to create, delete and update the data in the profile. Other authenticated users can read the data when they visit the user's profile.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="194" data-original-width="700" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEil9nZgIRgb7yeLH_eGvuyCzWTCJpAvQ7fH6ZO5kKDILJzidoaBo7UN9cLarqosLvOvHig_0ZlalMFsbzy-PKjhGOZrcW9QkRQAY4rHWHlv2NJMUgMLu0jCTnlUfWGYtndWAumyrZGnic8/s320/graphql-usersettings.png"/>
</div>
<figcaption>The schema files belongs to the backend, and lives in the amplify folder of your repo.</figcaption>
</figure>
<h3>async ClojureScript</h3>
<p>
The Amplify Client Library handles the communication with the backend services. All calls are asynchronous. In ClojureScript, you can use <i>core.async</i>. I haven't yet dig into the <i>core.async</i> stuff, so I'm using Promises here. I think it's a quite straight forward approach.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="173" data-original-width="650" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhxC_x8vO1oUGtSmGqtcxjdO_nVmqTVZ4xbKufhWdvznri8gHTa8Lx_urMTff1SjK2sP4zq9O77CYQR6GBLLFvsCX_MxCvvm0XiLgHcCCpZuPkDG1fAko8afhDe6cKJdQvx_AlHWK8QcI/s320/amplify-fetch-settings-api.png"/>
</div>
</figure>
<p>
The data going through the wire is JSON. To make the developer experience better, I think it might be a good idea to add conversion functions to-and-from Clojure data structures. Like this one, converting user settings JSON to Clojure data:
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="215" data-original-width="700" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx6EQ0FMmCPIOaBV3G3dIKhlEqj3wYoJABZby8rEmuFF5N7vecSYzzlfmr12JtPsqWhD1VwEngmG8Sv4UqUxZoXk3KvlTx5pPHvGkNXNdPS3vo98y8h-cvRqQr_gEvMROW2MChicwoM9M/s320/settings-data-to-clj.png"/>
</div>
<figcaption>Read more about <a href="https://shadow-cljs.github.io/docs/UsersGuide.html#infer-externs" target="_blank">the ^js type hints</a> in ClojureScript.</figcaption>
</figure>
<h3>Finally: Ship it! 🚢</h3>
<p>
Here's an excerpt from an <i>amplify.yml</i> that defines the build steps. If you add a definition file in your repo, AWS Amplify will use it in the CI/CD process.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="320" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHqPHB8JTMyprrYcvoysTg6fxCx3b6Fhl7hu4kTE5IbKcRXQlyc3Oy9POiqwNPHW4kTCTSbPMgKWnBADCgWRclw3LZyc_sx-tWg5dTP7-3LkOnZzjMs0WgP3n7OdO7SpzkqZ9UaIZqH58/s320/amplify-build-config.png"/>
</div>
<figcaption>This is the <a href="https://github.com/DavidVujic/clojurescript-amplified/blob/main/amplify.yml" target="_blank">amplify.yml</a> that I use in my example code repo.</figcaption>
</figure>
<p>
There's more details at the official AWS docs about <a href="https://docs.aws.amazon.com/amplify/latest/userguide/build-settings.html" target="_blank">build settings</a> and <a href="https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/amazon-linux-install.html" target="_blank">about amazon-linux-install</a>.
</p>
<h3>Add UI building blocks to the amplified app</h3>
<p>
Now we have our amplified ClojureScript app. Let's add UI building blocks from Material-UI, a library that is very popular in the JavaScript & React ecosystem. Here, I'm using it via a ClojureScript library called <a href="https://github.com/arttuka/reagent-material-ui/#reagent-material-ui" target="_blank">reagent-material-ui</a>. The library makes it possible to write components using Clojure data structures. You can read more about Material-UI and ClojureScript in my previous post <a href="https://davidvujic.blogspot.com/2021/09/material-design-in-a-functional-world.html">Material Design in a Functional World</a>.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="362" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-A84N64Cp759uwLC7SlAVCQqzCCpn3qg66jCr9uvfZrWfDgSsrZiD6UqfEUf_TIBNjAg0Jo1C8-JUcKlPx5gFo6-ofIlfMGMWHkp4KoAbKcal_ajbIGvJmKEHJLACSBun0qFpPuJHbZc/s320/material-ui-button-cljs.png"/>
</div>
<figcaption>Totally unnecessary css styling used here, but why not?</figcaption>
</figure>
<h3>Testing the UI building blocks with ✨ Storybook ✨</h3>
<p>
I find Storybook very useful when experimenting with components and user interfaces in isolation. Being able to focus on one specific thing. There's many different aspects of developing an app, and it is sometimes difficult to handle all of them at the same time.
<br/><br/>
Storybook is a place for trying out things like css, styling, different viewports and UI events. You won't try out the AWS Amplify code here. The Interactive REPL is a better tool for that.
<br/><br/>
You'll find more about Storybook in <a href="https://davidvujic.blogspot.com/2021/08/component-driven-clojurescript-with-storybook.html">Component Driven ClojureScript with Storybook</a>.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="449" data-original-width="522" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqtfCwXS8f8Ao5TG8UsyA5T0IjGqptOC_28pw1D0wRBtf568vhCahBj7WA2u18xywlQZjqqwdGzkvB97PKAXuGTlT1SIx0fKA2WasL7NRt-QKaS4dtGuUe_MiVxBtCFJsfAM0_CFB-sM8/s320/storybook-actions-output_cropped.gif"/>
</div>
</figure>
<p>
Finally, check out <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">my GitHub repo</a> for ideas on how to create apps with the building blocks <strong>AWS Amplify</strong>, <strong>Material-UI</strong>, <strong>Storybook</strong> and <strong>ClojureScript</strong>.
</p>
<p>
That's a combination I'd like to call ✨ClojureScript. Amplified. ✨
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@drewpatrickmiller?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Drew Patrick Miller</a> on <a href="https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com2tag:blogger.com,1999:blog-1409352204799783992.post-19404759747468935702021-09-02T19:17:00.007+02:002021-09-12T21:46:26.392+02:00Material Design in a Functional World<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh190IizOdC0A_52mb_i123UzXARb_epdCsrQVqU6PfuCMiKVgR2TNybEC_uyTnFBqHdn9SBp3XXARqx3rwZMFQoE6E8iPt4tOQnu2egvGD2wr4HbNF2FqzXrNtD2_l_V-g99r02MrwRRA/s2048/thierry-lemaitre-DCTz78QCY24-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" height="320" data-original-height="2048" data-original-width="1638" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh190IizOdC0A_52mb_i123UzXARb_epdCsrQVqU6PfuCMiKVgR2TNybEC_uyTnFBqHdn9SBp3XXARqx3rwZMFQoE6E8iPt4tOQnu2egvGD2wr4HbNF2FqzXrNtD2_l_V-g99r02MrwRRA/s320/thierry-lemaitre-DCTz78QCY24-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<blockquote>
"... Material is a design system created by Google to help teams build high-quality digital experiences for Android, iOS, Flutter, and the web ..."
<br/><sup>from <a href="https://material.io" target="_blank">material.io</a></sup>
</blockquote>
<h3>ClojureScript & React is A 💕 Story</h3>
<p>
With Clojure, you can write functional & minimalistic code. With ClojureScript and <i>Reagent</i>, you can write functional & minimalistic <i>React</i> components. By adding <i>re-frame</i> to it, we also have a powerful way of handling state changes and triggering events.
</p>
<p>
The code you write in ClojureScript is instantly compiled to JavaScript React components that runs in your browser. The <strong>entire JavaScript ecosystem</strong> is available with this kind of setup.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="1226" data-original-width="1610" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrHMupEU_FJVen8Btr9aa7EwWhlz6MdQ77MzI6CTCwm0WMR0m1lRT4Ifl-YhPbv_ipBbNPVdX3ayG9RxI4xrKl3cEHpPsFMoUV1s87zsdFgKllJdFFnyZR4Pp29Ja1bKJOvFR77qJC94c/s320/Ska%25CC%2588rmavbild+2021-09-02+kl.+15.52.22.png"/></div>
</figure>
<blockquote>
"... React components for faster and easier web development. ..."
<br/><sup>from <a href="https://material-ui.com" target="_blank">material-ui.com</a></sup>
</blockquote>
<h3>Material-UI</h3>
<p>
Material-UI is a very popular <i>React</i> framework. And we can use it with ClojureScript. But the JavaScript interop can sometimes be bit of a hassle. Fortunately, there is a great ClojureScript library called <a href="https://github.com/arttuka/reagent-material-ui" target="_blank">reagent-material-ui</a> that simplifies this a lot.
</p>
<p>
One thing to notice is that the <strong>Material-UI</strong> components use <i>React Hooks</i> heavily to inject styling and theming into components. This is something to think about when using <i>Reagent</i>. Basically, just use the functional component syntax <code>[:f> my-component]</code> and you’re all good.
<br/><br/>
Have a look at the <a href="https://github.com/arttuka/reagent-material-ui#common-pitfalls-in-reagentreact-interop" target="_blank">reagent-material-ui docs</a> about common pitfalls in the React/Reagent interop and a fully working example <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank"> in this GitHub repo</a>.
</p>
<h3>Storybook: a playground for components</h3>
<p>
Here’s me playing around with the Card component, using ClojureScript and Storybook. I describe the Storybook integration in detail <a href="https://davidvujic.blogspot.com/2021/08/component-driven-clojurescript-with-storybook.html">in this post</a>.
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCU7vrt53tgLsX_EciuQWGGutaoD6eKYMpGBsEi9UTpEFZo-isyFjuvKMoel8MDXQJM-B1at-0A8z46cI4TdyCPBomIReVoDHvx9ogW6aPaokXsCVVE5arY2MsDabgQAfzmVUFTDzUVTQ/s320/clojurescript-and-material-ui.gif"/>
</div>
<figcaption>The source code is available <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">at GitHub</a>.</figcaption>
</figure>
<h3>No copy-paste?</h3>
<p>
I guess it can be a bit overwhelming for a Clojure developer to look at example code from the <a href="https://next.material-ui.com/components/cards/#MediaControlCard.js" target="_blank">Material-UI site</a>, with JavaScript/JSX syntax that is quite verbose (you’ll be even more overwhelmed by the TypeScript examples).
</p>
<p>
A JavaScript developer would probably <strong>copy-paste</strong> the example code and make some adjustments. We can’t do that. But it is worth the effort writing the code from scratch when discovering that the result is <strong>way more simplistic</strong> than the original. The amount of code written is usually half the size of the original JavaScript code. Yeah, that’s Clojure!
</p>
<figure>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="1133" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJq7uhlP0vfR_gXBitJ58Poy3pIHz3qS0etq2Eifvaky8jh0wwUAzRCI9BVv0EW-0up_X7bGWgqoZVxnIlosLeZUxYk_wTHArE62OBOxvCNdWATalgLhCSZFycfcFX0QvMpb4xR4YKrgM/s320/cljs-vs-js.png"/></div>
<figcaption>ClojureScript to the left, JavaScript to the right. Less is more.</figcaption>
</figure>
<p>
Check out <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">my GitHub repo</a> with examples on how you can implement Material-UI components in ClojureScript, with hooks and styles - while having <strong>fun</strong> writing functional code.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@tlemaitre?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Thierry Lemaitre</a> on <a href="https://unsplash.com/s/photos/playground?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-66441745015994087462021-08-31T18:17:00.014+02:002021-09-01T12:11:19.922+02:00Component Driven ClojureScript with Storybook<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwv67vBrMmg-0uzg7QhrbFuzFTOpL8PVsF45188TdSKlxfCdDR4w1vUNtqN-kJaFkIOnBaBMbkv7eN-klqpT83SO0QiPiHYsybYOBSxzzsVa6UackBVKucpyeAB5bb3scidjxgZpVx4n4/s1920/etienne-girardet-EP6_VZhzXM8-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1440" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwv67vBrMmg-0uzg7QhrbFuzFTOpL8PVsF45188TdSKlxfCdDR4w1vUNtqN-kJaFkIOnBaBMbkv7eN-klqpT83SO0QiPiHYsybYOBSxzzsVa6UackBVKucpyeAB5bb3scidjxgZpVx4n4/s400/etienne-girardet-EP6_VZhzXM8-unsplash.jpg"/></a></div>
<div style="font-family: 'Open Sans', sans-serif;">
<h3>Component Driven?</h3>
<p><strong>Start small.</strong> Develop components in isolation. Don’t forget to pause & reflect on the code you’ve written. Can it be split into several small components?
<br/><br/>
✨ When you’re ready, combine your components to create features.
<br/><br/>
✨ Add the features to a view.
<br/><br/>
✨ Add the view to your app.
<br/><br/>
You can read more about <a href="https://www.componentdriven.org/" target="_blank">Component Driven User Interfaces here</a>.
</p>
<h3>Storybook</h3>
<p>
I’ve just begun to use <a href="https://storybook.js.org/" target="_blank">Storybook</a> and currently learning about it. What I especially like so far, is that the tool encourages you to write UI components that are independent … and functional?
<br/><br/>
This makes it possible to develop user interfaces in isolation. Just pass data and actions as parameters into your components. <strong>Keep it simple</strong>.
This way of working reminds me a lot of <a href="https://polylith.gitbook.io/polylith/" target="_blank">Polylith</a>, an architecture I have used in backend focused Clojure projects and that I enjoy a lot.
</p>
<h3>ClojureScript & Storybook?</h3>
<p>
You will find a <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">fully working example in this repo</a>. I have added Storybook to a ClojureScript web app, using <i>shadow-cljs</i>, with additional <i>npm scripts</i> located in <i>package.json</i>.
</p>
<figure>
<div class="separator" style="clear: both;"><img alt="" border="0" width="400" data-original-height="438" data-original-width="1376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgj3qACfx_Y8Gb5mHyEYfDx4byzqsnJwtCTu95PxFoMX46sSBsSbUp5Q7cyyWjgfnnVajXaG5T2CZ9hsCiF3dDEBmsUJj-MbdGDgRxHQpxY7b_9ni5BH7GpiyuPFV4xVs3o0VID8zxeIo0/s320/shadow-cljs-storybook-with-regexp.png"/></div>
<figcaption>Update: a new version of shadow-cljs supports <strong>ns-regexp</strong> for the npm-module target. No need to manually add new stories!</figcaption>
</figure>
<p>
The code in my GitHub repo is a continuation of the work I’ve described in <a href="https://davidvujic.blogspot.com/2021/08/hey-webpack-hey-clojurescript.html">an earlier post</a>, about combining <i>shadow-cljs</i> with <i>Webpack</i> and <i>AWS Amplify</i>.
</p>
<figure>
<div class="separator" style="clear: both;"><img alt="" width="400" border="0" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilcG2_gUlIBP8KylKA5Jmjke1a5gBppBPYhb3OnTCSNpqLgedvaEtl6L69lixkeMgzv2T8Y-hgAywEpy0tz9rZUweNv1GBAtBAM5jHsjQMs6A6rYF9uHXbNJVz-dyOTDteE_oVwfvlaz8/s320/clojurescript-and-storybook.gif"/></div>
<figcaption>The Clojure Interactive REPL and Storybook working well together.</figcaption>
</figure>
<h3>No JavaScript quirks?</h3>
<p>
Well, yeah some. The JavaScript ecosystem is evolving fast, tools are updated with breaking changes and sometimes that causes compatibility issues. In this case it is Storybook and Webpack 5. But I have solved it with <a href="https://github.com/DavidVujic/clojurescript-amplified/blob/main/.storybook/main.js" target="_blank">a few extra rows</a> of configuration. <strong>Don’t worry</strong>.
</p>
<figure>
<div class="separator" style="clear: both;"><img alt="" border="0" width="400" data-original-height="1026" data-original-width="1438" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG2EdJnzkgSEKN148qVZbvvIBW5sgr_1cYfj7sht00KZ2H_8rwA37bwAIuIYegZdaXE_nFA-C7mgYNl38Y7oHQFfmbnrM-6aKwFzRFWSLAV_SHv5oIQL008LH5PlCir3TdXWgE5i1nXEs/s320/storybook-greeting.png"/></div>
<figcaption>An example of a Storybook story written in ClojureScript</figcaption>
</figure>
<br/>
Have a look at the <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">fully working example at GitHub</a>.
<p>
Many thanks to Shaolang Ai, that wrote <a href="https://shaolang.github.io/posts/2021-02-14-storybook.js-with-shadow-cljs/" target="_blank">this blog post</a> about shadow-cljs and Storybook that I’ve learned a lot from.
</p>
<br/><br/>
<sup>
<span>
Top photo by <a href="https://unsplash.com/@etiennegirardet?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Etienne Girardet</a> on <a href="https://unsplash.com/@etiennegirardet?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com3tag:blogger.com,1999:blog-1409352204799783992.post-41993433580940876292021-08-29T19:54:00.003+02:002021-08-30T07:05:01.937+02:00Hey Webpack, Hey ClojureScript<style>
blockquote.large {
font-size: 1.2em;
}
blockquote.large span {
font-style: italic;
}
</style>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLtRcFLpR89KDDe3jvR86yojKi2Mqma4HPoNeoNBMfZ78AKfTB1_dVaItAq8bCrtmQuVqAKfkdeIO-GgEX3N0O26xjH9Lmpfi6IbSWroc6x3gUEVICT_1Yana45LvqTj6kDBER-L7QCYM/s640/kamil-pietrzak-h2wAwxuCs0w-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLtRcFLpR89KDDe3jvR86yojKi2Mqma4HPoNeoNBMfZ78AKfTB1_dVaItAq8bCrtmQuVqAKfkdeIO-GgEX3N0O26xjH9Lmpfi6IbSWroc6x3gUEVICT_1Yana45LvqTj6kDBER-L7QCYM/s320/kamil-pietrzak-h2wAwxuCs0w-unsplash.jpg"/></a></div>
<div style="font-family: 'Open Sans', sans-serif;">
<p>
✅ ClojureScript is great.
</p>
<p>
✅ AWS Amplify is great.
</p>
<p>🤔 Why are there errors in my browser console?</p>
<h3>ClojureScript. Amplified.</h3>
<p>
Some time ago, I decided to give <a href="https://aws.amazon.com/amplify/" target="_blank">AWS Amplify</a> a try for a new project. With Amplify, you can quickly create and deploy web apps and AWS backend services by using your command line.
</p>
<p>Even though the Amplify code examples and docs are all about JavaScript, I wasn't too keen on abandoning my favorite REPL powered language: Clojure. 🤩</p>
<p>
So, I decided to combine two great things: AWS Amplify & ClojureScript!
</p>
<h3>Amplifed, yes. But with 🔔.</h3>
<p>
I have learned a lot from this <a href="https://www.omnyway.com/add-serverless-aws-amplify-authentication-to-a-clojurescript-re-frame-app/" target="_blank">blog post</a> by Robert J Berger about how to setup AWS Amplify with ClojureScript. New versions of the AWS Amplify libraries are released regurarly. The current Major version include breaking changes that probably will cause ClojureScript web apps to raise errors and stop working.
<br/>
<br/>
Until recently, I have had to rollback updates and use earlier versions of the Amplify libraries. Nothing wrong with that. But my inner Alarm Bells have been ringing constantly since noticing the word <strong>legacy</strong> in the official Amplify docs for those library versions (the <i>aws-amplify-react</i> library and the Amplify Auth feature in particular).
<br/><br/>
The newer Amplify libraries rely on <strong>Webpack</strong> specific features under the hood. The example setup described in the official Amplify docs include using <i>react-scripts</i>, that (if I'm not mistaking) will package code with Webpack. That kind of setup is not supported by <i>shadow-cljs</i>, a tool that is very common in ClojureScript projects.
</p>
<h3>Breaking things</h3>
<p>
In addition to being stuck using legacy code, the app will break each time I add a third party library from <i>npm</i>. I have learned that it is caused by not having a fixed version for amplify libraries defined in my <i>package.json</i> file. A quick fix is to set all AWS Amplify libraries to a fixed version and use that one forever. A version that is becoming more legacy for each day. 🔔🔔🔔
</p>
<h3>Fix the broken things</h3>
<p>
The main issue with AWS Amplify libraries, ClojureScript and shadow-cljs is described in <a href="https://github.com/thheller/shadow-cljs/issues/816" target="_blank">this GitHub thread</a>. There is also a suggested solution hinted by Thomas Heller, the creator of <i>shadow-cljs</i>.
</p>
<h3>A solution: shadow-cljs & Webpack working together</h3>
<p>
I have created an example app using AWS Amplify and ClojureScript <a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">in this repo</a>. Here, <strong>shadow-cljs</strong> will only create a target JavaScript file containing <strong>requires</strong> to all of the third party JavaScript dependencies.
</p>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="396" data-original-width="1380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjJARlEvJIG-L5iPgCQnzBKRK2vpqyFz2So_nlaDUpRhhAM1cVXPOiFtg266xQo6quODsC08aDMvQqSoXmFpCQPIrAixxt7jruY6hXvw7kokD29TiFNuZkSyCA3l6lgeWMtmqlTsYIznc/s320/shadow-edn.png"/>
</div>
<p>
<strong>Webpack</strong> will use that target to create a JavaScript bundle. In a continuous deploy environment, this can easily be automated by executing two scripts instead of one (the shadow-cljs build and webpack).
</p>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="438" data-original-width="1376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvYvUrJ6ku3OPhiKuoPOHP83Ov1rtKl9vQ3xsGTDVjxasiijoBHoJoUIOujk3-WJ9HHtgpM-dBVwI6pLk2slN_TeLw6qSaF-2lN38mCvIVg5yvSkgI_bsyaf-syteioLEZ2-1ZxiEP0AY/s320/ci.png"/>
</div>
<p>
<a href="https://github.com/DavidVujic/clojurescript-amplified" target="_blank">This solution</a> makes it possible to use the very latest versions of the AWS Amplify libraries, without any css quirks or being forced to use legacy code. Hey Webpack, Hey ClojureScript!
<br/><br/>
Nowadays, my inner alarm bells are nice & quiet.
</p>
<sup>
<span>
Photo by <a href="https://unsplash.com/@kpietrzakweb?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Kamil Pietrzak</a> on <a href="https://unsplash.com/s/photos/parachute?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0tag:blogger.com,1999:blog-1409352204799783992.post-83687991209416278792021-04-18T19:07:00.011+02:002021-04-20T20:05:16.030+02:00Confessions of an Open Source Contributor<style>
blockquote.large {
font-size: 1.2em;
}
p.big {
font-size: 5em;
margin-top: 0;
margin-bottom: 0;
}
blockquote.large span {
font-style: italic;
}
.tag {
display: inline-block;
height: 26px;
line-height: 26px;
padding: 0 20px 0 23px;
margin: 0 10px 10px 0;
}
.help-wanted {
background: green;
color: white;
}
.good-first-issue {
background: azure;
color: black;
}
</style>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL7lIVWiIHKqzI5NSBgkop_XSNxyTwBVSCftgdAlG_X2O76IVaMnBUmg2TnB38-MvVgj3MeIDsKHEh-Dz4hX64rdaCbk3DCEM72AekH-zmEDgbi703rso8WHngEIL1iB6pOVvtuA9TY58/s320/justin-lim-tloFnD-7EpI-unsplash.jpg" style="display: block; padding: 1em 0; text-align: center; ">
<img alt="" border="0" width="400" data-original-height="1440" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL7lIVWiIHKqzI5NSBgkop_XSNxyTwBVSCftgdAlG_X2O76IVaMnBUmg2TnB38-MvVgj3MeIDsKHEh-Dz4hX64rdaCbk3DCEM72AekH-zmEDgbi703rso8WHngEIL1iB6pOVvtuA9TY58/s320/justin-lim-tloFnD-7EpI-unsplash.jpg"/>
</a>
</div>
<div style="font-family: 'Open Sans', sans-serif;">
<p>
I am one of the contributors to an Open Source project called <strong>zookeeper</strong>. It's an Apache ZooKeeper client made for Node.js. You can find <a href="https://www.npmjs.com/package/zookeeper" target="_blank">the package on npm</a> and the <a href="https://github.com/yfinkelstein/node-zookeeper" target="_blank">source code at GitHub</a>.
</p>
<h3>What’s ZooKeeper?</h3>
<blockquote>"Apache ZooKeeper ... providing configuration information, naming, synchronization and group services over large clusters in distributed systems ..."</blockquote>
<p>
In short, <a href="https://zookeeper.apache.org/" target="_blank">Apache ZooKeeper</a> is made of two parts: a service, and clients that communicate with it. Included in the packaging from Apache, there is a client made for the Java platform and also a C based client. You will most likely find one for your favorite language out there as a third party Open Source project.
</p>
<h3>Why joining Open Source?</h3>
<p>
A couple of years ago, I was part of a team in an organisation that use Apache ZooKeeper heavily. The team maintains a Node.js based app that communicates with a ZooKeeper service.
<br/><br/>
One day, the app began to raise strange warnings and unexpected errors. We looked into it and found out that the reason was probably caused by the client no longer being compatible with the API of the service.
<br/><br/>
There wasn't much happening in the client's repository:
<br/>
unanswered questions from users, issues not taken care of and pull requests not being reviewed.
<br/><br/>
<strong>The project was abandoned.</strong>
<br/><br/>
So, we used a different client that seemed fairly updated and stable. The strange behavior in our app was now gone. Great! It worked on our machines and in production. All good!
</p>
<br/>
<blockquote class="large">🙋"Hey, team. I can't install your app on my Windows machine."</blockquote>
<br/>
<p>
Sometimes other teams need to run our app locally for testing and such things. It turned out that our new client didn't support Windows at all.
<br/><br/>
Wait a minute, isn't Node.js supported in all major platforms? Well, yes. But this one is building a Native Node.js AddOn using bash scripts. Windows don't have that.
</p>
<p class="big">🤦♂️</p>
<p>
Even though it wasn't my fault only, all I could think of is that <strong>it was my fault</strong>. I have ruined it for the teams at this workplace.
<br/><br/>
What else to do than try fix it on my spare time? So, I grabbed the source code to try to understand what's going on behind the scenes.
</p>
<p class="big">😭</p>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="414" data-original-width="1266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUMCIzansvWE5RIn1l3wz5YQly_ObAvHRWjsql3Otv9PuT01F6m8LJqa8CSh0F6_ZxRAbxuLQ0DmGIfXxOp514Nl-QHbAZuegMwFZBgnH-6fLaz8DP85Rwh06I2yDD6Q2j8cToiZWN5Dc/s320/gitclone.png"/>
</div>
<br/><br/>
And that's how my Open Source journey began: triggered by <strong>guilt</strong> and feeling <strong>ashamed</strong>.
<br/><br/>
<h3>Coding in the Dark</h3>
<p>
Late nights and weekends were spent trying to figure out how to solve this huge problem, including C/C++ code, CMake, bash and node-gyp.
<br/><br/>
One day, I figured it out. I had succeeded in making it work on Mac OS X, Linux and on Windows! The next day - a sunny morning in beautiful Stockholm - I went to the office smiling. Exhausted by the lack of sleep, but happy.
</p>
<p class="big">😎</p>
<div class="separator" style="clear: both;">
<img alt="" border="0" width="400" data-original-height="414" data-original-width="972" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTHeTxPabb3621ful-hjB20AP8NyQ1rEqWm0h-IW6JLKjSUJAHYqoWOjmXQd4OBY72wqC2v6EbztbLs7sux_xqV4-c-CVXrq3WlESAKyUeBP-qsmszdPDy5Gig_pZngayy_Oz7hOiPWEU/s400/gitcommit.png"/>
</div>
<br/>
<h3>Do I have to fix everything? 😟</h3>
<p>
While struggling with the source code, I realized there were more things that should be fixed. By then, I also realized that this Open Source project suffered from similar issues as the one we had replaced earlier: there wasn't really that much activity in the repo.
<br/><br/>
This is probably a common pattern: you work with something on a daily basis and contribute to Open Source projects. The years go by, you drift way from the project, simply because you now work with other things unrelated to the project.
</p>
<h3>How to maintain an Open Source project</h3>
<p>
Instead of grabbing the next task to spend late nights and weekends with, I decided to learn more about how to maintain Open Source projects.
<br/><br/>
I found a very useful <a href="https://docs.github.com/en/communities" target="_blank">guideline at GitHub</a>, and I began to add notes about all kinds of things I thought should be done. Other Open Source projects were using labels in a way that I liked, so I added labels like <span class="tag help-wanted">help-wanted</span> and <span class="tag good-first-issue">good-first-issue</span> to most of the tasks.
<br/><br/>
I think labels can help people getting an overview of the tasks and to understand the degree of difficulty.
<br/><br/>
Every time a question comes up, or someone is filing a bug report, I have replied to most of them within a day or two. I think this is important. The person reporting will know there is someone that has actually received and read the message.
<br/><br/>
As a result, really nice things have happened. The project has received Pull Requests from people located all over the world. Solving a broad variation of things from typos in the docs, to rewrites of the source code to modern JavaScript.
</p>
<h3>What's happening today?</h3>
<p>
Today, I think <a href="https://davidvujic.blogspot.com/2020/12/the-zookeepers-new-years-resolution.html">the project is stable</a> and working well. I don't work nights or weekends. Even though I have moved on and no longer work with ZooKeeper as before, the feedback from users is my main inspiration to keep on working. Also, the amount of users of the client looks pretty good.
<br/><br/>
It inspires me to do the best I can with the project.
</p>
<br/><br/>
<sup>
<span>
Photo by <a href="https://unsplash.com/@justinlim?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Justin Lim</a> on <a href="https://unsplash.com/s/photos/cartoon?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText" target="_blank">Unsplash</a>
</span>
</sup>
</div>
David Vujichttp://www.blogger.com/profile/05640815111394960040noreply@blogger.com0