yield code();

Farewell, Rust

Thu, 19 Feb 2026 18:00:00 GMT

Prelude

When I was in 9th grade — last year before high-school — my best friend persuaded me to join, together with him, the schools programming club. At first, I hesitated, but later agreed. I am immensely thankful to him for this.

There, step by step, we learned Pascal using Turbo Pascal. Little by little, we grasped the basics of the language: variables, operators, string manipulations, data structures — until eventually, remaking Conway’s Game of Life. And then, summer break came. Other than HTML, which is not a programming language, Pascal was the first real programming language I have learned and used.

But I did not stay with Pascal for very long time. After the summer break, in high-school, I have chosen the “Software Engineering” branch of studying, and there we learned C. Just like Pascal, we started step by step: basic operators, string manipulations, memory allocation, data structures, and the final boss, the void *.

I fell in love with C. The precise control over memory; passing variables by reference or pointer; the need to allocate memory for every data structure that you wanted to create.

After 3 more years of studying, I have graduated high-school. In that time, I have improved my C skills, learned some PHP, picked up C++, and tried to build a variety of programs (don’t judge me too hard, these are ~20 years old): a clone of BattleCity, 3D software renderer, IRC bot, unfinished operating system kernel, unfinished game engine, TTF renderer for OpenGL.

After high-school, I have enrolled in a two-year college program that would earn me a Practical Software Engineering Degree. During the summer break between the first and the seconds years, I had a choice to make: go work in McDonald’s (I had experience working as a waiter during high school summer breaks), or find a job in software engineering. After 2 seconds of hesitation, I have crafted a CV and started to send it to every position I saw online. And despite the fact that I really wanted to get a software development position in C or C++, nobody would hire me. Eventually, I have secured a web-development position in PHP (thank you very much my first employer for giving me a chance).

And this would be the last time I’d touch C or C++. Dynamic, high-level languages such as PHP, Python, and Ruby — are more suited to the dynamic nature of web development. You rarely need to squeeze the maximum performance from your hardware, since for every second you gain by optimizing data structures allocations in C, you lose 10x more waiting for network or disk requests to resolve. And so, collectively, we all agreed that the web is better to be written in dynamic languages.

But just like your first true-love, C would hunt me. I would obsess about micro-optimizations, and would get mad that I can’t control when variables are allocated, and how to pass things by reference or pointer without making a redundant copy. And then, Rust became a thing.

Web development in Rust

Just like every software engineer, I started learning Rust by building my own game engine. Do you even call yourself a software engineer if you never tried to build your own game engine? And I fell in-love with Rust. C was aging, and failed to catch up with the moving world of hipster development: modern tooling, linters, formatters, package management. Rust offered the best of both worlds: low-level control of memory allocation, and variable life-cycle, while having modern development practices: a simple-to-use compiler, built-in linter and formatter, and of course a modern package management tool and repository.

The community was loving Rust, and the language was growing very fast. And for the first time you could write a web application using a low-level systems-programming language. ¹ After picking up the basics of the language, and abandoning my game engine, I have decided to build a web application in Rust. And, in the end of 2023, I have succeeded. I have built, and shipped a fully operational web application that was making me money, in Rust.

Throughout the years, I learned from my mistakes, and shared my progress here, here, and here. Eventually, the application became a Frankensteins’ monster were the backend would be pure Rust, while the frontend was a statically generated Astro website. Until, in the end, I have reached a road-block and could no longer maintain or extend the application.

I took a painful decision to migrate everything to Node.js, and conclude this experiment with Rust. It was hard for me to make this decision. I really loved working with Rust, and I wanted to make it work. My work with Rust earned me valuable human connections, and opportunities to present my knowledge at two conferences, and one meetup. I am happy that I have took the risk with Rust, but…

Farewell Rust, at least for now…

I’m not going to go over the positive sides of Rust, since I covered them in other articles such as Building a web app in Rust and One year of Rust in production. But I do want to talk about some of the struggles I had, both as a reminder to my future self (you can take C from me, but you can’t take my love to low-level programming), and as a cautionary tale to everyone else.

Keep in mind that everything I share is based on my perspective: I’m a solo founder, building and running a web application. There is no heavy workload like video processing, nor a need to have low latency — two things at which Rust excels. From my experience, and my product, the bottleneck is always the database, disk, or network. I’m also, somewhat, experienced in Rust, so I’m not the “I tried to build a web app in Rust, but I don’t understand borrow checker, so Rust bad” kind of guy. Now, with that out of the way, let’s begin.

Templating

As I mentioned earlier, from pure Rust + server side generated templates with tera, I have switched to a Rust API server + Astro static website that calls the API. The reason I moved away from rendering HTML in Rust, is the fact that I am too spoiled with type-safe templates. With Astro, I can use the .astro components that are type-safe. Since I use, and advocate for, Typescript, all my templates are type-safe, and there is very little chance to mistype a variable, or miss it. If you have a component like this, it’s very hard to make it compile with the wrong type (unless you do stupid stuff like props as any):

interface Props {
    username: string;
    email: string;
}

export function UserCard(props: Props) {
    return (
        <div>
            Hello {props.username} ({props.email})
        </div>
    );
}

With libraries like tera, handlebars, or mrml — your view is essentially separated from the props, and if you rename a field in the template, you must remember to rename it in the model as well. It’s possible to solve it with tests at the cost of longer CI/CD pipeline, but I will talk about compilation time a bit later. Functions inside templates are basically wild-west of strings. And you need functions. It’s hard to make rich templates without functions.

But there are libraries like maud or askama — I hear you say. And I agree, there are. These are essentially type-safe HTML templates that use the macro mechanism from Rust. By building, basically, HTML DSL using Rust macro, you can create compile time, type-safe templates. But then again, compilation is expensive, but more on that later.

Localization and Internationalization

I have been talking about localization back in 2015. Node.js ships with full icu support, and a set of Intl.* APIs to format numbers, lists, currencies, country names, you name it. On top of that, with libraries like i18next, one can get type-safe auto-complete for translations, something that I was not able to achieve in Rust. Yes, Rust has support for fluent translation files developed by Mozilla, and a minimal support for number formatting. But Node.js has everything you need in order to build and ship fully localized and translated web-applications. It’s a know fact that i18n is lacking in Rust (see: AWWY: Internationalization), and bindings for icu4c are in progress, but they are nowhere near what Node.js can offer it that regard.

The web is dynamic by nature

Take it or leave it, but the web is dynamic by nature. Most of the work is serializing and deserializing data between different systems, be it a database, Redis, external APIs, or template engines. Rust has one of the best (de)serialization libraries in my opinion: serde. And yet, due to the nature of safety in Rust, I’d find myself writing boilerplate code just to avoid calling .unwrap(). I’d get long chain calls of .ok_or followed by .map_err. I defined a dozen of custom error enums, some taking other enums, because you want to be able to handle errors properly, and your functions can’t just return any error.

Similar thing can be said about writing SQL. I was really happy with using sqlx, which is a crate for compile-time checked SQL queries. By relying on macros in Rust, sqlx would execute the query against a real database instance in order to make sure that your query is valid, and the mappings are correct. However, writing dynamic queries with sqlx is a PITA, as you can’t build a dynamic string and make sure it’s checked during compilation, so you have to resort to using non-checked SQL queries. And honestly, with kysely in Node.js, I can get a similar result, without the need to have a connection to the DB, while having ergonomic query builder to build dynamic queries, without the overhead of compilation time.

Okay, I hear you, let’s talk about compilation time.

Compilation time

Rust achieves its safety at the expense of, somewhat long, compilation time. On modern hardware, the compilation time is not that bad. It’s not good enough to have uninterrupted change-f5-preview cycle, but it’s good enough especially with incremental compilation.

However, the more crates you have, the more macros you use, the slower the compilation time becomes. So the problem becomes is how to achieve fast compilation during CI/CD. Due to the dynamic nature of the web, you often times find yourself in a loop of: there is an error in production → fix it → deploy. And you want this loop to be as fast as possible, because you have customers who can’t do things with your app.

I have my own hardware to run CI/CD workers. A dedicated VM on an Intel Core i5-7500 with 32GB of RAM. The VM has access to all 4 CPU cores, and would take about 14 minutes from push to master until the docker container was deployed on the server (about 12 of it is the docker stage with compilation). That’s with multistaged docker file, and a cached builder layer; with no cache, it would probably take ~20-25 minutes. And it does not include running tests or clippy as part of the CI/CD. I simply gave-up trying to set-up proper caching so tests, clippy, and compilation would be able to use the same cache.

Node.js, on the other hand, takes on average 5 minutes, including linting, and tests. And since I moved to Node.js, I added another back-office service, so I deploy more code, 3 times faster, while actually relying on running tests and lint as part of the CI/CD pipeline.

Ecosystem maturity

Rust has a very mature ecosystem in most aspects, but it is lacking in the web aspect. Need an obscure third party API? It’s probably not there. So you end up implementing APIs instead of doing core business logic. Each API is additional code that you need to test and maintain.

It’s not hard to implement REST APIs, but the lack of maturity does slow you down. I had to implement, and test, third party APIs, queue mechanism on top of PostgreSQL, code to validate webhook signatures. It was fun, but it comes out of the box in other, “standard”, languages.

Node.js is good enough

And honestly, Node.js is good enough. People like to criticize Node.js or the npm ecosystem, implying that there is something fundamentally wrong with JavaScript. And sure, JavaScript is far from being a good language. There are many quirks in the language, which you can learn only with experience, pain, and tears. And I miss stuff from Rust like the Result and Option types; I miss the match statement; and I miss the enum as well. But in all honesty, Node.js ecosystem has matured and is stable to write web applications. You have libraries like zod to validate request/response JSONs, libraries like kysely, @kitajs/html, and my own @mjmx/core to write type-safe SQL, HTML, and MJML. And async/await is still better in Node.js than in Rust. It’s always weird to need to bring thirds party async-trait crate because you want to have an async method in a trait.

There are still issues with Node.js. The never-ending deprecation of cjs in favor of esm; it’s mostly good, until you encounter a cjs package, and then it’s hell. The cumbersome eslint and prettier setup where you need to juggle between 2 files and dozen of plugins to get basic functionality you get with clippy out-of-the-box; I wait for the day biome will close the gaps, and becomes the standard tool. Workspaces are still not solved, although pnpm is better in this regard, but it’s nowhere near the workspaces in cargo. And the occasional struggles with typescript where the runtime seems to be changing too often; is it ts-node? tsx? tsm? The built-in typescript runtime in node? deno? bun?

Oh, and I really wish we all would just agree that snake_case is simply the best, most aesthetically pleasing, and easy to read case for programming. I really hate the fact that JavaScript has adopted the camelCase.

Closing notes

Building solo is hard. You wear 10 hats as software engineer, in addition to the dozen hats as an entrepreneur. I really wanted Rust to succeed, but I also need to move forward. I found myself ignoring bugs in Sentry because it meant going back to long compile times. I was postponing feature development because it meant slow iteration speed, and trying to synchronize backend REST API with frontend, or the need to visually re-test every page after a variable rename in a template. Call me spoiled, but I don’t have the luxury of development team, and someone checking my code.

Rust excels at non-visual things, i.e. things you can write and wrap with tests. But when you need to develop UI, it becomes painful. You wait for the code the recompile; worried about passing the wrong variable to a view.

From pure Rust, I had to go to Rust with a statically generated HTML + Alpine on top of it. To make sure my email templates are safe, I was considering writing a dedicated mailing service in NodeJS, just so I could use react.email (or my own @mjmx/core), and get some type-safety. It’s as if I was moving backwards, breaking my Rust monolith and rewriting parts in a dynamic language that is better suited for the web.

No, Node.js is not perfect. But at least I have one stack, and funny enough, I get more type-safety than I had in Rust. Sure, it’s a fake type-safety, JavaScript is a dynamic language, but I no longer have to jump between my view files and my model and try to keep them in sync. I no longer misspell translation keys, resulting in blank words; emails do not come with Hello {{dearCustomer}}.

It seems like most of my issues boil down to dynamic things: templates, i18n, SQL. If I were to write an API service today, I would probably choose Rust again, since API services do not have to deal with views or translations. I still don’t know how I would handle SQL though. ORMs are not my cup of tea, and other than sqlx, Rust seems to lack a good type-safe query builder.

I also miss the small footprint of the application. With Rust, the containers would use between 60 and 80 MB of RAM; but with Node.js, the lowest I have is 117 MB of RAM for the back-office, without any load. And it’s true what they say: use the right tool for the job. Rust shines in CPU-heavy tasks, and for sure I will be using it when I will have such tasks.

But until then, farewell Rust.

Yes, there is Go. But I never really had the chance to like Go. ↩

Introducing mjmx: a custom JSX runtime for mjml

Fri, 06 Feb 2026 18:00:00 GMT

In my opinion, one of the best things that happened because of React, is JSX. I know many people seem to hate on JSX, but I like it. With JSX, or more specifically TSX, one can write typesafe html, instead of relaying on libraries like handlebars, which is a great library, but it keeps your code and your variables separated. So every change in variables needs to be synced to the view-model, or worse, risk in runtime errors / blank values.

Recently, I’ve been doing some server-side rendering, and was using @kitajs/html as HTML JSX runtime. @kitajs/html is great, because it does not bring react or react-dom with it. And when I got to emails, I decided to try react-email and mjml-react. But I was disappointed in both.

First of all, I like mjml. It’s a great syntax, and it’s battle tested. react-email seems to not use mjml, but instead re-implement email rendering themselves. Now, react-email is a product by Resend, and the guys at Resend are doing email sending, so I bet they know how to render emails properly. Despite this, there is a second issue: I don’t like React, and I don’t want to bring react and react-dom just to render a couple of emails.

So I decided to try to write a custom JSX runtime for mjml. And that’s how @mjmx/core was born: mjmx Github.

It’s a custom JSX runtime that has no dependency on react or react-dom. It implements the minimum needed runtime to support all the mjml tags, as well as the allowed HTML tags. And with it, writing mjml becomes a breeze:

const Email = ({ name }: { name: string }) => (
  <mjml>
    <mj-body>
      <mj-section>
        <mj-column>
          <mj-text font-size="20px" color="#333">
            Hello {name}!
          </mj-text>
          <mj-button href="https://example.com">Click me</mj-button>
        </mj-column>
      </mj-section>
    </mj-body>
  </mjml>
);

const { html, errors } = render(<Email name="World" />);

Under the hood, render is just a proxy to mjml2html from the mjml library. This means that mjml compilation happens with each email you render. I would like to outsource the mjml compilation into a build / transpile step, but with the current approach it is not possible, unless you decide to re-implement email rendering yourself.

I hope you will find this library useful. Don’t forget to star it on GitHub, and feel free to reach out to me for questions, or open an issue / PR for suggestions and improvements.

Isolating Claude Code

Sat, 24 Jan 2026 09:00:00 GMT

Recently, I have joined the vibe-coding club. I used to laugh and vibe-coders who use Claude Code while accepting everything it asks for, until eventually CC decides to delete their entire home directory, turning their computer into a brick. But, life has a funny way of turning around and coming back at you. Before this happens to me, I decided to take action.

Not today CC

Moreover, according to the creator of Claude Code, Boris Cherny, Calude Code writes Claude Code.

This is how skynet started

And we all know that we can’t really trust AI. So let’s talk about isolating CC.

Built in Claude Code sandboxing

CC has built it sandboxing. You can configure it with /sandbox inside CC chat. You can read about CC sandboxing here. Long story shorty, Claude is good at escaping the sandbox. Eventually, I would find myself glancing at the output of CC and seeing something long the lines of:

permission denied to access db:5432
[CC] sandboxing is on, trying without sandbox

I might have accepted a permission here and there that allowed CC to escape its sandbox, so I don’t trust the built-in sandboxing.

Existing tools

The first option I was thinking about was: let’s see if an existing tool is available. I asked Claude to tell me how I can isolate it’s brother Code, and Claude was glad to help. It came up with 3 tools: docker sandbox, cco, and claudebox. I saw claudebox mentioned on HackerNews the other day, so I though I might be onto something.

How I work

I have adopted a workflow where I run all my auxiliary services in docker compose. Every app / service that I develop, has its own docker-compose.dev.yml which brings all the needed services (database, redis, etc) up, so I can fully isolate versions, data, and accidentals fucks-ups. Instead of installing all these tools on my host machine, trying to find workarounds for supporting different versions, I can just bring an entire stack for each app. And when I’m done, I bring it down, and my machine is not wasting CPU cycles. Or, if the project is no longer in development, I just nuke the entire docker container, with all its data, leaving my machine clean.

This means that if I run CC inside a docker container, this docker container needs to be able to communicate with my dev environment. I want CC to be able to run tests against the database, or even inspect the database directly.

docker sandbox does not support this. You can --mount-docker-socket which basically means that the docker where CC runs has root-privileged access to the Docker daemon, which kind-of beats the purpose of isolation.

cco (which stand for “Claude Condom”, funny name), has:

Network access: Full host network access for localhost development servers, MCP servers, and web requests
source

This means that if my host has access to a Docker container named some-secret-app that CC should not access, cco won’t protect me.

Lastly, claudebox does not support joining an existing Docker network, hence I wasn’t able to find a way to give claudebox access to my auxiliary services.

Rolling my own

As an experienced software engineer, I know that the best practice is to always ~~use existing solutions~~ roll your own solution. I need CC to access my development Docker network, so the reasonable thing to do, was to put CC in the same docker compose file. Create a Dockerfile for claude:

FROM ruby:3.4.7

RUN apt-get update && apt-get install -y \
    nodejs \
    npm \
    libvips-dev \
    && rm -rf /var/lib/apt/lists/*

RUN npm install -g @anthropic-ai/claude-code

WORKDIR /app

CMD ["bash"]

NOTE

I use ruby as the source image, since in this particular project it was for Ruby on Rails development.

Then, I added a service into my docker-compose.dev.yml file:

  claude:
    build:
      context: .
      dockerfile: Dockerfile.claude
    stdin_open: true
    tty: true
    init: true
    environment:
      CLAUDE_CONFIG_DIR: "/claude"
      DATABASE_URL: "postgresql://user:password@db:5432/db"
      REDIS_URL: "redis://redis:6379/0"
      TERM: xterm-256color
    profiles:
      - claude
    volumes:
      - .:/app
      - ./.claude-credentials:/claude

It took me some time to figure out how to persist authentication between container re-runs, and the key was to mount claude credentials, but also specify CLAUDE_CONFIG_DIR. I also spent a lot of time trying to fix terminal issues, where key presses would not register, so it’s import to set tty, init, stdin_open and TERM. Finally, I created a bash script to run the docker container:

#!/bin/bash

case "$1" in
  continue)
    docker compose -f docker-compose.dev.yml run --rm claude script -q -c "claude --continue" /dev/null
    ;;
  *)
    docker compose -f docker-compose.dev.yml run --rm claude script -q -c claude /dev/null
    ;;
esac

Use ./claude.sh to start a new CC session, or ./claude.sh continue to resume an existing session. Since the docker container is ad-hoc, and is removed after each run, this means you can run as many containers as you want. This was it, and I thought I was happy.

But then, I started to question myself. Docker container shares the kernel with the host machine, meaning CC could still bring down my entire computer. Unlikely, but possible. But more importantly, I was running the dev server of my application on the host machine, rather than in the container. In general, I don’t like running my dev server in containers, since they overcomplicate stuff. I know some people do it, but for me, pulling, and rebuilding the container kills the entire DX of rapid development.

Since I was running the dev server on the host, and CC was editing the code in the container (which was a simple mount to the container), it is possible for CC to inject a code that would read an environment variable that was defined in my host machine, even though this variable is not available to the Docker container. This malicious code can, for example, dump all the environment variables and send them to some obscure server. Moreover, CC could still inject a variant of rf -rf ~ into the application code, which would still nuke my home directory, and turn my computer into a temporary brick.

And so, to mitigate these two issues, I had to go back to the drawing board.

Rolling my own, with blackjack and VMs

Doing your own “CC isolation” framework is a very common project many people undertake these days. And so, while browsing HackerNews, I came across a blog post by Emil Burzo: Running Claude Code dangerously (safely). And in this blog post, I saw something I haven’t seen for at least a decade: Vagrant.

I remember using Vagrant back in 2010s, when I was playing with VMs, as well as toying with writing my own Operating System Kernel. Eventually, I switched to using qemu, but nevertheless, seeing Vagrant again has brought some warm memories. And as Docker popularity peaked circa 2015-2016, Vagrant became less relevant, as container were faster, and lighter alternative.

Vagrant is basically “Virtual Machine as a code”. Think terraform, but for VMs. You write a Vagrantfile which describes your VM, and with commands like vagrant up you bring the VM alive, and vagrant ssh gives you ssh access to the machine. Unlike Docker containers, Vagrant VMs are fully isolated, and they run their own kernel, meaning that there is very little chance for malicious code to gain access to the host OS. Vagrant support various VM providers, but the most popular one is VirtualBox from oracle.

You define a VM by writing a Vagrantfile:

vm_name = File.basename(Dir.getwd)

NODE_VERSION = "24"

Vagrant.configure("2") do |config|
  config.vm.box = "bento/ubuntu-24.04"

  config.vm.synced_folder(".", "/vagrant", type: "virtualbox", SharedFoldersEnableSymlinksCreate: false)

  config.vm.hostname = vm_name
  config.vm.network("private_network", type: "dhcp")

  config.vm.provider("virtualbox") do |vb|
    vb.memory = "4096"
    vb.cpus = 2
    vb.gui = false
    vb.name = vm_name
    vb.customize(["modifyvm", :id, "--audio", "none"])
    vb.customize(["modifyvm", :id, "--usb", "off"])
  end

  config.vm.provision("shell", inline: <<-SHELL)
    export DEBIAN_FRONTEND=noninteractive

    curl -fsSL https://deb.nodesource.com/setup_#{NODE_VERSION}.x | bash -

    apt-get update
    apt-get install -y avahi-daemon \
        nodejs \
        git \
        unzip \
        curl \
        rustc

    if ! command -v docker &> /dev/null; then
      curl -fsSL https://get.docker.com | sh
    fi

    npm install -g @anthropic-ai/claude-code --no-audit

    usermod -aG docker vagrant
  SHELL

  config.vm.provision("shell", run: "always", privileged: false, inline: <<-SHELL)
    cd /vagrant

    bundle install
    npm i

    docker compose -f docker-compose.dev.yml up -d
  SHELL
end

Like with Docker, I mount the working directory on the host machine to /vagrant on the guest machine. I have a provision script that will run the first time I create the VM. The provision script is responsible for installing the needed dependencies (docker, NodeJS, and Ruby on Rails dependencies which I omitted from this code as they are specific to this particular project). I have another provision script that always runs when I bring the VM up, which installs the needed ruby and node dependencies for my project, and start the development docker (postgres and redis in my case).

NOTE

As of writing this blog post, it seems like the installation process for Claude Code has switched to “native install”. Instead of running npm i -g, you now need to run a shell script: curl -fsSL https://claude.ai/install.sh | bash and add ~/.local/bin to your $PATH like this echo 'export PATH="$HOME/.local/bin:$PATH"' >> /home/vagrant/.bashrc So make sure to replace the npm line with these two lines.

source

After the machine is up, I can ssh into it using vagrant ssh, and start my app development server. Then, I can open another ssh session, and start Claude Code. I can have as many ssh sessions, and as many CCs as I need. I can give CC full access to the VM, root permissions, whatever it needs, so it can install packages, run various tools, whatever. It’s all inside the VM, and if the VM gets infected, I just destroy it, and bring a new one.

Nothing app related runs on my host machine. The only thing that breaks the isolation level is the code which is mounted to the VM. For truly paranoid people, you can specify type: "rsync", in the config.vm.synced_folder call, thus taking a copy of the code into the VM. In that case, changes made in the VM are not synced back to the host machine. You either need to sync manually, or setup git flow from within the VM. Some people like to vibe-code 100%, so they don’t edit code manually, but instead let CC make the changes and open PRs. I still write code manually, and use CC for mundane tasks or code reviews, so I want to be able to edit the code myself, and control when and how I push to git, therefor I prefer to go with a synced folder approach even at the cost of penetrating the isolation level.

Some of you might have noticed that I also install avahi-daemon on the guest VM. What I wanted to achieve is that I will be able to open a browser on my host machine, and access the code that is running in the guest VM. This can be achieved by forwarding ports from the guest to the host, just like with docker, using the network configuration in Vagrantfile:

config.vm.network("forwarded_port", guest: 3000, host: 3000, auto_correct: true)

But I don’t like to expose too many ports, as well as being able to remember what port does each service use. If you run a PostgreSQL Docker in every VM, you eventually end up with port mappings like 54321, 54322, 54323, and so-on. I wanted to be able to access the machine by it’s name. It is possible to setup a static IP to the VM, and then access the services by the IP, for examples 192.168.10.100:5432 for PostgreSQL, but then again, I’d have to memorize the IPs of every VM I run. Another option is if you run a local DNS server (even PiHole or AdGuard Home), you can create a local DNS entry (or just edit /etc/hosts) and map a name to an IP. But, with mDNS you can do better. By running avahi on the guest machine, and properly setting the hostname (using config.vm.hostname = vm_name), you are able to “magically” access http://vm_name.local from your host machine. Just make sure to bind your services to 0.0.0.0 instead of 127.0.0.1 when you start them.

Why you probably shouldn’t copy my approach

Someone recently asked on HackerNews, why so many people rolling out their own AI sandboxing solution, and I think the best answer was given by varshith17, which basically boils down to:

Same reason everyone rolled their own auth in 2010, the problem is simple enough to DIY badly, complex enough that no standard fits everyone.

My solution is tailored to my needs. I run auxiliary services in docker containers, so I need a solution where CC can access these containers. I don’t trust CC to access anything else, and I don’t trust myself to be able to correctly control permissions for CC after a long day of programming. Eventually, I will slip and approve a request to Bash(rm -rf some-old-code ~/), hence rendering my machine unusable for some time.

My solution does not cover things like proxying requests to make sure CC can only access certain websites, or safeguarding environment variables, as I use only dev credentials that can barely cause any harm if they are leaked. If CC destroys my development database, I have a backup. Your needs might be different, and you might find existing tools, like the ones I mentioned earlier, useful. Or, you will decide to roll your own, and by having access to diverse information, and experience of others, you will craft a solution that suits your needs.

Safe vibe-coding!

I'm Netlify-Free!

Tue, 23 Sep 2025 19:00:00 GMT

Every time someone asked on HackerNews/Reddit “what’s your blog setup”, I would reply “Well, I use Astro hosted on Netlify, but I plan to move to a VPS”. It has been going for more than a year. More important things would be tackled first, and I kept promising the internet that I will move to my own VPS, eventually. Well, I’m glad to say that I finally did! Both this blog, and my personal website are now hosted on a VPS that I rent. I still use Astro, and they are still statically generated, but I’m no longer in the hands of Netlify.

What difference does it make?

The first question you might ask yourself: “What difference does it make?” And it’s true, there is no specific reason to move away from Netlify. It worked for me for a couple of years, and Netlify is a great tool. But, like any tool that offers convenience, it comes at the expense of price.

Sure, I never paid for Netlify since I was on their free tier. But it’s a matter of one viral post until you hit the free tier limits (and I was close). And then, it’s $9/mo, a bit more than 2 times the price of a VPS in Hetzner (not affiliated, just love their products). I already have a couple of Hetzner servers, so I figured, why not?

On top of that, “just for fun” as always a good excuse. Netlify trades configuration for convenience. Deploying Astro with Netlify is basically one-click. But the moment you need another step in you CI/CD pipeline, or want to customize something, you are basically out of luck.

So buckle up, and I will tell you how you too can free yourself from proprietary hosting providers.

How to do it?

You need 2 components:

A web server / reverse proxy - I use Caddy because Caddy is the GOAT, with minimal config file, and SSL out of the box
A CI/CD pipeline / runner - I use GitLab CI/CD, with a runner hosted on my hardware

I run Caddy on bare metal, because reasons, but you can containerize it. In my /etc/caddy/Caddyfile, which is the main caddy configuration file, I added this line import /etc/caddy/sites/*.caddy, so that every file I put in /etc/caddy/sites/ will be included as part of caddy configuration. This allows me to run both blogs, without messing with editing one config file, so everything is separate and nice. KISS.

Then comes the Caddyfile for each blog. Very simple:

yieldcode.blog {
    root * /var/www/yieldcode.blog/current

    header {
        /assets/* Cache-Control "public, max-age=31536000, immutable"
        /_astro/* Cache-Control "public, max-age=31536000, immutable"
        * Cache-Control "public, max-age=3600"
    }

    handle_errors {
        rewrite * /404.html
        file_server
    }

    file_server {
        precompressed br gzip
    }
}

www.yieldcode.blog {
    redir https://yieldcode.blog{uri}
}

Astro builds a new set of assets for each build, so it’s safe to cache /assets/* and /_astro/* forever. As for the rest, I cache everything for 1 hour. Error paths will be forwarded to 404.html, and everything is served precompressed with brotli or gzip.

Finally, we need a gitlab-ci.yml that describes our pipeline:

variables:
  NODE_VERSION: "24-alpine"
  SITE_NAME: "yieldcode.blog"
  DEPLOY_PATH: "/var/www/${SITE_NAME}"
  DEPLOY_DIR: "${DEPLOY_PATH}/${CI_COMMIT_SHORT_SHA}"
  COMMIT_REF: "${CI_COMMIT_SHORT_SHA}"

stages:
  - build
  - deploy
  - notify

.node:
  image: node:$NODE_VERSION
  cache:
    key: $CI_COMMIT_REF_SLUG
    paths:
      - .npm
  before_script:
    - npm ci --cache .npm --prefer-offline

.ssh:
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client rsync
    - eval $(ssh-agent -s)
    - install -m 600 -D /dev/null ~/.ssh/id_rsa
    - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
    - ssh-keyscan -H $SSH_REMOTE >> ~/.ssh/known_hosts
  after_script:
    - rm -rf ~/.ssh

build:
  stage: build
  extends: .node
  tags:
    - custom
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour
  only:
    - master

www:
  stage: deploy
  extends: .ssh
  dependencies:
    - build
  script:
    - ssh ${SSH_REMOTE} "mkdir -p ${DEPLOY_DIR}"
    - rsync -avz --delete dist/ ${SSH_REMOTE}:${DEPLOY_DIR}/
    - ssh ${SSH_REMOTE} "ln -sfn ${DEPLOY_DIR} ${DEPLOY_PATH}/current"

    - |
      ssh ${SSH_REMOTE} "
        cd ${DEPLOY_PATH}
        ls -td [0-9a-f]* | grep -v "current" | tail -n +4 | xargs rm -rf
      "
  only:
    - master
  environment:
    name: production
    url: https://yieldcode.blog

caddy:
  stage: deploy
  extends: .ssh
  script:
    - rsync -avz Caddyfile ${SSH_REMOTE}:/etc/caddy/sites/${SITE_NAME}.caddy
    - ssh ${SSH_REMOTE} "caddy reload --config /etc/caddy/Caddyfile"
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      changes:
        - Caddyfile

on_success:
  stage: notify
  image: alpine:latest
  when: on_success
  before_script:
    - apk add curl
  script:
    - |
      curl -s -F "token=${PUSHOVER_API_TOKEN}" \
              -F "user=${PUSHOVER_USER_TOKEN}" \
              -F "message=Pipeline succeeded for $CI_PROJECT_NAME on branch $CI_COMMIT_BRANCH@$CI_COMMIT_SHORT_SHA"\
              https://api.pushover.net/1/messages.json
on_failure:
  stage: notify
  image: alpine:latest
  when: on_failure
  before_script:
    - apk add curl
  script:
    - |
      curl -s -F "token=${PUSHOVER_API_TOKEN}" \
              -F "user=${PUSHOVER_USER_TOKEN}" \
              -F "message=Pipeline failed for $CI_PROJECT_NAME on branch $CI_COMMIT_BRANCH@$CI_COMMIT_SHORT_SHA"\
              https://api.pushover.net/1/messages.json

First, I define all the needed variables. Then, I have 3 stages:

build - builds the Astro code
deploy - copy the Astro artifacts to the server, as well as the Caddyfile (if it was changed)
notify - send a pushover notification to my phone that the deploy is done

Then, I define 2 common blocks: .node for NodeJS stages, and .ssh for stages that need SSH. build builds the Astro code and stores the artifact. www copies the artifacts from build stage over ssh to the remote server. Here I also do an interesting trick (which is very old, and the first time I learned about it is years ago when I played with Capistrano). To avoid corrupted state, each deploy copies the files to /var/www/yieldcode.blog/${CI_COMMIT_SHORT_SHA}, where $CI_COMMIT_SHORT_SHA is the first 8 characters of the git commit. Once the files are copied, I use a soft-link named current that points to this directory. Caddy points to the current directory, which always points to the lats commit. Finally, I do some magic to keep the last 3 directories (in case I want to rollback or something like that). This is how the directory tree looks:

 yieldcode.blog
 ├── 0c69eaa0
 ├── 4cdc8c1f
 ├── 6ddf000d
 └── current -> /var/www/yieldcode.blog/0c69eaa0

Hence, switching between versions is a simple matter of changing where current points to.

The caddy stage simply copies the Caddyfile to /etc/caddy/sites/${SITE_NAME}.caddy, and reload the caddy server to make sure it picks up the new configuration.

Lastly, I have two stages, one that runs on success and one that runs on failure, that send me a pushover notification when the deploy is done. Sure, I could have used one stage, that executes a script, which looks at some GitLab defined variable that tells me the state of the pipeline, and based on that constructs a message, but meh. It works like this as well.

That’s it. That’s Netlify (more or less).

Now sure, Netlify has more things like CDN, automatic forms, and serverless functions. But I don’t use any of that. Maybe if you do, then Netlify is for you. I never had problems with them, so I guess they are cool.

But in recent months I feel like the fun is being sucked out of being a software engineer, so doing small things like this, brings me a bit of joy in an otherwise, vast greyness of AI slopification.

Vibe Management

Tue, 01 Apr 2025 16:00:00 GMT

Vibe coding is taking the internet by storm. Popularized by Andrej Karpathy, it changes the entire industry. But why only engineers should have all the fun?

Andrej Karpathy Tweet

I’m in favor of making as many professions in tech redundant, as possible. I mean, if I can enjoy singularity while vibing my way through my code, why upper and middle management shouldn’t? And the reality is they should! And they can!

As of today, both ChatGPT and Claude are more than capable of replacing your average middle manager such as team lead or product manager. Here is an example of an interaction between a product manager and a potential customer that begs for a feature:

You are a product manager, and a customer requests a certain feature to be implemented. The truth is, you are over capacity, and don't really want to implement it, but it is your duty to make your customers happy, it's in the DNA of your company "Customers First". Write a short statement that will make the customer feel good, without making any commitment about deadlines, or even if the feature will be delivered at all.

Thank you for sharing your feedback! We truly appreciate hearing from our customers, as your insights help shape the future of our product. This is an interesting idea, and we’ll be sure to take it into consideration as we evaluate potential improvements. While I can’t provide a specific timeline, please know that our team is always looking for ways to enhance the experience. Keep the great ideas coming—we love hearing from you!

Delightful!

But we won’t stop there. Team lead is our next goal! Coming up with a good way to share negative feedback, is a very challenging thing. It can even harm your mental health, or turn your subordinates against you. But don’t worry, we can remove YOU from the equation, and instead let ChatGPT do the heavy lifting:

You are a team lead of in small engineering team. One of your peers underperforms. It is your duty to deliver negative feedback to them, without hurting their feelings or demoralizing them even more, as well as protecting yourself from getting labeled as a bad manager. Write a short statement that you can say during 1-on-1 with your subordinate, to make sure they will understand that they need to start performing, or there is a big line of candidates that can replace them.

Hey [Name], I wanted to take some time to check in with you. I’ve noticed that some of your recent work hasn’t quite met the expectations we set, and I want to understand if there’s anything blocking you or if you need support in any way. My goal is to see you succeed here, and I know you’re capable of more. That said, we do have high standards, and I need to see improvement moving forward. Let’s work together to make sure you’re on track because this is a competitive environment, and I want you to be in the best position possible. What can we do to help you step up?

Now that’s a banger! Who wouldn’t want to dedicate 24 hours a day to their corporate overlord after hearing such constructive and thoughtful feedback!

Let’s take it a step further. When there is no middle management, we can argue for the necessity of upper management. Undoubtedly, but CEOs perform a very important role in the company. But the truth is, with the help of ChatGPT, the CEO can forget that problems exist, and embrace exponentials. They no longer need PR teams, and they can create infinite number of iterations before they land on something they like. And they don’t even have to touch the keyboard, they can talk to Whisper. They can run a company, but it’s not really running a company — they just type stuff, copy stuff, and see stuff, and it mostly works:

You are the CEO of security company called CrawlStroke, and earlier today, your company have shipped a bricked update that broke the entire internet. Blue screen of death is present at airports, hospitals, emergency service, banks, etc. Everyone is angry, and shareholder are not happy. Write an apologetic press release, with emphasis on taking full accountability, fixing it as fast as possible, and offering a 10$ discount for Uber eats to your customers.

SUNNYVALE, Calif. — [DATE] — This morning, CrawlStroke issued a routine security update that, due to an unforeseen technical error, resulted in widespread system failures across multiple industries. We recognize the severity of this incident, and we take full accountability for the disruption this has caused to businesses, critical infrastructure, and individuals worldwide.

First and foremost, we deeply regret the impact this has had on hospitals, emergency services, airports, financial institutions, and countless other organizations that depend on stable, secure systems. Keeping the world safe from cyber threats is our mission, and today, we failed to uphold that responsibility. For that, we are truly sorry.

Our top engineers have been working around the clock to resolve the issue. We have already deployed a fix to affected systems, and we are providing direct assistance to organizations still experiencing issues. If your business is still impacted, we urge you to contact our emergency response team immediately at [support hotline/email].

While we cannot undo the disruption this has caused, we want to make things right. As a small token of our apology, we are offering a $10 Uber Eats credit to our affected customers. We know this doesn’t fix everything, but we hope it brings some relief during this challenging time. Details on how to redeem this offer will be provided via email.

We are conducting a full internal review to understand how this happened and to ensure it never happens again. Transparency and accountability are core to our values, and we will share our findings once our investigation is complete.

We appreciate the patience and resilience of our customers, partners, and the global community as we work swiftly to restore normalcy. We will provide ongoing updates at [company website/social media].

Sincerely, [Your Name] Chief Executive Officer CrawlStroke

OMG 🤩! Who wouldn’t want to make business with such an accountable CEO?

So here you have it! It’s a new kind of management and leadership that I call “Vibe Management” and “Vibe Leadership” where you just give in to the vibes. And I predict that it will replace 90% of upper and middle management in the next year, and within five years, all upper and middle management will be able to join us, software engineers, in the singularity, while all of us will be just vibing our way through our careers!

Bright future ahead of us!

Working with systemd timers

Mon, 10 Mar 2025 19:30:00 GMT

The other day I thought to myself that it would be a good idea to have some backups of my data. So I was wondering, how would I execute a periodic backup task?

Most of you are probably familiar with cron. cron is a Unix utility to run scheduled tasks. There is a file called crontab which you edit by issuing crontab -e command, and inside schedule tasks such as:

0 1 * * * /home/skwee357/backup.sh

The first 5 entries are what is known as a cron expression, and they dictate when the command will be executed. In the above example, the expression says “every day at 1 am”. There is a cool website called crontab.guru which allows you to compose various cron expressions. After the cron expression, we have the actual script to execute. This can be a bash script, or a binary like curl.

Cron is easy, very simple and robust mechanism to execute periodic tasks on a *nix server, and is available by default on all popular Linux distributions. However, cron suffers from some issues:

If the system is down when the cron needs to run, the cron will be missed
There is no built-in status monitoring
There are no built-in logs
If you want to execute pre/post commands (for example by pinging an external service for success/failure) you have to do it inside the script itself

Luckily, there is another utility called systemd that allows us to overcome all of the above issues. systemd is a relatively new utility that provides an array of components for Linux systems. It is essentially a system and service manager, which also replaces some daemons and utilities like device management, login management, network connection management, etc. systemd is adopted by most popular Linux distributions, but it’s important to note that it’s not adopted by other Unix-like system such as FreeBSD or OpenBSD. There are various unit-types inside systemd but the two we are going to focus on today are .service which define the actual service to execute, and .timer which acts as a cron-like job-scheduler. Let’s start with the timer.

systemd timers, and services, are located at /etc/systemd/{system/user} depending on whether it’s a system-wide service/timer or user-only system/timer. We will focus on system-wide services and timers, so all edits (unless instructed differently) happen inside /etc/systemd/system. First, we will create a timer for our service: vim /etc/systemd/system/backup.timer:

[Unit]
Description=Backup service timer

[Timer]
OnCalendar=*-*-* 1:00:00
Persistent=true

[Install]
WantedBy=timers.target

First, we define the unit, and it’s description. Then, we define the timer itself. The format for OnCalendar is actually DayOfWeek Year-Month-Day Hour:Minute:Second, but every component can be replaced by asterisk to signify any value. DayOfWeek can be omitted, and any two values separated by .. indicate a contiguous range, so for example Mon..Fri 22:30 means run at 22:30 on weekdays. We can use comma to include multiple values, for example: Sat,Sun 20:00 means run on weekends at 20:00. OnCalendar can appear multiple times to include multiple date ranges. Lastly, it is possible to also specify timezone in the end, for example: *-*-* 02:00:00 Europe/Amsterdam to run every day at 2 AM in Amsterdam time.

Then we have Persistent which controls whether the timer is persistent or transient. Transient timers are valid only for the current session, meaning if the system is powered off when the timer is due, it won’t run after system boot, just like cron. However, if the timer is persistent, this means that when the system boots, the timer will run immediately, which is great for backups that must run even if the system crashed during/before the expected backup.

Lastly, we have Install directives. Unlike crontab entries, systemd units (be it services which we will talk about in a bit, or timers) are NOT active by default, and needs to be installed. You have probably seen commands like systemctl enable redis.service at least once in your development career, this is the actual installation of the unit, which from now on will be active in systemd. When installing systemd units, we can optionally specify dependencies, and this is done by using WantedBy or Wants directives. In case of timers, it is common to specify WantedBy=timers.target in the Install section. timers.target is a special target unit that sets up all the timers that shall be active after boot. So if we want our timer to be active after boot, we have to make sure that timers.target wants it. Save the file, and exit vim. We are done with the timer, time to move onto the service.

Create a new file /etc/systemd/system/backup.service:

[Unit]
Description=Backup service

[Service]
ExecStart=/home/skwee357/backup.sh
Type=oneshot

As with the timer, we first define our Unit and it’s description. Then we define the service itself. In ExecStart we give the command that systemd should execute when the service is ran. Type is a bit tricky, and mainly controls how systemd acts after forking the processing and executing our command. There are various Type values and you can find them in man systemd, but the reason I have used oneshot is that it essentially allows us to specify multiple ExecStart directives, and executing all the commands serially. If you do not intend to run multiple commands, you can use Type=simple or omit the Type at all, as Type=simple is the default when no Type present, but ExecStart is present.

So far so good. Let’s spice it up. Let’s say we want to use a service such as Pushover to send a push notification when backup completes with either success or failure. For that, we can use ExecStopPost in the [Service] section. ExecStopPost will be called once ExecStart will finish running. In order to determine the result of the execution, we will have 3 variables available to use:

$SERVICE_RESULT - will be success for successful execution, and a bunch of other codes for non successful execution
$EXIT_CODE - the exit code, 0 for success, non zero for error
$EXIT_STATUS - somewhat different from $EXIT_CODE in a sense that if the process was killed by a signal (SIGTERM, or SIGKILL for example), this will contain the exit code of the termination (unlike $EXIT_CODE that contains the exit code of our process)

So we can write a script in bash something along the lines of ExecStopPost=/home/skwee357/pushover.sh $EXIT_CODE, and inside we will parse the exit code and send appropriate push notification. We can also use ExecStartPre= in order to execute some code prior to calling ExecStart. There are way, way more settings in a systemd service, so RTFM if you need more.

After we finished writing our timer and service, it’s time to install the service. This is done via systemctl enable backup.timer. A symlink will be created, and systemd will manage our timer which will trigger our service when the time comes. In order to disable the timer, we can execute systemctl disable backup.timer. And by executing systemctl status backup.timer we can view the status of our timer. It should be active (waiting) and show us how much time left until trigger. We can also execute systemctl status backup.service and view the status, as well as logs, for the service. In general, we can use journalctl -u backup.service to view the logs of our systemd unit (hence the -u), and combine it with -b to show logs only from current boot session, or -b -N where N is a number, for example 2, to show logs from two boots ago, etc.

The commoditization of AI

Thu, 30 Jan 2025 18:30:00 GMT

Earlier this week, a Chinese company released a new AI model called DeepSeek, that shattered the big tech giants that tried to own AI. Seeing this, brings some memories from the past.

In the 1990s, there was a tech revolution called “The Internet”. According to Our world in data, in 1990 there were about 2.6 million of global internet users. By the end of the decade, this number increased by more than 15,000% up to 416.2 million of global internet users. From that moment on, the increase was almost exponential. One of the companies that caught this trend of “the internet” was Cisco, and around 1997 their stock started to climbed and in March 2000 it’s price reached $82 per stock, putting it at a total market cap of $536B. This was the highest stock price, and market cap, that Cisco has ever reached. It stayed there for a year, until crashing back to $15 per stock on March 2001. Today, it’s stock trades at around $60, and its market cap is about $230B. What caused this fast climb and sharp decline, you might ask?

The 2000s were an interesting time, when the entire tech world has lost its mind, which eventually led to the dot-com bubble bursting. Seeing this new shiny thing called “the internet” becoming accessible to more and more people, led investors to seek the next unicorn to put their money in. Cisco was an established networking equipment company, and it continued to expand its offerings by acquiring other companies that were doing networking, voice over IP (VoIP), digital video, LAN switching, etc. And despite how much tech bros and investors will sell you the “we want a better world for everyone”, what they truly want (at least as number #1 priority) — are profits for themselves. Hence, they were looking for someone to “own” the internet. What they failed to account for is that the internet will become a commodity, which it did. But what it has to do with DeepSeek and AI?

You see, the entire AI tech bubble is built on the assumption that doing AI is hard, and it requires billions of dollars of investments. Just few days ago companies like Microsoft, OpenAI, and Oracle announced project Stargate which will require a $500B investment in order to build nuclear power plants, data centers, and acquire hardware to train AI models. AI seemed to be out of reach for the common folks, forever to be in the hand of the big corporations with deep pockets, who will profit massively from it. But then came DeepSeek, which released a model that is at least as capable as the top model from OpenAI, if not more capable. They not only set a new bar for AI models, but they also released the model as open source, and claim to have trained it on older hardware for less than $6M compared to the billions that US tech leaders are pouring into AI.

This caused Nvidia to lose almost $600B of market value, a 17% drop, one of the biggest market loss in history. There is no simple, or logical explanation as to why this happened, because Nvidia still seems to be the major provider of GPUs for AI purposes, and it’s not like AI is going anywhere. Nvidia had a sharp climb since mid 2023, just 6 months after releasing ChatGPT, so this might be the market correcting itself. Or as I always said, nobody knows what the stock market is, and why things happen as they do. And while DeepSeek claim to have used older hardware, which could be a plausible claim, there is no real way to validate their claim as China not supposed to have access to modern GPU chips due to export bans, and they might have inclination to hide such information.

Back in the early 1990s, the internet was not accessible to most people. Some businesses invested into getting equipment in order to be able to access the internet, and a select few from the public used to visit Internet Cafés — public places with computers connected to the internet. Personal computers were often sold without the expensive addition of Ethernet cards that allowed your computer to be connected to the modem. Moreover, adjusted to inflation, the cost of getting a modem varied anywhere between $600 to $1000, and that’s in addition to having an internet capable computer. In short, this technology was out of reach for most people. Fast-forward to today, Ethernet is built into every single computer, and you can get a router for as little as $50.

Just like with the internet, AI is going to transform. From being tightly controlled by a group of tech people to be run on expensive hardware, to being freely available and run on consumer grade hardware. There are people today who can run the most capable DeepSeek model at home. Of course, it is not affordable to many yet. Most use a cluster of 6 to 8 Mac Minis which can easily reach the cost of $10K; and those who run it on PCs, require server grade motherboards that support hundreds of GBs of RAM, as well as Nvidia GPUs which are price hiked at the moment.

But make no mistake, we will eventually arrive at a future where AI will be run locally on your devices. With Apple investing into Neural Engines in their M-series CPUs, to dedicated AI boxes from Nvidia and AMD, we will get there. And just like with the internet, the AI bubble will burst, and AI will become a commodity. You probably won’t be able to train an AI model at home, as it will still require money and hardware, but just like with the internet, you are not required to lay down your own undersea cables in order to speak with your friends on the other side of the world.

But what about AGI?

AGI, or Artificial General Intelligence, is something that many of these AI companies claim to move towards in their development. Thus, the first one to reach AGI, will be the first one to set the rules. It’s like the atomic bomb race — you either get it first and set the rules, or someone else does. AGI, if achieved, will be a big change in the way the world works. And so many of these tech companies seem to incorporate into their value the promise of reaching AGI. A promise that we’ve heard to come alive next year… every year. And yet there are no signs for AGI, and I don’t think we will see it, at least not as a by product of LLMs.

When ChatGPT came out I, like many others, have seen it as this magical thing. I didn’t really know how to treat it, or how it works. My view of LLMs swung all the way from “they are bad and stupid” to “they are good and useful”, and with every new model, and every new use-case, my opinion is still shifting. But it really clicked for me when I decided to learn, at least high level, what GPTs are and how do they work. I highly recommend this video by 3Blue1Brown about what are LLMs and how do they work.

And after viewing this video, I now see LLMs for what they are — a mathematical model to predict the next most probable word/token. This not only broaden my knowledge, but it also allowed me to use and prompt LLMs in a more efficient way, as I now understand what are their strengths and limitations. But what it has to do with AGI?

For as long as I remember myself, I was bothered with the hardest question of humanity: what is free choice and is it truly free, rather than preselected/preprogrammed by the environment? In short, can we predict certain human actions/interactions by observing/learning about their environment. I don’t have the answer, but it is an interesting thought experiment nevertheless, and it aligns perfectly with LLMs and AGI. You see, LLMs are predictable. I tried both ChatGPT and DeepSeek for the past couple of days, and both outputted similar answers to some of the question(s). Sure, some models perform better in specific tasks like coding, but the general gist is that they can’t think in a sense a human can. Nor they step outside their mathematical model.

Humans are predictable as well, but the difference between human behavior and a mathematical model is that we can break the rules. The other day I was sparring with my boxing coach. He threw a jab followed by a right hook, which I blocked. Then, he repeated the combination, which I blocked. He did it again, which I blocked again. And then he threw a jab, followed by right hook (which I blocked), followed by left hook which shoot me to the ropes. In a sense, he conditioned me by his behavior (input data) to a certain reaction (output data), which can be seen as a training of an LLM. For the same input, he got the same output, albeit a bit different as I was anticipating the combination, thus performing better/different defense sequences. However, him being an experienced fighter, he was able to see and read my reaction in order to take advantage of my situation. Should I have reacted differently, or been a more experienced fighter, he would probably fail to throw me to the ropes.

But boxing is not a mathematical function, and it requires many skills in coordination, reading the environment, and fast reaction. And there is always an element of surprise, or irrationality. This can be seen as randomness by the outside observer, but it’s not. It’s something in a though process of the brain where people are able to break outside the box (which LLMs are constrained to). Moreover, it’s something more than thinking. I don’t like to talk in meta terms (meta the word, not meta the company), but intelligence is not only the processes that happen inside our brain, but it’s also the effects of environment. Things like reading and understanding a situation, gut feeling, etc. Think about a time when you felt embarrassed to do or say something, or when you reacted instinctively in order to avoid danger. I’ve seen it many times throughout history: creative approaches to job searching, advantages on the battlefield, how certain political figures act, etc. And I don’t think that the current state of LLMs will lead us to AGI.

Don’t get me wrong, GPTs are an amazing piece of technology and research, but they are still a mathematical function, once you understand how they work. There is no magic or thinking behind that. And they are certainly not able to break outside the constraints of their mathematical function.

Ghost Engineers

Tue, 10 Dec 2024 16:30:00 GMT

There is a new viral tweet circulating, which states that 9.5% of all engineers—are ghost engineers. Their performance is <0.1x of a median engineer, they do virtually no work, and they might work multiple jobs, thus collecting 2x-5x of an average engineering salary. Usually, I’d ignore such statements, in the end, there is a lot of BS circulating in the World Wide Web. But this research seems to come from Stanford, and it has been picked up by news outlets such as BusinessInsider, 404 Media, and the Washington Post (which is behind a paywall, so I’m not linking it), to name a few. Surely, something is going on.

Ad Hominem

Before I dive into the research, I need to address something else. I don’t want this blog post to become an Ad Hominem argument, an argument directed towards the person rather than their position or stance. But nevertheless, I do want to address some concerns about the author.

Firstly, while there is a short paper explaining the methodology, it has not been peer-reviewed. The only credible mark of this research is the Stanford name on it. Academia has always been viewed as a source of credibility, but let us not forget that despite the fact that many good things came out of the academia, it has also served the interests of both individuals and corporations, so before this research has been peer-reviewed, we ought to view it as what it is—an infographic from a Stanford MBA.

Secondly, the author. I tried to find as much information about him as possible. But aside from the fact that he had/is having an e-commerce business, I couldn’t find any relation between the author and software engineering in general. And while it is not forbidden to give advice or express your opinion about things you do not understand, I also think it harms credibility when a person who has no experience in a particular field, tries to measure productivity in said field. It’s as if I would say that doctors should be measured by the amount of pills they prescribe per day, and will provide a nice infographic to support my claim. I can do it, but any reputable doctor will know to dismiss my opinion as it’s clear that my knowledge in medicine is limited to Googling my own symptoms.

With ad hominem out of the way, let us dive into the research.

The research

Findings

First, let’s talk about the findings. According to the author, they looked at private git commits, and fed the data to a panel of 10 experts who evaluated the quality of the code. How scalable was the code that was committed, how slow it was, and, yes, how complicated it was to write it. And the results were shocking!

The results

On average, 9%, or roughly 1 out of 10, of engineers are Ghost Engineers. Ghost, in this context, means that they produce less than 0.1x of a median engineer, and do virtually no work at all. This number goes down to 6% for office workers, and climbs up to 14% for fully remote workers.

In the next slide, the author goes to demonstrate that from a sample of 5,832 ~~slackers~~ ghost engineers, ~58% make less than 3 commits a month, while the other 42% make trivial changes like deleting a line or changing a character.

And the next 3 slides, and the most important in my opinion (I will share further in the article why), talk about the wasted $$ these ghosts create. In the top tech companies such as IBM, Microsoft, Amazon, Google, and others — ghost engineers are costing a whooping $500B. Thus, laying them off, will free these precious $$ money for… hyper growth? I don’t know.. Anyway.

Methodology

Looking at the published paper, we can get some more information about the methodology used. The research focused on Java as Java “is versatile and widely used across various domains, including mobile, web, enterprise, and data applications.” They used 10 Java experts consisting of:

3 Managers with 11 to 13 years of experience, team size of 1–50 people
2 Executive with 15 and 24 years of experience, team size of 1-10 and 201–500 people, respectively
3 Senior engineers with 16, 20 and 25 YOE, team size of 11-50, 51-200, and 1-10, respectively
1 Director with 23 years of experience, team size of 11–50 people
1 Vice President, with 24 YOE, team size of 51–200 people

Please tell me you can see that flaws in this, please!

The researchers then sampled, wait for it, 70 commits (some teams push that many commits in an hour, roughly a representative number), and asked the 10 experts to evaluate the commits by asking various questions. I won’t list the entire questionnaire, but here are some examples of the 7 questions asked:

How many hours would it take you to just write the code in this commit assuming you could fully focus on this task?
What is the experience level of the author?
How maintainable is this commit?

Then, they created an AI model (of course they did), that matches the output of the experts. That’s it. That’s the research.

My analysis

I am a software engineer with 14+ years of experience. Throughout my career I’ve been an interviewer, a tech lead, a mentor, and progressed from junior script kiddie to leading my own complex designs and implementations. Moreover, I’m a tech entrepreneur building, running and scaling my own online business.

When I talk about productivity, I, and people with my levels of knowledge and expertise, know that lines of code do not equal productivity. Sometimes, you have weeks of discussion that lead to few line changes. Consider an example where your contribution is commenting out a feature flag that held a global roll-out of a particular feature to your entire customer base.

Sure, from the commit itself, you have deleted one line (or flipped a boolean variable). But from the business perspective, it probably involved days, if not weeks of testing; discussions with PMs; alignment with other teams—for example DevOps engineers to support the increased traffic/CPU/memory; and for some engineers it might even involve speaking directly with customers.

And it’s ironic that they chose Java for their research. Considering the fact that Java null reference has been crowned as “The Billion-dollar mistake”, by the creator of the null reference himself, hence commits such as if x != null, could theoretically save millions to the companies running Java for their services. Yet, this “ghost engineer productivity” AI model would qualify such commits as “low value/low quality/super simple”.

The other flaw of the research is the panel selection. Among the 10 selected experts, only 3 are actual engineers. I do not disqualify managers/leadership as being unable to write/understand code, but do you really think that a VP with 24 years of experience, or an executive that manages 200+ people, have been writing code in their day-to-day job, for the past decade? Java is 29 years old, and assuming you are not born as a VP, this person probably wrote some java 20 years ago, before he was promoted to management and then leadership. I think it’s fine to include some managers/leadership in your experts panel, but don’t you think that for a panel that measures software engineering productivity you would want the ratio of hands-on employees to be, I don’t know, higher than 30%? Unless…

My speculation

…you try to appeal to management/leadership. Look, I don’t know the guy, and I don’t know what’s his motivation. Maybe he is trying to make the world a better place by pointing at inefficiencies in software engineering. And there are inefficiencies, no doubt in that, like in any other profession. Is 1 out of 10 engineers are not doing anything? Could be. I don’t know.

But this entire research seems like a sales pitch to appeal to the leadership to get rid of software engineers. The papers seem to focus on creating AI model that consists of 70% “business peoples’” opinion, while the infographic seems to focus on the monetary loss due to ghost engineers.

I’m sorry, I’m going to be blunt here: most people in leadership position know shit about what software engineers are doing. Sure, there are VPs or CTOs who were technical in their career, but most of them are either business people who are focused on streamlining the business side of operation, or they were engineers very long time ago, and they would think that React is a name of a spell from Harry Potter. They all are obsessed with the word productivity, but they are unable to measure it correctly. Therefor they employ companies like McKinsey, a global management consulting company, who published papers about developer productivity. Or they are looking for the next Stanford MBA guy to sell them on some AI, because everything is AI nowadays, on how to measure developers productivity. The deliberate use of business jargon like 0.1x engineer throughout the infographic is made to appeal to business people who like numbers. What does it even mean to be a 0.1x engineer? What is the baseline? Who is 1x engineer?

I worked with engineers who saved their companies millions of dollars per month. They were analyzing spending pattern across cloud providers, talking with different stakeholders, and eventually delegating their work to their peers who would go on and change a config value to create less resources on AWS. Such changes would result in decreasing the monthly bill to the cloud provider.

I, myself, spent weeks working with other people to optimize the interviewing process and on-boarding more interviewers, an action that would increase the amount of quality candidates while decreasing the monetary spending on interviewers who interviewed in pairs, thus costing the company two times more per interview. I spent weeks working on a design review that was so detail oriented that the development of the feature was almost on autopilot, allowing two teams to run in parallel. Heck, I mentored and on-boarded other engineers.

All these things resulted in 0 lines of code written or committed. Luckily, I worked with managers who appreciated people like us, who understood that the output of a software engineer is not the lines of code they produce. And in fact, this research seems to support my claim. You see, managers who are obsessed with productivity, seem to inflict this obsession on their subordinates. Hence, you see that an office environment has a lower percentage, in fact 2 times lower, of ghost engineers than fully remote positions. In the office, it’s harder for you to sit and think about a problem, especially if your manager expects you to write code. This is briefly mentioned in the infographics in this tweet:

Comparison between remote and office engineers.
On average, engineers working from the office perform better, but “5x” engineers are more common remotely.

And unintentionally, this research actually confirms what it tries to confirm, but from a different angle. Productivity raises when you are not spending time in an abusive office environment, or participating in meaningless corporate rituals. An environment that gives you peace of mind and ability to focus and think, which is your home, tend to have higher rate of “5x” engineers, unlike the uninviting and chaotic setting of the open office, with potentially abusive manager who breaths down your neck.

But until managers/leadership will understand this, it seems like the worst tech market since 2008, is yet full of surprises and a grim future where our productivity will be measured by AI models created by business people who never interacted with software engineering in their life.

How to be a great manager

Thu, 28 Nov 2024 18:00:00 GMT

An insight from a non-manager perspective.

Before joining Autodesk back in 2017, I didn’t really have a proper manager. I either worked in family-like companies, where hierarchy was somewhat flat, or early stage companies where hierarchy was just developing. It’s only at Autodesk where I’ve been exposed to career ladders, and proper company hierarchy.

And while I do have managerial and leadership experience, I believe that a good manager can be judged only by the team he manages, rather than by his manager. I was managed by a great manager at Autodesk. He was so good that I even submitted a talk proposal titled “How to be a great manager - an insight from a non manager perspective”, to the internal knowledge sharing conference at Autodesk. My talk was rejected (don’t know why), but my observations stayed with me for the past 7 years. And now, after 7 years, a few more managers in my career, and a fairly popular blog, I decided to share my insights here. Let’s begin.

Being a great manager is being down there with your team. I joined Autodesk at a stage where the office I worked at was transitioning from a startup like environment, to a proper chain of responsibilities, and new teams were forming. My manager was my co-worker. We’ve both been software engineers working on the same product. I don’t think it’s a necessity for your teammates to see how you are being promoted to a manager in order to be a great manager, but I do think it’s important that as a manager—either one who was promoted to from inside the team, or hired outside—you need to stay close to your team.

Many managers justify their work by doing “management” stuff. They are in meetings all the time, they barely interact with their teams outside the necessary corporate rituals like team status updates, periodic planning meetings, and 1 on 1s with their team members. I believe this mindset alienates the manager from the team that he/she manages. Sure, management feels like something more important than writing code, but managers forget who they “serve”. They often want to play with the “big boys”, the upper management above them, rather than interacting with the people who they need to grow—their team members.

For example, my manager despite being a manager who was no longer coding, was still involved and interested in what is happening in the team. He would ask questions about the architecture or would discuss new approaches he learned for software development. He wasn’t taking active part in development, aside from doing light pair programming here and there, but he knew how the product works in general terms. I know that for many managers it’s hard to stay somewhat technical, but ain’t this the reason many engineers become dissatisfied with their managers because “he/she doesn’t understand what I’m even doing or how the product works from a technical perspective”?

This necessity to be part of your team rather than hang-out with the “big boys”, translates to yet another important quality of a manager—being the firewall between their team and the rest of the organization. There is a saying that people leave companies more often due to bad managers than any other aspect. We can tolerate somewhat boring projects, in-capable teammates, being in disagreement with the tech choices made in the team. But we seem to be unable to tolerate bad management. And we should not.

The entire function of the manager is to focus on managing their team, and in this sense to also create a wall between the chaos that is happening in rest of the organization in order to protect the sanity, and productivity of their team. Many managers are like messengers of bad news. If the company underperforms, these managers would fail to absorb the negative feedback, and would pass it down to their team. They lack stoicism, and are unable to control their emotions, thus creating a sense of tension, or sometimes even fear, inside their teams. Great managers, on the other hand, are able to shield the entire corporate drama from their teams. Not even once I felt stressed or being involved in corporate drama that is beyond my scope. Sure, when I was underperforming, my manager let me know that, but anything that was not related to me, or my team—stopped at my manager. I would often see other teams being in stress, thus having reduced productivity, because their managers failed to be the firewall that they should have been.

I mentioned feedback, and a great manager is a manager that knows how to give feedback, and especially negative feedback. Being a manager is being responsible for peoples’ growth. The success of their team, is their success, and yet, many managers seem to forget this. They lack basic empathy, they see themselves as “superior” to their teams, and they give negative feedback in such a way that it causes you to quit.

Giving, and receiving negative feedback is a complicated social situation. Nobody likes to be judged, especially when people feel like they are doing good, and suddenly they discover that they were underperforming. If you did not expect to get a below average score on your last performance review, rest assured that your manager failed to communicate a proper feedback to you. On the other side, if after receiving negative feedback, you feel like your entire world has collapsed, then your manager failed in their task.

I have never felt down or devastated by negative feedback, and there were times when I was underperforming. But every time when I was underperforming, my manager knew how to communicate it to me, and how to find a solution. And their method of communication was unique for each team member, which brings me to yet another trait of a great manager—they understand people.

You would think it would be a trivial requirement, right? I mean, if you hire a NodeJS developer, you would check their NodeJS knowledge and understanding, so when you hire a manager, you would check how they understand people, right? Right? But many managers lack people skills. They are unable to provide negative feedback in a way that it can be received and worked on. And even if they can, they try to apply the same method to each and every team member. However, every person is different. Some are more resistant to negative feedback, while others expect to be told the bold truth in their face. You can’t, as a manager, treat these people the same way.

You need to build a connection with each and every team member. I loved my 1 on 1s with my manager. I waited for them. We enjoyed talking, and I was feeling that my manager truly cares about me not only as an employee, but more importantly as a human being. Often in our 1:1s I’d share some personal things that I go through in my life, which could be affecting my work, and my manager would care to listen, and even give advice. I never felt that our 1:1s are just a part of the “corporate rule book”, or it’s there because it needs to be there. He would listen, give advice, and become an enabler to my success.

Yes, being an enabler for my success is yet another quality of a great manager. My success is his success, and he knew it perfectly. If I wanted to do something, be it try myself in a new role, mentor someone, try out a new tech—he knew how to make it happen. And even if it leads to failure, we would still find a way for me to try it out, while also figuring out how to recover if this fails. There are limits of course, in the end, you can’t expect your manager to enable you to rewrite the entire product in Rust because you love Rust, but the company uses JavaScript. But for reasonable desires like being part of the technical interviewers pool, mentoring junior developers, or leading my own project—he was able to make it happen.

In conclusion, management is a leadership position. Whether you want to be a great people manager, or you want to focus on being a great tech manager such as Tech Lead/Staff Engineer, you need to have these 5 qualities. Let me repeat them:

Be there for your team, remember that their success is your success
Be Stoic enough in order to shield your team from irrelevant news/events (especially negative ones) while allowing them to focus on what they do best
Know how to provide feedback, especially negative feedback, in a way that it can be accepted and worked on
Learn, observe, and understand people. Find the right approach to every individual
Be an enabler for your team, identify how they want to grow, and find ways to make it happen

This post does not necessarily talk about a specific individual, but rather outlines the most important qualities needed in order to be a great manager based on my observations when being managed by different managers throughout my career.

Why DX doesn't matter

Tue, 05 Nov 2024 17:30:00 GMT

Developer experience, or shortly DX, seems to be the number one thing that people pay attention to in their chosen tools/frameworks.

Jump on any online discussion, and you will hear “DX that framework”, “DX this tool”, etc. DX, together with “fast prototyping”, “rapid development”, “developer ergonomics”, seems to be the only thing everyone cares about nowadays. I’m here to argue that neither DX nor any fast/rapid development/ergonomics enhancement tools/frameworks—truly matter.

The entire premise of DX enhancement frameworks/tools is that the biggest and most damaging aspect of writing software in a productive manner—is indeed the writing itself. However, I bet that if we observe a lot of developers and measure the time they spend between writing code, i.e. producing new lines of code, compared to reading code, i.e. understanding and navigating previously written code, reading documentations, reviewing code written by their peers, etc—we will see numbers in the vicinity of 20% being the writing part to 80% being the reading part.

Keep in mind that these numbers are for coding activities only, I omit any time spent on corporate rituals like meetings and technical discussions, producing technical documentation, chatting with co-workers, and general procrastination. From my experience, an engineer is lucky to spend 2 full hours out of an 8-hour day in coding related activities, the rest of the time is spent on other tasks. And out of these 2 hours, the majority of the time is spent on reading/understanding code, rather than producing code. This alone should be a strong indicator that DX doesn’t really matter, because if you want to optimize for productivity, you better optimize for ease of reading and understanding code (or abolishing meaningless corporate rituals, but the end of the world will arrive sooner than management will free us of meetings).

I believe that this was the purpose of Go. It’s an ugly, in my opinion, and very verbose language, but it’s super easy to read. But let’s say you still disagree with me. I shall demonstrate it by looking at one of the tools I like: Svelte/SvelteKit.

NOTE

My point is not to criticize Svelte/SvleteKit, I really enjoy the framework, but rather demonstrate a point with a tool I’m familiar with it.

Svelte and Kit are the king of DX. Jump on HackerNews or any web development related subreddits, and you will see how people praise Svelte and Kit. It is a great framework indeed, a very refreshing approach to web development. But, as many other frameworks, it optimizes for the wrong things. While it has a great DX, it breaks quickly when you are done developing and now need to move to production. You will, eventually, encounter weird bugs with hydration errors, or that window is suddenly undefined. Moreover, while you develop with a hot reloading server behind npm run dev, after you deploy your npm run build code, parts of your code won’t run, even though in development, your DX was great. With time, you will realize that there is a prerender = true in some obscure +layout.ts file somewhere in your hierarchy, and while npm run dev behaves in one way, thus allowing you to have great DX, the production version behaves in a completely different way.

You could say that I’m a terrible software developer, a noob with Svelte, and any experienced developer should have known these things. And by claiming that you will support my point. DX is great when you begin, even as a newbie. This is what allows you to move fast. But when time comes to actually run and maintain your code, you will spend a lot of time trying to understand why this particular piece of code does not execute. And it’s only after you’ve burned yourself, and now hold this important lesson, and contextual knowledge, in your head—you will be able to move at a decent speed post initial development.

I had similar experience with NextJS, where everything would work great during development, but in production I’d get hydration errors. And I bet people have similar experience with Laravel, Django, or Ruby on Rails—all of them claiming to have tremendous DX. But the moment you need to actually run your code, maintain it, and fix bugs in production, all your DX is thrown away with lack of strong typing, magic shenanigans, and the need to hold contextual knowledge related to the current minor version of your framework.

On the other side, however, we have poor DX in the form of Rust. Jump on the same forums I mentioned early, and you will see all kinds of accusations thrown at Rust: the borrow checker fucks you up, the language is hard to pick up, the DX is poor, etc. But once you finish your application, not without pain and tears, it will run pretty much forever. Yes, there could be logical bugs like in any software, but there won’t be surprises or weird behaviors. What you saw during development, will happen in production.

Moreover, when you come back to develop yet another feature, or fix yet another bug, there won’t be any surprises or hidden rocks. Refactoring will be a breeze; reading and navigating the code will make sense; and a chance for you to break something in an unrelated part of the application—is slim. All this without the need to hold enormous contextual knowledge, and without the fear that the next minor upgrade of your dependency—will spiral your productivity down the abyss.

Developers are obsessed with tools and frameworks that help them to be more productive. For years, if not decades, productivity meant speed of writing code, which is important, but it’s a very small part of the entire process of development in general, and if you want to optimize productivity, you better focus on other areas in order to gain more benefits.

Software Development Fragmentation

Wed, 30 Oct 2024 16:00:00 GMT

About 15 years ago, I got my first software engineering job, and the way things were done back then—was different.

I was working in tourism/hospitality, and we were building a B2B product targeted at hotel owners and travel agencies. The product was big and had vast feature set. But in addition to the main functionality of the product, we were also building an accounting and BI (business intelligence, a fancy word for analytics), submodules. We were also building tools that would export our accounting data to standardized industry tools like SAP, but nevertheless this practice of building in-house accounting or analytics is practically non-existent in modern software development.

If you Google for “how to code an accounting system”, you will land on a bunch of Reddit or HackerNews posts telling you “Don’t”. I’m not sure whether this advice stems from the incompetence of the current generation of developers, who are afraid to build anything more complex than saving a few JSONs in a database; or it’s a genuine concern to avoid wasting time on trivial tasks that can be outsourced to a third party SaaS. Sure, accounting systems are complex, and there are best practices to follow like “don’t use floats for money” and “please don’t use floats for money”, but nevertheless they are not impossible to build.

Despite this fact, today, the consensus seems to be to outsource any mundane task like accounting or analytics to a third party service that is designed for these particular tasks. Hence, such approach creates a fragmentation in software. Your product is no longer standalone, and in order to be used properly it needs to be tied to these third party SaaS products. This, often, leads to absurd scenarios where you collect analytics in tool A, but this tool is so general—in order to appeal to the masses it has to be—that you end up building complex data pipelines to move data from said tool, to yet another tool designed for visualization. In reality, you have all the data you need, you just need to augment it with some additional analytics collection and build a proper visualization that makes sense for you.

You might say that it’s a waste of developers’ time and company money to develop yet another analytics tool where there are a dozen existing solutions that cost a cup of coffee and a croissant a month. But this reminds me of the fallacies described in “Mythical Man Month”. Sure, you save time on development, but you did not account the time needed for training and mastering said tools. And let’s be honest: there is no such thing as due-diligence. You can’t possibly determine whether the tool is right for you by reading their landing page or talking with their sales rep. The only way to effectively evaluate the tool is by using it. And by the time you realize that it’s not exactly what you need, you are already knees deep in it, and the sunk cost fallacy starts to kick in, so you overengineer a solution to build a data pipeline to a data warehouse as if you are a warehouseman working in a sorting center at Amazon. But hey, at least you didn’t have to reinvent the wheel™ and waste company money and resources. Well done!

All this could have been avoided if you built a custom tool tailored for your needs. In the end, you have all the data, why spread it across multiple third party tools in order to later build a data pipeline that unifies it all together? But no, we need to fragment. We must not need to reinvent the wheel. And yes, sometimes it makes total sense. But I don’t know what is it with the modern software development practices, but we seem to rely too much on third party solutions, as if the role of a software engineer was reduced down to a procurement employee who stitches together a solution based on dozen third party services.

But you need two for a tango, and as the inefficiencies grow on the development side, so do SaaS companies tend to meet the demand (or maybe they induce it?) to create dull software offerings. I have yet to see a development company that settled on one particular tool, whether it’s for knowledge sharing, project management, or analytics. It’s always a combination of multitude of tools because “while Jira works good for us, we actually use Monday for our Gantt progress”; “well, yes, we collect analytics in Google Analytics, but then there is a process that moves data to snowflake to visualize it”, etc. SaaS models make sense from financial perspective, but they are far from being the most optimal technical solution, as they focus on wider distribution thus offering generalized solutions.

I’d argue that on-premise/self-hosted tools with access to source code behind a license that forbids redistribution, or dedicated development team, would be a better solution overall. A company would host the needed tool on their infrastructure—thus removing the need for the tool provider to comply with dozens of regulations (“oh, but we need you to have an EU datacenter because GOVERNMENT”)—while at the same time retaining the ability to modify the tool to suit their specific needs. In the end, you have a data store, you just need to enhance this data in order to make sense of it for you. Why would you move analytics data storage to a third party data broker requiring them to comply with whatever regulations you need to comply with, while accepting the risk of data mismanagement, downtime, or being a hostage of their pricing plan—when all this can be achieving by hosting your own, whether purchased or developed in-house, tool?

I know why. Because the fragmentation happens not only on the software level, but also on the infrastructure level. Modern software development is so convoluted, that nobody really knows how the deploy process works or why the fuck it takes 45 minutes for your code to reach production. Asking for a custom deploy solution, outside the established practices by the company, will result in a firm denial from DevOps as their custom terraform wrapper doesn’t know how to deploy Joe Shmoe’s analytics tools.

So we are left at the mercy of our Lord and Savior the REST API, which allows us to fragment our software further and further, while benefiting no one. Neither you, whose data is now being spread across N services and requires complex data pipelines, nor the SaaS developers, who try to appeal to the masses by building as abstract as possible—benefit from it. Really, an analytics tools that shows you how many people are online on your website for $20/m, or “CONTACT US FOR ENTERPRISE” pricing is the best we can do? Should I remind you that phpBB had this functionality built-in for decades?

One year of Rust in production

Sun, 22 Sep 2024 10:00:00 GMT

For the past months, I spent (almost) all of my free time working on my side business — JustFax Online. It all started with migrations from LemonSqueezy to Stripe. But like every refactor, or rewrite, it grew way bigger than I anticipated. A simple payment provider change required me to implement a job processing queue on top of SQL as well as doing a tiny accounting system. All in Rust of course. This resulted in one of the biggest merge-requests I have ever merged.

A merge-request from the new version of JustFax to the master branch

This merge-request comes just in time, and reminds me that I have started JustFax about a year ago (maybe 11 months). In addition to that, I’m working with Rust in production in my $MAIN_JOB, so I decided to give an overview of what it’s like to use Rust in production.

If it compiles, it runs

I mentioned this briefly in my initial impression of writing a web application in Rust, and I want to emphasize this again.

I don’t, usually, run my code after each line I change (unless I do annoying CSS adjustments). Most of the time, I finish a big chunk of refactor, or feature implementation, and then run everything to test. Moreover, with Rust, you can’t run a code that does not compile, so if your code has errors, you need to fix them, unlike JavaScript. And so I got carried away with the refactor, which span across endless amount of evenings, during multiples months. One thing led to another, and the code was never in a state when it could compile, until the very last moment when everything was done.

I was in fear running everything, but after setting up every component of the system, and issuing a first (local) request, I was surprised to see that everything works! I know that I’m a good developer, but I don’t fool myself that I can write code without mistakes. And apart from minor issues with dynamic aspects of the code, like wrong serialization/deserialization formats, the code worked flawlessly.

The first layer for this success, is type-safe, compiled nature of Rust. You can’t, accidentally, assign a i64 to uuid, Rust compiler won’t let you do that. Nor you can access a non-existing field in a struct, unlike JavaScript. And sure, there are people who claim to never experience cannot access property "foo" of undefined runtime errors, my experience, as well as the experience of my colleagues — is not the same. One of the reasons that we use Rust at my $MAIN_JOB, is because the existing backend, written in dynamic language, became unmanageable. Sure, you can hold the context of a small application in your head, but at some point it will outgrow your memory. I remember in the old days when I used to do PHP, at some point we had to resort to dumping the content of the array on screen in order to know what keys it has because some keys would be snake_case, while other camelCase, etc.

Second layer, and it’s kind of an extension of the first, is the obsession people have with type-safety in Rust community. And it’s a good obsession, because it gives birth to tools like sqlx — a compile time, type safe SQL wrapper that runs your queries against a real DB and won’t let your code compile if your query is wrong syntactically, or your try to insert a i32 into a TEXT column without explicitly casting it. I used to fear SQL, because it is usually a big part of every application, and it’s easy to get it wrong. Misspell a query, or mess the arguments, and good luck to you in production. Many people would write unit tests around SQL queries, while others would resort to using ORMs or code generators from your schema. But with sqlx I no longer have to worry. I get the full power of SQL without going to a high level, very abstracted ORM, or resort to code generation from the database schema. There are drawbacks, sure, and I’ll mention some of them further in the blog post.

Another set of tools that I can put this category, are type-safe templating engines like askama or maud. They did not find their place in my project, even though I toyed with them, but they nevertheless remove yet another element of unpredictability from your code. One of the things that make me chuckle, is when I get an email that starts with Hello {{userName}}. It’s funny to get it from a no-name service; it’s embarrassing to get it from a bank. With type-safe, compiled templates, such errors are non-existent.

The only thing I miss is a type-safe third-party API interface. Sure, you can generate an OpenAPI client from an OpenAPI/Swagger spec, but you are still left at the mercy of the third-party API provider, and the fact that they have a spec, does not necessarily mean that they will follow it and won’t break their API, so I’m not sure that we can solve this one.

And when it runs, it’s very stable

I have never ever experienced a Rust process crash. I did experience a Node process crash. Unless you pollute your code with .unwrap() (which basically says “if the result is an error, crash”), there is a high chance your process will never crash. I have a few .unwrap() calls in my code, mainly during initialization, so if the config file is missing, or the environment variable was not defined, there is nothing I can do except crash the process. But in general, Rust requires you to handle errors explicitly. Everything that returns a Result<Something, Error>, requires you either bubble up the error through the ? operator, or perform a match statement on it. In most cases it makes sense. And in some cases, it’s a bit annoying (for example when you know that the conversion from i64 to i32 will succeed because it holds a number that is well inside the boundaries of i32, you are still required to do try_from() which returns a Result). But even with the annoying cases, I tend not to resort to .unwrap(), and would rather log a warning and return some sane default value. Programming is unpredictable, so at least this way my program can continue to run, while logging the location of where I could have been stupid.

And the tooling is incredible

JustFax is not just Rust, it also has internal systems written in TypeScript, and other JS frameworks. And I swear that every time I create a new TypeScript project, something changes. Either the config file changed for some tool (looking at you eslint); or there is new boilerplate for typescript; or express is no longer cool and now everyone uses fastify; or express is cool again; or ts-node is bad, now we use tsx; or this module is esm only, while this one is cjs so go figure. There is always something with TypeScript. The way you work, the way you lint your code, the way workspaces work in vanilla JS compared to TypeScript.

But not with Rust. cargo init - boom you get a new project. Want workspaces? No problem, works flawlessly. Linting? clippy got you covered. It’s just so much less boilerplate and mental fatigue.

Compile time is still PITA

Undoubtedly the biggest drawback of Rust — is its compile time. Especially when you resort to using tools like sqlx or maud which rely heavily on macros, both the LSP during development, and the compiler — start to struggle. As the project grows, I collect more third-party dependencies. And while I try to optimize them by putting common dependencies between multiple packages, in the root Cargo.toml, as well as aggressively selecting only the needed dependency features, I still struggle with compile time. I mentioned how the initial compilation took about 6 minutes inside a CI/CD, which now takes about 20 minutes. It’s possible to optimize it according to different blogs online, but it requires a surgical-like procedure to aggressively cache the dependencies in a multistaged docker container. I don’t have time for that right now, and need to focus more on the business side, but I will get to it later.

NOTE

It has been pointed out to me, in a Reddit discussion about this post, that I use a wrong cache key for CI. Fixing this, should fix the long CI/CD pipeline, because then the build will use previously built/compiled artifacts. I haven’t tried it yet, due to lack of time.

Local compilation on a Mac M2 is manageable especially with incremental build, but it comes at a price of storage. Once in a while I run cargo clean, which results in tens of GBs of cache being removed. However, even with incremental compilation, it’s still far from instant hot reloading that you get in JS world. So circling back to my first point, a development cycle of “change small piece of code, test immediately” kind of breaks. You don’t get this instant feedback that you get with JS/TS, and that’s the reason some of the internal, and external, tools for JustFax are still written in TypeScript. Rust kind of guides you towards “write a lot of code, compile, check” flow, rather than the “change one line, alt-tab browser” workflow that you have with Node/JavaScript. I’m mostly fine with that. I get a lot in return, like real type-safety. The CI pipeline, I believe, can be fixed, I just need more time to work on it.

The tools are not quite the same, yet

There are things that Rust excel at, such as pure backend. Frameworks like axum provide a lot of needed parts to build an API server, and most (popular) external APIs have a client for their API in Rust. Sure, they are usually not maintained by the company who builds these APIs, but at least we can use them. However, most examples on the web do not include Rust. And so you are left on your own to figure out how to integrate with this or that API.

And it’s a recurring theme with development in Rust, at least for web, I can’t comment on other types of applications. I often find myself reading the source code, or navigating the GitHub issues for similar problems to the one I encounter. LLMs rarely help with a proper solution, as most of the packages are kind of niche. Rust community has great attention to documentations and examples for their packages, which is awesome, but inevitably you will end up with some edge case that is neither documented, nor has an example for it, and chances are you will have to resort to reading the source code in order to figure stuff out.

And some applications are not well suited for Rust, especially if you come from fast-prototyping environments. I still prefer to write frontend in TypeScript using Astro or Svelte. Rust DX is just not great in this department. With the need to recompile the code upon every variable change, it makes it too slow to iterate fast on frontend development.

In conclusion, I’m happy that I have chosen Rust one year ago. Not only it helped me secure a $MAIN_JOB that I really like, but it also helped me to build better software. And I’m looking forward for the second year of developing Rust.

Software development is hard

Sat, 17 Aug 2024 21:00:00 GMT

In my latest blog post, I compared software engineers to construction engineers. I said, that if we, software engineers, want to have the same level of responsibility as other engineers, we need to have the same level of respect other engineers get. But something was haunting me. I was thinking a lot about software engineering for the past month, and I couldn’t understand why software engineers are different. Could it be “just because”? Software is young, compared to other engineering professions. Could it be that we, as humanity, haven’t really understood software development to it’s fullest in order to hold software engineer with the same level of responsibility as we hold other engineers? But then, it all came to me.

I was writing code that integrates with another third-party API. As usual, I was consulting the API documentation. There were no official libraries, and I had to construct the API client using the documentation and trial and error. All went well, I did some testing, and eventually pushed the code to production. Sometime afterward, the code broke. My server would return 500 error code. Why? Because the API provider changes their API.

Who is at fault? Me, for not monitoring the API change logs? The API provider for not communicating these changes to all its consumers? Is it even possible? An average website communicates with a dozen of third party providers. We don’t think about it, and take a lot of these dependencies for granted.

Let’s look at a database for example. Nearly every web service, except for statically rendered web pages, depend on a database. Most of us take it for granted. Have you ever checked that your UPDATE table_name SET field = ? WHERE id = ? actually affect any rows? What if it doesn’t? What do you do? And what happens when pool.acquite() fails to acquire a connection from the connection pool? Sure, it almost never happens, but a sporadic network error could cause your code to fail. But none of us think of such cases.

Let’s look at another example: writing a file to the disk. During our day-to-day life, we rarely encounter errors when trying to write a file to the disk. But it can happen. A bug in the OS. Lack of disk space. Absence of writing permission for the process.

We take all these things for granted, as if they can’t fail, but they can, and they do. And then the question raises: how can we demand accountability from software engineers when they depend on systems they did not develop nor understand fully? An electrical engineer checks the wiring in your home, he is not responsible for the grid providing you 500 volts instead of 220 volts. When we develop code, we make assumptions. Assumptions that the network will function, because if it’s not, the IT engineer needs to fix it. Assumptions that the disk is not full, because we, hopefully, have monitoring alerts on disk size. And worse of all, we integrate with a dozen of providers, often times using lousy documentation in combination with trial-and-error or some test data, to determine the happy path flows.

It might sound like I’m whining. Like I’m trying to say “Hey, we are poor software engineers, don’t demand accountability from us!”. But I’m not. I truly think that software development is hard. Sure, it might sound like we put stuff in a database or move pixels on a screen, but it requires so much effort, and belief in others, that I don’t think it’s even possible to develop a fault-tolerant software. We can reduce the amount of bugs by, say, doing more tests, but PMs don’t have time for tests. Great software exist, but it comes with a cost. A deadly cost of time. When you don’t have the luxury of time, you end up with “Ops, something went wrong” type of errors.

Software development depends on a lot of verbal agreements, that rarely are respected. It depends on a lot of assumptions. Assumptions that the network will be there, that the database will be able to update the record, and that the user won’t take the hard drive out of the computer while running our software. Most of these assumptions are legit. That’s why most of the software works most of the time. But what do we do when we can’t acquire a database connection because the machine has reached its memory limit? Do we retry again? For how long?

Software development is hard.

Let's blame the dev who pressed "Deploy"

Sun, 21 Jul 2024 09:00:00 GMT

Yesterday I was browsing /r/programming and came across a blog post titled “After CrowdStrike, Programmers Deserve Consequences.” The author opens up with a comparison of Software Engineers to an Anesthesiologist or a Structural Engineer, and tells a personal story about his friend who got stuck in a hospital due to the CrowdStrike outage. And it’s true—software bugs can kill people. Mistakes in airplane autopilot software, or malfunction in equipment run by software—can cost the lives of people.

But then the author engages in an absurd rant about how the entire software engineering industry is a “bit of a clusterfuck”, and makes a delusional claim how software engineers should bear the responsibility for bugs and outages. It’s not the first time I hear these arguments. I remember when VW emission scandal was revealed, people had the same arguments: the developer(s) who coded the bypass—should be responsible. I’ll explain, in the end of the article, why I don’t think developers should be responsible, but for now, let me give you some more figures to blame.

We can start with the CEO. When you argue with CEOs why they deserve such a big pay and bonuses, they will usually say something along the lines of: “Entrepreneurship implies huge risk and lays the responsibility for failure on the shoulders of the founder/CEO”. And it’s true. Founders/Entrepreneurs bear a lot of risk. They go on raising money, getting into debt, take the role of running a company and they, usually, enjoy the result of their success—which is translated to a nice salary—or bear the consequences of failure. At least they used to.

I remember times when leaders had dignity and self-respect. They would go on stage and apologize. They would take responsibility and outline an action plan. Some even stepped down from their position as a sign of failed management. But today? Today, they hide behind an army of PR people who write blunt, soulless, unsympathetic press releases, blame their customers and compare their employees to a number in a balance sheet. They no longer care about serving the public, but rather generating profit for investors.

And, usually, they fail upwards. George Kurtz, the CEO of CrowdStrike, used to be a CTO at McAfee, back in 2010 when McAfee had a similar global outage. But McAfee was bought by Intel a few months later, Kurtz left McAfee and founded CrowdStrike. I guess for C-suite, a global boo-boo means promotion.

But if C-suite fail upwards, then the next obvious party to blame is the Customer. We could blame United or Delta that decided to run EDR software on a machine that was supposed to display flight details at a check-in counter. Sure, it makes sense to run EDR on a mission-critical machine, but on a dumb display of information? Or maybe let’s blame the hospital. Why would they run EDR on an MRI Machine?

We should blame the IT department in these companies. Unless we understand that the IT department was simply trying to check a box in a stupid regulation imposed by the government. So it makes sense to blame the Government. A government that creates absurd regulations by senile people whose familiarity with technology is limited to their TV remote. Have you seen the absurd questions by the US senate to tech entrepreneurs?

Politicians are good at what they do—creating regulations. The companies are then left at the mercy of implementing these regulations. I imagine an IT specialist trying to explain to his manager how it makes no sense to install EDR software on an MRI machine, but his manager has no clue about it and simply tries to follow the regulation. So maybe we should blame the middle management? The same middle management who cuts corners because they need to demonstrate KPIs, or fulfill unrealistic deadlines that they, themselves, created?

Or, we cane go a level higher and blame the board. The board who cares only about one thing: $$$. How can we save dollars? Let’s fire, sorry layoff, some engineers. Clearly, these overpaid peasants don’t deserve the money they are getting paid, and we can remove some of them. Or better, replace them with AI. I won’t be surprised if this outage was caused by CorwdStrike trying to “optimize” it’s pipeline by ~~firing~~ laying off some QA engineers and or replacing them with AI.

But the truth is, we can’t blame any of the above. They are the innovators, the entrepreneurs, the backbone of our economic systems. They all fail upwards. The white knights of our modern society. Smartest, most successful individuals whose words and practices should be taken without questions as if they spread the message of God himself.

No, we can’t blame them. Instead, we should blame the peasants. These dirty developers who make a lot of, unjustified, money, whose work can be replaced by AI.

You want to gate keep the word “Engineer”? You want to compare Software Engineers to Structural Engineers? Then I have a shocking discovery for your. A Structural Engineer won’t sign off a design that was made under pressure or with cut corners. He will shift the blame to the construction if they decide to use a different type of steel or cheaper screws. And if you decide to replace an Anesthesiologist with AI, then good luck to you.

The reason why Anesthesiologists or Structural Engineers can take responsibility for their work, is because they get the respect they deserve. You want software engineers to be accountable for their code, then give them the respect they deserve. If a software engineer tells you that this code needs to be 100% test covered, that AI won’t replace them, and that they need 3 months of development—then you better shut the fuck up and let them do their job. And if you don’t, then take the blame for you greedy nature and broken organizational practices.

Because blaming software engineers is nothing more than satisfying the bloodthirsty public for your organizational malpractices. Sure, you will get the public what they want, but you won’t solve the root cause of the problem—which is a broken pipeline of regulations by people who have no idea what are they talking about, to CEOs who are accountable only to the board of directors, to upper and middle management who thinks they know better and gives zero respect to the people who actually do the work, while most of the latter just want to work in a stable environment where they are respected for their craft.

The day of the blue screens of death

Fri, 19 Jul 2024 15:00:00 GMT

Earlier today, a broken update from CrowdStrike caused massive global outages in machines that are running Windows. A component named “Falcon”, that is responsible for endpoint protection, and runs in kernel-mode, causes machines to fail into blue screen of death (BSOD) and enter a boot loop. Among the industries that were affected are: airlines and airports, railways, healthcare, postal services, broadcasting, retails shops, banks, stock exchanges, etc.

While I could talk about the reason behind this outage, smarter people than me already analyzed it, so go read their reviews if you are interested. Instead, I want to talk about how we got there.

When you zoom out, and look at the situation from above, you realize two important facts:

A lot of critical infrastructure is, still, running on Windows
Windows is not secure enough

I haven’t used Windows in over a decade now. I first switched to Linux, and after getting a Mac from my former employee, I made a switch to macOS. Occasionally, I run Windows in an isolated VM due to development that I have to test on Windows. Back in the days I used to run Windows, I remember that the first thing I had to install on a fresh Windows machine — is an antivirus. Modern Windows still feels like a malware. It’s sad that Microsoft decided to focus on putting ads in their system, rather than making their system more secure by default. Instead, you have to reach to third party providers to protect your systems.

More over, these third party providers operate in a push manner, where updates are distributed automatically without any approval from the system administrator. In Linux based servers and systems, updates are, usually, performed in a pull manner, where the DevOps engineers or the system administrator goes and updates the system, and it’s underlying components. And the result of a push based system is evident in today’s outage. A private company that has access to most critical civilian infrastructures and can bring to an economical disaster. Hell, forget economy, I’ve read comments on HackerNews about hospitals going down during heart attack surgeries.

I could rant about how irresponsible CrowdStrike was during the rollout of this update. How they should have better rollout mechanisms, or how we should find the one’s responsible and prosecute them. But I hate it when after every incident people jump into “we need to find someone to blame/take responsibility” mantra. The sad reality is that nothing will happen, and no one will take responsibility. Instead, I want to talk about something that no one mentioned that much, and it’s the security through obscurity that many companies practice.

In order to understand, and prevent such cases in the future, we need to talk about why CrowdStrike is so popular. You see, as companies become bigger they start to drown in bureaucracy. Some of it is self-inflicted, while other is imposed by the government/regulating entity. To put it bluntly: companies need to check boxes.

The intention of “box checking” is good: we want to prevent outages and security incidents. But at some point, this ritual becomes so engraved in the minds of the developers and the management, that nobody dares to challenge it anymore. “We need to run security scans.” Why? Nobody knows, we just need in order to pass some audit. And sure, at the beginning you try your best to fix all vulnerabilities found by the scans. But then you meet a stubborn person, to whom you try to explain that this particular vulnerability can not be exploited because the package that is responsible for this vulnerability is used only during testing phase, and is removed from the final build. But someone needs to tick a box somewhere, and so you sweat and pull out your hair in a desperate attempt to fix this vulnerability, and eventually you end up with some hack, just to satisfy the scanner and your boss/auditor.

And sure, this example can be somewhat imaginary, but not far from the truth. Everyone who worked in medium to big sized corporate, have encountered horror stories about security through obscurity. A ritual, or a workflow that exist for the sake of a checkbox on some audit paper, but serves no real purpose in strengthening the security of the company and or the product. And so we end up outsourcing our security to third party companies who scare you with how 62 minutes of downtime can kill your business, and instead you should use their software. A sales rep that convinces a higher up manager, who convinces everyone else that we must use it for our own security. And there is no sane person in this chain to question the absurdity of the decision and the potential dangers it might inflict to your business, because they tell you: “we can’t fail”.

But the truth is: everyone can fail. Companies are made of people, and people make mistakes. The question is how you recover from it. And in the case of this outage, the recovery is very painful. It’s a long weekend where system administrators will need to recover every, single, machine. I don’t envy them. “Somebody should take responsibility.”

Tips for improving your CV

Mon, 17 Jun 2024 06:30:00 GMT

Recently, I’ve seen a lot of people online asking for CV improvement tips. Most of them make the same mistakes, and while I tried to help some of them, eventually I realized that I repeat myself too often. So I decided to share some tips for CV improvement from someone who’s been on both sides of the interviewing process, and is married to a tech recruiter. So let’s begin.

Who reads your CV?

Before we begin to talk about your CV, we need to understand who reads them, and what’s their purpose. Excluding any automation and AI tools that might or might not exist nowadays, the first person to view your CV—is a tech recruiter. Their role is to do a quick match between your skills and the position.

NOTE

In smaller companies, the first person to view your CV might be an engineer or a hiring manager. This does not change the role of the CV in any way.

A recruiter, on average, spends between 6 and 8 seconds per resume. Why? Because some positions get hundreds, if not thousands of resumes. Many of them are irrelevant for the position, location, experiences, skills, etc. Some CVs are needlessly long, or contain absolutely irrelevant information. Therefor, we, humans, develop optimization mechanisms.

A recruiter will try to scan your CV for recognizable patterns. They will then do an assessment based on these patterns, and if your CV passes the initial assessment, the recruiter will spend time reading the relevant parts.

It’s important to clarify that this does not tell anything about how good or bad the recruiter is. You might think that it’s unfair that recruiters resort to quick scanning rather that reading your entire CV, but think about it this way: do you really read every line of 1500 lines modification PR? Chances are, you skim. Because if you wouldn’t, we would live in a perfect world where there would be no bugs or security vulnerabilities.

So look at it as a fact of life. These are the rules of the game, and you can either play it, or go be angry at how unfair the world is.

How to style your CV to catch the eye

As I mentioned earlier, people who read your CV, resort to quick scanning. This is why we want to catch their attention. There are many ways to get the attention of the reader, and here are some examples:

Using bullet lists instead of long walls of text
Structuring your text logically with headers
Highlighting relevant achievements and skills by using text formatting such as bold or italic
Removing unnecessary information that shadows the important parts
Rephrasing complicated sentences (for example, instead of saying “in a logical way”, you can say “logically”, etc)

How to structure your CV?

Now that we understood who looks at your CV and in what manner, let’s talk about some basic structuring elements of the CV.

Header is the first thing that catches the eye of the reader. This is like your business card, and it should tell the person who sees it:

Who are you?
What do you do?
Where your work can be found?

Some people go crazy in their header and include things like their picture, full address, or hobbies. All these things are irrelevant, and some of them can even harm. I will discuss things you should not put in your CV later in this article.

Your header should include at minimum: Your full name, relevant title for the position, and contact details like email and/or phone number. It’s also advisable to add links to places that showcase your work or contributions. This could include links to:

LinkedIn
GitHub/GitLab/BitBucket—a place where you can showcase your projects
StackOverflow
Blog or personal website/portfolio

Use common sense. If you don’t have an active GitHub, don’t link it. Same for StackOverflow, LinkedIn and any other website. If you blog about food recipes with mushrooms, but applying for a software engineer position, your blog is a distraction, and should not be in your CV.

Last tip: links should be clickable and visually presented as links (for example: using blue text). The last thing you want is for the person to mistype your email, and you will never get your interview appointment.

Summary

Summary is a very contradicting block. The general advice is to skip summary all together, and instead use cover letter if the job application requires it. If you do want to include summary, and have the necessary space (I will discuss later about the length of the CV), then you should—(a) keep it short, and (b) explain why you are a good fit for the position, rather than talking about your life journey.

Education

If you have a relevant degree (read: computer science or mathematics), include them at the top of your CV (after header and summary). Irrelevant degrees (sociology, psychology, architecture), should be included in the end of the CV.

Make sure to highlight relevant honorary titles, and include any high scores, achievements, relevant publications, projects, and researches.

This section should be short, and to the point. Ideally a one-liner per degree for non internship or junior positions.

For people without degrees, remember that there are plenty of talented, self-taught developers who work, and are successful in the industry. You should still apply to positions that require degrees if you believe that you can compensate the lack of degree and showcase your strong side in the CV.

Work experience

This is the most important section of your CV, and should include, in reverse chronological order, all your work experience, internships, contracts, projects, etc.

Each position, including different roles in the same company, should be visually separated, while all roles in one company should be grouped together. This presents growth in the company in a visual way to the eyes of the reader. You should also include tenure for each role.

Each company block can include 1–2 sentences about the company and/or the products. Afterward, mention your role(s) together with these 3 points:

Things done
Achievements and impact
Tech stack

Things done are, well, things you’ve done while performing the role. It’s always better to back it up with numbers. Here are some great examples:

Reduced response time from 1.5s to 200ms in a critical part of the system
Increased sales by 30%
Implemented TDD which caused a reduction of 25% in runtime in our CI/CD pipeline

Notice how each sentence is (a) phrased in past tense, (b) is part of a bullet list, and (c) highlights important information. Now, instead of trying to explain you why this is better, let me just write the same information in a paragraph and let you be the judge.

I work at X for Y years, in a team. During my time there I managed to achieve a lot. One great example is when I was able to write a piece of code that reduced the response time drastically in a critical part of the system. This was great, because it helped the company to increase sales. I was also able to implement a test driven methodology which helped cut the run time of our CI/CD pipeline by 20 seconds.

One word: BORING. Ain’t nobody got time to read it. It’s like a dull novel. It says nothing by using so many words and mental capacity. There are no numbers and no reference points, but instead ambiguous words like “drastically” and “increase”. The only number here is “20 seconds”, but even then it’s useless, because if your pipeline was running for 17 minutes, 20 second reduction is not an achievement I’d put in my CV. 25%, on the other hand, is.

There is another psychological trick you can use called “framing”. For example, a reduction in response time from 600ms to 400ms, might not sound impressive. On the other hand, 33% reduction—is impressive, even though it’s the same information.

Before we move to the next section, one word about tech stack. There are two approaches:

Adding tech stack per position
Adding a separate CV section for tech stack

I recommend using the first approach, as this indicates two things: (1) showcasing your most recent tech stack, and (2) might show a progression and/or growth.

Additional information

Some more information that you can include in the CV:

Spoken languages
Openness to relocation, if applicable
Links to relevant publications, technical papers, patents, etc

Common mistakes in making your CV

Biased information

By far, one of the biggest mistakes people make in their CV, is to include biased information. They might think it’s cute that they can bake cupcakes, or like to put their photo in the CV, while in reality it might play against them.

I’m not going to talk in depth about biases, why they exist, and how the world is unfair. Instead, I’ll leave you with that: this is a game, and you need to play by certain rules in order to be able to compete and win this game.

We, humans, all have biases. And even though we undergo corporate trainings, you can’t go against your nature. Your interviewer might dislike dogs, or be allergic to cats. The fact that you dedicate a sentence, or God forbid a paragraph, of the CV to your beloved furry friend—might bias the interviewer to look for reasons to reject you.

Here is a non-exhaustive list of things to not include in your CV: photo, hobbies, marital status, location, sex, race, culture, religion, age (including date of birth), disabilities, etc. Having said that, there might be laws in your country, or the country where you apply in, which require you to disclose certain information, so get yourself educated on that matter.

Unnecessary information / Generalization / Too long of a CV

The second most common mistake—is a very long CV and/or CV that tells the reader nothing about you and your experience. Firstly, let’s talk about CV length.

Your CV must be 1 page long. If you have more than 8 years of experience, it’s fine if your CV becomes 1.5 to 2 pages. Here are some tips how you can reduce the length of your CV:

Removing irrelevant information
Replacing terminology with abbreviations: Database becomes DB, Pull Request becomes PR, etc
Avoiding “walls of text”
Playing with font size and formatting options

In case you still struggle with CV length, consider limiting information about your oldest experience. Remember, most interviewers are interested in your recent experience anyway.

Lastly, I want to touch the topic of unnecessary information and generalization. These two things contribute to CV length as well, and avoiding them can help you make your CV leaner while providing more context to the reader.

Avoid adding any information that is not relevant to the job/position/industry. Your future team lead does not care about the fact that you were the beer drinking champion in high school, despite the fact that you consider it an honorary badge. Instead, they care about your experience in impact in past jobs. This is why it’s important to make your CV concise and to the point.

Instead of writing vague statements like “Designed an internal system”, try to answer questions such as “What system?”, “Why it was needed?”, “What was the impact of this system on the product/company/team?”. Every sentence in your CV should show impact and explain—better with numbers—the value of the idea, task performed, or experience gained. Remember, you have to explain years of your work in a few sentences, why not focus on the most important things that really demonstrate your growth and the value your provide(d) for the company?

Design / Format / Spelling

The last mistake people make with their CVs is related to the design and the format of the CV.

The only acceptable format of the CV is PDF. Other formats depend a lot on the software in which they are viewed, so a .docx file created on your computer, might be completely messed up on the computer/software of your reader. PDF does not have these issues.

In terms of design, your CV should be monotonous color and text only. Avoid any graphics such as images, or complex formatting options such as tables or forms. Use neutral colors: white background, black text, gray headers, blue hyperlinks. Avoid using anything that can distract the reader from the main purpose of your CV.

Lastly, make sure that your CV is professional. This means two things. First is spelling and grammar—make sure to reread it before submitting, give someone to proofread it, or use spellchecking software. Second, make sure your CV is written in the expected language. Some companies want a CV in English, while others might request CV in country’s locale language.

Final words

Your CV is a gateway to your experience. On one hand you want to put as much information as possible in there; on the other hand, you need to understand that CVs are rarely read thoroughly, at least not in the screening phase. This is why it’s important to craft your CV in a way that you can both (a) present your experience, and (b) catch the eye of the reader. Your CV could be deciding factor of whether you will be contacted back by the company and invited for interview.

However, there is more to CV that what I wrote in this article. I didn’t touch things like LinkedIn, which is preferred nowadays more than CVs; covers letters; blogs and other social networks. Moreover, CV is just one step in the interviewing process.

This is the reason I teamed up with my wife, a Senior Talent Acquisition Specialist in tech, and we wrote the book From Applicant to Employee - Your blueprint for landing a job in tech. This book covers the entire interviewing process, and this article is based on chapter 2 from the book: Building your CV. If you believe this book can help you with your job search, consider buying it on Gumroad or LeanPub in order to support us and this blog.

Why your team might be holding you back

Fri, 10 May 2024 16:00:00 GMT

About a week ago, my wife and I went to our first escape room in the Netherlands. We’ve been in about 12-15 escape rooms before, and were always able to escape as a team of two. Not this time, however. This escape room was the first where we failed to escape.

Poisoned by the corporate world I decided to do a postmortem of what went wrong, and what we could have done better in order to increase our chances to escape. We both came to a conclusion that adding two more people would have increased our chances to escape, as the room relied a lot on multitasking in its first phase. When we tried to think who we would like to take with us, if we could, we came to a conclusion that most of our friends would not be helpful.

Some, we don’t know well enough. Others are as competitive as us, and a team of 4 leaders can lead to only one outcome: failure due to inability to follow rather than lead. This recent experience, together with some past thought about software engineering as a team sport, made me reconsider the accepted norm of software engineering teams.

Why your team might be holding you back

It’s pretty common to accept the fact that software is developed in teams. Solo developers are an exception, and any successful MVPs that are built by 2 people, are going to grow into a big development team eventually. However, developing software in teams is not the be-all end-all solution. In fact, there are some issues that arise strictly from the fact that we have a team. Let’s look at these issues.

There are two ways to define a team on a given project. A team can be a combination of multiple disciplines like backend developer, frontend developer, QA engineer, designer and DevOps. But a team can also be 7 backend engineers. The former are less prone to team issues, however some issues also applies to multidiscipline teams.

The first issue I would like to discuss is the obvious concept on stepping of each other toes. When a team is too big, people start to work on overlapping functionalities. We try to pretend that we can solve it with feature branches and occasional rebase—in reality we can’t. The bigger the team, the more they step on each other toes.

Another issue with team work is the fact that not all team members are on the same page. You see, in reality, if we look at software companies, we assume they have one goal—build product for their customers and make money. As an engineer who joined such company, ideally you should inherit this goal. In reality, however, this is not true. Due to corporate BS and things like career ladders, every team member has a hidden agenda. It’s like these board games when you all work as a team towards common goal, but the actual winner of the game is the one who fulfills his/her hidden agenda. One such game is “The Resistance”, where everybody claims to be member of the resistance who tries to overthrow the government, but at least two people (depending on the size of the group) are secret spies who try to sabotage the resistance.

I’m not saying that in a corporate world, members of a team try to sabotage the entire team, but I am claiming that people have hidden agendas. It’s even part the growth plan. So members of the team might get tasks like “lead the design of a complex feature” in order to score points in their leadership qualities so that they can get promoted in the next promotion cycle. This creates a weird dynamic inside the team because team members would focus on reaching their personal goals while putting the company goal on a second place.

Composition of the team also affects its success. If we look at job requirements, we can agree that everyone wants the best of the best. They want the top performers, and A-players, rock stars, and ninjas. And while in theory it sounds great, in reality it’s a recipe for disaster.

If we look back at my escape room example, my wife and I wouldn’t take any of our friends with us. Maybe to have fun together, but not if our goal is to escape. We both have successful friends who are motivated and posses leadership qualities. Smart and successful people, usually, don’t like authority. In order to be a leader you need to be able to question the norm, and you don’t like to follow orders.

The same can be said for top performing software engineers. They are top performers, or A-players, because they question authority and dare to change what’s accepted. Gather a team of such people together, try to appoint one to be the lead, and you will get into constant refactoring sessions as nobody will be satisfied with the code quality of his peers. But companies seem to ignore it.

Lastly, team size correlates with growth. The other day I was watching an interview with Pavel Durov, the creator of VK and Telegram messenger. Telegram is a messenger that is on a path to reach 1 billion monthly active users, while having only about 30 engineers according to Pavel. When asked by interviewer why other companies have so much engineers, Pavel shared an interesting insight. It’s his words, so take it as you wish. He said that he met with Jake Dorsey at a time and told him that Twitter does not need that many engineers in order to run. Dorsey agreed with Pavel, and in reply said that he can’t downsize the teams as it will signal that something is wrong with the company.

And while I can’t attest to the existence of such conversation between the two, I can agree with a clear fact that team size equals growth. The moment the project is deemed as important, higher management will allocate monetary resources primary for one use-case: hiring. It becomes the number one priority of the team lead—to grow his team. And it creates a double-edged sword. From inside the company, team growth correlates with success of the product. From outside, candidates don’t want to work in small teams as they might see the project not as important as project that has a bigger team.

The lone-wolf

People who like or want to work alone, are not viewed well in the eyes of the corporate world. Every job requirement wants people who can work alone if needed, usually because companies don’t want to babysit you or save cost to avoid investing in proper documentation and/or trainings; but you also need to be able to work in a team and be a great team player, which usually means that you need to play the corporate rules of pointless meetings and other rituals around development cycles.

Derek Sivers wrote a nice paragraph about it:

Nobody gives a novelist shit for writing alone. But an entrepreneur, programmer, or musician is expected to collaborate. I disagree, for me. I prefer the life of a novelist, whether I’m writing code, music, or systems.

And I’d like to agree with Derek here. The moment I switched to a small team (we are two people), I must say that I find it way more effective compared to 5+ people teams I had before. There are no hidden agendas as we don’t participate in the ladders games. Nor we step on each other toes, as the product scope is huge. While infrastructure related code affects both of us, individual feature development is diverse enough that we can operate each by himself.

I believe that modern software development practices were inherited from wrong disciplines. Software development viewed more as a factory assembly line where more people equals higher output, but it’s far from the truth. A lot of the modern team dynamics are flawed, and they tend to harm productivity of each individual, thus sabotaging the entire team. I think that software engineering would benefit more if we would focus on one-man-teams or small, less than 3–4 people, teams. This configuration provides higher development velocity, and reduces knowledge fragmentation.

Development velocity is improved because nobody has hidden agendas, people don’t step on each other toes, and there is less overhead in managing the team, thus you are able to get rid of rituals such as stand-ups, plannings, retrospectives, etc. Knowledge fragmentation is reduces because each individual has bigger knowledge of the system. With bigger teams, in order to avoid stepping on each other toes, each engineer is given a smaller subset of the system, thus known as “a cog in the machine”.

Consider the above the next time you want to join a big team or scale your own team.

xz backdoor

Thu, 04 Apr 2024 08:45:00 GMT

A malicious backdoor was discovered in xz library that implements LZMA compression. xz, among many other places, is used, indirectly, in sshd. My attempt to explain what happened.

WARNING

I’m not a security researcher. I did not analyze the backdoor, and this post is my attempt to make sense of what happened in simple words.

I’ll link to as much content as possible from people who are more competent than me.

Timeline

Around the year of 2005, Lasse Collin together with other people, creates the xz library that implements the LZMA compression algorithm. LZMA is a lossless compression algorithm. Over the time, xz becomes widely adopted in compressing data, and you’ve probably had to download at least one file that has the extension of .tar.xz. Most of the work on xz was done by Lasse Collin, who, as we learn now, had personal struggle and not a lot of time to maintain the project. This setup was perfect for some social engineering.

At the end of 2021, we meet another person named Jia Tan. Jia Tan starts to create little patches to xz, but they are not merged.

Forward to mid 2022, another person, named Jigar Kumar, complains that the development on xz has stalled, and that the original maintainer should give the project to someone else. An exchange of accusations from Jigar Kumar, and explanations from Lasse Collin, take place in the following months from April till early July 2022. Eventually, Jia Tan becomes a maintainer of xz.

It’s important to note few things.

First, Lasse Collin is a one person that maintains a free open source library. A lot has been said on the state of open source and how maintainers treated unfairly. Companies build a billion dollar products on top of free and open source software, while rarely giving anything back. While the accusations from Jigar Kumar were probably part of social engineering, it’s not uncommon that maintainers of OSS are being harassed and demanded to patch their projects or keep developing them. There are no indications that Lasse Collin was part of the execution, and everything points out to the fact that he is just a guy who maintained a fairly popular open source library, and had personal health issues and lack of time.

Secondly, there are no proofs that both Jia Tan and Jigar Kumar are real individuals (rather than state actors); not the same person; didn’t know each other; none of this was planned. Jigar Kumar does not appear anywhere else on the internet apart from a few mailing list exchanges where he pushes Lasse Collin to give up the project to another person. Jia Tan does not appear anywhere else on the internet apart from small commits to xz about 2 years ago in order to build credibility and gain trust.

I’ll skip some other technical milestones in the timeline. You can read a detailed explanation by Russ Cox.

The Backdoor

The TL;DR; version is that Jia Tan goes on to create a very sophisticated backdoor that hides inside compressed xz (irony?) files that are, presumably, used for testing. This is smart, because nobody would suspect a compressed file inside tests/ folder to be malicious. In fact, it’s common practice to have mock data for tests, and these things are rarely checked during PRs, as we assume they are just test data.

The backdoor, which is hidden in multiple compressed files, taps into the build process of the xz library—essentially replaces the RSA_public_decrypt function symbol with a malicious function that gives an attacker, who possess a particular private key, remote execution capabilities.

This means that when SSH server is launching, it tries to load a dynamic library and looks for RSA_public_descrypt function, which is a legit function. However, if the distributed xz library is malicious, then it modifies the way the dynamic library resolves RSA_public_descrypt function, and instead it loads a malicious code. This code listens to a particular payload that is encrypted by a private key, that apparently only Jia Tan have, and using the embedded public key inside the code, it validates the payload (by checking that it was signed by Jia Tan’s private key)—and if so, it executes whatever is in the payload.

SSH runs as root. It has to, because it needs to de-escalate your privileges down to the particular user account that you log in with. This means that using remote code execution backdoor, one could bypass the entire account management and gain root access to a Linux machine that runs SSH. In case if you are not aware, this means millions, if not more, of machines all around the world. From web servers, to networking equipment, industrial equipment management systems (manufacturing, power plants, military and civilian objects), personal and home appliances, etc. Everything that runs SSH using systemd, on a rolling release Linux distribution, is vulnerable.

The backdoor was assigned the CVE-2024-3094 identifier, and has the base score of 10.0 CRITICAL, the highest possible severity.

Discovery

The backdoor was, accidentally, discovered by Microsoft employee and PostgerSQL contributor Andres Freund, when he was benchmarking PostgreSQL and noticed that his SSH connection took a hundred milliseconds longer and used more CPU cycles. He first thought that it has to do with a compromised Debian package, but later found out that the entire upstream xz repository has been compromised. If he hadn’t discovered it, there is a good chance that this vulnerability could have been reached major Linux distributions, and even production servers and systems.

There are discoveries now that Jia Tan was in talks with major Linux distributions to push his malicious version of xz. One such example, is this bug request for Ubuntu to update to xz to 5.6.1.

Andres shared his finding in an Openwall mailing list.

Who is behind the backdoor?

As of the time of publishing this article, there are no evidence that Jia Tan/Jigar Kumar are real individuals, and the consensus in the security community, is that a state actor is behind this attack. It seems like this backdoor was at least 2 years in the making, carefully planned and executed. Neither Jia Tan nor Jigar Kumar exist outside the few posts on mailing list of xz. Jia Tan’s role was to write the backdoor, while Jigar Kumar’s role was to pressure Lasse Collin to pass xz to another maintainer.

While this attack could have been carried out by an individual, or two, the fact that it was almost three years in the making, and the level of sophistication, points out to either an organized group, or a state actor. I could name at least 6 state actors who have the ability and the desire to plan and execute such attack, given the geopolitical situation in the past 3 years. I will, however, refrain from doing so, as the situation is still unfolding.

What’s next?

I want to come back to XKCD #2347. It clearly illustrates the modern digital infrastructure and the role that OSS plays in it.

We tend to forget that, but most of the modern digital infrastructure is run on the backs of hobby projects that few people maintain in their free time. These people are prone to losing interest, burn out, and other personal problems. And they are, like everyone else, prone to social engineering.

It’s unfortunate that Lasse Collin found himself in that situation. At one point, his GitHub account was even suspended (but it seems like it was reinstated few days ago). I don’t think he had any malicious intents, and he was just a guy maintaining a fairly popular open source library.

But we also need to remember that while the world became a safer place, there are still malicious actors in it. It is impossible to audit every single line of code out there.

Time will tell how we will move forward with OSS and our digital infrastructure, but as of today, we are thanking Andres who cares about CPU cycles, even in a world where everyone seems to solve their performance problems with “just buy more RAM”, and “let’s spin another EC2 instance”.

The Curse of the Senior Software Engineer

Thu, 14 Mar 2024 18:00:00 GMT

I started my software engineering career more than 14 years ago. Back then I was doing some SEO script automation using PHP. Slowly, I integrated better into the company and started to work on more core features of the product, and eventually even leading my own product.

I worked 6 years in that company. While working there, the company grew by approximately 3 people. This is far from your average hyper growth startup. We had no titles, no promotions, no performance reviews. I was happy because I was doing software engineering and calling myself a software engineer. It’s only when I needed to revamp my CV for a new job search, I had to look back at these 6 years and try to put the “correct” titles to each period of time:

Web developer at the beginning
Senior software engineer afterward
Tech lead when I was leading my own product and a tiny team

And only when I joined Autodesk, an American corporate, I was exposed to titles, career ladder and promotions. And I learned that many companies, both small and big, have titles and career ladders.

What’s up with the titles?

There are multiple reasons why companies are obsessed with titles.

Firstly, titles help normalize pay range. You don’t want an engineer with 2 years of experience to be paid the same as one with 6 years of experience. So you create titles. The first one will be Junior, and the second one will be Senior. The pay range for junior is X and for senior is Y. Often times, companies will have multiple internal titles behind one external title. For example a Senior Software Engineer as an external title, could be translated to either Sr.SW1 or Sr.SW2 based on different factors. This allows for further fragmentation of the senior scale. Instead of putting everyone who has between 5 and 10 years of experience into one big bucket named “Senior”, you would split it between Sr.SW1 who has 5 to, say, 7 years of experience; and Sr.SW2 who has between 7 and 10. On the outside, both of them would call themselves Senior Software Engineer.

Second reason companies have titles is to allow for progression and evaluation. Titles are usually attached to a set of roles and responsibilities. A junior engineer might be someone who requires assistance and closer supervision. The assistance and supervision is usually provided by a senior engineer, thus becoming senior requires that you mentor someone less experienced than you. Other functions might include things like reach and scope. A junior engineers work output will impact only his assigned task. Senior engineer, on the other hand, might improve something in system as he/she works on the assigned task. A staff engineer is someone who improves something that affects not only his team, but other teams as well. Examples could be things like introducing a new monitoring tool that is adopted by multiple teams; improving some workflow; etc.

When the time comes to conduct performance review, your manager will look at your impact inside the team and organization, and based on that will decide whether to promote you to the next level. With promotion also comes the raise in pay, since levels are tied to pay ranges.

And when you are interviewing for a company, even though they interview you to an external title like “Senior Software Engineer”, internally they will assign you the company level title based on things like: your compensation, years of experience, how good you passed the interview, etc. There is a website called levels.fyi that allows you to look into different career ladders in different companies, and how they compare.

The promotion dilemma

Often times, failure to get promoted will lead to either you being put on a performance improvement process, or getting fired. If you look at a career of engineers who stayed for long time in one company, you will see progress. It could be junior to senior to tech lead. Or it could be senior to staff to engineering manager. It all depends on the route in which the engineer want to grow.

What you won’t see is someone who is stuck in one position for a very long time, usually more than 2–3 years. Instead, these people would move to a different company which sometimes come with a title improvement. And because title is tied to salary, this also means salary improvement. This is why they say that it’s better to go to a different company if you want a pay raise and/or title improvement. Some companies are either slow, or fail to promote you, while other companies would be happy to do so, based on your years of experience and how well you pass the interview.

But not everyone is a job hopper, and some people do get stuck in the same company and same title due to laziness, comfort, or other reasons. These people, unfortunately, are prone to a curse that I call “The Curse of the Senior Software Engineer”.

The Curse of the Senior Software Engineer

Before I continue to talk about the curse, and how to avoid it, I want to clarify why it applies mainly to senior engineers. And seniority, in this case, is referring to reach and scope based on company’s career ladder rather than years of service. If we look at an engineer’s career, we can split it in two (at least from my experience): Individual Contribution (known as IC) and Leadership. Everything up to, and including Senior level, falls under IC definition.

ICs are people who are doing their own work. They close tasks, fix bugs, carry team duties like on-call schedules. They might improve the product or the code base, bring their ideas, mentor less experienced engineers, voice concerns and propose resolutions. But their scope is, mainly, limited to the individual/team level. They do not have any cross team or organization level impact.

The moment you cross the boundaries of your own team, you are moving to a Leadership position. In a healthy organization, Leadership positions are broken into two: Technological Leadership and People Leadership. Technological leadership includes titles like Staff Engineer, Tech Lead, Principal Engineer, Architect, etc. People leadership includes titles like Team Lead, Engineering Manager, Group Lead, etc. You need to reach Senior title in order to be promoted to leadership position. If you see someone who was promoted to a leadership position from a junior title, this could indicate either unhealthy organization, or incorrect initial title assignment. Whether you will be promoted to technical leadership or people leadership, is determined by your desire and the career path you have built with your manager.

But what happens if you reach a Senior level and get stuck there? There are many reasons to get stuck in a certain level, but they all boil down to either of the two: you fail, or the company fucks up. You, failing, could be lack of ambition to fight for promotion; lack of desire to be promoted; being comfortable where you are; etc. A company fuck up might include things like lack of proper promotion protocols; frequent organizational changes which interrupt promotion cycles; in-ability to properly identify who to promote; etc.

Oh, and there is always a personal factor. Companies don’t like to discuss it, but we are all humans. It could be that your manager simply does not like you, and so he/she will try to do their best not to promote you. This could be a pure humane dislike (they don’t like the way you look, the way you talk, the way you dress, etc.); or it could be due to their own insecurities such as being afraid to become redundant or replaced by you, who is super successful and eager to conquer the world. And finally, some people are just bad managers. Companies should have protocols to avoid this from happening, but hey, we are all humans, and sometimes we fuck up.

So you stay Senior for too long, and you get cursed by the Curse of the Senior Software Engineer. The curse goes like this:

The Curse of The Senior Software Engineer

A cursed Senior Software Engineer, is an engineer who has been Senior for too long that upon his or her next job search, they find themselves in between two worlds:

They are too Senior to be hired as Senior (due to years of experience or salary expectations)
They have no Leadership experience, or very little of it, to be hired on a Leadership position (tech lead/staff/principal for tech; team lead/engineering manager for people)

And so they are cursed to roam the Senior valley, and have no ability to enter the tech market.

Stay senior for too long, and there is a high chance that you will get this curse. How do I know? Because I judge based on my experience. I’m a Senior engineer with 14+ years of experience, and was never interested in titles, nor practiced any job hopping for the sake of title improvement. I like what I do, and I think I’m good at what I do. But I never felt like playing the corporate game and climbing the career ladder. I spent the first half of my career in a-typical companies, without any corporate BS, and I can attest that I was successful in what I did.

However, as I found myself burnt out, I left the corporate world in pursuit of building a sustainable online business, with the idea to never return to a traditional workplace. I have failed however, and in the middle of my 6-month sabbatical, I started to look for a job. After applying to 100+ positions, I got a bunch of generic rejections. From those who did not reject me, some told me that I’m overqualified for Senior roles, but under qualified for Leadership positions. I had to make assumptions as to why I’m getting rejections. I could have blamed the 2023 tech-recession, but I like to blame myself, so I had to do some self-analysis.

And while it could be the 2023 tech-recession, if there is a slight chance that my theory is correct, I would like to spread it and inform other engineers to not make the same mistakes as I did.

How to avoid the curse?

There are two ways: you either play by the rules or you exit the game.

You have to play by the corporate rules and become obsessed with the corporate career ladder. You need to follow your promotion cycles and make sure you are not stuck at one level for too long. If you are, you need to stay proactive and consider quitting and moving to a different company. This move should include a title improvement.

It sounds cruel, I know. But this is your career. Companies, in general, won’t care too much about you. By the time you will naturally grow for promotion to next level, your manager might change, or the entire company might get restructured. And while other people, who were more pushy and demanding than you, will get promoted—you will stay behind, slowly acquiring the Curse of the Senior Software Engineer.

The other option is to exit the game completely. Remember, FAANG or MANGA, whatever the acronym right now is—dictate the way the industry operates. If they have titles and career ladder, other hype driven, VC funded startups will follow. The only way to exit the game, is to find a company that is not like them. A company that does not care about titles or have performance review rituals. It’s harder, but companies like this do exist. This is what I did. I just found a local startup company (well, in fact, they found me) that does not care about titles and other corporate rituals. I don’t have daily standups or monthly retrospectives. We just pull tickets from the backlog and do them.

Other companies you might consider are big-n-old companies like banks, railway operators, hospitals, etc. This also includes IT departments in boring standard companies. Remember, almost every company needs programmers nowadays, but most of them won’t take you to parties in Hawaii, have a hipster office in the coolest building in town with a bearded barista, and attach you the “Full Stack Ninja Squad Leader Engineer” BS title. This also means they won’t pay you the same money as FAMANGA and Co, and you won’t work with the latest and greatest tech (although exceptions exist).

Another option you might consider, is switching to self-employment either through freelancing or contracting. Nobody cares what your title is, or how many years of experience you have, as long as you can deliver the work in time at a reasonable price.

There is no better or worse decision. While I don’t see myself coming back to traditional corporate environment (unless forced to), I can understand why people would want to work in Meta or Google and Co. And if you do, remember that there are certain rules you need to play by. It’s not enough to “just write code”.

Serving Astro with Rust

Sun, 10 Mar 2024 15:00:00 GMT

As some of you might know, for the past few months, my free time is spent on JustFax Online. I also published few blog posts about this project: here, here, and here. Let’s try to milk this project some more, this time with Rust and Astro.

The entire website for JustFax is was fully server rendered. Like in the old times. You would hit a URL, and the server would respond with an HTML. Interactivity was handled by htmx and a bunch of vanilla JavaScript.

The problems with server side rendering

Templates are hard to maintain

Firstly, let’s not confuse Server Side Rendering (SSR) and Server Side Generation (SSG). The former renders the HTML on demand per request, while the latter generates HTML files beforehand to be served by a server.

I was using the tera templating engine. Tera works with a format similar to Jinja. Overall, the syntax is nice, and tera is great. I was even able to implement hot reloading of templates, so I could run the rust server and edit the HTML, without the need to restart the server.

However, as the project grew bigger, it became harder to maintain the HTML templates and, more importantly, the context I would pass to the templates. Templates need variables. Variables are passed from Rust to the tera engine by means of serialization (if I’m not mistaken via serde_json). In reality, they are completely detached from each other. And this brings me to the first issue of server side rendering: errors in rendering the templates are run time.

This means that if I misspelled a variable, the template would fail to render. Refactoring code, and renaming variables—became a nightmare. I needed to check each page manually before committing to deploying a new version. I was considering switching to askama, which is a type-safe, Jinja-like, compiled template engine. But eventually decided to generate all the HTML beforehand.

In addition to that, Jinja, or at least the way it’s implemented in tera, is very limiting. I would end up defining variables that are used purely for HTML purposes, in Rust, because I wasn’t able to achieve the same inside a template. Examples include scenarios like: converting a vector of structs into JSON; and various translation shenanigans. This is in no way a criticism towards tera; tera is a great templating engine, and I would definitely use it again in the future (if askama won’t prove to be better). I never worked with Jinja or Django, so I assume these might be limitations of the templating language itself, rather than the tool.

Oh, and one last annoyance. Jinja is poorly supported in vim. So I had to fight the editor with stupid errors. One great example is from this piece of code:

{% if country is defined and country is string -%}
const INITIAL_COUNTRY = "{{country}}";
{% else -%}
    {% if LANG == "en" -%}
      const INITIAL_COUNTRY = "us";
    {% elif LANG == "de" -%}
      const INITIAL_COUNTRY = "de";
    {% elif LANG == "fr" -%}
      const INITIAL_COUNTRY = "fr";
    {% endif %}
{% endif -%}

The LSP would yell at me for trying to re-declare INITIAL_COUNTRY.

Everything is a run-time problem

With server side templates, everything becomes a run-time problem. Forgot to declare a variable? Run time errors. Extending a base template that does not exist? Run time errors.

On a small website with 2–3 pages, it’s easy to check each page manually. But at some point it becomes too complicated, and I wish I could outsource this to some build step. Maybe compiled templates like askama would help here, but I haven’t tried them.

In addition to that, maintaining SSR requires a lot of tools. Rust is amazing in run-time, but not that in compile-time. Removing tera and other crates that I used that helps with template rendering—improved the compile time of the project. Previously, it would take about 4 minutes to compile the rust code in CI. After removing tera, markdown related crates, and other crates used for formatting (for example phone number formatting)—I was able to reduce the compilation time to 3 minutes, which might not sound like a lot, but this is a great improvement.

In the end, if I look at it, the HTML was completely static. There was no need to re-render it for each request, and the dynamic parts were small enough to be able to generate them in the browser without incurring any performance penalty. I also learned the hard way that Rust is way behind JavaScript in terms of l10n and i18n. While it has great support for translations (I personally like project fluent by Mozilla)—it has no crates to format dates/numbers/currencies. There are crates like icu that are in development, that essentially tap into the Common Locale Data Repository (I wrote about CLDR back in 2015). This would allow Rust to have the same capabilities as JavaScript has through the Intl API, but currently it’s still a work-in-progress.

And so, I decided to turn to my second favorite tool.

Hello Astro-naut

I’m a big fan of Astro, and in fact this blog is powered by Astro. Astro, unlike other frameworks, does not force you to use a particular frontend framework, you can use vanilla JavaScript or TypeScript, or you can mix them all together. It’s fast enough to generate hundreds of pages. It comes with a lot of tools for websites like automatic sitemap generation, assets optimization and bundling, etc.

Most of the issues I described above are also solved by Astro. Template data and representation, are connected, especially if you opt for TypeScript. It will be impossible to generate a page that includes a non-existing Astro component, for example. Astro comes with superb support for Markdown content collection, which makes it super easy to generate content. When I was using Rust, I had to sort-of reimplement what Astro does. It didn’t work well.

L10n and i18n are also solved with Astro. You can pick any translation library you want, including project fluent from Mozilla. And you have access to Intl for formatting numbers, dates, and currencies.

The only problem with Astro is that it’s a static site generator. Wait? What?! Yes, I know I said that I wanted to switch to SSG from SSR, how come it’s a downside?

Well, as I described in this blog post—Web app localization in Rust—I need a server in order to do locale negotiation. Before rendering the initial HTML, I need to query cookies and the Accept-Language header, and determine the preferred language of the visitor. This can be done with JavaScript, for example by using the window.navigator.languages property (which MDN claims to be the same values as Accept-Language header). However, this would harm usability, as Google won’t be able to index the website correctly.

Astro, does, however take care of this as well. If you decide to opt for hybrid or server output for your Astro website, you will gain access to Astro.preferredLocale and Astro.preferredLocaleList properties (as describe in the Internalization article from Astro Docs). But opting in for hybrid or server outputs creates a problem. You are now locked into either using Netlify or Vercel edge functions (which has official Astro adapters), or go back to Node. Neither of them I wanted to do. I have my locale negotiation code in Rust, I want it to stay in Rust. In addition to that, the locale negotiation by Astro does not take into account the cookie that I set in order to avoid sending an English speaking German person to the German version of the website, even though he insisted for English interface, unlike Google is doing.

And so, I was left with the only option, serve Astro website with Rust.

Serving Astro website with Rust

Astro generates, by default, your entire website structure in the dist folder. HTML pages as well as additional resources will be there. All is left, is just to expose this folder to the web.

One approach I wanted to use, is to serve the website by Caddy. I already use Caddy as reverse-proxy for the Rust server, and Caddy is capable of serving folders. However, figuring out the correct handlers order for Caddy—made my cabesa hurt. You see, theoretically I could point / to / on Rust server; /api to /api on Rust server; /en to /dist/en; /de to /dist/de; etc. But what do I do with static resources like JavaScript, sitemap, robots.txt, etc? You see, it makes my head hurt. So this option was out of the window.

Let’s now talk about Rust. I use axum as my web framework. Axum is capable of serving directories using the ServeDir service. You give it a path to a directory, and it will serve the files from there. So I have a regular axum route that handles the / url; route that handle /api url(s); and everything else is fallen back to ServeDir. This covers 99% of the files. The last 1% are 404 files. Astro will allow you to define 404.astro for a custom 404 page, and services like Netlify or Vercel will be able to route to it, but not axum.

If axum fails to find a matching route for the request, nor it finds a static file inside the provided ServeDir path, you are able to implement a custom 404 handler.

let fallback_service = ServeDir::new(public_path)
    .append_index_html_on_directories(true)
    .not_found_service(Handler::with_state(handle_404, context.clone()));

Inside handle_404 you can do all kinds of things. I do locale negotiation to decide which version of the 404 page to serve, but eventually it boils down to

match ServeFile::new(path.as_path())
    .try_call(Request::new(axum::body::Body::empty()))
    .await
{
    Ok(res) => res.into_response(),
    Err(e) => (StatusCode::NOT_FOUND).into_response()
}

While path is the path to [en|de]/404/index.html.

Another thing with Astro, is that it has a great asset management pipeline. Each static resource will be processed by Astro and a hashed name file will be put it _astro directory. Since the hash of the file represents the hashed content, if the file did not change, it means that it will have the same hash. And this means that _astro is a great directory to cache aggressively.

I had to fight a bit with axum in order to understand how to send headers alongside ServeDir file, but eventually I was able to figure it out.

let public_path = Path::new(&config.server.www_dir);
let fallback_service = ServeDir::new(public_path)
    .append_index_html_on_directories(true)
    .not_found_service(handle_404.into_service());

Router::new()
    .fallback(get(|req: Request| async move {
        let (mut parts, body) = req.into_parts();
        let uri: OriginalUri = parts.extract().await?;

        let req = Request::from_parts(parts, body);
        match fallback_service.oneshot(req).await {
            Ok(mut res) => match res.status() {
                StatusCode::OK => {
                    if uri.path().contains("/_astro/")
                    {
                        res.headers_mut().insert(
                            "Cache-Control",
                            "public, max-age=31536000".parse().unwrap(),
                        );
                    }
                    Ok(res)
                }
                _ => Ok(res),
            },
            Err(e) => {
                tracing::error!("fallback_service error: {}", e);
                Err(e)
            }
        }
    }))
    .layer(service_layer)

This essentially uses the ServeDir service through the oneshot request, and based on the request uri it sets the cache headers for _astro folder.

That’s it. That’s the blog post.

Rendering emails with Svelte

Sat, 24 Feb 2024 11:00:00 GMT

Emails play an important role in my project JustFax Online. Before I implemented emails, I would spend time answering customers’ requests for fax status updates. This means that I had to monitor each fax transmission (and some can take up to 30 minutes) and send an email when it succeeded or failed. And so the obvious solution was to implement email notifications.

I hacked something together using MJML, handlebars, fluent for localization and Postmark. Postmark is a great service by the way. I’m not affiliated with them, but they have superb customer support. I tweeted this:

My wish for the new year, is that @postmarkapp will approve my account today 🙃

And then, I’ll be able to ship something cool!
— Dmitry Kudryavtsev (@skwee357) December 29, 2023

And after 7 minutes they approved my account. Highly recommend! But let’s talk about emails.

Tables. Tables everywhere.

If you ever had a chance to design emails, you know that it’s one of the most terrible jobs to do. Emails are still designed using HTML tables, and getting them right on every client—is an impossible task. Luckily, there are tools like MJML which provide a DSL similar to HTML in order to design your emails. With compilers available for all major languages, it’s a great solution for responsive emails. Of course for the hip-and-cool there are React.email, VueEmail, and Svelte Email. But I chose MJML.

In addition to MJML, I used handlebars as a lightweight templating language. Emails rarely need complex logic, so handlebars with its simple syntax was a perfect choice. JustFax is available currently in 3 languages: English, German, and French. This means that emails need to be available in the same languages. I use Fluent by Mozilla as my i18n framework through a set of crates for Rust: the low level fluent, and the higher level fluent-templates which was designed specifically to be used in template engines like tera and handlebars.

MJML files would sit in the same monorepo as the rest of the code, and through the MJML npm package, they will be compiled to HTML in CI/CD. In runtime, Rust will load the HTML files, initialize Fluent and handlebars, and upon successful (or failed) fax delivery—would send the correct email. This was good enough for the MVP, but it had some problems I wanted to improve on.

Preview. Previewing emails was hard. While the MJML tool comes with an online app (and a one you can download), frequent readers of this blog know that I’m stuck in Vim and I won’t use any other editor. But even if you decide to use MJMLs own tool, or the live editor, you still can’t preview handlebars templates. So the entire process of previewing your emails would turn into a nightmare of copy-pasting stuff around.

Modification. Email modification is another thing that was hard. Because there was no proper preview, modifying the emails became a risky task. I would either break some variable, misspell a translation key, or would run into handlebars compilation errors, which happened in runtime.

Armed with the desire to improve these two problems, I started to look for solutions.

Attempt #01—compiled templates for Rust

Rust has this notion of compiled templates. There are two popular crates for this: askama and yarte. The former is for lovers of Jinja syntax, while the latter is better suited for handlebars/mustache enthusiasts. The idea behind compiled templates is that through macro programming, they will analyze the template code and extract all the needed variables to a generated struct. This means that you can’t accidentally forget to specify a variable or mistype it. If you do, your code won’t compile.

I tried yarte, but it was somewhat slow, and I got many errors with it. On top of it, while yarte solves the type-safety aspect of templates, it does not compile MJML to html, or allow me to preview my templates. And after some brainstorming, and inspired by the recent change I made for the frontend (article is coming, drop a follow on Twitter or Mastodon, or subscribe to the newsletter—to make sure you won’t miss it), I asked my self “what would happen if I try to generate emails statically, like with static site generators?”.

Attempt #02—statically generated email templates

What if we take the same principles behind static site generators like Astro or SvelteKit, and use them to generate emails?

Vite

My first attempt was using Vite. If you put all your templates in a Vite project, add some plugins—namely vite-plugin-mjml and vite-plugin-handlebars—you can achieve an amazing result!

Inside your Vite config, you can define the following:

export default defineConfig({
  plugins: [
    mjml({
      input: "src",
      output: "emails",
      extension: ".html",
      watch: true,
    }),
    handlebars(),
  ],
});

This will take all files inside src directory, pass them through MJML compiler and put the result with .html extension inside emails directory. It will also watch for changes, so if you change the template, you will get hot reloading. Navigating then to localhos:4321/emails/template_name.html will render the compiled MJML in the browser. Not bad! But if you actually try this, you will see that handlebars variables are all empty.

We can do better. Using the handlebars plugin, we can define pageData:

const pageData = {
  "/emails/template_name.html": {
    someVar: "someValue",
    foo: "bar"
  }
};

---
handlebars({
  context(path) {
    return pageData[path];
  }
})

If we go to the same path again now, we will see that our handlebars variables are properly replaced with the provided data from pageData! Specifying strict: true inside the handlebars plugin, will allow us to see errors about any missing variables from the template.

This is good enough for most use cases. But not for me! I wanted more.

Ability to view the email in different languages
Ability to modify the email data
Ability to preview email in desktop and mobile

And so, I embarked on my next journey—SvelteKit.

SvelteKit

Everything I described above requires a bit more than just static HTML served in a browser. It requires some sort of server and logic. And SvelteKit allows for that. I’ll first put a small gif of the resulting product, and then I’ll walk you through the important steps of how to achieve this.

Email preview with MJML, handlebars, and SvelteKit

Since SvelteKit is based on Vite, the build process remains as described above with the only exception that we no longer need the handlebars plugins. Having the MJML plugin will allow the build process to compile MJML into HTML, which I then can render in Rust with handlebars. During CI/CD, the resulting directory that contains the HTML of the emails, is simply copied into the Docker container, and read by Rust upon startup. However, in order to achieve the preview and navigation, we will have to implement some routes.

We will need two route: ’/’ and ‘/render’ with the last being server only route.

Main route ’/’

The purpose of this route is mainly UI. It should display a list of email templates, and allow the user to select a particular template. It consists of two files.

+page.server.ts

import type { PageServerLoad } from "./$types";

export const load: PageServerLoad = async () => {
  const files = import.meta.glob("../../mjml/*.mjml", { query: "raw" });
  const fileNames = Object.keys(files).map((name) =>
    name.replaceAll("../../mjml/", ""),
  );
  return { emails: fileNames };
};

There is nothing special going on there. We just import all the *.mjml files, clean their names, and return them to the page.

The other file is page.svelte, but I won’t include it here since it’s mostly UI. The important bit is to declare data import export let data: PageData; so that the result of the load function will be available to the page. After that, just render a button for each email template:

{#each data.emails as email}
    <button on:click={() => sendRequest(email, {})}
      class={`flex items-center font-medium gap-2 justify-between w-full my-1 borer border-gray-400 rounded-md hover:bg-gray-200 px-4 py-2 ${activeEmail === email ? 'bg-gray-200' : ''}`}>
        {email}
    </button>
{/each}

You might have noticed the sendRequest callback, let’s talk about it.

function sendRequest(file: string, context: object) {
  activeEmail = file;
  fetch("/render", {
    method: "POST",
    body: JSON.stringify({
      file,
      context,
    }),
  }).then((res) =>
    res.json().then((json) => {
      html = json.html;
      content = { json: json.context };
      validator = createAjvValidator({ schema: json.schema });
      updateFrameContent();
    }),
  );
}

Upon selection of the email template, we will issue a POST request to /render with the name of the template. The request is the name of the file, and a context object which will be used to render the template by handlebars. Initially, the context object is empty.

The response is a JSON that contains the parsed and rendered HTML; an updated context object; a JSON schema schema; and errors object that MJML produced. I ended up not using the errors, as I achieved what I needed, but in theory you can create a nice validation to inform the frontend that your MJML template is broken.

Render route `/render`

The render endpoint is purely server side, and resides inside a /render/+server.ts file. I’ll dump the code first, and then we will discuss the important parts.

import { type RequestHandler } from "@sveltejs/kit";
import fs from "node:fs";
import mjml2html from "mjml";
import Handlebars from "handlebars";
import { getTranslationFunction } from "../../i18n";
import Ajv, { type AnySchema } from "ajv";

interface RequestBody {
  file: string;
  context?: Record<string, string | number>;
}

const ajv = new Ajv({ useDefaults: true });

export const POST: RequestHandler = async ({ request }) => {
  const body: RequestBody = await request.json();
  const mjml = fs.readFileSync(`./mjml/${body.file}`);
  const res = mjml2html(mjml.toString());

  const schema = JSON.parse(
    fs
      .readFileSync(`./schemas/${body.file.replace(".mjml", ".json")}`)
      .toString(),
  );
  const validate = ajv.compile(schema as unknown as AnySchema);
  const context = body.context ?? {};
  if (!validate(context)) {
    return new Response(JSON.stringify({ errors: validate.errors }), {
      status: 400,
      headers: {
        "Content-Type": "application/json",
      },
    });
  }

  const t = await getTranslationFunction(context.lang?.toString() ?? "en");
  Handlebars.registerHelper("fluent", function (key) {
    return t(key);
  });
  const tpl = Handlebars.compile(res.html, {});
  const output = tpl(context);
  return new Response(
    JSON.stringify({
      html: output,
      schema,
      context,
      errors: res.errors,
    }),
    {
      status: 200,
      headers: {
        "Content-Type": "application/json",
      },
    },
  );
};

The code accepts a JSON object with file and context. file is a path to a template file inside the mjml folder (or any other folder name you decide to store your MJML files in). context is an object to render handlebars with.

The code first reads the template and parses the MJML into HTML. It then reads a json file from schemas folder that have the same name as the template file. This JSON file contains a JSON schema. This is the cool part, because this file acts as both documentation and for initial preview purposes. I can use this JSON schema as a reference in order to define the corresponding Rust struct, or if I want to take it further, I can add a compile step that will generate the struct for me during build based on the schema. At the moment, I do it manually.

Next, I compile the JSON schema and validate it. Since ajv is created with useDefaults: true, and all fields in the schema will have default values, this gives me the initial context object when I send {} in the first request. I then get my t() function which is used for translation, and bound to the context.lang or en. Since returning the t() requires promises to load the .ftl files and initialize FluentBundle, I had to redeclare handlebars fluent helper for each request, as handlebars does not support asynchronous helpers. It’s an MVP in the end ;)

Lastly, I compile the template with handlebars and send back the response, together with the schema, the updated context, and any errors from MJML.

Tying it up together

I won’t bore you with the frontend code, but I do want to mention some tools I used.

The emails’ HTML is rendered inside an iframe. Every time there is a new response, I call this function which updates the iframe content:

function updateFrameContent() {
  if (html !== null) {
    frame.src = "about:blank";
    frame.contentWindow?.document.open();
    frame.contentWindow?.document.write(html);
    frame.contentWindow?.document.close();
  }
}

frame is a variable that is bound to the iframe: <iframe class="w-full bg-white h-full mx-auto" title="email" bind:this={frame}></iframe>.

For the context preview and modification, I used svelte-jsoneditor packages. It has a lot of features, and even able to validate the JSON using JSON schema, which ties everything together nicely. I listen to onChange event of the svelte-jsoneditor, and upon every correct JSON, I reissue the request:

onChange={(c) => {
    if ('json' in c && typeof c.json === 'object' && c.json !== null) {
        onContextChange(c.json);
    } else if ('text' in c) {
        try {
            const json = JSON.parse(c.text);
            onContextChange(json);
        } catch (e) {
            console.error(e);
        }
    }
}}

There are some UI things to handle resizing the iframe from full width to 360px for mobile view, and another button to replace the iframe with a Highlight component from svelte-highlight for the HTML source code.

Conclusion

I’m very happy with this solution! I’m able to preview the emails with all kinds of different values. It’s even possible to create dynamic svelte forms based on the JSON schema, so if your template can accept an enum, you can have a drop-down list of all the possible values.

I’m also able to view emails in different languages, something that I previously was able to do only after the emails were sent, using Postmark dashboard. The JSON schema acts as a documentation to create proper types in Rust backend, and can be further processed in order to automate types creation. The only downside of this approach is that during build, the entire boilerplate is being built as well. In my CI/CD pipeline I copy only the resulting templates’ HTML, but nevertheless I spend computing power to generate code I don’t need. I bet it’s possible to configure Vite with some option or flag in order to only run the MJML plugin, but I didn’t bother with it since this process runs in parallel to Rust compilation, and that takes way more time that generating the HTML templates and everything else.

Emails are a pain in the butt to work with. I’m happy that we have tools like MJML, and I’m glad to share my knowledge and, hopefully, help you improve your email workflow.

How to stay junior forever

Tue, 30 Jan 2024 19:00:00 GMT

In my last blog post, I mentioned briefly the fact that today there are more engineers than meaningful jobs for them. This means that there is an immense pressure on junior engineers. Getting a job becomes a challenge.

The other day I stumbled upon a post in /r/Programming. The post was talking about how GitHub Copilot affects code quality. You can read the post here. The only important quote from the post is this:

We find disconcerting trends for maintainability. Code churn — the percentage of lines that are reverted or updated less than two weeks after being authored — is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of ‘added code’ and ‘copy/pasted code’ is increasing in proportion to ‘updated,’ ‘deleted,’ and ‘moved ‘code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don’t repeat yourself] of the repos visited.

This reminded me of something.

Copy-paste

I remember when I joined Autodesk, back in 2017, I got the experience to work together with different kinds of engineers. Prior to Autodesk, I worked mostly in small teams with little to no supervision. But at Autodesk I had responsibilities such as mentoring other engineers, as well as conducting pair-programming sessions. These practices allowed me to take a glimpse into how other engineers work and write code.

And I noticed that many engineers write code in, what I call, a copy-paste method. They would copy blocks of their own code around, and modify the necessary parts. Moreover, they would copy code snippets from Stackoverflow or other online resources.

I don’t practice copy-paste programming. Don’t get me wrong, I do copy and paste parts of my own code, as well as code found on the internet. But I never do it for code that I don’t understand. And this made me wonder, am I just being inefficient? It took more than 6 years to finally find the answer.

The power of writing

Last year I published one of my all times popular blog posts—Why engineers should focus on writing. It even reached the top page of HackerNews! One particular advice, that was copied into multiple publications, and mentioned by multiple people on social networks, was:

One last advice–abolish the copy-paste. So many developers, whom I mentored, simply copy-paste everything. Code snippets, function declarations, etc. I know how to initialize a git repository, because I do it by hand every time. Most people simply copy the instructions from GitHub or Google. And if you are scared about being unproductive–remember that you are not judged by how much code you write, or how fast you complete the assignment.

There is a reason why writing is a skill and not a habit. Habit is something that you do repetitively. You can’t perfect a habit, you just stay consistent for as long as possible. Skill, on the other hand, is something you can perfect. And perfection of skills is achieved by doing them over and over again, with slight improvement over time.

Software engineering is a skill, and in order to perfect it you have to write more code. The keyword is write. Not copy. Not generate. Write.

It might sound counter-intuitive, because one could argue: why should I spend time typing, while I can just copy and paste? And it’s a valid argument, but copy-pasting code is the same as watching how to drive a race car—it won’t make you into a race car driver. I know the things I know because I keep writing them over and over again. That’s why I hate all these gp and other shortcuts that people make for git commands. I type git push every time.

Not only it engraves these commands into my brain, so I won’t have to Google them every time, it also does something with the way I understand things. I can’t explain in, and it frustrates me because I’m a logical person, but writing is magical. Any type of writing. This is why students are asked to take notes because that’s how you learn. And that’s why good engineers write. Code, and blogs, and articles. The first advice we ever give to anyone who is struggling to solve a problem is: write it down, draw it on the whiteboard.

And this leads me to the main topic of this post.

How to stay junior forever

If you want to stay junior forever, keep using AI and other code assisting tools.

It’s a bold statement, but I’m willing to stand behind it. The more you use AI, Copilots, and other helpers alike—the less of an engineer you become, and the easier it is to replace you with AI. Software engineering is not only writing code. In fact, software engineers is everything except writing code. Writing code is the easiest part. Once you know 2–3 families of programming languages, all the rest are the same, just different syntax.

However, translating requirements into code, and more importantly understanding existing code—this is what software engineering is about. Writing code is the mechanical part of your hands. But in order to write it, you need a pre-trained brain that can translate the problem into code.

Now, it’s not that I’m against AI and code assisting tools. In the end, we all went up from using Notepad to using auto-completion and intelli-sense. But none of the pre-AI tools are crucial. They help me to remember what are the arguments for Array.slice function, but they do not replace my ability to understand what a given piece of code does.

With copy-paste and AI-like tools—it becomes too easy to abuse it. The best way to avoid eating sweets, is not to keep sweets at home. We are all humans, and it’s tempting to abuse what makes our life easier or brings us pleasure. Seneca used to say:

This is what I mean: pleasure, unless it has been kept within bounds, tends to rush headlong into the abyss of sorrow.

But the thing is, like with eating a piece of chocolate a day, you will pay for it in the future. And instead of becoming an engineer to whom colleagues will come for advice—you will be replaced by the same tool you adored, because you practiced the habit of typing, rather than mastering the skill of software engineering.

Farewell.

The Bloat in Software Engineering

Tue, 23 Jan 2024 18:00:00 GMT

I’ve been thinking recently about two things: The Roman Empire and the unnecessary bloat in Software Engineering. You see, about 7 months ago I quit my job due to burnout. Since then, I spent a lot of time thinking and analyzing what is so wrong with our industry. I wrote once about how our industry is broken, but that article was referring more to the technical side and code bloat. Today, I want to talk about the cultural and architectural sides.

Let’s build a Kafka queue with distributed microservices architecture

Seems like the microservices architecture hype is finally over. People are coming back to traditional monoliths, or not caring at all about labeling their architecture. But there still remains a trend that many follow. I call it “the MANGA trend”.

Inexperienced engineers, or engineers that are driven by ego/desire to be promoted (and sometimes boredom, I’ll talk about it later), come up with one solution for every problem. The “Kafka solution”. Kafka, in this case, is just a placeholder for “your favorite MANGA tech stack” (MANGA: Meta, Apple, Netflix, Google, Amazon—The holy quintet of software engineering).

What happens is that one of these “big tech” companies post on their super cool engineering blog, about a super hard problem they had, and how they solved it by building a reverse-proxy-on-top-of-a-potato-using-ipv6-over-telephone-wires, which is now open-source, and all inexperienced or ego driven engineers be like: “Wow! I want that too!”

Much wow. Such architecture

So they convince their fellow teammates, team leads, and engineering managers (using their superior writing skills which they acquired thanks to my recent book Technical Writing for Software Engineers)—that we, too, need to have such architecture. It’s good for Google, why would it be bad for us? Their plan could have been bulletproof, and would lead them to promotion, if only they didn’t forget one thing: they are not Google, and they have 12 MAU (monthly active users).

The desire to implement over-engineered solutions is caused by multiple factors. The need for social validation is one of them. We want to be as cool and as trendy as the other company. But we forget that we are not Google. We don’t have the same scale, and definitely not solving the same problem.

NOTE

It’s interesting because this trend is sort of catch-22. From the employee side, many engineers want to work in companies that work with the same tech as MANGA, so they would ask these question during the interviewing process. But from the employer side, the drive towards Cool Tech might be fueled by said potential employee, as well as internal desire inside the engineering team to adopt Cool Tech.

And then, you will meet the “anti-Donald” type of developers. In a paper titled “Structured programming with go to statements”, Donald E. Knuth famously wrote:

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

But then comes the “anti-Donald” developer who says:

We need to optimize for future. Our product is going to grow, and we need to support its future growth.

And while this statement by itself is not wrong (assuming it’s backed by actual data) there is another fallacy that comes attached to it. The fallacy that says that the existing tech we use—is not scalable. Which most of the time is wrong. The following statements will piss-off many engineers, so be warned:

Majority of the web could run on SQLite.
PostgreSQL can be used as a queue without the need to introduce a standalone broker.
And your bottleneck is not in your asynchronous processing technology. It is, in fact, in your managerial inefficiency, and technological bloat.

All aboard the hyper-growth train

There is a hard truth that needs to be said: there are more developers today than meaningful jobs for them. Here, I said it, you can cancel me now.

Companies are living organisms. And they have many of the same social interactions as humans have. One such interactions is mirroring.

When you, as a respecting and politically correct human being, find yourself in an environment of racists, you have two options:

Leave
Be racist (mirror your environment)

Yes, technically you have third option: to stand your ground and stay true to your principles. But com’on, I’m trying to make a point, so bear with me.

Companies rarely have the option to leave. They all compete on the same market, over the same talent (most of the time). And they all mirror each other. This is why when Google announces layoffs, all the other MANAs will follow it. And when one company releases an AI model, others will pivot to it as well. And finally, when one enters a hyper-growth mode, others will follow.

They do so for various reasons, which most of the time are unrelated to actually needing this talent for meaningful jobs. One reason is mimicking what others are doing. Another reason is to keep candidates off the market. Amazon would rather hire all the best AI engineers, so Nvidia wouldn’t get them, even though Amazon had no plans to release a general purpose LLM. But FOMO is real, and it’s better to insure myself then let a competitor get ahead.

This creates a weird situation where a company is now stuck with personnel which has no actual work to do. But nobody is getting paid to watch Instagram Reels all day, so we have to come up with some sort of work. And smart engineers, being smart, start to come up with over-engineered solutions. This is why we end up with Kafka infested development in every company.

In reality, it would be cheaper, and easier to implement Kafka when you are milking your resources to the max, instead of prematurely optimizing it. And the reasons being is…

Legacy

I wrote about legacy code in this blog post. And the truth is that legacy code is unavoidable in our modern engineering culture. Modern development practices prioritize endless feature list, while pushing technical backlog down to the abyss. But technical backlog is the building block of avoiding legacy code. You need to have constant refactoring sessions, and revisiting old code in order to avoid code rotting.

By the way, opposite scenario, where majority of the time is spent on refactoring, leads to another pandemic that I call “lets-rewrite-it-to-v2-using-new-cool-tech”. This also stalls healthy development and leads to over-engineered solutions that disregard any product features.

If instead of having a “premature-optimization” development cycles, we would concentrate on refactoring, you wouldn’t need to introduce Kafka “just-because”™. But due to the desire to do cool stuff, “like Google does”, paired with the fact that a lot of the engineers are over-hired—we end up with premature optimization. And in reality it always fails, because you are building for something you don’t have, hence you can’t know what do you need. It would be wiser to introduce a particular technology when you hit a bottleneck and understand what you need to fix.

The good news is that there seems to be a shift in the minds of developers and companies. I see more and more people sharing how they abolish their overly complicated AWS setups in order to go back to traditional VPS + DevOps setups. More engineers, myself included, are speaking about using simple technologies. The above-mentioned blog post by Adriano Caloiaro about using PostgreSQL as queue is a great example. The increased mentions of simple technologies such as SQLite and HTMX is another example. So there is hope.

I spent €200 on Google Ads so you won't have to

Thu, 11 Jan 2024 17:00:00 GMT

If you remember, in one of my past blog posts, I was talking about the fact that I want to try running ads for JustFax Online. In general, I don’t like ads. Something in the advertisement industry repels me. However, as I mentioned in that blog post, for some products running an ad campaign can actually benefit. And so I spent €200 on Google Ads, so you won’t have to. Let’s see what happened.

Segment properly

The first time I ran the campaign, I did a mess. I targeted multiple countries, with multiple speaking languages. This led to a lot of irrelevant clicks.

I wanted to target the US region, as well as some non-English-speaking countries. So I created one campaign, in English, and set the target audience to US, and few other countries who speak English and non-English. This turned out to be a bad decision as it’s preferably better to segment the audience with different campaigns. Not only it lowers the cost (more on that in a moment), but also generates clicks that are more likely to convert.

Second thing I didn’t take into account, is that the same keyword, but with different target audience—will have a different cost. This is why it’s important to segment properly.

Think about acquisition cost

When users started to flow, I got so happy! I had orders, made refunds, improved the system. But I forgot one small thing—I pay more per click than I make from converted customers.

This motivated me to try and lower the cost as much as possible. Since I don’t operate on a recurring subscription model, this means that I either need to have a very tight budget, or explore other channels of customer acquisition. Speaking about budget…

On daily budget

I’m not entirely sure how the whole advertisement system works, but from what I understand you bid for ad placement. This means that Google, and other ad networks Google work with, will determine in real time the highest bidder for a particular keyword/search term. If you win the bid, your ad will be placed. This means that your daily budget is not set in stone, and your CPC (cost-per-click) is not constant.

For some days you will pay more than your daily budget, sometimes up to 2x more, while on other days you won’t reach your budget. Same goes for CPC. Some clicks will be more expensive than others.

What Google does guarantee is that you won’t pay more than your daily budget times 30 days, per each month. So if your daily budget is €10, you won’t pay more than €300 per month. But it’s not evenly spread among the days. Some days you might get more traffic and pay more, while other days you get less traffic and pay less. So just keep an eye on it.

One more thing about daily budget. I was experimenting with different daily budgets ranging from €20 a day, all the way down to €2.5 a day. It’s important to note that every time you change your budget, Google does bid strategy adjustment. It takes anywhere between 3 and 5 days, and during that period your bid strategy is not optimal. So make sure to let it finish the strategy adjustment before jumping into conclusions. This is indicated in the status of the campaign.

1 out of 10 out of 100

Now, lets talk about conversion. My current click-through rate (CTR)—the amount of people who clicked on an ad as a percentage of total impressions—stands on 8.5%. That’s roughly 8 click per every 100 impressions.

My conversion rate is hard to tell, because I didn’t actively track conversions. I can dissect the traffic and make assumptions which will lead me to about 5% conversion rate. This means that out of 100 people who clicked the ad, only 5 made a purchase.

If I round the numbers a bit, out of every 1000 people, I got around 100 clicks, out of which I got about 5 sales. This means that my conversion rate from impression to purchase is 0.5%. It’s a bit lower, as I’ve heard of 1% number, but it depends a lot on how optimized the campaign is.

Campaign optimization

Google has nice tips to help you optimize your campaign. Some of them are useful, while others are just annoying (like the advertiser verification, which I never completed, and I doubt it affects the placement strategy). I would put attention to things like: keywords, titles, and descriptions.

I never had a perfect 100% campaign score, but I was around 82% most of the time. Keep in mind that more keywords is not necessarily better. Remember, your ad is not shown every time someone hit a keyword you are targeting, hence you might spend your successful bids for low performing keywords. It’s better to focus on fewer keywords, but ones that have bigger potential to drive sales.

Another tip is to pay attention to trademarks. I’m not entirely sure how it works, but when I tried to target keywords like “fax with paypal”, I got a warning that some of my ads are limited due to trademarks in the ad description (in that case: PayPal). So keep an eye on that, and make sure to remove keywords that might limit your ads.

In addition to all that you can also set up ad schedule. If you target the US, you might want to target US waking hours, instead of wasting your budget on a 3am guy who accidentally clicks your links while searching for “jet max”, but misspells it with “just fax”. The example is made up, but you get the point. Your goal should be to play with the parameters as you go, in order to squeeze the maximum from your daily budget. Monitor your campaign, analyze the data, and iterate.

Summary

I was terrified to start with Google Ads. When I first saw the recommended daily budget, it was around €300 per day, which means around €9,000 per months. As a bootstrapped solopreneur, I don’t have that money to blow on ads. But even if I did, I’m not sure whether it would be the best use for that money. This brings me to two lessons:

You can have a daily budget that is 10 or even 100 folds less than what Google recommends. It will drive less traffic, and less sales. So don’t afraid to start with less.
Ads are not for everyone. When you get into ads, you need to think about the LTV (lifetime value) you will get from your customers. My SaaS is a one-time use: people come, send fax, and go. They might return, they might not. If I make $4 per fax, but it cost me €10 in ads to acquire that customer, I’m loosing money. If, however, you operate on subscription model, or sell a very expensive product—your LTV might be way higher, and the cost of ads might be justified.

Oh, and the interface for Google Ads is terrible. It’s so bad, that I honestly don’t understand how people use it. And it’s not only Google Ads, I peeked at Meta Ads as well, and the interface is terrible as well.

One last thing. Google Ads is not the only place where you can advertise. More platforms exist such as: Meta Ads (which includes Facebook and Instagram), Reddit Ads, and even in Google Ads you can target different products such as Search, YouTube, etc. It’s important to understand where your target audience is. I doubt that people who want to send a fax online, hang out on Instagram, but if you sell a physical product, or a digital “guru”-like course—Instagram might be more suitable for you.

If you like this kind of content, I share my progress on a semiregular basis on Twitter and Mastodon, so consider following me there as well.

Using AsciiDoc to write my two books

Tue, 19 Dec 2023 18:00:00 GMT

In the past period of six months when I roamed around Central America, among the six projects that I launched—two where books that I wrote. The first book is about technical writing: Technical Writing for Software Engineer - A Handbook. The second one, which I wrote together with my wife, is about getting into tech: From Applicant to Employee - Your Blueprint for Landing a Job In Tech.

When you decide to write a book, you need to choose what tool to use. Among the most used one are Word (could be Microsoft or Google Docs), and Scrivener—which is the go-to app for writing. But I’m special, so at first I wanted to use Markdown.

Why Markdown, and why not?

Markdown is a fantastic format. It supports all the needed text styles such as italic, bold, and for non-supported styles like underline, you can always use HTML (in that case <ins> tag). And as some of you might know, this very blog you are reading at the moment, is written in Markdown. In short—I really like Markdown. Moreover, I wanted to use the same tools (NeoVim) as I do with my blog, so Markdown was the first obvious choice.

However, as it turned out, Markdown is awful for books, or any technical documentation longer than an average blog post. When writing a book, you need to have support for different elements which Markdown does not support natively, and without hacking. Examples could be: multipage documents, admonitions (those tip or warning boxes you see in many technical books), example blocks, etc.

Example of a "NOTE" admonition from my book Technical Writing for Software Engineers

I didn’t even mention things like: converting the Markdown files to PDF and EPUB, adding cover page, support for referencing other pages/chapters, etc. Of course, most of these issues can be hacked around. My blog has support for admonitions, but I use a custom remark plugin. Referencing other pages can be done using wiki-link styles, but they are not supported by Markdown parsers, and you need to bring in third-party plugins to your Markdown parser of choice.

And bundling a bunch of .md files into a PDF, can be hacked together with tools like pandoc—which is a great piece of software if you need to convert various text formats from one another. And in fact, Markdown + pandoc is what I first tried to do with my book. But quickly I learned that it’s too much hassle, and decided to explore other alternatives.

AsciiDoc

Before settling on AsciiDoc, I read about many different formats. Word and Scrivener were ruled out immediately as I didn’t want to learn new tools (in case of Scrivener), and preferred to stay in my environment of keyboard driver writing experience with NeoVim. I looked at tools like DocBook, which was ruled out because it’s heavily XML infested. And finally, I was set on AsciiDoc.

AsciiDoc is very similar to Markdown, but way more powerful. You have the standard elements of the page like headers (but using = instead of #); text styles (*bold*, __italic__); URLs (https://yieldcode.blog[yield code();]); etc. But in addition to the standard formatting elements, you get more things out of the box like admonitions, example blocks, variables that you can reference, table of contents, etc.

Example of an "Example" block from my, and wife's book From Applicant to Employee

The syntax of AsciiDoc is not that hard, especially if you know Markdown already. But one of the greatest things of AsciiDoc, is the software that comes with it: asciidoctor. Not only it has a great AsciiDoc syntax references, but it also acts as a publishing tool chain for converting AsciiDoc to PDF, HTML, DocBook, etc.

Using asciidoctor

I want to give a quick overview of how to set up asciidoctor, especially for PDF. On macOS, brew install asciidoctor will install the program itself, together with support for PDF. In order to add support for EPUB, you will have to install additional ruby gem using gem install asciidoctor-epub3. Consult the manual for your OS to get started.

Converting an AsciiDoc file to PDF, is a very simple task:

asciidoctor -r asciidoctor-pdf -b pdf -o out.pdf in.adoc

This command will convert in.adoc to out.pdf using pdf backend (-b) via requiring asciidoctor-pdf library (-r). Since AsciiDoc supports including other AsciiDoc files, you can have a one main.adoc that contains all your book information and metadata, and includes all other .adoc files which are chapters or appendixes.

I could end here, but I want to cover some more features. With EPUB you don’t have much room to play. The way EPUB is displayed in terms of fonts and theme—depends a lot on the device where it’s being rendered. PDF, however, is different, and you have more control over it.

When converting AsciiDoc to PDF, you can use a theme that will define how your PDF will be rendered. You control the theme with two CLI arguments:

pdf-themesdir: accepts a directory where a bunch of yml files named theme-{}.yml will exist
pdf-theme: accepts a string, which is the name of the theme

So, if inside your themesdir you have a file named theme-book.yml, you can pass pdf-theme=book to the tool in order to use your theme.

There is a default theme that asciidoctor will use if you do not specify one, but if you want to at least change the fonts, you will have to create your own theme. Luckily, you don’t need to create an entire theme from scratch, you can just extend the default theme and overwrite the parts you want/need. In addition to that, there are many themes you can find on the internet that you can use. And in case you do want to create a custom theme, here is the documentation.

Tying it all together with Makefile

The AsciiDoc for my two books lives inside a git repository. From here, I build it to PDF and EPUB using asciidoctor and upload to online stores. I want to create a CI/CD pipeline, at some point, that will build the books and upload them automatically, but for now I don’t have many revisions, so I do it manually.

However, I do automate the generation process with Makefile. For each book, I have a book.mk file that generates both EPUB and PDF files of the book. Then, each book.mk is included in the main Makefile which I run every time I want to generate a book.

Additional tools I use

In addition to AsciiDoc and NeoVim, I use some other tools that helped me with writing my books. Inside NeoVim I use ltex which is language server that runs spell checking using LanguageTools. I used macOS built it text-to-speech in order to do proofreading, in addition to proofreading myself multiple times.

For complicated sentences, I used Hemingway Editor in order to simplify them. The thing with Hemingway though, is to not just follow its recommendation. I believe that sometimes it’s fine to have complex sentences, or sentences in passive voice. But it’s a good tool nevertheless to improve your writing.

One last advice I want to give you. It’s not mine, I saw it from multiple blogs. Writing one sentence per line. With formats like AsciiDoc and Markdown, you can write one sentence per line, and multiple line will be rendered as one paragraph. You separate paragraphs with blank line.

Writing this way makes it easier to edit specific sentence, or scrap entire sentences. In NeoVim I would just dd or cc a line to delete/edit the sentence. One sentence per line is also a good visual indicator for long sentences, that are candidates for simplification. Lastly, gitdiff is easier on the eyes with one sentence per line. This is how it looks:

Imagine you want to tell a story.
You start with describing the setting.
A forest for example.

Inside this forest, Rust and Go are doing a picnic.
And they eat C++.

That’s it. I hope that if you would like to write a book in the future, that this article will be helpful to you.

6 lessons from building 6 projects, while nomading for 6 months

Sun, 10 Dec 2023 17:00:00 GMT

Six months ago, I boarded a one way flight to Costa Rica. A month prior to that, I quit my job after being burnt out. My apartment lease ended, I sold almost everything I own, and packed the rest into a backpack and a suitcase.

I had a simple plan: slow travel, work on my mental and physical health, and build a sustainable online business. I didn’t have a job, and was relying solely on my savings. This gave me all the time I need to focus on the above goals.

With time, from Costa Rica I moved to Guatemala, and then to Mexico. And after exactly 6 months, at the beginning of December, I boarded my last flight in 2023 for a permanent move to The Netherlands, a country to where my wife and I—relocated. I want to give you a short overview of my projects, and what I learned from them.

Disclaimer: I actually launched 8 different products. However, 4 of them can be put in 2 categories, so in total I launched 6 different product categories, each with different mindset. Felt like I need to clarify this, in order to avoid comment on the somewhat, misleading title.

Lesson #1: Build it, and they will come—is a lie

After landing in Costa Rica, and taking the first few days to adjust to the time difference, eventually I got to work on my first project: SilentFlip. I had the idea for SilentFlip in my head long before I decided to quit my job and nomad. The rationale was simple: take an existing product and build a copy that targets a different market. In my case, I wanted to build a remote feature toggling service that will target solopreneurs and small startups. Existing solutions are targeting big enterprises that can afford to pay per seat, which many solo builders can’t.

After spending some time building the MVP, and doing the landing page, I finally bought the domain, and shared the launch post on LinkedIn (where I’ve been somewhat active for the past 1.5 years). I got a modest amount of views to the post, and some small amount of likes. The landing page had no option to purchase the product, and instead I was collecting emails based on the desired price tier (I had 3 tiers).

My Post about SilentFlip on LinkedIn

I got this tip from Tim Ferriss’s book—The 4-hour work week—where he recommended setting up a landing page where you would collect customer intentions to buy the product. The idea behind this is that you don’t want to spend time developing a product if you can’t really sell/market it. However, looking back at it, I believe this is a wrong approach. I’ll cover this in a different lesson below.

But back to the original lesson. I, like many technical founders, believed at that time in the “build it, and they will come” mantra. I learned the hard way that this is not how it works. Unless you build a unicorn, there is a big chance that every market you will try—is saturated with competition. And when people search for a remote feature toggling service, there is a high chance they won’t land on my landing page.

In the end, nobody cares how perfect your product from a technical viewpoint, and this is what many technical solopreneurs/indie-hackers don’t get. You can build a product in a saturated market, and get customers, as long as you don’t just throw it in the void an expect people to magically find it because “it’s written using amazing technology!”

My Mastodon and Twitter post about "Build it, and they will come"

Lesson #2: Don’t build communities

Like a good indie-hacker, after seeing no traction from my first product, I ditched it and launched not one, but two, different products: Artisan Brew Coffee and PetAway.

Artisan Brew Coffee is a community for coffee lovers. Not the Nespresso type, but rather the pour over type. I got this idea after visiting a coffee plantation in Costa Rica, and as a coffee lover, decided to launch this community.

PetAway is a “scratch your own itch” type of product. Since I didn’t plan to come back to my home country, but rather wanted to end up in The Netherlands—my wife and I had to take our dog with us. And if you ever travelled with pets—you know what hell you need to go through. So I launched this community together with my wife in order to share knowledge for travelling with pets.

And here comes the second lesson: don’t build communities.

When you build a product that does not need a community, your success depends on two things: the quality of the product, and how widespread it is (read: marketing). It doesn’t matter how many people bought this product yesterday. As long as you are able to attract new customers, you will make profit.

With the community, on the other hand, you depend a lot on the members of the community. Nobody wants to be a member of a community which doesn’t have any members. So you are stuck in a catch-22 scenario: you need members for your community, but in order to gain more members, you already need to have members. There are ways to overcome this such as creating lots of content, or even fake profiles, in order to attract initial members. But it’s just another thing you need to take care of, in addition to moderating the community. And as a solopreneur—you already have a lot of things you need to take care of.

Lesson #3: Make sure to have a business plan

My two next products were LibrePress.io—an “own your content” type of newsletter distribution platform, I even wrote a blog post about it; and Remote or Not—A list of all companies by their remote and work-from-home policy. Both of them “failed” because I didn’t have a solid business plan.

LibrePress was born out of idealistic motivation: I’m a big believer in owning your content, and I wanted a newsletter platform that focuses on distribution without caring how you create the content. The idea was to distribute RSS feeds, or upload markdown files. I wanted to switch my newsletter to it, soon after launch. However, what I failed to take into account, is the running cost. Given the amount of newsletter readers I have, and the pricing for the hosting+email provider—I would end up in a range of $15-$20 per month, just to distribute my newsletter. This is too expensive for me at the moment, given the fact that currently I pay nothing (in monetary terms), and other services exist that charge less. Obviously, if I had paying customers, that would make it more cost-effective to switch, but I didn’t even come up with a pricing plan for LibrePress.

Remote or Not, on the other hand, was born out of my endless side-project ideas list. At the time of building it I was searching for a job with a strong preference for hybrid or fully remote policy, and couldn’t find a reliable source of such information. While I still maintain Remote or Not, it failed on two fronts: it’s a somewhat community based product (I depend heavily on contributions); and it didn’t have any clear business plan. It doesn’t have enough traffic to put ads. I didn’t want to charge job-seekers in order to get access to the information. As for companies, I’ve added a form for a company representative to reach out to me and provide official remote work policy for their company, with the intention to charge companies in the future. So far, no company reached out.

And, speaking about pricing plans…

Lesson #4: If you don’t charge, you don’t make money

Sounds obvious, right? But so many indie-hackers, including myself, ignore this lesson. There is this mental barrier to charge money for something. The moment you reach this milestone, you start to question yourself: am I good enough; is the product worth it; what if nobody buys it; etc. But the truth is, and circling back the advice from Tim Ferriss’s book, if you don’t charge—you don’t make money.

Collecting emails is not the same as collecting money. People are way more willing to sign up to alpha, beta, closed releases, and any other free services. But the moment they need to pay for something, it’s a completely different mindset. Tim’s advice was focusing more on physical products, where you have a high cost of manufacturing. In that case yes, I think it’s better to set up a fake shop, but guide the user through purchase flow, to clearly identify people who are willing to pay now, rather than collect emails of people who might pay in the future. But when you build a digital product, the cost of which is nearly zero, you need to charge money from the get-go. Otherwise, you are building a pet project, and not a business.

And it took me almost 4 months to realize it, and eventually launch JustFax Online—a super simple, e-fax solution. It was the first product I finished from A to Z. It had a working MVP, and I was charging money from the start. So far, JustFax and the two other products I’ll talk about in the next lesson, are the only products that make me money.

And coming back to the previous lesson, JustFax online was the first product where I actually had a business plan. I created an Excel spreadsheet and calculated different cost scenarios of operating such service: how much would I pay for sending the faxes, how much is the infrastructure, how much should I charge, etc.

My Twitter post, also available on Mastodon, about calculating the price for JustFax Online

Lesson #5: Think about distribution

Looking back at SilentFlip, Artisan Brew Coffee, PetAway, LibrePress, and Remote or Not, the reason they failed—is lack of distribution channels. Distribution channels can be advertisement, but it also can be a personal brand. And this is what happened to me when I launched my two books: Technical Writing for Software Engineers - A Handbook and From Applicant to Employee - Your Blueprint for Landing a Job in Tech.

Both of my books are available on Gumroad, which is a marketplace for digital goods. And while marketplaces and app-stores are a great way to distribute your products, they are, like anything else, saturated with content. There is a way to succeed on marketplaces/app-stores, but this is a topic for another blog post. Most of the sales of my books came from this blog, my newsletter, and my social network presence (Twitter, Mastodon, and LinkedIn).

I started to write my first book after the “failure” of Artisan Brew Coffee and PetAway, and at that time I started to be active on Twitter, Mastodon, and LinkedIn (where I’ve been active, but mainly promoting my blog). Being active on social media is a good distribution channel. If you are consistent, share quality content, and try to engage with the community—you essentially get “free” customers. The #buildinpublic hashtag is filled with solopreneurs and indie-hackers that are building products and sharing their experience.

But the problem is—you need to know your audience. If you are branding yourself as a software engineer, like I do, it becomes easier to distribute products targeting software engineers. And I saw it with both of my books. As I’ve said, most of my sales came through Twitter/Mastodon and my blog. However, if we look at JustFax—the target audience is not software engineers or indie-hackers. And while I got great feedback from both Twitter and Mastodon about JustFax, most of the sales didn’t come from these platforms.

And this is why it’s important to think about your distribution channels. Moreover, what worked for one product, will not necessarily work for another. This is why with JustFax, I’m focusing more on SEO and organic growth, and considering experimenting with advertisement.

Lesson #6: You only fail when you quit, or in other words: consistency wins

Few days ago I had a chat with my friend. He is also an indie-hacker who is trying to build a side-business. We talk once in a while and share ideas, and revelations. During our call, we were talking about consistency, and I came up with a great analogy.

Imagine you play a game. I grew up playing Dungeon Siege, so I’ll describe the setup from it. You are spawned as a peasant in burned down village. The entire kingdom is infested with evil creatures such as orcs, and the only hope to save the kingdom from evil force—is you. In this village, behind a burning house, you find a pitchfork. You take it and go kill your first two orcs. And get your first level-up to level two where you can assign skill points.

But then you quit the game, and the next time you relaunch it, you don’t start where you’ve quit—you start from the beginning, with the pitchfork behind a burned house, on first level, with both orcs alive, and zero skill points to assign.

This is how it works if you are not consistent. Every time you quit, or start a new project—you start from zero. Yes, you prefect some craft. I, for example, can build landing pages in less than a day now, because I built 6 of them. But only with JustFax I was able to get to a new level, unlock new abilities and knowledge. None of my other products reached a point where I had to take care of customers, collect payments, deal with bug fixes, focus on ads and SEO, etc.

If I had to start over again today, I would focus on one product for as long as possible, taking into account all the above lessons about community, distribution, and charging from the start. The moment you switch to a new product, and indie-hacker (myself included) like to switch often, you don’t start from the same point you left your last project—you start from zero. Yes you have prior experience, and starting over will be faster. But if none of your products reached a phase where you need to focus on SEO, you can start 100500 new products—but you won’t learn about SEO.

This is why it’s important to stay consistent. It’s perfectly fine to quit, and start a new product. But it’s only when you go through the whole journey, starting from idea, to product, to launch, to marketing and distribution, to taking payments, to fixing bugs and doing customer support; only then you are allowed to quit and start a new product. Otherwise, you risk running the same 100 meters of a marathon over and over again. Which is fine, if you want to be the fastest runner of 100 meters. But if you want to finish the marathon, you need to stay consistent.

I remember how scared I was before launching SilentFlip on LinkedIn. You get all these thoughts: you are not good enough; nobody needs it; you will just embarrass yourself. And I had the same feeling with every other product I launched afterward. It goes away little-by-little, but if you don’t share your work out there, you won’t succeed. No matter how stupid you feel, just share it.

Final Words

These were amazing 6 months. I’ve acquired an enormous amount of knowledge, and most importantly practical knowledge, which worth more than every theoretical knowledge you get. In addition to that, this has been a great time to cure my burnout and restart my career. I’m still analyzing everything from this trip, and planning to compile it to a few more blog posts, so stay tuned.

I would like to use the final words as an opportunity for self-promotion.

If you ever need to send a fax online, consider giving JustFax online a try. I’m open to feedback, so feel free to reach out to me.
If you are a software engineer who want to take your career to the next level, you need to be able to write technical documents. In that case, consider getting my handbook: Technical Writing for Software Engineers. It contains all my knowledge regarding technical writing.
If you found yourself laid off from your job; you want to switch to a new job; or you want to get into tech—consider getting my and my wife’s book: From Applicant to Employee: Your Blueprint for Landing a Job in Tech. It contains my and my wife’s (Senior Talent Acquisition, Technical/Executive Recruiter, and Career Coach) entire knowledge about technical interviews, how to pass them, and how to prepare. We used tips from this book in our recent search for a new job.
If you are interested in starting a side business, but don’t have ideas or ability to build your own product—I offer SilentFlip and Artisan Brew Coffee for sale. You can buy any part you want: domain, website, potential leads. I’m open to sell LibrePress.io and Remote or Not as well, as I don’t have much time to work on them. Reach out to me (email, or social networks), and let’s discuss it.

And as always, consider subscribing to my newsletter to never miss an update from this blog. Follow me on Twitter, Mastodon, and LinkedIn—where I share a lot about software engineering, as well as solopreneurship, indie-hacking, and building in public my projects.

Web app localization. In Rust.

Thu, 30 Nov 2023 16:30:00 GMT

Localization, often times stylized as l10n, is a process of adapting your website or web application to meet the requirements of specific target market (known as “locale”). It’s one of the 3 processes of going global, which include:

Translation (t9n)—the process of translating the content to a target language
Localization (i10n)—the process that I described above
Internationalization (i18n)-the process of adapting numbers, prices, dates, etc—to a target market/locale

Today, I want to talk about localization, why I think some websites do it wrong, and how I did it in my recent side project: JustFax Online. Are you ready? Bon voyage!

Oh, Rust code included 🦀!

Step one—understanding your users

I want to start with the obvious—how do you understand where your users are coming from?

What not to do

Many websites, Google and PayPal among them, are doing it wrong. For some reason, they are obsessed with GeoIP. But let me tell you a story.

For the past 6 months, I’m nomading in Central America. People in Central America—speak Spanish. I, however, hablo poco español (Spanish for “I don’t speak too much Spanish”). But both Google and PayPal (and I believe there are many other websites), insists on serving me their websites in Spanish.

With PayPal, it’s easy, they have a language switcher at the bottom, so I waste 2 seconds of my life every time I need to pay with PayPal. Google, on the other hand, is more complicated. The other day, I was given access to a Google Document through my non Google account, and after opening it, I saw that it was entirely on Spanish. I wasn’t able to find a way to revert it back to English. Luckily I know the Google Docs interface well enough that I was able to manage it.

The reason I got served the Spanish version of their websites—is because my IP is originated from a Spanish-speaking country, in my case Mexico. Switching on VPN to a different country, with different language, would serve their website in that particular language.

I understand the desire of whoever made this decision to have a good user experience, but in my opinion it’s horrible, and there is a better way to do it.

Meet the Accept-Language header

Accept-Language is a standard HTTP header that every browser sends. Instead of relying on GeoIP, the browser tells the server what languages I prefer, based on the settings of my device. My macOS is in English, therefor my Accept-Language looks like this: Accept-Language: en-US,en;q=0.5. Yes, Accept-Language can even have multiple languages. Every OS has a setting to set the preferred languages, and countries, and desktop applications seem to respect that. But web being web, prefers to use fancy GeoIP methods to wrongly guess my language.

But enough talking, let’s see how we can extract Accept-Language header with Rust and axum framework. If you are interested to read my take on Rust and axum for web development, make sure to check my last blog post Building a web app in Rust.

I always prefer to use TypedHeader.

NOTE

While writing this article, I found out that axum 0.7 was released. And it brings a breaking change to the header module. TypedHeader is no longer part of axum, and was moved to axum-extra. You can read more about it in the changelog. I haven’t upgraded to axum 0.7 yet, therefor the rest of this article relies on axum 0.6.x.

TypedHeader allows us to extract request headers in a type-safe way. axum has many headers defined that you can use with TypedHeader, but unfortunately, Accept-Language is not one of them. So I had to write my own. It’s not that hard.

Defining the struct for accepted language

First, we need to write a struct that will hold a pair of language identifier and quality values, essentially holding a string like this: en-US;q=0.5. Let me dump the code first, and then we will go over it.

#[derive(Debug, Error)]
pub enum AcceptLanguageError {
    #[error("Invalid value")]
    InvalidValue,
}

pub struct AcceptedLanguage {
    pub value: String,
    pub quality: f32,
}

impl Eq for AcceptedLanguage {}

impl PartialEq for AcceptedLanguage {
    fn eq(&self, other: &Self) -> bool {
        self.quality == other.quality && self.value.eq(&other.value)
    }
}

impl PartialOrd for AcceptedLanguage {
    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
        Some(self.cmp(other))
    }
}

impl Ord for AcceptedLanguage {
    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
        if self.quality > other.quality {
            std::cmp::Ordering::Greater
        } else if self.quality < other.quality {
            std::cmp::Ordering::Less
        } else {
            std::cmp::Ordering::Equal
        }
    }
}

It’s a very simple struct that holds a string and a float, and supports comparison and ordering (based on quality value).

Next, we need a way to create such structs. Since the most common scenario is creating them from a string, I’ve opted for implementing FromStr trait.

impl FromStr for AcceptedLanguage {
    type Err = AcceptLanguageError;
    fn from_str(s: &str) -> Result<Self, Self::Err> {
        let mut value = s.trim().split(';');
        let (value, quality) = (value.next(), value.next());

        let Some(value) = value else {
            return Err(AcceptLanguageError::InvalidValue);
        };

        if value.is_empty() {
            return Err(AcceptLanguageError::InvalidValue);
        }

        let quality = if let Some(quality) = quality.and_then(|q| q.strip_prefix("q=")) {
            quality.parse::<f32>().unwrap_or(0.0)
        } else {
            1.0
        };

        Ok(AcceptedLanguage {
            value: value.to_string(),
            quality,
        })
    }
}

Here, again, we don’t have anything special. We take a string such as en;q=0.5, and break it to its value and quality components. Keep in mind that quality is optional, and default to 1 if not specified.

Lastly, in order for TypedHeader to extract our custom header, we have to implement the Header trait.

pub struct AcceptLanguage(pub Vec<AcceptedLanguage>);

impl Header for AcceptLanguage {
    fn name() -> &'static axum::http::HeaderName {
        &header::ACCEPT_LANGUAGE
    }

    fn decode<'i, I>(values: &mut I) -> Result<Self, axum::headers::Error>
    where
        Self: Sized,
        I: Iterator<Item = &'i axum::http::HeaderValue>,
    {
        let value = values.next().ok_or_else(axum::headers::Error::invalid)?;
        let str = value.to_str().expect("Accept-Language must be a string");
        let mut languages = str
            .split(',')
            .map(AcceptedLanguage::from_str)
            .collect::<Result<Vec<AcceptedLanguage>, AcceptLanguageError>>()
            .map_err(|_| axum::headers::Error::invalid())?;

        languages.sort();

        Ok(AcceptLanguage(languages))
    }

    fn encode<E: Extend<axum::http::HeaderValue>>(&self, values: &mut E) {
        let val = self
            .0
            .iter()
            .map(|l| format!("{};q={}", l.value, l.quality))
            .join(",");

        let val = HeaderValue::from_str(&val).expect("Accept-Language must be valid");

        values.extend(std::iter::once(val))
    }
}

As I said, Accept-Header is an array of language identifiers and quality values, separated by comma. Therefor, I break the string by the comma character, and map it to my custom AcceptedLanguage struct.

Extracting preferred language

Next step, is to create a middleware that will extract the preferred language of the user. The purpose of this middleware, is to look at the Accept-Language header, and compare the values to our supported languages. JustFax is offered in English, German, and French, and their all defined in a toml config:

supported_languages = ["en", "de", "fr"]

NOTE

You can have a better user experience if you decide to target different countries, differently. For example American English (en-US), is different from British English (en-GB). I didn’t go that deep, and opted to offering American English to all my English-speaking audience. Apologies for that.

The middleware is not that hard:

#[derive(Clone)]
pub struct PreferredLanguage(pub Option<LanguageIdentifier>);

pub async fn extract_preferred_language<B>(
    request: Request<B>,
    next: Next<B>,
) -> impl IntoResponse {
    let config = get_config();

    let span = tracing::span!(Level::TRACE, "preferred language extraction");
    let _enter = span.enter();

    let (mut parts, body) = request.into_parts();

    let preferred_lang: Option<LanguageIdentifier> =
        if let Ok(TypedHeader(accept)) = parts.extract::<TypedHeader<AcceptLanguage>>().await {
            accept
                .0
                .iter()
                .filter_map(|lang| lang.value.parse::<LanguageIdentifier>().ok())
                .filter(|lang| config.locale.supported_languages.contains(lang))
                .collect::<Vec<LanguageIdentifier>>()
                .first()
                .map(|lang| lang.to_owned())
        } else {
            None
        };

    tracing::event!(
        Level::TRACE,
        "extracted preferred language: {:?}",
        preferred_lang
    );

    let mut request = Request::from_parts(parts, body);
    request
        .extensions_mut()
        .insert(PreferredLanguage(preferred_lang));

    next.run(request).await
}

NOTE

I use another library, called unic_langid that is able to parse strings such as en-US into their respective language, region, script and variants. It makes it easier to validate the locale strings, as well as comparing them.

The middleware takes the Accept-Language that we’ve created earlier, map each language to LanguageIdentifier, and filter out the ones that are not supported. If you look again at the Accept-Language header that we’ve implemented, it spits out a sorted array of languages based on quality. Therefore, we are guaranteed to pick the most preferred language that both the user, and our website, support.

Once we identified the preferred language, we put it in a axum extension, for future handlers to extract. Keep in mind that it can be None for cases where none of the languages in the Accept-Langauge header are supported by us (or the header is empty/invalid). In that case, I default to English as this is the main language of my application.

Step two—serving localized pages

We now have a way to extract user’s preferred language, using cutting-edge technology ~~GeoIP~~ Accept-Language header. The next step, is to serve the content in each language.

Defining URL scheme

There are 4 main ways to define your multilingual URLs. Using a country specific domain such as example.de; using a subdomain such as de.example.com; using a subdirectory such as example.com/de; and lastly, using query parameter such as example.com?lang=de.

Domains are expensive, so the first option goes to trash. Subdomains could be a good option, but I didn’t want to mess with DNS settings. Query parameters are bad option, and are not recommended by Google. I opted for using subdirectories.

NOTE

It’s funny, now I see it. On the above webpage by Google, they have a strong red warning that says: Don’t use IP analysis to adapt your content. But that’s what they are doing. Side effects of being a big-corp.

After deciding on the URL scheme, you need to decide what do you do with the root URL. Some websites leave the root URL to be the default language (for example, English). Others, might show a language chooser on the root URL (I don’t like that approach). I opted for a different approach: redirecting the root URL to a language specific URL.

So when a user navigates to justfaxonline.com/availability, he/she will be redirected (we will cover the redirection process in a bit), to a language specific subdomain, for example: justfaxonline.com/de/availability. In order to achieve this, I had to do some dancing with axum router.

I first needed to create the root routes:

fn make_root_router() -> Router<AppContext> {
    Router::new()
        .route("/", get(redirect_handler))
        .route("/availability", get(redirect_handler))
        .[some more routes]
        .layer(middleware::from_fn(middlewares::extract_preferred_language))
}

This router covers all the GET pages, and instead of serving the HTML, it calls a redirect_handler (hang on, we will get to it), which redirects to the appropriate locale subdomain. Then, I had to create the actual router:

let main_router = web::router()
    .merge(content::router())
    .layer(middleware::from_fn(middlewares::extract_preferred_language));

for lang in &config.locale.supported_languages {
    router = router.nest(
        &format!("/{}", lang.to_string().to_lowercase()),
        main_router
            .clone()
            .layer(Extension(DetectedLanguage(lang.clone()))),
    );
}

This router simply loops through all the supported languages from the config file, and creates a subdirectory for them. In addition to that, the main router also gets the preferred user language, as well as the detected language (which is the actual language of this particular route). We need them both in order to properly redirect the user, and provide a nice UX.

Lastly, we need to talk about user experience. First time users should be redirected based on their Accept-Langauge header. However, what if my computer is in Spanish, but when visiting JustFax Online, I prefer to use English? Meet: the cookie.

After the raise of SPAs (single page applications), it seems like people forgot about the existence of cookies. But cookies are very handy to store non-sensitive user information. They are being sent with each request, event if the request is made with Ajax. You don’t need any specific JavaScript code to send them, unlike LocalStorage.

On my website, I have a language picker at the bottom that allows you to switch language. Once you click it, there is some JavaScript on the client side that is used to set/overwrite the cookie, and then refresh the webpage. On the backend, I read this cookie and do the proper redirection. I’ll spare you the JavaScript code, but here is the backend code for redirect_handler:

async fn redirect_handler(
    Extension(PreferredLanguage(preferred_langauge)): Extension<PreferredLanguage>,
    OriginalUri(uri): OriginalUri,
    jar: CookieJar,
) -> impl IntoResponse {
    let config = get_config();

    let selected_language = match jar.get("lang") {
        None => None,
        Some(cookie) => cookie
            .value()
            .parse::<LanguageIdentifier>()
            .map(Some)
            .unwrap_or(None),
    };

    let mut lang = selected_language.as_ref().unwrap_or_else(|| {
        preferred_langauge
            .as_ref()
            .unwrap_or(&config.locale.default_language)
    });

    if !config
        .locale
        .supported_languages
        .iter()
        .any(|item| item.matches(&lang, false, false))
    {
        lang = &config.locale.default_language;
    }

    let url = get_url(uri.path(), lang);

    event!(Level::TRACE, "redirecting: {}", url);

    Redirect::temporary(&url)
}

It’s a simple code that reads from the CookieJar; if no cookie with the language exists, it reads the language from the preferred language (which comes from Accept-Language header); and if it doesn’t exist, or none of them are supported, it falls back to the default language.

Here you can also see the usage of unic_langid::LanguageIdentifier::matches function. This function is able to compare two languages, and based on the seconds and third arguments, to treat them like range. This allows comparing en and en-US easier.

NOTE

But I don’t like JavaScript!, I hear you say. And I’m with you. JavaScript is overused nowadays, and usually for bad purposes like annoying popup windows. However, I try to keep my JavaScript code to a minimum. In fact, aside from htmx and some event handlers, JustFax Online is mostly JavaScript free. If you browse the web with disabled JavaScript, you will lose functionality. I’d like to have you as a customer, but I also care about a nice user experience. So I always need to balance those to aspects.

A touch of UX

Some websites put the language switcher on the top; others at the bottom. And some, hide it in obscure places, never to be found again. I appreciate a UI that puts me, the user, in the center. This is why I added a nice touch to the UI of JustFax.

When your preferred language (the one that comes from the Accept-Header), and your detected language (the actual subdomain language)—differ—I show you a nice, unintrusive popup:

Language switch popup

You can either revert to your preferred language, or hide this popup, never to be shown again (for the next 30 days).

Step three—actual localization

I hope you are not tired, but we need to talk about how to actually localize your web pages.

There exist a multitude of tools and frameworks to do so. The lamest one look at a key, and it’s associated string. So for example you could have a en.toml with the content of:

features-title = Features

And a de.toml with the content of:

features-title = Funktionen

And a magic t() function in your templates, that will accept a key, and output its value.

This is a lousy solution, if you aim for perfection (like I do). Language is way more than just a collection of words. One obvious example, is when enumerating things. In English, you would say “Close one tab” and “Close n tabs” for any number of tabs other than one. However, in many Slavic languages, you would use a different imperative for the word “tabs”. In Polish for example, you would say “Zamknij kartę” for one tab; “Zamknij {$tabCount} karty” for few tabs (2, 3, and 4); and “Zamknij {$tabCount} kart” for many tabs (5, 6, and on). In addition to imperatives, you might have different variants of translation based on gender, how lower/upper case is handled, etc. So in short, languages are fun!

There exists a tool made by GNU called gettext. It’s very robust tool for l10n. The main feature of it, is that you will use an agreed function name, usually _(), and the parameter of this function would be an English phrase: _("Hello World"). You’d then run the gettext utility over your source code, and it will generate a .po file to which it will export all the strings in your source code. By using POEdit GUI application, you will be able to translate those files, or give them to a translator.

It’s a very robust tool that exists from 1990. It’s written in C, and there are Rust crates that interface with it. It doesn’t pollute your code with obscure keys like feature-title. I, however, dislike it due to the fact that it needs to go over my source code, which means I need to have another pre-deploy task to perform. Moreover, because the actual phrase is your key, when you find a typo, good luck fixing all your translations.

Luckily, I was able to find Project Fluent which is maintained by Mozilla, and is used in Firefox. It addresses many of the issues with gettext, and luckily for me, has a Rust implementation. The implementation is very raw though, and requires some juggling to make it work. But, as you already understood, I’m very lucky and there is another crate, called fluent-templates, which makes working with fluent-rs way easier. More luck on my side, it has a built-in implementation of the t() function for the tera templating engine that I use. Inside a tera template, I would call it like this:

<p>{{ t(key="hello-world", lang=LANG) }}</p>

LANG is a global variable defined in the tera context, based on the detected language of the route.

This article is already too long, so feel free to check fluent-templates documentation. It’s pretty straightforward.

Conclusion

Things like l10n and i18n are hard to get right. Even giants like Google mess up sometimes. I hope in this article I was able to cover the basic building blocks you would need for your own localized web applications in Rust. Bear in mind that I didn’t even touch things like numbers, currencies and dates, which have their own caveats. Maybe I’ll do one day in the future.

Building a Web App in Rust

Sun, 12 Nov 2023 19:20:00 GMT

I remember when I first tried Rust, I fell in love. It was like a breath of fresh air after JavaScript and NodeJS. TypeScript made the experience way better, and I believe that if you still write pure JavaScript, you are dooming yourself. I wrote about the importance of types here and here.

But back to Rust. My first experience with Rust—aside from trying to build my own game using ggez, bevy, and wgpu—was building a business logic layer in Rust which Electron app could call. I wrote about my experience here and here. This project, however, never saw light. I’m not disappointed because I managed to craft two amazing pieces of content, and present them in one meetup and two international conferences.

But I still wanted to write a web application in Rust, because as they say: “anything that can be written in Rust, will be written in Rust”. And on the 1st of November—I did! I launched JustFax Online. JustFax Online is a service that allows you to send a one-time fax without creating an account or committing to a subscription. It’s a perfect tool for people who need to send an occasional fax. And most important, in terms of technology, JustFax is a 100% Rust application.

The Architecture

Let’s talk about the architecture of JustFax. It consists of two OS processes: the web server, and the dispatcher that is responsible for sending the faxes.

The web server is written in axum, which is a very popular web framework. Axum is built on top of tokio, which is and asynchronous runtime for Rust; hyper, which is a low-level HTTP implementation in Rust; and tower, which is a collection of reusable components for building networking clients and servers.

On top of that, it uses the tera templating engine, which has a similar syntax to Jinja2. Storage is handled by SQLite, via the rusqlite library.

The dispatcher process is just an infinite loop that polls the database for changes. There are also some 3rd party integrations, which I won’t go into details here.

Let’s talk about my experience building a web application in Rust.

The Good

First, we will start with the good parts.

Memory usage. This is obviously the first advantage you get from Rust. I tweeted this image, shortly after deploying the first version.

Look Ma! A web app that doesn't take hundreds of MB of RAM#buildinpublic pic.twitter.com/Ymwt6sulTJ
— Dmitry Kudryavtsev (@skwee357) October 29, 2023

You can find the original post on Twitter and on Mastodon. Consider dropping a follow there as well ;)

Yes, it’s true. A web application + a job runner, all wrapped in a supervisord in Docker container, that takes ~30MB of RAM. Eat that NodeJS!

Safety. There is a saying in the Rust community that goes like this: “If it complies, it runs.” And in general, it’s true. You can, obviously, run into runtime issues with deserialization of 3rd-party API responses, but that’s just a side effect of building dynamic applications. Other than this, if you are able to compile it, it will run successfully, but not necessarily correctly.

Async. People in Rust community like to shit on async Rust. Coming from NodeJS, where async development is heavily embedded, I can’t seem to understand the hate. Sure, async Rust is not as mature as NodeJS. There are different macros to make asynchronous traits, but in general, I found async Rust to be nice. I haven’t had the chance to debug asynchronous code, which I believe can be a pain. And you always have the option to spawn a thread, should you need to.

Type safety. Oh, sweet Jesus! Real type safety! It feels incredible. You no longer need to hunt stupid bugs where your UUID string, can have, accidentally, a non-uuid value. A real, mature type system—is a welcoming addition to the development toolbox.

Having said that, not all is rainbows and chocolate.

The Bad

Rust is a complicated language. I’m still far from understanding all its features. And with such complexity, comes the downsides.

The borrow checker. One of the biggest struggle for new Rustaceans—is the borrow checker. In short: borrow checker is a Rust mechanism which is the heart of it’s safety and security. This is what prevents Rust from leaking memory, or referencing a null pointer.

Understanding the borrow checker is a complicated thing. I must admit that if you know concepts such as pass by references, pass by value, pointers, and when variables go out of scope—it helps a lot with understanding the borrow checker. In addition to that, you barely have to touch the borrow checker by building a web application on top of axum. Unless you have complex data structures that hold references to other types/objects, most of the code will look, and read like a regular code in any other language.

Starting is rough. While axum is very similar to your favorite web framework such as RoR, fastify, or django—it does have unique features to it. Things like request extractors, the way middlewares work, are error handling. This hurts DX and the speed at which you can deliver.

I was familiar with axum from a previous toy project I played with, but I must admit that I could have finished the project faster with NodeJS. It could be related to the fact that I have more experience with NodeJS, or it could be a caveat of Rust development. At that point, I no longer can answer this question reliably.

Compilation time. On my machine, compilation is relatively fast. However, the moment I switched to a CI/CD pipeline, I’ve encountered a relatively long compilation times. I tweeted and tooted about the initial compilation times of ~650 seconds (which is more than 10 minutes) in a CI/CD pipeline.

I was able to reduce the compilation time to around 3 minutes, after removing the multi-stage docker build, and instead compiling the code inside the CI docker, as well as caching the dependencies. As of today, the entire CI/CD pipeline from push to deployment takes about 6 minutes, which is not bad overall.

The ecosystem. Rust has a very big community around it, and a great ecosystem. The community maintains a bunch of arewe__yet websites, such as arewewebyet which list all the popular packages for a particular domain (in this case web development).

Having said that, the ecosystem is not as big as NodeJS or Python. Many semi-popular packages and libraries—will be missing. Most examples will be given in either JavaScript, Python, Ruby, and even PHP—but will not have Rust.

Many issues and problems you will encounter with, will require you to go through, mostly, unanswered Rust community questions; in many cases you will have to read the source code of a particular crate in order to understand how it works.

One great thing I love about the Rust community though—is that most packages are test covered, which serves as a great tool for examples. And almost every popular package comes with a examples folder inside the repo, where you can see popular use cases and build on top of them (example for axum).

Conclusion

I really enjoyed building this project in Rust. For a long time I looked for an excuse to launch a Rust project. There is no particular reason for using Rust over, say, NodeJS. But being and indie-hacker, I have the tendency to launch many projects without having $$ to throw on AWS or support teams. Knowing that my application takes ~30MB of RAM, means I can run a few of them on a 1GB RAM shared VPS. And Rust being a language that puts a lot of attention on safety, means I can run these projects without worrying of cannot read 'foo' of undefined errors, or my entire server is crashing due to an unexpected response from a third party service.

In case you need to send a fax, consider giving JustFax Online a try. The service is available in English, German, and French (with more languages to come). It supports sending a fax in 50 different countries. You don’t need to create an account or commit to a subscription model. It takes literally 2-minutes to send a fax. And most importantly, you get to support a fellow indie-hacker.

Follow me on Twitter and Mastodon to see more behind the scenes of my journey, as well as software engineering content.

Until next time!

Typescript Monorepo with NPM workspaces

Sun, 01 Oct 2023 23:00:00 GMT

During the development of my recent project (more info on my website), I wanted to make it more modular. It consists of 3 main components: api which is the API server; a process that is responsible for RSS feed discovery called observer; and a process responsible for sending out emails called distributor. They all depend on a database.

I wanted all of them be independent executables. Using a monorepo for such use case, is great. But settings up dependencies between them, is complicated.

Luckily, npm 7 has support for workspaces, which simplify the management of monorepos. Not only that, but using workspaces also helps you to keep your node_modules more lean (as much as possible).

How it works

Let’s say you have 3 projects:

infra which responsible for working with the database
api which is your API server
worker which is some kind of asynchronous processing worker

You want to keep them all separated, with their own set of dependencies. Both api and worker need to depend on infra. You could just create 3 different projects, publish the infra on some internal repository, and install it as a dependency for api and worker. But this really complicates things.

Instead, you can setup a workspace using npm (we will see in a moment how to do that). On top of all that, npm will optimize your node_modules. So if both api and worker depend on the same version of uuid package, instead of installing the package two times, it will be installed only once.

The general project structure also looks different:

.
├── node_modules
├── package-lock.json
├── package.json
├── packages
│   ├── api
│   ├── worker
│   └── infra
├── tsconfig.build.json
└── tsconfig.json

You get only one node_modules at the top level. All dependencies will be there, including symlinks to your internal packages:

.node_modules/
├── ...
├── api -> ../packages/api
├── worker -> ../packages/worker
└── infra -> ../packages/infra

So this means you can install them just like regular packages! Let’s dive in.

Setting up npm workspaces

We are also going to setup typescript along the way, as there are some caveats.

Inside your top level folder, you will need to create a package.json (either manually, or via npm init):

{
  "name": "my-app",
  "private": true,
  "scripts": {},
  "workspaces": ["packages/*"],
  "devDependencies": {
    "@tsconfig/recommended": "^1.0.2",
    "@types/node": "^20.6.0",
    "ts-node": "^10.9.1",
    "typescript": "^5.2.2"
  }
}

The most important line is "workspaces": ["packages/*"], which instructs npm to treat this package as a workspaces root. All the actual packages will be inside the ./packages folder. By the way, you can name this folder whatever you want. I also added some dev dependencies for typescript.

While we are inside the root folder, it’s also a good time to create some typescript boilerplate.

First, we need a base tsconfig.json.

{
  "extends": "@tsconfig/recommended",
  "compilerOptions": {
    "incremental": true,
    "target": "es2019",
    "module": "commonjs",
    "declaration": true,
    "sourceMap": true,
    "composite": true,
    "strict": true,
    "moduleResolution": "node",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  }
}

This is the base typescript configuration for all the packages. Since we want to keep everything consistent throughout the monorepo, we will change any project wide settings there. Next, we need another config, this time for building the entire monorepo: tsconfig.build.json:

{
  "files": [],
  "references": [
    {
      "path": "packages/infra"
    },
    {
      "path": "packages/api"
    },
    {
      "path": "packages/worker"
    }
  ]
}

And while we are here, let’s also add a build script to the root package.json:

...
"scripts": {
    "build": "tsc --build --verbose tsconfig.build.json",
}
...

This way, when we run npm run build in the root of the monorepo, the entire app will be built.

Creating a package

Now, we need to create individual packages. This can be done either manually, or via npm init. For example, in order to create the infra package, we will execute npm init --workspace packages/infra -y. This will create a default package.json inside the packages/infra directory.

NOTE

I recommend you use scoping for your project. Scoping helps you avoid any confusion with existing packages. In order to use scoping, when calling npm init provide a flag --scope @mycompany. This way, all your packages will be prefixed with @mycompany, and won’t cause any confusion.

Here, we work regularly. We install dependencies as if it was a standalone package. Any npm script can be executed from inside the packages/infra directory, or from the root directory using npm run <script-name> --workspace packages/infra.

Another tip. If you want to execute a script across all packages, you can also use npm run <script-name> --workspaces. This will iterate over all the packages in a workspace, and execute the said script (or emit error if no such script is defined). This is very handy for running tests, for example.

In order to install a dependency inside a package, we will execute npm install --save uuid --workspace/infra. However, as you will notice, there won’t be a node_modules inside packages/infra. Instead, all the dependencies will be put in a top level node_modules.

Finally, let’s answer the question of how to share infra with both api and worker. Repeat the above steps in order to create packages/api and packages/worker.

And now, when you have all of them ready, just execute npm install --save @mycompany/infra --workspace packages/api (and repeat it for worker as well).

NOTE

Here, scoping comes into play. The global npm registry has a package called infra, so the above command is ambiguous without scoping.

Setting up typescript

Lastly, we want to set up typescript for the individual packages. Inside every package, create a tsconfig.json that looks like this:

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "rootDir": "src",
    "outDir": "dist",
    "paths": {
      "@mycompany/*": ["./packages/*"]
    }
  },
  "references": [
    {
      "path": "../infra"
    }
  ]
}

First, we extend the global tsconfig.json that we created. This is necessary in order to keep the general guidelines among all packages—the same.

Next, we override some compilerOptions. It’s needed in order to tell typescript where are the source files and where is the output directory. We can’t include it in the base config, since it’s relative to the package path.

On the api and worker packages, we need two more things: paths and references.

paths allows us to use scoped imports such as import x from @mycompany/infra.

references allows us to use the .d.ts files of the dependent package. If you want to learn more about references, consider reading the official documentation.

NOTE

The infra package does not need to have paths and references since it does not depend on any internal packages.

Now, when we execute npm run build, all the packages will be built. According to our internal tsconfig.json files, the output will be placed inside packages/<package-name>/dist.

Caveats

While workspaces in npm are great to keep things clean, they come with some caveats.

One problem I’ve read about online, but haven’t encountered myself, is improper dependency resolution. Since all dependencies are flattened and put into a global node_modules directory, some people reported that it can cause bugs when the wrong dependency is imported.

Another downside of this approach, is that you won’t be able to produce atomic units for deployment. Since node_modules is shared among all packages, it needs to be included with each and every executable unit.

Consider a scenario where api depends on a lot of packages, but worker depends only on @mycompany/infra (which in turn depends on some database package). In order to deploy worker independently, you will have to copy the entire node_modules along with it. This is less than ideal if you deploy api and worker on separate docker containers or machines. However, if you have one Docker container that runs both using supervisord or pm2—it’s not a problem.

On APIs and their responses

Mon, 25 Sep 2023 02:00:00 GMT

Since the dawn of ~~times~~ the web, humans created CRUD APIs. And we were instructed that modification verbs should return the modified resource in response. But, should they?

A basic CRUD API consists of 5 verbs. Let’s go over them quickly.

The 5 verbs

GET

The GET verb is used to retrieve the resource, or a collection of resources. It should not make any modification on the server, and therefor it’s idempotent.

NOTE

Before moving on, we need to talk about idempotent verbs. In regard to HTTP verbs, a verb is idempotent only if subsequent calls to the same endpoint, with the same verb—will produce the same result. GET is idempotent, because if I execute GET method on a particular endpoint, let’s say /post/on-apis-and-responses—I’ll always get the same result, and the state on the server does not change.

POST

The POST verb is used to create a resource on the server. It also can be used to create a collection of resources. From what you understood, POST changes the state on the server, and is not idempotent.

NOTE

Why POST is not idempotent? Because subsequent POST request will either fail, in case of duplicate identifier, or will create another, different, resource.

PUT

Contrary to POST, PUT is used to replace a resource, or a collection or resources. Like POST, PUT modified the state on the server. However, unlike POST, PUT is idempotent.

NOTE

Why PUT is idempotent? Since PUT fully replaces the resource, subsequent calls will end-up with the same state—the replaced resource.

PATCH

PATCH is not a well known verb. Some confuse PATCH with PUT, but there is a difference. While PUT replaces the entire resource, PATCH is designed to update parts of a resource.

To give you a better example, imagine we have the following resource:

const resource = { email: "contact@yieldcode.blog", handle: "@skwee357" };

In order to update the email, I have two options.

I can issue a PUT request. But this will require me to submit both email and handle fields, because PUT used to replace the resource.
I can issue a PATCH request. This way, I can submit a partial JSON that will contain only the new email.

PATCH, as you probably guessed, modifies the state on the server. However, if you assumed that PATCH is idempotent, just like PUT, you are wrong. I won’t go into details here, but you can check this 8-year-old StackOverflow question, where I asked about the same thing.

DELETE

Last, but not least, is DELETE. It’s used to delete a resource from the server. It modifies the state on the server, and is idempotent.

NOTE

Why DELETE is idempotent? While the first call to DELETE will, most likely, return 200 Ok—subsequent calls will, most likely, return 404 Not Found. However, it’s not about the response. The state on the server remains the same—the resource no longer exists.

On POST, PUT, and PATCH and their responses

There is a verbal agreement that POST, PUT, and PATCH operations should return the created/updated resource. In the rest of this article, I want to provide reasons why they shouldn’t.

NOTE

Everything I’ll talk about, applies to standard CRUD APIs. If you develop a specialized API that does not adhere to the CRUD structure—do what you want, and what suits your needs best.

Reason 1: API consistency

Consistent APIs are easier to work with. It doesn’t matter if you plan on third-parties to integrate with your API, or it’s used internally by your frontend/mobile teams.

By returning a newly created resource inside POST request—you will have to return a response in all POST requests. Otherwise, the people who integrate with your API, will have to handle endpoint-by-endpoint use-cases.

Returning response in every POST endpoint is, however, unfeasible. Why? Here are some examples.

What would your return on POST endpoint that accepts batch data from CSV? Let’s say you are able to process the CSV in-place, without delayed processing. Are you going to return all the newly created resources? What if there are hundreds of them? Do you return them paginated?
What would you return on a POST endpoint that schedules an async job? Let’s say you send s CSV, but instead of parsing it in-place, you schedule the parsing for a later stage using some queue. Do you return a job object? A job ID? Do you even want to expose your API consumers to the concept of asynchronous processing?
Imagine you have a resource such as blog post. This blog post can have tags. After issuing a PATCH request to add a new tag, what would you return? Do you return the entire blog post object, or only the array of tags? Or maybe the newly created tag?

As I demonstrated, there are some ambiguities you need to solve in order to support returning the resource on create/update. This complicates your API, and makes it harder for your consumers.

What is the alternative? There is one, but first, let’s discuss some more issues with returning resources.

Reason 2: Offline support

One reason to return the resource on create/update operations—is to avoid the additional GET call to retrieve the newly created/updated resource, that the client will have to do. However, when designing an API to be consumed by a client that can work in offline mode, the client needs to backfill the newly created resource anyway.

When a user creates a resource, but the application is offline, you want to have optimistic flow. The UI updates accordingly, but the resource is queued to be created when the client goes back online. In that case, the application already knows the structure of the POST response. Hence, it’s unnecessary to return anything when issuing create/update.

When the client will be back online, it will execute all the create/update operations to let the server know what happened, and will issue a GET request to update the view for the user.

But Dmitry, I don’t care about offline! Too bad. But let’s say you really don’t care about offline. In today’s world, of zero patience, users expect stuff to happen instantly. And even if you issue a POST request, and wait for a response from it, there could be a noticeable latency that your users won’t like.

One way to fix this latency, is to do an optimistic update. The user fills the form, and click submit. Your UI framework creates a dummy resource and immediately appends it to the list of resources, as if it was created successfully, hence updating the UI. The user is happy.

In the background, you issue a POST request to create the resource on the server. There is a high probability that the request succeeds, hence the name—optimistic update. If it fails, we can notify the user via a pop-up, or retry again up to a certain number of retires.

Reason 3: It violates DRY

DRY, which is an abbreviation for don’t repeat yourself, is a principle that states that your code should be free of repetitions. Repetition make the code hard to maintain, as now there are few places that do the same, or depend on the same method.

By introducing a response to create/update operations, you violate DRY, because you have to return the same resource as the one you return in GET. And if the resources are different between create/update and GET calls, your API becomes inconsistent, which violates the first reason about API consistency.

Reason 4: It violates SRP

SRP, which is an abbreviation for single responsibility principle, states that each module in your code should be responsible for one action. GET returns the resource. DELETE removes the resource. But when you introduce response to create/update operations, they no longer have single responsibility. This can get complicated.

Consider a use-case of adding a new tag to a blog post. The entire logic (excluding authentication and validation), can be narrowed to one particular SQL:

INSERT INTO blog_post_tags (post_id, tag) VALUES ($1, $2) ON CONFLICT IGNORE;

So when your POST operation finishes, you don’t hold the entire blog post resource. Therefore, in-order to return a proper response during POST, you either need to call GET on the same resource internally (many frameworks support issuing internal HTTP requests). Or you need to refactor the resource retrieval, and serialization, to an external function which both the GET and the POST handlers will call.

Now, they need not only create or update the said resource, but also serialize it in the same way as GET would, and return it to the client.

What to do instead

Well, nothing! You just don’t return anything. The same way we do DELETE requests, which return 200 OK. However, for resource creation you should use 201 Created or 202 Accepted if the resource will be created asynchronously.

If you want a really nice API design, you can return a header named Content-Location which will point the client to the location of the newly created resource. This creates a very nice decoupling inside your code, because retrieving the resource becomes a matter of reading this header, rather than hard-coding a specific end-point.

So, instead of

const { id } = await api.post("/task", body);
const task = await api.get(`/task/${id}`);

It becomes less coupled:

const { headers } = await api.post("/task", body);
const task = await api.get(headers["content-location"]);

This, obviously, won’t work for batch operations. While you can use another header, Link, this will quickly get ugly if you have more than a few resources. You can return a Content-Location header that points to the collection, instead of an individual entity. However, I think it’s better to not return anything.

But what about the subsequent GET request? Well, for starters, if you rely on optimistic update (and you should), your UI already reflects the change. There is no need to fetch the created resource.

In general, GET on collection should be performed only once—when the user opens the list of tasks/todos/whatever. GET on a particular resource, should be performed when the user opens the page of that particular resource. In all other cases, you better rely on optimistic update, especially if you have mobile clients with offline capabilities.

And if you really have to fetch the resource, just issue the GET request. It’s not that big of deal.

That’s it. That’s all I wanted to share with you today.

Let me know what do you think about this approach. I’m available via Email, Twitter, Mastodon, and (for the brave corporate folks among you) LinkedIn.

On types and JavaScript drama

Sat, 09 Sep 2023 20:00:00 GMT

Earlier this week, the entire JavaScript community was in shambles over a controversial decision in a popular open-source project.

What happened?

David Heinemeier Hansson (known as DHH)—who is the creator of Ruby on Rails (among many other things)—approved a PR that removes Typescript from turbo framework.

This decision upset a lot of people, and created a big controversy. It even reached trolling inside the project’s issues/pull requests sections on GitHub. Many people, some of them never even heard about turbo, were really upset.

Here are 3 lessons we can learn from it.

Lesson #1: Caring less

While DHH is a popular figure inside the development community, he is just a regular guy with his own opinions and views on software development. The fact that he decided to go back to vanilla JavaScript—should not, in any way, affect your development preferences.

Clarification: I don’t suggest you dismiss any opinion which does not align with yours. It’s important to listen to other opinions and review your development preferences from time-to-time. What I do suggest is to not follow, blindly, opinions of other people just because they are well known in the industry.

Lesson #2: How to (not) treat the community

Many people expressed a very valid concern that I agree with. Turbo is an open-source package. And while DHH can do whatever he wants, the success of turbo is thanks to a lot of contributors. Some of them had open PRs, not yet merged, that are now obsolete. I didn’t dig too deep, because I don’t like drama, but from my understanding—the change was not communicated to contributors. I can see why active contributors are getting upset—they did not get the respect that they deserve.

It seems like this is a common pattern in some open-source projects. Project that are led by people who care only about themselves and their agenda. We should remember that the success of OSS depends a lot on the community, and the people who, often times without any monetary gains, dedicate their time for the project.

Lesson #3: Types are important

I strongly disagree with DHHs decision. Going back to vanilla JavaScript—is a big step backwards. It is now up to DHH to sort out his relationships with contributors, but I suggest you do not follow his lead. Type safety, even at compile time, is a big guarantee for maintainable, and correct code.

While DHHs arguments were simpler code, and less type acrobatics¹—in reality, I believe it will bite him back on day. The amount of can not read foo of undefined errors, that I was dealing with back in the days of vanilla JavaScript, is too much. And considering the fact that you need to have a healthy development cadence, dealing with such errors—hurts DX a lot.

I suggest you check one of my old blog posts about type safety titled It's better to be (type)safe than sorry.

In the meantime, stay safe (pun intended) and enjoy the weekend.

https://world.hey.com/dhh/turbo-8-is-dropping-typescript-70165c01 ↩

Own your content

Wed, 30 Aug 2023 15:00:00 GMT

For the past years, we’ve witnessed a rise of new type of social media platforms. Platforms that compete for the ownership of your content. And the attention of your consumers.

From owning to renting

Before I’ll talk about why, and how you should own your content—let me briefly cover the transition we’ve made from owning stuff, to renting stuff. Feel free to skip to #Why you should own your content if you are not interested in a bit of history.

As consumers, we used to own the things we paid for. If I bought a DVD with The Matrix movie—it’s mine forever. I can re-watch it as many times as I want. I can create copies from it, convert it to different formats, and watch it on different computers ¹. The only way someone can take it from me—is by stealing, which is punishable by law.

I remember the days when we used to buy new music albums of our favorite bands. These days are over, because everything is now for rent. I was very surprised when I learned that even the Kindle books I buy—are rented to me. Yes, I knew that Kindle books are protected by DRM, and it’s hard (not impossible) to read them on a non kindle device. It’s not perfect, but I’m okay with such arrangement.

However, when I checked the terms and conditions of Kindle books, I found out that the content is not sold, but rather licensed to us. I tweeted about it back in December.

I kind of swallowed the pill that I no longer own music or video content. I don't watch that much TV shows or movies, and anyway I rarely rewatch them. And so losing video content due to platform's removal - is sad, I won't cry over it. But I discovered something interesting.👇🧵
— Dmitry Kudryavtsev (@skwee357) December 22, 2022

And in case you are not aware: your Netflix shows, Spotify playlists, etc—are all rented to you. There are cases when content that was available in your geographic region, or even content you bought—was removed from the platform.

But I don’t want to ramble too much about content ownership from a consumer perspective. It is important, don’t get me wrong. But I’m here to talk about content ownership as a producer, which also participated in the “rental revolution”.

Why you should own your content

Content was, and still is, one of the biggest assets you can create. This blog is visited by hundreds of people daily—because it has the knowledge that I possess, presented in textual format. Content is the reason we have the Internet in the form we have it today. And it’s incredible.

I’ve learned so much from other people’s blogs and posts. And I’m doing my best to spread my knowledge further, to more people. And while I want to do it on my terms, there are companies that want to monopolize this market. They create tools for easy content creation, and distribution—but in return they gather most of the rewards.

These companies will lock content behind a pay-wall. They will create their services is such way to make sure their users—the content consumers—will stay on their platform.

$$$

Gee, Dmitry!! You are so anti-capitalism! No I’m not. I believe that people who produce content—should be able to get paid for their content, if they choose to do so. I don’t mind pay-walls. I just don’t want to pay to big corporations that will cut their chunk of the profit. Supporting creators directly would be my preferred way.

We already have a model where big corp gets the payment. The music industry is a good example. Small creators rarely see any real profits from their work, because labels take most of the money. Do we really want to create yet another model like this?

The real cost of outsourcing your content

Paying for the content, however, is not the biggest problem with these platforms. The real cost comes in a simple fact—you no longer own your content. It’s left to the mercy of the platform.

I saw a LinkedIn post from Gergely Orosz recently. In this post, Gergely said that his Google Document was blocked due to “violations of the ToS”. Gergely is a paying customer. And even though he pays Google, Google didn’t bother to contact or notify Gergely that his document might violate their ToS. Here is the LinkedIn post.

The internet is filled with examples of people who were blocked from their accounts, or whose work was removed from the platform. YouTube, Medium, Spotify, Google—you can find examples of all of them.

And what happens when a platform goes down? Do you remember Revue? It was a newsletter platform, which was owned by Twitter. Shortly after Elon purchased Twitter, Revue stopped existing. Luckily for me, who used this platform, and many others—Revue offered an option to take out the data. Do they all offer such option? Don’t know.

You are your brand

It’s hard to talk, or in my case write, about branding without sounding like a wanna-be-twitter-influencer. But in reality branding, and especially personal branding—existed for many years, decades even, if not centuries. And for many content producers, their content becomes their brand. I know of technical bloggers who are getting jobs because of their blogs.

In his book, Developer Hegemony: The Future or Labor (not an affiliate link), Erik Dietrich talks a bit about branding. One of the traits of the “opportunist developer”, as Erik calls them, is that they market themselves. Here is what Erik writes about marketing and branding:

Everyone I interviewed markets themselves, with varying degrees of deliberateness. But all of them have made names for themselves. Blogging is, perhaps, the most common vehicle and was frequently mentioned. But podcast appearances, user group participation, video creation, and speaking at conferences all factor heavily into the equation for the developer opportunists. They have made themselves known by marketing themselves.
— Erik Dietrich, Developer Hegemony

But guess what happens to your brand when you let someone else own your content? You are essentially creating a brand for the hosting platform. When I land on a Medium article, I don’t care who wrote it. I just learn that a lot of people write on Medium.

But when I land on a personal blog, I explore a bit. I check who is the author. Where does he work? What else did he write? I subscribe to his newsletter; favorite his blog.

How to own your content

I hope I was able to convince you that owning your content, rather than renting it to a third-party platform, is the better alternative. The next question you might ask is How. I can’t give you a definitive answer for every use-case. But I can provide some guidelines, mainly for written content.

Host it yourself

The reason people resort to big name platforms—is because they might lack the technical know-how of hosting their own blog. But since I posted this in my technical blog, rather than the second one, I assume you are a technical person. Hosting your own blog is not complicated. Writing your own platform—takes a couple of evenings, and there are existing platforms like hugo or Jekyll. And if you are really lazy, you can self-host WordPress.

And it’s cheap. Like $0 cheap. You can host it on Netlify (not affiliated). And if Netlify takes down your site, or it becomes expensive, you can move to any other place, because you own your content. By having your own domain, pointing to your own content, you will never lose SEO or branding due to movement from one hosting provider to another. I honestly don’t understand why people choose existing platforms—that lock your readers in their ecosystem, and end up owning your content—rather than self-hosting.

POSSE

POSSE? Yew, sounds gross! But stay with me for a moment. POSSE is an abbreviation for Publish (on your) Own Site, Syndicate Elsewhere. It’s a common term in the IndieWeb movement with a simple mission: because you self-host your content, you shouldn’t deprive consumers from your knowledge.

Our content is worth as long as it’s useable for our readers. And, as the Stoics believe, I can only control my actions. I can’t force everyone to go to my blog. Some people do like the ease-of-use of existing platforms. And they deserve to enjoy your knowledge as well.

So the premise behind POSSE is to always publish on your own website—for the benefits of owning your content, and building your brand. But syndicate this content to other platforms. And remember to always use the correct canonical URL, so search engines will prefer your self-hosted copy, over syndicated one. And platforms that do not support canonical URL—should cease from existing.

Final words

Owning your content concept extends way beyond blogs posts. But this post is already at ~1700 words, and I want to make it short and to the point. If you are interested in other forms of content ownership, and my approach to them—feel free to reach out to me via LinkedIn, Twitter, Mastodon, or Email.

”You are a lair!”

Yes I am. While I do believe in owning my content, I’m not without sins myself. My newsletter is still powered by Substack. However, I’m in the process of moving away from Substack.

I’ll use those last words as shameless self-promotion. I’m building a newsletter platform for the privacy centered people. People who care about owning their content, rather than participating in the creation of yet another big-tech social-network.

It’s currently under heavy development, and I share a lot of the process in my Twitter and Mastodon. I encourage you to follow me there. And if you don’t have any of them, then subscribe to my newsletter in order to be the first to know when it launches.

Own your content.

That might not always be true. Some countries might forbid you from re-encoding the files you own. Crazy, I know. IMNAL. ↩

How to auto-generate OpenGraph images

Fri, 28 Jul 2023 01:00:00 GMT

Ever seen those nice image previews when you share a link on Twitter, LinkedIn, Mastodon, etc? Those are called OpenGraph images, more on that in a minute. Creating those images, is however, time-consuming. Especially if you are focused on writing. I used to use Unsplash images, which are free, up until the moment I shared this very important issue, that I’ve opened on GitHub, to my LinkedIn followers.

The image you see when you share any GitHub issue—is auto generated, and contains the information about the issue itself. Ain’t this cool? We can do the same!

What is OpenGraph?

OpenGraph is a protocol that enables any web page to become a rich object in a social graph - https://ogp.me/

The idea was created, and popularized by Facebook, but in a nutshell—it’s a bunch of meta tags, that different platforms such as Twitter, or LinkedIn—can read, and generate preview cards based on them. They were initially made to index the content inside a graph, but we care only about the preview card.

You can visit the official website to learn more about all the meta tags. We, however, will focus only on one: og:image. And because back in the days when Facebook was popular, Twitter tried to compete with it, it’s no surprise that Twitter has its own set of meta tags. For our use case, we will focus on twitter:image meta tag. You can read more about Twitter tags here. Also keep in mind that Twitter can parse OpenGraph tags, in case Twitter tags are not present.

How to auto-generate the image

Meet satori. Satori is an extremely cool library that converts HTML and CSS into SVG images. One thing to keep in mind is that satori supports a sub-set of the CSS properties, so you can’t go too wild. Refer to this table to learn more.

satori also supports JSX, but I don’t see a reason to bring an entire Frontend framework, just to render one image. Instead, we will use satori-html which is an HTML adapter this will allow us to write plain HTML.

First, install the libraries:

yarn add satori satori-html

As an example, we will use my website’s OpenGraph image which can be found here. I’ll spit the code first, and then we will discuss it.

const markup = html`
  <div
    style="width: 1200px; height: 630px; display: flex; flex-direction: column; flex-wrap: wrap; justify-content: center; align-items: center; padding: 0 24px;"
  >
    <img
      src="https://www.yieldcode.blog/images/cover.jpg"
      width="500"
      height="500"
      style="width: 150px; height: 150px; border: 1px solid white; border-radius: 50%;"
    />
    <h1 style="color: rgb(17,24,39); font-size: 5.5rem; font-weight: 700">
      ${SITE_AUTHOR}
    </h1>
    <h2 style="color: rgb(82,82,82); font-size: 2.5rem; font-weight: 700">
      ${SITE_AUTHOR_TITLE}
    </h2>
    <h3 style="color: rgb(82,82,82); font-size: 1.5rem; font-weight: 400">
      ${SITE_DESCRIPTION}
    </h3>
  </div>
`;

const svg = await satori(markup, {
  width: 1200,
  height: 630,
  embedFont: true,
  fonts: [
    {
      name: "Inter",
      data: interRegularData,
      weight: 400,
      style: "normal",
    },
    {
      name: "Inter",
      data: interBoldData,
      weight: 700,
      style: "normal",
    },
  ],
});

The first thing we notice straight away, is that satori does not support classes, and instead requires the usage of style attributes.

The html` thingy comes from satori-html:

import { html } from "satori-html";

Handling fonts

The other thing we notice—is how fonts are handled. Since we do not render complete HTML page, fonts are not handled through the head tags. Instead, we need to load the fonts manually, and feed them to satori. In my case I use the Inter font in its regular and bold variations. This means that I need to get the font files and read them.

import fs from "fs/promises";

const [interRegularData, interBoldData] = await Promise.all([
  fs.readFile("./src/assets/fonts/inter/Inter-Regular.woff"),
  fs.readFile("./src/assets/fonts/inter/Inter-Bold.woff"),
]);

To SVG or not to SVG?

We could stop there, but Twitter being Twitter:

URL of image to use in the card. Images must be less than 5MB in size. JPG, PNG, WEBP and GIF formats are supported. Only the first frame of an animated GIF will be used. SVG is not supported.

So for the sake of compatibility, it would be best to convert the SVG to PNG. Meet sharp—a high performance Node.js image processing library. Sharp can do a lot of things such as: converting one type to another, resize, and rotate images.

The conversion is straight forward:

const png = await sharp(Buffer.from(svg)).png().toBuffer();

That’s it. What’s left now, is sending the png as a body of the response, together with the correct header: Content-Type: image/png. If you are familiar with express-like libraries, this should be easy for you.

Bonus - Astro

I <3 Astro. I think it’s a great framework to build static websites. And it’s not a surprise that all my websites, including this blog, are powered by Astro. It can be a bit tricky to add this functionality to Astro, so I decided to dedicate a section for this.

Astro has a notion of APIRoute. APIRoute is a away to generate non HTML pages, such as: RSS feeds, sitemaps, and images. Following our example image, we would like to create a new file, inside our pages folder, named og.png.ts. The content of this file is very simple:

import type { APIRoute } from "astro";

export const get: APIRoute = async function get() {
  // read fonts

  // created markup

  // generate svg with satori

  // convert svg to png with sharp

  return new Response(png, {
    status: 200,
    headers: {
      "Content-Type": "image/png",
    },
  });
};

And that’s it. Upon building your static website, Astro will also generate a file named og.png which will be the image that satori generated from your HTML markup, and sharp converted to PNG.

Bonus 2

It’s even possible to generate multiple images for an Astro collection. The same way Astro generates HTML pages from your collection, you can generate a cover image for each post. Suppose you have your posts inside src/pages/[slug]/index.astro, you can create a new file named src/pages/[slug]/cover.png.ts with the following content:

export async function getStaticPaths() {
  const posts = await getCollection("post");
  return posts.map((entry) => ({
    params: { slug: entry.slug },
    props: { post: entry },
  }));
}

export const get: APIRoute = async function get({ props: { post } }) {
  // do whatever you want with post - extract the content, excerpt, date, tags
  // load fonts
  // create html markup
  // create svg with satori
  // create png with sharp
  // send response
};

The `meta` tags

One really final touch—don’t forget to add the proper meta tags:

<meta property="og:image" content="https://www.yieldcode.blog/og.png" />

<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:image" content="https://www.yieldcode.blog/og.png" />

Why engineers should focus on writing

Sun, 09 Jul 2023 20:00:00 GMT

All engineers are good writers… of code. But I believe that in order to a become better engineer–you should improve your writing skills.

From the dawn of times, people were writing. We have written using symbols, like in Ancient Egypt. And we have written using letters, like in Renaissance times. And all of us, got at least one writing assignment in school, without the “Why?” And yet, today writing is so underrated, that most people want to avoid it. But the truth is–you will have to write. Comments, documentation, design documents, presentations. Whether you like it or not. So why not become better at it?

Why write?

Writing is a way to organize your brain

For some reason we, humans, think that knowledge equals reading stuff. I’ve read the starting guide for Rust, so now I know this topic. I’ve read a book about how TCP/IP works–so now I know this stuff. But it’s not true. If it was true, we’d all be super-stars.

What helps us cement our knowledge–is writing. That’s why I believe in writing code, as opposed to copying code snippets. Because when you type it–you cement that knowledge.

Writing is a way to learn something

If you want to learn a new topic–write about it. When I want to learn a new programming language, I write a short program using that programming language. When I want to understand how something works–I write an article on this blog about it.

Real writing, where you try to dig the truth, will require you to try and experiment. I’ve learned more by writing articles on this blog, than by reading programming books.

Writing helps you identify mistakes

How many times we are asked to prepare a design document, while we think to ourselves, “What for? The design is so simple, I can hold it in my head easily.” This is a big misconception, we all fall for. If we would actually write the design document, we would have identified so many problems with our “simple” design. Things like–inconsistencies, missing details, or simply “lazy” thinking our brain did, which in reality make no sense.

How to write more?

Remember–reading is a habit, writing is a skill. And in order to prefect your skill, you need to write more. One simple way to write more–is to approach design reviews differently. Instead of hating them, and doing them like a homework assignment, try to approach them with enthusiasm. Each time you write a design review, try to improve something. Make it shorter–without missing the point. Make it longer–in order to cover more use-cases.

Writing design reviews, and documentations–at your workplace–is an easy way to get into writing. You will have to do it anyway, so why not improve while writing? However, if you want to improve your writing further, outside your workplace, consider starting a blog. If writing a blog is intimidating for you, consider answering questions on Stackoverflow–but focus on providing textual content, rather than copy-pasting code snippets.

One last advice–abolish the copy-paste. So many developers, whom I mentored, simply copy-paste everything. Code snippets, function declarations, etc. I know how to initialize a git repository, because I do it by hand every time. Most people simply copy the instructions from GitHub or Google. And if you are scared about being unproductive–remember that you are not judged by how much code you write, or how fast you complete the assignment.

Writing is unavoidable. Whether you like it or not, if you want to become a 10x engineer, you will have to write. Write documentations, write design reviews, write presentations. The sooner you get better at writing, the faster you will become a better engineer.

An Essay on Burnout

Sun, 28 May 2023 15:00:00 GMT

14 years ago, I’ve got my first programming job. For 14 years, I worked mainly non-stop. I had some short breaks in-between jobs, but usually I’d end one job, and the next day would start a new one. I was literally waking up, and going to a different office the next day.

I really loved what I do, and couldn’t wait to start working in a new and exciting company. I was unable to understand people who took breaks in-between. I was thinking to myself—“Do people really hate their profession that much that they need to take breaks?” Moreover, any time someone mentioned the phrase “burnout”, I would cringe. I denied the existence of burnout as a concept. Sure, you could land on a bad manager, bad team, bad company, or exhaust your potential in a particular company—but those were perfectly fine reasons to move to a new workplace.

I saw every person who said that they burnt out—as weak. If you hate your profession, then maybe you should do a career change. But life’s a funny thing. It will eventually bring you down in order to teach you a lesson. And here I am—burnt out. Ah, the irony.

How did I get there

Before my last job as software engineer at Forter, I worked for almost 5 years at Autodesk. It was my first big corporate job, although the Tel-Aviv office, where I worked, felt more like a well funded startup. I came to Autodesk as a contractor, and eventually moved to an FTE. I occupied different positions such as senior software engineer, and tech lead. I interviewed people; I mentored people; I helped onboarding new hires; I worked on at least 5 different projects, and changed 2 amazing managers; and I had the opportunity to work with talented people.

But somewhere after 4 years, I was feeling that I reached the ceiling of my growth there, and it’s time to move on. It was also a crucial step for my career, because life gave me the book Developer Hegemony: The Future of Labor by Erik Dietrich, which I bought on 7th of October 2021, just 4 months before I’d find myself in a completely different company. It took me 17 days to finish this book, which for me, as a slow reader, is very fast. I recommend everyone who feels similar to what I’ll describe here—to read this book.

After about 2-3 month of interviews with many interesting companies, I found a job with Forter. I handed out my 30-day notice to Autodesk, officially finished my relationship with them on 31st of January 2022, and on 2nd of February same year—I had my onboarding day at Forter. Fun fact, I did not intentionally take one day of rest. If I could, I would start on the 1st of February, but Forter has two days a week when they do onboarding, and 1st of February wasn’t one of them. So I had one day of rest between jobs. Crazy me.

I was super excited to start my new journey. I really wanted to get into Forter, as they have a very strong engineering culture. Furthermore, I got into a new team with a new set of products, in a completely different market. A lot of new projects, and great opportunity to design, and architecture them from scratch. I’ve asserted my position as a strong engineer, with a clear desire to move up the career ladder.

Fast-forward to one month ago, I handed out my 30-day notice, and at the time of publishing this essay—I’m officially jobless. My wife and I sold everything, took essential items, and booked a one-way ticket to Costa Rica. I’ve officially burnt out.

What went wrong

Burnout is hard to diagnose and distinguish from general exhaustion or stress. People change jobs for different reasons. Sometimes they are offered a more interesting product, other times a more senior title. And sometimes it’s about money. Not every software engineer is motivated by passion—for some, it’s just a job that pays well. I get it. It is possible to make these people stay at the company rather than switch, but I’ll leave that discussion to a different article.

However, when I left Autodesk, I had different motives. I started to have a strong feeling of dissatisfaction. I had low energy, and lack of motivation. A feeling of exhaustion, as in “I have given everything I could, and I took everything I was able to take—and there is nothing else left for me there”. I started to distance myself from the company. I started to have dissatisfaction with the profession. I remember when I decided to quit, I had repulsion towards programming in both a professional environment, as well as a hobby. In my free time, I focused more on other hobbies such as reading, or writing—rather than coding.

I noticed that I start to fall behind in performance. Tasks that I would to ace, now take more time to complete. I was no longer excited about my job, and I felt tired from just commuting to the office, let alone being there. And I had a feeling that if I won’t do something about it, I’ll end up in a deep rut, that might even lead to depression. Looking at it now, I was burnt out at that moment.

Each single symptom does not necessarily mean that you have burnt out. However a full house might be a good indicator that you need to change something, either do a break or try a slight career change (for example moving away from high pressure FAANG like company, to something smaller).

However, I failed to recognize this in time, and I’m bad with communicating my feelings to my manager. So I ended up in a corner, where quitting was the only sensible solution to me. And so I did. I found myself in a new company, and the excitement came back. I was devouring knowledge, proving myself in a new environment. The passion to coding came back. I’ve given talks to the community, I’ve learned new tools such as vim and tmux in order to improve my workflow.

But quickly enough, I was back at the same place I’ve been at for the last 5–6 months in Autodesk. I got dissatisfaction with my work. I hated being in the office, and would feel sick there. I started to fantasize about having a different career such as: car mechanic, police detective, movie actor, podcaster, DJing or music production. Anything, but please not writing code anymore. Eventually it grew into apathy, and it took me a lot of mental energy just to care.

I didn’t plan to quit, but life gave me an opportunity I couldn’t pass. Looking in retrospective, I probably had burnout by the time I left Autodesk. Switching jobs always creates this feeling of newness. But your wounds remain untreated.

What causes burnout

It’s hard to understand why burnout happens. In the end, some people stay for 10, 15, or even 20 years in a company. They do, usually, move through the career ladder, but nevertheless they stay at the same company. However, I believe there are two main factors that contribute to burnouts.

Hustle

Work, today, is glorified. What do you do in life?—is often the first question we ask new people we just met. And you better have an answer other than just traveling and enjoying life, or you will be labeled as lazy, unmotivated, indolent child. We live in a hustle culture. If you don’t have a job—you are an immature adult.

Hustle culture is on the raise. Wherever you look, you get the message that you need to hustle. Become someone. Work harder than anyone else. Stay up late. Show up earlier. In the end you will be rewarded for your hard work. Truth is—meritocracy does not exist. I used to believe it does, but the ones who are getting promoted, are rarely the ones who work the hardest.

Toxic positivity

We also live in an era, of what I call, toxic positivity. We tend to avoid negative emotions, and try to avoid people who share their struggles. While I never felt it directly, its societal existence had an impact on me. The desire to run away from negative emotions, leads us to believe that the problem is external. It’s not me, who burnt out, but rather the company that is not valuing me, promoting me, appreciates me, etc.

Moreover, there is a societal judgment toward the ones who make good money, by doing, what seems like, an easy job, in-between playing ping-pong, and drinking beer. You often hear, or read phrases like “oh those poor developers who complain about the fact that have to carry their shiny MacBooks on a Saturday evening, because they are on-call—there are people who are stuck in a dead end jobs, getting paid minimum wage.” And I agree, the latter is terrible. But it does not deduct any pain, or suffering, from me. My feelings can’t be ignored because somebody has it worse than me. No one should have his, or her feelings—ignored. That’s not how we create a healthy society.

And so we are not allowed to feel bad, or express struggle. And we find someone else to blame—an external entity, such as an employer. Therefor, the solution is also external—change job. By changing jobs, I’ll get a fresh start to prove myself, and get what I deserve. And I better make the change ASAP, without any breaks in between, because I don’t want to be labeled as lazy, unmotivated, and indolent child who is just enjoying life, while figuring what to do next. Yikes!

But in the end, work is just work. Recent layoffs, and economic recession, proved—once again—that people, at large, are a disposable resource. When times are good, we will be partying in Hawaii and posting inspiring pictures in LinkedIn, promoting our fun office vibe. But when times are thought, we will be hoping that we won’t be laid off, because being able to pay mortgage is, apparently, way more important than drinking Piña colada in Hawaii.

Tech Industry

Apart from those two factors, there are other, unique, factors specific to the tech industry. Because this industry is extremely fast-paced, this leads to a situation where one could be working on a project with no clear definition. Companies like to put can work in an uncertain, fast-paced environment, in their job descriptions, and while it is ok to have such environment for a limited period of time—if an individual works on an unclear goal, or his project gets canceled every time, this might create a situation where he or she starts to feel disengaged, and unfulfilled at work.

And this puts on a lot of stress on the individual. He can’t fulfill himself at work, which in turn leads to a feeling of emptiness, which slowly creates detachment from work. On a professional level, he could feel stress because on the upcoming performance review he has nothing to show, since his project got cancelled again.

Apart from the uncertain environment, and unclear goals, another stress factor could be the abundance of meaningless rituals like daily standups, weekly retrospectives, and monthly status updates—which create a mental fatigue that feeds this cycle of inability to do any work, which leads to stress, and eventually burnout.

How to prevent burnout

I remember we had a Tech Lead forum at Autodesk, where all tech leads would gather, on a weekly (or it was bi-weekly) basis, and discuss tech lead stuff. Sometimes it would be struggles from inside the teams, other times we would talk about software architecture.

I remember one session where somehow we came to the topic of problem prevention versus problem solving, and the entire forum got split into one of two camps. Long story short, I don’t believe in prevention. This is simply not how life works, in my experience. However, I can shed some light on how to identify and cure burnout.

One pretty accurate model, that I read about on HackerNews, says that there are 5 phases to burnout. First, you have the honeymoon phase. You join a company, everything is pink and chocolate. Then comes the realistic phase—you realize that not everything is good, and there are bad sides as well. The next phase is team-isolation phase—my team is great, but everyone else is stupid/lazy/ignorant. The fourth phase is self-isolation—I’m doing my best, but everyone else is stupid/lazy/ignorant. And the last phase is apathy—I don’t care anymore.

This model was described by sz4kerto on HackerNews, and I think it’s very accurate, based on my experience. That’s exactly what I felt when I quit Forter. I remember the apathy. I just scheduled 1:1 with my manager and told him that I quit. No emotions. Funny thing is, he told me he had a feeling. When I asked him why, he told me I had an attitude change.

Each stage by itself does not indicate that you are going towards burnout. It is possible to feel that your team is good, but other teams are not supportive. It could just mean that you’ve ended up in a bad company, and not that you’ve burnt out. And it is only when you look back, and you are able to identify all the 5 stages—you realize that you have burnt out. That’s why it’s so hard to prevent burnout.

At that point, I’d recommend a change of some sort. Maybe a break, or downgrade to a less demanding job. Also, try to take care of yourself. More often than not, burnout at work, also leads to deterioration in your day-to-day life. You eat shit, because you are working all the time (or try to appear like you are working). You don’t take care of your physical state. And during any free time you get, you feel exhausted, and unable to do any hobbies. Your mind is occupied by the future thought of going back to work tomorrow, or on Monday. Stress levels are also high, so read about stress management, and apply it.

If you are unable to take a break, and don’t want to or can’t find a less demanding job, then maybe you do need a career change.

And please talk. Talk to your manager. Talk to your friends and family. Reach out to a professional therapist if you can’t handle it yourself. Unresolved burnout can lead to depression.

What is next

For the first time in my adult life—I’m jobless. I got my first real job when I was still in college, and since then I’ve been working. It’s a weird feeling, to not having to go to work. And there is a strange post-traumatic syndrome. I remember a day or two after I was already jobless, my wife asked me if I want to go watch a movie midday. My first reaction was a mini panic, as-in: “How am I going to disappear from work for 2 hours, in the middle of the day?”. Work is an important part of our life.

As for me, I’m going to go inwards. The problem was never Autodesk, or Forter, or the industry. The problem—is me. And accepting that is not a shame. I’m going to focus on projects that excite me. I have few interesting ideas for tools for developers. I’m going to focus on this blog, as well as my second blog—Jiko Kaizen. I’m going to focus on my health, both physical and mental. And I’m going to spend time with my wife, exploring together new places. I’ll see where all this takes me. I might return to a traditional workplace, however I plan not to.

I lived in denial, looking from above at all those people who burnt out. In denial of the existence of burnout. But it’s real. And I want to raise awareness to that. You, yes you! You might be experiencing burnout right now! But the societal norm—or the story you created to yourself, about yourself—are preventing you from realizing it, because you need to be strong, quit whining, and work hard to become someone in this world. But the truth is—you don’t. You deserve to be happy.

Stay safe. Take care of your mental health. Nos Vemos!

The need for a more semantic web

Fri, 17 Feb 2023 08:00:00 GMT

The web has come a long way. From the early static pages, through the evolution of reactivity, and towards the biggest knowledge pool for all of humanity.

A little history of the web

The World Wide Web (WWW), which is commonly known as the Web, was invented by Tim Berners-Lee at CERN in 1989. It’s an information system that enabled documents and other web resources such as—images, videos, music, etc—to be accessed over the internet.

I, myself, being an early user of the web, saw it transform through different generations and paradigms. I like to call them Web 1.0, Web 2.0 and Web 3.0. Let’s explore the unique characteristics of each of those eras.

Web 1.0

When I talk about Web 1.0, I refer to the early state of the Web. The state where there was practically no standard and no interactivity. Web pages, back then, used to be completely static and written entirely in HTML (which was poorly defined as a standard back then). CSS was non-existent (it was created later, in 1996). JavaScript, which appeared in 1995, was also unheard of.

This was the era of the static, generic looking, web. This is how the first ever website looked like.

First ever website#1424x546

Go on, check the source of it.

Web 2.0

With the birth of CSS and JavaScript, the world entered a new era of the Web. The Web 2.0. This web provided more colors. This web provided interactivity.

Suddenly, people started to care about separation between structure (the HTML side) and appearance (the CSS side). HTML elements such as <font>, <marquee>, and <strong>—became deprecated, and were replaced with their CSS and JS counterparts.

The Web 2.0 also saw the birth of the famous XMLHttpRequest and the Ajax technology. You could replace an entire webpage, or parts of it, without navigating away from the page—which created a feeling of what today is known as SPA—Single page application.

On the HTML side, we saw the appearance of some semantic attributes such as <article>, <section>, <aside>, and others. Finally, in 2008, HTML5 was released, which among many things, added a support for Microdata. Remember this, as we will talk about Microdata later.

With the rise of the Web 2.0, came the rise in content published in the World Wide Web. Suddenly it became easier, cheaper, and more accessible to have a web page. People, and companies, started to care about web presence. And more over—the Web became a place of knowledge. This era gave birth to concepts such as blogging and forums. In 2001—Wikipedia was born. And as the amount of content grew, the need to find content became a necessity. In 1998 Google was created, and later became the dominant search engine till this day.

Web 2.0 was a great era. An era of revolution in the way we present our webpages, as well as revolution in the way the Web is being used. But something happened, which ended the Web 2.0 era, and Web 3.0 became the new kid in the town.

Web 3.0

When we hear the term Web 3.0, most of us think about decentralized web, Bitcoin, NFTs, and blockchain. But in my opinion those technologies are unrelated to the Web. The Web was, and still is, a decentralized system. Yes, it is controlled by few major corporations. But nothing forbids me to buy a server, put it in a rack at my home, and plug in into the internet in order to serve my blog from it.

For me, Web 3.0 means the birth of the semantic web. Somewhere around the peak popularity of Facebook, the Web stopped being about webpages, and started to become more about things. When I search for @skwee357, I don’t want to find webpages that contain this obscure phrase/word. I want to find a person behind this handle (which is me). When I search for “Fight club”, I want to find the movie by David Fincher. The search term, “how to make pancakes”, should ideally yield a set of recipe results for pancakes.

Web 3.0 is about things. It’s no longer about webpages and content. It’s about people, movies, recipes, tutorials, places, events, books. And the current state of things with Web 3.0—has one drawback.

The birth of the semantic web

As I’ve noted earlier, during the Web 2.0 era, HTML got some new elements that aimed to make it more semantic. Here is a list of some of them. But have you ever wondered what’s the different between a <div> element and an <article> element? <span> vs <div> is a simple answer. The first does not create a new block, and can be incorporated inside a sentence without breaking it. But most of the other elements, such as <article> or <header>, do not have different behavior than say a <div>.

They do serve one major role—to help make the web more semantic. With the increased usage of the Web, came increased production of web content. And the latter created a demand for increased searching and indexing capabilities. And these semantic HTML elements—help search engines to classify your content. When a crawler sees an <article> element—it knows that the text, inside of it, is indeed an article. Compare that to a generic <div> which does not provide any information about the nature of it’s content. It could be an article, but also could be a table of contents.

Have you noticed how Google sometimes provide a snippet from Wikipedia, or recommend you a video from YouTube or Vimeo? How does Google, or any other search engine, knows to classify between a general section on your web page, and the content of the blog post? Probably by looking at the <article> HTML element. But there is more. How does a search engine know that “Brad Pitt” is an actor, “Fight Club” is a movie, and “1984” is a novel?

Semantic semi standards

Like with any new and established standard, someone first created a semi-standard. And this is what happening to Web 3.0 now. I believe that Facebook was the first to create the og meta attributes to classify content. A webpage with correct <meta> tags with og attributes—will provide semantic information to those who parse the og meta attributes. One could provide information such as site name, description, associated image, etc. The og attributes are responsible for content sharing. When you post a link to Facebook or LinkedIn, those platforms query the og attributes in order to extract the title of the webpage, description, as well as cover image.

But meta attributes are limiting. And so other formats were created. I believe the first among them was RDF—Resource Description Framework. RDF never gain much popularity, and was served as a foundation to newer semi-standards such as JSON-LD and Microdata, which I mentioned earlier when I wrote about HTML5. Those two formats took a different approach.

Microdata

Microdata was targeting to enhance existing HTML elements. By using Microdata, one can provide a set of special attribute to any HTML element. Combining with a set of predefined schemas, such as event, product, and review—one can provide additional context to crawlers about the nature of the content on a webpage. A geolocation with Microdata will look like this:

<div itemprop="geo" itemscope itemtype="https://schema.org/GeoCoordinates">  
	<span itemprop="latitude">52.48</span>
	<span itemprop="longitude">-1.89</span>
</div>

Interestingly enough, according to MDN, Microdata support was removed from Firefox 49. According to Wikipedia—no major browser support the Microdata DOM API. Google developers portal tells us that Google, still, does care, and parse, Microdata.

Microformats

Somewhere around the time Microdata became a thing, another semi-standard, that’s called Microformats—was created. Microformats took a similar approach to Microdata, but instead of relying on a good set of schemas to classify things (like the above event, product, or review I’ve mentioned), it went on to define custom CSS classes. So a geolocation would be marked as:

<span class="h-geo">
	<span class="p-latitude">52.48</span>,
	<span class="p-longitude">-1.89</span>
</span>

This approach created some problems. For starters, there was no schema definition for thing. A crawler would have to implement all the h-* and corresponding p-* classes. Extending Microformats was practically impossible. In addition to that, the usage of CSS classes to classify content semantically—feels wrong.

JSON-LD

Lastly, we need to talk about JSON-LD. JSON-LD is also a semantic web format. But instead of trying to build upon existing convention—like HTML element attributes, or CSS classes—JSON-LD relies on encoding your entire semantic information, in a script tag. Here’s how it looks for the homepage of my blog:

<script type="application/ld+json">
{
	"@context":"https://schema.org",
	"@type":"Blog",
	"@id":"https://www.yieldcode.blog",
	"name":"yield code();",
	"description":"Thoughts and stories on programming, the industry, and technology written by a software engineer",
	"author":{
		"@type":"Person",
		"@id":"https://www.kudmitry.com/",
		"name":"Dmitry Kudryavtsev",
		"sameAs":["https://www.kudmitry.com/","https://linkedin.com/in/kudmitry","http://github.com/skwee357","http://twitter.com/skwee357"],
		"image":{
			"@type":"ImageObject",
			"url":"https://res.cloudinary.com/dvp7olno1/image/upload/v1673035861/yieldcode/cover_eacuil.jpg",
			"height":"150",
			"width":"150"
		}
	},
	"keywords":["programming","software industry","software engineering","tutorials","programming consulting","tech lead","architect"]
}
</script>

JSON-LD is one of the most popular semantic web formats, and it is the one recommended by Google. RDF and Microdata never became popular. Microformats has a small community among IndieWeb people, but it’s hardly a specification. It’s a gross hijack of CSS classes—that were created to style the HTML elements—not provide semantic information.

Making Web 3.0 great

HTML is an established format and specification. By looking at this webpage, you already have most of the semantic information you need. It makes no sense to duplicate the same semantic information inside a JSON in the HTML <header> tag, for the sake of crawlers. This creates unnecessary duplication, which in turn makes the webpage heavier.

However, with the rise of Web 3.0 and the content era, we have to help crawlers understand our content better. The web needs to become cleaner for machines, so it will become cleaner for humans. If we introduce HTML elements such as tag, author, recipe, and others—we can create a better web for machines, which will help to create a better web for us—humans.

Call to action

Web Hypertext Application Technology Working Group, also known as WHATWG, is a community of people interested in evolving the HTML. About a month ago, I’ve created a proposal at WHATWG Github: Elements for a more semantic web.

The web is built on open formats. The evolution of HTML is needed in order to keep up with the changing nature of the Web. I hope I was able to convince you that we need to move HTML forward, and adapt it to the new reality of the Web 3.0—the semantic web. Failing to do so, will force us to use formats that were created, and controlled, by corporations—better built to serve their needs, rather than ours. In addition to that, by failing to provide semantic information to crawlers, we will either have to place our hope in the hands of AI classification tools (which are far from optimal), or worse—drown in a sea of search results that make no sense.

If I did convince you, then please go to my proposal and show your support. Together we will move HTML to its next level.

Inheritance, Composition, and everything in–between

Wed, 25 Jan 2023 11:00:00 GMT

There are many concepts in the OOP paradigm. Inheritance is the most known, and it allows us to model software as relations between objects, just like in real life.

But it has some drawbacks, which alternative paradigms can fix. Let’s dive into OOP, inheritance, and everything in–between.

Object-oriented programming

Object-oriented programming, or OOP in short, is probably one of the most popular programming paradigms. It’s an easy paradigm to understand, because it follows the way we, humans, are seeing the world.

For us—the world is a collection of objects that can be modeled in software. Let’s say a car. A car consists of the engine, a set of tires, a chassis, a steering wheel, seats, etc. In code, this would look something like this:

class Engine {
	function goVroom() {}
}

class Tire {}

class SteeringWheel {}

class Car {
	private engine = new Engine();
	private tires = [new Tire(), new Tire(), new Tire(), new Tire()];
	private steeringWheel = new SteeringWheel();
}

Inheritance

We can’t talk about OOP without mentioning inheritance. Let’s look back at our car example. Say we want to output the parameters of our car. For the sake of simplicity, let’s omit the Engine for now.

class Tire {
	private radius: number;

	public function getInfo() {
		return `Tire Radius: ${this.radius}`;
	}

	get circumference() {
		return 2 * Math.PI * this.radius;
	}
}

class SteeringWheel {
	private radius: number;

	public function getInfo() {
		return `Steering Wheel Radius: ${this.radius}`;
	}

	get circumference() {
		return 2 * Math.PI * this.radius;
	}
}

We can notice some repetition in there. If only there was a way to specify that both the Tire and SteeringWheel are forms of a Circular object. Welcome—Inheritance.

abstract class CircularObject {
  constructor(protected radius: number) {}

  get circumference() {
    return 2 * Math.PI * this.radius;
  }
}

class Tire extends CircularObject {}
class SteeringWheel extends CircularObject {}

Inheritance allows us to define common behavior and functionality, as well as common interface. We can enforce, through inheritance, that all object that inherit from CircularObject will have a radius property, and a circumference property, which is calculated. I’m not going to go too deep into inheritance in this post, though.

Where inheritance fails

Inheritance, however, has one big pitfall—multiple inheritance.

Let’s say we have the following code:

abstract class Fruit {}
class Plum extends Fruit {}
class Apricot extends Fruit {}
class Pluot extends Plum, Apricot {}

NOTE

Pluot is a real crossover between plum and apricot. It has high sugar levels and extraordinary sweet taste.

If you try to “compile” that code, you will get an error:

Classes can only extend a single class.(1174)

And it’s worth mentioning that this is not some weird constraint of a Typescript compiler. Apart from C++ (and apparently python), no other major language supports multiple inheritance. Why you ask? There is one problem with it.

The diamond problem

One of the biggest drawbacks of multiple inheritance, is that it can lead to one particular problem knows as—the diamond problem. Let’s try to visualize the above inheritance as a relations graph:

	Fruit
	 /\
	/  \
   /    \
 Plum  Apricot
   \    /
    \  /
     \/
    Pluot

As you can see, this inheritance creates a diamond-like shape. The shape itself is not the problem, but the problem lies in an underlying implementation ambiguity one can introduce. Imagine that everything we inherit from Fruit, needs to implement an abstract method called getColor:

abstract class Fruit {
	public abstract getColor(): string;
}

class Plum extends Fruit {
	public getColor() {
		return 'purple';
	}
}

class Apricot extends Fruit {
	public getColor() {
		return 'yellow-orangish';
	}
}

class Pluot extends Plum, Apricot {}

Since Pluot extends non-abstract classes, it is not obliged to implement the getColor method. Assuming multiple inheritance was possible in Typescript, what would be the output of:

const pluot = new Pluot();
console.log(pluot.getColor());

It’s impossible to know. This ambiguity is hard to mitigate. In complied language—it’s usually forbidden at the compilation level, either via explicit error during class inheritance definition, or during invocation. The latter enforces the invocation to be explicitly cast to a specific type, for example:

Pluot pluot = new Pluot();
System.out.println((Apricot)pluot.getColor()); // yellow-orangish

NOTE

Java does not support multiple inheritance either, so this code is for demonstration purposes only.

Another approach some languages take—is a compile time requirement to re-implement the method in the grand-child class, thus eliminating the ambiguity.

However, in case of dynamic languages—nothing guards your types. In–fact, the JavaScript runtime doesn’t know about your Apricot and Pluot objects. All it sees are Object types with a function on their prototypes. In this case, Typescripts compiler is the one guarding you. And Typescript, as well all know, is just a syntactic sugar on top of JavaScript.

But what about interfaces?

Apart from abstract classes that can contain both interface and functionality, there are also what is known as Interfaces or Protocols. An interface is just a definition of an agreed upon API. It defines the API, but does not allow implementation in the body of the interface. It’s up to the object that implements the interface—to provide the actual implementation.

Therefor, it’s possible to create a diamond like relation graph with interfaces. Consider the following example:

type Target = ...;

interface Clickable {
	getTarget(): Target;
}

interface Moveable {
	getTarget(): Target;
}

class Ball implement Clickable, Moveable {
	getTarget() {
		const target = ...;
		return target;
	}
}

Since neither Clickable, nor Moveable provide the actual definition, it is perfectly fine that the ambiguity exists, and it does not create any problems, since the compiler has no confusion.

The same is true for this scenario:

interface Moveable {
	move();
}

class Ball {
	move() {
		...
	}
}

class Basketball extends Ball implements Moveable {
}

console.log((new Basketball()).move());

This code complies and runs perfectly, because from the view point of the compiler, Basketball conforms to the interface Moveable since it inherits the move method from Ball.

There are some more interesting cases, such as: what happens if the signature of move in the Moveable interface—is different from the move in Ball? Would you be able to both extend from Ball and implement Moveable? Which version of move then will be called?

I don’t want to spend more time on OOP and inheritance. Drop me a message/email (contact details down below this post) if you are interested to read more about those weird cases in Typescript. For now, let’s move on.

Mixins

As we’ve seen earlier, multiple inheritance is a big no–no. However, there are legit cases where we want to inherit functionality from multiple classes. A great example is the game development industry. It’s common to think of game objects as, well, objects. Let’s say we are trying to build a modular system of NPCs (Non Playable Character—represents a game character that is controller by the game/AI and not by the player).

We have some NPCs that can move. Some NPCs can take damage. Not all NPCs are going to take damage. The local town potion seller—should not be attackable by the player. An evil orc or mimic chest—should be able to take damage. The chest however can’t move, because it’s, well, a chest.

I bet in your head you already have an inheritance graph. Let’s try to model it.

class Moveable {
	constructor(private positionX: number, private positionY: number) {}

	public moveTo(x: number, y: number) {
		this.positionX = x;
		this.positionY = y;
	}
}

class TakesDamage {
	constructor(private health: number) {}

	public takeDamage(damage: number) {
		this.health -= damage;
	}

	public isDead() {
		return this.health <= 0;
	}
}

class MimicChest extends TakesDamage {}

class PotionsSeller extends Moveable {}

class EvilOrc extends TakesDamage, Moveable {}

Oh-no! The evil orc is trying to do multiple inheritance. Even though it does not introduce any ambiguity, like we’ve seen before, multiple inheritance is still forbidden. What can we do?

Meet—Mixins. Mixin is a way to have reusable code, like our Moveable and TakesDamage classes, without having the end class to inherit both, hence avoiding the diamond problem.

A common way to implement a mixin, is by creating functions that return an anonymous class. In our case, it will look something like this:

type Constructor = new (...args: any[]) => {};

function Moveable<TBase extends Constructor>(Base: TBase) {
  return class extends Base {
    _positionX = 0;
    _positionY = 0;

    moveTo(x: number, y: number) {
      this._positionX = x;
      this._positionY = y;
    }

    get x() {
      return this._positionX;
    }

    get y() {
      return this._positionY;
    }
  };
}

function TakesDamage<TBase extends Constructor>(Base: TBase) {
  return class extends Base {
    _health = 0;

    takeDamage(damage: number) {
      this._health -= damage;
    }

    get health() {
      return this._health;
    }

    get isDead() {
      return this._health <= 0;
    }
  };
}

class NonPlayableCharacter {
  constructor(private name: string) {}

  getName() {
    return this.name;
  }
}

const EvilOrc = Moveable(TakesDamage(NonPlayableCharacter));

const orc = new EvilOrc("sluggish");
orc._health = 10;

console.log(`Orc ${orc.getName()} is located at ${orc.x},${orc.y}`);

orc.takeDamage(20);

console.log(
  `Orc ${orc.getName()} took 20 points of damage and is now ${
    orc.isDead ? "dead" : "still alive"
  }`,
);

The output of this program will be:

Orc sluggish is located at 0,0
Orc sluggish took 20 points of damage and is now dead

And as you can see, our EvilOrc class was able to inherit from two classes! There are some drawbacks, as you probably noticed. For one—any mixin must have a generic constructor with ...args. Attempt to add a non-generic constructor to a mixin will result in error TS2545 that says:

A mixin class must have a constructor with a single rest parameter of type ‘any[]’.

So all the benefits of hidden fields, or initialization—are thrown away. One way to at least remove some code duplication, is to have factory methods:

type Vector2D = { x: number; y: number };

function createOrc(name: string, initialHealth: number, initialPos: Vector2D) {
  const orc = new EvilOrc(name);
  orc._positionX = initialPos.x;
  orc._positionY = initialPos.y;
  orc._health = initialHealth;

  return orc;
}

const orc = createOrc("sluggish", 10, { x: 10, y: 10 });

But data access is still not protected. Fields can’t be marked as private, because then you lose access to initialize them, and need to provide a setter method—which in this case beats the entire purpose of making the field private in the first place.

However, if you have common functionality that you want to share to between multiple objects, and don’t care much about encapsulating the properties—then mixins are a great way to do that.

NOTE

There is a way to define mixin through prototype inheritance. It’s ugly. But if you want to take a look, there is an example in the Typescript Handbook.

As we’ve seen, with mixins you lose the encapsulation, and more over, you can encounter weird bugs like prototype overrides! Consider the following example:

type Constructor = new (...args: any[]) => {};

function Plum<TBase extends Constructor>(Base: TBase) {
  return class extends Base {
    getColor() {
      return "purple";
    }
  };
}

function Apricot<TBase extends Constructor>(Base: TBase) {
  return class extends Base {
    getColor() {
      return "yellow-orangish";
    }
  };
}

class BasePluot {}
const Pluot = Plum(Apricot(BasePluot));
const p = new Pluot();
console.log(p.getColor());

What would be the output? The correct answer is purple. However, if we switch the order if the mixin composition from Plum(Apricot(BasePluot)); to Apricot(Plum(BasePluot));, the output now changes to yellow-orangish!

And what if we want to define a color for our Pluot?

class BasePluot {
  getColor() {
    return "red";
  }
}

You won’t be able to access it, since mixins are based on prototype overriding.

Last, but not least—have you noticed that we need to have 2 classes for every object we want to construct? In order to create a Pluot, which is composed of Apricot and Plum, we need to have both the BasePluot class, and the result of the mixin composition call. And how the heck do we create an instance of Plum or Apricot? That’s how:

class ApricotBase {}
const ApricotClass = Apricot(ApricotBase);
const a = new ApricotClass();

Such a mess. Imagine testing a mixin, you will need to define a dummy class just to be able to instantiate the mixin itself. Mixins are better reserved for small pieces of reusable code that needs to be composed. In a real scenario, like a game—they are hard to maintain.

Composition

Lastly, we come to Composition. Composition, as the name implies, is used to compose objects from reusable code pieces, rather than leveraging inheritance. Mixin is a form of composition, but it still relies on inheritance to combine shared functionality into one class.

You’ve probably used composition many times in your development, without even knowing it. The Car example at the top of the article uses composition, to compose one specific object—the Car, using other reusable objects like Tire, Engine, etc. But this composition approach does not help us with our EvilOrc example. We could do something like this:

class Moveable {
	...

	moveTo(pos: Vector2D) {
		...
	}
}

class EvilOrc {
	private moveable = new Moveable();

	moveTo(pos: Vector2D) {
		this.moveable.moveTo(pos);
	}
}

But it’s ugly, and repetitive.

When building an architecture around composition, we no longer think of objects and their properties—as inheritance tree. We don’t say that an Orc is a Moveable object, and an object that can take damage. Instead, we say that an Orc has the ability to move, and to take damage. It’s a small change in wording, but a totally different concept. Let’s try to redefine our Moveable and TakesDamage classes, to better suit composition.

First, we need to define some utility types:

abstract class Component {}
type ComponentFunction = Function;
type ComponentClass<T extends Component> = new (...args: any[]) => T;

With composition, we need to distinguish between the composable objects, and the composed objects. In our case, Moveable and TakesDamage are reusable classes that have no meaning outside a context of a game object. We can create an instance of Moveable class, but it has no meaning by itself. Therefor, we will separate those containers of common logic, into what I’ll call a Component. Hence, the abstract Component class. The two types are used for some type inference, and their role will be clear in a bit.

Next, let’s rewrite our Moveable and TakesDamage as components that will be composed, and at the same time, rename TakesDamage to Health:

class MoveableComponent extends Component {
  constructor(private position: Vector2D) {
    super();
  }

  public moveTo(pos: Vector2D) {
    this.position = pos;
  }
}

class HealthComponent extends Component {
  constructor(private health: number) {
    super();
  }

  public takeDamage(damage: number) {
    this.health -= damage;
  }

  public isDead() {
    return this.health <= 0;
  }
}

Now, we marked some classes as components. Before we can finally define our EvilOrc class, we need one more utility class:

abstract class GameObject {
  private components = new Map<ComponentFunction, Component>();

  protected addComponent(component: Component) {
    this.components.set(component.constructor, component);
  }

  hasComponents(components: Iterable<ComponentFunction>): boolean {
    for (const c of components) {
      if (!this.components.has(c)) {
        return false;
      }
    }
    return true;
  }

  getComponent<T extends Component>(componentClass: ComponentClass<T>): T {
    return this.components.get(componentClass) as T;
  }
}

This class will serve as the base class for all our game objects, and it has a list of components. Hence, composing them into a game object. Here, the types ComponentFunction and ComponentClass are used for some Typescript magic.

Lastly, we need to define our EvilOrc and MimicChest:

class EvilOrc extends GameObject {
  constructor(initialPosition: Vector2D, initialHealth: number) {
    super();
    this.addComponent(new MoveableComponent(initialPosition));
    this.addComponent(new HealthComponent(initialHealth));
  }
}

class MimicChest extends GameObject {
  constructor(initialHealth: number) {
    super();
    this.addComponent(new HealthComponent(initialHealth));
  }
}

And voilà! As I’ve said earlier: EvilOrc is not a Moveable and Health object. Instead, EvilOrc has the ability to move, and has health (and hence—ability to take damage, in our case).

Okay—you say. But what do I do with this? How do I operate on this? Let’s say we are trying to build a sequence that will perform an attack by the player, on a NPC. This might look something like this:

function performAttack(go: GameObject, damage: number) {
  if (!go.hasComponents([HealthComponent])) {
    throw new Error("game object does not have health component");
  }

  const h = g.getComponent(HealthComponent);
  if (h.isDead()) {
    throw new Error("game object already dead");
  }

  h.takeDamage(damage);
}

It’s a very modular system that allows you to compose all kinds of behavior. It’s widely used in game development, and is knows as ECS—Entity Component System. I was never able to find use-cases for it outside of game development, but recently I had the chance to use a similar implementation at my work. I thought it’s worth sharing this with you, and maybe you can find a good use cases for it as well.

Imagine for example that you want to build a highly reliable weather service. It has to depend on multiple external weather APIs, because you want to fallback to a different API if one is down, or takes too much time to answer. But not all weather APIs might support the features you need. One might support temperature and wind conditions; another one will support temperature and precipitation; while the last might support all of them: temperature, wind condition, and precipitation. This example might be a prefect use case for the aforementioned composition. The implementation of such system, using composition—will be clean and easy to maintain.

Remember—good developers have a lot of tools, and try to use the best one for each scenario. Happy coding!

How to get into programming in 2023

Thu, 05 Jan 2023 13:00:00 GMT

It’s 2023 already, you’ve been a responsible individual and created a new year’s resolution. One of the items in your resolution is a career change to become a software engineer. So how do you start?

Becoming a software engineer is 2023, is no different than in 2022, or 2021. But for some unexplainable reason—new year is considered to be a good reset point when we decide to pursue new endeavors or set goals. It doesn’t matter if you are fresh out of school, or looking for a career change after many years in a particular industry—becoming a software engineer is a great way to up your skills as well as pursue a new career.

Let this be your guide on how to make your first steps towards your desired change. I’m not going to teach you programming, or telling you what path to pursue among the many. Instead, I just want to give you my personal recommendations on what to focus on during your learning.

Ain’t programming dead?

Haven’t the demand for programmers declined rapidly? Aren’t the AI going to replace programmers?

No, and no. The end of 2022 was rough on the industry, with massive layoffs in many companies. But developers are still needed. As our world becomes more and more dependent on software, the need for good software engineers will persist. And as for AI, I’ve blogged earlier why I don’t think ChatGPT will replace developers any time soon.

Now that we’ve cleared out those two misconceptions, let’s discuss all the possible paths you can take, in no particular order.

The gamer

So you want to be a game developer? This is a very popular and demanding path. Many newcomers to this profession want to build games. In fact, the first serious things I’ve built when I was learning C in school—are games. Conway’s game of life and snake, to be exact.

Game development is a very broad umbrella term. It covers many disciplines such as: audio programming, graphics programming, physics programming, AI, game logic scripting, etc. Each of these disciplines can be their own career path. I’d say that graphics, AI, and game logic are the most interesting, but you might have interest for other things.

However, if you really want to become a game developer, I would suggest you focus on building games. Many beginning game developers are getting caught in the low level stuff. How to build a game engine, how to draw triangles, etc. And while this is fun, it does not bring you closer to actually building games. The ability to finish and ship a product is more important than building a game engine, which chances are you won’t be doing in your first years of this career path anyway.

The easiest, and fastest way to build games today—is to use a ready engine such as Unity. You will be able to utilize the amazing building blocks provided to you by Unity, while focusing on building your game. Development in Unity is done mainly in C#, which is a good overall programming language to learn. Unity can be used to build desktop and mobile games, and many AAA titles are built with Unity (I’m not an avid gamer, but Pillars of Eternity is one that comes to mind).

However, even though Unity abstracts a lot of the basics of graphics, audio, physics, and AI concepts—you will still need to know and understand them. And most of them are based on math and algorithms. Ray casting, path finding, vectors, quaternions. All these words will need to become your best friends. Also keep in mind that the game development industry is known to be a tough path. Crunching is real, and according to international game developers’ association survey, done in 2019, 41% of respondents indicated that their job involved crunch time, with 38% indicated working between 50 and 59 extra hours per week ¹.

The mobile developer

Smartphone took over the world. It became the main device through which we consume media. Even my blog was visited primarily (51%) via mobile device, in the past 12 months. And so becoming a mobile developer, could be a viable career path to choose.

In order to get into mobile development, you need to decide what platform you want to start with: Apple iOS or Google Android. While emulation software exists, it’s always better to test on a live device, and so my recommendation would be to go with the one you already own. Keep in mind that with Apple it’s a bit more problematic as you need access to macOS in order to develop for the iPhone.

As for languages, the choice is very limited. If you plan to develop iOS applications (or any applications for the Apple ecosystem, for that matter), your two choices are Objective-C and Swift. Objective-C is old and somewhat outdated. Swift is a way nicer language, according to Apple it’s faster than Objective-C, and on top of that it can be used with SwiftUI, which is a modern way to declare user interfaces. For Android, on the other hand, Java and Kotlin are your only options. Java is much more verbose than Kotlin, and I, personally, prefer Kotlin. Both of them are good languages to know, as they can be used in many industries.

However, as smartphones become more and more important, and companies are looking to save cost on developers, another viable option could be—write once, run everywhere approach. In this approach, you develop one code base which is then compiled to run on both iOS and Android. The two main frameworks for that are React Native developed by Facebook, and Flutter developed by Google. React Native uses JavaScript, an insanely popular language (more on that later), while Flutter uses Dart—a somewhat niche language developed by Google.

Flutter should be more performant than React Native, but Dart is not that popular. So it’s up to you to do the research on what technology is demanded in your area. Keep in mind that you will probably compete with the cool kids (read about them in the next section) who are doing web development. Having a mobile application is a must for a successful product, but many companies will cheap out on proper mobile developers, and instead will repurpose a good frontend engineer, who knows JavaScript, to do a semi-native development using React Native.

The cool kid

The most popular path to choose in 2023—is no doubt web development. If smartphones became our main devices of choice, then the web became our main operating system. I bet most computers today run mainly Chrome or any other browser, and all our interactions are done through the web. The browser can run video conferencing, graphic design, and even gaming. There are two main careers you can take there. Frontend development, and backend development.

Frontend development focuses mainly on what we see. When you go to YouTube and interact with the video, you are interacting with the works of a frontend engineer. For years, frontend was relatively simple. But recently it got a lot more challenging, at least as challenging as backend, if not more challenging. The most popular and recommended way to get into frontend development—is the holy trifecta of HTML, CSS, and JavaScript.

Once you know how to build a web page using HTML, how to style it using CSS, and how to make it interactive using JavaScript, the next step would be to pick up a JavaScript framework. React is by far the most popular JavaScript frontend framework. Others are Vue and Svelte, so I recommend you to check if they are more popular in your region, and maybe pick one of them instead.

It is also possible to become a backend developer. Backend developers are the ones responsible for feeding the data to and from the frontend. In the example of YouTube, backend development might include things like storing the videos, processing them, streaming them into the browser, authenticating users, etc. The most popular framework for building backend applications is NodeJS, which is written in JavaScript. So if you want to be a backend developer, I suggest you learn JavaScript and then pick up NodeJS. To further improve your employability, you should definitely know how to work with Databases such as PostgreSQL and or Mongo.

However, it is possible to write backend in other languages such as Ruby, Python, or PHP. If you do decide to chose one of those, I’d recommend you go with Python as it is a more versatile language that can be used in different fields like Machine Learning or Data Analytics. If you plan to work primarily with WordPress, then PHP is the obvious choice. Ruby is still used in some companies, but it’s slowly phasing out, and finding Ruby developers becomes harder and harder. This can be an opportunity for you, if your city or country has demand for Ruby developers. And according to the latest StackOverflow developer survey, Ruby is one of the top paying languages. ²

Web development is by far the most popular route as of today. It has the biggest market, and plenty of learning materials. On the other side, it creates a strong competition as everyone is trying to become a web developer. And you always need to be one step ahead of everyone else in order to raise your chances of success.

Everything else

The above careers paths are not the only ones you can take. You can decide to go the route of data analytics or machine learning, where Python is dominating. If you like servers and infrastructure, you can choose the path of a DevOps engineer, where languages like Shell scripting, Ruby, and Python as well as tools like Chef, Ansible, Terraform, Docker, and Kubernetes are the most demanded.

Desktop application development, while slowly dying, is still needed for some industries. There, like with mobile development, a lot of the decision depends on the operating system you are planning to develop for. For macOS, as with iOS development, Swift is the de facto standard. For Windows, you can use C# or C++. For Linux, it’s primarily C++. And as with mobile, you can also do cross-platform development with Java/Kotlin, C++ and a framework like Qt, Flutter and Dart, or Electron and JavaScript.

Aside from all that, system development is a viable career path as well. The world runs on operating systems and networking equipment, and someone has to write drivers for those. More over, the market for Internet-of-Things (IoT in short) is rapidly growing, with devices like smart home appliances, or embedded systems that run on a SoC like Raspberry Pi. As it’s the closest you can get to hardware, and performance matters a lot, development in this area is done mainly in C and recently Rust. If you do decide to go that route, keep in mind that you need to understand how hardware works and be prepared to work with manual memory management, as well as understand how to write and debug low level code.

Beyond the basics

A bit more than twenty years ago, the world was switching from Windows 95 to Windows XP, and the internet was growing in popularity. At that time, a career in software engineering wasn’t something as common as today. I bet that today it’s one of the most popular choices for a career. Our increased dependency on technology created the increased demand for software engineers. The more people take this career path, the harder it will be to get there. And the only way to increase your chances—is to be curious.

A specific language or framework might help you land your first job, but vast knowledge in how computers work—will help you to build a career. Software engineering is fun. It’s well paid. But it’s not easy. And it will be harder each year, as more people decide to choose this path. If you do it to escape your soul sucking career, without any passion or desire to learn and stay curious—you will eventually find your self in the same soul sucking career you wanted to escape from. However, if you like to solve problems, passionate about learning and exploring new topics—welcome aboard! And enjoy your stay there!

Happy 2023!

The missing ingredients from your monitoring alerts

Thu, 22 Dec 2022 13:00:00 GMT

There is one thing engineers hate the most—waking up at night and troubleshooting production incidents. What if I told you I know how to make it less painful?

Nobody wants to be on–call. Back in the old days of desktop applications, the on–call duty was reserved mainly to network and infrastructure engineers, who were responsible to make the internet work at all times. In today’s world, where every single app—is a SaaS served via the only operating system you will ever need, the browser—on–call duty has been propagated towards software engineers as well.

It doesn’t matter if you are a frontend or a backend engineer, if you are junior or senior—chances are you will, eventually, find yourself as part of an on–call rotation.

What it means to be paged?

Before I move on to your broken alerts, let’s set the record clear on the meaning of being paged. Paging an engineer means that your system performance has been degraded. This might mean a total degradation—like inability of users to access your system, or it might mean partial degradation—like inability to send registration emails.

From a business perspective, this usually means that your business is losing money. Every time your customers are experiencing service degradation, they will demand compensation in accordance with your SLA. Inability to acquire new customers, means they will switch to a different service—hence depriving you of their money.

From an engineering perspective, this means an immense pressure. During an on–call page, the number one priority of the paged engineer is to get the system back and running as fast as possible. This is not a time for nice solutions, or a major refactor. This is a time when you need to move fast, and with confidence. Keep that in mind, we will circle back to this later.

Why you are being paged?

If we look at all the reasons for paging an on–call engineer, we can group them in two main categories: manual and automatic.

Manual paging is self-explanatory. Your customer calls the customer support and notifies of a problem she is having. The customer support representative usually goes through a series of predefined steps, as he lacks the technical knowledge of the internals of the system, to either mitigate, or escalate the issue. After failing to mitigate the issue, and upon realization that the problem is critical, he might decide to page the on–call engineer. Depending on the nature of the organization, the customer support rep might page a SOC engineer, who will in turn might escalate to either infrastructure or a software engineer. In such organizations, software engineers are typically the last in line to be paged. In a smaller, less defined organizations, customer support might page the engineer directly.

With manual paging, there is not a lot to do. Each case is usually unique and requires a thorough investigation by the paged engineer to mitigate the issue. What can be improved, however—are automatic pages.

Automatic paging is triggered by your monitoring system. With proper monitoring in–place, you can define alerts that will be triggered upon different events. For example: an alert that triggers when the amount of free disk spaces reaches below certain point, indicating a potential failure due to inability to store data. Or an alert that monitors the RAM usage of a machine and triggers at a specific threshold, which might indicate a potential memory leak which will lead to an out of memory exception, and eventually—a painful death of your process by the operating system.

And from my experience, automatic paging is where most people, and alerting systems—get it wrong.

The missing ingredients from your monitoring alerts

During an on–call trigger, the paged engineer experiences a context switch. Mental context switch, just like the context switch done by OS schedulers, is an extremely expensive and productivity taxing operation for the human brain. Combined with the fact that from the moment you’ve been paged—your company starts to, potentially, lose money—it’s a terrible situation to be in.

The last thing you want to do under such circumstances—is to start understanding what happened. At the moment you’ve been paged, you already should know what’s the issue. And so, the first missing ingredient is—clarity.

Have clarity in your alerts

Imagine the following PagerDuty alert: Database disk reached 90% usage. How clear is this alert? The only information I can gather from the alert itself—is that disk space is running out. Here are some of the questions that I have no answers for:

What database is it? (I could have multiple)
What region? (I could have multiple)
How fast is it filling up?

By getting the answers to the first 2 questions, I eliminate the unnecessary time spent by browsing the terrible AWS console for all my databases, in an attempt to find the affected one.

By knowing the rate of filling, I know what are my time constraints. If I’m going to run out of space in 7 days, maybe I should not be paged at 2AM. However, if there is an automatic process that got stuck, and in about 1 hours I’m going to run out of space—I better hurry up before my entire database crashes.

A good alert should have clarity. It should point you to the alerting resource, give you time constraints, and if possible direct you towards the offending service or sub system. For example: [us-east-1] Database pikachu is filling at a rate of 1 GB per hour and will run out of space in ~1 hour.

I know that I have 1 hour max. I know the exact resource and where it’s located. This automatically might help me identify the potential issue, without even looking at the code. I might remember, out of the blue, the Jake pushed a new cron earlier today, that writes to this database. And so this might be a potential vector of investigation. This gives me a starting point, as opposed to blindly browsing AWS for all my databases, in all my regions, to find the offending database, while wasting my sleep hours as well as precious time I’ve been given to resolve the issue.

However, I still lack one major ingredient.

A checklist

Aviation is one of the safest forms of transportation. There are 3 ingredients that make it safe:

Well trained pilots
A lot of automation
A checklist

If you’ve ever looked out of the window of an airplane, just before takeoff, you probably noticed that all the flaps are suddenly going up and down, back and forth. The reason for that—pilots are running a pre–takeoff checklist. It doesn’t matter if you are a novice pilot, or a veteran one, before every takeoff—you run a checklist to check the critical systems of the airplane.

For some mysterious reason, we, engineers—don’t do that. Imagine the following alert: Subsequent requests to service xyz failed continuously for the past 15 minutes. This alert monitors the status of some xyz service, by issuing requests to it, and now indicate that those requests have failed. For the sake of example, let’s assume that xyz is an external service you depend on. It could be an email provider, or an API that gives you weather information.

One potential approach would be to go and start looking at the logs. Chances are, if this alert is infrequent, you won’t remember the exact log index or query you need to execute. And so you will try to fish the right query, to understand why requests to this service—fail. After eventually finding the correct log entry, and seeing that the error code is 503, you now start to defragment your fragile memory, and the neurons in your brain are starting to catch fire—while you try to remember who is the relevant person to reach out to in order to verify whether xyz is not having any issues on their side. After aging faster than you should have, you open a support ticket to xyz customer support, and they notify you—yeah, we have some issues on our side.

Humans are imperfect. And even though we—engineers, work with the same system every day, just like pilots who fly every day, we might make mistakes, just like pilots. But pilots have checklists. And instead of remembering a bunch of pre-takeoff procedures, they just need to remember one—run the damn checklist!

This entire issue could have been resolved if we had a checklist like this:

Identify the error code returned from xyz by following [this link] to the logs and looking for the field response_code.
If the code is 503—open a support ticket, using [this form] and verify whether xyz is having problems on their side
…

Not only this guides you step–by–step on how to identify, and possibly resolve the issue—it also functions as future documentation for new engineers that enter your on–call schedule.

I don’t know the best way to manage those checklists. I tend to hate Wiki pages as they become a pile of unsearchable garbage, but if you are able to keep a clean Wiki—it could work for you. You can have it pinned in your teams’ Slack channel. If you have other creative solutions, please feel free to share them with me (contact details are at the bottom of this post).

The best place would be to have it inside the alert itself. Once I get paged, I just go to the alert and can see the checklist there. Moreover, I can edit and fine tune the alert checklist as I go, or after I’ve finished with the incident. I was surprised that no monitoring system offers such capabilities.

Remember—incidents and pagers are stressful events, and people are imperfect. We tend to forget how things work, even if we work with them on a daily basis, and especially under stress. By eliminating as much decision–making during the incidents, as possible, we can create a better experience and faster resolution of production incidents.

I use checklists a lot in my life. This blog post, and all others I write, follow an always–evolving checklist during all the publication stages. If you are interested to know how I manage my writing—drop me a comment in one of the contact channels below. Until next time.

Living side-by-side with an AI

Wed, 14 Dec 2022 13:00:00 GMT

2022 is, undoubtedly, the year of the AI. It started with DALL-E, then GitHub Copilot, and now—ChatGPT. AI is here to stay, whether you like it or not, and we ought to find a way to live with it.

Earlier this month, the OpenAI team released ChatGPT—a GPT-3 based AI chat assistant that can do a lot of things like drafting a blog post idea, writing essays, explaining various topics, writing code etc. I’ve shared my initial impression about it in my previous blog post—ChatGPT, AI, and the future of tech. In the past week, I’ve used ChatGPT to perform some mundane tasks, which I usually use Google for. In this post, I want to raise some questions on how we, humans, are going to co-exist with AI.

Immediate impact of ChatGPT

In my previous post, I said that I don’t think software engineers or content creators will lose their jobs, at least not in the near future. I still stand behind this opinion. However, as with any technological progress, there are those who are going to be impacted—for good and for bad.

When the assembly line was introduced, and later on—perfected with automation, people who used to do what robots do now—lost their jobs. That’s the price we pay for technological advancement. Mundane, and repetitive jobs—are always the first in–line to be replaced. Things like low quality content production for SEO purposes, or researching keywords for blog posts. All these can be easier done by ChatGPT with minor corrections from a human expert.

ChatGPT generates me some keywords for a blog post#859x728

Yet, there are always people who look for low-hanging fruits. In today’s world—content is highly rewarded. No matter what niche you take, be it software engineering or self development, the successful people in that niche produce content. Blog posts, podcasts, YouTube videos, books, courses etc. You can say we do it for ourselves, but the truth is—we all, including myself, want to share our content with the world. Those who write for themselves—keep a journal, not a blog. We all do it for different reasons. Some to learn, others to teach. Some to share with others, others to enrich themselves. I’m not the one to judge, and I believe people should be paid for their knowledge—if they desire so.

However, the thing with new content producers is they think it’s easy. Just slap in some words in a text editor and publish it to WordPress, or film yourself talking to a camera and upload to YouTube. But in reality—content production is a hard job that involves not only the ability to write or speak, but also the ability to distribute your content and build an audience.

And while most of the creators take the hard route of producing content, there are people who will always try to find shortcuts. Like with money making—some people go to work every day, while others try to set up a get-rich-fast scams. And the big beneficiaries of AI technologies today—are those who are looking for shortcuts.

I’m relatively active in communities like HackerNews and Reddit, and I can see the explosion of ChatGPT content on those platforms. I even had an answer to one of my questions, copy–pasted from ChatGPT. Granted, the author of the answer disclosed he/she used ChatGPT. Notion, the highly popular note–taking app, that I no longer use, recently released Notion AI—an ability to generate content right inside the editor itself, leveraging GPT-3 model. They did not work on the countless issues that their users have with the software, such as lack of offline support or E2EE—instead, they’ve put their efforts into AI.

And it’s a smart move because I believe there will be a huge demand for AI, due to the important role of content creation in our current society. Websites, like StackOverflow, have now implemented a temporary policy to ban ChatGPT generated content. Getty bans AI generated content. And I’ve read somewhere that Google is working on, or already have, a policy to punish websites that use primarily AI generated content. I can’t find the source, so don’t quote me on that.

The human touch

From all the examples I’ve seen, and based on my personal experience—ChatGPT is often times wrong. It gets a lot of the things right, but usually make a mistake here and there. And this is one of the dangers of using its raw output. A person, who has no understanding of the topic, can not effectively use ChatGPT. I can ask ChatGPT how a nuclear bomb works, but I’m not a nuclear scientist to verify that the answer is correct. I can, however, go on Quora or Reddit and spam answers at a rate of one per minute, hoping that 90% of them will be correct—while ripping the benefits.

The power of communities like StackOverflow or Reddit—is the positive feedback loop we get from our fellow humans. The ability to audit a solution, or provide multiple answers from multiple people—is priceless in the world of self development and knowledge acquiring. People make mistakes—that’s why we have protocols in place, as well as fellow human beings, to correct our mistakes. ChatGPT is free, for now. If I want to read AI generated content, I can use ChatGPT directly. And so, flooding the internet or the above communities with AI generated content, makes it harder to maintain the quality of the content as well as getting real human answers that come from their personal experience and knowledge.

What is creativity?

When we talk about AI generated content, we need to talk about creativity. If we put aside for a moment, the fact that ChatGPT produces inaccurate content in most cases, one could ask then—what’s the difference between a content written by a human and a content created by a machine?

Creative process is not something that can be described mathematically. Humans, usually, describe their creative process as a-ha! moments. Something like: I was taking a shower, while thinking about the solution to that problem, and then it hit me! I need to use JavaScript!

We then go on and do research on that topic, by reading materials, and eventually summarizing the entire process in a unit of production—be it a system design, a piece of code or art. If we look beyond technical fields, and gravitate towards more creative fields like writing or singing—we can learn that many writers or poets produce content based on their personal experience and background. A complicated child might become a rapper who will rap about his though childhood, which might resonate with children in a similar situation. A WW1 veteran—might write about his horrible experience during the war, as a way to ease his pain and help him cope with that experience, while passing onto the reader the horrors of those events.

The word creativity in itself is associated with something human. AI has no experience. It has no notion of past. It’s not having childhood traumas. Nor it can think like a human can. Therefor, I believe it has no creativity—it can only remaster. And there is something humanly when we look at art or read books and can feel the same things the artist or the author were feeling. I can’t have sympathy towards machine. I usually kick my robot vacuum cleaner when it’s stuck. Please don’t call the robots abuse hot line.

ChatGPT talks about creativity#878x680

Something from nothing

One could, rightfully, ask—Humans rarely create anything from nothing. How is that different from an AI?

I do want to point out that there are great discoveries made by humans. My point is not to say that all we create—is a remix of existing things. But as I’ve said earlier, most of what we do create—is based on past knowledge and experience. A student of Rembrandt will be inclined to create art in a similar style to his teacher—unless he or she decides to explore other styles or artists. The same goes to all other types of content creation. What I write—is based on what I read and the experience I have.

The difference between me and AI—is that I have context and understanding. I know that 2+2=4. A good mathematician, which I am not, also knows why. AI—does not. It simply fed this fact into its neural network. If you feed it with another fact, it will produce answers based on that fact. This creates a negative feedback loop. I’ll talk a bit later about the negative feedback loop.

What is copyright?

When we talk about creativity and remastering—we have to talk about copyright. As I’ve said in the beginning of this post—2022 is the year of the AI. During a very short period of time, something like 3–4 months, we’ve gone from closed-behind-doors-deep-learning-ai-that-beats-humans-in-go-or-chess, towards AI that generates art, code, and poems.

But you know what’s missing? AI that generate music or videos. Yes, there is OpenAI Jukebox, but it didn’t get as much attention as DALL-E, Stable Diffusion, or ChatGPT. While the results of it are amazing—they are not on the same level as other AI tools. And there is no, yet, AI that can generate video content such as movies. I think it’s related to the fact, that existing AI models are trained on existing data. Access to royalty free music or films—is limited. GitHub copilot was trained on public GitHub repositories, while ChatGPT was trained on crawling the internet. Those are huge datasets. GitHub has nearly 128 million of public repositories; while the GPT-3 model was trained on nearly 400 billion websites.

I don’t believe knowledge or art should be safeguarded, unless the author decides so. But I do believe that people have the right to retain copyrights for their work. I’m not a copyright lawyer, so I don’t know exactly where is the line between original work and remixed work. But I know for sure that I can not publish a YouTube video with Britney Spears song in it. This video will be removed, for respect to Britney’s copyright. The same respect that other authors, or open source developers—have. Is it right to train an AI model on an open source project, just because it’s open source? Is it right to train an AI model on a website, just because it’s publicly available? Is it moral to charge money for an app that uses an AI model, which is, most likely, trained on a publicly available art, that generates avatars for you—without giving credit or attribution back to the artists that “inspired” the AI model?

I don’t know. I know there is a lawsuit against copilot. And I know that we must have a discussion as to how we should train those AI models. The fact that some datasets used to train the models—are public, does not mean it can be used without implications, credits, or attributions. The fact that one owns a bunch of legally obtained DVDs with movies, does not grant them the right to open a private movie theater. Images, articles, and blogs posts—sometimes have licenses. And those licenses should be respected by humans as well as AI. And unless we start to raise awareness for this issue, and have an open discussion—we are, potentially, violating the rights of content producers and artists.

Negative Feedback Loop

I’ve mentioned it earlier, and I want to expand on this topic. In one of my previous posts, The software industry is broken, I’ve discussed the fact that software today—is not as optimized as it used to be. One of the reasons I’ve outlined—is the fact that writing software became easier, and in a way dumber. Developers today don’t need to know how memory is managed. There are high level languages that manage it for you. And so, as tools become more available to the masses—they’re ought to be simplified. And simplification, often times, leads to loss or deterioration of knowledge.

A similar thing can, and will, happen to our knowledge if we do not control the spread of AI content. AI is great today because of two reasons—(1) the engineers behind it and (2) the entire knowledge of human beings. Every time an AI generates a piece of code, art, or a text paragraph—it’s because there is work made or knowledge shared by humans, that the AI was able to train on.

I’ve said earlier that, as of today, ChatGPT gets most of the things right, and some things wrong. If we don’t set up protocols in place to distinguish human generated content from AI generated content, AI generated content will flood the internet. Look at how the standards of beauty have changed, because everyone is posting unrealistic photos of themselves, by using a bunch of photo retouching, auto-magical, tools.

There are people who embrace the easy model of producing free content. And by flooding the internet with potentially inaccurate content, we will not only bias the future training process of AI—we will damage the future knowledge of human beings.

The real danger is not that computers will begin to think like men, but that men will begin to think like computers.

— Sydney J. Harris

Conclusion

I tend to think of myself as a realist. Although, if you ask some of the people who know me well—they might say I’m leaning towards pessimism. AI is here. We’ve opened that box. As with every tool ever created by human beings—it can be used for good, and it can be used for bad. People, usually, tend to focus on the good. I’m not going to unwrap my entire philosophical views here, but I tend to, typically, focus on the bad.

We reward content creation. Aside from Google, the 4 most visited websites are YouTube, Facebook, Twitter, and Instagram (source); and the 3 most downloaded apps are TikTok, Instagram, and Facebook. Those are all content creation tools. I’ve read somewhere that today’s kids, when you ask them who they want to be when they grow up, answer a YouTuber or a TikTok/Instagram influencer. And we value knowledge.

ChatGPT, DALL-E, Stable Diffusion, Midjourney, Copilot etc. Those are all great tools for both content creation and knowledge sharing. Every one of us will have to find a place for them in our daily toolbox. ChatGPT crossed 1 million users in less than a week—this is a clear sign that it’s not just a tool that will be forgotten soon.

The striking thing about the reaction to ChatGPT is not just the number of people who are blown away by it, but who they are. These are not people who get excited by every shiny new thing. Clearly something big is happening.

— Paul Graham, Computer Scientist, Entrepreneur, Venture Capitalist, and Author, Twitter

AI here to stay—and more will come. But together with the adoption of those tools into our daily toolboxes—we need to have a discussion about topics such as creativity, attribution, respect for human work and the effects those tools might have on the future of humanity and our knowledge.

#WrittenByHuman

Cursor based pagination

Wed, 07 Dec 2022 13:00:00 GMT

Traditional pagination existed for many years. But it’s not useful for platforms with huge amount of user generated content. Luckily, there is an alternative solution for such systems.

Traditional pagination

Before we jump into the solution, let’s look at traditional pagination and it’s drawbacks. Let’s say we have a database table for forum posts that looks like this:

ID	Title	Content	LastActiveDate
1	Some post	Some content	2022-11-26T14:00:00

In this case, LastActiveDate—indicates the date when the post was last active (i.e., a comment was published).

We want to display a front page with all the posts sorted by LastActiveDate. Since we don’t know how much data we can have, we decide on an arbitrary page size of 10. And then, in order to display the posts on a particular page, we will use the following query:

SELECT * FROM posts
ORDER BY LastActiveDate
LIMIT ${pageSize}
OFFSET ${(currentPage - 1) * pageSize}

The math is very simple. Given a specific page, say 2, we will select all items starting from offset (2 - 1) * 10 = 10, limiting the selection to 10 elements, and the result will be all the items in group of [10,20).

Usually, with traditional pagination, we also need to have another query in order to generate the visual pagination itself. And for that we need to know the total amount of pages we have, which can be done by:

const totalItems = await sql("SELECT count(ID) FROM posts") as number;
const totalPages = Math.ceil(totalItems / pageSize);

Drawbacks

This pagination style, however, has one major drawback. Let’s say I’ve navigated to page 1 of the forum and then decided to go make some tea. After coming back and seeing that no posts on the first page are of interest to me, I navigate to the second page and find out that it includes 2 posts from the first page.

This happens because during the time I was making and drinking my tea, some older posts that were not among the 10 posts on the first page — were updated, hence changing their LastActiveDate to be the most recent. Upon navigating to the second page, I now see 10 more posts, but because of the shift in order, 2 posts from the first 10 — were pushed into the second 10 on the second page.

Usually, on platforms with low update rates, like a small forum — this drawback is not that critical. Occasionally users will see content on the next page, which appeared on the previous page — but the frequency during which it happens — is low. We can increase the page size to say 15, 20, 25, or even 30 posts per page — and since forums often have activity concentrated in few popular or new posts on the first page, the chance of seeing the same data on next pages — is small, and we can live with that.

However, for high throughput platforms like Twitter, or machine-to-machine streaming services — this case is unacceptable. Image navigating Twitter only to see the same content over and over again, because the rate at which content is created on Twitter is rapid, and thus content from the previous page is always pushed down to the next page.

In order to solve this problem, let me introduce — Cursor based pagination.

Cursor based pagination

Let’s say we want our forum pagination to never display data from previous pages. When a user navigates to the second page — she will always see data that suppose to be on that page. This can be achieved with cursor based pagination.

How cursor based pagination works

In cursor based pagination, we do not have logical pages. Instead, each page returns the data that suppose to be displayed on that page as well as a token to retrieve the next page. The token is usually opaque, and for the end user it has no meaning — but our system knows to read it and generate the next page based on it.

This brings me to the first drawback of cursor based pagination:

WARNING

Cursor based pagination is forward. It does not support going back (i.e. navigating from second page back to first, or from third back to second); nor it supports skipping pages (i.e. going to page 5 from page 2).

On a platform like forum, this might not be very useful. Forums are typically meant to be browsed, so users always can jump to a specific page, navigate backwards and forwards, or event jump to the last page. However, — on platforms with infinite scrolling navigation, like feeds, this solution is perfect.

Cursor based pagination example

Let’s say our product manager came up with the terrible idea of turning our 90s looking forum with traditional pagination, to 2000s looking forum with infinite scroll feed. We’ve tried to warn him that this is a bad idea, but he insisted that users wanted it. And so let’s turn our forums’ traditional pagination — to cursor based instead.

First, we need to select the cursor. Since forum posts are often sorted in chronological order from newest to oldest post, it makes sense to use the LastActiveDate as a cursor. Hence, the retrieval of a particular pages’ data can be done like this:

async function getPosts(cursor: null | Date) {
	const posts = await table("posts").select("*").orderBy("LastActiveDate", "DESC").limit(PAGE_SIZE).where(q => {
		if(cursor) {
			q.where("LastActiveDate", "<", cursor);
		}
	});

	return {posts, nextPageCursor: posts.length ? posts[posts.length - 1]["LastActiveDate"] : null}
}

Disclaimer: The query builder is a pseudocode, but can be achieved with tools like KnexJS

For every call, we fetch the first PAGE_SIZE posts and provide a cursor to the next page, which is equal to the LastActiveDate of the last post in our current batch (or null if no posts exist). If the cursor is null — we start from the first page.

Since our posts are always sorted in chronologically descending order, our pagination becomes consistent.

Let’s try to do a visualization. Let’s imagine we have the following database:

ID	Title	LastActiveDate
1	Title 1	2022-01-10T00:00:00
2	Title 2	2022-01-09T00:00:00
3	Title 3	2022-01-08T00:00:00
4	Title 4	2022-01-07T00:00:00
5	Title 5	2022-01-06T00:00:00

NOTE

I’ve intentionally sorted the posts by LastActiveDate to easier visualize the process. Obviously, the data does not need to be sorted in advance, since SQLs SORT BY LastActiveDate DESC handles the sorting.

For the sake of simplicity, let’s set our page size to 2. Upon first execution, we will get posts 1 and 2 and our nextPageCursor will be equal to 2022-01-09T00:00:00 (which is the LastActiveDate of the second post).

As we would like to retrieve the next page, we will provide the nextPageCursor to our function, and since the query looks at posts older than 2022-01-09T00:00:00 we will get two more posts — IDs 3 and 4.

Now — let’s come back to our tea scenario. Imagine I’m looking at the forum’s first page with posts 1 and 2, and before clicking next, I go to drink tea. During that time, someone posts a new comment in post ID 3 — thus altering its LastActiveDate to be today (let’s say today is 2022-01-11T00:00:00). The table will now look like this:

ID	Title	LastActiveDate
3	Title 3	2022-01-11T00:00:00
1	Title 1	2022-01-10T00:00:00
2	Title 2	2022-01-09T00:00:00
4	Title 4	2022-01-07T00:00:00
5	Title 5	2022-01-06T00:00:00

When I come back to the computer, I click next page. In our traditional navigation — my action would have returned posts with IDs 2 and 4, thus “leaking” the second post from the first page to the second.

Cursor based navigation, with the cursor of 2022-01-09T00:00:00, would however return posts with IDs 4 and 5 as expected. Notice how post ID 3 is skipped and will never be seen by me, unless I start over from page 1. It’s important to point out that this flaw is not a problem of the cursor based pagination — but rather a problem with the way we’ve implemented cursor based pagination. Different use cases — have different implementations. It is possible to implement cursor base pagination which takes into account newer content as well. It, however, makes little sense to do so in a context of a forum, since in forums — going forward in pagination — means seeing older content.

As I’ve said before, we do lose some functionality — namely: going backwards (from page 2 to page 1 for example) and skipping pages (from page 2 to page 7 for example). So this approach is not useable for browsing content in a paginated manner, but it is a good approach to implement infinite scrolling without encountering duplicate content.

Bonus—don’t use dates and or timestamps as cursor

I could have ended the article here, but I’m pretty sure if I did so — I’d get criticized for using date as a cursor. And this will be a well deserved criticism.

You see — the entire point of cursor based navigation — is to create a consistent stream of data without repetitions. Yes — you give away some features, but in exchange you get fresh data for each page. As I’ve said — some systems might benefit from it.

However, — by using dates as cursors — you run into the same problem. You can encounter content repetition. Let’s assume we have a high traffic forum with a very active community. Let’s look at a subset of such forums posts table:

ID	Title	….	LastActiveDate
675	Title 1	…	2022-01-10T12:31:22
123	Title 2	…	2022-01-10T12:30:48
534	Title 3	…	2022-01-10T12:30:23
301	Title 4	…	2022-01-10T12:30:23
231	Title 5	…	2022-01-10T12:30:23
945	Title 6	…	2022-01-10T12:30:17

Disclaimer: as earlier, the data is sorted by LastActiveDate for demonstration purposes only.

For the sake of example, let’s say our page size is 2. And since we are looking at a subset of data, let’s say we hold a nextPageCursor of 2022-01-10T12:31:22. What will be the result of the invocation of getPosts("2022-01-10T12:31:22");? If you’ve read through the article and understood it well, you would be correct to answer: posts 123 and 534. And what will be the nextPageCursor value for that invocation? The correct answer is 2022-01-10T12:30:23.

Now, here’s the problem. Let’s say you want to navigate to the next page using nextPageCursor = '2022-01-10T12:30:23'. What would be the result? That’s true — post ID 945. Since the query we’ve written earlier, translates to:

SELECT * FROM posts
ORDER BY LastActiveDate DESC
WHERE LastActiveDate < '2022-01-10T12:30:23'
LIMIT 2

We are skipping posts ID 301 and ID 231. The hasty among you might just run and replace < to <=, but this introduces another issue. Now, instead of getting post ID 945 you get posts ID 534 and ID 301. So not only you are getting post ID 534 again, your nextPageCursor is set to 2022-01-10T12:30:23 so you’re essentially stuck in an endless loop (unless someone decides to update one of the posts).

Dates, although provide chronological order, are not providers of uniqueness. In a system with high rate of content, you might end up with collisions. And I know what you are thinking about: I’ll just add milliseconds! While milliseconds might help you to prevent this issue in human generated content (although there is still a chance of collision, depending on the traffic and the nature of the system) — they are not a solution for machine generated content in systems with high rate of content generation. There are other solutions, like content versioning, to provide uniqueness of the cursor.

Also keep in mind that cursors do not have to be bound to one specific dimension. I once designed a system where the cursor was a JSON that held multiple vectors for data retrieval. This JSON was then base64 encoded and was used as a cursor for navigation. So you could combine multiple fields, and any other data you would want, inside the cursor. As long as you can pass the cursor in the URL and your system can decode it — it’s a viable solution.

ChatGPT, AI, and the future of tech

Mon, 05 Dec 2022 20:00:00 GMT

Unless you’ve been living under a rock, you probably heard about OpenAI releasing ChatGPT for a free research preview.

After the release, a bunch of different posts across Reddit and Hacker News started to emerge — outlining all the possible reasons why AI and ChatGPT will now replace jobs such as: Product managers, software engineers, tech bloggers etc. Usage of ChatGPT exploded, leading to sites like StackOverflow — to implement temporary policy to ban ChatGPT generated answers (source).

Look — I’m a technology lover. I like messing with technology; I like using technology to improve my life. I automate certain tasks using software; I adore my robotic vacuum and window cleaners, and after getting a dishwasher last year — I don’t understand why I didn’t get one earlier. But, I, personally, can’t share the enthusiasms or doom–like prophecies that we all, are soon, going to be replaced by AI.

Understanding AI

I’m not an AI engineer, and apart from messing a little with artificial neural networks and genetic algorithms many years ago, my AI knowledge is very limited. According to my understanding, AI, and especially the GPT–3 model — built upon feeding on initial content. This means that those models are analyzing existing texts or source code — and learn to generate meaningful content based on that. According to Wikipedia, GPT–3 was trained mostly on crawled websites (source). The famous GitHub CoPilot was trained on source code from GitHub repositories. And so, in my uneducated opinion, current AI models are limited by the knowledge we — humans — have created.

Taking AI for a ride

At first I was skeptical, but ChatGPT was topping Reddit and Hacker News front pages, as well as talked a lot among my colleagues, so I’ve decided to take it for a ride. It’s open for public, for a free research preview: https://chat.openai.com/.

Hey AI – write me a blog post

Before jumping to find whether I’ll be replaced as software engineer, I was wondering will I be replaced as a blogger. On Wednesday, I’m going to publish a new blog post that talks about cursor based pagination. You should totally subscribe to my newsletter to be notified when it’s published, as well as stay updated about any future posts. But in the meantime, I wanted to see — would ChatGPT be able to generate a blog post on that topic. And so I’ve asked AI the following:

write a blog post about cursor based pagination and comparison with traditional pagination, including examples for usage as well as cons and pros of each approach

It generated a nice post, but when it came to comparing pros and cons — the results became messy. It listed 3 pros for cursor based pagination, and while the first two were okay-ish, the third one is a big miss:

It’s more user-friendly. With traditional pagination, it’s difficult for the user to know how many pages there are in total, or to go directly to a specific page. With cursor based pagination, the user can see the cursor value and use it to navigate to a specific position in the dataset.

This is a complete lie. There is an easy way to both (1) know how many pages are there in total, and (2) go directly to a specific page. We’ve been doing it for decades. Moreover, depending on the implementation, the user is not always able to know and use the cursor as he desires. On complex datasets — cursors are opaque and meaningless to the user.

You could say that this is a mistake that can be easily fixed by a human later on, but wait! There’s more! When ChatGPT generated the cons of cursor based pagination, it wrote this:

It’s less predictable. In traditional pagination, the user always knows how many records will be on each page. With cursor based pagination, the number of records on each page can vary depending on how many records have been added or deleted.

The first sentence contradicts the pro outlined earlier. If I know how many records are on each page, I can know how many pages there are. The second sentence — is not entirely true, in fact it is possible (and recommended) to include a limit on the amount of records you retrieve using any type of pagination.

While I am amazed by the ability to generate, generally, good structured and informative post — it’s far from replacing a human, such as myself, who performed a good research combined with personal, and professional experience.

sudo ai – generate me some code

I’m safe with writing blog posts. Let’s see if I can be replaced as a software engineer.

I’ve asked ChatGPT to generate me some code:

Write a function that streams data from external API endpoint and caches it, using typescript and nodejs

And it produced the following result:

import * as request from 'request';
import * as fs from 'fs';

// Function for streaming data from external API endpoint and caching it.
async function streamData(url: string, cacheFile: string) {
	return new Promise((resolve, reject) => {
		// Check if the cache file exists.
		fs.access(cacheFile, fs.constants.F_OK, (err) => {
			if(err) {
				// Cache file does not exist, so create it and steam data from the API
				const fileStream = fs.createWriteStream(cacheFile);
				request(url)
					.pipe(fileStream)
					.on('finish', () => {
						resolve(true)
					})
					.on('error', (err) => {
						reject(err);
					});
			} else {
				// Cache file exists, so read data from the cache file.
				fs.readFile(cacheFile, 'utf8', (err, data) => {
					if(err) {
						reject(err);
					} else {
						resolve(data);
					}
				});
			}
		});
	});
}

ChatGPT suggested using a deprecated module — request. Also, the code itself will fail linting because streamData is defined as async but there is no need for that function to be async, since the code uses callback style fs and has no await statements. Nitpicking, I know.

When I asked ChatGPT, what is wrong with the above code — it was smart enough to output this:

One potential issue with this code is that it uses the request module, which has been deprecated and is no longer recommended for use. Instead of using the request module, the code should use a more modern HTTP client, such as the axios library, to make HTTP requests.
Another potential issue with this code is that it does not handle errors properly. In the case where the request module throws an error, the code does not reject the promise, which means that the error will be silently ignored. To fix this, the code should add a catch block to the request call, and reject the promise with the error if one occurs.

It was able to catch the deprecated request module, but said nothing on the unnecessary async keyword, or suggested to use fsPromises instead of callbacks for a more readable code.

Programming is more than just writing code

For some reason, people keep thinking that a programmer’s job is to write code, while in reality — our job is translating client requirements into code as well as designing, and maintaining complex systems. Writing new, fresh code — is relatively small part compared to maintenance and modification of existing code.

Questions, such as how to sort an array, were long ago solved with all possible algorithms. ChatGPT is no different from Google search for such question, except for being more convenient than performing your own search, while potentially producing wrong or outdated code. The power of communities such as StackOverflow — is that answers are audited and updated. I keep getting notification about new comments or updates to a question I’ve asked more than 7 years ago!

Don’t get me wrong — those are amazing results. I’m happy for the OpenAI team, and the people behind GPT–3 are, undoubtedly, incredible and talented engineers. But I believe that current AI models are limited by our existing knowledge. They are unable to create new things. They are able to remix existing knowledge. Takes bits from different sources, and try to put them in context. Those tools, in their current state, can be a great addition to other tools like searching in Google or using StackOverflow. But I don’t believe we are anywhere close to having an AI that can truly create new things — and thus replacing us in our jobs. Don’t trust me? At least trust an AI, then.

ChatGPT on replacing software engineers#890x417

#WrittenByHuman

Why you should adopt Makefile in all of your projects

Wed, 23 Nov 2022 13:00:00 GMT

GNU Make. A software that is, most likely, older than you. It’s so simple, so standard, and so ignored. I’m here to provide a case in favor of make and Makefiles.

What is `make` and Makefile?

Before I even start to build my case, I need to explain to you what is make and Makefile.

make — is a build automation tool. It is written in C and was first released in April 1976. On Linux, make is usually available as part of the build tools dependencies that includes stuff like gcc — the C compiler etc. On Ubuntu, it’s under build-essentials; on Arch Linux, under base-devel. MacOS users can install it with brew. And on Windows… I honestly have to clue what is going-on on Windows.

But make does not work by itself. It needs instructions. And those instructions are stored in a Makefile.

Makefile 101

Makefile is a file named… Makefile. Makefile consists of rules. Each rule has a target, optional dependencies (which can be other targets or files — more on that later) and a set of commands.

A simple Makefile that complies index.ts into index.js might look like this:

build:
	npx tsc index.ts

In order to compile the project, you execute make with a target name:

make build

build — is the name of the target. Then each next line that starts with a tab is a command that make will execute. The above example will compile index.ts and place the resulting index.js in the same folder. I can modify the commands and move the resulting file into dist directory:

build:
	mkdir -p dist
	npx tsc index.ts
	mv index.js dist

Of course, in case of TypeScript — you should avoid this practice and instead set up a proper tsconfig.json, since TypeScript compiler handles that for you. But for the sake of demonstration, I’ll continue with this example as it’s easier to understand.

I’ve mentioned earlier that make targets can have dependencies which might be other make targets or files. Let’s explore this concept. Say we have the following directory:

~/workspace/make_test ❯ tree -I node_modules
.
├── Makefile
├── package.json
├── src
│   └── index.ts
└── yarn.lock

1 directory, 4 files

A typical TypeScript project. We want to compile it with make and put the resulting files in dist folder (which might or might not exist and might or might not have previously compiled files). Here’s a Makefile that allows this.

prepare:
	mkdir -p dist

clean: prepare
	rm -rf dist/*

dist/index.js: src/index.ts
	npx tsc $< --outfile $@

build: clean dist/index.js

When I run make build, I get the following output:

~/workspace/make_test ❯ make build
mkdir -p dist
rm -rf dist/*
npx tsc src/index.ts --outfile dist/index.js

And running tree reveals that I now have a compiled file in dist directory.

.
├── Makefile
├── dist
│   └── index.js
├── package.json
├── src
│   └── index.ts
└── yarn.lock

2 directories, 5 files

Let’s understand step by step what’s going on.

First, we have a target named prepare. This target has only one command that creates a directory named dist.

Then, we have a target named clean. The purpose of this target is to delete the contents of dist. But if you’ve noticed, unlike prepare which does not have anything after the double colon, clean specifies prepare. This is called a dependency. In order to run clean, it will execute prepare first.

Then we have another target called dist/index.js. This time it’s a file, and it depends on… you’ve guessed it — another file called src/index.ts. Targets in make, as I’ve said earlier, can depend on other targets or on files. In order to satisfy the file dependency, make makes sure that the file exist. Make does not really differentiate between targets and files. If you create a file named prepare and run make build again, you can see that this time the output is different:

~/workspace/make_test ❯ make build
rm -rf dist/*
npx tsc src/index.ts --outfile dist/index.js

Make is no longer executing mkdir -p dist, because make sees that a file named prepare already exists, therefor this target is already satisfied. The same is true for src/index.ts. In practice — it’s a target. But our Makefile does not have a definition for that target. If you go on and delete (or rename) src/index.ts, and run make build again — you will get an error:

~/workspace/make_test ❯ make build
mkdir -p dist
rm -rf dist/*
make: *** No rule to make target `src/index.ts', needed by `dist/index.js'.  Stop.

Because make can’t find the file src/index.ts, nor it can find a target named src/index.ts in our Makefile — it fails. Remember — make xyz first checks for the existence of a file named xyz, and if no such file is found — it looks for a target named xyz, and if none found — make fails. Also if you’ve noticed — make does not do rollbacks. It simply executes commands until it’s done or until failure.

Side note: there is a way to instruct make that a target is not a file by using what is known as phony targets, but for the sake of simplicity, I’m not going to cover that here. You can read more about phony targets if you want.

Going back to our dist/index.js. This target executes one command npx tsc $< --outfile $@. There are two variables there: $< — is the name of the first dependency, in our case src/index.ts while $@ is the name of the target (dist/index.ts). And so running this command translates to npx tsc src/index.ts --outfile dist/index.js.

Make has a robust mechanism for variables and looping over files. You can, for example, pick all the *.ts files and compile each and one of them using the $< and $@ parameters. I’m not going to cover it here since it’s not a tutorial about make. Drop me an email if you want me to write one (my contact details are at the bottom of this post).

Finally, we have the build target. This target depends on two other targets — the clean target and the dist/index.js target, and by itself it executes no commands.

Makefile has more cool features such as support for variables and macros, including sub-makefiles etc. For the sake of simplicity, I’m not going to cover them here. You can take a look at my decade old Makefile for a WIP OS I was once writing. It includes variables, macros, sub-makefiles etc.

But… but… this looks like [your favorite build tool name]

Exactly! If you are familiar with JavaScript then you are probably familiar with npm scripts. In Rust we have cargo. In Go, we have the go build tool. When using Python, we have pip, pipenv, potery and conda for dependency management. For Java there are maven, gradle and ant. In Ruby, dependencies are managed with gem. And honestly — it’s too much.

Here’s the thing. Developers are lazy. When you jump from project to project — managing dependencies and compiling the code becomes a headache. Even if you don’t change the language, only in JavaScript we had grunt, gulp and bower which are dead long ago, but might still be in use for some legacy project you are working with; but even today we are torn between npm and yarn, and I didn’t mention frontend where there are multiple bundlers to choose from. I tend to jump between projects written in JavaScript, TypeScript, Java, Python, Ruby, and Rust. So I need to remember a handful of script runners and dependency management tools.

And I don’t want to do 2 things: (a) Write a comprehensive README.md file that describes how to install, build and run my project, and (b) Read a comprehensive README.md file that describes how to install, build and run your project. I want to be productive by simply jumping into the codebase and producing code, rather than spending my time on trying to figure out what dependency management system the project uses and how to run tests.

And here’s where make and Makefile comes in. make exists for 46 years. It’s not going away. By providing Makefile and common make targets like build, compile, lint and test — I can easily jump into a project and start being productive. I don’t need to scout multiple files, trying to come up with the winning combination. Aha! You have a package.json, but a yarn.lock so this means I need to use yarn install. It’s a mental overhead I don’t want to deal with.

Moreover, designing a proper CI pipeline becomes a breeze. My CI will execute make targets, and under the hood I can use whatever dependency management / script runner tool that I like. Hell, I can even change them every 5 days when a new hyped, blazingly fast, build / script runner tool appears.

Use whatever you like. You like yarn? Go ahead! Use yarn. You can even put your commands in package.json if you want to (for example to utilize pre* or post* hooks), but please provide a Makefile that acts as a facade to your npm scripts. That way, all we need to look at is only one single Makefile to understand how to run, build, compile and test the project.

4 Ways to Minimize your Dependencies in Node.js

Thu, 03 Nov 2022 17:30:00 GMT

We all know the joke about how node_modules is the heaviest object in the universe.

For example, a project that uses only fastify, knex, typescript, and uuid generates an 83MB node_modules folder! That’s huge! And those four packages are far from a complete set for a relatively small back-end. A more realistic size for node_modules is north of 100MB, in some cases reaching 1GB.

In this post, we’ll explore four methods to minimize your code’s dependencies, resulting in faster CI/CD execution and safer code.

But first, let’s touch on some of the problems with heavy node_modules.

The Issues with Heavy `node_modules` in Node.js

Heavy node_modules can cause slower CI/CD pipelines, as dependencies are usually installed during those pipelines, and require network calls to a registry (whether npm or your mirror of it). This affects your development experience.

Moreover, package creep can introduce serious issues like security vulnerabilities, as you don’t own the code that resides inside the packages.

Let’s jump into how to minimize the dependencies in your Node.js code.

Method One: Check the Age of Node.js Packages

Imagine a situation where you have a function that looks like this:

function findSomethingByIds(ids: number | number[]) {
  ...
}

This function’s purpose is to either find a single entity or a list of entities by their id. It’s common practice with different repository patterns.

However, let’s say you want to know whether you’ve requested a single entity or a list of entities. A quick npm search leads us to npm/isarray, an insanely popular package with over 62 million weekly downloads! You know the drill now:

npm i isarray

Don’t forget:

npm i -D @types/isarray

But wait! Before you do that, have you noticed that this package is three years old? And if you do a better search, you will discover that isArray is now a part of the JavaScript core, can be invoked using Array.isArray, and is perfectly supported in Node.js version 10 and up.

Package age is a good first indicator. Many old (2+ years) packages might have security vulnerabilities or be outdated. Some of the outdated packages are also merged into the official JavaScript spec.

Another example is the trim package. It has more than 4 million downloads, and while it’s not that old (only 1), a native solution exists in JavaScript core: String.prototype.trim().

So always try to find a native solution first, as JavaScript evolves fast and its standard library is always expanding. Don’t be fooled by the weekly downloads counter in npm.

Method Two: Using ‘One-liner’ Node.js Packages

Many ‘simple’ packages are actually what I call ‘one-liners’. A one-liner is a package that contains very few lines of code.

If we continue with our isArray example from above, by examining the content of index.js we can see that it’s a simple one-line function:

var toString = {}.toString;

module.exports = Array.isArray || function (arr) {
  return toString.call(arr) === '[object Array]';
};

Source: GitHub

I always recommend you look at the source code of packages, at least simple ones because they can teach you a lot. Looking at the above source code, we discover one important thing: that Array.isArray exists (the same conclusion we came to with method one).

However, even if we did not have a native is-array method, the entire function is a simple if statement. And instead of requiring this package as our dependency, we can simply write the code as part of our project. It’s worth stopping for a minute and discussing the pros and cons of such an approach.

Using Npm Vs. Writing Your Own Code

Npm is a public registry, and the code contributed to npm comes from people worldwide. It is amazing that we have such a big repository of free and open-source code. However, this also comes with some disadvantages.

Npm, unfortunately, is known to be at high risk of attack. Packages might get compromised, whether by third parties or by developers themselves.

We all remember the scandal around left-pad and the recent rise in crypto-mining and crypto-stealing code that resides inside popular npm packages.

While it’s impossible to audit every single package we install, we can lower the attack vector by using fewer packages in our projects. One great way to reduce the amount of packages we depend on is to write trivial packages ourselves.

On the other hand, by writing some code ourselves, we deprive ourselves of the community’s sheer knowledge. Multiple people maintain popular packages, so they can quickly react to new vulnerabilities (for example, by monitoring the GitHub issues page of their package). This is something you, as a sole developer or part of a small organization, might lack the resources to do.

So the next time you are eager to install a package, check its source code. It might be a one-liner you can write instead of introducing a potential attack risk and slowing your CI/CD pipeline.

But don’t go too far with this method: you don’t want to reinvent the wheel or entirely deprive yourself of the community’s support.

Method Three: Extracting Sub-packages with `lodash`

Imagine we have the following object interface:

interface SomeObject {
  foo?: {
    bar?: {
      baz?: string;
    }
  }
}

And we have a function that accepts an object with that interface.

function getBazOrDefault(obj: SomeObject, defaultValue: string)

As you’ve guessed from the function’s name, it will give us the value of baz, or defaultValue, if the path to baz is undefined. Here is one implementation of that function:

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  if(!obj.foo || !obj.foo.bar || !obj.foo.bar.baz) return defaultValue;
  return obj.foo.bar.baz;
}

Ugly, right? It will get more ugly if you need to deal with arrays. Luckily, we can use the popular lodash library!

Install lodash, and the code becomes nice and easy to read:

import _ from "lodash";

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  return _.get(obj, "foo.bar.baz", defaultValue);
}

Neat!

However, if we install lodash:

npm i lodash

And its type definitions (because we use TypeScript):

npm i -D @types/lodash

We will introduce 8.4MB of dependencies to our node_modules. That’s a lot!

❯ du -sh node_modules
8.4M    node_modules

We have some experience now, so let’s put it to practice!

Package age — lodash is relatively old - it was published a year ago. I couldn’t find any JavaScript core functionality to get a nested value from an object by a string key. Let’s move on.
One-liner package — lodash is not a one-liner. It has tens of files and lots of tests. Even looking at the functionality of _.get, it’s not exactly a one-liner. While it’s a simple two-line function, it has an internal dependency on ./internals/baseGet.js, which in turn depends on ./castPath.js and ./toKey.js - and each depends on more files! It’s too much to be a candidate for extracting standalone code.

So it seems we are stuck with lodash then. But wait! There is another trick I want to show you! Sub-package extraction. If we read the lodash readme carefully, we will notice that every single functionality lodash provides is extracted into its own sub-package, including the _.get function!

Instead of installing the entire lodash, we can install lodash.get for just the _.get function (of course, don’t forget the type definitions). This reduces the node_modules size from 8.4MB to 3.6MB. It’s still a large folder, but a 57% reduction from the original size! I’ll happily take such a percentage reduction.

❯ npm i lodash.get
❯ npm i -D @types/lodash.get
❯ du -sh node_modules
3.6M    node_modules

The getBazOrDefault code will look like this:

import get from "lodash.get";

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  return get(obj, "foo.bar.baz", defaultValue);
}

In conclusion, be aware that many packages, especially collections of utilities like lodash, can be published to npm as individual packages. Instead of installing the entire 312 different methods from lodash, we can install the ones we actually need, reducing the weight of our node_modules.

Method Four: Do It Yourself

Last but not least is the DIY method.

Let’s say we need a method to capitalize the first letter of each word in a given string. We can run an npm search for the term capitalize and be presented with 266 different packages (not all relevant).

We can spend some time looking for a relevant, relatively maintained package and add it as a dependency to our project. Or we can write it ourselves!

Why, you ask? Software engineering is the art of problem-solving. To become a great software engineer, you need to be able to solve software engineering problems.

If you only know how to use third-party packages and copy code from Stack Overflow, you will be a software engineer, but not a great one. And you do want to be great, don’t you? Then let’s write our own capitalize function!

Let’s start with a signature:

function capitalize(str: string)

First, let’s identify words from our string. We can do that by splitting the string with whitespace:

const words = str.split(" ");

Then we need to capitalize the first letter for each word:

for(let i = 0; i < words.length; ++i) {
  words[i] = words[i].charAt(0).toUpperCase() + words[i].substring(1);
}

And lastly, we need to join the words back to the string:

const capitalizedString = words.join(" ");

Here’s the entire function:

function capitalize(str: string) {
  const words = str.split(" ");
  for(let i = 0; i < words.length; ++i) {
  	words[i] = words[i].charAt(0).toUpperCase() + words[i].substring(1);
	}
  return words.join(" ");
}

If you run this with some example strings:

console.log(capitalize("hello world")) // Hello World
console.log(capitalize("how are you?")) // How Are You?

Great success!

Note - this function is for demonstration purposes only. It does not consider scenarios like quoted words - for example, hello "world" will be capitalized incorrectly. However, this is a great opportunity to up your problem-solving skills! Go on and figure out how to capitalize quoted words as well!

As with the second method, you’ll want to avoid ‘reinventing the wheel’ here. On the one hand, you don’t want to write the same functionality over and over. On the other hand, you don’t want to introduce a dependency for every small functionality you need.

Always analyze and evaluate each solution. Many packages do way more than you need, so it’s better to implement the code yourself. However, there are good quality packages for common utils like slugify, which is well-maintained and provides basic functionality you probably wouldn’t want to implement yourself in a production environment.

Wrap Up: Decrease your Dependencies in Node.js

In this post, we ran through four methods to analyze packages and shrink your node_modules:

Checking the age of packages
Using on-liner packages
Extracting sub-packages
Doing it yourself

These methods should make your development process faster, reducing the amount of packages you need to pull every time you run your CI/CD pipeline.

All in all, the npm ecosystem is great! It is the biggest package registry as of today. But installing packages shouldn’t be a panacea. Our project is not only the code we write — it’s also the packages it consists of. Knowing what goes into our node_modules makes our projects more resilient and makes us better developers.

Until next time, happy coding!

This post was initially published in AppSignal Blog

Implementing Promisable setTimeout

Thu, 25 Aug 2022 16:00:00 GMT

In a world controlled by the event loop, setTimeout is the king of execution control. It allows you to schedule function calls to be executed later. However, it can’t do one thing - suspend the execution for a certain amount of time. Think of an equivalent to sleep(ms) from different languages.

Let us now explore how we can implement it.

Promisifying setTimeout

In order to make setTimeout behave similar to sleep we first need to promisify it. For that we will use the Promise constructor.

A short recap. new Promise((resolve, reject) => {}) accepts two arguments: a resolve function that we need to call when the promise resolves successfully (we can also pass the resolution value); and a reject callback that is called if the promise is rejected (we can also pass the rejection error).

Let’s look at a promisified setTimeout():

function setTimeoutPromise<T = void>(
	cb: () => T,
	ms: number
): Promise<T> {
	return new Promise((resolve) => {
		setTimeout(() => resolve(cb()), ms);
	});
}

We then can execute it like this:

setTimeoutPromise(() => "hello", 1000).then(console.log)

Voilà! And if we remove the callback, we can get a basic sleep function

function sleep(ms: number): Promise<void> {
	return new Promise(resolve => setTimeout(resolve, ms));
}

Which can be used as promise chain with .then() or as await-able inside an async function.

We now have a basic sleep(ms) function. I could end the article here, but setTimeout has some other cool functionality like the ability to be canceled. And so we can implement a cancelable sleep!

Imagine the following scenario. You want to do some async call to… let’s say an external API. But if the response takes too long, you want to return some known value. This is easily achievable with our setTimeoutPromise function.

const TIMEOUT = 1500;

// ....

const result = await Promise.any([
	getValueFromAPI(),
	setTimeoutPromise(() => KNOWN_VALUE, TIMEOUT)
]);

// do something with value

Now, we will either get the value from the external API or if it takes too much time (over 1.5s in this example), we will return a hard-coded value.

Ideally, after we’ve got the value, i.e. Promise.any was resolved, we would like to cancel the calls to the API and as well as the timeout because we don’t want to have unneeded things in our event loop. Most API calls can be canceled using AbortController (more on that later), but what about the setTimeout()?

Cancelable setTimeout

Not everyone knows, but setTimeout actually returns a value.

The returned timeoutID is a positive integer value which identifies the timer created by the call to setTimeout(). This value can be passed to clearTimeout() to cancel the timeout.

— MDN#setTimeout

So we get a unique timeoutID that can be used to cancel the timeout. It’s all nice and easy when the code is synchronous:

const timeoutId = setTimeout(doSomeWorkLater, 1500);

// ... some more code

clearTimeout(timeoutId);

How can we cancel a promiseable setTimeout? Well, our setTimeoutPromise function, instead of returning a promise function, can return an object that will contain two keys: the promise itself and a function that we can call to cancel the timeout, like this:

interface ReturnValue<T> {
	timeout: Promise<T>;
	cancel: () => void;
}

function setTimeoutPromise<T = void>(
	cb: () => T,
	ms: number
): ReturnValue<T> {
	let timeoutId: number;
	
	const timeout = new Promise((resolve) => {
		timeoutId = setTimeout(() => resolve(cb()), ms);
	});
	
	return {
		timeout,
		cancel: () => timeoutId && clearTimeout(timeoutId)
	};
}

Our then previous example turns into something like this:

const TIMEOUT = 1500;

// ....

const {timeout, cancel} = setTimeoutPromise(() => KNOWN_VALUE, TIMEOUT);

const result = await Promise.any([
	getValueFromAPI(),
	timeout
]);

cancel();

// do something with value

But what about `AbortController`?

I briefly mentioned AbortController, but for those who are not familiar with it, it’s a mechanism to abort web requests. It became a standard way to abort requests made with the request library and was adopted by axios as well.

In a nutshell, AbortController consists of two parts: the controller itself and a signal known as AbortSignal. The signal is given to the abortable targets while the controller remains in the hands of the one who wishes to abort the request.

const controller = new AbortController();
const signal = controller.signal;

doSomeAsyncWork({signal});

controller.abort();

It would be nice to rewrite our setTimeoutPromise to use the AbortController instead of the inconvenient cancel method we need to deal with.

Set timeout but with `AbortController`

The AbortController itself exposes nothing except for the signal and the abort method. Remember that we’ve said it’s just a tool to signal for abortion. Therefore we need to look at the signal itself, namely the AbortSignal.

If we look at the AbortSignal we can see it has two read-only properties: a boolean aborted to indicate if the signal was aborted, and reason which can be any value to indicate the reason the signal was aborted (it’s taken from the first argument passed to controller.abort(reason)). It also has a method throwIfAborted() which throws the reason. I suspect it’s just a shorthand for

function throwIfAborted() {
	if(this.aborted) {
		throw this.reason;
	}
}

None of them are helpful for what we need. Luckily for us, AbortSignal is also an EventTarget which means it can listen to events. One specific event that we are interested in - is the abort event. And with the help of addEventListener() method provided by the EventTarget - we can actually implement our setTimeoutPromise!

First, we need to make sure our function is able to accept the AbortSignal. The usual convention with AbortSignal is not to pass it as a standalone argument but as part of options.

Let’s define it

interface Options {
	signal?: AbortSignal;
}

function setTimeoutPromise<T = void>(
	cb: () => T,
	ms: number,
	{ signal }: Options = {}
): Promise<T> {
	// logic here
}

We still need to return a promise. Also, since it is possible to pass an already aborted signal, we need to have some edge case checking here

return new Promise((resolve, reject) => {
	if(signal?.aborted) {
		return reject(signal?.reason || new Error('Aborted'))
	}
});

However, if the signal was not aborted, we need to create the timeout and capture its id

const timeoutId = setTimeout(() => resolve(cb()), ms);

Now, what’s left is to subscribe to the abort event of our signal, clear the timeout and reject the promise

signal?.addEventListener('abort', () => {
	clearTimeout(timeoutId);
	reject(signal?.reason || new Error('Aborted'));
});

The final result looks like this

interface Options {
	signal?: AbortSignal;
}

function setTimeoutPromise<T = void>(
	cb: () => T,
	ms: number,
	{ signal }: Options = {}
): Promise<T> {
	return new Promise((resolve, reject) => {
		if(signal?.aborted) {
			return reject(signal?.reason || new Error('Aborted'))
		}

		const timeoutId = setTimeout(() => resolve(cb()), ms);

		signal?.addEventListener('abort', () => {
			clearTimeout(timeoutId);
			reject(signal?.reason || new Error('Aborted'));
		});
	});
}

We can rewrite our example from earlier like this:

const TIMEOUT = 1500;

// ....

const controller = new AbortController();

const result = await Promise.any([
	getValueFromAPI({signal: controller.signal}),
	setTimeoutPromise(() => KNOWN_VALUE, TIMEOUT, {signal: controller.signal})
]);

controller.abort();

// do something with value

Hence, no matter what promise resolves first, after we get the value, we can abort the execution of all the remaining promises.

And so, we’ve created our very own, promisified, cancelable version of setTimeout.

But… You don’t need all this

NodeJS v16 introduced Timer Promises API which are accessible under the timers/promises module. So you can get a promisified version of all the different timers such as setTimeout, setImmediate and setInterval:

import { setTimeout } from 'timers/promises';

const val = await setTimeout(1000, 'hello world');

console.log(val); // prints 'hello world' after 1s

And they even accept signal as the last argument to control their cancellation.

Outro

You might think “well if it’s available natively in NodeJS, why do I need to implement this myself?”. And you are right - you should not. If you need timers with promises, use them from timers/promises module.

But, I’m a big believer in learning through understanding how things work. Most of the knowledge I’ve acquired is because I was curious to know how things work: how operating system kernel works; how IRC protocol works; how instant messengers work; how a compiler works; etc.

When you take an existing, working solution, and try to reimplement it - you learn a lot about why and how decisions were made. And this makes you a better developer.

This is the first article in a series of Understanding Implementations, where I’m going to uncover how things are implemented, in order to help you become better developers. If you have suggestions on what other topics I should cover in this series, feel free to drop me an Email. Until next time!

Legacy is where companies go to die

Sat, 09 Apr 2022 16:00:00 GMT

Legacy. The one word engineers cringe when they hear. Nobody wants to work with legacy code or legacy systems. And yet someone has to. Can we do something about it?

The evolution of software

Let’s look at the evolution of software. It all starts with an idea. After some market research and market fit, a small team is formed to develop an MVP. The MVP then gets a few real users, which sets the ball rolling for team expansion and accelerated feature development. And as new features are being shipped, old features are becoming what we call legacy, and the backlog in Jira is getting filled with Tech Debts that are never prioritized because We must develop this feature ASAP!.

Eventually, the code becomes very hard to maintain. Most of the original authors already left the company and new engineers with enthusiasm to make the code a better place are met with criticism such as

This code is the core of the system, and any change there needs to be carefully reviewed by Amy’s team. But her team does not have time for it in this quarter.
Nobody really knows what’s going on there. Dany was the last to work on it, but he now left for UniCo(rn) which raised gazillion dollars.
Yeah, we can do that, but maybe in the next quarter. We really need you to work on Feature X for ThisBigClient.
I’m in! Let’s create a ticket in Jira Backlog and prioritize it in our next grooming!

And hence, their passion and enthusiasm slowly dies, until one day they become the ones who say the above phrases.

Legacy is the place where companies go to die.

If you fear legacy — you are doomed to fail

Code is just a means to produce products. The harsh truth is that your code quality doesn’t really matter, as long is it produces the product that your customers are happy to pay for. And in a very competitive world we live right now, you don’t really have time to produce maintainable code. New startups don’t have time or resources to focus on code quality — they need to provide POC as fast as possible to acquire that one customer. Established companies already have customers who give zero concerns about your legacy code, and are mainly interested in features.

However, we are not in the 15th century, where you work as blacksmith your entire life, and the cost and time you will spend investing into your workshop will eventually pay for itself. In the modern world, people are not their companies. It’s very common for engineers to change jobs every 3 to 5 years. And investing in a “better code” will not benefit future you, because future you will work in a different company in few years. Same goes for your manager, his manager and your product manager. Even the most veteran people, who work for 10 or 15 years at the same place, will usually occupy a completely different position by the time you approach them to ask for help about the legacy they’ve written 9 years ago.

Because good companies will usually outlive their entire employees, legacy creates a big problem for them. If you want your company to be good, you must have 2 magic components:

Ability for new employees to support and maintain your codebase without hunting for knowledge from veteran employees or those who already left
Ability to innovate

FooBaz Automobiles Inc.

I like to provide analogies from other industries, and I’m somewhat enthusiastic about cars, so here is one. Let’s say you start a car company and you rush your small engineering team to develop an efficient engine. Once your engine has been developed and you’ve provided a first prototype of your car, you begin to work on customer requests: better seats, nicer dashboard, more safety features etc. However as time passes, your engine starts to show its age.

Your new enthusiastic employee proposes to redesign the engine, but he is met with criticism: the current engine fits in the chassis we are developing; the engine is a critical part of the car; very few people know how the engine works and it’s better not to touch it; we promised our customers a big screen with navigation etc. As time passes, Acme Automobiles Inc is founded, and your enthusiastic employees are moving there, so they can innovate on their engine design, which proves to be twice as cheap and efficient as yours. And so little by little your company is phased out from existence.

Legacy code is like a viral infection. If you leave it untreated, it will spread and infect your entire codebase.

Version 2

Another side effect of legacy is what is known as “We can no longer develop new features in this codebase. It’s written in Ruby and it’s impossible to find any Ruby developers, so the Architects Council has come up with a decision to rewrite the software to Node.JS. We will call it V2“.

And while a rewrite sounds like a good idea from an engineering perspective, it’s a bad idea from a product perspective. There are many nuances you need to think about when doing v2.

Are you going to hire an entirely new team to work on v2, or are you going to repurpose the existing team from v1 to v2?
What is happening to v1? Are you deprecating it? Are you going to invest development time in it? What happens to customers of v1?
How are you going to sell v2 to existing customers in v1? Nobody wants to move from a working product to a half-baked prototype with lacking feature parity.

V2s are great for engineers, especially those who are going to lead the development. You get to work with new technologies (usually), greenfield a product, the feature set is usually clear from the beginning (because you are essentially remaking an existing product, but in a maintainable way). In the end, after tears and blood, you get a new, somewhat maintainable, codebase with no legacy. But make no mistake, the life path of V2 will follow V1s - eventually it will become legacy. Because V2s are no more than a bandaid on the big “legacy infection” of your codebase.

Oh no! What can we do?!

Have no fear! Legacy fighter is here.

The truth is — you can’t avoid legacy. I don’t believe in problem prevention, especially in software engineering. Problem prevention is equal to magic crystal reading. But this is a topic for another post. Let’s for now agree that legacy is the byproduct of software development. And the same way as a mess in the kitchen is a byproduct of Shabbat dinner preparations — it needs to be taken care of as soon as possible, before it piles into something you can’t handle.

This means that you need to have tasks that are prioritized in your sprint (or whatever system you use) in which you can tackle regions of code that are prone to become legacy. You don’t just fill your backlog with “Improve X so that it will do Y“ tasks and never work on them. You need to dedicate time. Find a balance between features and tech debts. Every sprint should have, let’s say, 80% features development and 20% tech debts. This also needs to be understood by your (a) Engineering Manager and (b) Product Manager. It must be a rule. There should be no exceptions. No “but maybe in this sprint we can focus more on features since we really promised this client that it will happen”. It’s not your problem, as a software engineer, that your product manager makes promises he can’t fulfill. It’s his job to balance the management of the product. Your job is to balance development of the product. And part of the development is making sure that your code continues to live after you decide to go to a different company; it’s your job to make sure that new developers are not faced with oh no, this is a core functionality, we don’t touch that in our shire.

Apart from fixing legacy, you need to make sure legacy is identified. Legacy does not just appear out of nowhere. Legacy is a good code that was piled on with a bunch of quick fix for client and a patch to make it work in Explorer type of development. Every feature and bug fix is a potential to identify legacy. If something is hard to understand, hard to work with — you need to document it. You need to make sure it’s known as a problem to the team, engineering manager and product manager. They need to be aware of the fact that you’ve tried to implement X but encountered a code that you’ve had to debug with other developers, because this code is hard to understand. And once they are aware of it, you prioritize it and start to work on it.

It’s an endless cycle. And it’s a healthy cycle. It’s part of development. And there will be sprints when you have a lot of features and very few legacy code, and there will be sprints where you have a big refactor to do and hence, you freeze feature development.

Legacy is not a boogie man. You don’t need to be afraid of it. But if you let it grow, it will grow. And will eventually kill your product and maybe even your company.

It's better to be (type)safe than sorry

Sat, 19 Feb 2022 16:00:00 GMT

Take a good look at the following function and try to understand what it’s doing.

function do_magic(a, b) {
  return a + b;
}

Is it summing two numbers? Or maybe it’s doing string concatenation? The answer is: you can’t know. And this beats the entire purpose of programming languages – to be easily understandable by humans.

A brief history of dynamically typed languages

There is a long list of dynamically typed languages, but for simplicity, I’ll focus on a few of the most popular.

PHP

PHP first appeared in 1995, and is officially categorized as a scripting language. It was created by Rasmus Lerdorf to maintain his personal homepage, which Rasmus wrote as CGI programs in C. It was never intended to be a programming language and Rasmus even said: “I don’t know how to stop it, there was never any intent to write a programming language […] I have absolutely no idea how to write a programming language, I just kept adding the next logical step on the way.”

It is now powering 78.1% of all websites, according to W3Techs report from January 2022.

Python

First appeared in 1991. Its development started 2 years before that, in 1989, by Guido van Rossum. He was working with Amoeba operating system and needed a better language to do system administration, rather than writing C programs.

Python consistently ranks as one of the most popular programming languages.

Lua

Created by Roberto Ierusalimschy, Luiz Henrique de Figueiredo, and Waldemar Celes in 1992, Lua was primarily designed to be embedded in other applications.

Today, Lua is the leading scripting language for games.

JavaScript

Designed by Brendan Eich and appeared in 1995, JavaScript was created for one sole purpose – add dynamic capabilities to boringly static HTML pages.

Today, JavaScript is one of the core technologies of the World Wide Web and over 97% of websites use JavaScript.

As you can see, none of these languages were designed to be robust and replace the old C or Pascal. They all managed to find their niches, but they were all languages designed to either be used for personal needs, or with the main purpose of extending other applications which were written in statically compiled languages.

None of these languages should be used, in their pure state, as the primary languages for application development. Let me elaborate on that.

The human factor

If you’ve ever worked on a serious project, you know that extending and maintaining it – is a challenging task.

How often you’ve looked at your code, or code you peer reviewed, and you said to yourself, “What a nice, clean solution”? But after a few months, when you’ve revisited the code, you thought to yourself “How this piece of crap ended up in our code base?”.

When we write code, we are unable to keep the entire context of “why” that we have in our head at the moment of coding; hence we end up looking at old code and crying. And the fact that we take away types – only makes things worse.

One little bug

I remember hunting a bug in one of the system I was working on. The system had entities that were versioned, and the version was universal across the entire entity domain. Since the number of entities was huge, and with each entity modification, we needed to increase the version, we could easily overflow the 64bit integer. So we’ve decided to use BigInt column in the database to track the version.

For each create or update operation on an entity, we would call the following function:

async function getCurrentVersion(howMany = 1) {
  const currentVersion = await magically_get_current_version_from_db();
  const nextVersion = currentVersion + howMany;
  await magically_set_current_version_in_db(nextVersion);

  return currentVersion;
}

Note: I’m reconstructing the code from memory.

The entire function was wrapped in a database transaction (to avoid race condition), and the “*magically_**” functions were actually calls to our ORM (please don’t use ORMs) which would fetch the version from the DB, then reserve 1 or more (we could have batch updates) versions, and save it as next.

What we saw in our development environment was that entities were out of order, and we saw a considerable increase in version numbers. We’ve reached massive numbers very fast.

Just for an exercise, try to understand what was the problem. Continue reading once you’ve shed some tears.

It turned out that our ORM returned BigInt as strings. So if our current version in the DB was 123, it would come back from the ORM call as "123", and adding an integer (let’s say 1) to it, turned out to be… poopity whoopity… "1231".

I’ll pause for a minute, so you could enjoy your moment of pseudo-superiority and express your nasty comments about my mediocre programming skills, because we all know that you write perfect code, and compile it in your head before V8’s runtime even finishes interpreting it.

My point is – it would not have happened in a statically typed language. No compiler in the world would just let you sum a string with an integer, or would at least warn you that you are about to do something stupid.

And yet, when I proposed people that we need to switch to Typescript, I’ve encountered comments such as: “but I want fast iteration”, “I had enough of Java headaches”, “But then I have to define every type” and my favorite “But we don’t have errors related to dynamic types”.

This bug is only one of many I’ve dealt with in my life. I don’t have enough bits to count how many times I had to print objects just to see their contents. Is it obj.foo_bar or obj.fooBar? Not having types is like not having street names. You need to memorize the entire city and knock on each door and ask what street and what number is it. You have to keep the entire project and all of its dependencies in your head (and we all know that node_modules is like a black hole, so it can’t be kept in your head). Otherwise, you risk forgetting that some library decided not to use native BigInt, but instead return strings 🤷‍♂️.

The cost of bugs

With the shift from desktop application towards SaaS software, and because of humans losing patience, developing software became a 24/7 task. You no longer have engineers who write code from 9to5, but you also have 24/7 on-call engineers (which nobody really wants to be) – hence the cost of such bugs is colossal.

Not only it harms your customers who cannot access your products; it harms your company, which loses money. And it harms you – because Murphy hates you, and it’s your turn to be on-call. So, suddenly, you find yourself half drunk in a bar, on Friday evening, trying to understand why your BigInt is overflowing, while your friends take Vodka shots off your MacBook Pro. But hey, at least you don’t need to define EVERY TYPE, like a bizarre monster who enjoys writing interfaces in Java.

They aren’t made for it

None of the dynamically typed languages we are using in our production today are made to be used in such way. You keep hearing about how PayPal or Uber are switching to NodeJS, while forgetting about the fact, that their adoption of NodeJS is for a tiny API Gateway that proxies requests to their huge Java monoliths (probably; I’ve never worked in those companies, but I have 10+ years of software development experience, and know a thing or two about maintaining legacy code).

You keep embracing dynamic languages, while the world keeps deprecating them. Facebook developed Hack to introduce types into PHP. Even PHP now has support for types. Microsoft came up with TypeScript to ease the pain while writing JavaScript applications; Python 3 has support for type hinting; and surprisingly, but in Lua "123" + 1 produces 124, and "123" + nil throws an error. Want to guess what’s the output of "123" + null in JavaScript?

It’s better to be type safe than sorry

A world of fast iterations, race to market and other marketing terms – has no place for dynamically typed languages for business critical applications.

Don’t get me wrong, dynamic languages are great at what they were designed to do. Python excels at AI and ML applications; Lua is perfect as embeddable scripting language for a type safe application; and JavaScript… it’s still terrible, so please, at least, use TypeScript.

Nah. I don’t make stupid type related errors – Well, go to your logs and search for how many cannot read property x of undefined you have. Chances are you have a few of those logs, and your code is scattered with const bar = foo || {} - because computers are made to be deterministic, but you are a free spirit and your foo might or might not be an actual object.

I have tests to make sure I never sum integers with strings – Well, guess what. If you had let compilers do their job, you wouldn’t have to spend developer’s time on writing guard tests for type related problems. Java developers don’t spend time writing tests to make sure int x is an integer.

Type safety is not some esoteric obsession of programming geeks in basements. It’s business critical for every company that want to stay in business. And I’d even radically say to you, and ask you to politely tell your manager, that making your code type safe should be treated the same way you treat customer bugs, and not buried in a technical debt pile. Use type hinting, switch to TypeScript or a real statically typed language – I don’t really care, as long as you do it.

You owe it to yourself and your sanity. And if not yourself, then for the well-being of your company. And if not your company, then, at least, for the satisfaction of your customers.

The Software Industry is Broken

Wed, 26 Jan 2022 15:00:00 GMT

It’s that time again. You come to your Toyota dealership to buy a new Toyota Corolla. You’ve been waiting to upgrade your old, fuel inefficient and slow model, with this year’s new model. But the new model is more fuel inefficient! Wait! What?!

The evolution of products is very clear. Newer should be better. Better equals more fuel efficient, faster, safer. Otherwise, why would you upgrade? Have you ever heard of your friend bragging about his new fridge, telling you how roomy it is, how it has an RGB light inside, but it uses more electricity? 😦

Or get this. Have you ever heard about a major airline, purchasing a new fleet of Boeing (or Airbus) airplanes, just so they will be less fuel efficient, make your flight longer and the airline needs to rebuild their hub airport because the airplane now can’t taxi to the gate?

No, you haven’t. Because that’s not what progress is. Progress is a process towards better, not worse. You don’t buy a new car to drive slower or pay more for gas.

But you know what? In software, that is what we are doing. I’m writing this article, while Spotify is playing in the background, and a quick look at Activity Monitor reveals Spotify has 6 processes running, with an average of 2.2% of CPU usage, and a combined total of 470MB of memory usage! 470MB!! Winamp uses around 49MB (with a VM size of 60MB) source. And Winamp has visualizations ffs!

Wina-Winamp! It really whips the llama's ass!

Product Segmentation

I’m a developer. I try to have the best machine I can. Max memory, top CPU, best GPU (if I’m working with 3D graphics). We expect it from professionals. The same way professional car drivers try to get the best car; professional mountain bikers own the best bikes; pro photographers get the best camera and lenses.

But every product has segmentation. Some products are entry level, other aim at professionals. You can get a decent DSLR camera or an entry level mountain bicycle. And you can get an entry level PC or aim at a beefy machine with 64GB RAM and Ryzen 9 5900x (should you need that). But while your entry level DSLR will snap exceptional pictures, and your beginners mountain bicycle will get you through beginner singles, your entry level PC will struggle to keep up with basic software like music players or instant messaging.

640K ought to be enough for anybody

It’s Saturday and I’m on my home PC, so I don’t have Slack open, but a quick Google reveals that slack uses at max around 1GB of RAM. Add to it Spotify, a Chrome with few tabs, the OS itself and you’ve maxed out 4GB of RAM easily.

But Slack is a professional tool! - no. No, it’s not. It’s a glorified IM with slightly more features that mIRC. And mIRC uses anywhere between 50MB and 250MB (source). Slack is used in a professional environment, yes, but it does not justify the fact that it eats a good chunk of my memory that I need for my professional work such as an IDE, running compilations and VMs.

We are in an era when software grows way bigger and heavier, but does not add any features or improvements. Spotify, for example, does not have an equalizer, a feature that Winamp does. Instead, it has friend’s activity, so I can see what are my friends listening to, because I have nothing else to do. Games used to be distributed on single CDs, then multiple DVDs and now you need to download tens of gigabytes over network in order to enjoy the last AAA titles. But with the increase in size and memory usage, we also got an increase in quality and visuals. We went all the way from rectangular faces like this:

Busted

To near realistic graphics like this:

Toss a coin to your Witcher

640k ought to be enough for anybody - is a quote falsely attributed to Bill Gates ¹. Although he never said that, and obviously staying within the limits of 640K is not progress, the current status quo is not acceptable! We can’t, and shouldn’t, argument everything with “But RAM is cheap”.

Today’s software gets heavier and ships with fewer features. There is a word for this process in the English dictionary: regress. It turns out that it is the antonym of progress.

Why it happens?

Compared to 20-30 years ago, today software is everywhere. It’s even in your fridge. Think of anything, and high chance it has software in it. And with a significant increase in software availability, a matching demand for software engineers came along.

And as with any factory production line, optimizations were made. New languages were created. From an old and clunky Fortran, where you write code such as:

INTEGER(4) :: i,io_unit
REAL(8) :: pmin
STACK_PUSH

All the way to the new and shiny JavaScript where you write poetry:

request(app)
  .delete("/something")
  .send()
  .expect(400)
  .end((err, res) => {
    err.should.have.proptery("message").equal.to("Bad Request");
  });

As something becomes more mainstream, it takes a simpler form. First cars were hard to drive, today you don’t even have to change the gears. Same with software. Back then you had to understand what are registers and how many bits your integers take. Today, everything is dynamic and the compiler or the runtime knows better than you what types to use.

While I struggle with this concept sometimes, because I grew up on understanding the difference between little and big endian; I understand the reasoning behind it. When a specialty tool is being adopted by the masses, it will be simplified to suit everyone as much as possible. As long as the adaptation includes improvements, I’m fine with it. This is progress, and the reason we, as humanity, can enjoy technology in the way we enjoy it.

Creating software today is simpler than ever before, with high level, garbage collected, dynamic languages and block-like programming. And the world of software creation opened a lot of business opportunities. You practically can create anything from nothing.

A personal story

Around the age of 16, I had to choose a specialization in school. Once you reach 9th grade in school, you need to decide what specialization you want to get based on your desire and grades. There were specializations like psychology (focusing less on math and more on literature, psychology and philosophy); robotics (with focus on math and programming, and less on literature or psychology); computer science (with higher focus on math and programming, and less on literature and psychology) and others.

I was torn between robotics and computer science. I was sure it’s going to be the biggest decision of my life (boy, I was wrong). I remember the summer break between 8th and 9th grade, where I was struggling to decide what do I want to do, where do I want to go. I enjoyed creating things by writing code and my major struggle was to decide whether I want to create virtual things like software, or make software create real things like robots.

Programming was already a hobby of mine. And I choose computer science. One of the key factor for my decision was the fact that I come from an immigrant family. We are not poor, but I couldn’t allow myself asking my parents for money to buy resistors, breadboards and soldering station. I’ve already had a computer and knew programming and with that I could create software.

Software production is relatively cheap

My “biggest” life decision boiled down to the fact that I had a PC. The upfront cost of software creation is relatively cheap. You don’t need to buy programming languages like you do with heavy machinery, nor you have to rent an office, like you do with a factory or a warehouse.

And the rewards you can get from software production are massive. We are in a midst of a software revolution, where VC funded startups are being spawned like zombies in Left 4 Dead 2.

Look! Startups are everywhere

Nobody cares

Software surrounds us, and yet, in my opinion, it’s one of the inefficient industries we have. The solution to slow software is to buy more RAM or spin up yet another AWS instance. Ain’t nobody got time to figure out what’s wrong. Optimization became a hated word used by heretics. All our optimization problems are solved by increasing the instance type or instances count. In the end, everybody suffers and nobody really cares.

Users suffer. But they don’t care. They have nothing to do. Their choice is mostly between either using the product or not, or between two different software when neither of them is optimized enough or lack features.

Developers don’t care because it’s too hard to write proper software. It’s too hard to do proper error handling or debugging. They prefer to spend their time in creating FizzBuzz architectures and appearing smart in front of their colleagues during design reviews. And their simple To-Do app is grown to a pile of shitty abstractions that nobody can understand how to debug and optimize.

Imagine your car asking you to pull aside, turn off and then turn on because 'something went wrong'

Business and VCs obviously don’t care as well. They care only about 3-letter-abbreviates such as TTM and ROI. The lower the first and higher the second - the happier they are. Tech debts are being buried beneath piles of optimization bugs that are fixed by occasionally restarting the servers. At least we do a retrospective!

When I type google.com in my browser, my computer issues a network request that, according to traceroute tool, travels between 20 different network hops and takes about 75ms to reach Google servers. 75ms. Let that number sink for a moment. It travels between 20 different routers, most of them are with shitty (compared to your phone, yeah?) 1GHz CPUs and 1GB of RAM, serving millions of requests, via fiber or copper wiring, to a different continent! LET THAT SINK! And yet, it takes few seconds from the moment I open slack till the moment it’s useable, and I have fully spec-ed Mac Mini. 🤯

The future

We had 4KB or RAM when we landed on the moon², but today it takes 100000 more to send your colleague a message. Yes, RAM is cheap. Yes, machines are powerful. But progress is about doing the best you can. And I’m pretty sure that 400MB IM is not the best we can.

There is beautiful software out there. Efficient, snappy, feature-rich. There are engineers who care about performance, who care about getting the maximum out of the available technology. Not wasting resources. But they are in the minority.

Imagine a world where you can open a software and it’s useable right away. You don’t need to wait for it to load the redundant friend’s activity. It doesn’t have useless features. And it works on your entry level PC. Such a world is possible, and we are responsible for making it.

I encourage everybody to take an action. Refuse to write inefficient software. Demand time for Tech Debts that focus on optimization. Be open-minded and choose the best tool for the problem. Together we can fix the software industry.

NodeJS Native Module vs WASM

Sat, 11 Dec 2021 22:00:00 GMT

In my previous post about Native Rust Modules for NodeJS, people asked me how neon bindings would compare to WASM. Let’s check!

What is WASM?

WebAssembly (sometimes abbreviated Wasm) is an open standard that defines a portable binary-code format for executable programs, and a corresponding text format, as well as interfaces for facilitating interactions between such programs and their host environment.The main goal of WebAssembly is to enable high-performance applications on web pages, but the format is designed to be executed and integrated in other environments as well, including standalone ones.

— Wikipedia

WASM (or WebAssembly) is a binary code. It is executed by a portable VM that is currently implemented in all major browsers.

However, as Assembly, WASM is not that pleasant to write in and therefore other languages, and Rust among them, support WASM as compilation target.

The purpose of this article is to:

Do a quick introduction to compiling Rust into WASM
Benchmark Pure JS, Native Module and WASM Module implementations of the same function
Provide basic guidelines towards when to choose either implementation

Please make sure you’ve read my previous article, because this one is built on it. You can find the full code for this tutorial on GitHub. Let’s dive in.

How to compile Rust to WASM

For native modules we have Neon, for WASM we have wasm-bindgen. wasm-bindgen gives the tools needed for importing JS functions into Rust and exporting Rust function to JS. Let us look at how we can create a WASM version of our fibonacci function.

First, we need to make sure our Cargo.toml lists wasm-bindgen as dependency like this:

wasm-bindgen = { version = "0.2.78"}

After that, everything that is left, is to mark our function with a special macro, like this:

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
fn fibonacci(n: i32) -> i32 {
    return match n {
        n if n < 1 => 0,
        n if n <= 2 => 1,
        _ => fibonacci(n - 1) + fibonacci(n - 2)
    };
}

And that’s it! 🎉

Thank you for reading. See you in the next one.

Not really. I suggest you read the wasm-bindgen Guide, there is a lot of information about how to interact with JS from Rust and vice versa and information regarding what types are supported and how you can transfer them between the two languages.

In order to build the WASM, we need another tool, called wasm-pack, We then can execute the following command wasm-pack build --target nodejs, which will produce a NodeJS compatible WASM (you can read more about different targets here). The resulting code will be placed in pkg directory and will contain 4 files:

index.d.ts – Typescript definitions for our module
index.js – The main file that will initialize the WebAssembly module and create a WebAssembly instance that we can interact with (feel free to take a look inside, its very readable)
index.wasm – The actual machine code of the WebAssembly
index.wasm.d.ts – Typescript definitions for the WASM file

You then simply require the files from pkg dir and use them as regular JavaScript (or TypeScript).

const {fibonacci_wasm} = require("./pkg/index");
const result = fibonacci_wasm(number);

What about Performance?

Ah. Performance. The thing that Javascript developers care for the most.

I’m going to use the hyperfine tool, run each fibonacci with 3 warmup runs, computing different numbers, taking the mean running time, and present you the results. Please note: These are not laboratory grade benchmarks.

All benchmarks are run on 2020 Mac Mini M1 with 16GB of memory.

Runtime	30th Fibonacci	44th Fibonacci	45th Fibonacci	46th Fibonacci
JavaScript (NodeJS)	165.2ms	5.846s	9.358s	15.038s
Native Rust	161.5ms	2.271s	3.578s	5.721s
Rust WASM	163ms	3.286s	5.207s	8.317s

Analysis

I hope no body is surprised that Rust took the 1st place, followed by WASM, while JavaScript was finishing last.

It’s interesting though to see that on low numbers, such as 30th Fibonacci number, all 3 methods performed roughly the same, with Rust being 2.23% faster and WASM being 1.33% faster. This proves again that you always need to benchmark a specific function / method before assuming that a switch to low-level languages will perform better.

However, once we go higher in the Fibonacci numbers, we can see a clear difference between the 3 methods.

We can see that for JavaScript, the jump between 44th and 45th Fibonacci numbers resulted in an increase of 60.07% in time, while the jump from 45th to 46th number resulted in 60.69% increase.

For Native Rust module, the increase resulted in 57.55% and 59.89% respectively, while for WASM the numbers were 58.46% and 59.72% respectively.

Conclusion 1: Rust was more efficient in computing the next number in the chain followed by WASM while JS taking bronze medal in this race.

Let’s continue. Native Rust was 61.15% faster than JS in computing the 44th number, while WASM was 43.79% faster. For 45th and 46th numbers, Rust was 61.76% and 61.95% faster respectively, while WASM was 44.35% and 44.69% faster, respectively.

Conclusion 2: Opting for Native Rust module, increased the performance by 60% on average! Opting for WASM, increased the performance by almost 45%.

Rust was 44.69% faster than WASM in computing the 44th number; 45.52% faster in computing the 45th number, and 45.37% faster in computing the 46th number.

Conclusion 3: Rust on average was 45% faster than WASM.

Summary

I hate benchmarks. Especially on useless functions like Fibonacci. You should always do side-by-side comparison for your use case, rather than rely on benchmarks like this one. However, this benchmark can give us some base line assumptions. We can see that JavaScript is fast enough on lower Fibonacci numbers, however, it struggles as we go higher.

It’s no surprise that WebAssembly is taking a strong second place. One of its goals was to provide near-native code execution speed. However, considering the fact that WebAssembly is executed by a VM, it will be hard to achieve real native performance, like with Rust.

What you should choose and when?

I won’t discuss when and why you should choose JavaScript. Chances are, if you are reading this article, that you’ve already chosen JavaScript but looking at how to improve performance in critical parts of your application. Therefore, I’ll focus on providing the differences between Native modules in Rust and WebAssembly.

Native or WebAssembly?

First, you need to understand the major difference between the two. Native modules are modules that are written in a compiled language like Rust or C/C++ and are imported via node-ffi to use in NodeJS applications. They are no different from .so or .dll files that are loaded by, say, Java JNI or any other language with FFI support.

WebAssembly on the other hand is a compilation target. What it means is that the best way to write WASM is to actually compile another language, such as Rust or C/C++ into WebAssembly. Yes, you can write WASM directly, by writing a WAT (WebAssembly Text) files and then translating them to WASM by using wabt, but why would you do that? I won’t argue that x86 Assembly is a useful language to know for certain applications (for example extreme optimization cases for Games or Digital Audio Workstations), WASM, on the other hand, as a language is useless, because many higher-level languages can be compiled to WASM. And since WASM is mainly a compilation target, I assume the difference in performance between a C++ or Rust implementation compiled to WASM will boil down to the actual performance of the code written in C++/Rust, and how good are the tools that compile said code to WASM (Emscripten in that case). Other than that, there should be no performance difference whether the WASM was compiled from C or from Rust.

Having said that, there is one big difference between the two. When I’ve started to do the research for this article, I wanted to focus on a more realistic problem rather than Fibonacci. I’ve downloaded at 14.7MB CSV file that contains ranges of IPv4 addresses that are mapped to different countries. My idea was to parse this CSV using the 3 methods, load the data into memory as an array, and then perform a lookup of 50 random IPs to find out to which countries they belong to. This would simulate a scenario where you have huge chunk of data being loaded into memory and you need to scan it to find a specific value.

I’ve successfully completed the task for JavaScript and for Native Rust module, but when I’ve finished writing it for WASM, upon opening the CSV file from the Disk - I’ve got a panic call. This was the first time I’ve realized the biggest difference between Native Rust module and WASM.

A short trip to the land called `std` - The standard library

You see, compiled languages can be compiled for different operating systems and target architectures. The two biggest architectures as of today are x86 / x86_64 (The standard 32bit and 64bit architecture we know, that are implemented by Intel and AMD) and armv7 thanks to Smartphones, Raspberry PI and the recent Mac M1 processors. Each architecture handles things like floats and math differently, therefore gcc (the C compiler) and rustc (the Rust compiler) need to know how to produce the machine code suited for each architecture. On top of all that, each operating system such as Linux, MacOS and Windows, has it’s own way to manage things like file descriptors and network operations. When you write File::open in Rust, what it does is actually calling the OS defined method to handle file opening. It fact, the File struct in Rust is no more than simply

pub struct File {
    inner: fs_imp::File,
}

While fs_imp::File being the actual filesystem implementation (be it NTFS or ext4) which is handled by the OS . ¹

std is very interesting, I even had some small experience implementing my own memset, around 10 years ago when I was toying with my own OS kernel. But what’s the connection to WASM?

Well, remember I’ve said that WASM is a compilation target? So essentially, when you are running wasm-pack somewhere down the line it calls cargo build (which in turns calls rustc) and it passes cargo a special argument, --target that is equal to wasm32-unknown-unknown. It mens that we are compiling a wasm 32 bit code on an unknown vendor with unknown system (other examples of target might include things like x86_64-pc-windows-msvc meaning that we are compiling to an Intel/AMD 64bit architecture, on PC, for Windows using the MSVC ABI. You can list all supported targets with rustc --print target-list).

And since WASM is a compilation target with unknown vendor and system, there is no stdlib!

In fact, we can go to the unsupported system in Rust source code, and see that File::open actually calls to unsupported() ² function, which simply panics.

And this leads me to the biggest difference between the two — if you need to use any of the stdlib utils such as accessing the filesystem, accessing network, threads and anything related to the OS - chose native Modules. WASM simply can’t support this functionality, since it was designed to be executed by a VM that is running in a sandboxed environment.

Correction

Upon reading more, I’ve realized that there are 2 compilation targets for WASM: wasm32-unknown-unknown which we’ve already seen and wasm32-wasi. WASI or Web Assembly System Interface is, a still in development, standard to get safe access to some resources of the OS such as fd_read, fd_write and etc. Some WASM VMs provide support for WASI, but for my understanding, no browser currently supports WASI as it’s not fully standardized. Back to the article.

Below is a simple guideline when to choose Native vs WebAssembly (as I don’t have vast experience with both methods in order to give you a definitive flowchart).

Comparison between Native Modules and WebAssembly

Performance

Due to the fact that WASM is executed by a VM, native modules will, most likely, be more performant that their WASM counterpart.

Reusability

Native modules are also reusable. We can use them in any other language that supports FFI. So, for example, if you have a shared logic in Rust native module, you can load it from Nodejs and from Python (using CFFI) or Ruby (using Ruby FFI). WebAssembly on the other hand, can only be run by WASM VM.

Ergonomics

In my opinion, the ergonomics of wasm_bindgen are way better than neon. I like neon, don’t get me wrong, but all you need to do in order to export a function from Rust to WASM is to add #[wasm_bindgen] before the function. With neon you need to mess with conversions.

Portability

Native modules depend on the host machine. WASM being run by a VM is very portable format — get a .wasm file and it’s guaranteed to run in any environment that has a WASM VM. Native modules need to be recompiled for each host. It’s not that big of a deal if you are using lean docker containers such as alpine but it is something you need to be aware of, and the “I don’t know, it works on my machine” becomes a real issue.

Entry barrier for new developers

Native modules can be written, mainly, in two languages: C/C++ or Rust. C/C++ get a lot of hate, while Rust is the most loved language ³. But Rust is not simple. In C/C++ you fight with core dumps, in Rust you fight the borrow checker. It’s not the most simple language to grasp for someone who is not familiar with the concepts of memory management and pointers.

WASM being a compilation target, can be produced from many languages. The main ones are C/C++ and Rust, but there is also partial support for Python, Java, Ruby and Go. And once WASM will support multi-threading and garbage collection, C# will be a candidate as well. More over, there is a special language, with TypeScript like syntax, called AssemblyScript, that was created with one purpose — to be compiled to WASM.

So the entry barrier to WASM is simpler, in my opinion.

Node vs Browser

While both this post and my previous one focuses mainly on NodeJS, we can’t forget the browser. JavaScript is still, and probably forever will be, king in the Browser. And Native modules can’t be executed in the browser. Or any other place that is not NodeJS runtime. JavaScript does not (yet?) support FFI.

So in case you are developing for the browser, or any other system that has access to JavaScript runtime and WASM VM – your only choice is WebAssembly.

Conclusion

Native modules and WebAssembly are both performance optimizations to JavaScript, but they are here to solve different problems. Native modules are here to extend NodeJS with performant code while giving you full access to stdlib. WebAssembly is here to replace non-performant JavaScript code with a near-native performance binary that is executed in a sandboxed environment.

JetBrains Fleet Preview - First Impression

Fri, 10 Dec 2021 12:00:00 GMT

Last week JetBrains launched their new Fleet IDE¹ and I’ve been lucky enough to get the Preview version. What do I think about it?

I must admit that I adore JetBrains tools. I’m using them since the days they had ReSharper and PHPStorm. It is the single most productive piece of software I own for programming. I’ve used almost all of their tools including IntelliJ IDEA, PHPStorm, WebStrom, CLion (with Rust plugin), GoLang, PyCharm, Resharper (which is now more or less obsolete since they have Rider), and DataGrip.

Of course, when I’ve read the news that they released a new IDE called Fleet, I had to get it.

Fleet at a high level

Fleet is a very lightweight IDE. I never complained about JetBrains tools, I know some people consider their tools to be heavy, I never had any problems with them, but Fleet feels more like a text editor rather than an IDE, in terms of speed.

Fleet is also very simple. I don’t know if this is intentionally or just limitations of the Preview version. In terms of architecture, Fleet is a total rewrite and it builds upon great tools such as Rust, Kotlin, and Skija + AWT ² as well as on 20 years of JetBrains expertise in writing IDEs. Fleet has a distributed architecture which you can read more about here.

Fleet is polyglot meaning it supports the following languages and technologies: Java, Kotlin, Python, Go, JavaScript, Rust, TypeScript, and JSON. According to JetBrains announcement, support for PHP, C++, C# and HTML is coming soon.

And this marks a step in the right direction. Development today is a lot more than just Java on the backend and JavaScript on the frontend. Developers are writing API proxies in NodeJS, microservices in Go, and business logic in Java. The need to constantly switch between different IDEs is not that comfortable, and with Fleet, you can have your entire codebase in one IDE.

Let’s take a deeper look at some of the features of Fleet.

The UI

Fleet has 2 main components: The Editor and Panels.

The editor is, well, for editing your code. Apart from the editor, you can have up to 3 panels: left, right and bottom. Each panel can be configured to show multiple tabs that can show one of the following:

Files Tree - the files tree of your project
Debug - debug output when debugging the application
Git - the Git changes with an option to commit them
Git history - the history of the git repository
Search - search results
Terminal - the terminal

Fleet UI with 2 panels

The UI is also context-aware and can show you information such as how many implementations your Rust struct has.

Fleet showing "Code Vision"

There is also an overview window in which you can switch between files, perform actions, jump between different tools and perform a search. It reminds the catch-all window you’d use by double-tapping the Shift key in their other IDEs.

Overall the UI is very minimal and clean which again makes you feel like you are using a sophisticated text editor rather than an IDE.

Few things I miss from the original JetBrains IDEs, and I hope that those are just limitations of the Preview version:

I’d really like to see Dracula theme. I believe this is the best IDE theme that was ever created, and the default dark theme of Fleet is not appealing.
In the files tree, I’d like to have a different color for Git ignored paths. Original JetBrains IDEs tend to color ignored folders as well as mark library roots (such as node_modules). This makes project navigation way easier.

Collaboration

The years 2020 and 2021 marked a new era in the software engineering industry - the era of remote collaboration. Fleet is built with collaboration in mind. It allows multiple people to work simultaneously on the same files, rut test, and more.

I never worked on collaborative IDEs, but I believe it’s a nice addition for distributed teams or as a remote pair-programming.

Focused more on hacking than UI

I’m not sure if it’s intentional or just the limitations of MVP, but in Fleet, you don’t have fancy configuration windows. Want to enable or disable an IDE setting? Create a .fleet/settings.json file and use this reference to configure Fleet.

What about Run Configuration? Create .fleet/run.json and consult this reference for settings.

A great usage for this is actually checking in your run.json to git so that everyone on the team can have the same run configurations for the project. I know you can technically do the same with their other IDEs, but I have never seen anyone commit their .idea directory.

Some people might like this approach, others might not. I, personally, feel comfortable with configuring my IDE via JSON files.

What about VS Code?

A Fleet review without mentioning VS Code? 😱

Well, the truth is… I don’t like VS Code. I never understood the hype around it. When I tried it, it was slow, it was clunky.

BUT TEH PLUGINZZZZ!!!!111. Yes, looks like the selling point of VS Code. Plugins. I personally don’t see it as a selling point. I want to open my IDE and code, not mess with plugins and incompatibilities. (Yes, I use plugin in JetBrains IDEs. But those plugin are mainly language additions like Rust or TOML support, and not make-your-text-editor-ide-like sort of plugins).

So sorry folks, no VS Code comparison today.

Final notes

I’m excited about Fleet. It’s a refreshing IDE that suits the new realities of the world. It’s built on Skia, a C++ graphics library as opposed to Electron. It’s fast. It’s snappy and it’s minimal.

I won’t, however, ditch my other JetBrains IDEs, yet. Fleet is still missing some things I use daily such as auto-import in Rust; support for HTML/CSS (which should come soon); reopening previous files after restarting Fleet; confirmation windows for deleting files. I’m sure Fleet will improve. JetBrains has the tools and knowledge to do so.

For me, Fleet takes a great place as an IDE instead of Sublime. It’s perfect for things like messing with my blog (which is powered by Hugo), or doing a quick code for an article (new post coming soon, subscribe to my newsletter to get updates).

JetBrains did not say anything about pricing as well. As someone who pays for the All Products Pack, it will be great to see Fleet being offered as part of it, otherwise users like myself will have a hard time justifying paying for another IDE while having access to all the other JetBrains tools. On the same note, it looks like Fleet is entering the ground of VS Code as its main rival. And since one of the main selling points of VS Code for developers, is that it is free, Fleet will have a hard time competing with VS Code if it will be paid.

I’ll keep following Fleet. I hope it will position itself correctly in the toolbox of IDEs.

[Fleet] 1 ↩
[Twitter] 2 ↩

How to be a Great Technical Interviewer

Sat, 27 Nov 2021 17:00:00 GMT

The path to becoming a great technical interviewer is full of doubts, tough decisions, and self-discovery. But I believe every engineer should try to walk it.

In my career, I’ve interviewed close to 150 people by phone, on-site, and recently — via zoom. I’ve criticized the existing on-site interview and took ownership to reshape it. I’ve inspired others to reshape the phone interview. I am mentoring and onboarding new interviewers to the pool. And I’ve realized that the technical interview is not perfect. Before we jump to explore the problematic points, let’s first learn how we came to it in the first place.

It all started with a light bulb

Or to be precise, with the person who invented the light bulb — Thomas Edison.

Before Thomas Edison and the Industrial Revolution, there was no concept of Job or Workplace. If you were born to a blacksmith — you’d become a blacksmith. Professions were inherited from father to son, and occasionally via an apprenticeship.

Then came the Industrial Revolution, which brought means of mass production and a Human Workforce. People started to become educated and were able to choose careers.

And Thomas Alva Edison decided that he needs people to help him with his work. He was self-educated and, rumors say, was not impressed with the knowledge that people acquired through formal education. Thus, he required a way to screen potential candidates.

Edison decided to create a test to evaluate candidates¹. His test included questions related to the field Edison was working in, and general knowledge questions on topics such as history and cosmology. The test then leaked in the New York Times², and thus more companies started to practice a similar process. Fun fact: Albert, yes Einstein, allegedly couldn’t pass this test.

Modern Days

Back in the modern days, not much has changed. Few studies concluded that the technical interview is useless at predicting job performance³. While I don’t believe that we should abandon the technical interview altogether, I think I’m in a position to outline a few of the problems with the technical interview and propose alternative solutions.

Problem #1: 🧠 Brain Teasers

It’s worth noting that brain-teasers slowly die, and yet some companies continue to practice them. They include questions such as How many ping pong balls you can fit in Boeing 747 or How many cows are in Canada? — and similar.

Unless your company is trying to ship ping pong balls via airplanes, or do a count of the cow population in Canada - the primary reason to ask those questions is …to make the interviewer feel smart - Former Google SVP Laszlo Bock⁴.

Problem #2: 👩‍💻 On-site coding / Home assignment

This one is still common. Companies are afraid to hire candidates whose coding abilities are mediocre, so they ask them to perform an on-site coding task or submit a coding assessment done at home. There are two main problems with those.

On-site coding does not simulate a real environment

Let’s say you ask the candidate to implement a To-Do app. You provide him with basic scaffolding of an API Server and DAL and give him 1.5 to 2 hours to implement 3 endpoints: add a to-do, complete a to-do and get a list of all the non completed to-dos.

Now, let’s ask a question — what are you trying to test? What’s the Definition of Done for such an assignment? Successful completion in the allocated time frame? A clean code and maintainable architecture?

I want to see how he writes code under stressful situations! — you say. And I ask Why? Is your company writing a lot of code under stressful situations? Then I’d rather move on to the next company.

I want to test his code qualities! — you tell me. And I ask — By putting the candidate in a constrained environment and limited time frame? That’s hardly an environment that produces quality code. In a timed environment, such as a production incident, my number one priority is to resolve the problem ASAP and get my product back online so that customers continue to be happy, not to write nice code, whatever the hell this means. I’d refactor the solution tomorrow when I have time, and I’ve done my postmortem.

Home assignment requires a time investment

As with on-site coding, home assignments goals are vague. What are you trying to test? That the candidate can write a code that adheres to your coding standard? Your coding standards are something that can be learned, not something that the candidate should guess.

And moreover, home assignments require a time investment. I only want candidates that are willing to show their dedication towards the position. Of course you do! And candidates want a fair process in which an employer shows dedication towards them, not a home assignment that will take them 2 to 3 hours of unpaid work time, while you will throw a 7 minute glimpse at the code and conclude whether it adheres to your coding standards.

Do coding question instead

Instead of performing an on-site coding that takes 1.5 to 2 hours, or throwing a home assignment — do a coding question.

The combination I’ve come to love is to design an algorithm and then translate it to code. Let’s say your algorithm is finding whether a given word is a palindrome. You can start by providing a definition:

A palindrome is a word that reads the same backward as forward. Examples of palindromes are madam or racecar. Now try to describe an algorithm that will identify if the given word is a palindrome.

The candidate then tries to come up with a solution, while you can guide her if needed. In the end, you ask her to implement the algorithm by providing a basic scaffolding of the function signature such as: function is_palindrome(word: string): boolean and a set of tests that should all pass once she completed the implementation.

This combination achieves multiple goals.

It tests the candidate’s thinking abilities

By giving a problem and asking to find a solution, you simulate a real-life scenario in which the candidate requires to think in order to find the right solution. Palindrome is not the best example, but you can come up with any other algorithm (or use Google if you lack imagination). You can understand how the candidate thinks, how he approaches unfamiliar problems.

You can complicate it with Computer Science

Algorithms can be solved in multiple ways. Some are more performant than others. You can complicate this question by asking the candidate to try to come up with a better complexity. You can ask her how she would implement it in a real-life scenario rather than laboratory conditions. The latter gives you more understanding of tools and technologies she’s familiar with and how she evaluates them for a given problem.

You see how she translates words to code

Coding is, essentially, translating algorithms and requirements to code. This skill is way more valuable than simply writing clean code. But please use an actual computer and an IDE. Nobody ever writes code on Whiteboard during their career (unless they are University Professors or writing pseudocode).

It’s simple and adaptable

Unlike a 1.5-2 hours of on-site coding or home assignment, algorithm implementation has an obvious goal — make the tests pass by implementing the missing function. There is no ambiguity in terms of should I add log calls or how performant my code should be. You’ve already discussed the solution, and now you need to implement it. On top of all that — function implementation (which should be limited to a couple of lines of code) is adaptable. It can be done in any language, and in case you are not looking for a “native speaker” of your chosen language, giving him or her option to use a more familiar language — will make the process more smooth.

Problem #3: 👑 Manifestation of your superiority

I’ve mentioned this briefly in the Brain Teasers section. Let me do a quick overview of the path from a Trainee to an Interviewer.

The newly onboarded technical interviewer will be shadowed for a couple of interviews. She starts by being completely quiet and observing her senior counterpart leading the interview. In the next interview, she gains some more confidence and asks a question or two. By third or fourth interview - I, as a Senior Interviewer, take a leap of faith and let her lead the interview by making myself completely quiet (interrupting only if I see that there is a misunderstanding between her and the interviewee).

At every step, it’s important to have a feedback call right after the interview. I want to understand how she was feeling during the interview, are there any questions. I want to listen to her decision on the interviewee. I want to give her the tools to navigate the interview better. And I want to understand whether she is ready for her first solo interview.

The transition between being a Trainee to becoming an Interviewer

At that point, the well-shadowed trainee interviewer is on his own. It’s now his path and his decisions (I’m always here for him to support and mentor should he need that).

His first interviews will be a hit or a miss. Occasionally, he will come to you with questions such as I’ve interviewed this engineer, she looks great but lacks X and Y. What do you think? I’ve been like this at the beginning of my path. But like with anything new, we adapt and start to make decisions on our own. And there comes a time when each of us is going to have to make a choice.

Either bring balance to the Force or leave it in darkness⁵

Everything about the technical interview can be summarized by one word — Control. The interview is about having control over the outcome of the hiring process. We want to have the control that we won’t hire a candidate that has bad coding standards, is unable to perform architecture design, or is not proactive enough. This is the reason some companies have 3, 4, or even 5 interviews, each focusing on different aspects starting from coding and going to architecture, proactivity, and even personal skills such as conflict resolution, or working in a team.

And while having control is important, control can also be abused. And the easiest way to abuse control — is to feel superiority. Consider the following, somewhat hypothetical, somewhat real, scenario: An interviewee is coming to your company for a technical interview. He is greeted by 2 senior engineers with grumpy faces who are tired of this process because the last 5 people they interviewed were trying to impose a schema on a by-design schema-less database (Sorry, it’s a personal pain point for me).

Constant failure brings frustration and builds resentment towards the process. Don’t be like that. Don’t be the person who signals that the potential candidate is a failure just because your last interviews were not successful. Yes, smile. Yes, be nice. The interview is not about you and your frustration with the process. Keep it professional. Remember, behind every candidate might hide a great engineer who can bring value to your company and your product, as well as your team and even yourself.

When you start asking questions, please remember the following — You’ve asked them hundreds of times and know to navigate yourself from every possible answer and situation. The same is not true about the candidate, avoid making yourself the smartest man in the room. Remember — it’s not about you.

The fix is in framing

Recently, I’ve read an article by Mayank Verma about how you should treat the Technical Interview as more of a discussion rather than an interview. The change is not radical — it’s in framing. Instead of treating the interview as a process in which you judge the interviewee by his answers, we can turn the interview into a discussion where both the Interviewee and the Interviewer are working towards a joint solution with the purpose of understanding whether the Interviewee is suitable for the position.

This can take form in different representations. For example: it can be a discussion about the interviewee’s experience. An exercise I like to do is to ask the candidate to pick a feature or an architecture that she considers to be interesting, challenging, or comfortable to discuss. This achieves 2 things:

It gives the candidate freedom and control over the interview. Instead of throwing her into the water and asking her to implement some sort of system I’ve designed, I let her talk about something that she did — thus transferring the control from myself to her. This makes the relationship between us equal, instead of Interviewer / Interviewee.
This gives me a glimpse at what is the most challenging or interesting thing the candidate worked on.

I then proceed to a discussion. We discuss who was responsible for this feature, who decided on the architecture, what was her role in that. What would she change now, if she had the time and resources? This gives me way more insight about her as a candidate, instead of a generic design question such as Let’s now design {Some System}. The former is familiar to the candidate, and she probably knows the in-s and out-s of the architecture; while the latter is… you can’t expect a candidate to do a system design in 1.5 hours and have a great design. Moreover, the latter might be a hit for someone who worked with systems in that particular domain, and a big miss for someone who did not.

🗒 The Manifest of a Great Technical Interviewer

As I’ve mentioned earlier, I’m getting close to the mark of 150 interviewed people. I have an unexplainable passion for interviewing. I have a great track record for post-interview feedback. Furthermore, I learn a lot from the interview as an interviewer, and I think every engineer should try interviewing. Below is a summary of traits that I consider mandatory to become a Great Technical Interviewer.

Be curious. By being curious, you arrive at the interview with the purpose to discover. Curiosity allows you to evaluate the candidate without prior biases.
Evaluate instead of judging. By judging, we search for imperfections; by evaluation — we try to find a match. Approach the interview from an evaluation standpoint rather than from a judging one.
Let go of the ego. The interview is not about you or your knowledge. You are not here to show the candidate how awesome you are. Let go of the ego and focus on evaluating the candidate’s skills for the position.
Listen first, only then speak. In an ideal interview, the candidate should talk most of the time. Your role is to listen and guide her towards the skills you are looking for.
Understand his motivation. Many candidates submit their CVs to multiple jobs in different domains. Remember that your role is not only to find the best engineer for your team, but to also make sure that your team is the best fit for the candidate. Listen to what he is looking for and be honest with whether you can provide that.
Be an expert in your field (or admit your lack of knowledge). If you are evaluating a candidate for great RDBMS knowledge, then you should be close to an expert in that field. Finding yourself arguing with a candidate on a specific matter, only to later find out that she was right, and you were wrong, due to lack of knowledge, — is an unpleasant situation to be in. If in doubt, you can always say I don’t know, but I’ll definitely check that after the interview (and obviously go and check).
Seek to learn. Every person I’ve interviewed — taught me something new. It might be filling a technical gap I’ve had or acquiring new communication skills. Always seek to learn, even from candidates that do not pass your interview.
Have a structure, but know to be flexible. You must lead the interview. It should have structure. It should evaluate the candidate based on the position you are interviewing for. But you should also remember that everyone is unique, and occasionally, you will encounter candidates with a unique experience that will break your interview structure. You must know how to navigate yourself back towards the qualities you are looking for, without making the process feel robotic or scripted.
Accept the fact that you are going to fail. No matter how great you are as an interviewer, eventually you will miss a great engineer or hire an unsuitable one. Your goal should be to minimize those mistakes, but you should accept the fact that they will happen.
Mentor others. When you feel comfortable with the interview, mentor other developers so they, too, will become Great Technical Interviewers.

This path is hard, but it is rewarding

Interviewing is a hard process. There are times when the hiring is on fire, and you will have 2-3 interviews a week, each taking 1.5 to 2 hours. Don’t overload yourself. Pushback if you feel you’ve had too many interviews and need a rest. Always continue to improve yourself and the process. Questions the existing status quo — always remember that the status quo became the status quo because someone questioned it. And enjoy the process — it’s way more fulfilling once you do. Good luck, young Padawan.

Bonus — Tips for candidates from an Interviewer’s perspective

Once I became an interviewer, I’ve realized some things that I wish I knew when I was an interviewee. Below is a summary of them.

Embrace rejections. Rejection is not necessarily a red mark that says you are not a good engineer. Rejection simply means the qualities you possess do not suit our position. I’ve interviewed talented engineers that I’ve had to reject because they were not suitable for the position.
Always ask for feedback. I was surprised how few engineers ask for post-rejection feedback. If you are interviewing for a company and get rejected — always ask for honest feedback. Don’t just settle on We’ve decided to proceed with other candidates. Ask what you can improve so that the next time you apply — you will succeed. (Not all companies are willing to provide such feedback. If you request one and get a generic response or no response, consider yourself lucky — you’ve avoided a workplace that isn’t concerned with your development).
Avoid biased information. Your CV is not your Facebook page. You think that the fact that you play the guitar is cute — but we are all humans, and we can be biased. You want to avoid blowing your interview just because you fell on an interviewer who prefers drums to guitar. Interviewers and HR does a lot of work to eliminate bias from our decision-making, but we are all humans. Avoid including any personal information in your CV.
Interview the company. Remember, the interview is not only about them interviewing you — but it’s also about you interviewing the company. Make a list of things that are important to you in a company, and make sure to ask questions and understand whether this particular company can provide what you are looking for.
Don’t be afraid to say I don’t know. I’d have more respect towards a candidate that says “I don’t know, I never worked with X. I’d take some time and read about it”, instead of a candidate that will try to confuse me with buzzwords.
Have fun. Treat interviews as games. See how well you can pass them and what you can learn from them.

Supercharge Your NodeJS With Rust

Sat, 23 Oct 2021 07:00:00 GMT

Node isn’t the fastest framework out here. It’s not the slowest either, v8 is doing wonders to its speed, but nevertheless, if we setup an unfair battle between Node and say Rust; Node will lose.

If you are interested to compare this approach to WebAssembly, I’ve written a new article on WASM in comparison to native modules.

What is Rust?

Rust is a multi-paradigm, high-level, general-purpose programming language designed for performance and safety, especially safe concurrency. Rust is syntactically similar to C++, but can guarantee memory safety by using a borrow checker to validate references.

— Wikipedia

Rust is a strongly typed and compiled language. It supports a lot of, so called, modern features such as closure and anonymous functions, a rich standard library, types and polymorphism as well as modern tooling such as Cargo which is an equivalent to npm in the world of Node.

Rust is also Memory Safe language. All safe Rust code is memory checked during compilation so unless you opt to use unsafe Rust, you won’t encounter the famous Segmentation Fault you’d get in C or C++ by accessing a pointer that is pointing to a memory you no longer own.

All this makes Rust a very attractive replacement to C/C++.

Ok great and all, but I write Javascript!

Yeah, me too. And here comes the great part. We can use all the beauty of Rust in Javascript!

Oftentimes we encounter a situation when we need to run a heavy task such as PDF generation or some sort of computation. Most of the modules in NPM are Javascript based, meaning they will suffer a penalty at runtime. Usually in such situation we opt to “creative” solutions such as offloading the heavy process to a Lambda function and turning the entire process into asynchronous for the user. And yet, I rarely hear people suggest or even evaluate the solution of embedding a native module that most of the times will be faster.

Another great usage for Rust in the Javascript ecosystem - is providing shared backend for Desktop applications. Let’s say you are developing an MVP in Electron and want to use SQLite for data storage. While you can use the sqlite3 package which is native C++, you can also create a dynamic library (also knows as .so or .dll file in Linux and Windows respectively) in Rust. What this gives you, is the ability to later ditch Electron in favor of Native Framework for each OS while doing 0 changes to your Backend code.

So how do I use Rust from Javascript?

I’m glad you’ve asked! Please meet Neon.

Neon Bindings#774x300

Neon is a library and toolchain for embedding Rust in your Node.js applications and libraries. It is similar to creating native modules with C/C++ but with none of the fear and headaches associated with unsafe systems programming.

Let’s try now to create a native Fibonacci function in Rust and call it from Node. Disclaimer: This is not a tutorial on Rust, so I’m not going to explain Rust specifics here. You can always go to the Rust Book to learn about Rust, however the syntax I’m gonna use here should be easy to understand to anyone familiar with Typescript, or any other statically typed language like C/C++, Java, Kotlin or C#. So let’s dive in.

Step 1 - Setup a project

Let’s start by creating a Rust project. We will start by creating Cargo.toml file. It’s a file that mimics the role of package.json:

[package]
name = "fibonacci_rs"
version = "0.1.0"
edition = "2018"

[lib]
crate-type = ["cdylib"]

[dependencies]
neon = { version = "0.9", default-features = false, features = ["napi-6"] }

First, we define our package. We give it a name, we define the version and what edition of Rust we are going to use.

Then we define the type of our package. Remember, Rust is a compiled language and it can produce all kinds of binary outputs. They could be executables (such as .exe files in Windows) or they could be library files such as .so or .dll. In our case we create a library, hence the [lib] declaration. There are 2 types of libraries that most languages (including Rust) can produce: dynamic and static. In short:

Static libraries usually have the extension of .a or .lib in Linux/MacOS and Windows respectively, Static libraries are compiled into the final executable. So if I produce a static library and then link my executable against it - the final output will be one file, where the static library is embedded into the executable itself. Their main purpose is to create reusable code but once the binary is compiled, you can’t change the implementation of a static library without recompiling the binary.
Dynamic libraries have the extensions of .so, .dylib and .dll in Linux, Mac and Windows respectively. The difference is that dynamic libraries are not compiled into the final binary file and instead are loaded at runtime by the OS. So when I link my binary against a dynamic library, I’ll have 2 files in the output - the dynamic library and my binary. Their main use is to create a reusable code between multiple applications without locking the implementation. Also, since dynamic libraries are not compiled into the executable, I can reuse their code by having any number of executables that depend on the same dynamic libraries. Libraries such as Microsoft C/C++ runtime are dynamic libraries that are downloaded once by the user to their machine, and then used by countless applications that depend on MSVC.

Since we are not going to recompile Node.js itself, we are interested in producing a dynamic library that can be linked at runtime, rather than embedded directly into Node.js runtime. Therefore we define our crate-type as cdylib meaning that we want to create a dynamic library with C ABI (hence the “c” in the beginning). I’m not going to go into details what ABI is and what’s its purpose, I’ll instead drop a link to Wikipedia. Note that we can also create dylib which will have unstable Rust ABI that can be changed across Rust releases.

Last but not least - we define our dependencies. In our case, we depend only on neon library.

Step 2 - Write our Rust Fibonacci

Since we are creating a rust library, we need to create the following file in the root of our project: src/lib.rs. In this file we are going to create our Fibonacci function:

fn fibonacci(n: i32) -> i32 {
	return match n {
		n if n < 1 => 0,
		n if n <= 2 => 1,
		_ => fibonacci(n - 1) + fibonacci(n - 2)
	}
}

A simple, recursive function that accepts a 32bit integer and returns a 32bit integer.

Step 3 - Create a binding for our Rust Fibonacci into Javascript

Currently, we’ve created a Rust function. We can run cargo build in order to compile our code. This will download and compile neon and any of its dependencies, and will compile our lib.rs file. We will get a compilation warning saying that fn fibonacci is dead code, since no body uses it, but nevertheless, if you’ve done everything correctly, you should end up with a new directory tree of target/debug and inside it you will find a file named libfibonacci_rs.{dylib,so,dll} (the extension will depend on what type of OS you are compiling on).

Since its a dynamic library that does not extern any functionality, it is now useless. But let’s put some Neon lights onto it. Edit src/lib.rs and add the following function:

use neon::context::{Context, FunctionContext};
use neon::types::JsNumber;
use neon::result::JsResult;

fn fibonacci(n: i32) -> i32 {
	return match n {
		n if n < 1 => 0,
		n if n <= 2 => 1,
		_ => fibonacci(n - 1) + fibonacci(n - 2)
	}
}

fn fibonacci_api(mut cx: FunctionContext) -> JsResult<JsNumber> {
	let handle = cx.argument::<JsNumber>(0).unwrap();
	let res = fibonacci(handle.value(&mut cx) as i32);
	Ok(cx.number(res))
}

We’ve added some use neon statements. They are similar to import or require statements in Javascript. We’ve also added a new function called fibonacci_api. This function is the conversion layer between Javascript and Rust. It accepts a FunctionContext since it’s a function and returns a JsResult of JsNumber meaning that on successful execution, it will return a value that is equivalent to number type in Javascript.

cx is used to interface with the Javascript world. We first get a Handle to JsNumber which is the first (index 0) argument that was passed to our function. Handle is a safe reference to a Javascript value that is owned and managed by the Javascript engine’s memory management. Since Javascript is a dynamic language and Rust is statically typed, we need to manually downcast the javascript value to a number. Since fibonacci_api can be called with anything (a string, an object or array) from Javascript, this downcast can fail, therefore we need to handle a situation where it fails. In my example, I just .unwrap the result of the downcast, since this is a tutorial and not a real world example, but in real world you’d need to handle a situation where the first argument is not a JsNumber.

Just a few more words on values and down/up casting and we will continue. Everything you get in Rust via Neon, initially has the type of JsValue. JsValue is sort of a catch all type that can be anything: JsNumber, JsObject, JsNull, JsBuffer and etc. A Handle with JsValue can be downcasted to a specific type by using Handle::downcast() or upcasted by using Handle::upcast(). There is a nice diagram on the neon::types Documentation that shows the relation between JsValue and other Javascript types. I’ll include it here:

javascript type hierarchy#960x480

Anyway, back to the code.

So now we have a Handle<JsNumber>. We can access the internal value by calling Handle::value() method, which in case of a JsNumber handle, returns f64 (since number in Javascript is represented as float), but we safely convert it to i32 for our Fibonacci function.

We then safely return the result as JsNumber by calling the .number() method of the Function Context. And we are done.

Once again, we can compile the code by running cargo build. It will once again warn us about dead code, but this time it will mark both fibonacci and fibonacci_api function as dead code, since no body calls them.

Step 4 - Export our Rust function to Javascript world

We are almost done with Rust. Bear with me for a little bit more.

Right now we have a function that converts a JS call into Rust call, but we have no way to call that function from Javascript. Enter the main!

If you are familiar with C or C++, every executable has a main function. This is a special type of function that the OS will run once it loads your binary into memory. Libraries however do not have main functions, since libraries are not meant to be executed by themselves. However you’ve probably seen a Javascript module. It usually ends with module.exports = {}. We can treat it as main. And that’s what Neon does.

Once again, edit our src/lib.rs file:

use neon::context::{Context, ModuleContext, FunctionContext};
use neon::types::JsNumber;
use neon::result::JsResult;
use neon::result::NeonResult;

fn fibonacci(n: i32) -> i32 {
	return match n {
		n if n < 1 => 0,
		n if n <= 2 => 1,
		_ => fibonacci(n - 1) + fibonacci(n - 2)
	}
}

fn fibonacci_api(mut cx: FunctionContext) -> JsResult<JsNumber> {
	let handle = cx.argument::<JsNumber>(0).unwrap();
	let res = fibonacci(handle.value(&mut cx) as i32);
	Ok(cx.number(res))
}

#[neon::main]
fn main(mut cx: ModuleContext) -> NeonResult<()> {
	cx.export_function("fibonacci_rs", fibonacci_api)?;
	Ok(())
}

We’ve added a small function named main and marked it with neon::main attribute to tell Neon that this is the main entry of the module. This function also accepts a context, but not FunctionContext like our previous function, instead it accepts ModuleContext. While function context is used to interface with js functions and have operation like getting arguments of a function, up/down casting JsValues into specific types and generating results; Module context is used to interface with Javascript modules, and the main methods in ModuleContext are ModuleContext::export_function() and ModuleContext::export_value(), which are used to export functions and values from a module.

In our case we export a function and name it fibonacci_rs, while the actual implementation of it is fibonacci_api we’ve written earlier.

In Javascript world, this code would look something like this:

function fibonacci(n) {
  ...
}
  
module.exports = {
  fibonacci_rs: fibonacci
}

Running cargo build once again will produce our library without any warnings. We are now done with Rust, let’s move to Javascript.

Step 5 - Fibonacci Javascript

Let us now create a new Javascript file: fibonacci.js:

const value = process.argv[2] || null;
const number = parseInt(value);

if(isNaN(number)) {
	console.log("Provided value is not a number");
	return;
}

console.log(number);

This is a simple file that reads the first argument we pass to it from CLI, validates that it is a number and prints it back. You can execute it by node fibonacci.js 10 .

Now we somehow need to call the fibonacci function we’ve created in Rust. For that we will have to create a package.json.

{
  "name": "fibonacci_rs",
  "version": "1.0.0",
  "private": true,
  "main": "fibonacci.js",
  "scripts": {
    "build": "cargo-cp-artifact -nc index.node -- cargo build --message-format=json-render-diagnostics"
  },
  "dependencies": {
    "cargo-cp-artifact": "^0.1.5"
  },
  "devDependencies": {}
}

Run npm install to install the dependency and then run npm run build to generate the native module. If everything worked correctly, you should end up with index.node file in your root directory.

Let’s explain. cargo-cp-artifact is a package maintained by Neon team. What this package does, is handling the extraction of the library produced by cargo build. Remember, cargo build generates different files depending on the host OS. In my case it generated libfibonacci_rs.dylib since I’m on Mac, but it could also generate .so or .dll files if you are on Linux or Windows. Theoretically, you can go manually to the target directory and copy the library file to the root of the project and name it index.node, but cargo-cp-artifact is a handy utility to do that for you.

Also note: index.node is not a special name. You can call it what.ever, but the .nodeextension is used to represent a native Node.js module.

Now lets go back to our fibonacci.js:

const {fibonacci_rs} = require("./index.node");

const value = process.argv[2] || null;
const number = parseInt(value);

if(isNaN(number)) {
	console.log("Provided value is not a number");
	return;
}

const result = fibonacci_rs(number);
console.log(result);

You can run it with node fibonacci.js 10 and you should get the result of 55. Note: do not input high numbers. Since this is a very inefficient and recursive implementation of Fibonacci, it will hang on numbers above 40ish.

Let’s explain. We require our index.node as if it was a regular Javascript module. We know that it exports only 1 function: fibonnaci_rs since that’s what we did in cx.export_function("fibonacci_rs", fibonacci_api)?; in src/lib.rs , hence we require it and then execute it.

That’s it! We’ve created a native module in Rust.

You can clone the entire code of this tutorial from my GitHub. You can also explore the examples repository by Neon for more robust examples like gzip-stream or async sqlite. Feel free to explore the neon repo as well, they also have a link to their Slack channel for questions and support.

What you can do with it?

Many things. You can access native implementations that are not available in Node. You can use any Rust crate (equivalent to npm package) from crates.io (provided you implement the required neon bindings), or you can have another strong tool to evaluate when your Javascript implementation is not fast enough for your needs. I’m myself currently experimenting in writing a Desktop application using Electron for the Frontend and Rust for the Backend. This allows me to isolate my backend code into a reusable library that I can plug into any more performant UI Framework such as Qt, GTK and SwiftUI, as well as letting me access SQLite using native sqlite implementation written in C rather than Javascript.

Things to take into account

If you are not coming from the world of compiled languages, or you knowledge of it became rusty (no pun intended), you might need to pay attention to some caveats with using native modules written in any language and not just Rust.

With Javascript, Python or Ruby, I can give you a code and you can run it (provided you have the runtime installed). Dynamic languages are easily shared between people and machines. Compiled languages are not. If I give you my binary index.node from this tutorial, you might or might not be able to use it. It depends on the host OS you are running. The safe bet is to compile the code yourself on the OS that you intend to run the code. This means that native modules should not be committed into your source control, but instead be built during your CI/CD. And they must be built on the same OS that they are going to be executed on. So if your Node application is running in Alpine Linux Docker container, it is very much advised that you compile your native modules on the same docker image.

That’s it for today. I hope this newly acquired knowledge will help you to become a better Node developer.

Wrap your gifts not your dependencies

Mon, 08 Feb 2021 10:00:00 GMT

We’ve all been there. Its time to introduce a new package / dependency to our code base, be it a HTTP request library, a logger or something else, and the question we ask ourselves “Should I wrap it?”

The main reason we ask this question is a hidden fear that maybe in the future we would like to change the underlying implementation to a different package. We, humans, like to keep our options open.

But in reality, by creating wrappers around your dependencies, you are actually narrowing down your options and limiting your self.

Lets say you use axios package and instead of using it directly, you decide to wrap it. Your wrapper will depend on axios and will expose some generic API that uses the internal functionality of axios.

You need to find the exact balance between what you want to expose and what you want to abstract. You don’t want to create a dumb proxy in front of axios because you want to keep your options open in case you decide to switch underlying functionality, so you try to create something abstract. And by doing so you encounter the first problem of wrappers:

Wrappers take a feature rich library and dumb it down a set of simple functions

You don’t have time to create a proper wrapper. You don’t know what functionality you might need from it, so you focus on wrapping what you need now. In the end, you are making a product for your clients, not developing a library. You don’t have time to correctly wrap the underlying dependency. So you expose a few methods, those you need right now, in some abstract way, type git commit and go to drink your coffee with a smile on your face.

Later on, when one of your team mates needs to develop a new feature, he will happily look at your axios wrapper, expecting it to contain a method he needs and will be disappointed to find out that this method is not there. Now instead of focusing on delivering value to your customers, your team mate will have to focus on creating a useless abstraction for a greater cause of “we might replace it in the future”, and this brings me to the second and biggest problem of wrappers:

Wrappers create the illusion of control. The main argument for creating a wrapper is that “we might change to a different underlying dependency in the future”, but guess what? No body, ever, changed from one logger to another; changed from postgres to mysql. And if they did, their wrapper probably caused them more troubles than help.

In programming there is a nice principle called YAGNI — You Aren’t Gonna Need It. YAGNI suggest that instead of implementing a bunch of abstractions with the hope that “one day I might change the underlying implementation” you actually focus on implementing what you need now. Don’t be afraid of refactoring — its a healthy process in software development. Focus on implementing real safety nets such as continuous automated unit and integration tests and look at type checking, either via statically typed languages or by using tools like TypeScript to help you with refactoring. Remember — wrappers create the illusion of safety, as they rarely help you with transition from one underlying library to another.

Donald Knuth once said

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

Wrapping your dependencies is a premature optimization for a dooms day when you decide to switch to a different logger. This day will either never come or if it does, your wrapper will not save you from tons of refactoring and fixing broken tests anyway. And in the mean time, you’ve wasted time on creating your useless wrapper but instead you’ve needed to focus on proper refactoring and tools to help you with them (such as type checking), proper automated unit and integration testing while in same type enjoying the full potential of a library that’s been co-created by 260 different contributors (https://github.com/axios/axios/graphs/contributors), but hey! It is sure better to roll your own, dull wrapper, ain’t it?

So stop creating wrappers. Instead focus on refactoring (by prioritizing it as tech debt in your development life cycle), focus on tests. Focus on creating middlewares to common functionality that can be quickly plugged to existing libraries, instead of wrapping those libraries. Focus on bringing value to your customers instead of reinventing the wheel over and over again.

Context Switch - The only operation that kills productivity

Fri, 26 Apr 2019 09:00:00 GMT

In computing, a context switch is the process of storing the state of a process or of a thread, so that it can be restored and execution resumed from the same point later. This allows multiple processes to share a single CPU, and is an essential feature of a multitasking operating system.

— Wikipedia

See, a single core CPU can not multitask. It is all a lie. What really happens when single core CPU multitasks, is that it runs small parts of each program for a small period of time. The higher the IPS of the CPU, the more “multitasking” the CPU looks.

But in order to “multitask”, the OS needs to perform one very expensive operation — Context Switch. What happens in that operation is that it needs to store all the data of the current running program: its CPU registers and flags. Then unload the current program from the CPU, load the next program to the CPU, load its CPU registers and flags, and let it run for some small amount of milliseconds (somewhere around 100, depends on the OS scheduler implementation), until the OS decided to perform another context switch.

This is probably the most interesting and critical part of designing an operation system. If your context switching algorithm sucks, your OS becomes irresponsive, slow and eventually hangs.

We are like single core CPUs

Humans can not multitask. The brain is created to perform one task at a time.

It looks like you multitask sometimes, but what actually happens is that you, like an OS, perform context switch. And as in OS, your context switch is a very expensive operation.

Try to answer a phone call during intensive, brain required, work. Chances are you will forget most of the content of the call, because it takes your brain time to leave what it was doing (thinking and analyzing) and switch to another task (listening and remembering), not to mention that once the call is over, it will take you on average 25 minutes (No Task Left Behind? Examining the Nature of Fragmented Work) to go back to the task you were doing.

Some people are better at it (especially women), some are less, but for everyone — context switch is the productivity killer. It doesn’t matter if you wake up at 4am every day, eat healthy, do sports and read books — if you do a lot of context switching during the day, you won’t be as productive as you could be.

And it seems like no one in the modern world understands it.

Instant messaging, annoying notification pop ups, frequent meetings, open space work environment — all those and others encourage context switching. And thus — reduces productivity.

What can we do?

As someone who needs his full brain power, you need to tweak your working environment towards less context switching. Disable most of the notification — honestly they are not that important most of the times. Put your phone on silent; disable notification pulse lights and sounds; use headphones with relaxing, instrumental music; avoid meetings; practice concentration through meditation.

And avoid instant messaging — not only it distracts you from your tasks, but it is not an effective way for communication. It is great for things like “see you tomorrow at 6pm on main street 1 cafe”, but its awful tool for real conversations. You never have the full brain capacity to read, analyse, sympathize and respond correctly. In order to actually do all this, you need to dedicate time to real communication as opposed to communication “by-the-way”. This is also true for offline communication — never go watch a movie if what you want is to communicate, instead prefer a walk or a dinner with drinks; those are better for communication rather than the constant context switch between the movie and your partner / friends / family.

And if you are a manager who looks to increase the productivity of your team — promote quite times; drop those useless meetings or schedule a meetings day — a day that is dedicated entirely to meetings; prioritize — not every small email is important right now; promote documentation of your team’s members work, so team members wont spend time asking one another about parts of team’s work they were not involved in; promote non peer to peer communication — avoid IMs, prefer issue tracker that can be handled when the person has time to.

The key is FLOW

In positive psychology, flow, also known as the zone, is the mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by complete absorption in what one does, and a resulting loss in one’s sense of space and time. — Wikipedia

Flow is the new trend today. Be in the flow. Whether its work flow, conversation flow, listening flow, watching flow, reading flow. Your brain is not capable of doing multiple things at once, it performs best when its in a flow state. Embrace the flow. There is a reason why people experienced flow state describe it as the most productive work they’ve done. It is because in the flow you have only the sense of the task, you do not spend any energy thinking about or concentrating on different tasks or areas.

Multitasking is not the answer

We keep hearing a lot about how multitasking is important; the world has gone crazy and we need to do a lot of stuff in order to succeed. “Being able to switch between different tasks fast”, “Can work in a fast moving environment” — those are one of the common job descriptions you can find today, and the ones you need to avoid.

The only way to succeed is to devote your full attention to the particular activity, whether its work, communication, reading, doing sports, meditating. Really successful people know how to plan — plan work, plan leisure, plan everything. And they know how to disconnect — leave your work behind when you spend time with family; leave your trip planning behind when you work and etc.

Avoid multitasking; avoid being flexible; avoid working in a fast moving environment. You will not succeed there. You might feel like you are, but what actually will happen is that you will break down. Break down because multitasking does not work. The only thing that multitasking does to you (apart for the health problems it is associated with such as: brain harm, memory problems, increased distractibility, chronic stress, depression and social anxiety)— it makes you do mediocre work in more time; makes you spend your potential on stress; makes your brain keeps wasting its power as well as your energy, to a very expensive operation — Context Switching.

React.JS Localization v0.2

Mon, 24 Jul 2017 10:00:00 GMT

Remember my last post about React.js localization? Not 2 years passed yet and I’m happy to share with you the “proper” way to do localization in React.js.

Globalize.js

Yes, I still use globalize. As I said in my previous post, the Common Locale Data Repository is maintained by Unicode Consortium and have support for every possible locale. Please don’t re-invent the wheel. Languages are hard (believe me, I know Russian) and I’ve seen a lot of people trying to do their own localization and plurals. Kudos to them, its a valuable experience to get, but when you are doing a professional web site, you want to use proper plurals.

But while CLDR is only data (usually JSON, but if you are old and nostalgic you can get it in XML format), globalize is the tool. You feed globalize with data and it gives you way to produce shiny localized numbers or dates. Globalize API is very extensive and modular, the core library is only 1KB minified + gzipped and you can add other modules depending on your needs.

Getting CLDR

The way you get CLDR did not change from my previous post. I still think it should be part of your build step. The only difference is that I no longer use cldr-data-downloader since it is limited in the way it downloads the CLDR. Instead I use the combination of gulp-download and gulp-decompress as well as a custom mapping JSON file to map the different packages and versions of CLDR.

What I do recommend you pay attention to, is that CLDR package is split into two “modules”: Locale files and Supplemental files. It is important that you do not ship everything as one json file. Supplemental data is shared across all locales, so if a user switches between different locales, he will use the same supplemental file, but different locale file. This will allow the browser to cache the supplemental.json file if it was already requested once. So while you package your cldr, split it into at least 2 files: {locale}.json and supplemental.json, where {locale} is being the locale you are shipping.

Here are the 2 gulp tasks that I use to create supplemental.json and {locale}.json for the locales I support:

gulp.task('cldr:bundle:supplemental', ['cldr:bundle:clean'], function () {
    return gulp
        .src(CLDRConfig.files.supplemental.map(function (f) {
            return path.join(CLDR_SOURCE_PATH, 'supplemental', f);
        }))
        .pipe(extend('supplemental.json'))
        .pipe(gulp.dest(CLDR_DEST_PATH));
});

gulp.task('cldr:bundle:locale', ['cldr:bundle:clean'], function () {
    const streams = Config.i18n.locales.map(function (locale) {
        return gulp
            .src(CLDRConfig.files.locale.map(function (f) {
                return path.join(CLDR_SOURCE_PATH, 'main', locale, f);
            }))
            .pipe(extend(locale + '.json'))
            .pipe(gulp.dest(CLDR_DEST_PATH));
    });

    return merge(streams);
});

Localization

Now that you have your CLDR ready, you can start with the actual localization.

The main and the most important file is the GlobalizeProvider:

import React from "react";
import PropTypes from "prop-types";
import Globalize from "globalize";

export default function provide(locale, supplemental, data) {

    Globalize.load(supplemental);
    Globalize.load(data);
    const globalize = new Globalize(locale);

    const GlobalizeProvider = class extends React.Component {

        static childContextTypes = {
            globalize: PropTypes.object
        };

        getChildContext() {
            return {
                globalize: globalize
            };
        }

        render() {
            return this.props.children;
        }

    };

    return GlobalizeProvider;

}

locale — is the actual locale name (e.g. “en”, “ru” etc), supplemental — is the content of supplemental.json and data is the content of {locale}.json.

You can theoretically load ALL locale data you will ever support, but its a huge huge huge waste. Locale is not something that users change every 5 minutes, so if I prefer an en_US version of you website, most likely Ill never switch to ru or en_GB, and even if I will switch, I’m ok with you reloading the website and loading the locale data again.

So we have the GlobalizeProvider. It uses context to pass the globalize instance (you can read about context in one of my posts). We will need one more Container component that handles the locale loading based on any strategy you want (guess it from the browser, headers, make the user select; whatever, I prefer to start with en_US and let the user select). Here is a snippet of my RootContainer component that handles the locale loading (I’m using redux here):

import React from "react";
import {connect} from "react-redux";
import {BrowserRouter as Router, Route, Switch} from "react-router-dom";

import {Actions as LocaleActions} from "../actions/locale";

import GlobalizeProvider from "../globalize/Provider";

import ErrorComponent from "../components/Display/Error";
import LoadingComponent from "../components/Display/Loading";

import "../../styles/style.sass"

class Root extends React.Component {

    componentWillMount() {
        const {dispatch, locale} = this.props;

        if (locale.supplemental === null) {
            dispatch(LocaleActions.loadSupplemental());
        }

        if (locale.data === null) {
            dispatch(LocaleActions.loadLocale(locale.locale));
        }
    }

    render() {
        const {locale} = this.props;

        if (locale.error) {
            return <ErrorComponent>Failed to load locale data</ErrorComponent>;
        }

        if ((locale.supplemental === null) || (locale.data === null)) {
            return <LoadingComponent/>;
        }

        let GlobalizeProviderContainer = GlobalizeProvider(locale.locale, locale.supplemental, locale.data);

        return (
            <GlobalizeProviderContainer>
                <Router>
                    <div>
                        <SignNDAtoViewThisComponent/>
                    </div>
                </Router>
            </GlobalizeProviderContainer>
        );
    }

}

const select = state => {
    return {
        locale: state.locale
    }
};

export default connect(select)(Root);

And now we have GlobalizeProviderContainer that wraps our whole application, and since it uses context we can pass initialized globalize instance to every leaf component.

Localized Leaf Components

The only thing left is to implement Localized leaf components.

Here is an example for DateTime component to display date & time in given format:

import React from "react";
import PropTypes from "prop-types";

export default class DateTime extends React.Component {

    static propTypes = {
        date: PropTypes.instanceOf(Date).isRequired,
        format: PropTypes.string.isRequired
    };

    static contextTypes = {globalize: PropTypes.object.isRequired};

    static _formatters = {};

    static getFormatter(format, globalize) {
        if (!(format in DateTime._formatters)) {
            DateTime._formatters[format] = globalize.dateFormatter({skeleton: format});
        }
        return DateTime._formatters[format];
    }


    render() {
        const formatter = DateTime.getFormatter(this.props.format, this.context.globalize),
            formattedDate = formatter(this.props.date);

        return <span className="component localized datetime">{formattedDate}</span>
    }

}

Notice the static contextTypes declaration. This is what gives us access to the globalize instance provided from GlobalizeProviderContainer (for more info read my context post).

Inside the Localized component, you use pure globalize API to access the different formatting methods.

You can even query the CLDR itself for data. Take a look at Currency component below:

import React from "react";
import PropTypes from "prop-types";

import Currencies from "../../../../data/currencies.json";

export default class Currency extends React.Component {

    static contextTypes = {globalize: PropTypes.object.isRequired};
    static propTypes = {
        display: PropTypes.oneOf(['name', 'symbol']),
        currency: PropTypes.oneOf(Currencies).isRequired
    };
    static defaultProps = {display: 'name'};

    render() {
        switch (this.props.display) {
            case 'name':
                return <span
                    className="component currency name">{this.context.globalize.cldr.main(`numbers/currencies/${this.props.currency}/displayName`)}</span>;
                break;

            case 'symbol':
                return <span className="component currency symbol">{this.props.currency}</span>;
                break;

            default:
                return null;
        }
    }

}

This component, when rendered like this:

<Currency currency="USD" display="symbol"/>

Will produce:

<span class="component currency symbol">USD</span>

But when rendered like this

<Currency currency="USD" display="name"/>

Will produce

<span class="component currency name">US Dollar</span>

For en_US locale, and for the ru locale it will produce

<span class="component currency name">Доллар США</span>

You can now go on and implement as much Localized Leaf components as you wish, depending on your needs. I have Country and Currency Selects that are entirely localized, Date Picker that is based entirely on CLDR and Globalize (as opposed to a lot of date pickers that use internal localization files) and etc.

Hope this helped you to become a better React.js developer. Best of luck :)

Validating Requests With a Simple Middleware for Express

Sat, 27 Aug 2016 10:00:00 GMT

A key to a good API design is good control of what data your controllers or API end points accepts. Another important key to a good design — is that you can take a brief look at your API end point and immediately understand how your request should look.

Now, of course you can validate all the request parameters inside the API end point itself. But this makes your end point messy and sort of violates the SRP (your end point should not validate requests. It should only work with them and provide a response). Now, if you are old fashioned and work with XML requests / response, you can of course use XSD to validate your incoming requests. But we are in 2016 where JavaScript runs in your browser, your desktop environment, your media center, you smart watch, your smart home and soon in your brain implants. So we gonna use JSON!

But, how do I validate JSON? I’m glad you’ve asked. Meet joi by hapijs. This small and awesome library can validate your JavaScript objects. It can do all kind of stuff like making sure that a specific key has only specific set of values, or that specific key is only required if another key exists and contains a specific value. So as always, RTFM.

We now have joi to validate our JSONs. But we still need to find an elegant way to put it inside the life cycle of our specific API end point. I’m too lazy to create a specific, generic NPM module so I’ve just dumped the current code I use in my project to a gist. Feel free to use it or create a proper NPM module (don’t forget to mention me for being your inspiration ;)

The middleware is very simple. It defines a Bad Request Error that will be thrown if the incoming request is malformed, and what it does, it simply validate the request against a given schema (a bit later on that) and assigns the validated object (essentially this is req.body but with type conversions and modifiers applied, more on this in joi API documentation) to the req.schema (not the best name) so it can be accessed later on inside the API end point (or any next middleware in the chain).

Now lets talk about the schema and how to actually use this. Schema is just a JSON with joi rules. Lets say we have 2 API end points: One for user creation and one for user update. We define then 2 schema:

module.exports = {
  create: Joi.object().keys({
    username: Joi.string().min(3).required(),
    password: Joi.string().min(3).required(),
    email: Joi.string().email().required(),
  }),
  update: Joi.object().keys({
    username: Joi.string().min(3).optional(),
  }),
};

In order to create a new user we need to provide 3 required keys: username (at least 3 characters), password (at least 3 characters) and email (should be a real email). And in order to update a user we can just provide an optional new username (at least 3 characters) (side note: an empty request in such case is a valid request, it is up to your end point to make sure to do no-op if the request is empty). We then define our API end points like this:

var SchemaValidationMiddleware = require("path/to/schema/middleware"),
  UserSchema = require("path/to/schema/user"),
  UserController = require("path/to/user/controller");

router.post(
  "/users",
  SchemaValidationMiddleware.validate(UserSchema.create),
  UserController.createNew,
);

router.patch(
  "/users/:id",
  SchemaValidationMiddleware.validate(UserSchema.update),
  UserController.edit,
);

And inside the controller / API end point we can use req.schema to access the validated schema and perform the required operations. And we can be sure that the controller / end point will be executed only if our request meets the schema we defined for it. The UserController.createNew will never be called if one of the keys we defined as required in our schema — missing, if they are mistyped, if there are keys that we are not expecting and etc.,

This gives the answer to the 2 design keys of a good API that I’ve outlined in the beginning:

Controllers / end points only manipulates the request and provide a response. They are not used to validate the request. We can be sure that our controllers / end points will never be called with an invalid request.
By looking at our schema/user.js we can immediately understand how our request should look. We also can give those files to our customers / put them in our API documentation.

You can extend this idea and provide validation for responses as well. By writing a joi schema for the response and adding another middleware after your actual API end point that will validate the response you want to send against a response schema, so this can help you to avoid cases where developers might mistype a response key for examples usrename instead of username. And also by having request and response schema — your API is essentially self documented.

Hope this helped you to become a better JavaScript Jedi! Please share and like! Leave a comment or hit me on Twitter if you have any questions.

The land of undocument react.js: The Context

Thu, 03 Sep 2015 10:00:00 GMT

Русская версия на Хабре. Перевод не мой, спасибо webMarshal-у за перевод.

If you have any questions, or want to discuss some points — leave a comment here, or tweet me.

If we take a look at React component we can outline some properties.

State

Yes, each react component has state. State is something internal to the component. Only the component it self can read and write into its own state and as the name implies, the state of the component is used to store state (captain obvious here). Not interesting. Let’s move next.

Props

Or shall we say properties. Props is the data that affects the component display and behavior. Props can be optional or mandatory, and they are provided by the parent component. Ideally, when you provide your component with the same props — it renders the same. Not interesting as well. Let’s move on.

Context

Meet context, the reason I’m writing this post. Context is an undocumented feature of react.js and it is similar in a way to props, but the only difference is that props are passed explicitly by parent to child, and they do not propagate down the hierarchy, while context can be requested by children component implicitly.

But why?

Good question. Lets draw!

React state props tree#363x313

We have a Grandparent component that renders Parent A component that renders two child components: Child A and Child B.

Lets say Grandparent component knows something that Child A and Child B wants to know as well, but Parent A doesn’t care about. Lets call this piece of data Xdata. How would Grandparent component give ChildA and ChildB access to Xdata?

Well, using Flux architecture we can store Xdata inside a store and let Grandparent, Child A and Child B subscribe to that store. But what if we want Child A and Child B to be pure dumb components that only renders some markup?

Well, we than can pass Xdata as prop to Child A and Child B. But Grandparent can not pass props to its grand children without going through Parent. And its not that big deal if we have only 3 level nesting, but real applications have a lot more nesting levels where top components acts as containers and leaf components are just markup. Well, we can use mixins that will automatically pass the props down the hierarchy for us, but its not elegant.

Or we can use context. As I said earlier, context allows children component to request some data to arrive from component that is located higher in the hierarchy.

Here is how it looks:

var Grandparent = React.createClass({  
  childContextTypes: {
    name: React.PropTypes.string.isRequired
  },  getChildContext: function() {
    return {name: 'Jim'};
  },
  
  render: function() {
    return <Parent/>;
  }
    
});

var Parent = React.createClass({
 render: function() {
   return <Child/>;
 }
});

var Child = React.createClass({
 contextTypes: {
   name: React.PropTypes.string.isRequired
 }, render: function() {
  return <div>My name is {this.context.name}</div>;
 }
});

React.render(<Grandparent/>, document.body);

And here is a JSBin with the code. Change Jim to Jack and you see how your component re-renders.

WHAT IS HAPPENING?!

Our Grandparent component says two things:

I provide my children with context type named name of type string. This is what happening in childContextTypes declaration.
The value of context type named name is Jim. This is what happening in getChildContext function.

And our child component just says “Hey I expect to have context type named name!” and it gets it. As far as I understand (and I’m far from being expert in react.js internals) when react renders children components it checks which components want to have context and those that want context are provided with context if parent supplies them.

Cool!

Yep, expect when you encounter the following error

Warning: Failed Context Types: Required context `name` was not specified in `Child`. Check the render method of `Parent`.
runner-3.34.3.min.js:1

Warning: owner-based and parent-based contexts differ (values: `undefined` vs `Jim`) for key (name) while mounting Child (see: http://fb.me/react-context-by-parent)

Yes of course I checked the link, its not very helpful.

Here is the code that causes this (JSBin):

var App = React.createClass({
  render: function() {
    return (
      <Grandparent>
        <Parent>
          <Child/>
        </Parent>
      </Grandparent>
    );
  }
});

var Grandparent = React.createClass({  
  childContextTypes: {
    name: React.PropTypes.string.isRequired
  },  getChildContext: function() {
    return {name: 'Jim'};
  },
  
  render: function() {
    return this.props.children;
  }
    
});

var Parent = React.createClass({
  render: function() {
    return this.props.children;
  }
});

var Child = React.createClass({
  contextTypes: {
    name: React.PropTypes.string.isRequired
  },  render: function() {
    return <div>My name is {this.context.name}</div>;
  }
});

React.render(<App/>, document.body);

It doesn’t make sense now, but Ill explain a setup where this is a viable hierarchy in the end of this post.

So what is happening?

It took me a lot of time to understand why this happens. Googling the error only brought me to discussions by people who also wonders why this happens. I looked at other projects like react-router or react-redux that uses context to pass data down the component tree, when eventually I realized what is wrong.

Note: I said I’m not expert in react internals, so everything I’m going to say now is purely what I understood about react internals. Leave a comment if you think I’m wrong (preferably with a link that backups your comment).

Remember I said that each react component have state, props and context? Well, each component have as well so called parent and owner. And if we follow the link from the earlier warning (so yes, it is useful, I lied) we can understand that:

In short, the owner of a component is whomever creates the component, while the parent of a component is whomever would be the containing ancestor in the DOM hierarchy.

It took me some time to understand this statement.

So in my first example, the owner of Child component is Parent, as well as the parent of Child is Parent. While in the second example the owner of Child component is App while the parent is Parent.

Context is something that propagates in a strange way to all the descendant components, but will be assigned only to those components who explicitly asked for context. But context does not propagate from the parent component as you’d expect but from owner! And since the owner of Child is App, React tries to find the “name” attribute in context of App instead of Parent or Grandparent.

Here is the relevant bug report in React. And here is the relevant pull request that should fix this to Parent based context as opposed to Owner based context in React 0.14.

However React 0.14 is not there yet, so here is a fix (JSBin):

var App = React.createClass({
  render: function() {
    return (
      <Grandparent>
        { function() {
          return (<Parent>
            <Child/>
          </Parent>)
        }}
      </Grandparent>
    );
  }
});

var Grandparent = React.createClass({  
  childContextTypes: {
    name: React.PropTypes.string.isRequired
  },  getChildContext: function() {
    return {name: 'Jack'};
  },
  
  render: function() {
    var children = this.props.children;
    children = children();
    return children;
  }
    
});

var Parent = React.createClass({
  render: function() {
    return this.props.children;
  }
});

var Child = React.createClass({
  contextTypes: {
    name: React.PropTypes.string.isRequired
  },  render: function() {
    return <div>My name is {this.context.name}</div>;
  }
});

React.render(<App/>, document.body);

Instead of instancing the Parent and Child components inside App, we return a function. Then inside the Grandparent we call this function hence making Grandparent the owner of Parent and Child components. Context propagates as expected now.

Ok, cool, but… WHY?!

Remember my previous post about localization in react.js? Consider the following hierarchy:

<Application locale="en">
  <Dashboard>
    <SalesWidget>
      <LocalizedMoney currency="USD">3133.7</LocalizedMoney>
    </SalesWidget>
  </Dashboard>
</Application>

This is a static hierarchy, but usually you will have some routing and eventually you will create a situation where the owner and parent of your leaf components — differs.

Application is responsible for loading the locale data and initializing an instance of jquery/globalize. But it doesn’t use it. You don’t localize your top level components. Localization affects usually leaf components like localized text nodes, numbers, money display or time. And I’ve discussed earlier about the 3 possible ways to pass the globalize instance down the tree. Either we store it in a store and let leaf components subscribe to the store, but I think its incorrect. Leaf components should be pure dumb components.

Passing the globalize instance as props can be tedious, imagine ALL your components requires globalize instance. Its the same as making globalize instance a global variable and let those who needs it — use it.

But the most elegant way is using context. Application components says “Hey I have a globalize instance here, anyone who needs it let me know” and any Localized leaf components yells “Me! Me! I want it!”. This is an elegant solution. Leaf components stays pure, they dont depend on stores (yes they depend on context, but they have to, because they need it to render correctly). Globalize instance is not getting passed as props to your entire hierarchy. Every body is happy.

That’s it my friends!

May the force be with you.

A different approach for localizing react.js app

Tue, 18 Aug 2015 09:00:00 GMT

Howdy!

I’ve heard you use ReactJS? Good! I also heard you want to localize your application? That’s great! Well proper localization is hard, there are many js libraries for client side localization as well as yahoo/react-intl that provides localized react components based on JavaScript Intl API.

I my self use es6 syntax and avoid using mixins, because mixins are dead (and well, because there is no easy way to integrate mixins into es6 syntax). So react-intl didn’t work for me. But damn I wanted those localized numbers! In this article I’m gonna show a different (not bad neither good, just different) approach on how I did it.

Localize all the things!!

I do like the idea of Intl API, why reinvent the wheel when we have 51M of compressed localized data with support for dates, numbers, currencies and messages. So we do have jquery/globalize on client side to use all this data. The question is: How do I use it in a react.js application?

Well, here is the general idea

We need a way to identify user locale. Either we can guess it from the browser or (a preferred method, in my opinion) let the user choose his locale.
Once we know the locale, we need to load the needed cldr json data (we can load everything, or just the parts we really need) into Globalize.
….
PROFIT!!!!

Simple. Yet a bit complicated.

One thing I disliked in react-intl is that each component knew about the locale data. If you have a nested tree of: App => Dashboard => SalesWidget => FormattedMoney, even though the locale data is needed only in FormattedMoney component, its passed to all the child components of App (react-intl handled that automatically using mixins). I offer a different solution. Locale data is no more than just… data! And data should be stored inside… yes right a Store!

Show me the code!

Note: I use alt as my flux implementation. Ideally my locale info should be fetched from the user, but since I do a MVP, I omitted this step.

Application component issues a UserFetch action that goes to the API and fetches the user details. Locale store listen to UserFetchCompleted action and then gets the locale from the user, and loads the needed cldr data.

Here is how it looks

handleSetupStarted() {
    this.waitFor(ProfileStore.dispatchToken);

    let locale = 'en', //@TODO get from profile
        self = this;
    request({url: CLDR_URL + 'supplemental.json', type: 'json'})
        .then(function (supplemental) {
            request({url: CLDR_URL + locale + '.json', type: 'json'})
                .then(function (localeData) {
                    self.handleSetupFinished(locale, supplemental, localeData);
                })
                .fail(self.handleSetupFailed);
        })
        .fail(self.handleSetupFailed);
}

handleSetupFinished(locale, supplemental, localeData) {
    Globalize.load(supplemental, localeData);
    this.setState({translator: Globalize(locale)});
}

(Yes, I know, I do an ajax call from the store. I couldn’t find a better way to do it. For some reason I wans’t able to dispatch an action inside the ProfileStore once the user data loaded because I got this weird message of: Dispatch.dispatch(…): Cannot dispatch in the middle of a dispatch. and was lazy to investigate on how to fix it. Ideally this should be inside an action or an alt term called source. I’ll figure it out later).

See, the handleSetupStarted waits for the ProfileStore to finish loading the user, and once its done, we get its locale and load 2 files: supplemental.json (which is a common set of data for every locale) and en.json (which is data related to the en locale). Once it finished loading, it creates a new instance of Globalize.

Here is an example of a LocalizedMoney component

class Money extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      locale: LocaleStore.getState(),
    };
  }

  render() {
    let amount = parseFloat(this.props.children),
      translator = this.state.locale.translator,
      cls = classNames({
        money: true,
        positive: amount >= 0,
        negative: amount < 0,
      });

    return (
      <div className={cls}>
        {translator.formatCurrency(amount, this.props.currency)}
      </div>
    );
  }
}

And a usage example

render() {
    return (
        <FormattedMoney currency="USD">1337.93</FormattedMoney>
    );
}

See! No need to pass the locale data from the Application component to the LocalizedMoney component! LocalizedMoney just uses the Locale Store. You can even setup listener on localized components, to listen to changes on the Locale store, and when the user fires a Change Locale Action, you can change the locale on the fly without reloading the page. But I wouldn’t mind reloading the page. Locale is not something you change twenty times a day.

You can as well implement caching for different formatters inside the LocaleStore as well, and instead of using the translator (which is just an instance of Globalize) directly, you can use something like

LocaleStore.getCurrencyFormatter(“USD”).format(123.45);

Which internally looks like (pseudo code):

getCurrencyFormatter(currencyCode) {
    if(!this.state.currencyFormatters[currencyCode]) {
        this.state.currencyFormatters[currencyCode] = this.state.translator.currencyFormatter(currencyCode);
    }
    return this.state.currencyFormatters[currencyCode];
}

Note: formatter identified not only by currency, but by options as well. You can have two USD formatters: One that shows the number as

$1,000.00

And another one as

1,000.00 US Dollars

In short: RTFM.

A note about CLDR shipping

The process of shipping the locale data should be a build step of your application. There is cldr-data-downloader. Its a node module to download the CLDR data from unicode.org.

I then use gulp to bundle the supplemental (common) and locale based data into merged json file.

Less talk, more code:

gulp.task("cldr:download", function (cb) {
  cldrDataDownloader(
    "http://www.unicode.org/Public/cldr/" + CLDR_VERSION + "/json.zip",
    cldrPath,
    function (err) {
      if (err) return cb(err);
      cb();
    },
  );
});

gulp.task("cldr:bundle:supplemental", function () {
  var files = [
    "likelySubtags.json",
    "currencyData.json",
    "numberingSystems.json",
    "plurals.json",
    "ordinals.json",
  ];
  return gulp
    .src(
      files.map(function (f) {
        return path.join(cldrPath, "supplemental", f);
      }),
    )
    .pipe(extend("supplemental.json"))
    .pipe(gulp.dest(path.join(distDir, "cldr")));
});

gulp.task("cldr:bundle:locale", function () {
  var files = ["currencies.json", "numbers.json"];

  var tasks = locales.map(function (locale) {
    return gulp
      .src(
        files.map(function (f) {
          return path.join(cldrPath, "main", locale, f);
        }),
      )
      .pipe(extend(locale + ".json"))
      .pipe(gulp.dest(path.join(distDir, "cldr")));
  });
  return gulpMerge(tasks);
});

At this point I need only currency formatting, so I use the required files in both supplemental bundle and locale bundle. jquery/globalize actually giving a nice table that shows, what json files from the cldr you need in order to use different localization modules.

A final note

For now I am happy with this approach. In my opinion it decouples the components from the actual localization implementation. Components stays reusable even if you decide to switch to a different localization implementation. Your store can expose an API like the getCurrencyFormatter() I showed earlier, and actual implementation can be based on Intl, or your own. FormattedMoney component doesn’t care.

Intl API is supported in all major browsers and those that do not support it, can be polyfilled. Of course Intl API can be an overkill for most applications, especially if all you want is translating strings. But if your application requires displaying localized numbers, dates, times — I think there is no better solution other than Intl API.

Best of luck and may the force be with you.

yield code();

Farewell, Rust

Prelude

Web development in Rust

Farewell Rust, at least for now…

Templating

Localization and Internationalization

The web is dynamic by nature

Compilation time

Ecosystem maturity

Node.js is good enough

Closing notes

Footnotes

Introducing mjmx: a custom JSX runtime for mjml

Isolating Claude Code

Built in Claude Code sandboxing

Existing tools

How I work

Rolling my own

Rolling my own, with blackjack and VMs

Why you probably shouldn’t copy my approach

I'm Netlify-Free!

What difference does it make?

How to do it?

Vibe Management

Working with systemd timers

The commoditization of AI

But what about AGI?

Ghost Engineers

Ad Hominem

The research

Findings

Methodology

My analysis

My speculation

How to be a great manager

Why DX doesn't matter

Software Development Fragmentation

One year of Rust in production

If it compiles, it runs

And when it runs, it’s very stable

And the tooling is incredible

Compile time is still PITA

The tools are not quite the same, yet

Software development is hard

Let's blame the dev who pressed "Deploy"

The day of the blue screens of death

Tips for improving your CV

Who reads your CV?

How to style your CV to catch the eye

How to structure your CV?

Header

Summary

Education

Work experience

Additional information

Common mistakes in making your CV

Biased information

Unnecessary information / Generalization / Too long of a CV

Design / Format / Spelling

Final words

Why your team might be holding you back

Why your team might be holding you back

The lone-wolf

xz backdoor

Timeline

The Backdoor

Discovery

Who is behind the backdoor?

What’s next?

Further reading

The Curse of the Senior Software Engineer

What’s up with the titles?

The promotion dilemma

The Curse of the Senior Software Engineer

How to avoid the curse?

Serving Astro with Rust

The problems with server side rendering

Templates are hard to maintain

Everything is a run-time problem

Render route `/render`