Working with systemd timers

The other day I thought to myself that it would be a good idea to have some backups of my data. So I was wondering, how would I execute a periodic backup task?

Most of you are probably familiar with cron. cron is a Unix utility to run scheduled tasks. There is a file called crontab which you edit by issuing crontab -e command, and inside schedule tasks such as:

0 1 * * * /home/skwee357/backup.sh

The first 5 entries are what is known as a cron expression, and they dictate when the command will be executed. In the above example, the expression says “every day at 1 am”. There is a cool website called crontab.guru which allows you to compose various cron expressions. After the cron expression, we have the actual script to execute. This can be a bash script, or a binary like curl.

Cron is easy, very simple and robust mechanism to execute periodic tasks on a *nix server, and is available by default on all popular Linux distributions. However, cron suffers from some issues:

1. If the system is down when the cron needs to run, the cron will be missed
2. There is no built-in status monitoring
3. There are no built-in logs
4. If you want to execute pre/post commands (for example by pinging an external service for success/failure) you have to do it inside the script itself

Luckily, there is another utility called systemd that allows us to overcome all of the above issues. systemd is a relatively new utility that provides an array of components for Linux systems. It is essentially a system and service manager, which also replaces some daemons and utilities like device management, login management, network connection management, etc. systemd is adopted by most popular Linux distributions, but it’s important to note that it’s not adopted by other Unix-like system such as FreeBSD or OpenBSD. There are various unit-types inside systemd but the two we are going to focus on today are .service which define the actual service to execute, and .timer which acts as a cron-like job-scheduler. Let’s start with the timer.

systemd timers, and services, are located at /etc/systemd/{system/user} depending on whether it’s a system-wide service/timer or user-only system/timer. We will focus on system-wide services and timers, so all edits (unless instructed differently) happen inside /etc/systemd/system. First, we will create a timer for our service: vim /etc/systemd/system/backup.timer:

[Unit]
Description=Backup service timer

[Timer]
OnCalendar=*-*-* 1:00:00
Persistent=true

[Install]
WantedBy=timers.target

First, we define the unit, and it’s description. Then, we define the timer itself. The format for OnCalendar is actually DayOfWeek Year-Month-Day Hour:Minute:Second, but every component can be replaced by asterisk to signify any value. DayOfWeek can be omitted, and any two values separated by .. indicate a contiguous range, so for example Mon..Fri 22:30 means run at 22:30 on weekdays. We can use comma to include multiple values, for example: Sat,Sun 20:00 means run on weekends at 20:00. OnCalendar can appear multiple times to include multiple date ranges. Lastly, it is possible to also specify timezone in the end, for example: *-*-* 02:00:00 Europe/Amsterdam to run every day at 2 AM in Amsterdam time.

Then we have Persistent which controls whether the timer is persistent or transient. Transient timers are valid only for the current session, meaning if the system is powered off when the timer is due, it won’t run after system boot, just like cron. However, if the timer is persistent, this means that when the system boots, the timer will run immediately, which is great for backups that must run even if the system crashed during/before the expected backup.

Lastly, we have Install directives. Unlike crontab entries, systemd units (be it services which we will talk about in a bit, or timers) are NOT active by default, and needs to be installed. You have probably seen commands like systemctl enable redis.service at least once in your development career, this is the actual installation of the unit, which from now on will be active in systemd. When installing systemd units, we can optionally specify dependencies, and this is done by using WantedBy or Wants directives. In case of timers, it is common to specify WantedBy=timers.target in the Install section. timers.target is a special target unit that sets up all the timers that shall be active after boot. So if we want our timer to be active after boot, we have to make sure that timers.target wants it. Save the file, and exit vim. We are done with the timer, time to move onto the service.

Create a new file /etc/systemd/system/backup.service:

[Unit]
Description=Backup service

[Service]
ExecStart=/home/skwee357/backup.sh
Type=oneshot

As with the timer, we first define our Unit and it’s description. Then we define the service itself. In ExecStart we give the command that systemd should execute when the service is ran. Type is a bit tricky, and mainly controls how systemd acts after forking the processing and executing our command. There are various Type values and you can find them in man systemd, but the reason I have used oneshot is that it essentially allows us to specify multiple ExecStart directives, and executing all the commands serially. If you do not intend to run multiple commands, you can use Type=simple or omit the Type at all, as Type=simple is the default when no Type present, but ExecStart is present.

So far so good. Let’s spice it up. Let’s say we want to use a service such as Pushover to send a push notification when backup completes with either success or failure. For that, we can use ExecStopPost in the [Service] section. ExecStopPost will be called once ExecStart will finish running. In order to determine the result of the execution, we will have 3 variables available to use:

• $SERVICE_RESULT - will be success for successful execution, and a bunch of other codes for non successful execution
• $EXIT_CODE - the exit code, 0 for success, non zero for error
• $EXIT_STATUS - somewhat different from $EXIT_CODE in a sense that if the process was killed by a signal (SIGTERM, or SIGKILL for example), this will contain the exit code of the termination (unlike $EXIT_CODE that contains the exit code of our process)

So we can write a script in bash something along the lines of ExecStopPost=/home/skwee357/pushover.sh $EXIT_CODE, and inside we will parse the exit code and send appropriate push notification. We can also use ExecStartPre= in order to execute some code prior to calling ExecStart. There are way, way more settings in a systemd service, so RTFM if you need more.

After we finished writing our timer and service, it’s time to install the service. This is done via systemctl enable backup.timer. A symlink will be created, and systemd will manage our timer which will trigger our service when the time comes. In order to disable the timer, we can execute systemctl disable backup.timer. And by executing systemctl status backup.timer we can view the status of our timer. It should be active (waiting) and show us how much time left until trigger. We can also execute systemctl status backup.service and view the status, as well as logs, for the service. In general, we can use journalctl -u backup.service to view the logs of our systemd unit (hence the -u), and combine it with -b to show logs only from current boot session, or -b -N where N is a number, for example 2, to show logs from two boots ago, etc.