Restic: send an email on backup failure #202

reynir · 2024-03-03T20:24:20Z

reynir commented

2024-03-03 20:24:20 +00:00

The restic docker image has a mechanism for specifying scripts to execute on success and failure. https://github.com/djmaze/resticker?tab=readme-ov-file#execute-commands-after-backup

However, a challenge is that the image comes with very few utilities. You are expected to use curl to notify a web service that then can send an email for you. Thankfully, curl can do everything including sending mail.

There is some tomfoolery in order to setup noreply.data.coop as a general purpose domain for sending notification emails. We can't use data.coop (I think) because mailu manages that.

I have manually tested the curl invocation in a random container that had curl installed and was on the postfix network. First attempt I sent it from noreply@services.data.coop, but that is not in our allowed sender domains. Then I (ab)used noreply@member.data.coop with success. The remaining changes are untested.

I also set restic to exclude backup of its own files just because they are all generated by ansible.

The restic docker image has a mechanism for specifying scripts to execute on success and failure. https://github.com/djmaze/resticker?tab=readme-ov-file#execute-commands-after-backup However, a challenge is that the image comes with very few utilities. You are expected to use curl to notify a web service that then can send an email for you. Thankfully, curl **can do everything** including sending mail. There is some tomfoolery in order to setup noreply.data.coop as a general purpose domain for sending notification emails. We can't use data.coop (I think) because mailu manages that. I have manually tested the curl invocation in a random container that had curl installed **and** was on the postfix network. First attempt I sent it from noreply@services.data.coop, but that is not in our allowed sender domains. Then I (ab)used noreply@member.data.coop with success. The remaining changes are untested. I also set restic to exclude backup of its own files just because they are all generated by ansible.

reynir added 1 commit 2024-03-03 20:24:21 +00:00

Restic: send an email on backup failure 6982d0feaa

samsapti requested changes 2024-03-03 20:43:52 +00:00

Dismissed

samsapti left a comment

Some suggestions and minor things that need to be fixed, otherwise great work!

roles/docker/defaults/main.yml Outdated

					
				@ -53,0 +53,4 @@

				    # mail dance

				    domain: "noreply.{{ base_domain }}"

				    allowed_sender_domain: true

				    mail-from: "noreply@noreply.{{ base_domain }}"

samsapti commented

2024-03-03 20:38:08 +00:00

I suggest restic@noreply.{{ base_domain }} or backup@noreply.{{ base_domain }} instead. Also, let's keep the syntax style with underscores, so mail_from instead of mail-from.

I suggest `restic@noreply.{{ base_domain }}` or `backup@noreply.{{ base_domain }}` instead. Also, let's keep the syntax style with underscores, so `mail_from` instead of `mail-from`.

reynir marked this conversation as resolved

roles/docker/templates/compose-files/restic.yml.j2 Outdated

					
				@ -15,6 +15,7 @@ services:

				        --tag datacoop-volumes

				        --exclude '*.tmp'

				        --exclude '/mnt/volumes/mastodon/mastodon_data/cache/'

					--exclude '/mnt/volumes/restic/'

samsapti commented

2024-03-03 20:39:02 +00:00

Please change this line to use spaces instead of tabs, and have it on the same indentation level as the above lines.

reynir marked this conversation as resolved

roles/docker/templates/compose-files/restic.yml.j2 Outdated

					
				@ -22,9 +23,13 @@ services:

				        --keep-weekly 5

				        --keep-monthly 12

				      TZ: Europe/Copenhagen

				      POST_COMMANDS_FAILURE=/run/libexec/failure.sh

samsapti commented

2024-03-03 20:40:01 +00:00

This needs to be with a : instead of = like the rest of the variables.

This needs to be with a `: ` instead of `=` like the rest of the variables.

reynir marked this conversation as resolved

roles/docker/templates/restic/failure.sh.j2 Outdated

					
				@ -0,0 +1,14 @@

				#!/bin/sh

				curl smtp://postfix --mail-from {{ services.restic.mail-from }} --mail-rcpt admin-hold@data.coop --upload-file . << END_OF_MAIL

samsapti commented

2024-03-03 20:41:17 +00:00

Our email is admin@data.coop 🙂
Also, mail-from -> mail_from like above.

Our email is `admin@data.coop` 🙂 Also, `mail-from` -> `mail_from` like above.

reynir commented

2024-03-04 07:48:14 +00:00

Actually we have both for some reason

reynir marked this conversation as resolved

reynir added 1 commit 2024-03-04 08:22:07 +00:00

Address comments by @samsapti 9fb16d3a69

We need to use ':' instead of '=' in yaml for environment variable
bindings.
Spurious tab where it should be all spaces
Rename variable mail-from to mail_from to align with existing code style
Nit: change email addresses

samsapti requested changes 2024-03-04 11:22:17 +00:00

Dismissed

samsapti left a comment

One last thing, then it's approved 💯

roles/docker/templates/restic/failure.sh.j2 Outdated

					
				@ -0,0 +1,14 @@

				#!/bin/sh

				curl smtp://postfix --mail-from {{ services.restic.mail_from }} --mail-rcpt admin@data.coop --upload-file . << END_OF_MAIL

samsapti commented

2024-03-04 11:21:45 +00:00

Let's use {{ smtp_host }} here 😄

Let's use `{{ smtp_host }}` here 😄

benjaoming commented

2024-03-04 11:39:54 +00:00

You mean like smtp://{{ smtp_host }} ?

You mean like `smtp://{{ smtp_host }}` ?

reynir commented

2024-03-04 11:52:00 +00:00

FINE!

😆 1

samsapti marked this conversation as resolved

reynir added 1 commit 2024-03-04 11:48:59 +00:00

. ac64706fcb

samsapti requested changes 2024-03-04 12:05:57 +00:00

Dismissed

samsapti left a comment

One last thing (v2.0) 😃

roles/docker/templates/restic/failure.sh.j2 Outdated

					
				@ -0,0 +1,14 @@

				#!/bin/sh

				curl smtp://{{ smtp_host }} --mail-from {{ services.restic.mail_from }} --mail-rcpt admin@data.coop --upload-file . << END_OF_MAIL

				From: Restic backup <{{ services.restic.mail-from }}>

				To: admin-hold@data.coop