allgood.systems: get a Slack message when your server goes down

I was recently asked if it’s possible to post a message to a Slack channel whenever the state of a monitor on allgood.systems changes (for example, a web server goes down, a job starts failing, etc.).

The answer is “yes”, and it’s quite easy to set this up once you have a Slack app configured.

All you need to do on allgood’s side is to add a web hook to your notification group, enter the URL you get from Slack, and define what message you want to send.

Creating a Slack app

In the past you could just add Slack’s “Incoming WebHook” to your workspace and were – more or less – done.

A while ago, however, this method was deprecated, and Slack now recommends to create your own “App” instead.

“Creating an app” sound more involved than it actually is. You just have to click through a few menus to “build” one.

Let’s follow the Quick Start Guide and take the following actions:

1) Go to Apps and click “Create New App” -> “From scratch”. Then give your app a name, for example “allgood bot”, and choose the workspace to use it in.

2) Go to “OAuth & Permissions” -> “Scopes”, and then add the following “Bot Token Scopes”: chat:write.public and chat:write. This will allow your bot to post messages.

3) Go to “Incoming WebHooks” and set it to “enabled”.

4) Go back to “Basic Information” and click “Install to Workspace”. You will be prompted to allow the bot’s actions and have to select a channel for your bot to post in (for example #general).

5) Go back to “Incoming WebHooks”, and copy the URL that is referenced for the channel you selected.

Now, log back into allgood.systems and continue the setup there.

Setup on allgood.systems

First, you go to notification settings and then either select an existing notification group or create a new one.

For the existing or new group, you can then click on “Add member”, select “Web hook” as type and “POST” as method:

Now, you can copy the web hook URL from Slack and enter it into the URL field.

In the JSON payload you can define your desired message structure and make use of the available placeholders (such as the monitor url).

For example:

{
    "blocks": [
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "{{subject}}"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "👉  <{{monitor_url}}|View Monitor>"
            }
        }
    ]
}

If you’ve attached the web hook to a notification group already linked to your monitors, your setup is now complete.

If you instead created a new notification group, go to one of your monitors, select “Edit” and choose the new group in the monitor’s notification settings.

Now, whenever a notification is triggered, you will see a new message in your configured Slack channel.

And, if you enabled repeated down notifications, you will also get these notifications repeatedly in Slack.