EES-5685 - added many additional details to alerts being sent through to Slack

[duncan-at-hiveit]

Overview

This PR adds additional useful contextual information to alerts coming through to Slack from Azure Monitor. This is achieved by amending the Logic App to post additional details of the alert to Slack.

Before

Once the alerts were drab, lifeless and unhappy:

After

But now, they spring with colour, details and joy!

Changes to Logic App workflow

Because getting alert details from several layers deep in the alert metric JSON body is very verbose in the Logic App, we now capture a few useful variables ahead of time before creating the Slack message.

We then include:

• colour-coding based on the severity of the alert if it is firing (red for critical and error, yellow for warning, and green for informational and verbose).
• colour-coding based on resolved alerts (green)
• a detailed message about the condition that caused the alert e.g. thresholds, values and operators that are being applied
• details about the time of the alert testing, firing and resolving
• links to view the affected resource
• links to view the timeline of the alert rule that got fired (I wanted to include a link to the actual time of firing that specific alert, but it was a very tricky URL to try to create!)

Testing

The Logic App is currently expecting alert message payloads of the form documented here: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-payload-samples#metric-alert-with-a-static-threshold-and-the-monitoringservice--platform

We can manually use this example payload and amend it in different ways within Azure Portal's Logic App Designer Run with payload feature, and test how the Logic App responds to changes to severity, monitoring conditions etc.