feat (monitoring): [alerts] enable new recommended experience for aks clusters #435
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pod level alert: at least one Job instance did not complete successfully for the last 6 hours.
Pod level alert: The average CPU usage per container exceeds 95% for the last 5 minutes.
…erAverageMemoryHigh Pod level alert: The average memory usage per container exceeds 95% for the last 5 minutes
Pod level alert: One or more pods is in a failed state for the last 5 minutes
Platform level alert Node cpu percentage is replacing this
no replacement available
Node level alert: A node has been unreachable for the last 15 minutes
Platform level alert Node memory working set percentage is greater than 100% is replacing this
…edCount Cluster level alert: One or more containers within pods have been killed due to out-of-memory (OOM) events for the last 5 minutes
Pod level alert: The average usage of Persistent Volumes (PVs) on pod exceeds 80% for the last 15 minutes
Pod level alert: The percentage of pods in a ready state falls below 80% for any deployment or daemonset in the Kubernetes cluster for the last 5 minutes
…rRestart Pod level alert: One or more containers within pods in the Kubernetes cluster have been restarted at least once within the last hour
johndowns
approved these changes
Nov 18, 2024
There was a problem hiding this comment.
@ferantivero thanks for this! I can see a ton of work here - thanks so much. The changes all make sense.
We should ensure we link to the Recommended alert rules for Kubernetes clusters in the reference architecture too, to make sure it's clear where these came from.
|
#sign-off please let's consider merging this once we landed the desired changes at the RA level |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WHY?
We do wanted to switch from legacy/retired container insights metric alert rules to the new recommend Prometheus metric alert rules
WHAT Changed?
replace legacy Container CPU % CI alert w/ KubeContainerAverageCPUHigh
replace legacy Container working set memory % CI alert w/ KubeContain…
replace legacy Failed Pod counts CI alert w/ KubePodFailedState
replace legacy Node NotReady status CI alert w/ KubeNodeUnreachable
replace legacy OOM Killed Containers CI alert w/ KubeContainerOOMKill…
replace legacy Persistent Volume Usage % CI alert w/ KubePVUsageHigh
replace legacy Pods ready % CI alert w/ KubePodReadyStateLow
replace legacy Restarting container count CI alert w/ KubePodContaine…
add extra recommended Prometheus Pod level metric alert rules
add extra recommended Prometheus Node level metric alert rules
add extra recommended Prometheus Cluster level metric alert rules
disabling legacy Node CPU % CI alert
disabling legacy Node Disk Usage % CI alert
disabling legacy Node working set memory % CI alert
Test