Feature: AWS CloudWatch Container Insights Addon

docs/addons/aws-cloudwatch-insights.md

@@ -0,0 +1,99 @@
+# CloudWatch Insights Add-on
+
+The Cloudwatch Insights add-on adds support for [CloudWatch Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) to an EKS cluster.
+This replaces the current [Container Insights](./container-insights.md) add-on and all customers on it, should migrate to
+CloudWatch Insights.
+
+Customers can use CloudWatch Insights to collect, aggregate, and summarize metrics and logs from your containerized 
+applications and microservices. Container Insights collects data as performance log events using an embedded metric
+format. These performance log events are entries that use a structured JSON schema that enables high-cardinality data
+to be ingested and stored at scale. From this data, CloudWatch creates aggregated metrics at the cluster, node, pod,
+task, and service level as CloudWatch metrics. The metrics that Container Insights collects are available in
+CloudWatch automatic dashboards, and also viewable in the Metrics section of the CloudWatch console.
+
+**IMPORTANT**
+
+CloudWatch does not automatically create all possible metrics from the log data, to help you manage your Container
+Insights costs. However, you can view additional metrics and additional levels of granularity by using CloudWatch Logs
+Insights to analyze the raw performance log events.
+
+Metrics collected by Container Insights are charged as custom metrics. For more information about
+[CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/), see Amazon CloudWatch Pricing.
+
+Note: that this add-on cannot co-exist with `adot-addon`, `cloudwatch-adot-addons` or `cloudwatch-logs` on same EKS
+cluster as they have conflicting and redundant interactions.
+
+## Usage
+
+Add the following as an add-on to your main.ts file to add CloudWatch Insights to your cluster
+
+```typescript
+import 'source-map-support/register';
+import * as cdk from 'aws-cdk-lib';
+import * as blueprints from '@aws-quickstart/eks-blueprints';
+
+const app = new cdk.App();
+
+const addOn = new blueprints.addons.CloudWatchInsights();
+
+const blueprint = blueprints.EksBlueprint.builder()
+  .version("auto")
+  .addOns(addOn)
+  .build(app, 'my-stack-name');
+```
+
+## Prerequisites
+
+Once the CloudWatch Insights add-on has been installed in your cluster, validate that the
+`AWS Cloudwatch Observability Controller` is installed and FluentBit is installed: 
+
+```bash
+kubectl get pods -n amazon-cloudwatch
+```
+
+You should see output similar to the following respectively:
+
+```
+NAMESPACE           NAME                                                 READY   STATUS    RESTARTS        AGE
+amazon-cloudwatch   amazon-cloudwatch-observability-controller-manager   1/1     Running   1 (4d22h ago)   5d
+amazon-cloudwatch   cloudwatch-agent                                     1/1     Running   1 (4d22h ago)   5d
+amazon-cloudwatch   fluent-bit                                           1/1     Running   1 (4d22h ago)   5d
+```
+
+## CloudWatch Dashboard
+
+Once enabled, enhanced container insights page looks like below from AWS console, with the high level summary of your
+clusters, kube-state and control-plane metrics. The Container Insights dashboard shows cluster status and alarms.
+It uses predefined thresholds for CPU and memory to quickly identify which resources are having higher consumption,
+and enabling proactive actions to avoid performance impact.
+
+![CloudWatch Insights Dashboard](./../assets/images/cloudwatch-insights-dashboard.png)
+
+Additional functionality of the addon is listed in this blog [Introducing CloudWatch Insights with Enhanced Monitoring](https://aws.amazon.com/blogs/mt/new-container-insights-with-enhanced-observability-for-amazon-eks/).
+
+### View Top 10 Lists
+
+The CloudWatch Insights Dashboard can also let you quickly view the Top 10 lists of Cluster, Nodes, Pods, Workloads, and
+Containers as shown below. Based on their consumption, you can set up critical charts that let you identify risky components
+without using alarms and before you are resource constrained.
+
+![CloudWatch Top 10 Views](./../assets/images/cloudwatch-insights-top10.png)
+
+### Cluster Overview
+
+The dashboard also lets you quickly view the consumption of your cluster, with the clusters ranked on their "criticality"
+where they're top ranking if they're in alarm, and then all the others are ranked based on resource consumption.
+They're visible in a list view as shown below.
+
+![CloudWatch clusters overview](./../assets/images/cloudwatch-insights-clusters.png)
+
+### Popular Workloads
+
+CloudWatch insights has pre-built automatic dashboards and alarms for popular workloads, and it allows you to drill into
+logs generated by those workloads as well.
+
+
+### Metrics namespaces
+
+CloudWatch insights also exposes the new "ContainersInsights" metrics namespace, it contains all the various dimensions 
+of EKS clusters and the data that's exported from the CloudWatch agents.

docs/addons/container-insights.md

@@ -1,5 +1,8 @@
 # Container Insights Add-on
 
+**!!  This add-on is deprecated in favour of CloudWatch insights to provide deeper integration with EKS through CloudWatch.
+Please use CloudWatch Insights instead linked here: [AWS CloudWatch Insights](./aws-cloudwatch-insights.md) !!**
+
 The Container Insights add-on adds support for [Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) to an EKS cluster.
 
 Customers can use Container Insights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices. Container Insights collects data as performance log events using an embedded metric format. These performance log events are entries that use a structured JSON schema that enables high-cardinality data to be ingested and stored at scale. From this data, CloudWatch creates aggregated metrics at the cluster, node, pod, task, and service level as CloudWatch metrics. The metrics that Container Insights collects are available in CloudWatch automatic dashboards, and also viewable in the Metrics section of the CloudWatch console.

docs/addons/grafana-operator.md

@@ -22,7 +22,7 @@ const blueprint = blueprints.EksBlueprint.builder()
 
 - `createNamespace`: (boolean) If you want CDK to create the namespace for you.
 
-- `values`: Arbitrary values to pass to the chart. Refer to the Grafana Operator [Helm Chart documentation](https://grafana-operator.github.io/grafana-operator/docs/installation/helm/) for additional details. It also supports all [standard helm configuration options](https://github.com/aws-quickstart/cdk-eks-blueprints/blob/main/docs/addons/index.md#standard-helm-add-on-configuration-options).
+- `values`: Arbitrary values to pass to the chart. Refer to the Grafana Operator [Helm Chart documentation](https://grafana.github.io/grafana-operator/docs/installation/helm/) for additional details. It also supports all [standard helm configuration options](https://github.com/aws-quickstart/cdk-eks-blueprints/blob/main/docs/addons/index.md#standard-helm-add-on-configuration-options).
 
 ## Validation
 

docs/addons/index.md

@@ -17,6 +17,7 @@ The framework currently supports the following add-ons.
 | [`ApacheAirflowAddOn`](./apache-airflow.md)                                           | This add-on is an implementation of Apache Airflow on EKS using the official helm chart.                                                  | ✅ |
 | [`ArgoCDAddOn`](./argo-cd.md)                                             | Provisions Argo CD into your cluster.                                                                                                 | ✅ | ✅ |
 | [`AWS Batch for EKS`](./aws-batch-on-eks.md)                              | Enables EKS cluster to be used with AWS Batch on EKS                                                                    | ✅ | ✅ |
+| [`AWS CloudWatch Insgihts`](./aws-cloudwatch-insights.md) | Provisions CloudWatch Insights to be used with the EKS cluster. | ✅ | ✅ |  
 | [`AWS for Fluent Bit`](./aws-for-fluent-bit.md)                           | Provisions Fluent Bit into your cluster for log aggregation and consumption.                                                                        | ✅ | ✅
 | [`AWS Load Balancer Controller`](./aws-load-balancer-controller.md)       | Provisions the AWS Load Balancer Controller into your cluster.                                                                        | ✅ | ✅ |
 | [`AWS Node Termination Handler`](./aws-node-termination-handler.md)       | Provisions Node Termination Handler into your cluster.                                                                        | ✅ |

examples/blueprint-construct/index.ts

@@ -68,6 +68,7 @@ export default class BlueprintConstruct {
             new blueprints.addons.XrayAddOn(),
             // new blueprints.addons.CloudWatchAdotAddOn(),
             // new blueprints.addons.ContainerInsightsAddOn(),
+            // new blueprints.addons.CloudWatchInsights(),
             new blueprints.addons.IstioBaseAddOn(),
             new blueprints.addons.IstioControlPlaneAddOn(),
             new blueprints.addons.CalicoOperatorAddOn(),

lib/addons/cloud-watch-insights/iam-policy.ts

@@ -0,0 +1,14 @@
+import {PolicyDocument, PolicyStatement} from "aws-cdk-lib/aws-iam";
+
+export function ebsCollectorPolicy(): PolicyDocument {
+  return new PolicyDocument({
+    statements: [
+      new PolicyStatement({
+        actions: [
+          'ec2:DescribeVolumes',
+        ],
+        resources: ['*']
+      })
+    ]
+  });
+}

lib/addons/cloud-watch-insights/index.ts

@@ -0,0 +1,88 @@
+import {Construct} from 'constructs';
+import * as eks from "aws-cdk-lib/aws-eks";
+import * as iam from "aws-cdk-lib/aws-iam";
+import {ClusterInfo} from "../../spi";
+import {conflictsWith, createNamespace, supportsALL} from "../../utils";
+import {CoreAddOn, CoreAddOnProps} from "../core-addon";
+import {ebsCollectorPolicy} from "./iam-policy";
+import {ManagedPolicy} from "aws-cdk-lib/aws-iam";
+
+/**
+ * Configuration options for AWS Container Insights add-on.
+ */
+export type CloudWatchInsightsAddOnProps = Omit<CoreAddOnProps, "saName" | "addOnName" | "version"> & {
+  /**
+   * Gives CloudWatch agent access to EBS performance systems by adding an IAM role as defined here:
+   * https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Observability-EKS-addon.html#install-CloudWatch-Observability-EKS-addon-configuration
+   */
+  ebsPerformanceLogs?: boolean,
+  /**
+   * Custom CloudWatch Agent configuration, specifics can be found here:
+   * https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Observability-EKS-addon.html#install-CloudWatch-Observability-EKS-addon-configuration
+   */
+  customCloudWatchAgentConfig?: string,
+
+  /**
+   * Define the CloudWatch Agent configuration
+   */
+  version?: string,
+};
+
+const defaultProps = {
+  addOnName: "amazon-cloudwatch-observability",
+  version: "v1.1.1-eksbuild.1",
+  saName: "cloudwatch-agent",
+  namespace: "amazon-cloudwatch"
+};
+
+/**
+ * Implementation of AWS CloudWatch Insights Addon
+ */
+@supportsALL
+export class CloudWatchInsights extends CoreAddOn {
+
+  readonly options: CloudWatchInsightsAddOnProps;
+
+    constructor(props?: CloudWatchInsightsAddOnProps) {
+      super({ ...defaultProps, ...props });
+
+      this.options = props ?? {};
+    }
+
+    @conflictsWith("AdotCollectorAddon", "CloudWatchAdotAddon", "CloudWatchLogsAddon")
+    deploy(clusterInfo: ClusterInfo): Promise<Construct> {
+      const cluster = clusterInfo.cluster;
+      const context = clusterInfo.getResourceContext();
+
+      const insightsSA = cluster.addServiceAccount("CloudWatchInsightsSA", {
+        name: defaultProps.saName,
+        namespace: defaultProps.namespace
+      });
+
+      const insightsNamespace = createNamespace(defaultProps.namespace,  cluster);
+
+      insightsSA.node.addDependency(insightsNamespace);
+
+      insightsSA.role.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy'));
+      insightsSA.role.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('AWSXrayWriteOnlyAccess'));
+
+      const insightsAddon = new eks.CfnAddon(context.scope,  "CloudWatchInsightsAddon", {
+        addonName: defaultProps.addOnName,
+        clusterName: cluster.clusterName,
+        addonVersion: defaultProps.version,
+        serviceAccountRoleArn: insightsSA.role.roleArn,
+      });
+      insightsAddon.node.addDependency(insightsSA);
+      insightsAddon.node.addDependency(insightsNamespace);
+
+      if (this.options.ebsPerformanceLogs != undefined && this.options.ebsPerformanceLogs) {
+        insightsSA.role.attachInlinePolicy(
+          new iam.Policy(context.scope, "EbsPerformanceLogsPolicy", {
+            document: ebsCollectorPolicy()
+          })
+        );
+      }
+
+      return Promise.resolve(insightsAddon);
+    }
+}

lib/addons/container-insights/index.ts

@@ -6,6 +6,9 @@ import { HelmAddOn, HelmAddOnUserProps } from "../helm-addon";
 import { ValuesSchema } from "./values";
 import { conflictsWith, createNamespace, supportsALL } from "../../utils";
 
+/**
+ * @deprecated use CloudWatch Insights add-on instead
+ */
 export interface ContainerInsightAddonProps extends Omit<HelmAddOnUserProps, "namespace"> {
     values?: ValuesSchema
 }
@@ -19,6 +22,9 @@ const defaultProps = {
     repository: "https://aws-observability.github.io/aws-otel-helm-charts"
 };
 
+/**
+ * @deprecated use CloudWatch Insights add-on instead
+ */
 @supportsALL
 export class ContainerInsightsAddOn extends HelmAddOn {
 

lib/addons/index.ts

@@ -14,6 +14,7 @@ export * from './calico';
 export * from './calico-operator';
 export * from './cloudwatch-adot-addon';
 export * from './cloudwatch-logs';
+export * from './cloud-watch-insights';
 export * from './cert-manager';
 export * from './cluster-autoscaler';
 export * from './container-insights';

mkdocs.yml

@@ -36,6 +36,7 @@ nav:
     - Calico Operator: 'addons/calico-operator.md'
     - Certificate Manager: 'addons/cert-manager.md'
     - CloudWatch ADOT: 'addons/cloudwatch-adot-addon.md'
+    - CloudWatch Insights: 'addons/aws-cloudwatch-insights.md'
     - Cluster Autoscaler: 'addons/cluster-autoscaler.md'
     - Container Insights: 'addons/container-insights.md'
     - CoreDns: 'addons/coredns.md'