kubernetes cluster monitoring

kubernetes cluster level monitoring, including api server and etcd cluster monitoring.

Features

Many of the useful settings come from the kube-prometheus project,and we added some new features.

• Enable pv for grafana

• Enable pv for prometheus

• Add kubernetes cluster monitoring dashboard

• Add kubernetes node overview dashboard

• Enable systemd service monitoring

• Add API server dashboard

• Enable etcd cluster monitoring

• Log collection with ElasticSearch and Fluent-Bit

Install

There are two ways to customize and install, choose one you prefer.

• Install from the generated YAML

• Install from jsonnet

And then install logging services with helm:

• Install logging services with helm

Install from the generated YAML files in manifests directory

We have generated all YAML files in manifests directory, if you have no jsonnet env, you can modify these files to customize your configuration.

Step 1: Customizing

• Overwrite etcd cluster certificate base64 data in prometheus-secretEtcdCerts.yaml
• Set etcd server ips in prometheus-endpointsEtcd.yaml
• Put your alert manager configuration in alertmanager-secret.yaml

Step 2: Apply to kubernetes

kubectl apply -f manifest/

Note: If some resources created failed, wait for prometheus operator up and then apply again.

Install from jsonnet

Step 0: Install jsonnet env

• Install jsonnet

go get github.com/google/go-jsonnet/cmd/jsonnet

• Install jb

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb

• Install gojsontoyaml

go get github.com/brancz/gojsontoyaml

• Download jsonnet dependencies

#run in monitoring directory
jb install

Step 1: Customizing

• Put etcd cluster certificate path and IPs in config.libsonnet

{
   etcd+:: {
      // Configure this to be the IP(s) to scrape - i.e. your etcd node(s) (use commas to separate multiple values).
      ips: ['172.16.4.155', '172.16.4.156', '172.16.4.157'],

      // Set these three variables to the fully qualified directory path on your work machine to the certificate files that are valid to scrape etcd metrics with (check the apiserver container).
      // All the sensitive information on the certificates will end up in a Kubernetes Secret.
      clientCA: importstr 'etcd/etcd-client-ca.crt',
      clientKey: importstr 'etcd/etcd-client.key',
      clientCert: importstr 'etcd/etcd-client.crt',
      insecureSkipVerify: true,
   }
}

• Put your additional dashboard name and path in config.libsonnet

{
  grafanaDashboards+:: {
          //Configure this to be the dashboard definitions, keep the name unique
          'k8s-cluster-monitoring.json': (import 'dashboards/k8s-cluster-monitoring-dashboard.json'),
          'k8s-node-dashboard.json': (import 'dashboards/k8s-node-dashboard.json'),
          'api-server.json': (import 'dashboards/api-server-dashboard.json'),
  }
}

• Put your alert manager configuration in alertmanager-secret.yaml after step2

Step 2: Generate YAML files

./build.sh true|false // the variables means that whether include etcd monitor and service of kube-scheduler and kube-controller-manager

Step 3: Apply to kubernetes

kubectl apply -f manifest/prometheus-operator

kubectl apply -f manifest/prometheus

Note: If any resource failed to create, wait for prometheus operator to be up and then apply again.