- Added configs to deploy Prometheus, Prometheus Operator & Grafana w…

…ith out of the box defaults that collect k8s, istio & node metrics. (knative#189)

- Added a sample app to demonstrate adding app specific metrics and how they can be collected.

DEVELOPMENT.md

@@ -112,3 +112,53 @@ Running tests as you make changes to the code-base is pretty simple. Just run th
 ```shell
 ./testing/unit_tests.sh
 ```
+
+## Collecting and generating telemetry
+
+To see k8s, Istio & node metrics, deploy Prometheus & service monitors:
+
+```shell
+bazel run config/prometheus:everything.create
+```
+
+To debug Prometheus installation and configuration, enable Prometheus UI on a public IP and browse to the IP.
+Keep in mind that running below creates a public IP and and all the data in Prometheus is exposed to outside.
+DO NOT run this in production environments or environments with sensitive metrics data.
+
+```shell
+bazel run config/prometheus:everything-public.create
+```
+
+To see Prometheus UI, get load balancer service IP:
+
+```shell
+# Put the Ingress IP into an environment variable.
+$ kubectl get service prometheus-system-public -o jsonpath="{.status.loadBalancer.ingress[*]['ip']}"
+```
+
+And browse to <IP_ADDRESS>:30800.
+
+To see dashboards, deploy Grafana and expose it on a public IP:
+```shell
+bazel run config/grafana:everything.create
+bazel run config/grafana:everything-public.create
+```
+
+Configuration above installs Grafana with a hard coded admin username and password:
+
+```shell
+Username: admin
+Password: admin
+```
+
+This default configuration should only be used in development environments with no sensitive metric data.
+To login to Grafana UI, get load balancer service IP: 
+
+```shell
+# Put the Ingress IP into an environment variable.
+$ kubectl get service grafana-public -o jsonpath="{.status.loadBalancer.ingress[*]['ip']}"
+```
+
+And browse to <IP_ADDRESS>:30802.
+
+To generate metrics, see [Telemetry Sample](./sample/telemetrysample/README.md)

config/grafana/BUILD

@@ -0,0 +1,45 @@
+load("@k8s_object//:defaults.bzl", "k8s_object")
+
+k8s_object(
+    name = "grafana-deployment",
+    template = "deployment.yaml",
+)
+
+k8s_object(
+    name = "grafana-service-public",
+    template = "service-public.yaml",
+)
+
+k8s_object(
+    name = "grafana-dashboards",
+    template = "dashboards.yaml",
+)
+
+k8s_object(
+    name = "grafana-dashboard-definitions",
+    template = "dashboard-definitions.yaml",
+)
+
+k8s_object(
+    name = "grafana-datasources",
+    template = "datasources.yaml",
+)
+
+load("@io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
+
+k8s_objects(
+    name = "everything",
+    objects = [
+       ":grafana-datasources",
+       ":grafana-dashboards",
+       ":grafana-dashboard-definitions",
+       ":grafana-deployment",
+    ],
+)
+
+k8s_objects(
+    name = "everything-public",
+    objects = [
+        ":grafana-service-public",
+    ],
+)