Istio Mixer help needed - how to scrape application metrics?

[darrenhaken]

Background
I have a K8 cluster deployed with Istio which is collecting pod metrics from services which are pushed into Prometheus and rendered in Grafana.

We are running a suite of Java services using Spring Boot which can expose metrics in a Prometheus format and covers metrics like JMX, response time etc.

We are using Micrometer with Prometheus as the backend. This exposes a set of the metrics on {application}/actuator/prometheus.

We want to monitor our services and get a holistic view of how they are performing.
I can imagine we will also emit custom metrics from our applications based on the domain.

Problem
I can see two approaches and I'd like some advice on the best approach:

• Envoy scrapes {application}/actuator/prometheus, push the metrics to Mixer which is then scraped by Prometheus. If this is the right way to go how do we do this? Does this matter that not all pods running on the cluster will be JVM and emit these additional metrics?

• Prometheus scrapes the application pods directly. This will result in a duplication of metrics from Envoy/Mixer and the application e.g. response time.