Exclude health checks from telemetry

[jsw]

Feature request

Describe the feature:

Add ability to exclude GLB and kubernetes health checks from telemetry.
The paths could either be inferred from the environment and/or allow users to specify a set of paths or path expressions.

[jsw]

On further thought, this is probably better done later in the flow, such as in the Grafana charts.

[sakshigoel12]

@douglas-reid can you take a look at this request

[douglas-reid]

It should be possible to use Mixer config and various attributes to handle (at least some of ) this via config today (depending on deployment).

match: request.path != "/healthz"
...

It would be nice to have a way to annotate, in service config, paths not meant for monitoring and have that pass down into Envoy filter configuration.

[catalinpan]

I'm trying to exclude the health checks from tracing using the example from the documentation but it doesn't really work for me https://istio.io/help/faq/telemetry/#controlling-what-the-sidecar-reports
Can I have some help please on where this flag should be set?

[douglas-reid]

@catalinpan is there a given path (or pattern of paths) that you use for health checking? the example on the FAQ uses the source.name attribute, but that could easily be switched to something like request.path (as in my example above).

[catalinpan]

@douglas-reid thanks for the reply. The path is FQDN/_health but I'm not sure where I should add that flag. Is part of a rule similar to this example https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/policy/mixer-rule-deny-label.yaml or should be set somewhere else?
When I use source.name as a part of a rule (like on the above mixer example) it blocks everything from accessing the health check endpoint.

[douglas-reid]

@catalinpan are you using Mixer for trace span generation, or do you have the proxies (envoy) configured for tracing in your install?

[catalinpan]

@douglas-reid if you are referring on how the http headers are set on each request, the apps are setting the tracing http headers based on the istio docs from here https://istio.io/docs/tasks/telemetry/distributed-tracing/#understanding-what-happened
I have tested today also this without any luck #335 (comment)

Is there any sample configuration I can adapt to use the match from the istio documentation?

[douglas-reid]

@catalinpan i'm wondering if you are using Mixer for tracing (with explicit config for tracespans) or if you are using Envoy. If you are using Envoy (the default case), then there are no match clauses etc that you can use, as Mixer isn't in the path.

[catalinpan]

@douglas-reid it looks like we are using the default Envoy. I will check what options there are for this. Thanks for all your help!