Possibility to label or ignore requests in metrics

[jan-stanek]

Describe the feature

Currently there is no possibility to ignore health check metrics in knative metrics. We have multiple knative services with different health check URL's which are called more often than other service endpoints, so the metrics are unusable for us, becase we can't distinguish between real traffic and healthchecks (we use knative as a part of https://github.com/kserve/kserve and want to have just traffic to the model in dashboards).

Related issues: #14581 #15646

Proposed solution: #15653, knative/docs#6177 (health check request have special header and these requests are ignored in metrics), another possibility is to add some label to metrics and filer them in tools like Grafana.

I hope that it's pretty easy and no google doc is needed.

@evankanderson

[dprotaso]

I hope that it's pretty easy and no google doc is needed.

I'd prefer a more flushed out proposal for this feature see: https://github.com/knative/community/blob/main/mechanics/FEATURE-TRACKS.md

The goal of the proposal is to surface questions which Evan has been doing in GitHub Issues: knative/docs#6177 (comment)

Part of this is defining what happens when revisions are scaled to zero etc