How to create slo for kafka latency?

[ashishvaishno]

Hello,

I wanted to create an slo for kafka for the below metrics kafka_network_requestmetrics_totaltimesms_produce_99th (https://docs.confluent.io/platform/current/kafka/monitoring.html#totaltimems)

I am using promethues with thanos where enabling experimental histogram is not an option as thanos compactor is yet to release downsampling of histogram thanos-io/thanos#6350
So usage of latencyNative in my situation is out of question. What would be a better way to handle this use case?

Regards
Ashish

[metalmatze]

Can you send more information on the kafka_network_requestmetrics_totaltimesms_produce_99th metric? Then we might be able to help.
You don't need native histograms for Pyrra and latency SLOs. The classic histograms are totally fine too.

What would be the time window and the latency objectives?

[ashishvaishno]

@metalmatze Sure :)
The metrics is mentioned here -> https://docs.confluent.io/platform/current/kafka/monitoring.html#totaltimems
I wanted to create the slo for all 3 type of request i.e Produce|FetchConsumer|FetchFollower
I would like to use it to see if the 99th Percentile is above 300ms for the latency SLO. The time window will be for a month

I started of with the below SLO

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: kafka-network-request-metrics-producer-99-latency
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  target: '95'
  window: 30d
  alerting:
    disabled: true
  indicator:
    ratio:
      errors:
        metric: kafka_network_requestmetrics_totaltimesms_produce_99th > 300
      total:
        metric: kafka_network_requestmetrics_totaltimesms_produce_99th

But that didnt work. Error : failed to get objective: ratio total metric is not a VectorSelector

[ashishvaishno]

@metalmatze do you need any other information?

[metalmatze]

I need time and energy between SRECon, KubeCon, and having a cold. A proper reply might need some time still. Sorry.

[ashishvaishno]

@metalmatze No worries :) All the best for the conferences and wish you a healthy recovery 💪