[FEATURE] Add redelivery rate to metrics

[fkollmann]

During operations of LavinMQ, the re-delivery rate is an important indicator, if sth is wrong with the overall client application.

Currently the metrics do not show the re-delivery rate (or count?).

An alternative could be to check if the minimum lavinmq_queue_messages never reaches 0 within 1 hour, but this could easily be true in larger applications, anyway.

Please add it to the /metrics endpoint, which provides the Prometheus export.

For reference, see the RabbitMQ metric: rabbitmq_global_messages_redelivered_total

Best Regards, Felix

[fkollmann]

Example for current export:

# TYPE lavinmq_identity_info gauge
# HELP lavinmq_identity_info System information
lavinmq_identity_info{lavinmq_version="2.1.0", lavinmq_cluster="lavinmq-2"} 1
# TYPE lavinmq_connections_opened_total counter
# HELP lavinmq_connections_opened_total Total number of connections opened
lavinmq_connections_opened_total 956
# TYPE lavinmq_connections_closed_total counter
# HELP lavinmq_connections_closed_total Total number of connections closed or terminated
lavinmq_connections_closed_total 885
# TYPE lavinmq_channels_opened_total counter
# HELP lavinmq_channels_opened_total Total number of channels opened
lavinmq_channels_opened_total 1313
# TYPE lavinmq_channels_closed_total counter
# HELP lavinmq_channels_closed_total Total number of channels closed
lavinmq_channels_closed_total 1244
# TYPE lavinmq_queues_declared_total counter
# HELP lavinmq_queues_declared_total Total number of queues declared
lavinmq_queues_declared_total 68
# TYPE lavinmq_queues_deleted_total counter
# HELP lavinmq_queues_deleted_total Total number of queues deleted
lavinmq_queues_deleted_total 42
# TYPE lavinmq_process_open_fds gauge
# HELP lavinmq_process_open_fds Open file descriptors
lavinmq_process_open_fds 216
# TYPE lavinmq_process_open_tcp_sockets gauge
# HELP lavinmq_process_open_tcp_sockets Open TCP sockets
lavinmq_process_open_tcp_sockets 71
# TYPE lavinmq_process_resident_memory_bytes gauge
# HELP lavinmq_process_resident_memory_bytes Memory used in bytes
lavinmq_process_resident_memory_bytes 365166592
# TYPE lavinmq_disk_space_available_bytes gauge
# HELP lavinmq_disk_space_available_bytes Disk space available in bytes
lavinmq_disk_space_available_bytes 83558301696
# TYPE lavinmq_process_max_fds gauge
# HELP lavinmq_process_max_fds Open file descriptors limit
lavinmq_process_max_fds 1048576
# TYPE lavinmq_resident_memory_limit_bytes gauge
# HELP lavinmq_resident_memory_limit_bytes Memory high watermark in bytes
lavinmq_resident_memory_limit_bytes 2147483648
# TYPE lavinmq_connections gauge
# HELP lavinmq_connections Connections currently open
lavinmq_connections 71
# TYPE lavinmq_channels gauge
# HELP lavinmq_channels Channels currently open
lavinmq_channels 69
# TYPE lavinmq_consumers gauge
# HELP lavinmq_consumers Consumers currently connected
lavinmq_consumers 47
# TYPE lavinmq_queues gauge
# HELP lavinmq_queues Queues available
lavinmq_queues 46
# TYPE lavinmq_queue_messages_ready gauge
# HELP lavinmq_queue_messages_ready Messages ready to be delivered to consumers
lavinmq_queue_messages_ready 0
# TYPE lavinmq_queue_messages_unacked gauge
# HELP lavinmq_queue_messages_unacked Messages delivered to consumers but not yet acknowledged
lavinmq_queue_messages_unacked 0
# TYPE lavinmq_queue_messages gauge
# HELP lavinmq_queue_messages Sum of ready and unacknowledged messages - total queue depth
lavinmq_queue_messages 0
# TYPE lavinmq_uptime counter
# HELP lavinmq_uptime Server uptime in seconds
lavinmq_uptime 323409
# TYPE lavinmq_cpu_system_time_total gauge
# HELP lavinmq_cpu_system_time_total Total CPU system time
lavinmq_cpu_system_time_total 50187270
# TYPE lavinmq_cpu_user_time_total gauge
# HELP lavinmq_cpu_user_time_total Total CPU user time
lavinmq_cpu_user_time_total 21170274
# TYPE lavinmq_stats_collection_duration_seconds_total gauge
# HELP lavinmq_stats_collection_duration_seconds_total Total time it takes to collect metrics (stats_loop)
lavinmq_stats_collection_duration_seconds_total 0.000484732
# TYPE lavinmq_stats_rates_collection_duration_seconds gauge
# HELP lavinmq_stats_rates_collection_duration_seconds Time it takes to update stats rates (update_stats_rates)
lavinmq_stats_rates_collection_duration_seconds 0.000234087
# TYPE lavinmq_stats_system_collection_duration_seconds gauge
# HELP lavinmq_stats_system_collection_duration_seconds Time it takes to collect system metrics
lavinmq_stats_system_collection_duration_seconds 0.000250375
# TYPE lavinmq_total_connected_followers gauge
# HELP lavinmq_total_connected_followers Amount of follower nodes connected
lavinmq_total_connected_followers 2
# TYPE lavinmq_follower_lag_in_bytes gauge
# HELP lavinmq_follower_lag_in_bytes Bytes that hasn't been synchronized with the follower yet
lavinmq_follower_lag_in_bytes{id="rtpypj"} -1665
# TYPE lavinmq_follower_lag_in_bytes gauge
# HELP lavinmq_follower_lag_in_bytes Bytes that hasn't been synchronized with the follower yet
lavinmq_follower_lag_in_bytes{id="4cdik5"} -1665
# TYPE telemetry_scrape_duration_seconds gauge
# HELP telemetry_scrape_duration_seconds Duration for metrics collection in seconds
telemetry_scrape_duration_seconds 0.00053323
# TYPE telemetry_scrape_mem gauge
# HELP telemetry_scrape_mem Memory used for metrics collections in bytes
telemetry_scrape_mem 2496