[RDP-1913]Reduce metadata refresh interval

[tkornai]

Context

According to https://wise.slack.com/archives/G01P8RBLGCC/p1693400446185769

The goal with this change is to provide more time for https://github.com/transferwise/kafka-health-checker to demote unhealthy brokers. We assume that faulty broker is in a zombie state, so it won't return PARTITION_MIGRATED exception that would force the metadata update. Default metadata.max.age.ms is 5 min. Producer’s delivery timeout is 7 min. Let’s say trouble starts at 13:01, health checker reacts to this and demotes the broker at 13:04, if Kafka client’s metadata was refreshed at 13:03, then next metadata refresh will be at 13:08, by that time we would already hit delivery timeout, which would be at 13:08. If producers producing to a changelog topic fail, then it forces the whole Kafka streams task to migrate to another instance, hence the rebalancing. Problem will be that exception will be thrown when produce fails within the delivery timeout, leading the Kafka Streams thread to be moved to another instance. Producer metadata is refreshed periodically, or if there’re certain exceptions returned by the broker.

Checklist

• Change meets or does not compromise the Baseline Security Requirements

Details from ticket: RDP-1913

Reduce metadata refresh interval for Service Kafka clients

When we implement the {{kafka-health-checker}} features for the Service Kafka cluster, we should also make sure that clients set {{metadata.max.age.ms=120000}} (two minutes) to reduce impact time.