org.apache.kafka.clients.admin.Admin or KafkaAdmin or org.apache.kafka.clients.consumer.Consumer

[midovlvn]

My application needs:

• list topic partitions
• list topic partitions end offset

In PartitionFinder example one-time consumer is used:
try (Consumer<String, String> consumer = consumerFactory.createConsumer()) {

If want use this pattern beyond application initialization, I have doubts:

• should I use one consumer per operation or reuse consumer across many operations
• if I want reuse, how to care that consumer is not thread safe? Should I protect any access with synchronized or something similar?
• is it a good idea to use Consumer when I do not consume anything? I see that Spring provides KafkaAdmin. Admin sounds better than consumer
• but KafkaAdmin does not expose everything Kafka can provide. It seems Spring Kafka designers decide to expose limited features. Should I raise a ticket: please add partition offsets to KafkaAdmin? Does not look smart that for every feature exposed by Kafka separated ticket to KafkaAdmin must be raised
• Kafka defines org.apache.kafka.clients.admin.Admin

The administrative client for Kafka, which supports managing and inspecting topics, brokers, ...

Sounds good. Rich API comparing to KafkaAdmin. That looks like something I should use. But how arrange Admin in Spring Application?
KafkaAdmin use one-time org.apache.kafka.clients.admin.Admin. Is that reasonable? I found in https://stackoverflow.com/a/57047821/2365727

AdminClient instantiations are expensive as they need to establish a new connection. Add in SSL handshakes and the likes, and you can see why it makes sense to reuse an existing AdminClient. Hence, it's not a bad idea to have multiple threads access the same instance

Considering all above, I think Kafka Admin should have reusable org.apache.kafka.clients.admin.Admin and expose it to allow application ask Kafka for everything

[garyrussell]

If you are using Spring Boot; you can use the auto configured KafkaAdmin and call its getConfigurationProperties() to create your own AdminClient and use it as you wish.

KafkaAdmin cannot (will not) expose all the AdminClient functionality; it is simply for what the framework needs (to provision topics, etc).

As we state in the documentation: https://docs.spring.io/spring-kafka/docs/current/reference/html/#configuring-topics

For more advanced features, you can use the AdminClient directly. The following example shows how to do so:

...

[midovlvn]

Thanks for showing me right documentation chapter.

Should AdminClient be permanent Spring bean or created ad-hoc per request?
I found here transformation from ad-hoc to permanent AdminClient

AdminClient is thread safe: looks like something designed to be permanent...

[midovlvn]

I'm worried.
If I have 20 topics and each ephemeral client costs me 400 ms, it cost me 8 seconds.
I could live 8 seconds shorter, but I can't live with annoying logs, which each new AdminClient outputs (all properties are logged).

In my application I created permanent AdminClient and I use with great satisfaction (for example in PartitonFinder)

AbstractMessageListenerContainer#checkTopics is the only place which use ephemeral AdminClient. If AbstractMessageListenerContainer is left unchanged, I have nothing to improve

[artembilan]

Your concerns is not clear.
The KafkaMessageListenerContainer has this logic:

	if (this.clientIdSuffix == null) { // stand-alone container
			checkTopics();
		}

So, it calls that checkTopics() only if this container is used as an independent.
The @KafkaListener is based on a ConcurrentMessageListenerContainer which does that only once in its start.

Do you mean that you have 20 @KafkaListener methods?
Do you want to reuse a provided KafkaAdmin in all your listener container?
That probably really makes sense and from there we might need to check for provided one or create an ephemeral if missed.

I'm not sure why we create a clone of an admin in that obtainAdmin(), but from here I fee like Spring Boot should be improved to inject its auto-configured KafkaAdmin in its ConcurrentKafkaListenerContainerFactory.

[midovlvn]

I have 4 @KafkaListener methods (not 20) which call 4 times ConcurrentMessageListenerContainer

@Override
protected void doStart() {
	if (!isRunning()) {
		checkTopics();

Before I start to think how to optimize existing code, I would like to know if KafkaAdmin is mandatory bean or not

[garyrussell]

We can add a KafkaAdmin to the container factory (and container). Boot can (eventually) wire one in, but probably not until 3.3 at this point. In the meantime, users can use a container customizer bean to add the admin to the container(s).

The container code should fall back to using an ephemeral admin if no KafkaAdmin is present.

[garyrussell]

Alternatively, the container could look for a single KafkaAdmin bean (via ObjectProvider) in the application context, but we should still add a KafkaAdmin field.