spring-cloud-stream-binder-kafka 4.1.4 breaks natural compatibility of Spring Cloud 2023.0.x with Spring Boot 3.3.x

[Amuerte]

Describe the issue
By forcing spring-kafka version to 3.3.0 to benefit the fix of org.springframework.kafka.support.converter.BatchMessagingMessageConverter, compatibility with Spring Boot 3.3.x is broken as it brings transitive dependencies to Spring 6.2.0 into the classpath.

That fix should be backport to spring-kafka 3.2.x and then used into Spring Cloud Stream 4.1.x.

• https://github.com/spring-cloud/spring-cloud-stream/blob/v4.1.4/binders/kafka-binder/spring-cloud-stream-binder-kafka/pom.xml#L48
• https://github.com/spring-projects/spring-kafka/blob/v3.3.0/build.gradle#L73C2-L73C15

To Reproduce
Steps to reproduce the behavior:

1. Create a Spring Boot 3.3.6 application
2. Add BOM org.springframework.cloud:spring-cloud-dependencies:2023.0.4
3. Add dependency org.springframework.cloud:spring-cloud-starter-stream-kafka (version 4.1.4 from Spring Cloud BOM )
4. Any piece of code or library using classes removed from Spring 6.2.x will fail with ClassNotFoundException. Like spring-vault added with org.springframework.cloud:spring-cloud-starter-vault-config

Wrapped by: j.l.NoClassDefFoundError: org/springframework/util/Base64Utils
	at a.AwsIamAuthentication.<clinit>(AwsIamAuthentication.java:83)
	at c.ClientAuthenticationFactory.awsIamAuthentication(ClientAuthenticationFactory.java:308)
	at c.ClientAuthenticationFactory.createClientAuthentication(ClientAuthenticationFactory.java:126)
	at c.VaultAutoConfiguration.clientAuthentication(VaultAutoConfiguration.java:232)
	at r.DirectMethodHandleAccessor.invoke(Unknown Source)
	at j.l.reflect.Method.invoke(Unknown Source)

Version of the framework

• Spring Boot 3.3.6
• Spring Cloud 2023.0.4
• Spring Cloud Stream Kafka Binder 4.1.4

Expected behavior

• Clean classpath with no Spring 6.2.x libraries.

Screenshots

Additional context
The version have been removed in 4.2.x that is compatible with Spring Boot 3.4.x.

[sobychacko]

@Amuerte Thanks for the report. We will look into it and get back to you soon.

[sobychacko]

@Amuerte It sounds like we should backport that fix to Spring Kafka 3.2.x and then remove manually setting the Spring Kafka version to 3.3.0 in Spring Cloud Stream 4.1.x. Do you think that assumption is correct? If so, we can backport and remove that hard dependency in the binder. Please let us know. Thanks!

[Amuerte]

Hey @sobychacko , that's what I had in mind to fix. Sounds perfect 👍

[sobychacko]

Fixed it via: 124dac5

Also, backported the change mentioned above to Spring Kafka 3.2.x: spring-projects/spring-kafka@c7ee2f4