Unified HTTP Client Configuration for All Providers

[RikJux]

Hi everyone,
I am currently working on a project that leverages multiple providers, each with its own starter module, such as GEMINI, OPENAI, AZURE, and BEDROCK. Each module contains a chatModel that follows the same interface but is specific to each provider. Each chatModel relies on different HTTP clients to make the actual requests.

For example:

• GEMINI uses VertexAI
• OPENAI uses RestClient.Builder and WebClient.Builder
• AZURE uses OpenAIClient
• BEDROCK uses Anthropic3ChatBedrockApi

The OPENAI implementation appears to be the most integrated with the Spring ecosystem, as it leverages Spring's RestClient and WebClient, which are part of the Spring Framework. In contrast, the other providers rely on their own custom HTTP client implementations, which are specific to each provider's SDK.

Problem Statement

I want to have a unified way to define HTTP client configurations and use Rest and WebClient for all providers. This would simplify the configuration and make it easier to manage multiple providers in a single project. Here is an example of what I have in mind:

public static WebClient createWebClient(ClientConfig config) {
    WebClient.Builder builder = WebClient.builder();
    PoolingAsyncClientConnectionManager connectionManager = PoolingAsyncClientConnectionManagerBuilder.create().build();
    connectionManager.setMaxTotal(config.getMaxTotal());
    connectionManager.setDefaultMaxPerRoute(config.getDefaultMaxPerRoute());
    CloseableHttpAsyncClient httpAsyncClient = HttpAsyncClients.custom()
        .setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_2)
        .setIOReactorConfig(ioReactorConfig(config))
        .setDefaultRequestConfig(requestConfig(config))
        .setConnectionManager(connectionManager)
        .build();
    ClientHttpConnector connector = new HttpComponentsClientHttpConnector(httpAsyncClient);
    return builder.clientConnector(connector).build();
}
 
private static IOReactorConfig ioReactorConfig(ClientConfig config) {
    return IOReactorConfig.custom()
        .setConnectTimeout(config.getConnectTimeout())
        .setSoTimeout(config.getSocketTimeout())
        .build();
}
 
private static RequestConfig requestConfig(ClientConfig config) {
    return RequestConfig.custom()
        .setConnectionRequestTimeout(config.getConnectionRequestTimeout())
        .setConnectTimeout(config.getConnectTimeout())
        .setSocketTimeout(config.getSocketTimeout())
        .build();
}

In this example, ClientConfig is a custom configuration class that holds various HTTP client settings such as timeouts and connection pool sizes. This method creates a WebClient instance with a custom ClientHttpConnector that uses Apache HttpComponents.

Then, you can reuse this method to inject different versions of the WebClient like:

@Configuration
public class OpenAIConfig {
    @Bean(name = "openAiClientConfig")
    @ConfigurationProperties(prefix = "openai.webclient")
    public ClientConfig webClientConfig() {
        return new ClientConfig();
    }
 
    @Bean(name = "openAiWebClient")
    @ConditionalOnMissingBean(name = "openAiWebClient")
    public WebClient openAiWebClient(@Qualifier("openAiClientConfig") ClientConfig config) {
        return WebClientFactory.createWebClient(config);
    }
}

Questions

1. Is there a plan to address the unification of HTTP client configurations across different providers?
2. Would it be possible to refactor the existing modules to use a common HTTP client configuration method like the one proposed above?
3. Are there any known challenges or limitations in implementing a unified HTTP client configuration for all providers?
4. Would the community be open to contributions that aim to standardize the HTTP client configurations?

I believe a unified way to make HTTP calls would be highly beneficial for projects that require more than one provider. Looking forward to your thoughts and suggestions!

[ThomasVitale]

You can follow the conversation and progress about the HTTP Client configuration enhancements in #512