http-client-benchmark with Apache SDK client added #6256

joviegas · 2025-07-14T19:19:30Z

Motivation and Context

Add JMH benchmark module to compare performance between Apache HttpClient 4.x and 5.x implementations in the AWS SDK. This will help us measure the impact of upgrading to Apache HttpClient 5.x and evaluate virtual threads performance.

Modifications

Added new http-client-benchmarks module with JMH framework
Created benchmark classes for Apache4 and Apache5 HTTP clients
Implemented S3 GET/PUT operations for single and multi-threaded scenarios
Added CloudWatch metrics publishing for performance tracking
Created unified benchmark runner to compare all client implementations
Configured Maven build with proper JMH annotation processing

Testing

Verified benchmark compilation and JAR generation
Tested individual benchmark execution via JMH
Confirmed CloudWatch metrics publishing functionality
Validated S3 operations setup and cleanup

Screenshots (if appropriate)

Benchmark complete! CloudWatch metrics published with run ID: 2025-07-14T23:30:27.276404225Z

============================================================================================================================================
BENCHMARK RESULTS SUMMARY
============================================================================================================================================
Client Type          | Benchmark                                          | Throughput      | Avg Latency     | P99 Latency     | Threads   
--------------------------------------------------------------------------------------------------------------------------------------------
Apache4              | multiThreadedGet                                   |         22.87/s |         43.73 ms |         65.60 ms |         10
Apache4              | multiThreadedPut                                   |          8.31/s |        120.35 ms |        180.52 ms |         10
Apache4              | simpleGet                                          |         34.15/s |         29.28 ms |         43.93 ms |          1
Apache4              | simplePut                                          |         13.81/s |         72.40 ms |        108.59 ms |          1
Apache5-Platform     | multiThreadedGet                                   |         22.90/s |         43.67 ms |         65.50 ms |         10
Apache5-Platform     | multiThreadedPut                                   |          8.04/s |        124.32 ms |        186.48 ms |         10
Apache5-Platform     | simpleGet                                          |         33.75/s |         29.63 ms |         44.45 ms |          1
Apache5-Platform     | simplePut                                          |         13.29/s |         75.23 ms |        112.84 ms |          1
Apache5-Virtual      | multiThreadedGet                                   |         22.02/s |         45.41 ms |         68.12 ms |         10
Apache5-Virtual      | multiThreadedPut                                   |          9.45/s |        105.79 ms |        158.68 ms |         10
Apache5-Virtual      | simpleGet                                          |         34.08/s |         29.35 ms |         44.02 ms |          1
Apache5-Virtual      | simplePut                                          |         13.15/s |         76.06 ms |        114.09 ms |          1
============================================================================================================================================
Total benchmark results: 12
============================================================================================================================================

PERFORMANCE COMPARISON (Apache5 vs Apache4):
================================================================================

multiThreadedPut:
--------------------------------------------------------------------------------
  Apache5-Platform    : -3.2% throughput, -3.3% latency improvement
  Apache5-Virtual     : +13.8% throughput, +12.1% latency improvement

simpleGet:
--------------------------------------------------------------------------------
  Apache5-Platform    : -1.2% throughput, -1.2% latency improvement
  Apache5-Virtual     : -0.2% throughput, -0.2% latency improvement

multiThreadedGet:
--------------------------------------------------------------------------------
  Apache5-Platform    : +0.1% throughput, +0.1% latency improvement
  Apache5-Virtual     : -3.7% throughput, -3.8% latency improvement

simplePut:
--------------------------------------------------------------------------------
  Apache5-Platform    : -3.8% throughput, -3.9% latency improvement
  Apache5-Virtual     : -4.8% throughput, -5.1% latency improvement

================================================================================

Java 17 (Virtual thread skipped)


============================================================================================================================================
BENCHMARK RESULTS SUMMARY
============================================================================================================================================
Client Type          | Benchmark                                          | Throughput      | Avg Latency     | P99 Latency     | Threads   
--------------------------------------------------------------------------------------------------------------------------------------------
Apache4              | multiThreadedGet                                   |          2.28/s |        437.73 ms |        656.60 ms |         10
Apache4              | multiThreadedPut                                   |          2.33/s |        429.30 ms |        643.95 ms |         10
Apache4              | simpleGet                                          |         16.96/s |         58.96 ms |         88.44 ms |          1
Apache4              | simplePut                                          |          7.26/s |        137.77 ms |        206.66 ms |          1
Apache5              | multiThreadedGet                                   |          4.57/s |        218.95 ms |        328.43 ms |         10
Apache5              | multiThreadedPut                                   |          4.31/s |        232.23 ms |        348.34 ms |         10
Apache5              | simpleGet                                          |         15.53/s |         64.38 ms |         96.57 ms |          1
Apache5              | simplePut                                          |          7.81/s |        127.98 ms |        191.97 ms |          1
============================================================================================================================================
Total benchmark results: 8
============================================================================================================================================

PERFORMANCE COMPARISON (Apache5 vs Apache4):
================================================================================

multiThreadedPut:
--------------------------------------------------------------------------------
  Apache5             : +84.9% throughput, +45.9% latency improvement

simpleGet:
--------------------------------------------------------------------------------
  Apache5             : -8.4% throughput, -9.2% latency improvement

multiThreadedGet:
--------------------------------------------------------------------------------
  Apache5             : +99.9% throughput, +50.0% latency improvement

simplePut:
--------------------------------------------------------------------------------
  Apache5             : +7.7% throughput, +7.1% latency improvement

================================================================================

Java 21 With Virtual threads

============================================================================================================================================                                                                                                     
BENCHMARK RESULTS SUMMARY                                                                                                                                                                                                                        
============================================================================================================================================                                                                                                     
Client Type          | Benchmark                                          | Throughput      | Avg Latency     | P99 Latency     | Threads                                                                                                        
--------------------------------------------------------------------------------------------------------------------------------------------                                                                                                     
Apache4              | multiThreadedGet                                   |          0.42/s |       2358.29 ms |       3537.43 ms |         10                                                                                                   
Apache4              | multiThreadedPut                                   |          0.38/s |       2637.07 ms |       3955.61 ms |         10                                                                                                   
Apache4              | simpleGet                                          |          1.58/s |        630.98 ms |        946.47 ms |          1                                                                                                   
Apache4              | simplePut                                          |          3.35/s |        298.71 ms |        448.07 ms |          1                                                                                                   
Apache5              | multiThreadedGet                                   |          0.72/s |       1382.25 ms |       2073.37 ms |         10                                                                                                   
Apache5              | multiThreadedPut                                   |          0.70/s |       1438.79 ms |       2158.18 ms |         10                                                                                                   
Apache5              | simpleGet                                          |          1.55/s |        645.84 ms |        968.76 ms |          1                                                                                                   
Apache5              | simplePut                                          |          3.57/s |        280.41 ms |        420.61 ms |          1                                                                                                   
Apache5-Virtual      | multiThreadedGet                                   |          0.80/s |       1243.96 ms |       1865.94 ms |         10                                                                                                   
Apache5-Virtual      | multiThreadedPut                                   |          0.75/s |       1326.85 ms |       1990.27 ms |         10                                                                                                   
Apache5-Virtual      | simpleGet                                          |          1.58/s |        632.60 ms |        948.90 ms |          1                                                                                                   
Apache5-Virtual      | simplePut                                          |          2.83/s |        352.90 ms |        529.35 ms |          1                                                                                                   
============================================================================================================================================                                                                                                     
Total benchmark results: 12                                                                                                                                                                                                                      
============================================================================================================================================                                                                                                     
                                                                                                                                                                                                                                                 
PERFORMANCE COMPARISON (Apache5 vs Apache4):                                                                                                                                                                                                     
================================================================================                                                                                                                                                                 
                                                                                                                                                                                                                                                 
multiThreadedPut:                                                                                                                                                                                                                                
--------------------------------------------------------------------------------                                                                                                                                                                 
  Apache5             : +83.3% throughput, +45.4% latency improvement                                                                                                                                                                            
  Apache5-Virtual     : +98.7% throughput, +49.7% latency improvement                                                                                                                                                                            
                                                                                                                                                                                                                                                 
simpleGet:                                                                                                                                                                                                                                       
--------------------------------------------------------------------------------
  Apache5             : -2.3% throughput, -2.4% latency improvement
  Apache5-Virtual     : -0.3% throughput, -0.3% latency improvement

multiThreadedGet:
--------------------------------------------------------------------------------
  Apache5             : +70.6% throughput, +41.4% latency improvement
  Apache5-Virtual     : +89.6% throughput, +47.3% latency improvement

simplePut:
--------------------------------------------------------------------------------
  Apache5             : +6.5% throughput, +6.1% latency improvement
  Apache5-Virtual     : -15.4% throughput, -18.1% latency improvement

================================================================================

License

I confirm that this pull request can be released under the Apache 2 license

* Add initial module for Apache5x for seting up package * Add based on new module checklist

…pache5SDKHttpClient (#6088) * Add initial module for Apache5x for seting up package * Add based on new module checklist * Baseline all the classes from Apache4 SDK client to the new Apache5 module

…aring Checkstyles and spotbug issues (#6100) * Phase 2 , getting Apache 5 compilation and Junit ready along with clearing Checkstyles and spotbug issues * Handle comments from review * Handle comments from Zoe

…reamRequestEntity repeatability (#6132) * Fix HTTP authentication retry failures by improving RepeatableInputStreamRequestEntity repeatability * Upated test cases * Handled comments

This reverts commit 64f7d33.

* Fix architecture test failures for apache5.x * Checkstyle issues

…ientConnectionManager for Connection Manager (#6147) * Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager

…6154) * Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager * Fix stream reset failure in RepeatableInputStreamRequestEntity by storing content reference to avoid multiple ContentStreamProvider.newStream() calls that cause IOException when retrying requests with non-resettable streams * writeTo_ConcurrentWrites_HandlesCorrectly no longer needed since even Apache 4.x doesnot suports this

…zil.json (#6191) * Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager * Fix stream reset failure in RepeatableInputStreamRequestEntity by storing content reference to avoid multiple ContentStreamProvider.newStream() calls that cause IOException when retrying requests with non-resettable streams * writeTo_ConcurrentWrites_HandlesCorrectly no longer needed since even Apache 4.x doesnot suports this * Fix connectionPoolingWorks by setting skipping setConnectionTimeToLive is value is set to 0 since 0 is treated as Infinite timeToLive in Sdk and Apache 4.x but treated as immediate closeConnection in apache 5.x * disableAutomaticRetries in Apache 5.x since SDK handles retries , also define Apache5 dependencies in .brazil.json * Added Test case for Async , handled review ocmments

…-v2 into feature/master/apache5x

* Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager * Fix stream reset failure in RepeatableInputStreamRequestEntity by storing content reference to avoid multiple ContentStreamProvider.newStream() calls that cause IOException when retrying requests with non-resettable streams * writeTo_ConcurrentWrites_HandlesCorrectly no longer needed since even Apache 4.x doesnot suports this * Fix connectionPoolingWorks by setting skipping setConnectionTimeToLive is value is set to 0 since 0 is treated as Infinite timeToLive in Sdk and Apache 4.x but treated as immediate closeConnection in apache 5.x * disableAutomaticRetries in Apache 5.x since SDK handles retries , also define Apache5 dependencies in .brazil.json * Added Test case for Async , handled review ocmments * Donot do buffer the response using BufferedHttpEntity since it might cause memory issue, this behaviour is same as Apache4.x * Fix compilation issues * Fix checkstyle issues * Remove test which are specific to apache http

* Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager * Fix stream reset failure in RepeatableInputStreamRequestEntity by storing content reference to avoid multiple ContentStreamProvider.newStream() calls that cause IOException when retrying requests with non-resettable streams * writeTo_ConcurrentWrites_HandlesCorrectly no longer needed since even Apache 4.x doesnot suports this * Fix connectionPoolingWorks by setting skipping setConnectionTimeToLive is value is set to 0 since 0 is treated as Infinite timeToLive in Sdk and Apache 4.x but treated as immediate closeConnection in apache 5.x * disableAutomaticRetries in Apache 5.x since SDK handles retries , also define Apache5 dependencies in .brazil.json * Added Test case for Async , handled review ocmments * Donot do buffer the response using BufferedHttpEntity since it might cause memory issue, this behaviour is same as Apache4.x * Fix compilation issues * Fix checkstyle issues * Remove test which are specific to apache http * Add benchmark for Apache5 and add Streaming Api test cases

… alternatives (#6211) * Clean up unused APIs and add test to make sure it can be handled with alternatives * Added NTCredentials to keep backward compatibilty with Apache4.x

… version (#6214) * Fix architecture test failures for apache5.x * Checkstyle issues * Update to use PoolingHttpClientConnectionManager class reference that is implementation of HttpClientConnectionManager * Fix stream reset failure in RepeatableInputStreamRequestEntity by storing content reference to avoid multiple ContentStreamProvider.newStream() calls that cause IOException when retrying requests with non-resettable streams * writeTo_ConcurrentWrites_HandlesCorrectly no longer needed since even Apache 4.x doesnot suports this * Fix connectionPoolingWorks by setting skipping setConnectionTimeToLive is value is set to 0 since 0 is treated as Infinite timeToLive in Sdk and Apache 4.x but treated as immediate closeConnection in apache 5.x * disableAutomaticRetries in Apache 5.x since SDK handles retries , also define Apache5 dependencies in .brazil.json * Added Test case for Async , handled review ocmments * Donot do buffer the response using BufferedHttpEntity since it might cause memory issue, this behaviour is same as Apache4.x * Fix compilation issues * Fix checkstyle issues * Remove test which are specific to apache http * Add benchmark for Apache5 and add Streaming Api test cases * Update Apache5 to 5.5

…-review

…nce and one sent to cw

…-review

sonarqubecloud · 2025-07-16T02:03:02Z

Quality Gate failed

Failed conditions
2 Security Hotspots
0.2% Coverage on New Code (required ≥ 80%)
12.3% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

joviegas and others added 30 commits April 30, 2025 08:27

Add initial empty module for Apache5x for seting up package (#6075)

f3fc974

* Add initial module for Apache5x for seting up package * Add based on new module checklist

Baseline or Copy all the ApacheSDKHttpClient classes to newly added A…

5092cc4

…pache5SDKHttpClient (#6088) * Add initial module for Apache5x for seting up package * Add based on new module checklist * Baseline all the classes from Apache4 SDK client to the new Apache5 module

Phase 2 , getting Apache 5 compilation and Junit ready along with cle…

6e9939f

…aring Checkstyles and spotbug issues (#6100) * Phase 2 , getting Apache 5 compilation and Junit ready along with clearing Checkstyles and spotbug issues * Handle comments from review * Handle comments from Zoe

Merge branch 'master' into feature/master/apache5x

4dda3f7

Update the snap shot

3053d09

Fix HTTP authentication retry failures by improving RepeatableInputSt…

985ca35

…reamRequestEntity repeatability (#6132) * Fix HTTP authentication retry failures by improving RepeatableInputStreamRequestEntity repeatability * Upated test cases * Handled comments

Merge branch 'master' into feature/master/apache5x

b97c9da

Updated snap shot after merge from master

64f7d33

Revert "Updated snap shot after merge from master"

24438e8

This reverts commit 64f7d33.

Updated snap shot after merge from master

62150d6

Fix architecture test failures for apache5.x (#6140)

fbfb70f

* Fix architecture test failures for apache5.x * Checkstyle issues

Merge branch 'master' into feature/master/apache5x

3c4433c

Updated snap shot after merge from master

6d8b5f4

Merge branch 'master' into feature/master/apache5x

0bdf878

Merge branch 'master' into feature/master/apache5x

4671831

Merge branch 'master' into feature/master/apache5x

2f72282

Merge PR#6165 #6165

79f6f0f

Merge branch 'master' into feature/master/apache5x

716a6db

Update snapshots

09e8ee4

Merge branch 'feature/master/apache5x' of github.com:aws/aws-sdk-java…

ad30c25

…-v2 into feature/master/apache5x

Merge branch 'master' into feature/master/apache5x

959605c

Merge from master

a959581

Clean up unused APIs and add test to make sure it can be handled with…

3cc94bb

… alternatives (#6211) * Clean up unused APIs and add test to make sure it can be handled with alternatives * Added NTCredentials to keep backward compatibilty with Apache4.x

Merge branch 'master' into feature/master/apache5x

45f630d

joviegas added 7 commits July 11, 2025 22:24

Removed some dependencies

8e7ce75

Merge branch 'master' into feature/master/apache5x

ee93292

updated performance comparison logic

5061901

updated readme

4d88a3e

Merge branch 'feature/master/apache5x' into joviegas/post-surface-api…

0dd8786

…-review

updated warmup time and iterations

6dda6bd

remove performance comparison since we already have printed performam…

3f4ba25

…nce and one sent to cw

joviegas requested a review from a team as a code owner July 14, 2025 19:19

Review from github

4833db9

joviegas force-pushed the joviegas/post-surface-api-review branch from a1d775b to 4833db9 Compare July 14, 2025 20:17

joviegas added 6 commits July 14, 2025 14:30

Review comment

061b5fb

updatded prints

59ac21f

Handled Review comments

f26f7bb

Merge branch 'master' into feature/master/apache5x

dfcfdcb

Added performance comparison for Apache

7d27844

updated README

305c62b

joviegas changed the title ~~Joviegas/post surface api review~~ http-client-benchmark with Apache SDK client added Jul 15, 2025

joviegas added 5 commits July 15, 2025 11:28

remove preview buid

1acd510

Merge branch 'feature/master/apache5x' into joviegas/post-surface-api…

5d78ede

…-review

added changed to skpi http client benchmark

807ad9f

updated the snapshot version

d382185

updated to run on jdk8

78ac2e2

joviegas changed the base branch from feature/master/apache5x to master July 15, 2025 20:58

joviegas added 6 commits July 15, 2025 14:04

merge from master

9d1255b

merge from master

422d197

updated README.md

832b52c

fixed checkbuild issues

29ab8d9

fixed checksyle issues

99e697e

fixed issue where secondary benchmarks were not found

b536571

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

http-client-benchmark with Apache SDK client added #6256

http-client-benchmark with Apache SDK client added #6256

Uh oh!

joviegas commented Jul 14, 2025 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Jul 16, 2025

Uh oh!

Uh oh!

http-client-benchmark with Apache SDK client added #6256

Are you sure you want to change the base?

http-client-benchmark with Apache SDK client added #6256

Uh oh!

Conversation

joviegas commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Java 17 (Virtual thread skipped)

Java 21 With Virtual threads

License

Uh oh!

sonarqubecloud bot commented Jul 16, 2025

Quality Gate failed

Uh oh!

Uh oh!

joviegas commented Jul 14, 2025 •

edited

Loading