Performance Issue

[asal-hesehus]

Elastic.Clients.Elasticsearch version: 8.17.1

Elasticsearch version:8.16

.NET runtime version:.NET core 9.0

Description of the problem including expected versus actual behavior:
We du performance test for searches in elastic for each of our versions. When upgraded Elastic.Clients.Elasticsearch from 8.15.10 to 8.17.1 We noticed that our queries was 4-6 times slower and that CPU usage was also about 3 times slower.

Steps to reproduce:
We do not have any steps to reproduce yet. But will provide them if needed.

Expected behavior
No performance dip.

[flobernd]

Hi @asal-hesehus,

We noticed that our queries was 4-6 times slower and that CPU usage was also about 3 times slower.

This statement is very confusing to me.

Is that the server or client CPU usage? How do you measure it? You mention the CPU usage is "slower". Does that mean it's higher or less than it used to be?

Is that the time a search query request takes when executing it while using the client? How do you measure that? Did you try to run the same query in Kibana or curl?

There are no fundamental changes in the client from 8.15.10 to 8.17.1 which could explain such performance drop.

I'm sorry, but this issue requires way more context to find the bottleneck. It could be anything from the client (unlikely), changes to the code that are not related to the client update, network, server, etc.

[asal-hesehus]

Hi sorry for the a bit complete support issue.

It is the client side that we noticed the higher CPU usage and the increased response times. The performance test was our servers complete time to process the responses.

Downgrading the client from 8.17.1 to 8.15.10 returned our response times to the "Normal". So we have isolated that the client is the issue but not what in client, if it serialization or if the queries that the client outputs have changes and that leads to worse performance.

I will try to dig a bit deeper and see if we can find the issue.

[mvkhesehus]

Hi - I work in the same team as @asal-hesehus

So i'll try to provide more info from what we saw in APM and from our load tests. The only change in the in the last deployment is the downgrade to version 8.15.10 nothing else. APM metrics are aggregated across several production deployments, so it indicates its a general issue.

From our load test where the only change is downgrading from 8.17.1 to 8.15.10. Also shows a significant change in performance. Both response times and number of request per second is affected.

[flobernd]

Hi @asal-hesehus @mvkhesehus

I will try to dig a bit deeper and see if we can find the issue.

Happy to help you finding the problem! For this I would need a standalone project that includes only your load test code and a bootstrapping routine that creates the required indices with proper mappings, some data seeding, etc.