benchmark: part of threads never stopped when run with memtier_benchmark #33
Comments
|
I'm unable to reproduce this on my laptop. The fact that memtier_benchmark fails to finish it's run is odd. Even if the server was not responding for some reason, it would ideally handle that and finish the run. Can you tell us more about the environment you're testing in? Might be helpful to know what OS/Version/Kernel/... and what hardware. I use rpc-perf for my testing: https://github.com/iopsystems/rpc-perf |
|
@brayniac Thanks for your quick answer! Here is my environment info: hw: Intel Alder Lake The output from memtier_benchmark side is as follows. There is one thread never stop until you manually kill it. Please be sure to set config/segcache with 6 worker threads or higher, and run memtier_benchmark with 20 threads or higher. Additionally, I've tried ubuntu 20.04 with kernel 5.4, the problem is also able to be reproduced. |
|
One more thing that the performance test was previously running on host directly, both pelikan and memtier_benchmark is installed on host. But when I run pelikan/pelikan_segcache_rs in docker image while memtier_benchmark is still on host, the issue disappears. |
|
@bjzhjing does this happen every time? |
|
Is it possible you're running out of available open file handles? It's
strange that memtier_benchmark would hang like that. But it's hard for me
to believe this is a Pelikan issue and not just a system configuration and
test setup issue. I think memtier_benchmark needs to be made more resilient
too.
For the record, I tested with even high number of Segcache workers and
memtier_benchmark threads than you and the test completed just fine. That's
on MacOS though. But we have run Segcache on production systems handling
10k concurrent connections without any observed issues.
You probably want to check `ulimit -n` with the numbers you provided, I
suspect that value is going to be too low. Default is 50 clients per thread
times 20 memtier_benchmark threads is 1000.
…On Thu, Feb 2, 2023, 4:42 AM bjzhjing ***@***.***> wrote:
One more thing that the performance test was previously running on host
directly, both pelikan and memtier_benchmark is installed on host. But when
I run pelikan/pelikan_segcache_rs in docker image while memtier_benchmark
is still on host, the issue disappears.
—
Reply to this email directly, view it on GitHub
<#33 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACP6HFYRTWIIUNN3S7IQKLLWVOTSVANCNFSM6AAAAAAUNKQYUY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@bjzhjing what specific git SHA are you using? I am going to try to reproduce this as well |
|
@bjzhjing I am quite certain at this point that @brayniac hit the nail on the head. I dropped my ulimit to 900 I haven't bothered to see if it's required that the ulimit be lower than necessary for both segcache and memtier in order to reproduce the hang, but I can't reproduce it without segcache being given too low of a ulimit. |
We are measuring pelikan performance with memtier_benchmark, but encounter the following issue.
Problem description
Run performance test with memtier_benchmark, when --test-time is reached, the output from memtier_benchmark indicates some threads never stop, and it has no chance to print the statistical data finally.
How to re-produce
The problem happens with the following settings. If memtier_benchmark threads < 20, there is no such problem.
With some debugging and code analysis, we see that those threads are looping in epoll_pwait(), waiting for events from pelikan. The soft stack:
client_group::run() (memtier_benchmark/client.c) -> event_base_dispatch() (libevent/event.c) -> event_base_loop() -> epoll_dispatch() (libevent/epoll.c) -> epoll_pwait2() (kernel systemcall)Question
The text was updated successfully, but these errors were encountered: