SYSDB: perf improvements in sysdb_add_group_member_overrides(), part 2

[alexey-tikhonov]

• Most impactful patch is "Replaced sysdb_search_entry() with sysdb_cache_search_entry()
  to avoid sysdb_merge_msg_list_ts_attrs()"
• Second impactful is "Avoid logging to the backtrace unconditionally in hot paths"
• All other patches optimize code under sss_nss_protocol_fill_members() (those helpers are also used in other cases), but I admit impact is pretty contained (2..3% of time in my test setup), so if you think some of patches make code readability worse - I'm fine to drop those.

Testing with the same setup as in #7841 (comment) but default debug settings:

time SSS_NSS_USE_MEMCACHE=NO getent -s sss group [email protected] > /dev/null

2.11.0-0.250306.171312 (vanilla) : 2.207 .. 2.434

sssd-9.pr7866-06233 (don't read ts) : 1.142 .. 1.271
sssd-9.pr7866-06235 (debug) : 1.035 .. 1.119
sssd-9.pr7866-06249 (fill-members): 1.007 .. 1.086

[alexey-tikhonov]

@joakim-tjernlund, since you tested previous PR in this ares, would you be interested to try this as well (it should be applied on the top of #7841)?

[alexey-tikhonov]

@marco-kusa, one of original patches in #7841 had to be dropped during review. Would you be able to test this PR in your env ('ignore_group_members = false' code path)?

[joakim-tjernlund]

@joakim-tjernlund, since you tested previous PR in this ares, would you be interested to try this as well (it should be applied on the top of #7841)?

Added an top of master on one machine for now

[joakim-tjernlund]

@joakim-tjernlund, since you tested previous PR in this ares, would you be interested to try this as well (it should be applied on the top of #7841)?

Added an top of master on one machine for now

Now on several machines(5-10)

[alexey-tikhonov]

@joakim-tjernlund, since you tested previous PR in this ares, would you be interested to try this as well (it should be applied on the top of #7841)?

Added an top of master on one machine for now

Now on several machines(5-10)

And what are observations so far?

[aplopez]

I tested this PR against the current master branch (459cc6b) in the following scenario:

• Running in the sssd-ci-containers,
• Using the IPA server,
• 2000 users, each with its private group,
• 1 extra group, all 2000 users are members of it,
• A loop calling id for each user.

I wanted to see how SSSD would behave in this case, not expecting any significative improvement. I ran the loop twice for each case. Before each loop, I deleted the logs and cache, and restarted SSSD.

Master

[root@client /]# rm -f /var/log/sssd/* /var/lib/sss/db/*; systemctl restart sssd.service
[root@client /]# time for ((i=2001; i <= 4000; i++)); do id u${i}@ipa.test > /dev/null; done

real	3m24.460s
user	0m4.097s
sys	0m6.317s
[root@client /]# rm -f /var/log/sssd/* /var/lib/sss/db/*; systemctl restart sssd.service
[root@client /]# time for ((i=2001; i <= 4000; i++)); do id u${i}@ipa.test > /dev/null; done

real	3m19.424s
user	0m4.402s
sys	0m6.403s

This PR

[root@client /]# rm -f /var/log/sssd/* /var/lib/sss/db/*; systemctl restart sssd.serviceq
[root@client /]# time for ((i=2001; i <= 4000; i++)); do id u${i}@ipa.test > /dev/null; done

real	3m38.338s
user	0m4.800s
sys	0m6.633s
[root@client /]# rm -f /var/log/sssd/* /var/lib/sss/db/*; systemctl restart sssd.service
[root@client /]# time for ((i=2001; i <= 4000; i++)); do id u${i}@ipa.test > /dev/null; done

real	3m36.469s
user	0m4.819s
sys	0m6.669s

My test is slower with this PR than without it. May I have done something wrong?

[alexey-tikhonov]

Before each loop, I deleted the logs and cache

I'm pretty sure most of time is spent in 'sssd_be', not in 'sssd_nss'...
Let me see if I can reproduce this with LDAP.

[alexey-tikhonov]

Let me see if I can reproduce this with LDAP.

I can't:

time for ((i=1000001; i <= 1002001; i++)); do id u${i}@ldap.test > /dev/null; done

sssd-2.11.0-0.250314.164005.git459cc6b15.fc42.x86_64

• power saver:
  • 3m28.476s
  • 3m36.098s (light web browsing in parallel)
• performance (+perf attached)
  • 3m6.477s

sssd-9.pr7866-06249.fc42.x86_64

• power saver:
  • 3m27.411s
  • 3m28.687s
• performance (+perf attached)
  • 3m7.599s

The only case where I got any difference - while I was web-browsing while running the test on the same laptop.
Flamegraphs also look pretty much the same.

This is inline with my expectations: the only large group in this setup is "1 extra group, all 2000 users are members of it". This group is actually read only once, all other reads hit mem-cache.

If you can reliably reproduce this performance degradation (while making sure overall load of the host stays the same), please capture logs with debug level 9 and microseconds, a perf flamegraph and share those.

[alexey-tikhonov]

Still, you can use your setup to actually test code paths being touched in this PR - just resolve this large group in a loop with default debug settings and mem-cache disabled:
time for ((i=1; i < 1000; i++)); do SSS_NSS_USE_MEMCACHE=NO getent -s sss group [email protected] > /dev/null; done

That's what I got with LDAP:

sssd-2.11.0-0.250314.164005.git459cc6b15.fc42.x86_64

• 1m0.583s
• 0m59.860s
• 0m59.597s
• 1m5.433s
• 1m0.360s

9.pr7866-06249.fc42

• 0m22.878s
• 0m25.566s
• 0m23.319s
• 0m23.352s
• 0m23.477s

[alexey-tikhonov]

Testing with the same setup as in #7841 (comment) but default debug settings:

In the #7793 environment results are much more modest: merely 10% perf gain.

That's somewhat expected because in that env - a lot of users but only a (small) fraction are members of a given group - it is search-by-memberof what takes the time, not resulting list processing (what is being optimized in this PR).

(But, as expected, #7872 makes it blazing fast - ~ x600 faster).

[aplopez]

Still, you can use your setup to actually test code paths being touched in this PR - just resolve this large group in a loop with default debug settings and mem-cache disabled: time for ((i=1; i < 1000; i++)); do SSS_NSS_USE_MEMCACHE=NO getent -s sss group [email protected] > /dev/null; done

I don't have exactly this setup. In my case (2000 members in the group) I saw no improvement nor degradation.

[sumit-bose]

Hi,

there is a typo in the "'tmp_ctx' was removed as it wasn't really used anyway" commit message, 'aboid' vs 'avoid'.

bye,
Sumit

[alexey-tikhonov]

there is a typo in the "'tmp_ctx' was removed as it wasn't really used anyway" commit message, 'aboid' vs 'avoid'.

Thank you, fixed.

[sumit-bose]

Hi,

thank you for the updates, ACK.

bye,
Sumit