speedup of pluggy

[RonnyPfannschmidt]

there is multiple levels of speedups possible/for consideration

• external tools (mypyc/cython)
• the hook call mechanism (attr getter in c for the arguments)
• setup/teardown of hooks
• behaviour of the limited/scoped hook callers (subset callers used in pytest)

to better gauge impacts we need benchmarks stressing

• nesting of calls
• number of calls
• number of plugins
• surrounding tools like tracing/hook exclusion
• number of arguments

[wahuneke]

Would you be interested in seeing a POC version using a 'dynamic code' approach? The concept here would be: at the first attempt to run hooks through HookRelay, we 'precompile' the whole stack of hooks so that less logic needs to be evaluated during the hook call and subsequent calls. Modifying the stack (by (de)registering plugins, etc) would invalidate the cached 'compiled' version.

By preprocessing / 'compiling' the stack we accelerate some of the slower aspects of calling hooks, specifically:

• assembling the correct args for each hook's call ( a significant portion of the total overhead, I think)
• multiple ifs and nested function calls and their associated attribute lookups

In my local, rough draft implementation, I was able to speedup benchmarks by 10-20% with only a small modification to the current code. More speed improvement is probably possible under this approach.

In some ways, I think, generated code is a great fit for what pluggy wants to do. On the other hand, I'm not sure it is ever a good idea to use it in a real project :P . Any thoughts?

[RonnyPfannschmidt]

It's not quite clear what you mean by that,

Cython is currently Most likely to create a spedup

Certain refactoring need to be added first tho

[wahuneke]

It's not quite clear what you mean by that,

Cython is currently Most likely to create a spedup

Certain refactoring need to be added first tho

Sorry @RonnyPfannschmidt ! That was a terrible explanation. I've taken one more run at it. And, I've moved the discussion to a separate ticket in order to avoid polluting this one (issue #449)