Missing DTrace probes #104280
Comments
|
I think DTrace was intentionally removed because it was not being used. I forgot where I read that, but Mark will have a better idea of it. |
|
I think a bigger issue is that I'm pretty sure it isn't tested on any CI jobs or buildbots, so stuff can break (like #98894) without us noticing. If it's not tested, it effectively isn't supported. Not sure whether the right path forward is for some motivated person to fix and maintain it, for us to remove the broken stuff and keep the working stuff (with a big disclaimer in the docs), or to just rip it out entirely. Anecdotally, it's always been a bit uncomfortable to touch this stuff when modifying the interpreter, since I'm not very sure how it works or how to test it. @ambv might have an idea of what the right path forward is here. |
Dtrace support was actually being used. DTrace markers, also known as "probes", are not only used by DTtrace. For example, these probes can be used with bpftrace. There are tools written using these probes. I will even give a talk about these probes at EuroPython 2023 :) |
|
Just chiming in here: we use these probes in anger in production, and their absence is a blocker preventing us from upgrading to 3.11. If tests are needed to make sure dtrace support is accounted for in the release process, I am happy to contribute time to write tests and set them up. |
|
@ambv @brandtbucher I wrote a test case to check for the presence of probes. Could you review the PR? |
(cherry picked from commit a1c737b) Co-authored-by: Furkan Onder <[email protected]>
…7125) (cherry picked from commit a1c737b) Co-authored-by: Furkan Onder <[email protected]>
(cherry picked from commit a1c737b) Co-authored-by: Furkan Onder <[email protected]>
|
The newly added tests fail in Fedora: Full log: https://kojipkgs.fedoraproject.org//work/tasks/6927/104486927/build.log Here's a readelf output printed out: I'm unfortunately unfamiliar with DTrace. Is this expected? How can I debug this? |
|
Are you building your Python When I build |
|
BTW, this says "dtrace" but those probes are also SystemTap / eBPF. |
|
Yes, |
|
OK, sorry for the trouble. I will look into this today. |
|
No trouble at all! |
Did you run the readelf tool with the -n parameter? I don't see the probes in your output. Now I have build cpython and I can see the probes. [ted@rasp4 cpython]$ readelf -n ./python
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: ac4de5af42642e3a69705be808d9c8a73aeee03f
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.7.0
Displaying notes found in: .note.stapsdt
Owner Data size Description
stapsdt 0x00000043 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: import__find__load__done
Location: 0x00000000002bad4c, Base: 0x0000000000478105, Semaphore: 0x00000000006c3c7c
Arguments: 8@x0 -4@x1
stapsdt 0x0000003e NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: import__find__load__start
Location: 0x00000000002bb084, Base: 0x0000000000478105, Semaphore: 0x00000000006c3c7a
Arguments: 8@x0
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: audit
Location: 0x00000000002ef1e0, Base: 0x0000000000478105, Semaphore: 0x00000000006c3c7e
Arguments: 8@x21 8@x20
stapsdt 0x00000036 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x000000000030a3c8, Base: 0x0000000000478105, Semaphore: 0x00000000006c3c76
Arguments: -4@[sp, 132]
stapsdt 0x0000002f NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x000000000030a8e4, Base: 0x0000000000478105, Semaphore: 0x00000000006c3c78
Arguments: -8@x19
[ted@rasp4 cpython]$ I didn't see a problem in the tests. [ted@rasp4 cpython]$ sudo ./python -m test test_dtrace
0:00:00 load avg: 0.46 Run tests sequentially
0:00:00 load avg: 0.46 [1/1] test_dtrace
== Tests result: SUCCESS ==
1 test OK.
Total duration: 4.1 sec
Tests result: SUCCESS
[ted@rasp4 cpython]$ |
|
Your readelf versions differ. Maybe there's a difference in output between 2.38 and 2.40? |
|
Oh, got it! We build with Getting to that library might be complicated -- I can't think of something easier than |
|
Note: This affects Fedora also on 3.11.5. We will skip the |
|
dtrace is supposed to have zero (or very close to it) overhead when not in use. I realize that's not the ideal solution for existing users of |
|
It also looks like dtrace function entry robes are broken in 3.11, as they are only invoked if |
|
With dtrace support effectively removed in 3.12, a few thoughts. My initial read of Is there room in Python for both visions of tracing? If not, why? If yes, what kind of resourcing would be needed to make sure that dtrace isn't removed again from the build? |
This is a little bit weird, the DTrace and the PEP 669 are different features. the DTrace API is for people who want to trace the Python process without restarting the process and injecting some code for the PEP 669 We can solve the process tracing problem by using #96123 in some circumstances. But there are still some issues.
If we decide to remove DTrace Support officially, we may need to update the documentation https://docs.python.org/3/howto/instrumentation.html |
|
When building python 3.11.7 from source, I ran into this error: Fixed using: |
|
I took a look at this yesterday in 3.12 as part of my current effort to package 3.12 for OpenIndiana (illumos) and I got the function entry/exit probes restored and working well enough to pass the test_dtrace.py test. I'll share when I get a chance - I'm doing this with original dtrace on illumos (the open-source continuation of opensolaris) but it would likely also work for those who want to use these probes via systemtap. I'll also look at the line probes when I get a chance. |
|
https://github.com/Bill-Sommerfeld/cpython/tree/dtrace-3.12 has patches that restore the I'd appreciate it if others could give these a whirl on other platforms. |
furkanonder
added
3.12
Bug report
On Linux, you can list available DTrace probes by running /bcc/tools/tplist.py tool;
$ ~/bcc/tools/tplist.py -l cpython/python cpython/python python:function__entry cpython/python python:function__return cpython/python python:line cpython/python python:import__find__load__done cpython/python python:import__find__load__start cpython/python python:audit cpython/python python:gc__start cpython/python python:gc__doneThe
function__entry,function__returnandlineprobes appear to be missing after the GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython update.$ ~/bcc/tools/tplist.py -l cpython/python cpython/python python:import__find__load__done cpython/python python:import__find__load__start cpython/python python:audit cpython/python python:gc__start cpython/python python:gc__doneYour environment
Linked PRs
The text was updated successfully, but these errors were encountered: