offcputime.py and profile.py can now be stopped by SIGUSR1 signal

[jirka-h]

Hello,

I was missing an option to stop offcputime.py and profile.py when the benchmark stops.

I have added an option to send the SIGUSR1 signal to stop these applications. Here is the pseudocode showing a usage from Bash script:

# Start the profile tool in the background
/usr/share/bcc/tools/profile.py -adf > profile.folded &
PROFILE_PID=$!

# Run the benchmark
./run_benchmark   # Replace with the actual benchmark command

# Stop profile with SIGUSR1. It will shut down correctly, producing a full output. 
kill -USR1 $PROFILE_PID

# Wait for profile.py to finish
wait $PROFILE_PID

Please consider adding this feature.

Thanks a lot
Jirka

[yonghong-song]

I don't know what is the purpose for your benchmarking, maybe testing bpf overhead? For offcputime.py and profile.py, ctrl-C is needed to end the tool (after printing out the necessary information). You can just filter out these two tools when you are doing your benchmarking. I don't want to add these SIGUSR1 things for benchmarking purpose.

[jirka-h]

Hi, these changes are not to benchmark the BPF overhead. The purpose is to automate the collection of call traces in the folded format for Flamegraphs generation.

With perf record, I collect the call traces like this:

perd record --call-graph fp -o ${out_name}.perf -- ./test_binary
perf script -i "${out_name}.perf" | "${SCRIPT_DIR}/FlameGraph/stackcollapse-perf.pl" > "${out_name}.perf-folded"
"${SCRIPT_DIR}/FlameGraph/flamegraph.pl" --minwidth 1 "${out_name}.perf-folded" > "${out_name}.svg"

The point here is that I collect the traces just for the test_binary. Sometimes, I need to use -a to collect the traces from the whole system. The key idea is that it's fully automated, and you collect the traces only while test_binary is running.

The issue with perf-record is its significant overhead, which leads to the collection of vast amounts of data that must later be filtered. To address this, Brendan Gregg developed these scripts. However, the current limitation is that there is no way to use these tools from a fully automated script. You either need to specify the duration in advance or run it in the foreground, stopping the collection with Ctrl-C. Both approaches are impractical.

SIGUSR1 is a simple method to stop the collection of call traces as soon as the test program ends. The perf-record command above can be rewritten as follows:

/usr/share/bcc/tools/profile.py -adf > profile.folded &
PROFILE_PID=$!

# Run the test
./test_binary   # Collect call traces for your binary

# Stop profile with SIGUSR1. This is to ensure the profiling will stop instantly after the test_binary finishes.
kill -USR1 $PROFILE_PID

# Wait for profile to finish
wait $PROFILE_PID

#Create Flamegraphs
"${SCRIPT_DIR}/FlameGraph/flamegraph.pl" --colors=java --minwidth 1 "profile.folded" > "${out_name}.svg"

Perhaps this approach is too cumbersome, and we should implement a method similar to perf-record, where you are allowed to specify the program to be launched and profiled and run it like this?

/usr/share/bcc/tools/profile.py -adf -o profile.folded -- ./test_binary

Please share your thoughts. I believe a method similar to perf-record would be the best solution. If you agree, I can prepare the PR. Nonetheless, I still think it would be valuable to have a way to stop profile.py when running in the background and output the data collected so far. Perhaps instead of using SIGUSR1, we should switch to the more standard signal SIGINT? At the moment, SIGINT stops profile.py instantly, and no output is generated.

Thank you!
Jirka