High UHD CPU usage when idle, networked X310

[afountain-brsc]

Issue Description

When creating a multi_usrp object targeting one or two X310 radios (with two UBX160 daughtercards each), one CPU core is completely maxed out on our system when we are not doing anything with the multi_usrp object.

Setup Details

We have seen this happen with UHD v4.8.0.0, as well as v4.4.0.0 and have not tried other versions. We use the default X310 XG firmware image for each release, obtained by using the uhd_images_downloader utility. We have been building the UHD from source with the following set of steps:

wget https://github.com/EttusResearch/uhd/archive/refs/tags/v4.8.0.0.tar.gz
  tar -xf *.tar.gz && cd uhd*/host
  mkdir build && cd build
  cmake ../ \
    -DENABLE_X300=ON \
    -DENABLE_UTILS=ON \
    -DENABLE_EXAMPLES=ON \
    -DENABLE_E300=OFF \
    -DENABLE_E320=OFF \
    -DENABLE_B200=OFF \
    -DENABLE_B100=OFF \
    -DENABLE_USRP2=OFF \
    -DENABLE_N300=OFF \
    -DENABLE_N320=OFF \
    -DENABLE_X400=OFF \
    -DENABLE_MAN_PAGES=OFF \
    -DENABLE_DOXYGEN=OFF \
    -DENABLE_MANUAL=OFF \
    -DENABLE_TESTS=OFF \
    -DENABLE_USB=OFF \
    -DENABLE_OCTOCLOCK=OFF \
    -DENABLE_MPMD=OFF\
    -DENABLE_SIM=OFF\
    -Wno-dev \
    -DCMAKE_C_FLAGS="-O3 -mtune=intel" \
    -DCMAKE_CXX_FLAGS="-O3 -mtune=intel" \
    -DENABLE_STATIC_LIBS=OFF \
    -DENABLE_PYTHON_API=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DLIB_SUFFIX=64 \
    -DCMAKE_INSTALL_PREFIX=/usr/local/uhd4
    make -j $(nproc) && make install

OS: Ubuntu 20.04 LTS
Network card: Intel X710

Expected Behavior

Low CPU usage when not transmitting or receiving with the multi_usrp object.

Actual Behaviour

100% CPU usage of one core when idle

Steps to reproduce the problem

The following program reproduces the issue:

#include <uhd/usrp/multi_usrp.hpp>

#include <thread>
#include <chrono>

int main()
{
  uhd::usrp::multi_usrp::sptr usrp = uhd::usrp::multi_usrp::make(std::string("addr=192.168.30.2"));
  while (true)
  {
    std::this_thread::sleep_for(std::chrono::milliseconds(250));
  }
}

Compiled with:

g++ high_cpu_usage.cpp -g -std=c++17 -fpic -Ofast -mtune=intel -I /usr/include/boost1.78 -I /usr/local/uhd4/include/ -lpthread -luhd -L /usr/local/uhd4/lib64 -Wl,-rpath=/usr/local/uhd4/lib64 -o high_cpu_usage.exe

and run with ./high_cpu_usage.exe.

While running, the following is observed with htop:

and it produces the following output on the console:

# ./high_cpu_usage.exe 
[LOG] Failed to set log level to: 6[INFO] [UHD] linux; GNU C++ version 13.2.1 20231205 (Red Hat 13.2.1-6); Boost_107800; UHD_4.8.0.0-0-unknown
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 1472 bytes.
[WARNING] [X300] For the 192.168.30.2 connection, UHD recommends a send frame size of at least 8000 for best
performance, but your configuration will only allow 1472.This may negatively impact your maximum achievable sample rate.
Check the MTU on the interface and/or the send_frame_size argument.
[WARNING] [X300] For the 192.168.30.2 connection, UHD recommends a receive frame size of at least 8000 for best
performance, but your configuration will only allow 1472.This may negatively impact your maximum achievable sample rate.
Check the MTU on the interface and/or the recv_frame_size argument.
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929b
[INFO] [X300] Radio 1x clock: 200 MHz

[mbr0wn]

100% seems weird, but I wonder if this is a duplicate of #514. Can you show the thread names in htop? Also, what hardware is this running on?

[afountain-brsc]

Thank you for the response! For one of our systems with only one X310 attached, here is an image of htop with the thread names included:

This is running on a machine with:

• CPU: i7-11700K
• RAM: 32GB DDR4 2666MHz
• NIC: 1x Intel X710-DA4

I also ran that program after modifying the device args on another machine using UHD 4.4 that is networked to two X310s, and it looks like it is using one core per idle USRP:

and this machine has:

• CPU: i9-10980XE
• RAM: 64GB DDR4 3200MHz
• NIC: 2x Intel X710-DA2

[mbr0wn]

@afountain-brsc Thanks for getting back to us. The uhd_ctrl_epXXXX thread is indeed the suspected culprit, so I'm going to merge this bug report with #514 (i.e., I'll close it, but we can continue the conversation in the other issue).

Now you might wonder when and if we're going to do something about it, if the other issue is three years old. The thing is, while this is a known problem, it doesn't usually cause a CPU to run at 100%. On my system, for example, the CPU usage of that thread hovers around 8%, and on a different system it was around 30%. If it's at 8%, we can tolerate it and therefore the pain threshold isn't high enough to fix it (this is not a trivial fix). So I wonder what the difference is.

Maybe I'll try your compiler flags. There is a busy polling loop in the uhd_ctrl_* thread, which is the root cause of this issue, but on most systems it doesn't cause the 100% usage that you're seeing, and I'm not sure what the difference is (I have a very similar CPU to yours, i9-9900 series, and neither in WSL nor native Linux do I see this high CPU usage). I wonder if some optimization leads to the busy loop being handled differently.