pybind11-vendor wants to fetch code during the compile stage #1279

vermaete · 2025-01-14T14:16:45Z

On scarthgap and ros2-jazzy the recipe pybind11-vendor wants to fetch code during the compile stage.
What should be during configuration.

I see some examples how to solve this in the CMakefile and bbappend.
But for now my solution is not yet working.
But is there already a solution available?

mark-bchoi · 2025-01-17T02:54:07Z

Hi @vermaete

I don't try to build jazzy yet, but I found there are some difference dependency in the recipe between humble and jazzy which git-native is missed in the jazzy version.

jiaxshi · 2025-01-17T06:37:46Z

Hi
If you encounter an ethernet issue, you can try with
do_configure[network] = "1"

vermaete · 2025-01-19T17:17:25Z

I assume the issue is here: https://github.com/ros2/pybind11_vendor/blob/rolling/CMakeLists.txt#L9
@jiaxshi : Yocto should build without having network access (except fetching step). For those poor people with air-gapped networks at work :-(

mark-bchoi · 2025-01-20T03:44:29Z

@vermaete

I think python3-pybind11 would not be found in your build system correctly. because find_package mode is set to QUIET, you could not any logs for it at the temp/log.do_configure.

You can see logs if you change find_package mode from QUIET to REQUIRED temporarily for further debugging.

vermaete · 2025-01-22T11:54:10Z

@mark-bchoi I hope I have collected all info you need.

I have changed the CMakeLists.txt file after doing a devtool modify pybind11-vendor

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 69666b1..e724fbb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -4,7 +4,7 @@ project(pybind11_vendor CXX)
 find_package(ament_cmake REQUIRED)
 find_package(ament_cmake_vendor_package REQUIRED)

-find_package(pybind11 QUIET)
+find_package(pybind11 REQUIRED)

 ament_vendor(pybind11_vendor
   SATISFIED ${pybind11_FOUND}

The log of the 'bitbake pybdind11-vendor`:

ERROR: pybind11-vendor-3.1.2-2-r0 do_configure: ExecutionError('/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/run.do_configure.3950729', 1, None, None)
ERROR: Logfile of failure stored in: /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/log.do_configure.3950729
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/var/tmp/user/x/yocto/sources/poky/../meta-xilinx/meta-xilinx-core/recipes-devtools/python/python3-pybind11_2.13.1.bb:do_populate_sysroot', '/var/tmp/user/x/yocto/sources/poky/meta/recipes-core/glibc/glibc_2.39.bb:do_populate_sysroot', '/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/cmake/cmake-native_3.28.3.bb:do_populate_sysroot', '/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/gcc/gcc-cross_13.3.bb:do_populate_sysroot', '/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/gcc/gcc-runtime_13.3.bb:do_populate_sysroot', 'virtual:native:/var/tmp/user/x/yocto/sources/poky/../meta-ros/meta-ros2-jazzy/generated-recipes/ament-cmake/ament-cmake-vendor-package_2.5.2-1.bb:do_populate_sysroot', 'virtual:native:/var/tmp/user/x/yocto/sources/poky/../meta-ros/meta-ros2-jazzy/generated-recipes/ament-cmake/ament-cmake_2.5.2-1.bb:do_populate_sysroot', 'virtual:native:/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/ninja/ninja_1.11.1.bb:do_populate_sysroot', 'virtual:native:/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/var/tmp/user/x/yocto/sources/poky/meta/recipes-devtools/python/python3_3.12.6.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['python3-pybind11', 'openssl', 'expat', 'gettext-minimal-native', 'glibc', 'libxcrypt', 'ncurses', 'util-linux-libuuid', 'zlib', 'binutils-cross-aarch64', 'cmake-native', 'gcc-cross-aarch64', 'gcc-runtime', 'libgcc', 'libedit', 'libtool-native', 'm4-native', 'opkg-utils', 'python3', 'bzip2', 'libnsl2', 'libtirpc', 'texinfo-dummy-native', 'xz', 'linux-libc-headers', 'gdbm', 'libffi', 'sqlite3', 'python3-catkin-pkg-native', 'python3-vcstool-native', 'libyaml-native', 'ament-cmake-core-native', 'ament-cmake-export-definitions-native', 'ament-cmake-export-dependencies-native', 'ament-cmake-export-include-directories-native', 'ament-cmake-export-interfaces-native', 'ament-cmake-export-libraries-native', 'ament-cmake-export-link-flags-native', 'ament-cmake-export-targets-native', 'ament-cmake-gen-version-h-native', 'ament-cmake-include-directories-native', 'ament-cmake-libraries-native', 'ament-cmake-python-native', 'ament-cmake-target-dependencies-native', 'ament-cmake-test-native', 'ament-cmake-vendor-package-native', 'ament-cmake-version-native', 'ament-cmake-native', 'ament-package-native', 'openssl-native', 'expat-native', 'ncurses-native', 'util-linux-libuuid-native', 'zlib-native', 'flex-native', 'gnu-config-native', 'libedit-native', 'make-native', 'ninja-native', 'perl-native', 'pseudo-native', 'python3-build-native', 'python3-cython-native', 'python3-flit-core-native', 'python3-importlib-metadata-native', 'python3-installer-native', 'python3-more-itertools-native', 'python3-packaging-native', 'python3-pathlib2-native', 'python3-pip-native', 'python3-pyparsing-native', 'python3-pyproject-hooks-native', 'python3-pyyaml-native', 'python3-setuptools-scm-native', 'python3-setuptools-native', 'python3-six-native', 'python3-toml-native', 'python3-tomli-native', 'python3-typing-extensions-native', 'python3-wheel-native', 'python3-zipp-native', 'python3-native', 'bzip2-native', 'libnsl2-native', 'libtirpc-native', 'unzip-native', 'xz-native', 'zstd-native', 'attr-native', 'gdbm-native', 'gmp-native', 'libffi-native', 'libmpc-native', 'mpfr-native', 're2c-native', 'sqlite3-native']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing python function externalsrc_configure_prefunc
| DEBUG: Python function externalsrc_configure_prefunc finished
| DEBUG: Executing python function fetcher_hashes_dummyfunc
| DEBUG: Python function fetcher_hashes_dummyfunc finished
| DEBUG: Executing shell function do_configure
| -- The CXX compiler identification is GNU 13.3.0
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Check for working CXX compiler: /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/usr/bin/aarch64-xilinx-linux/aarch64-xilinx-linux-g++ - skipped
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- Found ament_cmake: 2.5.2 (/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/opt/ros/jazzy/share/ament_cmake/cmake)
| -- Found Python3: /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/usr/bin/python3-native/python3 (found version "3.12.6") found components: Interpreter
| -- Found ament_cmake_vendor_package: 2.5.2 (/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/opt/ros/jazzy/share/ament_cmake_vendor_package/cmake)
| -- Configuring incomplete, errors occurred!
| CMake Error at CMakeLists.txt:7 (find_package):
|   By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
|   asked CMake to find a package configuration file provided by "pybind11",
|   but CMake did not find one.
|
|   Could not find a package configuration file provided by "pybind11" with any
|   of the following names:
|
|     pybind11Config.cmake
|     pybind11-config.cmake
|
|   Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
|   "pybind11_DIR" to a directory containing one of the above files.  If
|   "pybind11" provides a separate development package or SDK, be sure it has
|   been installed.
|
|
| WARNING: /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/run.do_configure.3950729:174 exit 1 from 'cmake -G 'Ninja' -DCMAKE_MAKE_PROGRAM=ninja $oecmake_sitefile /var/tmp/user/x/yocto/build/workspace/sources/pybind11-vendor -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_INSTALL_BINDIR:PATH=bin -DCMAKE_INSTALL_SBINDIR:PATH=sbin -DCMAKE_INSTALL_LIBEXECDIR:PATH=libexec -DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc -DCMAKE_INSTALL_SHAREDSTATEDIR:PATH=../com -DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/var -DCMAKE_INSTALL_LIBDIR:PATH=lib -DCMAKE_INSTALL_INCLUDEDIR:PATH=include -DCMAKE_INSTALL_DATAROOTDIR:PATH=share -DPYTHON_EXECUTABLE:PATH=/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/usr/bin/python3-native/python3 -DPython_EXECUTABLE:PATH=/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/usr/bin/python3-native/python3 -DPython3_EXECUTABLE:PATH=/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot-native/usr/bin/python3-native/python3 -DLIB_SUFFIX= -DCMAKE_INSTALL_SO_NO_EXE=0 -DCMAKE_TOOLCHAIN_FILE:FILEPATH=/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/toolchain.cmake -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DFETCHCONTENT_FULLY_DISCONNECTED=ON -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON -DCMAKE_PREFIX_PATH='/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot/opt/ros/jazzy;/var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot/usr' -DCMAKE_INSTALL_PREFIX:PATH='/opt/ros/jazzy' -DBUILD_TESTING=OFF -DPYTHON_SOABI=cpython-312-aarch64-linux-gnu -DAMENT_CMAKE_ENVIRONMENT_PARENT_PREFIX_PATH_GENERATION=OFF -Wno-dev'
| WARNING: Backtrace (BB generated script):
|       #1: cmake_do_configure, /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/run.do_configure.3950729, line 174
|       #2: do_configure, /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/run.do_configure.3950729, line 153
|       #3: main, /var/tmp/user/x/yocto/build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/temp/run.do_configure.3950729, line 197
ERROR: Task (/var/tmp/user/x/yocto/sources/poky/../meta-ros/meta-ros2-jazzy/generated-recipes/pybind11-vendor/pybind11-vendor_3.1.2-2.bb:do_configure) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2266 tasks of which 2116 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /var/tmp/user/x/yocto/sources/poky/../meta-ros/meta-ros2-jazzy/generated-recipes/pybind11-vendor/pybind11-vendor_3.1.2-2.bb:do_configure

Just in case you need this:

user@server:/var/tmp/user/x/yocto$ find . -name pybind11Config.cmake
./build/tmp/work/cortexa72-cortexa53-xilinx-linux/pybind11-vendor/3.1.2-2/recipe-sysroot/usr/lib/python3.12/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake
./build/tmp/sysroots-components/cortexa72-cortexa53/python3-pybind11/usr/lib/python3.12/site-packages/pybind11/share/cmake/pybind11/pybind11Config.cmake

mark-bchoi · 2025-01-22T12:28:10Z

@vermaete ,

python3-pybind11 should be built with cmake. if you did so, you can see pybind11Config.cmake in you build directory.

tmp/work/armv8a-oe4t-linux/python3-pybind11/2.11.1$ tree image

    ├── lib
    │   └── python3.12
    │       └── site-packages
    │           ├── pybind11
    │           │   ├── __init__.py
    │           │   ├── __main__.py
    │           │   ├── __pycache__
    │           │   │   ├── __init__.cpython-312.pyc
    │           │   │   ├── __main__.cpython-312.pyc
    │           │   │   ├── _version.cpython-312.pyc
    │           │   │   ├── commands.cpython-312.pyc
    │           │   │   └── setup_helpers.cpython-312.pyc
    │           │   ├── _version.py
    │           │   ├── commands.py
    │           │   ├── include
    │           │   │   └── pybind11
    │           │   │       ├── attr.h
    │           │   │       ├── buffer_info.h
    │           │   │       ├── cast.h
    │           │   │       ├── chrono.h
    │           │   │       ├── common.h
    │           │   │       ├── complex.h
    │           │   │       ├── detail
    │           │   │       │   ├── class.h
    │           │   │       │   ├── common.h
    │           │   │       │   ├── descr.h
    │           │   │       │   ├── init.h
    │           │   │       │   ├── internals.h
    │           │   │       │   ├── type_caster_base.h
    │           │   │       │   └── typeid.h
    │           │   │       ├── eigen
    │           │   │       │   ├── common.h
    │           │   │       │   ├── matrix.h
    │           │   │       │   └── tensor.h
    │           │   │       ├── eigen.h
    │           │   │       ├── embed.h
    │           │   │       ├── eval.h
    │           │   │       ├── functional.h
    │           │   │       ├── gil.h
    │           │   │       ├── iostream.h
    │           │   │       ├── numpy.h
    │           │   │       ├── operators.h
    │           │   │       ├── options.h
    │           │   │       ├── pybind11.h
    │           │   │       ├── pytypes.h
    │           │   │       ├── stl
    │           │   │       │   └── filesystem.h
    │           │   │       ├── stl.h
    │           │   │       ├── stl_bind.h
    │           │   │       └── type_caster_pyobject_ptr.h
    │           │   ├── py.typed
    │           │   ├── setup_helpers.py
    │           │   └── share
    │           │       ├── cmake
    │           │       │   └── pybind11
    │           │       │       ├── FindPythonLibsNew.cmake
    │           │       │       ├── pybind11Common.cmake
    │           │       │       ├── pybind11Config.cmake
    │           │       │       ├── pybind11ConfigVersion.cmake
    │           │       │       ├── pybind11NewTools.cmake
    │           │       │       ├── pybind11Targets.cmake
    │           │       │       └── pybind11Tools.cmake

You can check python3-pybind11 recipe.

My recipe is as below which have python3-cmake-native and inherit cmake. I think they are involved in creating $PNConfig.cmake.

bchoi@bchoi-Hyper-V-UEFI-Release-v4-1:/mnt/nvme1n1/deployment/yocto/scarthgap/bchoi/nvidia_jetson-orin-nano-devkit/feature/nvidia-yocto-bsp/layers$ cat ./meta-openembedded/meta-python/recipes-devtools/python/python3-pybind11_2.11.1.bb
SUMMARY = "Seamless operability between C++11 and Python"
HOMEPAGE = "https://github.com/pybind/pybind11"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://LICENSE;md5=774f65abd8a7fe3124be2cdf766cd06f"
DEPENDS = "\
    python3-cmake-native \
    python3-ninja-native \
"

SRCREV = "8a099e44b3d5f85b20f05828d919d2332a8de841"
SRC_URI = "\
    git://github.com/pybind/pybind11.git;branch=stable;protocol=https \
    file://0001-Do-not-strip-binaries.patch \
"

S = "${WORKDIR}/git"

inherit cmake python_setuptools_build_meta

EXTRA_OECMAKE = "-DPYBIND11_TEST=OFF"

do_configure:append() {
    cmake_do_configure
}

do_compile:append() {
    cmake_do_compile
}

do_install:append() {
    cmake_do_install
}

BBCLASSEXTEND = "native nativesdk"
bchoi@bchoi-Hyper-V-UEFI-Release-v4-1:/mnt/nvme1n1/deployment/yocto/scarthgap/bchoi/nvidia_jetson-orin-nano-devkit/feature/nvidia-yocto-bsp/layers$

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pybind11-vendor wants to fetch code during the compile stage #1279

pybind11-vendor wants to fetch code during the compile stage #1279

vermaete commented Jan 14, 2025

mark-bchoi commented Jan 17, 2025

jiaxshi commented Jan 17, 2025

vermaete commented Jan 19, 2025

mark-bchoi commented Jan 20, 2025

vermaete commented Jan 22, 2025

mark-bchoi commented Jan 22, 2025

pybind11-vendor wants to fetch code during the compile stage #1279

pybind11-vendor wants to fetch code during the compile stage #1279

Comments

vermaete commented Jan 14, 2025

mark-bchoi commented Jan 17, 2025

jiaxshi commented Jan 17, 2025

vermaete commented Jan 19, 2025

mark-bchoi commented Jan 20, 2025

vermaete commented Jan 22, 2025

mark-bchoi commented Jan 22, 2025