meta-searcher in unified.xml missing

[deeglaze]

Attempting to debug build with systemd-stub, instructions to use unified.xml fail with "error: Cannot fetch meta-searcher from https://github.com/flashbots/meta-searcher"

[deeglaze]

Switching out unified with tdx-base.xml, I get the following error

NOTE: Starting bitbake server...
ERROR: Unable to parse /opt/src/yocto-tdx/srcs/poky/bitbake/lib/bb/parse/__init__.py
Traceback (most recent call last):
  File "/opt/src/yocto-tdx/srcs/poky/bitbake/lib/bb/parse/__init__.py", line 120, in handle(fn='conf/machine/tdx-gcp', data=<bb.data_smart.DataSmart object at 0x7fa6f6c0f2d0>, include=True, baseconfig=False):
                     return h['handle'](fn, data, include, baseconfig)
    >    raise ParseError("not a BitBake file", fn)
     
bb.parse.ParseError: ParseError in conf/machine/tdx-gcp: not a BitBake file

I'm attempting to build for MACHINE=tdx-gcp bitbake core-image-minimal but tdx-gcp is not selectable. "tdx" similarly so:

ERROR: Unable to parse /opt/src/yocto-tdx/srcs/poky/bitbake/lib/bb/parse/__init__.py
Traceback (most recent call last):
  File "/opt/src/yocto-tdx/srcs/poky/bitbake/lib/bb/parse/__init__.py", line 120, in handle(fn='conf/machine/tdx', data=<bb.data_smart.DataSmart object at 0x7f19816c74d0>, include=True, baseconfig=False):
                     return h['handle'](fn, data, include, baseconfig)
    >    raise ParseError("not a BitBake file", fn)
     
bb.parse.ParseError: ParseError in conf/machine/tdx: not a BitBake file

I don't use yocto for making images, so I need more basic instructions to repro.
Here's the Dockerfile I've put together trying to follow given instructions (locale stuff added for Debian)

FROM marketplace.gcr.io/google/debian12:latest                                                                                                                                                                                                       
                                                                                                                                                                                                                                                     
RUN apt-get update && apt-get install -y \                                                                                                                                                                                                           
  gawk curl wget git diffstat unzip texinfo gcc build-essential chrpath socat \
  cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
  python3-git python3-jinja2 python3-subunit zstd liblz4-tool file locales \
  locales-all libacl1
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
RUN dpkg-reconfigure locales
RUN mkdir ~/bin ; curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo && chmod a+x ~/bin/repo
#ENV MACHINE "tdx-gcp bitbake core-image-minimal"
ENV MACHINE "tdx bitbake core-image-minimal"
RUN mkdir -p /opt/src/yocto-tdx
RUN (cd /opt/src/yocto-tdx ; ~/bin/repo init -u https://github.com/flashbots/yocto-manifests.git -b main -m tdx-base.xml )
RUN (cd /opt/src/yocto-tdx ;  ~/bin/repo sync )                                                                                                                                                                                                                                                     
WORKDIR /opt/src/yocto-tdx

ENTRYPOINT ["bash", "-c", "source ./setup", "-i"]                                                                                                                                                                                                    

[fnerdman]

You can ignore the meta-searcher repo for your use case.
After init, before sync, run

sed -i '/<project.*meta-searcher.*\/>/d' .repo/manifests/unified.xml

In the work directory.

[fnerdman]

@deeglaze totally of topic but raising this here in case my email doesn't get through: https://www.blackhat.com/eu-24/briefings/schedule/#aml-injection-attacks-on-confidential-vms-42723

[deeglaze]

Yup, got the email. I’ll keep working on all my obligations and keep fighting for time to work on transparency. Glad to see more educational materials for potential CoCo customers to make transparency a priority. Y’all are ahead of the curve.

[deeglaze]

So the sed command doesn't change the missing bitbake file problem.

[fnerdman]

This is how it works for me:

~$ mkdir tdx-yocto && cd tdx-yocto/
~/tdx-yocto$ repo init -u https://github.com/flashbots/yocto-manifests.git -b unified -m unified.xml
Downloading Repo source from https://gerrit.googlesource.com/git-repo

Your identity is: fnerdman <[email protected]>
If you want to change this, please re-run 'repo init' with --config-name

repo has been initialized in /home/frieder/tdx-yocto
~/tdx-yocto$ sed -i '/<project.*meta-searcher.*\/>/d' .repo/manifests/unified.xml
~/tdx-yocto$ repo sync
Fetching: 100% (11/11), done in 10.254s
Checking out: 100% (11/11), done in 0.508s
repo sync has finished successfully.
~/tdx-yocto$ source setup
You had no conf/local.conf file. This configuration file has therefore been
created for you from /home/frieder/tdx-yocto/poky/meta-poky/conf/templates/default/local.conf.sample
You may wish to edit it to, for example, select a different MACHINE (target
hardware).

You had no conf/bblayers.conf file. This configuration file has therefore been
created for you from /home/frieder/tdx-yocto/poky/meta-poky/conf/templates/default/bblayers.conf.sample
To add additional metadata layers into your configuration please add entries
to conf/bblayers.conf.

The Yocto Project has extensive documentation about OE including a reference
manual which can be found at:
    https://docs.yoctoproject.org

For more information about OpenEmbedded see the website:
    https://www.openembedded.org/

This is the default build configuration for the Poky reference distribution.

### Shell environment set up for builds. ###

You can now run 'bitbake <target>'

Common targets are:
    core-image-minimal
    core-image-full-cmdline
    core-image-sato
    core-image-weston
    meta-toolchain
    meta-ide-support

You can also run generated qemu images with a command like 'runqemu qemux86-64'.

Other commonly useful commands are:
 - 'devtool' and 'recipetool' handle common recipe tasks
 - 'bitbake-layers' handles common layer tasks
 - 'oe-pkgdata-util' handles common target package tasks
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Specified layer directory /home/frieder/tdx-yocto/meta-searcher doesn't exist
patching file /home/frieder/tdx-yocto/build/conf/local.conf
~/tdx-yocto/build$ sed -i '$ s/ bob//' conf/local.conf 
~/tdx-yocto/build$ MACHINE=tdx-gcp bitbake core-image-minimal
Loading cache: 100% |                                                                                                                                                                                                        | ETA:  --:--:--
Loaded 0 entries from dependency cache.
NOTE: /home/frieder/tdx-yocto/meta-confidential-compute/recipes-core/date-sync/date-sync.bb: Time sync interval set to: 600 seconds                                                                                          | ETA:  --:--:--
NOTE: /home/frieder/tdx-yocto/meta-evm/recipes-security/ssh-key-injection/ssh-key-injection.bb: No SSH_PUBKEY is set. The built image will have no SSH access!                                                                | ETA:  0:00:32
NOTE: /home/frieder/tdx-yocto/meta-confidential-compute/recipes-core/date-sync/date-sync.bb: Time sync interval set to: 600 seconds                                                                                           | ETA:  0:00:28
NOTE: /home/frieder/tdx-yocto/meta-evm/recipes-security/ssh-key-injection/ssh-key-injection.bb: No SSH_PUBKEY is set. The built image will have no SSH access!                                                                | ETA:  0:00:27
NOTE: /home/frieder/tdx-yocto/meta-confidential-compute/recipes-core/disk-encryption/disk-encryption.bb: Disk LUN number is set to 10                                                                                         | ETA:  0:00:24
NOTE: /home/frieder/tdx-yocto/meta-confidential-compute/recipes-core/disk-encryption/disk-encryption.bb: Disk LUN number is set to 10                                                                                         | ETA:  0:00:23
Parsing recipes: 100% |#######################################################################################################################################################################################################| Time: 0:00:25
Parsing of 6012 .bb files complete (0 cached, 6012 parsed). 9910 targets, 1364 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies

Build Configuration (mc:default):
BB_VERSION           = "2.8.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-22.04"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "tdx-gcp"
DISTRO               = "cvm"
DISTRO_VERSION       = "5.0.3"
TUNE_FEATURES        = "m64 x86-64-v3"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "HEAD:c5627ab06fc07221237b9a7059ac378ea80578f8"
meta-oe              
meta-python          
meta-networking      
meta-filesystems     = "HEAD:4a7bb77f7ebe0ac8be5bab5103d8bd993e17e18d"
meta-virtualization  = "HEAD:54b806b1985f3989722ee308e1073530fe3328c1"
meta-tpm2            = "HEAD:adbc1d929c4470c260464215b80f3968ea6d4564"
meta-confidential-compute = "HEAD:3a6ad2a707704d97cad03bbbbd263cea6f1394de"
meta-evm             = "HEAD:12e2d2f27180a99a1a0f942f315f47a6202df8d1"
meta-rust-bin        = "HEAD:061d78b05f2afb995d2d8f0cf7f403b14f89c92a"
meta-clang           = "HEAD:e7dceb1c92caf7f21ef1d7b49c85328c30cffd90"
meta-observability   = "HEAD:8e725fcc720295f63c33cd931addc5b9293b171d"
meta-custom-podman   = "HEAD:0f06d33f769f819ccd49388bab093b6247e216ab"

NOTE: Fetching uninative binary shim http://downloads.yoctoproject.org/releases/uninative/4.5/x86_64-nativesdk-libc-4.5.tar.xz;sha256sum=43ee6a25bcf5fce16ea87076d6a96e79ead6ced90690a058d07432f902773473 (will check PREMIRRORS first)

Build Configuration (mc:rbuilder):
BB_VERSION           = "2.8.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-22.04"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "tdx-gcp"
DISTRO               = "cvm"
DISTRO_VERSION       = "5.0.3"
TUNE_FEATURES        = "m64 x86-64-v3"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "HEAD:c5627ab06fc07221237b9a7059ac378ea80578f8"
meta-oe              
meta-python          
meta-networking      
meta-filesystems     = "HEAD:4a7bb77f7ebe0ac8be5bab5103d8bd993e17e18d"
meta-virtualization  = "HEAD:54b806b1985f3989722ee308e1073530fe3328c1"
meta-tpm2            = "HEAD:adbc1d929c4470c260464215b80f3968ea6d4564"
meta-confidential-compute = "HEAD:3a6ad2a707704d97cad03bbbbd263cea6f1394de"
meta-evm             = "HEAD:12e2d2f27180a99a1a0f942f315f47a6202df8d1"
meta-rust-bin        = "HEAD:061d78b05f2afb995d2d8f0cf7f403b14f89c92a"
meta-clang           = "HEAD:e7dceb1c92caf7f21ef1d7b49c85328c30cffd90"
meta-observability   = "HEAD:8e725fcc720295f63c33cd931addc5b9293b171d"
meta-custom-podman   = "HEAD:0f06d33f769f819ccd49388bab093b6247e216ab"

Checking sstate mirror object availability: 100% |############################################################################################################################################################################| Time: 0:00:28
Sstate summary: Wanted 1294 Local 0 Mirrors 586 Missed 708 Current 0 (45% match, 0% complete)
NOTE: Executing Tasks
WARNING: busybox-1.36.1-r0 do_fetch: Failed to fetch URL https://busybox.net/downloads/busybox-1.36.1.tar.bz2;name=tarball, attempting MIRRORS if available
WARNING: ovmf-native-edk2-stable202402-r0 do_fetch: Failed to fetch URL gitsm://github.com/Zeex/subhook.git;protocol=https;name=UnitTestFrameworkPkg/Library/SubhookLib/subhook;subpath=UnitTestFrameworkPkg/Library/SubhookLib/subhook;nobranch=1;lfs=True;bareclone=1;nobranch=1, attempting MIRRORS if available
WARNING: linux-yocto-tiny-6.10+git-r0 do_kernel_configcheck: [kernel config]: This BSP contains fragments with warnings:


[INFO]: Fragments with badly formatted configuration options:
    - fragment configs/v6.10/standard/tiny/./security-mitigations.cfg has the following issues: # CONFIG_FINEIBT should consider enabling this at some point

[INFO]: the following symbols were not found in the active configuration:
     - CONFIG_EMBEDDED

NOTE: Tasks Summary: Attempted 2794 tasks of which 1301 didn't need to be rerun and all succeeded.

Summary: There were 3 WARNING messages.