Skip to content

cross-compiling regression #1469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
classabbyamp opened this issue May 10, 2025 · 9 comments
Open

cross-compiling regression #1469

classabbyamp opened this issue May 10, 2025 · 9 comments

Comments

@classabbyamp
Copy link

When cross-compiling openssl-sys from x86_64-unknown-linux-gnu to aarch64-unknown-linux-gnu (and other host/target combinations) with cc 1.2.15 or higher, cc fails to use the correct compiler, leading to compilation failures because it uses the host's compiler with the target's cflags.

By bisecting, I have determined the issue was caused by #1401.

Build log showing failure 
warning: [email protected]: cc1: error: bad value 'armv8-a' for '-march=' switch
warning: [email protected]: cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server cascadelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids graniterapids-d arrowlake arrowlake-s lunarlake pantherlake bonnell atom silvermont slm goldmont goldmont-plus tremont gracemont sierraforest grandridge clearwaterforest knl knm x86-64 x86-64-v2 x86-64-v3 x86-64-v4 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui yongfeng k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 znver5 btver1 btver2 native
error: failed to run custom build command for `openssl-sys v0.9.108`

Caused by:
  process didn't exit successfully: `/builddir/sequoia-sq-1.3.1/target/release/build/openssl-sys-03b8be8f4cdf3f69/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-check-cfg=cfg(osslconf, values("OPENSSL_NO_OCB", "OPENSSL_NO_SM4", "OPENSSL_NO_SEED", "OPENSSL_NO_CHACHA", "OPENSSL_NO_CAST", "OPENSSL_NO_IDEA", "OPENSSL_NO_CAMELLIA", "OPENSSL_NO_RC4", "OPENSSL_NO_BF", "OPENSSL_NO_PSK", "OPENSSL_NO_DEPRECATED_3_0", "OPENSSL_NO_SCRYPT", "OPENSSL_NO_SM3", "OPENSSL_NO_RMD160", "OPENSSL_NO_EC2M", "OPENSSL_NO_OCSP", "OPENSSL_NO_CMS", "OPENSSL_NO_COMP", "OPENSSL_NO_SOCK", "OPENSSL_NO_STDIO", "OPENSSL_NO_EC", "OPENSSL_NO_SSL3_METHOD", "OPENSSL_NO_KRB5", "OPENSSL_NO_TLSEXT", "OPENSSL_NO_SRP", "OPENSSL_NO_RFC3779", "OPENSSL_NO_SHA", "OPENSSL_NO_NEXTPROTONEG", "OPENSSL_NO_ENGINE", "OPENSSL_NO_BUF_FREELISTS", "OPENSSL_NO_RC2"))
  cargo:rustc-check-cfg=cfg(openssl)
  cargo:rustc-check-cfg=cfg(libressl)
  cargo:rustc-check-cfg=cfg(boringssl)
  cargo:rustc-check-cfg=cfg(awslc)
  cargo:rustc-check-cfg=cfg(libressl250)
  cargo:rustc-check-cfg=cfg(libressl251)
  cargo:rustc-check-cfg=cfg(libressl252)
  cargo:rustc-check-cfg=cfg(libressl261)
  cargo:rustc-check-cfg=cfg(libressl270)
  cargo:rustc-check-cfg=cfg(libressl271)
  cargo:rustc-check-cfg=cfg(libressl273)
  cargo:rustc-check-cfg=cfg(libressl280)
  cargo:rustc-check-cfg=cfg(libressl281)
  cargo:rustc-check-cfg=cfg(libressl291)
  cargo:rustc-check-cfg=cfg(libressl310)
  cargo:rustc-check-cfg=cfg(libressl321)
  cargo:rustc-check-cfg=cfg(libressl332)
  cargo:rustc-check-cfg=cfg(libressl340)
  cargo:rustc-check-cfg=cfg(libressl350)
  cargo:rustc-check-cfg=cfg(libressl360)
  cargo:rustc-check-cfg=cfg(libressl361)
  cargo:rustc-check-cfg=cfg(libressl370)
  cargo:rustc-check-cfg=cfg(libressl380)
  cargo:rustc-check-cfg=cfg(libressl381)
  cargo:rustc-check-cfg=cfg(libressl382)
  cargo:rustc-check-cfg=cfg(libressl390)
  cargo:rustc-check-cfg=cfg(libressl400)
  cargo:rustc-check-cfg=cfg(libressl410)
  cargo:rustc-check-cfg=cfg(ossl101)
  cargo:rustc-check-cfg=cfg(ossl102)
  cargo:rustc-check-cfg=cfg(ossl102f)
  cargo:rustc-check-cfg=cfg(ossl102h)
  cargo:rustc-check-cfg=cfg(ossl110)
  cargo:rustc-check-cfg=cfg(ossl110f)
  cargo:rustc-check-cfg=cfg(ossl110g)
  cargo:rustc-check-cfg=cfg(ossl110h)
  cargo:rustc-check-cfg=cfg(ossl111)
  cargo:rustc-check-cfg=cfg(ossl111b)
  cargo:rustc-check-cfg=cfg(ossl111c)
  cargo:rustc-check-cfg=cfg(ossl111d)
  cargo:rustc-check-cfg=cfg(ossl300)
  cargo:rustc-check-cfg=cfg(ossl310)
  cargo:rustc-check-cfg=cfg(ossl320)
  cargo:rustc-check-cfg=cfg(ossl330)
  cargo:rustc-check-cfg=cfg(ossl340)
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=SYSROOT
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rustc-link-lib=ssl
  cargo:rustc-link-lib=crypto
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-changed=build/expando.c
  OPT_LEVEL = Some(0)
  OUT_DIR = Some(/builddir/sequoia-sq-1.3.1/target/release/build/openssl-sys-fba1018ce15cddbc/out)
  TARGET = Some(x86_64-unknown-linux-gnu)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = Some(gcc)
  cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
  CC_KNOWN_WRAPPER_CUSTOM = None
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a   -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.)
  cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
  CC_SHELL_ESCAPED_FLAGS = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = Some(-O2)
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  CARGO_ENCODED_RUSTFLAGS = Some()
  cargo:warning=cc1: error: bad value 'armv8-a' for '-march=' switch
  cargo:warning=cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server cascadelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids graniterapids-d arrowlake arrowlake-s lunarlake pantherlake bonnell atom silvermont slm goldmont goldmont-plus tremont gracemont sierraforest grandridge clearwaterforest knl knm x86-64 x86-64-v2 x86-64-v3 x86-64-v4 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui yongfeng k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 znver5 btver1 btver2 native

  --- stderr

  thread 'main' panicked at /host/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/openssl-sys-0.9.108/build/main.rs:315:13:

  Header expansion error:
  Error { kind: ToolExecError, message: "command did not execute successfully (status code exit status: 1): LC_ALL=\"C\" \"gcc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-gdwarf-4\" \"-fno-omit-frame-pointer\" \"-m64\" \"-I\" \"/usr/aarch64-linux-gnu/usr/include\" \"-fstack-clash-protection\" \"-D_FORTIFY_SOURCE=2\" \"-O2\" \"-pipe\" \"-march=armv8-a\" \"-I/usr/aarch64-linux-gnu/usr/include\" \"-ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.\" \"-O2\" \"-E\" \"build/expando.c\"" }

  Failed to find OpenSSL development headers.

  You can try fixing this setting the `OPENSSL_DIR` environment variable
  pointing to your OpenSSL installation or installing OpenSSL headers package
  specific to your distribution:

      # On Ubuntu
      sudo apt-get install pkg-config libssl-dev
      # On Arch Linux
      sudo pacman -S pkgconf openssl
      # On Fedora
      sudo dnf install pkgconf perl-FindBin perl-IPC-Cmd openssl-devel
      # On Alpine Linux
      apk add pkgconf openssl-dev

  See rust-openssl documentation for more information:

      https://docs.rs/openssl

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
@madsmtm
Copy link
Collaborator

madsmtm commented May 11, 2025

The problem is that setting CFLAGS="-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=." is not enough for cc-rs to know which target those flags apply to. So we have to assume that it applies to all configurations. Your setup only worked before because you happened to set HOST_CFLAGS, but that was brittle.

I will close this as intended behaviour, you should use one of the following workarounds instead:

  1. Set HOST_CFLAGS in a manner that "unset"s the flags from before. Something like HOST_CFLAGS="-fno-stack-clash-protection -U_FORTIFY_SOURCE -march=x86-64", depending on exact compiler errors you're seeing. Might not always be possible.
  2. Set CFLAGS_aarch64_unknown_linux_gnu instead of CFLAGS.

@madsmtm madsmtm closed this as not planned Won't fix, can't repro, duplicate, stale May 11, 2025
@classabbyamp
Copy link
Author

HOST_CFLAGS was already being set to -O2

if CFLAGS_$target is set (and CFLAGS is also set), it also does not work. Why does the more specific variable not override CFLAGS? That would make sense, instead of requiring that it be unset.

is this behaviour documented somewhere other than the build.rs scripts?

@madsmtm
Copy link
Collaborator

madsmtm commented May 12, 2025

Rethinking a bit, maybe I am wrong here? The old behaviour somewhat matched build.rustflags in Cargo, though that has quite confusing behaviour too (it seems to be dependent on the --target flag).

For a bit more prior art, we have Bazel's --copt and --host_copt flags, I'm pretty sure that --copt only applies to target binaries.

WDYT @NobodyXu? Reopening for discussion.

@madsmtm
Copy link
Collaborator

madsmtm commented May 12, 2025 •
edited
Loading

Let me outline various scenarios, might make it easier to talk about what the desired behaviour is.

There are effectively three ways of invoking Cargo:

  1. With --target $TARGET.
  2. With --target $HOST.
  3. Without target flag.

And cc-rs can be used in a few places:

  1. In build.rs script for a crate that will be used in the application itself.
  2. In build.rs script for a crate compiled for the host (such as a proc-macro dependency).
  3. Outside build.rs, as part of the application itself.

@madsmtm
Copy link
Collaborator

madsmtm commented May 12, 2025

Perhaps the desired behaviour would be:

let cflags = if cargo_target_flag_is_specified {
    if cross_compile {
        CFLAGS + CFLAGS_$TARGET
    } else {
        CFLAGS_HOST + CFLAGS_$TARGET
    }
} else {
    CFLAGS + CFLAGS_HOST + CFLAGS_$TARGET
};

But I don't think that is implementable with the information that Cargo provides us.

@NobodyXu NobodyXu reopened this May 12, 2025
@NobodyXu
Copy link
Collaborator

Hmm so I see that the old behavior is more reasonable for some use cases, so maybe we can introduce CFLAGS_$target_OVERRIDE to override default CFLAGS? I don't want to change the semantics of CFLAGS_$target again as it might break someone again.

@NobodyXu
Copy link
Collaborator

But I don't think that is implementable with the information that Cargo provides us.

Is that because cargo_target_flag_is_specified is hard to probe?

We do have code for probing cross compilation

cargo_target_flag_is_specified

@madsmtm
Copy link
Collaborator

madsmtm commented May 12, 2025

Yeah, only Cargo knows whether the --target flag has been specified (and that's probably for the best).

Hmm so I see that the old behavior is more reasonable for some use cases, so maybe we can introduce CFLAGS_$target_OVERRIDE to override default CFLAGS? I don't want to change the semantics of CFLAGS_$target again as it might break someone again.

I'd really like to avoid introducing yet another obscure env var. I'd rather revert to the old behaviour if we decide that's how we want it, I'm not too worried about the breakage from that, it's recent enough that I'm fairly sure it's only rustc's bootstrap that relies on it (and I'll fix that before landing any such revert).

@classabbyamp
Copy link
Author

classabbyamp commented May 12, 2025 •
edited
Loading

In our (Void Linux's) build system, when cross-compiling, the CFLAGS variable is set to the target's CFLAGS, as that is the primary CFLAGS programs and libraries should be using.

For various kinds of build systems, we also set:

CFLAGS_FOR_BUILD=-O2 -pipe
CFLAGS_target=-O2 -pipe -march=armv8-a  
BUILD_CFLAGS=-O2 -pipe
HOST_CFLAGS=-O2
CFLAGS=-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a   -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.
CFLAGS_host=-O2 -pipe

but CFLAGS contains all the flags that should be used for the target.

It's probably reasonable to assume that CFLAGS are for the target if something like HOST_CFLAGS or CFLAGS_host exists

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@classabbyamp @madsmtm @NobodyXu