Unable to cross compile for musl

[detly]

This happens the same way under cross with a custom docker image, or bare cargo on an Ubuntu env. To simplify things, I'll just give the local command (for Ubuntu 21.10). This is a GNU host, but what I want is a statically-linked-with-musl-libc binary.

Assume I've done sudo apt install pkg-config libssl-dev musl-tools.

⚬ OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu/ OPENSSL_INCLUDE_DIR=/usr/include/ OPENSSL_STATIC=1 cargo build --target x86_64-unknown-linux-musl
   Compiling openssl-sys v0.9.72
   Compiling socket2 v0.4.4
   Compiling signal-hook-registry v1.4.0
   Compiling atty v0.2.14
   Compiling mio v0.8.2
   Compiling idna v0.2.3
   Compiling syn v1.0.92
   Compiling nix v0.23.1
   Compiling regex v1.5.5
The following warnings were emitted during compilation:

warning: build/expando.c:2:10: fatal error: openssl/opensslconf.h: No such file or directory
warning:     2 | #include <openssl/opensslconf.h>
warning:       |          ^~~~~~~~~~~~~~~~~~~~~~~
warning: compilation terminated.

error: failed to run custom build command for `openssl-sys v0.9.72`

Caused by:
  process didn't exit successfully: `/home/jason/Code/[...]/target/debug/build/openssl-sys-51dce0970b29e049/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR = /usr/lib/x86_64-linux-gnu/
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR = /usr/include/
  cargo:rustc-link-search=native=/usr/lib/x86_64-linux-gnu/
  cargo:include=/usr/include/
  cargo:rerun-if-changed=build/expando.c
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-musl")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-unknown-linux-musl = None
  CC_x86_64_unknown_linux_musl = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-musl = None
  CFLAGS_x86_64_unknown_linux_musl = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "/usr/include/" "-Wall" "-Wextra" "-E" "build/expando.c"
  cargo:warning=build/expando.c:2:10: fatal error: openssl/opensslconf.h: No such file or directory
  cargo:warning=    2 | #include <openssl/opensslconf.h>
  cargo:warning=      |          ^~~~~~~~~~~~~~~~~~~~~~~
  cargo:warning=compilation terminated.
  exit status: 1

  --- stderr
  thread 'main' panicked at '
  Header expansion error:
  Error { kind: ToolExecError, message: "Command \"musl-gcc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-fno-omit-frame-pointer\" \"-m64\" \"-I\" \"/usr/include/\" \"-Wall\" \"-Wextra\" \"-E\" \"build/expando.c\" with args \"musl-gcc\" did not execute successfully (status code exit status: 1)." }

  Failed to find OpenSSL development headers.

  You can try fixing this setting the `OPENSSL_DIR` environment variable
  pointing to your OpenSSL installation or installing OpenSSL headers package
  specific to your distribution:

      # On Ubuntu
      sudo apt-get install libssl-dev
      # On Arch Linux
      sudo pacman -S openssl
      # On Fedora
      sudo dnf install openssl-devel

  See rust-openssl README for more information:

      https://github.com/sfackler/rust-openssl#linux
  ', /home/jason/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.72/build/main.rs:162:13
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

Fundamentally the problem seems to be:

• opensslconf.h is at /usr/include/x86_64-linux-gnu/openssl/opensslconf.h
• other OpenSSL headers are at /usr/include/openssl

So there is no way to set OPENSSL_INCLUDE_DIR to a value that satisfies this, and pkg-config doesn't seem to help.

This seems similar to #603, but (a) I would be happy with a statically linked OpenSSL, and I don't think they're doing that and (b) I can't even do this via cross with a custom image containing OpenSSL, I get the same error.

[sfackler]

The system's copy of OpenSSL is built against glibc, not musl. If you want a fully static binary, you need to build OpenSSL against musl as well. You could do this yourself or by enabling the vendored Cargo feature.

[detly]

You are right of course, but for this project openssl is a transitive dependency. I don't want to add it as a primary dependency just to enable this feature, especially since it builds fine in CI (what I am trying to do is document development processes as well). The solution I have grudgingly settled on is a convoluted process involving docker to do the cross-compile.

[defi-degaulle]

@detly mind sharing some of the details?

[detly]

@defi-degaulle I haven't tried to reproduce this recently, so you might need to work on this a bit, but basically:

• start from muslrust (I used 1.62.0-nightly-2022-05-02)

• customise it if you need to (eg. with cargo-chef and other build deps you need)

• build with a command that's something like

  docker run -it --rm --mount type=bind,src="$(pwd)",dst="/volume" muslrust-custom env CARGO_MANIFEST_DIR=/volume CARGO_TARGET_DIR="/volume/target-musl" cargo build
  

Presto, the properly linked binary should appear in target-musl/.

[seeekr]

for anyone stumbling across issues like this: If you just want to compile with musl, but openssl is getting in the way, another tip is that most Rust libraries have features that allow you to select rustls over openssl (also called "native-tls" frequently), which can be another easy solution to the openssl/musl compilation problem. (Sometimes you need to set default-features = false for a dependency and then manually enable the features you need, including for rustls, e.g. with the reqwest crate: default-features = false, features = ["rustls", ... .)

[halvors]

For reference i was able to link against openssl in Alpine both statically and dynamically using this Dockerfile
I previously was able to link against it dynamically without any workaround on Rust 1.71.1, but on Rust 1.72.1 it requires the workaround below.

To link statically against openssl from alpine

FROM rust:1.72.1-alpine3.18

ARG BUILD_DIR=/tmp

RUN set -x && \
    apk add --no-cache musl-dev openssl-dev openssl-libs-static

# statically link against openssl
ENV OPENSSL_STATIC=1 

COPY ./ ${BUILD_DIR}

RUN set -x && \
    cd ${BUILD_DIR} && \
    cargo build --release

To link dynamically from openssl from alpine (NOTE: you need the "libgcc" package installed in runtime container)

FROM rust:1.72.1-alpine3.18

ARG BUILD_DIR=/tmp

RUN set -x && \
    apk add --no-cache musl-dev openssl-dev

# dynamically link against openssl (libgcc package required in runtime container)
# see: https://rust-lang.github.io/rfcs/1721-crt-static.html
ENV RUSTFLAGS="-C target-feature=-crt-static"

COPY ./ ${BUILD_DIR}

RUN set -x && \
    cd ${BUILD_DIR} && \
    cargo build --release

[polarathene]

ENV OPENSSL_STATIC=1

Your requirement for it might be due to something else going on? I could not reproduce with rust:1.72.1-alpine3.18.

EDIT: For reference:

• This was likely due to a bug with +crt-static that affected the x86_64-unknown-linux-musl target (and any other targets that default to static linking).
• It was resolved in Oct 2023 with Rust 1.72 (which raised several bug reports about compilation failures with this release, but those reporters would find out via ldd that previous builds were unintentionally dynamically linked)

Static vs Dynamic link reproduction

Minimal example with openssl crate directly via shell commands / script for troubleshooting, rather than the Dockerfile examples above.

# Rust alpine container with extra deps added:
$ docker run --rm -it rust:alpine ash
$ apk add musl-dev openssl-dev openssl-libs-static

# Minimal project example:
$ cargo init /example && cd /example
$ cargo add openssl

# All one command (can paste into terminal without needing `\`)
$ cat > src/main.rs <<EOF
fn main() {
    println!("Version: {}", openssl::version::version());
}
EOF

# Check builds successfully:
$ cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023

# Check built binary is static linked:
$ apk add file
$ file target/release/example
target/release/example: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=7f5e22856b30e8134301eb470397c1894d90890a, with debug_info, not stripped

# Nothing is linked (musl ldd outputs ld-musl regardless, can ignore):
$ ldd target/release/example
/lib/ld-musl-x86_64.so.1 (0x7fdcc5a0e000)

# vs dynamic link:
$ RUSTFLAGS='-C target-feature=-crt-static' cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023

$ file target/release/example
target/release/example: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, BuildID[sha1]=d35b6310ac85417c2dbb64ee599e1c7eb745e152, with debug_info, not stripped

$ ldd target/release/example
/lib/ld-musl-x86_64.so.1 (0x7f3f14586000)
libssl.so.3 => /lib/libssl.so.3 (0x7f3f144a9000)
libcrypto.so.3 => /lib/libcrypto.so.3 (0x7f3f1408c000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f3f14068000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f3f14586000)

---

OPENSSL_STATIC=1 when necessary to avoid Segmentation fault

I have however observed that OPENSSL_STATIC=1 is required if using the ENV to configure OPENSSL_DIR:

Avoiding "Segmentation fault" with `OPENSSL_STATIC=1`

$ OPENSSL_DIR=/usr cargo run --release
Segmentation fault

$ OPENSSL_STATIC=1 OPENSSL_DIR=/usr cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023

# Ignored:
$ OPENSSL_LIB_DIR=/etc cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023
$ OPENSSL_INCLUDE_DIR=/etc cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023

# Segmentation fault when used together (/usr/lib is important, /usr/include/openssl can be any path):
$ OPENSSL_INCLUDE_DIR=/usr/include/openssl OPENSSL_LIB_DIR=/usr/lib cargo run --release
Segmentation fault

# Success (despite include path not being correct, lib path must remain correct):
$ OPENSSL_STATIC=1 OPENSSL_INCLUDE_DIR=/etc OPENSSL_LIB_DIR=/usr/lib cargo run --release
Version: OpenSSL 3.1.4 24 Oct 2023

• AFAIK OPENSSL_INCLUDE_DIR isn't relevant for the static builds since we just need the lib dir for OPENSSL_STATIC=1?
• I'm not sure why OPENSSL_DIR or OPENSLS_INCLUDE_DIR (paired with OPENSSL_LIB_DIR) cause a segmentation fault unless setting OPENSSL_STATIC=1 🤷‍♂️ (this comment suggests they're only intended to be used with OPENSSL_STATIC=1?)

When on a host that isn't Alpine, such as Ubuntu with glibc, you can create a static glibc build with -gnu target with OPENSSL_STATIC=1 and the libssl-dev package by pairing that ENV with OPENSSL_LIB_DIR + OPENSSL_INCLUDE_DIR to the correct locations.

For Fedora there doesn't appear to be any equivalent package support for libssl.a that is AFAIK I needed. So you must either dynamic link or build from source (aka vendored), otherwise static builds will fail as the link demonstrates. OPENSSL_STATIC=1 support can also fail when the locations are considered system libs, preferring to dynamic link instead without an opt-out for this behaviour.

Regardless, for cross-compilation the OPENSSL_STATIC=1 isn't useful unless you have openssl built for that target to statically link against. vendored is useful for this again (although there doesn't appear to be an equivalent ENV for opt-in, like there is for opt-out with OPENSSL_NO_VENDOR=1).

---

OPENSSL_NO_VENDOR=1 to opt-out of crates that enabled vendored feature

One issue I have encountered that may be worth adding here (although not musl specific AFAIK), some projects may have added openssl crate with the vendored feature enabled (or indirectly through another crate, like native-tls with it's own vendored forwarded feature).

You can override that when building via the OPENSSL_NO_VENDOR=1 ENV, I'm not sure what else would otherwise be needed to get that to avoid the build failure I encountered when using the rust:alpine image.

Build from source via vendored feature (cross-compile compatible)

UPDATE: The rust:alpine concern I mentioned above was because the vendored feature builds from source.. while the error output of the build failure wasn't communicating that very well. It just needed the perl package (possibly a few others depending on build host), which the crate docs I linked for the vendored feature does communicate well.

# Alpine (musl)
docker run --rm -it rust:alpine ash
cargo init /example && cd /example
cargo add openssl --features vendored

# Extra packages required to build for vendored (_`openssl-dev` + `openssl-libs-static` not required since `vendored` builds from source_):
# NOTE: When building from source (via the "vendored" feature), some builds may also require the package `linux-headers`
apk add make musl-dev perl
cargo build --release

# Debian (glibc)
docker run --rm -it rust:latest bash
cargo init /example && cd /example
cargo add openssl --features vendored

# This image already has `make` and `perl` installed, so works fine out of the box:
cargo build --release

# Cross compile to musl target:
rustup target add x86_64-unknown-linux-musl
# Needs `musl-gcc` package to successfully build, `musl-tools` provides `musl-gcc` command + `musl-dev` package:
apt update && apt install musl-tools
cargo build --release --target x86_64-unknown-linux-musl

# Static gnu target build:
RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target x86_64-unknown-linux-gnu

# Another glibc image example instead of an official Rust image:
docker run --rm -it fedora:40 bash
# `gcc` will bring in `make`:
dnf install -y gcc perl rustup
rustup-init -y && source "$HOME/.cargo/env"

# Prepare project
cargo init /tmp/example && cd /tmp/example
cargo add openssl --features vendored
echo 'fn main() { println!("OpenSSL version is: {}", openssl::version::version()); }' > src/main.rs

# Works:
cargo build --release --target x86_64-unknown-linux-gnu

# Static build needs extra package:
dnf install -y glibc-static
RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target x86_64-unknown-linux-gnu

# Cross compile to musl target requires `musl-gcc` package (`musl-gcc` command + `musl-devel` package):
rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl

There is also another scenario where a proc macro (built on host, separate from build target) may require adding openssl package with the vendored feature in [build-dependencies] of Cargo.toml (similar info here with a different workaround using separate target ENV).

---

Cross-compiling via xx

For those interested in cross compiling rust with Docker, you might be interested in the docker init command which has a Rust template with xx for building each target (cross compiling different archs may be blocked on this PR being merged).

xx is a bit more hands-on than cross as you're managing your cross-compiling Dockerfile directly and having it run the xx-* commands within the image build, rather than cross command from the host managing Docker.

---

Alternatively, cargo-zigbuild is useful here. It can cross-compile to different platforms than the host. Normally a x86_64 host cannot build for a aarch64 target when building openssl, hence the need for xx / cross tooling, but cargo zigbuild makes this much more accessible.

I have some minimal examples with Fedora and Ubuntu here as Docker container reproductions. You need the cargo-zigbuild command + zig (Fedora provides a package, while a common install method elsewhere is via Python pip, you can alternatively curl a release and extract that, adding the location to your PATH). Add your target and zig will handle the C/C++ compiles similar to Rust for cross-platform compilation 🎉