Custom cc toolchain with libunwind creates link problems

[criemen]

Hi,

I'm using a custom (bundled) cpp toolchain on Linux, which ships (besides other things) LLVM's libunwind.
I was excited to see that my bundled toolchain is also used for rust compilations - both as part of dependant C/C++ code, and for linking rust executables. This is key for me, as we're shipping an old glibc we want to link against, so our executables are compatible a wide range of distros.

However, I'm running into a problem already when building the process wrapper included in rules_rust:

The rustc invocation coming out of rules_rust looks like this:

  bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.sh -- bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/bin/rustc external/rules_rust/util/process_wrapper/main.rs '--crate-name=process_wrapper' '--crate-type=bin' '--error-format=human' '--out-dir=bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper' '--codegen=opt-level=3' '--codegen=debuginfo=0' '--remap-path-prefix=${pwd}=' '--emit=dep-info,link' '--color=always' '--target=x86_64-unknown-linux-gnu' -L bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib '-Cstrip=debuginfo' '--edition=2018' '--codegen=linker=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/clang' --codegen 'link-args=-Wl,--hash-style=both -m64 -static-libgcc --rtlib=compiler-rt --unwindlib=libunwind -fuse-ld=lld --ld-path=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/ld.lld -ldl -lpthread -lm -pthread -ldl --sysroot=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain -l:libc++.a' '--extern=tinyjson=bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib' '-Ldependency=bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust_tinyjson' '--sysroot=bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain')

Note that it passes all the linker options from my custom toolchain, included --unwindlib=libunwind and -static-libgcc to rustc's --codegen 'link-args... option.

rustc invokes clang as

  = note: LC_ALL="C" PATH="bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/bin:/bin:/usr/bin:/usr/local/bin" VSLANG="1033" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/clang" "-m64" "/tmp/rustcwA6AdR/symbols.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.0.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.1.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.10.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.11.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.12.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.13.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.14.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.15.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.2.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.3.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.4.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.5.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.6.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.7.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.8.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.process_wrapper.1b271f66-cgu.9.rcgu.o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper.2omey7wc3bx5ld07.rcgu.o" "-Wl,--as-needed" "-L" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust_tinyjson" "-L" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/execroot/semmle_code/bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-bc6b80525d6b1f3b.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-dbb416fff97e9855.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-af60be54961a030f.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-1303bc5098cb2f44.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-96ca4807f9d03fdf.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-03b108942351d49a.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-b348df34b7d8ac11.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-26a06d9c5ec29d3a.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-b5295fdab67e4cf6.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-b257ed099e7f67d0.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-c27b5dca54e295d8.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-f6c8245d52afa66d.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-642c68f15c02cc52.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-eecd84150c4ad967.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-123ffa13a38501db.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-2177aff67f4e9999.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-2298a66e03bd0fd2.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-f3c3b25345711552.rlib" "/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-10f041ff25bad5f3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs" "-Wl,--hash-style=both" "-m64" "-static-libgcc" "--rtlib=compiler-rt" "--unwindlib=libunwind" "-fuse-ld=lld" "--ld-path=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/ld.lld" "-ldl" "-lpthread" "-lm" "-pthread" "-ldl" "--sysroot=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain" "-l:libc++.a"

which fails with

  = note: clang: warning: argument unused during compilation: '-static-libgcc' [-Wunused-command-line-argument]
          clang: warning: argument unused during compilation: '--unwindlib=libunwind' [-Wunused-command-line-argument]
          clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
          ld.lld: error: unable to find library -lgcc_s
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

This is because before passing in our (custom) link flags, we're getting a bunch of linker flags supplied from somewhere else (I've not been able to find out where, despite searching in rust-lang/rust), including -lgcc_s, and the linker also (now correctly) warns that some of our options aren't used.
As our toolchain uses libunwind for unwinding, libgcc_s is not present in the toolchain, and the linker complains correctly about its absence.

Is there any way to tell rustc not to insert these linker flags?

The only really hacky workaround I see here is a linker-wrapper script that detects being invoked from rustc, and then filters out the harmful CLI arguments, but if there's any other chance of solving this, I really don't want to go there.

[daivinhtran]

Have you tried enabling --experimental_use_cc_common_link flag? This avoids setting link-args and use cc_common.link which pull the registered cc_toolchain for linking instead.

[criemen]

Thanks a lot!
I had set this on my rust_binary, but obviously that doesn't help with the process wrapper.
Also, my linkerflags (and clang executable used for linking) do come from my custom toolchain, so I hadn't thought that there's one step further.
The process wrapper links, and rules_rust gets to compiling crates instead.

Now it gets further, but at

ERROR: /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/python_tsg_crate_index__thiserror-impl-1.0.29/BUILD.bazel:20:16: Compiling Rust proc-macro thiserror_impl v1.0.29 (8 files) [for tool] failed: (Exit 1): process_wrapper failed: error executing command (from target @python_tsg_crate_index__thiserror-impl-1.0.29//:thiserror_impl) 
  (cd /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/sandbox/linux-sandbox/61/execroot/semmle_code && \
  exec env - \
    CARGO_CFG_TARGET_ARCH=x86_64 \
    CARGO_CFG_TARGET_OS=linux \
    CARGO_CRATE_NAME=thiserror_impl \
    CARGO_MANIFEST_DIR='${pwd}/external/python_tsg_crate_index__thiserror-impl-1.0.29' \
    CARGO_PKG_AUTHORS='' \
    CARGO_PKG_DESCRIPTION='' \
    CARGO_PKG_HOMEPAGE='' \
    CARGO_PKG_NAME=thiserror_impl \
    CARGO_PKG_VERSION=1.0.29 \
    CARGO_PKG_VERSION_MAJOR=1 \
    CARGO_PKG_VERSION_MINOR=0 \
    CARGO_PKG_VERSION_PATCH=29 \
    CARGO_PKG_VERSION_PRE='' \
    PATH=/bin:/usr/bin:/usr/local/bin \
    SYSROOT=bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain \
  bazel-out/k8-opt-exec-A914AEDA/bin/external/rules_rust/util/process_wrapper/process_wrapper --arg-file bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__proc-macro2-1.0.29/proc-macro2_build_script.linksearchpaths --arg-file bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__syn-1.0.76/syn_build_script.linksearchpaths --subst 'pwd=${pwd}' -- bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/bin/rustc external/python_tsg_crate_index__thiserror-impl-1.0.29/src/lib.rs '--crate-name=thiserror_impl' '--crate-type=proc-macro' '--error-format=human' '--codegen=metadata=-40918192' '--codegen=extra-filename=-40918192' '--out-dir=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__thiserror-impl-1.0.29' '--codegen=opt-level=3' '--codegen=debuginfo=0' '--remap-path-prefix=${pwd}=' '--emit=dep-info,link' '--color=always' '--target=x86_64-unknown-linux-gnu' -L bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib '--cap-lints=allow' '--edition=2018' '--codegen=linker=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/clang' --codegen 'link-args=-shared -Wl,--hash-style=both -m64 -static-libgcc --rtlib=compiler-rt --unwindlib=libunwind -fuse-ld=lld --ld-path=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain/bin/ld.lld -ldl -lpthread -ldl -lpthread -ldl -lpthread -lm -pthread -ldl --sysroot=/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/cc_linux_toolchain -l:libc++.a' '--extern=proc_macro2=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__proc-macro2-1.0.29/libproc_macro2-1905888690.rlib' '--extern=quote=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__quote-1.0.9/libquote-1287025313.rlib' '--extern=syn=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__syn-1.0.76/libsyn-1120817977.rlib' '-Ldependency=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__unicode-xid-0.2.2' '-Ldependency=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__proc-macro2-1.0.29' '-Ldependency=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__quote-1.0.9' '-Ldependency=bazel-out/k8-opt-exec-A914AEDA/bin/external/python_tsg_crate_index__syn-1.0.76' --extern proc_macro '--sysroot=bazel-out/k8-opt-exec-A914AEDA/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__stable_tools/rust_toolchain')
# Configuration: 31b3dcb0a39a5ac59b9fa50512a4935dc9aed1c13cf3c28eb45bf004176494c3
# Execution platform: //toolchain/platforms:bundled_toolchain_x86_64

it still fails with the same clang error.
Is there anything specific about the compilation of proc-macros that the codepath using cc_common.link isn't being used there?

[daivinhtran]

Would you mind sharing how your rust toolchains are set up? One problem I ran into before is that host toolchain needs to match with @local_config_platform//:host for it to build pro-macro successfully.

Now that you separate linking into a separate action by enabling --experimental_use_cc_common_link, the problem is at the compiling step (aka rustc) which is before cc_common.link is necessary.

[criemen]

This is indeed a bug (I believe):

rules_rust/rust/private/rustc.bzl

Line 1147 in bbe2cf4

if hasattr(ctx.attr, "experimental_use_cc_common_link"):

checks whether the rule context has the experimental_use_cc_common_link attribute, and if it doesn't, cc_common.link is never considered even a possibility.
Checking rust_proc_macro in https://github.com/bazelbuild/rules_rust/blob/main/rust/private/rust.bzl#L985, we see that it's lacking _experimental_use_cc_common_link_attrs in its attrs, and therefore, the aforementioned check is always false.

Now I don't know your codebase well enough, whether _experimental_use_cc_common_link_attrs should be included in more rule definitions, or whether the if check should be relaxed to take the value from the toolchain, regardless of individual rule support. I'd assume the former.

(my rust toolchain setup is rather boring, I'm downloading a stable 1.62.2 with the auto-setup toolchain)

[daivinhtran]

Hmm you're right. If you force experimental_use_cc_common_link to True

rules_rust/rust/private/rustc.bzl

Lines 1146 to 1153 in bbe2cf4

	experimental_use_cc_common_link = False
	if hasattr(ctx.attr, "experimental_use_cc_common_link"):
	if ctx.attr.experimental_use_cc_common_link == 0:
	experimental_use_cc_common_link = False
	elif ctx.attr.experimental_use_cc_common_link == 1:
	experimental_use_cc_common_link = True
	elif ctx.attr.experimental_use_cc_common_link == -1:
	experimental_use_cc_common_link = toolchain._experimental_use_cc_common_link

), what will happen?

[criemen]

Unfortunately, it's not so easy.
If I just flip that boolean, all sorts of things break, as the necessary attrs for the malloc implementation are missing, and those are accessed later unconditionally.

However, adding the attrs more targeted doesn't help either:
It seems that experimental_use_cc_common_link doesn't support any kind of library linking.
I.e. flipping that boolean already breaks rust_library_without_process_wrapper, and adding the attrs to the respective rules doesn't help.

The problem here is that rust_proc_macro also (effectively) compiles a library. The reason proc_macro fails and not something else earlier is that building static (rlib) libraries with rustc works, and doesn't require my custom toolchain, but proc_macro is built into a shared library (so), presumably so that the compiler can load it (I know very little actual Rust, I'm just the build engineer).

When enabling the code path just for rules_proc_macro, I get

ERROR: /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/python_tsg_crate_index__thiserror-impl-1.0.29/BUILD.bazel:20:16: in rust_proc_macro rule @python_tsg_crate_index__thiserror-impl-1.0.29//:thiserror_impl: 
/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rules_rust/rust/private/rust.bzl:121:5: The following files have no generating action:
external/python_tsg_crate_index__thiserror-impl-1.0.29/libthiserror_impl-40918192.so
ERROR: /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/python_tsg_crate_index__thiserror-impl-1.0.29/BUILD.bazel:20:16: Analysis of target '@python_tsg_crate_index__thiserror-impl-1.0.29//:thiserror_impl' failed

as that code path doesn't support building (?) and linking shared libraries. The whole rustc_compile_action code is somewhat convoluted, and I don't yet understand how it all hangs together, so I don't really know how to fix that.

I can workaround this whole problem by specifying a separate exec toolchain that doesn't inject all these linker flags (as proc macros get compiled for the host, not the target), but that comes at the cost of maintaining that separate toolchain, and a (potentially) slow autodetect. I'd rather not do that.

[daivinhtran]

It is typical (always) that we have at least two toolchains in place: one for target and another for exec. The exec toolchain is responsible for building things like process_wrapper.

These toolchains are resolved automatically by Bazel. See https://bazel.build/extending/toolchains#toolchain-resolution.

So I do not expect an abnormal slow down when you have 2 toolchains to resolve.

Can you elaborate why adding a exec toolchain will slow down the build?

[criemen]

I know - we just usually have target=exec, so we configure the same C++ toolchain here, which is a self-contained toolchain we build ourselves.
That toolchain works for both the target and exec configurations, except in conjunction with rules_rust, as rustc linking together with the linker flags from our self-contained toolchain is broken (as discussed above).

Now there are two ways forward:

1. Disable rustc linking everywhere, which is what experimental_use_cc_common_link seems to be for. This works great, but is limited by the lack of support for experimental_use_cc_common_link for linking dynamic shared libraries (i.e. the result of a rust_proc_macro rule).
2. Configure a second (exec) toolchain that is different from the host toolchain, and doesn't interject all the incompatible linker flags. For my testing, I tried using the standard autodetected C++ toolchain. This has the drawbacks of breaking build hermeticity (after all, we worked hard on our C++ toolchain to avoid depending on the host tools!), and slows the build down, as we now need to run the toolchain autodetection code which we otherwise could avoid.

[daivinhtran]

(I think) I see what you mean now. If you already have a host toolchain, you shouldn't need another one. In this case since target=exec (always?), the toolchain is just building host tools, correct?

This works great, but is limited by the lack of support for experimental_use_cc_common_link for linking dynamic shared libraries (i.e. the result of a rust_proc_macro rule).

I believe this is something we can easily add. rust_proc_macro might just have missed it initially.

If you don't mind, can you run the build with --toolchain_resolution_debug=@bazel_tools//tools/cpp:toolchain_type and --toolchain_resolution_debug=@rules_rust//rust:toolchain_type to see the resolutions?

[criemen]

In this case since target=exec (always?), the toolchain is just building host tools, correct?

Yes exactly. The peculiarity here (why this is even a viable workaround!) is that rust_proc_macro compilese for the exec environment, not target, as it builds a compiler plugin, and the compiler obviously runs on the host (i.e. exec).

I believe this is something we can easily add. rust_proc_macro might just have missed it initially.

That'd be great! I couldn't figure it out myself, but with a pointer or two about how to implement this in rustc_compile_action, I could also take a stab at this if you want.
cargo_build_script might be another macro that is missing support for this btw.

I'll supply the toolchain resolutions on Monday.

[daivinhtran]

Can you try applying this patch to rules_rust repo locally?

diff --git a/rust/private/rustc.bzl b/rust/private/rustc.bzl
index eff542eb..49b259d5 100644
--- a/rust/private/rustc.bzl
+++ b/rust/private/rustc.bzl
@@ -1150,6 +1150,8 @@ def rustc_compile_action(
             experimental_use_cc_common_link = True
         elif ctx.attr.experimental_use_cc_common_link == -1:
             experimental_use_cc_common_link = toolchain._experimental_use_cc_common_link
+    elif crate_info.type == "proc-macro":
+        experimental_use_cc_common_link = toolchain._experimental_use_cc_common_link

[criemen]

That will lead to the problem of the missing malloc attr. If I add the whole attr set to the rule, then that yields

ERROR: /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/python_tsg_crate_index__thiserror-impl-1.0.29/BUILD.bazel:20:16: in rust_proc_macro rule @python_tsg_crate_index__thiserror-impl-1.0.29//:thiserror_impl: 
/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rules_rust/rust/private/rust.bzl:121:5: The following files have no generating action:
external/python_tsg_crate_index__thiserror-impl-1.0.29/libthiserror_impl-40918192.so
ERROR: /home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/python_tsg_crate_index__thiserror-impl-1.0.29/BUILD.bazel:20:16: Analysis of target '@python_tsg_crate_index__thiserror-impl-1.0.29//:thiserror_impl' failed

showing that the code is somehow not ready to link dynamic (shared) libraries.

[daivinhtran]

Ok yeah. This seems like a bug in the rules. I'll dig more into this on Monday. In the meantime, if you can provide some steps to reproduce the issue, I appreciate it.

[criemen]

There's a full repro in https://github.com/criemen/rules_rust_bug_repro, which configures a C++ toolchain that doesn't ship libgcc_s.so.1, which in turn exhibits the problem reported in this issue. This is as close as I can get without providing our (quite involved) custom toolchain setup and toolchain binaries.
For the toolchain to work, I had to install ncurses5-compat-libs on Arch, on Ubuntu you might need libtinfo5.
Otherwise, the example works out of the box.

There's some options in .bazelrc you can tweak to observe different behaviour - i.e. if you disable the cc_common.link linking, compilation will fail much earlier, and if you disable registering additional toolchain, compilation will work.
I assume that with a patch that allows rust_proc_macro to use the C++ toolchain linker, that that project would compile without errors.

When building that workspace with the current version of rules_rust and

bazelisk build -j 8 //:hello_world

I get

  = note: ld.lld: error: external/org_chromium_sysroot_linux_x64/usr/lib/gcc/x86_64-linux-gnu/6/libgcc_s.so:4: unable to find libgcc_s.so.1
          >>> GROUP ( libgcc_s.so.1 -lgcc )
          >>>         ^
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

as described in the bug report above, too.

[criemen]

I investigated this more, and now I don't know what to do anymore:
A preliminary fix for linking dylib and proc-macro with cc_common.link can be found here: https://github.com/criemen/rules_rust/tree/criemen/fix-proc-macro

However, the fix doesn't work.
In order for the compiler to load the proc-macro so, it needs to have a special .rustc section, as also documented by https://rustc-dev-guide.rust-lang.org/backend/libs-and-metadata.html#dylib.
This section is not present in the output of rustc --emit obj as far as I can see, and therefore, the result of linking the .so from my fix can't be loaded as a compiler plugin by rustc.
To me, it seems that rustc wants to stay in charge of producing libraries that are then used again by rust code. I don't see a way around this, unless we can coax rustc to provide an object file with the rustc section for the crate, so that we can provide that to the C++ linker.

There's a long, somewhat relevant discussion about linking C++ and rust, and not having rustc as the final linker here, but it doesn't speak to creating rust crates with a non-rustc linker, which seems to be unsupported, and which is the issue here.

My takeaway from this is that it's not possible to support proc-macro (or dylib, should rules_rust ever support that) together with cc_common.link, and I need rather make my toolchain (somehow) compatible with rustc driving linking - either by providing libgcc_s.so in our selfcontained toolchain, or by overwriting the exec toolchain for cpp.