significant mul_add perf regression since nightly-2025-03-06

[sarah-quinones]

rust uses compiler_builtins fma instead of the one from libc since nightly 2025-03-06 (2025-03-05 is fine)

my suspicion is linkage changes somewhere but i haven't been following the changes in compiler_builtins

code

i ran this code under perf on linux x86_64 (perf record -g cargo run)

the code loops forever so i interrupt it manually after a couple seconds

use core::hint::black_box as bb;

fn main() {
    loop {
        bb(f64::mul_add(1.0, 1.0, bb(1.0)));
    }
}

on 2025-03-05, the profiling (perf report -g) shows that most of the time is spent in __fma_fma3. on 2025-03-06 it's now spending most of the time in compiler_builtins::math::libm::fma::fma, which is a lot slower since it emulates fma with the limited sse instructions instead of vfmadd231sd

version it worked on

nightly-2025-03-05

version with regression

nightly-2025-03-06

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

[nikic]

@tgross35

[tgross35]

Could you clarify what flags this was built with if any?

The idea was that LLVM should usually be able to lower the intrinsics directly to asm if known to be available. If it just happens to be missing a case that it should know about, then we can add it with asm in libm for now. But if this is one of the cases where glibc is using ifuncs to select features at runtime, things get a bit less clear.

Here is the full list of functions that will likely be using our version now, for reference https://github.com/rust-lang/compiler-builtins/blob/fdbefb39d5bb0b95b29b821247044c8aaf436160/compiler-builtins/src/math/mod.rs#L21

[Amanieu]

It's using ifunc to select an implementation: https://github.com/bminor/glibc/blob/77930e0447e0b37a129db0e13c6c6f5e60a3019e/sysdeps/x86_64/fpu/multiarch/s_fma.c#L49

[tgross35]

@Amanieu is there any reasonable way to do this in our libm? We could do runtime feature detection via cpuid, I’m not sure of any easy way to emulate ifuncs.

[tgross35]

Double checking, with +fma the single instruction is used https://rust.godbolt.org/z/YzTfT5Poa

[sarah-quinones]

what one could do is make fma a static AtomicPtr that gets initialized with a function that updates the pointer, then returns the result

subsequent calls will use the updated pointer value which can go directly to the vfmadd path. i don't know how this compares to ifuncs in terms of perf but it should be better than the status quo

[sarah-quinones]

Could you clarify what flags this was built with if any?

no specific target feature flags. i don't want to rely on target-cpu=native

[tgross35]

Candidate patch so hopefully we can avoid reverting anything rust-lang/compiler-builtins#896

[tgross35]

Builtins version sync #140648