Crash when enabling FIPS in OpenSSL

[parsley72]

Describe the bug

We've had the SDK working with our app fine, but we've recently added FIPS support to OpenSSL. When we run our app and it initialises the SDK it crashes.

Expected Behavior

The SDK should work as before.

Current Behavior

.s2n_init() failed: 402653221 (Error using Deterministic Random Bit Generator)
Fatal error condition occurred in /opt/linux-gnu/aws-sdk-cpp-1.11.404/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:228: 0 && "s2n_init() failed"
Exiting Application
########################
Stack trace:
########################
/opt/MyApp/libaws-c-common.so.1(aws_backtrace_print+0x5d) [0x7f0b7563682d] /opt/MyApp/libaws-c-common.so.1(aws_fatal_assert+0x48) [0x7f0b75614c98]
/opt/MyApp/libaws-c-io.so.1.0.0(+0x2a9ea) [0x7f0b7022a9ea]
/opt/MyApp/libaws-c-io.so.1.0.0 (aws_io_library_init+0x48) [0x7f0b7021bef8] /opt/MyApp/libaws-c-mqtt.so.1.0.0 (aws_mqtt_library_init+0x20)
[0x7f0b71217e20]
/opt/MyApp/libaws-crt-cpp.so(_ZN3Aws3Crt9ApiHandleC1EP13aws_allocator+0x44) [0x7f0b75a42b94] /opt/MyApp/libaws-cpp-sdk-core.so(_ZN3Aws13InitializeCrtEv+0x54) [0x7f0b7d902324]
/opt/MyApp/libaws-cpp-sdk-core.so(_ZN3Aws7InitAPIERKNS_10SDKOptionsE+0x1eb) [0x7f0b7d90050b]
./MyApp(+0xfabba) [0x550108fabba]
./MyApp(+0x187a96) [0x55010987a96]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0b7623fac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f0b762d1850]

Reproduction Steps

We're using Ubuntu 22 with OpenSSL v3.0.2 and we've built the OpenSSL FIPS library from v3.0.0:

$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.0.2
    status: active
  fips
    name: OpenSSL FIPS Provider
    version: 3.0.0
    status: active

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.404

Compiler and Version used

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Operating System and version

Ubuntu 22

[DmitriyMusatkin]

This might be an issue for https://github.com/aws/s2n-tls. Looks like their init is failing to validate DRBG in the fips version of openssl. Im not seeing any tickets on their side for it, so might be a bug or some sort of limitation in s2n integration with v3.0.2 (which itself is pretty old at this point)

[parsley72]

Seems that s2n-tls isn't included by this project directly. This project includes https://github.com/awslabs/aws-crt-cpp.git which then includes s2n-tls.

I'm confused because the documentation for aws-crt-cpp says:
https://github.com/awslabs/aws-crt-cpp?tab=readme-ov-file#openssl-and-libcrypto-unix-only

aws-crt-cpp does not use OpenSSL for TLS. On Apple and Windows devices, the OS's default TLS library is used. On Unix devices, s2n-tls is used. But s2n-tls uses libcrypto, the cryptography math library bundled with OpenSSL. To simplify the build process, the source code for s2n-tls and libcrypto are included as git submodules and built along with aws-crt-cpp. But if your application is also loading the system installation of OpenSSL (i.e. your application uses libcurl which uses libssl which uses libcrypto) there may be crashes as the application tries to use two different versions of libcrypto at once.
Setting -DUSE_OPENSSL=ON will cause aws-crt-cpp to link against your system's existing libcrypto, instead of building its own copy.

I think aws-sdk-cpp does set USE_OPENSSL=ON here:

aws-sdk-cpp/CMakeLists.txt

Line 50 in a35dba1

option(USE_OPENSSL "Set this if you want to use your system's OpenSSL 1.0.2/1.1.1 compatible libcrypto" ON)

So why is s2n-tls being used? This comment mentions older versions of OpenSSL, is this still ok for OpenSSL v3?

[DmitriyMusatkin]

Openssl has 2 parts to it - libssl and libcrypto. Historically, libssl had issues discovered in it periodically, while libcrypto has been fairly stabe.
CRT does not support usage of libssl at all on linux and instead always relies on s2n-tls. This behavior cannot be modified by compile flags. But at the same time multiple different flavors of libcrypto are supported as long as cpp sdk and all of its dependencies are using the same flavor of libcrypto.
aws-crt-cpp does default to aws-lc (aws implementation of libcrypto based on top of boringssl), but for cpp sdk we default it to openssl libcrypto (mostly due to majority of cpp deps built against it and mixing libcryptos can lead to "fun" segfaults).
USE_OPENSSL is a bit confusing in that it only indicates with libcrypto to use and has no effect on which ssl impl is used

[sbiscigl]

Hey thanks for reaching out. Elaborating on what Dmitriy said I spent some time looking into this and I could not reproduce. Using a custom crypto with the SDK has a lot of gotchas and without being able to view your entire environment I cannot diagnose it. However i spent time and created a workspace that will work with fips enabled openssl. I would also highly suggest looking at our developer guide

My project where it works as expected:

Project Structure:

~/sdk-usage-workspace ❯❯❯ tree -L 1
.
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

Dockerfile

# Using offical Amazon Linux 2023 image from public ECR
FROM public.ecr.aws/amazonlinux/amazonlinux:2023

# Install compiler et al.
RUN yum groupinstall "Development Tools" -y

# Install required dependencies
RUN yum install -y ninja-build cmake3 wget perl

# Install openssl in fips mode
RUN wget https://github.com/openssl/openssl/releases/download/openssl-3.0.9/openssl-3.0.9.tar.gz &&\
    tar -xf openssl-3.0.9.tar.gz &&\
    cd openssl-3.0.9 &&\
    ./Configure enable-fips --prefix=/fips-install --openssldir=/fips-install &&\
    make &&\
    make install

# Verify install
RUN cd openssl-3.0.9 &&\
    ./util/wrap.pl -fips /fips-install/bin/openssl list -provider-path providers -provider fips -providers

# Build and install curl targeting openssl with fips provider
RUN git clone --depth 1 -b curl-8_10_1 https://github.com/curl/curl && \
    cd curl && \
    autoreconf -fi && \
    mkdir build && \
    cd build && \
    cmake -DCMAKE_PREFIX_PATH="/fips-install" -DCMAKE_INSTALL_PREFIX="/curl-install" .. && \
    cmake --build . && \
    cmake --install .

# Install sdk
RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake -G Ninja -DBUILD_ONLY="core" \
      -DCMAKE_PREFIX_PATH="/curl-install;/fips-install" \
      -DAUTORUN_UNIT_TESTS=OFF .. && \
    cmake --build . && \
    cmake --install .

## Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -G Ninja -DCMAKE_PREFIX_PATH="/curl-install;/fips-install;" .. && \
    cmake --build .

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS core)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

main.cpp

#include <aws/core/Aws.h>

using namespace Aws;

auto main() -> int
{
    SDKOptions options;
    Aws::InitAPI(options);
    Aws::ShutdownAPI(options);
}

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# run example
docker run \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    -e LD_LIBRARY_PATH="/fips-install/lib;/curl-install/lib64" \
    --name test-image test-image /sdk-example/build/sdk_usage_workspace

when we run with ./replicate.sh we see that it exits successfully

~/sdk-usage-workspace ❯❯❯ docker ps -a
CONTAINER ID   IMAGE        COMMAND                  CREATED         STATUS                     PORTS     NAMES
134cb08bba23   test-image   "/sdk-example/build/…"   5 seconds ago   Exited (0) 3 seconds ago             test-image

additionally when we shell into the enviornment we can see the project setup as expected

bash-5.2# LD_LIBRARY_PATH="/fips-install/lib;/curl-install/lib64" ldd sdk_usage_workspace
	linux-vdso.so.1 (0x0000ffff81cae000)
	libaws-cpp-sdk-core.so => /usr/local/lib64/libaws-cpp-sdk-core.so (0x0000ffff81aad000)
	libaws-crt-cpp.so => /usr/local/lib64/libaws-crt-cpp.so (0x0000ffff819fb000)
	libaws-c-mqtt.so.1.0.0 => /usr/local/lib64/libaws-c-mqtt.so.1.0.0 (0x0000ffff81989000)
	libaws-c-event-stream.so.1.0.0 => /usr/local/lib64/libaws-c-event-stream.so.1.0.0 (0x0000ffff81948000)
	libaws-c-s3.so.0unstable => /usr/local/lib64/libaws-c-s3.so.0unstable (0x0000ffff818f6000)
	libaws-c-auth.so.1.0.0 => /usr/local/lib64/libaws-c-auth.so.1.0.0 (0x0000ffff818a4000)
	libaws-c-http.so.1.0.0 => /usr/local/lib64/libaws-c-http.so.1.0.0 (0x0000ffff81821000)
	libaws-c-io.so.1.0.0 => /usr/local/lib64/libaws-c-io.so.1.0.0 (0x0000ffff817be000)
	libs2n.so.1 => /usr/local/lib64/libs2n.so.1 (0x0000ffff8166e000)
	libcrypto.so.3 => /fips-install/lib/libcrypto.so.3 (0x0000ffff81228000)
	libaws-c-cal.so.1.0.0 => /usr/local/lib64/libaws-c-cal.so.1.0.0 (0x0000ffff811e7000)
	libaws-checksums.so.1.0.0 => /usr/local/lib64/libaws-checksums.so.1.0.0 (0x0000ffff811a6000)
	libaws-c-compression.so.1.0.0 => /usr/local/lib64/libaws-c-compression.so.1.0.0 (0x0000ffff81165000)
	libaws-c-sdkutils.so.1.0.0 => /usr/local/lib64/libaws-c-sdkutils.so.1.0.0 (0x0000ffff81124000)
	libaws-c-common.so.1 => /usr/local/lib64/libaws-c-common.so.1 (0x0000ffff810a2000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000ffff80e5e000)
	libm.so.6 => /lib64/libm.so.6 (0x0000ffff80dad000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000ffff80d6c000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffff80b9e000)
	libcurl.so.4 => /curl-install/lib64/libcurl.so.4 (0x0000ffff80a8a000)
	libz.so.1 => /lib64/libz.so.1 (0x0000ffff80a49000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff81c61000)
	libssl.so.3 => /fips-install/lib/libssl.so.3 (0x0000ffff80994000)

So with this we know it works as intended in a isolated environment. If I were to guess there is something incorrectly set up in your environment that is leading to the issues that you are seeing. If you can update the example above to replicate your issue we would be happy to look into but it looks to be working as intended.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[parsley72]

Strangely we found the crash went away when we used an unstripped binary for FIPS.

[jmklix]

I'm glad that you were able to get it working without crashing. Did you have questions/problems with using FIPS with this sdk?

[parsley72]

Sorry I'm late, just created #3198