Skip to content

What’s New in AWS SDK for CPP Version 1.8

Jump to bottom
Pushen Wang edited this page May 4, 2020 · 1 revision

Getting Started with AWS SDK for C++ Version 1.8 Developer Preview

The following is a list of changes for version 1.8 of the AWS SDK for C++. Items with an * include a breaking change from the previous version.:

  1. ENABLE_CURL_LOGGING is now ON by default in CMake. *
  2. ENABLE_UNITY_BUILD is now ON by default in CMake. *
  3. The deprecated version of function MakeRequest() was removed. This function takes a reference to an HttpRequest object as its argument, which is unsafe. *
  4. Client configuration now reads environment variables, configuration file and EC2 metadata to get the default AWS region*
  5. Exceptions may include more service and operation specific details now.
  6. New pseudo region: aws-global to make cross region requests. *
  7. S3 client in us-east-1 now uses regional endpoint by default. *
  8. Improvements on the underlying HTTP client override.

Turn on ENABLE_UNITY_BUILD by default

When ENABLE_UNITY_BUILD is turned on, most SDK libraries will be built as a single, generated .cpp file. This can significantly reduce static library size as well as speed up compilation time.

The following examples compare the compile time and binary size with ENABLE_UNITY_BUILD set to ON and OFF. First, with the following cmake flags to turn off ENABLE_UNITY_BUILD and then build it:

cmake <path-to-source> -DENABLE_UNITY_BUILD=OFF -DBUILD_ONLY=s3 -DBUILD_SHARED_LIBS=OFF
make -j8

It takes around 280 seconds and the binary size of libaws-cpp-sdk-s3.a is 9.3MB. And then turn on this option:

cmake <path-to-source> -DENABLE_UNITY_BUILD=ON -DBUILD_ONLY=s3 -DBUILD_SHARED_LIBS=OFF
make -j8

It takes around 145 seconds and the binary size of libaws-cpp-sdk-s3.a is 4.6 MB.

The result depends on the build machine, but it can still show the differences here.

Turn on ENABLE_CURL_LOGGING by default

When ENABLE_CURL_LOGGING is turned on, Curl's internal log will be piped to the SDK's logger if the logging level is greater than or equal to DEBUG. As this is turned on by default in version 1.8, your logs will be similar to the following:

[DEBUG] 2020-04-07 19:58:06.792 CURL [0x10d6c65c0] (Text) Rebuilt URL to: https://s3.amazonaws.com/
[DEBUG] 2020-04-07 19:58:06.824 CURL [0x10d6c65c0] (Text)   Trying 52.216.29.46...
[DEBUG] 2020-04-07 19:58:06.824 CURL [0x10d6c65c0] (Text) TCP_NODELAY set
[DEBUG] 2020-04-07 19:58:06.924 CURL [0x10d6c65c0] (Text) Connected to s3.amazonaws.com (52.216.29.46) port 443 (#0
[DEBUG] 2020-04-07 19:58:06.924 CURL [0x10d6c65c0] (Text) ALPN, offering h2
[DEBUG] 2020-04-07 19:58:06.924 CURL [0x10d6c65c0] (Text) ALPN, offering http/1.1
[DEBUG] 2020-04-07 19:58:06.924 CURL [0x10d6c65c0] (Text) Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
[DEBUG] 2020-04-07 19:58:06.931 CURL [0x10d6c65c0] (Text) successfully set certificate verify locations
[DEBUG] 2020-04-07 19:58:06.931 CURL [0x10d6c65c0] (Text)   CAfile: /etc/ssl/cert.pem
  CApath: none
[DEBUG] 2020-04-07 19:58:06.931 CURL [0x10d6c65c0] (Text) TLSv1.2 (OUT), TLS handshake, Client hello (1):
[DEBUG] 2020-04-07 19:58:06.931 CURL [0x10d6c65c0] (SSLDataOut) 222bytes
[DEBUG] 2020-04-07 19:58:07.034 CURL [0x10d6c65c0] (Text) TLSv1.2 (IN), TLS handshake, Server hello (2):
[DEBUG] 2020-04-07 19:58:07.034 CURL [0x10d6c65c0] (SSLDataIn) 91bytes
...
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text) SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text) ALPN, server did not agree to a protocol
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text) Server certificate:
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=s3.amazonaws.com
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  start date: Nov  9 00:00:00 2019 GMT
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  expire date: Dec  2 12:00:00 2020 GMT
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (Text)  SSL certificate verify ok.
[DEBUG] 2020-04-07 19:58:07.140 CURL [0x10d6c65c0] (HeaderOut) GET / HTTP/1.1
host: s3.amazonaws.com
Accept: */*
authorization: AWS4-HMAC-SHA256 Credential=AKIAJYTC5UXRVQGPK73Q/20200407/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=62e7d2da86ec118357ee86aa7deda5afafb93936ee46b59299881683ded50d5a
user-agent: aws-sdk-cpp/1.7.266 Darwin/18.7.0 x86_64 Clang/10.0.1
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20200407T195806Z
[DEBUG] 2020-04-07 19:58:07.284 CURL [0x10d6c65c0] (HeaderIn) HTTP/1.1 200 OK
[TRACE] 2020-04-07 19:58:07.284 CurlHttpClient [0x10d6c65c0] HTTP/1.1 200 OK
[DEBUG] 2020-04-07 19:58:07.284 CURL [0x10d6c65c0] (HeaderIn) x-amz-id-2: i4Y+p7ujlZjUYoLsYSEx0hlfYaJJil3JBqaumJXjbUCgUEZwKETc3rvOrfjnpE31tqgRC9SpZeY=
[TRACE] 2020-04-07 19:58:07.284 CurlHttpClient [0x10d6c65c0] x-amz-id-2: i4Y+p7ujlZjUYoLsYSEx0hlfYaJJil3JBqaumJXjbUCgUEZwKETc3rvOrfjnpE31tqgRC9SpZeY=
...
[TRACE] 2020-04-07 19:58:07.285 CurlHttpClient [0x10d6c65c0] Server: AmazonS3
[TRACE] 2020-04-07 19:58:07.285 CurlHttpClient [0x10d6c65c0] 
[DEBUG] 2020-04-07 19:58:07.285 CURL [0x10d6c65c0] (HeaderIn) 
[DEBUG] 2020-04-07 19:58:07.287 CURL [0x10d6c65c0] (DataIn) 1664
[DEBUG] 2020-04-07 19:58:07.287 CURL [0x10d6c65c0] (DataIn) 0
[DEBUG] 2020-04-07 19:58:07.287 CURL [0x10d6c65c0] (Text) Connection #0 to host s3.amazonaws.com left intact

With this now enabled, you get more raw data about headers, bytes read/written, secure channels and so on to help you debug and track the communication with server.

Remove unsafe version of MakeRequest()

In the class Aws::Http::HttpClient, a version of its member function MakeRequest() is defined as following:

virtual std::shared_ptr<HttpResponse> MakeRequest(HttpRequest& request,
                Aws::Utils::RateLimits::RateLimiterInterface* readLimiter = nullptr,
                Aws::Utils::RateLimits::RateLimiterInterface* writeLimiter = nullptr) const = 0;

This is unsafe, as it takes a reference to an HttpRequest object as a parameter and when this object is out of scope outside the function, the HTTP client will no longer be able get access to the original request.

We fixed that by adding another safe version of this function by passing a shared pointer to the HttpRequest object and marked the legacy one as “deprecated”, starting from version 1.4.46.

For version 1.8 we removed the deprecated functions. This will break your code if you have a subclass of HttpClient that has your own implementation of MakeRequest().

Client configuration now reads environment variables, configuration file and EC2 metadata to get default AWS region

With previous version of AWS SDK for C++, there are only two ways to specify regions for service clients with client configurations:

  1. By specifying region in ClientConfiguration explicitly:
Aws::Client::ClientConfiguration config;
config.region = Aws::Region::US_WEST_2;
Aws::S3::S3Client s3Client(config);
  1. By specifying profile in ClientConfiguration explicitly and define the region in the configuration file:
Aws::Client::ClientConfiguration config("default");
Aws::S3::S3Client s3Client(config);

Where your default configuration file (~/.aws/config) looks like:

[default]
region=us-west-2

A popular feature request has been to determine the region automatically by checking environment variables, configuration files and EC2 metadata. This has been added for version 1.8.

With version 1.8, here’s how ClientConfiguration determines the AWS region:

  1. If you specify a region in ClientConfiguration, it will always override region from other sources.
  2. If you specify a profile with ClientConfiguration, it will search configuration file for region associated with that profile. So far it’s the same as that of the previous version of SDK.
  3. If neither are specified, the SDK will try to determine the AWS region automatically based on environment variables. It will check AWS_DEFAULT_REGION first, then AWS_REGION. In some cases, the environment variables could be pre-defined by something other than the SDK (for example, in a CodeBuild project or a Lambda function).
  4. Next, the SDK will check the configuration files. The default profile is default and the default configuration file is ~/.aws/config. You can also specify these with environment variables. AWS_DEFAULT_PROFILE, AWS_PROFILE specify the profile name, and AWS_CONFIG_FILE specifies the configuration file.
  5. As the last step, the SDK will check EC2 metadata for AWS region information, if your application is running on an EC2 instance. You can set environment variable: AWS_EC2_METADATA_DISABLED as true to disable it.
  6. And finally, if you do nothing with ClientConfiguration, environment variables or configuration files, and your code is not running in any pre-configured environments, like EC2 instances or CodeBuild projects, the default region is us-east-1.

Exceptions may include more service and operation specific details now

In the previous version of the SDK, the data structure of AWSError, only returned limited information about exceptions. This included error type, exception name, and the error message. Some services return other useful information that could not be deserialized. For example, the Amazon Elastic File System operation CreateFileSystem returns the following response body for the FileSystemAlreadyExists exception:

{
    "ErrorCode": "FileSystemAlreadyExists",
    "FileSystemId": "fs-some-id",
    "Message": "File system 'fs-some-id' already exists with creation token 'basic-file-system-creation-some-token'“
}

With the previous version, the SDK can only get the ErrorCode and Message from the payload.

To support modeled exceptions, we introduced a similar interface, without breaking changes. The interface is easy to use. Just provide the type when getting the error: outcome.GetError<ERROR_TYPE>(). Here is an example of how to use this new interface:

CreateFileSystemRequest createFileSystemRequest;
createFileSystemRequest.SetCreationToken(FILE_SYSTEM_CREATION_TOKEN);
auto createFileSystemOutcome = efsClient.CreateFileSystem(createFileSystemRequest);

if (createFileSystemOutcome.IsSuccess())
{
    fileSystemId = createFileSystemOutcome.GetResult().GetFileSystemId();
    std::cout << "Succeeded to create file system with ID: " << createFileSystemOutcome.GetResult().GetFileSystemId() << std::endl;
}
else if (createFileSystemOutcome.GetError().GetErrorType() == EFSErrors::FILE_SYSTEM_ALREADY_EXISTS)
{
    std::cout << "File system with ID: " << createFileSystemOutcome.GetError<FileSystemAlreadyExists>().GetFileSystemId() << " already exists." << std::endl;
    std::cout << "Failed to create file system. Error details:" << std::endl;
    std::cout << createFileSystemOutcome.GetError() << std::endl;
}
else
{
    std::cout << "Failed to create file system. Error details:" << std::endl;
    std::cout << createFileSystemOutcome.GetError() << std::endl;
}

The most interesting part is createFileSystemOutcome.GetError<FileSystemAlreadyExists>(), which returns a FileSystemAlreadyExists object. Then you can call GetFileSystemId() to get the existing file system id.

The SDK can provide more common information about the exception, including remote host IP address and request ID if available.

For the example code, if you are reusing FILE_SYSTEM_CREATION_TOKEN to create a file system, then you will get the output that is similar to the following, with an FileSystemAlreadyExists error:

File system with ID: fs-ada1e82d already exists.
Failed to create file system. Error details:
HTTP response code: 409
Resolved remote host IP address: 52.94.226.247
Request ID: e831b56f-4261-48d9-a43c-a4a37daf38da
Exception name: FileSystemAlreadyExists
Error message: File system 'fs-ada1e82d' already exists with creation token 'file-system-creation-token-test'
6 response headers:
connection : close
content-length : 175
content-type : application/json
date : Mon, 23 Mar 2020 20:27:06 GMT
x-amzn-errortype : FileSystemAlreadyExists:
x-amzn-requestid : e831b56f-4261-48d9-a43c-a4a37daf38da

New pseudo region: aws-global to make cross region requests

For most AWS services, you have to know the region before accessing the resource, or it may encounter a signature mismatch error. The new pseudo region aws-global provides the flexibility to get resources without knowing the region. We are reusing the interface to specify a regular region in ClientConfiguration:

Aws::Client::ClientConfiguration config;
config.region = Aws::Region::AWS_GLOBAL;

Use cases include operations on an S3 bucket (such as ListObjects) without knowing the region. Also you don’t need to call GetBucketLocation to get the region. Instead, the SDK will help you solve the region issue by making two requests. With the first request, the SDK will get a response with status code 307 as well as the correct region and endpoint. Then the SDK will re-calculate the signature with the correct region and will use the correct endpoint to make the second request. If you don’t want the SDK to make a cross-region request, don’t specify aws-global as the client region. See the following example:

Aws::Client::ClientConfiguration config;
config.region = Aws::Region::AWS_GLOBAL;
S3Client s3Client(config);

// Create a bucket in us-west-2.
CreateBucketRequest createBucketRequest;
createBucketRequest.SetBucket(BUCKET_NAME);
CreateBucketConfiguration createBucketConfiguration;
createBucketConfiguration.SetLocationConstraint(BucketLocationConstraint::us_west_2);
createBucketRequest.SetCreateBucketConfiguration(createBucketConfiguration);
auto createBucketOutcome = s3Client.CreateBucket(createBucketRequest);

// S3 client with aws-global region should be able to get access to bucket in any region.
ListObjectsRequest listObjectsRequest;
listObjectsRequest.SetBucket(BUCKET_NAME);
auto listObjectOutcome = s3Client.ListObjects(listObjectsRequest);

Another breaking change is that the type of followRedirect in ClientConfiguration changed from a true or false boolean to the enum, FollowRedirectsPolicy, including the following options:

  • DEFAULT: Let the SDK decide if the underlying HTTP client should redirect a request if it receives a response with a 30x status code. With the aws-global region, the HTTP client will handle the 30x response manually. Otherwise, it will redirect the request by default.
  • ALWAYS: The underlying HTTP client will always redirect the request if it receives a 30x response. (It’s equivalent to true before version 1.8.)
  • NEVER: The underlying HTTP client will never redirect the request. (It’s equivalent to false before version 1.8.)

We made this change because of the underlying implementation of the aws-global region. If your service client tries to hit a resource in another region, for example getting a bucket in us-west-2 with a client in us-east-1, sometimes the service will return a response with status code 30x. Then we need to handle the request redirect carefully. For example, with the global region, the SDK will handle the 30x response manually, rather than depending on the underlying HTTP client doing redirects.

S3 client in us-east-1 now uses regional endpoint by default.

In the previous version of AWS SDK for C++, an S3 client with region us-east-1 will make requests to the legacy global endpoint: bucket-name.s3.amazonaws.com to hit the bucket with virtual hosted-style requests by default. With version 1.8, we are changing this behavior by making requests to regional endpoints instead, for example:

https://bucket-name.s3.us-east-1.amazonaws.com

You can see S3 Legacy Global Endpoint for more details.

However, you could still use legacy global endpoint for S3 client in us-east-1 with one of the following three approaches:

  1. In environment variables, set AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to legacy
  2. In the configuration file, under the profile you are using, specify s3_us_east_1_regional_endpoint=legacy
  3. Specify Aws::S3::US_EAST_1_REGIONAL_ENDPOINT_OPTION::LEGACY when creating an S3 client, for example:
Aws::Client::ClientConfiguration config;
S3Client s3Client(config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true, US_EAST_1_REGIONAL_ENDPOINT_OPTION::LEGACY);

Improvements on the underlying HTTP client override

With AWS SDK for C++, you can plug in your own implementation of the underlying HTTP client by extending HttpClientFactory. However, sometimes you want a minor code change in the configuration of the default HTTP client. There was not easy way to do that before version 1.8. We’ve improved the existing functionality by providing the virtual function: OverrideOptionsOn*Handle() so that you can add or override any configuration they want. The following is an example for disabling DNS caching for requests sent to S3 with curl HTTP clients.

First, we will need to create a subclass of CurlHttpClient, and then override the virtual function OverrideOptionsOnConnectionHandle to configure CURLOPT_DNS_CACHE_TIMEOUT with curl API:

class MyCurlHttpClient : public Aws::Http::CurlHttpClient
{
public:
    MyCurlHttpClient(const Aws::Client::ClientConfiguration& clientConfig) : Aws::Http::CurlHttpClient(clientConfig) {}

protected:
    void OverrideOptionsOnConnectionHandle(CURL* connectionHandle) const override
    {
        std::cout << "Disable DNS caching completely." << std::endl;
        curl_easy_setopt(connectionHandle, CURLOPT_DNS_CACHE_TIMEOUT, 0L);
    }
};

Then, as with the previous SDK version, create a custom HTTP client factory extending class HttpClientFactory:

class MyHttpClientFactory : public Aws::Http::HttpClientFactory
{
    std::shared_ptr<Aws::Http::HttpClient> CreateHttpClient(const Aws::Client::ClientConfiguration& clientConfiguration) const override
    {
        return Aws::MakeShared<MyCurlHttpClient>(ALLOCATION_TAG, clientConfiguration);
    }
};

And use it in your application:

SetHttpClientFactory(Aws::MakeShared<MyHttpClientFactory>(ALLOCATION_TAG));

When making requests with this custom HTTP client, the DNS caching will be disabled.

Clone this wiki locally