When using the path-style request URL under the v2 protocol, list_objects_v2 does not work.

[feiniks]

Describe the bug

When performing the list_objects_v2 operation using the s3v2 protocol with a path-style request URL, objects cannot be listed, and an exception is thrown: botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListObjectsV2 operation: None. However, it works normally when using a virtual host or s3v4.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When performing the list_objects_v2 operation using the s3v2 protocol with a path-style request URL, objects can be listed.

Current Behavior

The objects cannot be listed, and an exception is thrown: botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListObjectsV2 operation: None

Reproduction Steps

import boto3

def list_objs(client, bucket, prefix=None):
    paginator = client.get_paginator('list_objects_v2')
    if prefix:
       iterator = paginator.paginate(Bucket=bucket, Prefix=prefix)
    else:
        iterator = paginator.paginate(Bucket=bucket)
    for page in iterator:
        for content in page.get('Contents', []):
            tokens = content.get('Key', '')
            if tokens:
                obj = [tokens, content.get('Size', 0)]
                yield obj

addressing_style = 'path'
config = boto3.session.Config(signature_version='s3',s3={'addressing_style':addressing_style})
client = boto3.client('s3',
                                    aws_access_key_id=key_id,
                                    aws_secret_access_key=key,
                                    endpoint_url=endpoint_url,
                                    config=config)

objs = list_objs(client, "my-bucket")
for obj in objs:
    print(obj)

Possible Solution

The issue might be that the authPath is not handled correctly during authentication under the v2 protocol. I tried modifying the canonical_string method in the HmacV1Auth class of botocore/auth.py. If it's a path-style request, omitting authPath allows it to work correctly.

Additional Information/Context

No response

SDK version used

boto3 1.36.13

Environment details (OS name and version, etc.)

ubuntu 18.04

[khushail]

Hi @feiniks , thanks for reporting this. Although s3v4 is the supported one from AWS and s3v2 is being phased out, its recommended to use s3v4 and virtual addressing style -

https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

Note

Amazon S3 supports Signature Version 4, a protocol for authenticating inbound API requests to AWS services, in all AWS Regions. At this time, AWS Regions created before January 30, 2014 will continue to support the previous protocol, Signature Version 2. Any new Regions after January 30, 2014 will support only Signature Version 4 and therefore all requests to those Regions must be made with Signature Version 4. For more information about AWS Signature Version 2, see [Signing and Authenticating REST Requests](https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html) in the Amazon Simple Storage Service User Guide.

However I am able to repro the issue and getting this error while using signature_version=s3 -

botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the ListObjectsV2 operation: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

while it succeeds with signature_version=s3v4, listing the objects in S3 bucket.

This might be due to how the signature is calculated with V2 protocol but not really sure -

https://github.com/boto/botocore/blob/3ff30ef929da20bc83a1f21d1bd3ab6e408648fe/botocore/auth.py#L95

It would be helpful if you could mention what region you are using, just to make sure if this is not among the regions , which are not supported by V2, hence leading to the error.

[feiniks]

Hello @Khushai,
thanks for your reply.
The AWS region I am using is ap-southeast-1 .
I have also tested other non-AWS storage providers, such as Hetzner, which is located in the fsn1 region. Using v2 + path-style request, I am unable to list the bucket.
Of course, only the list request throws an exception, while get_object or put_object work normally.
Additionally, I also tried using the Go client minio-go, which can list the bucket correctly when using v2 + path-style request.

[khushail]

Hi @feiniks , thanks for sharing the required information. ap-southeast-1 is supported for V2 signature as listed here -https://docs.aws.amazon.com/general/latest/gr/s3.html. Looks like this is a bug on Python side. Let me check with the team and get back to you. Thanks for having patience !

[khushail]

@feiniks , as shared earlier s3v2 is legacy and would not work across APIs so its suggested to use the latest protocol s3v4 which is current standard supported version and should be used. s3v2 is used for presigned URLs in specific cases for backwards compatibility only. Hence we won't be able to take much action on this issue and don't recommend using this at all.

Hope that clarifies your ask. let me know if you have any other questions. Thanks.