Latest version requires newer boto3 than that provided by the Lambda runtime

[allanlewis]

The Lambda runtime provides boto3==1.26.90 but the latest Powertools requires a newer version. This means that if I want to exclude boto3 and botocore from my deployment package to keep the size down, then I can't use the latest Powertools. Is there a good reason to not limit Powertools' boto3 dependency to the version provided by the Lambda runtime?

I commented on this issue but I expect that won't be very visible as it's closed: #3632 (comment)

[leandrodamascena]

Hi @allanlewis! Firstly, I apologize for the delay in responding. We are working on some complex features and a lot is happening.

The boto3 dependency in Powertools is optional and in most cases it is unnecessary to install it as you can use the SDK provided by the Lambda runtime. Are you installing boto3 and Powertools in the same project and are experiencing dependency resolution issues? I would need a little more data/information to understand how we can help you with this problem, since Powertools does not include boto3 by default.

Just to give a context, the reason we fixed the boto3 version to ^1.26.164 is because we optimized the use of our Idempotency feature with the new ReturnValuesOnConditionCheckFailure (#3446). But even if you use Idempotency and boto3==1.26.90 it works because we use the old condition check.

Thank you and let's see how we can help.

[allanlewis]

Thanks for the context, that's very useful.

Our specific use-case is that we have a bunch of microservices using Poetry dependency management. Each service that uses Powertools has two entries in its pyproject.toml:

[tool.poetry.dependencies]
aws-lambda-powertools = "2.31.0"

[tool.poetry.group.dev.dependencies]
aws-lambda-powertools = {extras = ["aws-sdk"], version = "2.31.0"}

We've configured it this way based on the advice in your docs.

In order to be able to exclude boto3 and botocore from our deployment packages, we also have the following dependency group in each pyproject.toml:

[tool.poetry.group.lambda-runtime.dependencies]
boto3 = "1.26.90"
botocore = "1.29.90"

This deliberately restricts other dependencies to being compatible with the versions of the AWS SDK in the Lambda runtime; FWIW we're using Python 3.9 at the moment. This then prevents me from upgrading Powertools beyond v2.31.0 because you now depend on a different version.

Obviously, I could simply remove the aws-sdk extra from aws-lambda-powertools but then I'm using a version that you've declared incompatible with your library, which feels wrong. If this is what you expect people to do then I'd appreciate some language in the docs to that effect.

I hope that makes sense!

[leandrodamascena]

Thanks for the clear explanation on how you are using Powertools and resolving dependency issues.

Obviously, I could simply remove the aws-sdk extra from aws-lambda-powertools but then I'm using a version that you've declared incompatible with your library, which feels wrong.

I understand your perspective here, however, I don't think it's wrong to allow the user to bring their own version of boto3. The decision to make boto3 optional allows users the flexibility to utilize the version that best suits their needs. However, we recommend a specific version of the AWS SDK (as optional) in order to provide performance and cost optimization specifically for the Idempotency utility. Backwards compatibility with boto3 is maintained (minimum 1.26.90) in all features so that users can decide which version makes the most sense for their use case.

If this is what you expect people to do then I'd appreciate some language in the docs to that effect.

I'm thinking about adding the minimum version that we recommend to the documentation, but if the user wants they can use the Lambda Runtime version and exclude boto3 from their deployment package. Does it make sense to you?

Thanks

[allanlewis]

Thanks, Leandro.

I'm thinking about adding the minimum version that we recommend to the documentation... Does it make sense to you?

Yes, that sounds good.

I think what I'd really like is an extra that pulls in boto3 limited to the version in the Lamdba runtime as a minimum, so >=1.26.90 currently, and another extra that pulls in the "recommended" version, which can be any newer version you like. However, I accept this could get rather complicated so for now I'll remove the aws-sdk extra from my dependency specs and rely on the Lambda runtime version being compatible in perpetuity.

[leandrodamascena]

I think what I'd really like is an extra that pulls in boto3 limited to the version in the Lamdba runtime as a minimum, so >=1.26.90 currently, and another extra that pulls in the "recommended" version, which can be any newer version you like. However, I accept this could get rather complicated so for now I'll remove the aws-sdk extra from my dependency specs and rely on the Lambda runtime version being compatible in perpetuity.

Hi @allanlewis! I have an idea before we move forward to change anything. You have an important point about developer experience and compatibility between versions of this project and the Lambda runtime. So I'm going to contact the Lambda team to find out their plans to update the SDK in all Python versions of the Lambda Runtime. If they have plans to update it soon, we may have the same version of boto3 in Powertools.

I don't know when I will have this answer, but I believe it will take a few weeks. I'll keep this discussion open for interaction with you and update as soon as I have the answer, ok?

Thank you for your patience and for continuing this discussion. We appreciate you taking the time to share your perspective. It is helpful for us to understand how we can make adopting Powertools easier for others.

[allanlewis]

Thanks, that sounds like a good plan 👍

[leandrodamascena]

Hi @allanlewis, I have good news! For Lambda functions created and updated from now on, the new version of the runtime brings the SDK version 1.34.32, both botocore and boto3.
To access the new version of the Runtime with these SDK versions, you must create a new function or update an existing one.

Can you check if it works for you? We will plan a Powertools update to include these versions as the minimum required version.

Thanks

[allanlewis]

Thanks - I've put a ticket in our backlog to update. We've dropped the Powertools extra in any case so this shouldn't make any difference, but it will mean we're using the runtime versions in local dev.

[leandrodamascena]

Thanks for the feedback!

Please reopen this discussion if needed!