[Bug]: Failure to show drift when ephemeral storage has been manually set on a lambda

[eldondevat]

Terraform Core Version

1.9.8

AWS Provider Version

5.69.0

Affected Resource(s)

lambda_function

Expected Behavior

When the lambda function is manually assigned additional ephemeral storage and the optional ephemeral_storage setting is not present on the resource, terraform should recognize that the lambda has non-default ephemeral storage

Actual Behavior

When the lambda function is manually assigned additional ephemeral storage and the optional ephemeral_storage setting is not present on the resource, terraform should recognize that the lambda has non-default ephemeral storage.

Terraform Configuration Files

resource "aws_lambda_function" "bug_report" {
  filename = "lambda.zip"
  function_name = "bug_report"
  role = aws_iam_role.process_lambda_role.arn
  handler = "bug_report.handle_event"
  runtime = "python3.8"
}

Steps to Reproduce

terraform apply
update ephemeral storage value via console
terraform apply -> no changes

References

May be related to #29253

Would you like to implement a fix?

Maybe

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jeff-carey]

We just observed what may be related to whatever the underlying root cause of this bug is. We had added the optional ephemeral_storage block to allocate additional storage, which worked. We then removed that block assuming the storage would go back to the default value, but no change was detected by the plan. We had to add back the block and specify the default value explicitly to cause a plan to change the storage back to the default value.

[justinretzolk]

Hey @jeff-carey 👋 Thank you for taking the time to raise this! I attempted to reproduce this, but was unable to, however, I believe I may have found what may be misleading you. Apologies; this reply grew a bit long, but I wanted to be thorough.

First, my initial reproduction:

Configuration, most of which was pulled from the "basic example" configuration on the resource documentation (click to expand)

provider "aws" {
  region = "us-east-1"
}

data "archive_file" "lambda" {
  type        = "zip"
  source_file = "${path.module}/lambda.py"
  output_path = "${path.module}/lamabda.zip"
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "iam_for_lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

resource "aws_lambda_function" "main" {
  function_name = "jretzolk-test"
  filename      = data.archive_file.lambda.output_path
  runtime       = "python3.8"
  handler       = "lambda.lambda_handler"
  role          = aws_iam_role.iam_for_lambda.arn
}

# output value to make any changes easy to see
output "ephemeral_storage" {
  value = aws_lambda_function.main.ephemeral_storage
}

For the sake of completeness, I'm using this "hello world" example for the function code saved at ${path.module}/lambda.py:

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Reproduction notes:

1. Apply the above configuration. I used Terraform version
2. Navigate to the AWS console and update the ephemeral storage from the default of 512 to 1024.
3. Run terraform plan

The result of the third step was as follows:

$ terraform plan
data.archive_file.lambda: Reading...
data.archive_file.lambda: Read complete after 0s [id=6bfd5fdd62c3ca38857bce819c47ff43c3d35a9f]
data.aws_iam_policy_document.assume_role: Reading...
data.aws_iam_policy_document.assume_role: Read complete after 0s [id=2690255455]
aws_iam_role.iam_for_lambda: Refreshing state... [id=iam_for_lambda]
aws_lambda_function.main: Refreshing state... [id=jretzolk-test]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # aws_lambda_function.main has changed
  ~ resource "aws_lambda_function" "main" {
        id                             = "jretzolk-test"
        # (28 unchanged attributes hidden)

      ~ ephemeral_storage {
          ~ size = 512 -> 1024
        }

        # (2 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to
undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Changes to Outputs:
  ~ ephemeral_storage = [
      ~ {
          ~ size = 512 -> 1024
        },
    ]

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

After doing so, I destroyed the reproduction environment so I could start fresh, removed the output from the configuration, and ran an initial terraform apply to recreate the infrastructure. After the initial creation, I used terraform state show to inspect the Lambda function:

$  terraform state show aws_lambda_function.main
# omitted for brevity...

    ephemeral_storage {
        size = 512
    }

I then modified the Lambda function in the AWS console, again to 1024, and ran terraform apply again. The output indicated that the infrastructure was up to date:

$ terraform apply
data.archive_file.lambda: Reading...
data.archive_file.lambda: Read complete after 0s [id=6bfd5fdd62c3ca38857bce819c47ff43c3d35a9f]
data.aws_iam_policy_document.assume_role: Reading...
data.aws_iam_policy_document.assume_role: Read complete after 0s [id=2690255455]
aws_iam_role.iam_for_lambda: Refreshing state... [id=iam_for_lambda]
aws_lambda_function.main: Refreshing state... [id=jretzolk-test]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

However, there's an important line in there:

aws_lambda_function.main: Refreshing state... [id=jretzolk-test]

Inspecting the state again reveals that the value is indeed updated in the state:

$ terraform state show aws_lambda_function.main
# omitted for brevity...

    ephemeral_storage {
        size = 1024
    }

What's happening here is the result of a couple of behaviors of Terraform Core and the Terraform Plugin SDKv2.

The ephemeral_storage block is marked as both Optional and Computed. When both of those schema behaviors are specified, it means that if a value is provided in the configuration, it will be included in the calls to the upstream API. If a value is not provided, the value in the state will be set to whatever is returned from the upstream API.

With computed values, Terraform does not show changes in the plan unless it will require making changes. Otherwise, the value in the state is updated during that Refreshing state... step that I called out above. By introducing an output in my initial configuration, I introduced a "change" that needed to be made (an update to the output's value), and so the change appears in the plan output.

Regarding your follow up comment around adding and then removing the ephemeral_storage block, and why the Lambda's ephemeral storage was not set back to the default, this is again caused by the schema behaviors that are defined for the block. Because the block is Computed, it cannot have a Default value. With no default value, removing the ephemeral_storage block from the configuration means that the value will again be computed by reading the value from the upstream API. While the provider does check for a scenario where ephemeral_storage was defined, but was then removed from the configuration, using that to infer that the operator wants to return the value to the default of 512 may not be accurate in all cases. The underlying API doesn't default to 512 on update if a value isn't supplied, and closely matching the underlying API is one of the design principles of providers; differing is something we try to avoid where possible.

I hope that information helps clear this up a bit. I'll leave this issue open for now in case you have any follow up questions.