timeseries Lambda function logging of errors has changed

[nikki-t]

After deploying updates to how we handle errors the timeseries Lambda function logged errors in the following way:

2024-02-11T16:50:54.943-05:00 [ERROR] RequestMalformed: 400: This required parameter is missing: 'feature'
Traceback (most recent call last):
  File "/var/task/hydrocron/api/controllers/timeseries.py", line 286, in lambda_handler
    raise RequestMalformed(results['error_message'])

Recently it seems the way that the errors are logged has changed to:

2024-03-07T13:01:31.903-05:00
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] RequestError: 400: This required parameter is missing: 'feature'
Traceback (most recent call last):
  File "/var/task/hydrocron/api/controllers/timeseries.py", line 286, in lambda_handler
    raise RequestError(results['error_message'])

This new format does not impact existing functionality including the codes returned in the response but may prove difficult to parse for Cloud Metrics so we may want to change it if possible.

According to an AWS blog post in order for the API Gateway to pass the correct error code in the Gateway response, the Lambda function needs to raise an Exception which is what is being indicated by the LAMBDA_WARNING.

The developer guide section on errors also confirms this behavior. I believe that we can change this behavior by returning a custom error object documented in the guide.