Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug info seems to be lost when deployed to Lambda #49

Closed
maxcountryman opened this issue Sep 24, 2019 · 9 comments · May be fixed by #59
Closed

Debug info seems to be lost when deployed to Lambda #49

maxcountryman opened this issue Sep 24, 2019 · 9 comments · May be fixed by #59

Comments

@maxcountryman
Copy link

Hi there,

I've been back and forth with Sentry trying to figure out why debug information seems to be completely lost when in the deployed Lambda environment--very strangely, the same project compiled directly in the underlying Docker container built for release happily passes along its debug info. The trouble only starts when deployed.

Some interesting things about the project:

  1. We organize our functions via workspaces, so we've defined the profile.release overload in the root Cargo.toml.
  2. We only use Sentry in our library project crate and then use this crate in the other functions; this shouldn't be an issue as evidenced by the fact this works in virtually any environment aside from Lambda itself.
  3. Frustratingly things work as expected locally and during CircleCI builds and even from the build container; just not Lambda itself.
  4. As per Sentry's request we've tried uploading debug symbols manually, but this hasn't really helped.
  5. Sentry complains that libc.6.so is not included in the aforementioned uploaded debug files, but despite this things work in e.g. CircleCI, just again not in Lambda.

I'm wondering if this could be partly to do with the fact that the Lambda executable is packaged and renamed to "bootstrap"? If there's any light you can shed on that piece of things and how it might affect something like this, I'd really appreciate it! :)

serverless version: ^1.51.0

rust-plugin version: ^0.3.6
@softprops
Copy link
Owner

This is likely related to building a release binary. I think in those cases debug info gets stripped by default. serverless-rust is a Serverless framework front end for lambda-rust a docker builder. There's an open issue there about making the release vs debug build configurable softprops/lambda-rust#21 which may be helpful here. This defaults to release for efficiency of speed and size. I a believer that debug should also be an option though.

You mentioned something worked in a build container. Was that the lambda-rust container or a different container?

@maxcountryman
Copy link
Author

This is likely related to building a release binary. I think in those cases debug info gets stripped by default.

I think this is right. However my understanding was that you could workaround this by specifying something like this in your Cargo.toml:

[profile.release]
debug = true

You mentioned something worked in a build container. Was that the lambda-rust container or a different container?

This is in the lambda-rust container. Interestingly enough that was with a release build. Again I think that by specifying the debug = true configuration for the release profile the binary should be including debug info.

@softprops
Copy link
Owner

Post a link here for my own reference and testing
https://blog.sentry.io/2018/10/22/getting-started-with-rust-error-tracking

@maxcountryman maxcountryman changed the title Debug info seems to be lost when deployed to Lamba Debug info seems to be lost when deployed to Lambda Sep 25, 2019
@jan-auer
Copy link

jan-auer commented Oct 4, 2019

Debug information is stripped explicitly in the build script in the docker container:
https://github.com/softprops/lambda-rust/blob/d3e55ce8a436c79c32921740ce1eea3475b999e6/build.sh#L33

While this makes sense to reduce the size of the final binary, it might make sense to put debug information in a companion file instead so that it can be used for debugging later, like so:

objcopy --only-keep-debug "$file" "$file.debug"
objcopy --strip-debug --strip-unneeded "$file"
objcopy --add-gnu-debuglink="$file.debug" "$file"

@jan-auer
Copy link

jan-auer commented Oct 4, 2019

Also, it is a little strange that debug information is stripped even for debug builds. There, it should probably even be retained in the original file so that the builtin libbacktrace can symbolicate stack traces, etc.

@maxcountryman
Copy link
Author

Also, it is a little strange that debug information is stripped even for debug builds. There, it should probably even be retained in the original file so that the builtin libbacktrace can symbolicate stack traces, etc.

Thanks so much for spotting this, @jan-auer--I ran out of bandwidth on my end but this is pretty clearly the source of my issue.

@softprops
Copy link
Owner

I'll try to publish an update soon but I've updated the underlying container image to add support for this. I just haven't wired it into the serverless plugin

@maxcountryman
Copy link
Author

Fantastic! Looking forward to the update.

@jan-auer jan-auer mentioned this issue Oct 5, 2019
Merged
@maxcountryman maxcountryman mentioned this issue Nov 4, 2019
Open
@softprops
Copy link
Owner

Hi folks. It's been a while but I wanted to post and update on the latest release which now has experimental support for local builds which takes a slightly more hands off approach when it comes to stripping anything

https://github.com/softprops/serverless-rust#-experimental-local-builds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

allow for explicitly turning on debuginfo for builds maxcountryman/serverless-rust
3 participants
@softprops @maxcountryman @jan-auer