Bump Spring Boot to 3.4

[adinauer]

Description

Bump Spring Boot to 3.4

[bjornharvold]

Please see this issue as this might very well pertain to Sentry: spring-projects/spring-boot#43332

Using Sentry 7.18.1 and 8.0.0.rc1 produced different errors in our staging environment.

• 7.18.1 made the system unresponsive after authenticating
• 8.0.0.rc1 made it impossible to authenticate with Spring Authorization Server 1.4.0

Cannot confirm 💯 without disabling Sentry in our Spring Boot app completely but have a hunch.

Cheers,
Bjorn

[adinauer]

Thanks for reaching out @bjornharvold , what makes you think it's Sentry that's causing the issues? Just using different versions and having different outcomes?

Are you seeing any exceptions?

Do you have a repro we can use to look into this issue?

Can you please share more info on how you're using Sentry (which SDK, integrations, options, etc.).

[bjornharvold]

Hi @adinauer

What makes this look like an instrumentation issue, is that there is 0 errors. The apps just start hanging completely and being completely unresponsive even though we can see Spring Data MongoDb pinging MongoDB Atlas every few seconds.

Now it could be Spring Authorization Server but it wouldn't be giving us different errors based on the Sentry version we use.

Never seen anything quite like this actually.

Of course cannot be sure it's Sentry as we are unable to make a thread dump once the Cloud Run instance reaches this state.

Last indication that it is most likely Sentry is that our Spring Boot app (3.4.0) works locally and we have set sentry.logging.enabled=false in this environment. It's enabled in both staging and production.

Hope this helps,
Bjorn

[adinauer]

@bjornharvold did this start happening when you bumped Spring Boot to 3.4.0 and was fine before?

Can you please share more details on what Sentry integrations you're using and what config options you are setting?

How many servers are in play here and which one is causing troubles?

A reproducible sample would be the best way for us to help you.

[bjornharvold]

Hi @adinauer,

This started happening when upgrading Spring Boot to 3.4.0 and pushing to staging. Sentry was still on 7.18.0 at this point.

Here are the settings we use in staging and we haven't changed these settings for a long time (This is the configuration for a Spring Web MVC app FYI - maybe the reactive and backpressue properties are not valid here):
sentry.logging.enabled=true
sentry.environment=staging
sentry.dsn=HIDDEN
sentry.logging.minimum-event-level=error
sentry.logging.minimum-breadcrumb-level=debug
sentry.send-default-pii=true
sentry.max-request-body-size=ALWAYS
sentry.use-git-commit-id-as-release=false
[email protected]@
sentry.ignored-exceptions-for-type=
sentry.reactive.thread-local-accessor-enabled=true
sentry.enable-tracing=true
sentry.traces-sample-rate=0.2
sentry.enable-backpressure-handling=true

I am not asking for support on this ticket. I will create my own if I am certain that it's Sentry. Pointing out that this might be something to look at while testing support for Spring Boot 3.4.0. In your docs, there is no support for this version yet AFAIK.

[adinauer]

@bjornharvold can you please try with sentry.max-request-body-size=none and see if that changes anything?

[adinauer]

@bjornharvold have you had a chance to test the config change above? The problem might be related to us caching the request body. The suggested config disables attaching the request body to Sentry events as well as temporary caching of the request body.

[bjornharvold]

Hi @adinauer

Will try this next week when staging frees up.

Cheers

[bjornharvold]

Hi @adinauer,

We tested 2 things this time around:

• We enabled Sentry locally: sentry.logging.enabled=true
• We set sentry.max-request-body-size=NONE

1. Our local development environment works after enabling Sentry.
2. Our staging environment still hangs

Question:
If we set sentry.logging.enabled=false in our staging environment, would Sentry stay completely OFF. Would be great if we could rule out Sentry. Never seen anything like this and unable to get a thread dump from Cloud Run so it's a little bit like working in the dark.

Cheers

[adinauer]

Thanks @bjornharvold,

you can either not set the DSN (sentry.dsn=null) or use sentry.enabled=false to disable Sentry completely. You only disabled the logging integration but the Spring Boot part is still active.

[lbloder]

Hi @bjornharvold,
A quick update on our progress so far.
I've setup a small sample with the spring authorization server to investigate this issue.

The first tests indicate that the request caching we do seems to cause this issue with Spring Booot 3.4 and Sentry 8.0.0-rc.1.

This is only active when sentry.send-default-pii=true.

You could give disabling this a try.

Testing with 7.18.1 is going on today, will post the results here once done.

We also have a fix in mind, which I will test after that.

[lbloder]

Hi again @bjornharvold,
I'm done testing also with 7.18.1. While I was able to reproduce the not being able to login issue on 8.0.0-rc.1, I wasn't able to reproduce the unresponsiveness issue.

Maybe when you get around to testing again, try with sentry.send-default-pii=true as mentioned in my last comment to verify if it also gets unresponsive on Sentry 8.0.0-rc.1.

The login issue should be solved as soon as #3641 drops. The problem, it seems, had to do with how we did request caching and this caused the csrf filter to fail on the spring auth server side.

[bjornharvold]

Hi @lbloder,

Seems like setting sentry.send-default-pii=false fixed the hanging issue on 7.18.1.

What does that actually mean?

🫶🏼

[lbloder]

Hi @bjornharvold,

Seems like setting sentry.send-default-pii=false fixed the hanging issue on 7.18.1.

That's great news! Thanks for giving this a try and letting us know!

What does that actually mean?

This flags enables/disables sending what we consider to be PII (= Personal Identifiable Information). Which, for example, includes Request Payloads and known auth headers and so on. By default this flag is set to off, which is the safest. If you enable this, you need to consider the implications of which data is sent to sentry and filter it manually in beforeSend or beforeSendTransaction. Here is a link to our documentation that goes into more detail: https://docs.sentry.io/platforms/java/guides/spring-boot/data-management/sensitive-data/#personally-identifiable-information-pii

Hope that helps, if you have any questions, please let us know