Add graceful shutdown #10701

yawkat · 2024-04-11T11:05:01Z

This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated.

The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed.

There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new micronaut.server.netty.listeners.*.support-graceful-shutdown config option to disable graceful shutdown for a management port.

Graceful shutdown is triggered by application code through a new GracefulShutdownManager API.

Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed.

Docs are still pending, I want Graeme to review the API before I write it up.

This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated. The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed. There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new `micronaut.server.netty.listeners.*.support-graceful-shutdown` config option to disable graceful shutdown for a management port. Graceful shutdown is triggered by application code through a new GracefulShutdownManager API. Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed. Docs are still pending, I want Graeme to review the API before I write it up.

sonarqubecloud · 2024-04-11T12:12:50Z

Quality Gate failed

Failed conditions
1 New Bugs (required ≤ 0)
1 New Critical Issues (required ≤ 0)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownLifecycle.java

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

...src/main/java/io/micronaut/http/server/netty/configuration/NettyHttpServerConfiguration.java

.../src/main/java/io/micronaut/management/health/indicator/GracefulShutdownHealthIndicator.java

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

.../src/main/java/io/micronaut/management/health/indicator/GracefulShutdownHealthIndicator.java

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

graemerocher · 2024-04-16T20:29:09Z

Perhaps we should try and address #6493 in this PR as well

sdelamo

we need to include documentation about this new API in the main documentation.

andrebrov · 2024-09-11T13:32:38Z

@yawkat Hello!

Wanted to see if there are any blockers around this issue since it's a quite important feature.

yawkat · 2024-09-12T08:45:33Z

No blockers, it's just on backburner

altro3 · 2024-10-11T15:52:15Z

Any news about this feature?

graemerocher · 2024-10-14T07:33:43Z

@yawkat can you prioritise this next again after the client work

# Conflicts: # context/build.gradle # http-server-netty/src/main/java/io/micronaut/http/server/netty/HttpPipelineBuilder.java # http-server-netty/src/main/java/io/micronaut/http/server/netty/NettyHttpServer.java # http-server-netty/src/main/java/io/micronaut/http/server/netty/handler/PipeliningServerHandler.java # management/src/test/groovy/io/micronaut/management/health/aggregator/HealthAggregatorSpec.groovy

sonarqubecloud · 2025-02-19T17:27:09Z

Quality Gate failed

Failed conditions
1 New Critical Issues (required ≤ 0)
1 New Bugs (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

graemerocher

where is the logic that prevents new requests arriving and sends 503 back to the client except for management endpoints?

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownListener.java

graemerocher · 2025-02-24T15:17:09Z

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownListener.java

+@Singleton
+@Requires(bean = GracefulShutdownManager.class)
+@Requires(property = GracefulShutdownConfiguration.ENABLED, value = StringUtils.TRUE, defaultValue = StringUtils.FALSE)
+public final class GracefulShutdownListener implements ApplicationEventListener<ShutdownEvent> {


make non-public and annotate with @Internal

I kept this public so people could Replace it in the bean context, not sure if that is a useful thing to do

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

graemerocher · 2025-02-24T15:19:14Z

...ava/io/micronaut/scheduling/executor/GracefulShutdownCapableScheduledThreadPoolExecutor.java

+ * @author Jonas Konrad
+ */
+@Internal
+final class GracefulShutdownCapableScheduledThreadPoolExecutor extends ScheduledThreadPoolExecutor implements GracefulShutdownCapable {


should this only be a activated if graceful shutdown is enabled?

So far, all the GracefulShutdownCapable beans are enabled unconditionally. In fact, you can use GracefulShutdownManager without the config option. Only the ShutdownEvent listener is gated.

I think the overhead is negligible, and it allows for more flexibility for programmatic use for those that don't want to use the listener

yawkat · 2025-02-24T15:42:07Z

@graemerocher The HTTP listeners stop the server socket so that no new connections are accepted, but existing connections are still served. I think this is better than still accepting connections but returning a status

graemerocher · 2025-02-24T16:35:23Z

ok can you add docs? Thanks.

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownCapable.java

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java

context/src/main/java/io/micronaut/runtime/server/GracefulShutdownListener.java

yawkat · 2025-02-24T16:44:41Z

I'll do the docs tomorrow and then mark this as ready for review

yawkat · 2025-02-25T12:20:45Z

Not sure what's up with the build, sonar OOM?

graemerocher · 2025-02-25T12:29:38Z

http-server-netty/src/main/java/io/micronaut/http/server/netty/NettyHttpServer.java

@@ -687,6 +697,18 @@ private static String displayAddress(NettyHttpServerConfiguration.NettyListenerC
        };
    }

+    public static <T> CompletionStage<T> toCompletionStage(Future<T> future) {


does this need to be public? Should it be moved to utility class? Seems odd it is here

It's a utility yes, but it's used in two packages so it has to be public. I wasn't sure where to put it, so I just put it here. The class is internal anyway.

src/main/docs/guide/ioc/gracefulShutdown.adoc

yawkat added the type: enhancement New feature or request label Apr 11, 2024

yawkat added this to the 4.5.0 milestone Apr 11, 2024

yawkat requested review from graemerocher and sdelamo April 11, 2024 11:05

graemerocher requested changes Apr 16, 2024

View reviewed changes

sdelamo requested changes Apr 17, 2024

View reviewed changes

shutdown state reporting

8c36718

sdelamo linked an issue Apr 19, 2024 that may be closed by this pull request

Micronaut doesn't give the application code control of the order in which resources are gracefully shutdown/closed #6493

Open

rename

0b3ec41

sdelamo mentioned this pull request May 3, 2024

Micronaut doesn't give the application code control of the order in which resources are gracefully shutdown/closed #6493

Open

sdelamo removed this from the 4.5.0 milestone Oct 28, 2024

yawkat changed the base branch from 4.5.x to 4.9.x February 19, 2025 16:39

fix tests

a5b60b1

yawkat added 5 commits February 24, 2025 14:14

changes

eb0a071

move to less detailed shutdown state reporting

44fe380

Merge branch '4.9.x' into graceful

18730c1

detailing

7613b70

graceful shutdown for scheduler

08e32d5

graemerocher requested changes Feb 24, 2025

View reviewed changes

listener order

b9f17cc

graemerocher reviewed Feb 24, 2025

View reviewed changes

CRs

6d987bb

docs

e72ad61

yawkat marked this pull request as ready for review February 25, 2025 07:26

fix test

fd49a5f

graemerocher reviewed Feb 25, 2025

View reviewed changes

src/main/docs/guide/ioc/gracefulShutdown.adoc Show resolved Hide resolved

docs

cbe225e

graemerocher approved these changes Feb 25, 2025

View reviewed changes

graemerocher requested a review from sdelamo February 25, 2025 14:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add graceful shutdown #10701

Add graceful shutdown #10701

yawkat commented Apr 11, 2024

sonarqubecloud bot commented Apr 11, 2024

graemerocher commented Apr 16, 2024

sdelamo left a comment

andrebrov commented Sep 11, 2024

yawkat commented Sep 12, 2024

altro3 commented Oct 11, 2024

graemerocher commented Oct 14, 2024

sonarqubecloud bot commented Feb 19, 2025

graemerocher left a comment

graemerocher Feb 24, 2025

yawkat Feb 24, 2025

graemerocher Feb 24, 2025

yawkat Feb 24, 2025

yawkat commented Feb 24, 2025

graemerocher commented Feb 24, 2025

yawkat commented Feb 24, 2025

yawkat commented Feb 25, 2025

graemerocher Feb 25, 2025

yawkat Feb 25, 2025

Add graceful shutdown #10701

Are you sure you want to change the base?

Add graceful shutdown #10701

Conversation

yawkat commented Apr 11, 2024

sonarqubecloud bot commented Apr 11, 2024

Quality Gate failed

graemerocher commented Apr 16, 2024

sdelamo left a comment

Choose a reason for hiding this comment

andrebrov commented Sep 11, 2024

yawkat commented Sep 12, 2024

altro3 commented Oct 11, 2024

graemerocher commented Oct 14, 2024

sonarqubecloud bot commented Feb 19, 2025

Quality Gate failed

graemerocher left a comment

Choose a reason for hiding this comment

graemerocher Feb 24, 2025

Choose a reason for hiding this comment

yawkat Feb 24, 2025

Choose a reason for hiding this comment

graemerocher Feb 24, 2025

Choose a reason for hiding this comment

yawkat Feb 24, 2025

Choose a reason for hiding this comment

yawkat commented Feb 24, 2025

graemerocher commented Feb 24, 2025

yawkat commented Feb 24, 2025

yawkat commented Feb 25, 2025

graemerocher Feb 25, 2025

Choose a reason for hiding this comment

yawkat Feb 25, 2025

Choose a reason for hiding this comment