Cancel Running Simulations

[Mythicaeda]

• Tickets addressed: Closes Need a way to stop a running simulation #765
• Review: By commit
• Merge strategy: Merge (no squash)

Description

Adds a SimulationCancelListener that, upon receiving a signal to cancel a simulation from the DB, sets a flag to tell the Driver to stop before it either picks up or begins to perform more work. With this change, the only time a simulation will not be stoppable aside from shutting down the container is if the user code contains an infinite loop. It also alters how the canceled flag is set in the DB to accurately reflect that "if this flag is set, then the simulation was canceled".

Breakdown of commits:

• Commit 1 removes our auto-cancel triggers and sends a notification to the simulation_cancel channel when a simulation is canceled. The auto-cancel triggers were removed in order to give the users more control over when their simulations are canceled. This does mean that if anything is updated between a pending request and when it is picked up, it will now fail with an "Request is no longer relevant" message.
• Commit 2 adds Java code to listen for the new notification added in (1). Rather than using a separate DB connection for this, it was added to the list of subscribed channels in the Worker's Listeners.
• Commit 3 threads this listener through to the SimulationDriver. This is a breaking change, because it alters the signature of simulate.
• Commit 4 alters the behavior of the Driver to check if the simulation has been canceled before it fetches or runs a batch, and, if it has, to return the results up until when it was canceled with its reason field set to how far into simulation it was (using the same JSON structure as when simulation fails with an exception).
• Commits 5-7 update our tests.

Verification

• A new e2e test was added to check that canceling works and returns incomplete results. This test uses the foo model.
• DB Tests checking the removed triggers were also removed
• SimulationDatasets in our e2eTests now contain the information from the status and reason columns.
• Unit tests have been updated with the new simulate syntax. All of them were set to be uncancelable.

Documentation

No docs need to be updated.

Future work

• Is there a generic name for SimulationFailure? Prior to this PR, it was exclusively being used to fill in the reason column for failed simulations, but it's now also filling in the reason column for canceled sims.
• Scheduling ticket: Need a way to stop a scheduling run #1241
• UI needs to be updated to 1) allow canceling an inprogress simulation and 2) to properly display what a canceled simulation looks like (Useful note: canceled sims can only be incomplete or pending. A failed or success sim with canceled set to true implies that the simulation either finished or failed before it received the canceled signal)

[mattdailis]

Awesome work! No major concerns

Key:
🔻 Issues to address before merge

🔶 Requests that should not block merge, but should at least be discussed

🔵 Recommendations that can be ignored if desired

[mattdailis]

Did a quick manual test, re-approving 👍