Add timeout to RMQ

[kasesalum]

Requests taking too long and never timing out could cause incidents,
hence adding a configurable timeout. In case the timeout isn't added,
no timeout is enforced.

[indrekj]

Missing the reason why and this may break things

[kasesalum]

@indrekj Added the reason, changed the code in a way that should not affect its current usage. I'm having trouble creating tests for it, though. Do you have any suggestions?

[indrekj]

options is not documented

[indrekj]

I'm still not convinced that having this kind of high-level timeout is a good idea. See:

• https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/
• https://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/

Maybe if it's large enough and is triggered on very exceptional cases, it could be ok.

cc @sviik

[kasesalum]

@indrekj Siim is on vacation for 2 weeks. Do you maybe have any suggestions on how to better approach the ticket?

[indrekj]

Hm... If we would go with a timeout then that has to end up as an error and also in the sentry (and opsgenie). The current "logger.warn" will get ignored and if we get into a corrupted state then that would be really bad. We probably would need to make sure the error reaches the actual application and then the application can send it to Sentry.

I also looked into Rack. We don't have a timeout there either.

Rack has "Rack::Timeout" module, but it also warns about the corrupt state. One solution to avoid that is to restart the whole process on timeout.

I'm leaning toward not adding the timeout at all. Or if you do, then let's start on the application level, instead of adding it to the library immediately. And we probably should restart the process.