Hot reload of secrets via environment files

[mannharleen]

Both docker swarm and k8s allows mounting secrets as volumes. These secrets can then be altered at the swarm/k8 level, but the swarm task/k8 pod needs to be reloaded/restarted to apply new secrets. This is != hot reload.

Within benthos we already have the ability to hot reload the config.yaml using watch flag. Can we extend it to be used with secrets as well?

PS. Can hot reload also be extended to ressurces and templates?
PS. k8s also allows configuring secrets as docker env variables, but that is out of scope for this discussion (I think)

[Jeffail]

Is the requirement here that we watch for environment variable changes and reload as appropriate? If so I'm not sure that's actually possible from within the process itself (unless that process is what's modifying the variable).

Resource configs will get reloaded the same as the regular config.

[mannharleen]

Ok, so resource configs do reload, but still templates don't!

Regarding the process environments,
What do you mean by "unless that process is whats modifying the variable" ? The process is setting the variables in the first place anyway:
https://github.com/Jeffail/benthos/blob/4e741c2ca5914196ce143125ddcd68d1e4c8027f/lib/service/run.go#L183

Why I think its important: Currently the only way to pass credentials/secrets to benthos process is via env-variables (** see below). Now think of DB hostname changes or password/key rotations. If we dont want to have any down-time we need some kind of hot reload for secrets.

** I wish there were a inbuilt secret manager (backed by a configurable cache/DB) and then if we could reference the secrets manager within bloblang and interpolation. e.g. ${! get_secret("XXX", TTL) } and voila!