Configure loki

[nbenmoody]

Description

This PR collects up work required to install and configure a Loki server on a new Droplet. Given that so much peripheral work was needed amongst these scripts, since the new Droplet for Loki is being created using the new terraform provisioning process, there are a lot of changes across these scripts along with the Loki configuration.

Related Issues

• 21 - Set up alerts for relay.nos.social

Changes Made

• I created a new role to handle all the tasks required for ssh keys and hardening, to replace the original harden role.
• I added a draft playbook/inventory/role for Loki installation and configuration.
• I deprecated the old new-do-droplet playbook.

Additional Notes

This installs and configures the Loki server itself, but integration with our existing Grafana instance is another step (to be carried out manually, for now). This also does not configure the agents that will push logs to Loki, which will come as a next PR here.

[mplorentz]

LGTM so far.

[nbenmoody]

@mplorentz - This PR is now ready for full review. The Loki server has been configured using it, and is standing up at loki.planetary.tools, and integrated as a datasource in Grafana now.