Hetzner Cloud integration

[bastelfreak]

Purpose

the purpose of this beaker-plugin is to extend beaker in a way that it can automatically provision servers at Hetzner Hcloud.

• Hetzner Cloud: https://www.hetzner.com/cloud?country=de
• Main beaker project: https://github.com/voxpupuli/beaker

Background

Beaker, in combination with rspec and serverspec, is the acceptance testing framework. Beakers main job is to prepare an environment where we can run tests. The usual workflow is:

• Spin up a container or VM. This can be local or remote
• Use the hypervisor/API to enable automatic access (usually via ssh key)
• start a whole operating system
• install puppet
• rspec and our helpers will copy puppet code to the target, run it, then validate the system with serverspec

Beaker is very flexible and enhanced with plugins, usually one plugin per hypervisor. the most common ones are:

• Docker - https://github.com/voxpupuli/beaker-docker
• Vagrant (works fine with virtualbox and libvirt as backend) - https://github.com/voxpupuli/beaker-vagrant

We've also some libs:

• https://github.com/voxpupuli/beaker-puppet - brings puppet onto the target
• https://github.com/voxpupuli/beaker-rspec - rspec integration
• https://github.com/voxpupuli/beaker-module_install_helper - can install puppet modules onto targets
• https://github.com/voxpupuli/beaker-hostgenerator - generates a node set (information about the system beaker should create)

And some plugins that also work with other cloud vendors:

• AWS - https://github.com/voxpupuli/beaker-aws
• Google Cloud Platform - https://github.com/puppetlabs/beaker-google

tasks in this gem

• extend beaker-hcloud so it can interact with hetzner cloud
  • automatically create beaker instances with predefined OS templates from hetzner
  • support custom OS images
  • cleanup stale VMs
  • cleanup VMs after jobs are done
  • provide the desired availability zone
• extend beaker-hostgenerator
  • it probably needs some adjustments for the hetzner cloud specific os names: https://github.com/voxpupuli/beaker-hostgenerator/tree/master/lib/beaker-hostgenerator/hypervisor

Hetzner provides ruby API bindings: https://rubygems.org/gems/hcloud. Whereever possible they should be used instead of the API: https://docs.hetzner.cloud/

[oneiros]

Hi @bastelfreak! I prototyped this a little bit and wonder how to deal with SSH keys:

Similarly to AWS, Hetzner requires you to set up SSH Keys (manual upload, or via API) before they can be used to provision servers. The AWS integration for beaker has a nice implementation that generates a unique name for an SSH Key, creates that in AWS in the provisioning phase and then removes the key again during cleanup.

We could do the same, but there is a small caveat: Hetzner enforces uniqueness of keys. You cannot upload the same key with a different name. If you only ever run one beaker job at a time and always use a key that you never want to use within the same project in your hetzner account, this is not a problem. But when concurrent CI jobs are possible and/or you want to run jobs manually using the same key, things could get ugly.

A slightly more robust approach could be to:

1. Search for existing keys at hetzner
   1. If a matching key is found, use that
   2. If no matching key is found, set up a new one
2. Provision with the chosen key
3. During cleanup remove the key if and only if it was created during step 1.

This way we would not clobber existing keys and could safely support different use cases using the same key. But still not concurrently ☹️
For concurrent jobs using the algorithm above would present all kinds of race conditions.

Of course we could simply skip the cleanup of keys, but I would find that a bit messy.

A cheap solution could be to simply skip management of SSH keys altogether. In that case a user would have to set up their key at hetzner in whichever way they prefer and then configure within beaker which key to use. This way we would be rid of all the problems mentioned above, but it would require more manual setup and be less "plug&play".

What do you think?

[ekohl]

Is generation of SSH keys not cheap enough that you can generate a key for each individual run? beaker-vagrant does generate a Vagrantfile for every setfile. Perhaps a similar approach could be taken to generate a key in the local directory?

[bastelfreak]

@oneiros I agree with @ekohl here. I think generating ed25519 keys on the fly isn't that expensive. It doesn't require a lot of randomness and also not a lot of CPU time. We could generate one for each job, upload them to hetzner and maybe prefix them with beaker_ or something similar, and purge them after the job finished/failed. The prefix allows us to easily identify and purge keys manually.

[oneiros]

I agree and it makes total sense to me.

Being new to beaker and its ecosystem, I was not sure, what the expections were. beaker itself compiles a set of ssh options using existing keys from the user. The aws "hypervisor" builds on that and I thought it might be least surprising to stay as close to this behavior as possible.

But given hetzner's constraints and the fact that a truly "plug&play" solution is probably even less error-prone, I will gladly go ahead with your suggestion.

[bastelfreak]

awesome, thanks!

[oneiros]

I just opened PR #11 with my first attempt at an implementation that "works for me" ™️

Generating ed25519 keys from ruby turned out to be a bit trickier than expected. While there is the ed25519 gem to generate the raw key material and there are a couple of gems that can handle importing different key formats used by OpenSSH, none was able to serialize the private key to OpenSSH's format.

In the end I settled on Github's ssh_data gem and monkey patched the missing methods in, because that gem already had all the necessary low level encoding methods to create the private key file. Additionally, this SO answer was invaluable in figuring out, how OpenSSH's format works: https://stackoverflow.com/a/73902911

A couple of remarks with regards to the requirements mentioned above:

support custom OS images

For now I opted to specify OS images by name. According to the docs this should work for both the ready-made images Hetzner provides as well as custom ones, i.e. VM snapshots stored in your account. I did however not test this.

cleanup stale VMs

I am not sure how and if this can be done. So instead I opted to add a paragraph about cleanup to the README similar to what the beaker-google gem does.

provide the desired availability zone

Sorry, I am not sure what this actually means 🙂.

I did provide an option to specify a datacenter location. Maybe that is it?

(Note that Hetzner has multiple datacenters at these locations and indeed the API allows to name a specific datacenter instead of the location, but I thought giving the location should suffice as that is what you select when creating servers manually as well.)

[bastelfreak]

Thanks for all the work! I will try to review/test/respond in detail in the next couple of days.