Experiment with use of private VMs

[malachig]

Currently all VMs seem to be using ephemeral external IPs. Google charges for these similar to what they would charge if you actually reserve an IP, even if its in use.

It also theoretically makes the instance less secure. It not clear that we actually need to be able to access worker instances externally. Perhaps, only the cromwell server needs to be accessed externally. If one did need to get onto an individual worker instance, that could still be done by using the Cromwell server itself as a jump point.

Here is some discussion of how to do this.
https://github.com/atgu/cromwell_google_setup#using-vms-with-private-ip-addresses

Essentially to do this you can add noAddress: true to the runtime block of a task. We could test this and see if it breaks anything.

[Layth17]

Upon some testing, noAddress: true seems to break some tasks...

This runtime attribute adds support to disable assigning external IP addresses to VMs provisioned by the Google backend. If set to true, the VM will NOT be provided with a public IP address, and only contain an internal IP. If this option is enabled, the associated job can only load docker images from Google Container Registry, and the job executable cannot use external services other than Google APIs.

@tmooney suggested that "maybe it’d work if we used the docker image cache feature too (or if we pointed only to http://gcr.io images)"

[malachig]

It seems the GCP is transitioning from the "Container Registry" to the "Artifact Registry". I wonder if Cromwell already supports this.

[malachig]

If we wanted to try using the docker image cache feature crudely this involves something like this:

• Start a VM with Docker capabilities.
• Attach an ext4 disk to this VM from which you will create an image.
• Pull the docker images you want.
• Copy the Docker directory into the root of the ext4 disk.
• Shut down the VM
• Create an image from that disk. If the image has name: "immuno-docker-images", then the full name of that image will be "projects/griffith-lab/global/images/immuno-docker-images"

Then set up cromwell to look for images in this location as described here:
https://cromwell.readthedocs.io/en/latest/backends/Google/#docker-image-cache-support