[RFC] Add mesh VPN support to the CDH

[portersrc]

CoCo currently has no first-class support for securing the communication between enclaves. This responsibility has been left to the application, but CoCo could provide this optional feature. This RFC outlines what changes in the confidential data hub could support such a feature.

The following high-level diagram shows how this could work:

Orange nodes are where the main changes would be needed, including in the CDH. The main points are:

1. A "mesh daemon", in this case Nebula [7], would provide the fundamental mesh support (peer discovery, establishing routes, encryption). It runs inside the guest VM. This would be packaged with guest-components.
2. CDH must be enlightened with (a) general mesh setup support/structs and (b) specifically support to start a nebula mesh.
3. If this option is enabled by the user/workload owner, then kata-agent would invoke the CDH's mesh setup at start.
4. trustee must be able to supply any needed secrets for such a mesh [4].
5. The encrypted mesh would co-exist alongside the untrusted kubernetes overlay network, and both would be exposed and usable by the workload pod.

---

Proposed changes to guest-components

• Add a mesh VPN daemon, Nebula, to guest-components.
• Augment CDH with an encrypted mesh folder (alongside kms, secret, and storage)
• Within that mesh folder, provide a Nebula implementation. Mesh set up entails:
  • Requesting a key, cert, and CA cert from trustee
  • Starting the mesh daemon

External changes

Changes outside of guest-components that would be needed for this approach include:

• kata-agent needs to be able to conditionally invoke the CDH's mesh setup at start-up
• trustee needs to be able to serve mesh-related secrets [4]

Alternative designs and frameworks

Alternative designs that were considered, each with varying merits:

• Traditional VPN implementations such as Wireguard, OpenVPN, IPSec (strongswan, libreswan)
• Istio + Envoy

References

[1] August 15, 2024 community meeting with VPN talk
[2] May 9, 2024 community meeting with VPN talk
[3] guest-components PR 634 (query string support)
[4] trustee Issue 396 (extend KBS with plugin and mesh plugin support)
[5] trustee PR 451 (address 396)
[6] Secure Comms for peer pods
[7] Nebula docs

[Xynnn007]

Great. Nice to see that CoCo stepped image, memory, and now we are conquering networking and storage...

Some questions at first glance

1. How to ensure the availibility of the Nebula VPN daemon?
2. Once initdata is handled, probably we do not need to do any change upon kata-agent?

[ChengyuZhu6]

Great. Nice to see that CoCo stepped image, memory, and now we are conquering networking and storage...

Some questions at first glance

1. How to ensure the availibility of the Nebula VPN daemon?
2. Once initdata is handled, probably we do not need to do any change upon kata-agent?

It would be better if Nebula could be hidden behind CDH, allowing CDH to control its availability and making Nebula transparent to the kata-agent.

[portersrc]

Thanks for the initial feedback.

How to ensure the availability of the Nebula VPN daemon?

Currently I launch the Nebula daemon from the encrypted-mesh module that I added to guest-components. If it dies, the VPN won't work. This shouldn't lead to any leaked traffic, because the application should fail to send any secure data on that interface at that point. That said, it would be much better to have some kind of way to restart it, like with systemd restart policies.

Once initdata is handled, probably we do not need to do any change upon kata-agent?

That would be ideal. With initdata, is the idea to make all of it available inside of the CDH for other guest-components modules to access?

It would be better if Nebula could be hidden behind CDH, allowing CDH to control its availability and making Nebula transparent to the kata-agent.

I completely agree. If initdata can remove this dependency, then kata-agent will not need any mesh-related changes.