Ephemeral Service Evolution

[strieflin]

There is an emerging demand for integrating other MPC runtimes into Carbyne Stack (CS), e.g.:

1. Cybernetica has announced to integrate Sharemind MPC with CS.
2. We at Bosch are working together with the team at Boston University on integration of their Secrecy MPC DBMS into CS.
3. In our discussions with other organizations, we heard many times that more flexibility regarding the MPC runtime would be very much appreciated.

At the moment, however, Ephemeral, like the rest of the CS platform, is deeply "entangled" with the MP-SPDZ MPC runtime - and more specifically with its SPDZ2 active security dishonest majority protocol implementation. This post should be seen as a starting point for evolving the Ephemeral service towards extensibility in terms of support for additional MPC runtimes.

To get the discussion started, we came up with a first high-level proposal how the desired extensibility can be achieved. The key idea is to decouple the orchestration logic that is dealing with, e.g., networking setup, authorization, and the interaction with other CS services (Castor for fetching correlated randomness and Amphora for I/O) from the MPC runtime. This is facilitated by introducing an API (called the Ephemeral Integration Interface) between the orchestrator and the runtime. This API will initially provide the following functionality:

• Providing setup configuration (e.g., the network endpoints the runtime should use to connect to its peers).
• Providing Correlated Randomness to the MPC runtime
• Providing input secret shares to and consuming result output shares from the MPC runtime

Technically speaking, Ephemeral will be split into two parts (see the figure below): The orchestrator and an MPC runtime. The later might be complemented by a sidecar that translates interactions via the API into something that can be understood by the MPC runtime. the Both will run inside the same pod.

As a first iteration towards a general concept, we propose to reuse the interaction mechanisms that are implemented in Ephemeral today, i.e., using named pipes to transfer correlated randomness and sockets for handling I/O.

This is a still very sketchy high-level draft of what we at Bosch plan to implement in 2024. Any feedback is highly appreciated.

[ktali]

The new API would give integrators more clarity on the capabilities of the system. Further, having the MPC runtime as a container could indeed make the extension process simpler. I assume the goal is to have the Orchestrator as a foundational component of Carbyne Stack, with Runtime containers defined by the integrators.

I'm however concerned about the long term sustainability of such API.

• The interface would need to be iterated upon as future runtime integrators will have exotic requirements. The Orchestrator component would accumulate business logic that serves only a subset of runtimes, making it less generic.
• Changes in the existing API put pressure on existing integrators to stay compatible with the latest Orchestrator/Carbyne Stack. Needs constant involvement by all in the decision process.

The key idea is to decouple the orchestration logic that is dealing with, e.g., networking setup, authorization, and the interaction with other CS services (Castor for fetching correlated randomness and Amphora for I/O) from the MPC runtime.

I'm curious whether the Runtime<->Orchestrator API for interaction with other CS services is necessary for extensibility. Since it's just indirection between cluster services' APIs and the runtime, why not access them directly? Networking setup and authorization are more fitting to leave to the orchestrator as these are inherent to any MPC runtime or task, but intra-VCP service access is not orchestration in my opinion.

Redistributing the responsibilities that Ephemeral currently has could be done as so:

EII would contain only the bare necessities in terms of orchestration, like publishing and consuming some task lifecycle events, reading network configuration, inputs and results (Amphora UUIDs), etc. However I think the discovery service is a more fitting component to mediate these messages between the orchestrator and Runtime instead.