Parameterized Graphs (aka Modules)

[liamgriffiths]

Warning

This branch is still a work-in-progress.

We'd like the ability to be able to re-use Substate graphs and use them as nodes in other graphs. There's a two main components to this that I've been working on in this branch: parameterizing graph "inputs" and representing a graph as another node.

We're kicking around the idea of naming this Module for now - this will be exposed as a node that can be used in other graphs.

Right now this change would support using a subgraph through the JSON serialization directly in the Module node, but when we implement publishing these modules we'd also like to reference them by id (or possible name or uri)

The result will enable the following sort of SDK usage:

#!/usr/bin/env -S npx ts-node --transpileOnly

import { Substrate, Box, Module, sb } from "substrate";

async function main() {
  const SUBSTRATE_API_KEY = process.env["SUBSTRATE_API_KEY"];
  const substrate = new Substrate({ apiKey: SUBSTRATE_API_KEY });

  const x = sb.var({ type: "string", default: "hello" });
  const y = sb.var({ type: "string" });
  const z = sb.var({ type: "object", properties: {} });

  const a = new Box({ value: { a: x, z: z, array: [x, x, x] } }, { id: "A" });
  const b = new Box(
    { value: { b: sb.interpolate`x=${a.future.value.get("a")}, y=${y}` } },
    { id: "B" },
  );

  // publish the module on substrate.run
  const publication = await substrate.module.publish({
    name: "my reusable graph",
    nodes: [a, b],
    inputs: { x, y, z },
  });
  console.log("published:", publication.json);

  // using the module from JSON
  const mod = new Module({
    module_json: substrate.module.serialize({
      nodes: [a, b],
      inputs: { x, y, z },
    }),
    inputs: {
      // when commented will use "hello" because it is defined as the default above
      // x: 123,
      y: "yyy",
      z: {
        arr: ["123"],
      },
    },
  });

  // using the module from publication/module id
  // const mod = new Module({
  //   module_id: publication.id,
  //   inputs: { y: "yyy", z: { arr: ["123"] } },
  // });

  const c = new Box(
    {
      value: {
        "1": mod.future.get("A.value.z.arr[0]"),
        "2": mod.future.get("B.value.b"),
      },
    },
    { id: "C" },
  );

  const res = await substrate.run(mod, c);
  console.log(JSON.stringify(res.json, null, 2));
}
main();

Related server implementation: https://github.com/SubstrateLabs/substrate/pull/1401