gnd: Support multiple subgraphs, grafting, subgraph composition in dev mode #6000
Conversation
…kResolver with the right base dir for a subgraph
incrypto32
changed the title
incrypto32
force-pushed
the
krishna/graph-dev-composition-2
branch
from
0826b2a to
b5bbf93
Compare
There was a problem hiding this comment.
This is nice; I feel that we need a more general concept around finding related subgraphs in the code, maybe some kind of LinkMapper that is pretty much a noop for IPFS, and knows where different subgraphs are in the filesystem for dev mode. If we had that, I think we could set these up in the respective main methods, and wouldn't have to guess as much in the link resolver.
| // The manifest path is the path of the subgraph manifest file in the build directory | ||
| // We use the parent directory as the base directory for the new resolver | ||
| let base_dir = canonical_manifest_path | ||
| .parent() |
There was a problem hiding this comment.
I am a bit confused now, isn't this basically base_dir/deployment_str/.., i.e. base_dir ?
There was a problem hiding this comment.
deployment_str can be something like "../subgraph2/subgraph.yaml" in that case the new base_dir is parent of "base_dir/../subgraph2/subgraph.yaml" which is "../subgraph2"
When deployment_str is an absolute path its simply the directory in which the subgraph.yaml is
| @@ -50,6 +50,8 @@ impl FileLinkResolver { | |||
|
|
|||
| /// This method creates a new resolver that is scoped to a specific subgraph | |||
| /// It will set the base directory to the parent directory of the manifest path | |||
| /// This is required because paths mentioned in the subgraph manifest are relative paths | |||
| /// and we need a new resolver with the right base directory for the specific subgraph | |||
There was a problem hiding this comment.
I haven't followed the flow of the code overall too closely, but is there any way where we could first find the manifest we are interested in and then create the FileLinkResolver based on the location of the manifest?
There was a problem hiding this comment.
@lutter I dont follow your suggestion here. Are you suggesting getting rid of the aliases and instead just relying on the path of the manifest for creating the FileLinkResovler?
| /// For other resolvers, this method will simply return the current resolver | ||
| /// This is required because paths mentioned in the subgraph manifest are relative paths | ||
| /// and we need a new resolver with the right base directory for the specific subgraph | ||
| fn for_deployment(&self, deployment: DeploymentHash) -> Result<Box<dyn LinkResolver>, Error>; |
There was a problem hiding this comment.
Reading the comment, it might make sense to rather have a method for_manifest(manifest_path: PathBuf). By using the deployment here, we are still encoding a convention about where deployments live relative to each other. It would be nicer if that convention was only used to find manifests and everything else follows from there. Worst case, we could even store the path from which a manifest was read in the manifest itself to make it easier to get the path whereever it is needed. In a way, that would nicely express what changes with this PR: we used to know implicitly where manifests live because they were all on IPFS under their hash. Now, a manifest can live there, but also in the filesystem, and I think we should just express that explicitly
| false | ||
| } | ||
|
|
||
| /// Redeploys all subgraphs in the order defined by the BTreeMap |
There was a problem hiding this comment.
I assume what's meant is the order in which they appear in manifest_paths
| @@ -330,6 +331,7 @@ where | |||
| self.version_switching_mode, | |||
| &resolver, | |||
| history_blocks, | |||
| ignore_graft_base, | |||
There was a problem hiding this comment.
Instead of threading ignore_graft_base through, you could also just manipulate the raw here and remove any mention of a graft
Supporting multiple subgraphs in dev mode is a bit tricky since FileLinkResolver needs a base dir, this works well for a single subgraph since we can scope the resolver to the build directory of that subgraph, but when there are multiple subgraphs, we need to set the base_dir of the FileLinkResolver dynamically. This PR implements the mechanisms for that.
This is done by using a
link_resolver_overridethats passed upto the subgraph runner which switches to using it when available.For supporting subgraph datasource, aliases are used users can declare their subgraph datasources in the manifests normally like below example.
When they run gnd they have to pass in the flag