Visual Node Editor

[vvzen]

Hi @PaulSchweizer !
This is an awesome project which we're still using heavily at Framestore IA, so thanks for that. :)
Today I was letting my imagination run wild and I started to think.. what if you could have a visual node graph editor that would let create your custom graph, using your predefined nodes?
Then you could plug things in a intuitive visual way, and once you're happy with your graph, you could then "bake" it to a declarative format (json, yaml, GraphML.. etc).

The reason for that is that I feel that sometimes implementing the graph in python doesn't really feel natural, and it may be hard to decide what should be the exact scope of a node (it's easy to get too much stuff going on or get too granular and only implement a function in a node) - the result is that sometimes people have different way of seeing whaat a node should be, and you end up with graph that doesn't just connect nodes but also hosts a major part of your business logic.. But if you had the chance to plug your things visually, then, imho, writing the api all of each nodes would be much easier, and a graph would only be the deterministic result of a bunch of nodes plugged together. No logic required.
Also, this could become an incredibily powerful tool for tecnical people to provide a powerful abstraction for creating complex automations!
(Or maybe it's just that I've used Quartz Composer too much before I started programming :D )

So I was just wondering if you ever thought about implementing something like that, or it's too much outside the scope of the library.

Thanks,

Valerio

PS: as a reference, this is kinda what I'm referring to: https://github.com/paceholder/nodeeditor

[PaulSchweizer]

Hi @vvzen,

That's nice to hear, send my regards to the team at Framestore, I miss 'em!!!

As far as the node editor goes, you're absolutely right, this would be the next level!

In terms of visualization, @anton4 has done some great work on it, the discussion is here: #39
And his code is here: https://github.com/ml2grow/graph-flow-vis
His solution is web-based which makes it very versatile and extendible, I really like that approach but unfortunately I'd (massively) need to brush up on my web dev skills in order to get anywhere with it anytime soon, but I think this would be a great solution when extended with editing capabilities. Web app is a great choice because you "just" need a browser to use it and it could be wrapped into a desktop app with electron to provide more integrated features.

I personally have tinkered with a nice PySide2 library: https://github.com/jchanvfx/NodeGraphQt. That lib is really well done and actively being developed. I managed to create a very basic proof of concept with it but haven't uploaded it anywhere yet. When I have some time (next week) I'll dig upwhat I have and upload it so you can give it a go yourself.

In short: If you want to contribute to (or start) such a project, you have my full support. First step would be deciding the technology (web or python) I guess.

[vvzen]

Hi @PaulSchweizer !
They definitely miss you too! :)

https://github.com/ml2grow/graph-flow-vis seems like a very nice project, is based on D3 which is a great data viz library for the web, even if the API it's quite convoluted compared to something like PaperJS or even ThreeJS. I've used to work with d3 in the past and also other canvas/svg web libraries so I would be happy to give a hand in this direction!
I feel a web app would be a wise choice especially since it could be easier to integrate with Shotgun/Ftrack for people in vfx, and generally speaking it's way more portable, even though web technologies fly at a much greater pace than desktop so it would probably need a constant flow of love to be working across browsers :)
Performance-wise I've seen mind blowing thing done with WebGL so it shouldn't be an issue.

On the other hand, PySide2 seems a more robust and solid approach, but I would personally pick Qt Quick 2 and QML for this kind of tasks. It's designed with a clear separation between the logic and the presentation, and also makes it easy to create reusable UI components. Also, I think the QML scene graph optimizes a lot of draw calls (only redraws what has changed, instead of always redrawing everything) compared to the standard PySide2 widgets, which could prove useful to retain a nice FPS even on complex graphs.

So I guess it's really a matter of which direction you feel is worth following.
I would be happy to pick one or the either, it's been a while since I've done something like this and it could be a nice challenge either way! :)
I'd be curious to see your proof of concept, too!

Thank you!

Valerio

[PaulSchweizer]

Hi @vvzen,

Here's my proof of concept with PySide2: https://github.com/PaulSchweizer/flowpipe_editor , give it a try.

I think the web app is the preferred solution for most users, but it takes take more time to implement (at least for me).
The Pyside version is quicker to implement but less attractive to the general user base of flowpipe since you need to have Qt/Pyside installed on your system.

I personally am in favor of the web app solution, also because I can learn some new stuff along the way.

[vvzen]

Hi @PaulSchweizer !
cool, I'll start to work on the web version. For now, I think I'll gather ideas on which library it's better for this scope ('cause I definitely don't want to start writing WebGL stuff from scratch!).
I was looking at Twojs, which seems nice given the support for canvas/webgl/svg, and the fact that the API is inspired by Threejs, which I really like; it also uses a nodegraph which makes it easier to work with parents, children, etc..

I was trying to test the flowpipe_editor, but for now it's throwing me this error:

QWidget: Must construct a QApplication before a QWidget
[1]    45238 abort      python test_flowpipe_editor.py

I've been able to track it back to what you're doing in the __init__.py of floweditor :
QTGRAPH = NodeGraph() (which I guess creates a QtWidget behind the scenes?)

..I was starting to patch it when I realised that probably you could do it faster than me. :D
And also that my weekend energies would have been better spent on looking for a js library :)

PS: I'm currently testing from macos=10.14.6, python=2.7.15, pyside2=5.13.2, qt.py=1.2.2

[PaulSchweizer]

Ah, sorry, forgot to put the QApplication in there, should work now.

As for the js libs, I wonder if there might already be a library for graph networks that we could use?

[vvzen]

Still happening, I tried the same thing before having a deeper look at what was causing it. If it was so easy I would have done it myself 😛

There's one js library specifically made to handle graphs, but I feel it's too specific for what could be the aim of the node editor: https://github.com/ericdrowell/ElGrapho .
Same thing for http://sigmajs.org/ .
They are more aimed at generating graphs from data (basically data viz of networks).
While I think we could need something that offers enough flexibility in terms of drawing and handling the events.
Also, I feel that, as powerful as these two libs may be, they're lacking a good UX (and design choices, at least for my taste :) )

EDIT: after more searches, maybe Fabric.js and Konvajs are also worth a look, given their focus on user interaction

[vvzen]

Hi @PaulSchweizer !
I just wanted to let you know that I've just started to work on the web editor in my spare time: https://github.com/vvzen/flowpipe-webeditor .
I could only dedicate to it a couple of hours while commuting, but so far Fabric.js seems a very nice choice: it comes with a lot of very useful features (zooming and object interaction, to name a few) and I really like the OOP design of the API. Let me know if there's anything specific we should aim to have in a first iteration.

Valerio

[PaulSchweizer]

Thanks for getting it started @vvzen!

fabric.js seems like a good choice, let's run with it unless we come across something even better.

For the beginning, I'd say we should simply focus on displaying an arbitrary graph of nodes without much styling or finesse and no interactivity other than what already comes with fabric.js, like zooming and scrolling. That should be challenging enough for a start while still achievable and should provide the necessary groundwork for the next steps, what d'you think?

[vvzen]

Cool!

In order to make a dev plan for the next weekends (thanks to the corona 🦠 I'll probably have more weekends at home) I've outlined some steps as a general recap, I would like to know your opinion on them:

General workflow

1. The webapp would be used to create & connect a graph of nodes
2. Then the graph would be serialized to a static file
   2.1 There are many formats that we could support (dot, yaml, GraphML). For now, maybe we should simply focus to JSON (with the same kind of schema that you already support in flowpipe)?

Nodes
One thing that I'm not too sure about is how the web editor will know what nodes to use.
What I mean is that we need a mechanism, that somehow, given a python module, exposes the flow pipe nodes described in the python code as nodes that can be actually used in the web editor. I would call this binding a node.
Because the way I was thinking about it is that:

1. Developers will write a set of flow pipe nodes in a module
2. The web editor is somehow made aware of these nodes which are then available to be used
3. The graphs can then be created by users/developers

Maybe we could use the python module ast (abstract syntax tree) to do some of this stuff.
In theory we only need to know the number of inputs, number of outputs, and name of each node.
(then a totally different chapter is whether we want to also add support for the type of each plug, so that you can only plug a string into a string, etc..)

UI/UX
On the interactivity layer, fabricjs already gives us:

1. Grouping objects together
2. Dragging objects to move them
3. Zoom (needs to be refined)
4. Pan

My next step would have been to implement a mechanism to plug the nodes

Cheers!

PS: also for now I wouldn't focus on supporting subgraphs. It could be done in the future once the main functionality is there

[PaulSchweizer]

General workflow:
Agreed, let's focus on recreating the json format that is already implemented in flowpipe for now so we can immediately execute the graph. Other formats can follow at a later stage if there is a need.
We will also need a way to load existing graphs here.

Nodes:
We're on the same page here too. I think if the backend has access to the python library that contains the nodes, we won't have an issue here. I'm imagining running this as a flask app, like @Anton-4 's https://github.com/ml2grow/graph-flow-vis
The easiest way would then be to just pre-register all nodes in a simple list and just use that. Later, we can add a "plugin" architecture that allows us to load the nodes from file paths that the user can specify as env vars for the backend, but I think a simple list would suffice for now. As you say, the nodes would then be converted to a json dict containing the attributes necessary for the frontend to work with them (name, inputs, outputs, later the types ...)

UI:
Sounds good, I guess we'd also build some sort of node and plug "widgets" for/with fabric.js, right?

And yes, let"s ignore subgraphs and compound plugs and any other complex concepts for now.

And also, let's move further detailed discussion to your repo and only add important updates to his ticket, I'd say. We can set up tickets and a road map in there as well then.

I'm looking forward to this!

[vvzen]

Just as an update - I had to shift priorities a little bit since luckily I got some more work to do in the weekends. But I've reached a initial state which I'll try to document soon (this weekend?) and add some gifs/videos to show what currently works and what doesn't - then we can maybe also set milestones?