A discussion on scripting with Molecular Nodes and Blender more generally

[BradyAJohnston]

This is an attempt to collect thoughts and projects related to scientific scripting and rendering with Blender. It would better fit in a more general-purpose place for scientific interaction & visualisation with Blender (like a GitHub organisation for such), but as it currently doesn't exist I figured here would be a good a place as any to collect links, thoughts and comments from projects and people working in the space.

The text below on this initial post will be updated with additional links to projects that are relevant as they are found.

Molecular Nodes related projects

• Molecular Nodes
• ggmolvis: potential for more scripting and API usage with Molecular Nodes
• prettymol: potential for more scripting and API usage with Molecular Nodes
• basimol: experiment with scripting with Molecular Nodes via baslisp language
• experiments with GPU module: using the gpu module inside of Blender with MDAnalysis to visualise data & results in the viewport, enabling better interaction & feedback
• Visualising Viruses: Molecular Nodes talk at BCON22
• MN & MDAnalysis workshop: recording of workshop showing intro to MDA and MN

Molecular data related:

• Microscopy Nodes: add-on focused on microscopy data import
• batoms: more focused on small-molecule and material science rendering of atomic data in Blender
• Atomic Blender: original import of molecular data including .pdb files into Blender (used to ship with Blender, now an extension)
• pyrite: older add-on for import MD data into Blender
• upy & AutoPack: older add-on for importing molecular data, coupled with CellPack usage
• CellBlender: simulation of molecular data in Blender

General science-related:

• Surf Blender Course, website: General course and introduction to using and programming with Blender for scientists and scientific visualisation
• blender.science Discord: A discord server for scientific data visualisation in Blender
• MedBlender: radiation & DICOM visualisation in Blender
• BioxelNodes: volumetric data visualisation in Blender
• databpy: more data-oriented wrappers for interfacing with Blender & mesh objects that work well with numpy arrays
• blendernc: netCDF, grib, and zarr datasets in Blender
• blext: blender extension building & management
• extensions: A custom Blender Extensions website, built & hosted via GitHub and GitHub actions. Potential to host extensions that wouldn't otherwise be accepted into the more general Extensions website (i.e. for requiring external scientific dependencies)
• bpy_jupyter, previous: blender_notebook, notebookconnector: Linking a Jupyter Notebook to a live Blender session, enabling scripting from within the notebook that is reflected in the Blender scene
• typst importer: import .typ files for rendering text, math & equations
• csv_importer: general CSV importer powered by databpy and polars
• CG Figures: Many different approaches to modelling various scientific lab equipment and data in Blender
• bpy-gallery: results from different scripting experiments with the bpy API and their results
• plotoptix: Not actually for Blender but python package for data viz using CUDA & OPTIX
• SRTM-Terrain: Importer for terrain data

General and useful Blender related:

• bpy: Blender built as a stand-alone python module
• harry blends: for good overview videos of how Geometry Nodes logic works
• CGPython: General scripting and API usage in Blender for artistic rendering
• blenderless easy headless rendering with Blender
• blendify: high level API for creating and renderings scenes with Blender
• bpy-build: a build system for creating Blender addons
• pytest-blender: pytest plugin for addon testing using Blender's bundled python
• nodepad: generating documentation for custom node groups in Blender
• geometry-script: a scripting API for geometry nodes
• geonodes: create geometry node trees with python scripts
• Note to Python: turn node trees into python scripts to create node trees

What I want to try and get out of creating this is to for myself, but also for others collate what people are trying out and how they are approaching different data and visualisation tasks inside of Blender. For a more specific point currently, there are now several different approaches to potentially scripting with Molecular Nodes and Blender, as well as several different approaches to linking Jupyter Notebooks and Blender. I think it better if we tried to coordinate and work on one single approach, rather than replicating work in different projects.

This is something that I have attempted to remedy with databpy, splitting out the data & mesh handling code into it's own package for use with other projects.

The first step in trying to better coordinate these efforts is for everyone to become more aware of what other approaches are being attempted, and I'm hoping that here we can try and discuss what might be the best thing to do going forward. I'm not looking to try and police anybodies projects or there approaches, but if people are open to working collectively then it'll likely work out better than working separately.

I'm tagging some users to ping and potentially get some discussion going, but if you aren't interested then please mute the thread & I can remove you from the post. Please feel free to ping others to get them to join the conversation also.

Pinging: @yuxuanzhuang @PardhavMaradani @kolibril13 @jojoelfe @corredD @so-rose @zachcp @paulmelis @

[kolibril13]

Great idea! I would also add video tutorial resources, e.g.
harry blends- https://www.youtube.com/@harryblends/videos
cgfigures - https://www.youtube.com/c/cgfigures
CGPython - https://www.youtube.com/c/CGPython

and here's bpy gallery with scripts and their corresponding visual output:
https://kolibril13.github.io/bpy-gallery/n1objects/

edit: we could even bring this list to a website similar to https://matplotlib.org/thirdpartypackages/#animations

[zachcp]

Brady, thank you for this post and for all of your great work. And thank you for connecting this talented group; I admire the work of the other coders here.

I will respond in more depth but at a minumum I would connect to this discussion I had started for prettymol. In my case I am intending to use Blender/MN as the render platform but otherwise leave selection control to the user letting them get most of Blender's benefit without having to learn more that small additional API.

[zachcp]

Brady,

I've tried a few different approaches to scripting in Blender on top of Molecular Nodes and have formed some opinions on how to get the 80% benefit with 20% effort. Here are a few thoughts.

Blender Creates Opportunity

• fantastic rendering capability.
• infinite customization.
• extensive plugins and plugin system/
• beautiful existing nodes including MN
• python API for scripting.
• beautiful Node-based system allows flourishing creativity and fast iteration from power users.

Blender Presents Challenges

• WHL file distribution
• Install a full python environment that is seperate from system/conda/venc/etc.
• pinned python // numpy // boost etc libraries conmplicates compatability.
• node systems are in-theory programmable but in practice very difficult beyond the basics.
• GUI requires learning Blender which is steep; rewarding but steep.

Molecular Nodes Opportunity/challenge

Molecular Nodes masterfully embraces Blender's power but comes with all of its challenges. From a scripting perspective there are the installation/environment challenges, as well as the ergonomic challenge of programming against 1) the generic trickiness of node programming and 2) that the nodes themselves are not normal python objects - e.g. they are accessible as Blender components loaded from a .blend file.

I've tried a few different approaches to solving the problem and have come to believe the ideal scripting setup for Molecular Nodes is as follows:

• Commonly used, awesome visualizations should be able to be distributed as scripts (important for adoption; reproducibility; extension )
• A good IPYTHON/repl experience that dramatically lowers the need to learn Blender.
• We should not aim to completely recreate all aspects of the the GUI. Embrace the GUI for interactive dev, but provide a way to capture and reuse important bits (view coordinates, materials, etc) as pure code/data.
• Node-based programming in Blender via BPY is really, really tricky - there is way too much state/context/callbacks etc. So don't go that route.
• if we capture all of our relevant state as code and then use blender only as a VIZ/render tool is allows us to do all of our login in the familiar world of python.

ZCP's Approach.

My brief MN scripting trajectory is as follows:

• attempt to recreate the figure from the Evolutionary Scale front page using a script. struggle for a while and give up
• try to recreate all of Molecular Nodes as python classes with molnodes-utils and wobble but find that really unsatisfactory as an approach.
• stumble accross basilisp and basilisp-blender which let me have a very nice interactive experience with the Blender UI. Write basimol as a thing MN wrapper on top. Main benefirt of basimol is data-based-representaiton of what I want to render. the entire API inclduing styles is a few hundred lines.
• Realize through that effort that there are Ipython-Blender bridges that provide similar interactive experience with Pyhton as a backend. That can provide similar basimol-like REPL experience but not introduce a new language. Thats how prettymol was born.

How I hope prettymol can fit the bill of the points above:

• Distribution. Currently a work in progress but this runs.
  uv run --python 3.11 --with prettymol prettymol --code 1FAP --output prettymol.png
• REPL Experience.
  • to start IPYTHON in blender: pixi run jupyer
  • to get started in the repl from prettymol import repltools; rt=repltool(); See more here
• Params as data. See example here and code. Styles, Materials, and Lights are immutable data classes.
• Nodes: there are only a few functions relates to nodes. Everything else is data. Node creation is the last step before vizualiation. No attempts at conneccting nodes.
• Python centric. This GFP growing example is made with this code.

General Thoughts.

• Further embrace biotite. Its a great foundation and their new interface model opens the opportunity to define a 'to_blender' method.
• Define a schema for the Blender data objects and write a diagram that provides a mental model of what a Molecular Node is from the perspective of the programmer. E.g. what is the minimal info and calls that converts Data->to->mesh
• Consider integrating a standard path for IPYthon usage directly in the MN docs (I guess thats what this discussion is for!)
  • I ended up settling on the following:
    • deleting the Blender python ENV and
    • using my local PIXI venv as the blender env.
  • I don't know if thats the best approach for all users though.
• Continue to Embrace the beauty of nodes but consider that collections of nodes defined in Blender are really fantastic ways to extend python visualization more broadly. For example, can we come up with ways for defining and representing:
  • intermoleculer bonds
  • backbone "frames" .e.g. planar representation of peptides
  • expose style nodes specific to Nucleic Acid.
• Also continue to embrace the power of scripting:
  • have a molview spec compataiblity layer that allows MOL*->blender
  • have a pymol-> blender (maybe bia biotite)
  • provide a way to define style FUNCTIONs

This got a bit longer than I expected so I'll close with a thank you to Brady and an offer to help incorporate any of the ideas above as we see fit.

[jojoelfe]

Hi Brady,

thanks, as always, for putting so much thought into things.

I agree that being able to write scripts that produce visualizations is a great and important workflow. But being able to use the Blender GUI itself is nice, too. There are two workflows that I often use:

Create visualization using GUI and then convert to script.

• Come up with a visualization for a protein/map using the GUI, which allows for fast iteration and deciding for camera placement etc.
• Save that as a .blend file.
• Create a .py script that imports the bpy module and then opens the .blend file
• Swap out the underlying data via scripting and render in the script.

  Potential improvements:

  • It would be nice to be able to convert a .blend file into a .py script and then edit that.

Use the GUI to edit the underlying data

• Open up a dataset, such as a .star ensemble
• Modify the data, such as selecting a subset of the instances.
  • I most often do this by removing the Geometry Nodes modifier, so I have the original data as a mesh.
  • Then I create a visualization object that uses the Object Info node to get that mesh from the DataObject and create instances using that mesh
  • Now I can use 'Edit mode' to edit the DataObject. The visualization will update based on that.
  • Then one needs some custom python to save this mesh back into an appropriate format such as .star
• Since the bpy object does not come with the gui, any script that wants to incoporate this will have to start blender.exe through the subprocess module. The python code within blender then needs to be in a separate python file that gets called from within blender.

  Potential improvements

  • The most pain comes from the fact that the bpy module does not include the GUI, so all python code is either in the "project python" or the "blender python". That gets confusing fast.
  • It would be great to package this up into Blender Apps, but they are not official yet?

Sorry this is all a bit of a random mix of thoughts. Happy to follow up with more concrete examples.

[BradyAJohnston]

Thanks for the comments @zachcp and@jojoelfe!

I can see some main areas that need improvement, separate to MN, to better enable both MN and in general more incorporation of Blender into workflows:

An improved scripting API for nodes

This is a major pain point of the API. It's overly verbose and particular, with the user having to keep a mental model in their head of how the node tree is going to look at the end of it. There are a couple of projects that try to address this (geometry-script, geonodes) but I have found both of them to not quite work how I would like them to.

One possible solution is for another project to attempt to address this, creating our own API which better fits our use cases. Would require a lot of work though.

Scripting vs GUI

Blender itself is built first and foremost as a GUI tool for artists. While the python API and our ability to write and run scripts is still high up there on their priorities, it is still secondary to it being a GUI program, which is why we run into so many problems trying to approach this. The same applies for Molecular Nodes, with the primary design and tools being around GUI usage (mostly being Geometry Nodes based), which presents some quirks around scripting, but ultimately I would like to work them out as much as possible. Ideally everything would be able to be cleanly done through code, and the GUI is just point-and-click access to that.

Related to this, when I was first developing MN I had to discard all data and python objects after initial import, which is now no longer the case with the MNSession that can keep arbitrary python objects between operators. There is no potential for better embracing of using the abilities of biotite and MDAnalysis for more interactive workflows, both through the API and the GUI.

---

I'm interested to @jojoelfe to hear more about what kind of data editing you are doing. There has been a bunch of experimentation with the GPU module for drawing custom UIs the viewport, potentially making more of a custom GUI experience similar to a dedicated molecular graphics program. Blender Apps would in theory also solve that issue, but that project keeps getting pushed back because they don't have the development team to support it.

[zachcp]

Hi Team,

• I had previously explored converting MN nodes to python using geometry script but found the output to be a bit rough - e.g. 2500 lines for a node (https://github.com/zachcp/molnodes-utils/blob/main/molnodes/molnodes/helix_detect.py).
• I hadn't heard of geonodes and its looks very promising as a way to express relatively condensed collections of nodes - e.g. I can imagine using geonodes to make pure-python translations for all of MN's nodes where they are relatively condensed and comprehensible.
• To make it a bit easier to explore I made it a bit easier to install into my blender dev env. here is the pixi.toml for prettymol

[pypi-dependencies]
# note these git coords define a packageable fork of geonodes
geonodes = { git = "https://github.com/zachcp/geonodes", rev = "main" }
biotite = ">=1.1"
blender_notebook = "*"
bpy = "*"
databpy = "*"
molecularnodes = "*"
blendersynth = ">=0.3.0, <0.4"

• the tricky bit that I found about geonodes is that you run the code and it creates a modifer but then you have to apply that modifier to some existing object.

[BradyAJohnston]

Yes I've found both geonodes and geometry-script to be 90% of the way there, but not actually be setup to be used internally inside of other add-ons / packages.

I think we could take large portions / inspiration from each of them and combine into another package that is intended for this purpose. Would require a fair amount of work, but would be one aspect of improving this whole process so that node groups are ultimately saved as code.

[zachcp]

Potential Contributions I'd like to make to MN:

1. Direct Biotite --> Molecule layer. See Direct Biotite to MN method #750
2. Port prettmol selection to MN. e.g Make it easy to add programmable select portions of the Biotite data frame before conversion. (these can parallel similar pure-node selection)
3. Port / assist in repl tools (e.g. setting up the camera angle and defaults for orientation etc.).
4. Figure out an easy way to programmatically apply a color scheme.

@BradyAJohnston LMK if any of these seem reasonable an I can make specific issues to discuss implementation.