[WIP] Add architecture docs

[mfisher87]

Description

Resolves #401

Add some docs that help newcomers dive in to the code and make changes.

@martinRenou @brichet @gjmooney @Meriem-BenIsmail would love your input on this:

• What concepts are important to cover that we've missed?
• Are the things in the doc accurate?
• Any comments or thoughts on the TODOs in the doc?

Checklist

• PR has a descriptive title and content.
• PR description contains references to any issues the PR resolves, e.g. Resolves #XXX.
• PR has one of the labels: documentation, bug, enhancement, feature, maintenance
• Checks are passing.
  Failing lint checks can be resolved with:
  • pre-commit run --all-files
  • jlpm run lint

---

📚 Documentation preview: https://jupytergis--576.org.readthedocs.build/en/576/
💡 JupyterLite preview: https://jupytergis--576.org.readthedocs.build/en/576/lite

[github-actions]

👈 Launch a Binder on branch geojupyter/jupytergis/architecture-docs

[github-actions]

Integration tests report: appsharing.space

[mfisher87]

This documentation feels like it should be auto-generated from metadata files in the source code directories. For example, from front-matter in READMEs. @batpad @arjxn-py do you know of anything that can do that?

[brichet]

👍 to have the documentation in the READMEs.
myst_parser should be able to handle.

[mfisher87]

I was hoping for a way to generate a directory structure like the one @batpad and @arjxn-py and I wrote above from the actual directory structure. I don't see a way to do that from the myst_parser docs or sphinx docs, do you know what I might be missing?

[brichet]

Sorry, I misunderstood the question.
Maybe we could use an API doc generator, like typedoc. I don't know how it can be customized.

[mfisher87]

Looks like they ruled that out: TypeStrong/typedoc#1659

Maybe I can write an autodoc extension that does this? 🤔

[mfisher87]

Marking as ready for review knowing that it's incomplete. We'd just like some early feedback! Please see PR description for the type of feedback we're looking for.

[mfisher87]

TODO:

• Add @batpad and @arjxn-py as co-authors to commits

[mfisher87]

@martinRenou would love your help on this!

[mfisher87]

@martinRenou @batpad Would love your thoughts here! What JupyterLab architectural concepts are really important for new contributors to understand?

[kpdavi]

Martin, Sanjay, and Kristin reviewed the Architecture documentation with the goal of keeping important information for users but also keeping descriptions broad because code is changing quickly and likely will continue to do so.

Suggested change

	# Architecture overview
	JupyterGIS is a JupyterLab extension (based on the structure defined by
	[jupyterlab/extensions-cookiecutter-ts](https://github.com/jupyterlab/extension-cookiecutter-ts).
	Its architecture is based on QuantStack's
	[JupyterCAD](https://github.com/jupytercad/JupyterCAD) architecture.
	## JupyterLab
	### Backbone
	...
	## JupyterGIS components
	### Directory structure
	JupyterGIS is a monorepo containing TypeScript and Python packages.
	* `python/`: The Python packages! These may include some TypeScript as well (_TODO: why?
	Link to JupyterLab extension doc?_).
	* `jupytergis`: A metapackage including `jupytergis_core`, `jupytergis_lab`,
	`jupytergis_qgis`, `jupyter-collaboration`, and `jupyterlab`.
	* `jupytergis_core`: Python methods specific to JupyterLab (not JupyterLite).
	* :star: `src/jgisplugin/plugins.ts`: Where all the commands we've defined are
	registered in TypeScript land -- defines behaviors for menu items.
	* `src/jgisplugin/plugins.ts`: Where commands are linked to layer and source context
	menus.
	* `jupytergis_lab`: _TODO: Ask Martin_.
	* :star: `jupytergis_lab/notebook/gis_document.ts`: Entrypoint for Python API.
	* `src/notebookrenderer.ts`: Enables rendering JupyterGIS in a Notebook.
	* `jupytergis_lite`: A metapackage for deployment and testing of JupyterGIS in
	JupyterLite.
	* `jupytergis_qgis`: Enables importing and exporting QGIS project files.
	* `packages/`: The TypeScript packages.
	* `base/`:
	* `src/index.ts`: Entrypoint for Typescript code.
	* :star: `src/constants.ts`: Defines all the JupyterGIS command names and
	corresponding icons.
	* :star: `src/commands.ts`: For each command, defines the labels, the behaviors, and
	in what contexts each command is available.
	* :star: `src/mainview/index.ts`: Entrypoint for the main map view.
	* `src/toolbar/widget.tsx`: Where commands are linked to the toolbar.
	* `src/formbuilder/`: Base React components for forms.
	* :star: `schema/`: The source of truth for data structures in JupyterGIS. Python
	classes and Typescript types are generated from this.
	### Schema
	The fundamental source of truth for many other components of JupyterGIS.
	* Forms: Generated from e.g. `schema/src/schema/project/layers/vectorlayer.json`
	* Project file / shared model: `schema/src/schema/project/jgis.json`
	### Model
	Structure defined in schema `packages/schema/src/schema/project/jgis.json`.
	#### Shared model
	All collaborators share this and listen for changes to this. It mediates changes with
	Conflict-free Replicated Data Types (CRDTs), which is handled by `yjs`. It is the "magic
	sauce" that enables collaboration!
	:::tip
	You can view the shared model in many contexts by writing
	`console.log(model.sharedModel)` in a JupyterGIS TypeScript file!
	:::
	### Commands
	Many new features are a matter of defining a new command.
	### Forms
	Many forms are generated from `BaseForm`, but some forms use other classes which extend
	`BaseForm`. Each of these classes accepts the relevant schema a property in order to
	generate the form on-the-fly. The correct form class is selected in `formselector.ts`.
	_TODO: How much of this is specific to JGIS and how much is generic and could be
	extracted to a third-party package?_
	### Signals
	Signals are a [Lumino](https://lumino.readthedocs.io/en/latest/api/index.html) framework thing.
	_TODO: We need some help defining this section. Is this helping with listening to model
	changes? Do they carry CRDT payloads? Or simpler payload, e.g. `onLayerChange` signal
	carrying the layer ID of a changed layer?_
	Read more: https://www.youtube.com/channel/UCejhDXmzOrxhsTsQBWe-pww/videos
	### Map view
	Uses [OpenLayers](https://openlayers.org/doc/) as a rendering engine.
	`packages/base/src/mainview/index.ts`
	#### Swappable rendering engine?
	The Venn Diagram of the JavaScript map rendering engine ecosystem unfortunately looks
	like a bunch of disparate circles with few overlaps. The burden of understanding this is
	very high, and we hope to avoid shifting this burden on to our users.
	For example, OpenLayers has excellent support for alternative map projections and
	excellent low-level API, but lacks support for visualizing huge vector datasets with the
	GPU. DeckGL can quickly render huge datasets, but lacks projection support.
	# Architecture overview
	JupyterGIS is a JupyterLab extension (based on the structure defined by [jupyterlab/extensions-cookiecutter-ts](https://github.com/jupyterlab/extension-cookiecutter-ts).
	Its architecture is based on QuantStack's [JupyterCAD](https://github.com/jupytercad/JupyterCAD) architecture.
	### About Lumino and JupyterLab
	JupyterGIS is a JupyterLab extension. It may be useful to read more about the [extensions developper documentation](https://jupyterlab.readthedocs.io/en/latest/extension/extension_dev.html).
	The [Lumino](https://lumino.readthedocs.io/en/latest/api/index.html) library is a framework used to control the UI - i.e., tracks what changes in the UI and how it should react to that change.
	## JupyterGIS components
	### TypeScript Packages
	* `packages/`: The TypeScript packages.
	* :star: `schema/`: Defines our `.jgis` file format - as JSON schemas. The source of truth for data structures in JupyterGIS. Python classes and Typescript types are automatically generated from this, and not commited to the repository (using `json2ts` for TypeScript, `datamodel-codegen` for Python).
	If you want to add a new layer type, you'd add it to schema.
	If you change the schema, you have to rebuild JupyterGIS using `jlpm run build`.
	* :star: `base/`: Base contains everything that controls the map using Open-Layers, panels, buttons. Is a UI library, collection of tools - does not do anything by itself. We use it to make the Lab extension.
	Includes gallery of raster layers. If you want to add a new raster layer, this is where you need to add it. Map styling is in base package.
	Any new command / UI element gets added to the base package.
	It contains everything for creating layers and loading data.
	### Python Packages
	* `python/`: The Python packages! These may include some TypeScript as well.
	* :star: `jupytergis`: A metapackage including `jupytergis_core`, `jupytergis_lab`, `jupytergis_qgis`, `jupyter-collaboration`, and `jupyterlab`.
	* :star: `jupytergis_lite`: A metapackage for deployment and testing of JupyterGIS in JupyterLite including `jupytergis_core`, `jupytergis_lab`.
	* `jupytergis_core`: Gets the UI to do things - e.g., load / create JupyterGIS files, work with them. I.e., is the code for the GUI UI. Also has server endpoint for saving the created `.jgis` files to disk (not used in JupyterLite).
	* `jupytergis_lab`: Contains everything needed for JupyterGIS to work within a notebook - **the Python API**, the notebook renderer (the part that displays the JupyterGIS session in the notebook). **Might be worth considering renaming this folder? Current name doesn't reflect what it does**.
	* `jupytergis_qgis`: Enables importing and exporting QGIS project files. Requires a server component.
	### Schema
	The fundamental source of truth for many other components of JupyterGIS.
	* Forms: Generated from e.g. `schema/src/schema/project/layers/vectorlayer.json`
	* Project file / shared model: `schema/src/schema/project/jgis.json`
	### Model
	Structure defined in schema `packages/schema/src/schema/project/jgis.json`.
	#### Shared model
	All collaborators share this and listen for changes to this. It mediates changes with
	Conflict-free Replicated Data Types (CRDTs), which is handled by `yjs`. It is the "magic
	sauce" that enables collaboration!
	:::tip
	You can view the shared model in many contexts by writing
	`console.log(model.sharedModel)` in a JupyterGIS TypeScript file!
	:::
	### Commands
	Many new features are a matter of defining a new command.
	### Forms
	JupyterGIS uses automatically generated forms for creating/editing layers.
	Those forms are generated from the schema definitions of layers, meaning that adding a new entry in the schema will automatically create user-facing UI components when editing layers.
	An example of this was [adding a new "interpolate" parameter for raster sources](https://github.com/geojupyter/jupytergis/pull/522/files), the only required changes were to add the new schema entry, and react on the "interpolate" value in the Open-Layers viewer
	Many forms are generated from `BaseForm` (the default form implementation), but some forms use other classes which extend `BaseForm` in order to provide more advanced controls for editing layers. Each of these classes accepts the relevant schema a property in order to generate the form on-the-fly. The correct form class is selected in `formselector.ts`.
	### Map view
	Uses [OpenLayers](https://openlayers.org/doc/) as a rendering engine.
	`packages/base/src/mainview/index.ts`

[mfisher87]

keeping important information for users but also keeping descriptions broad because code is changing quickly and likely will continue to do so.

It's a big diff to read, and I may not have time to fully engage with this for a few more days, but my thinking on this is that we're at a very early stage, and our most valuable resource is contributors. Maybe it's worth the effort of being more specific in this doc, and paying the price of updating it more frequently, in exchange for lower barriers to entry for new contributors?

I'd love to hear thoughts from @rjpolackwich @YaoTingYao @tawoodard @batpad .

[brichet]

FYI there is also an AI generated documentation at https://deepwiki.com/geojupyter/jupytergis.
Perhaps it can give some indication, but I haven't read it in detail.
There are also some nice graphs, but I don't know if it can be legally reused in our documentation.

[mfisher87]

How do you feel about the accuracy of that documentation? I think I see some flaws in the diagrams.