Merge pull request #1 from flyconnectome/dev

Merge tons of changes from dev into master

.github/workflows/publish-to-pypi.yml

@@ -0,0 +1,34 @@
+name: Publish Python 🐍 distributions 📦 to PyPI
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  build-n-publish:
+    name: Build and publish Python 🐍 distributions 📦 to PyPI
+    runs-on: ubuntu-18.04
+    steps:
+    - uses: actions/checkout@master
+    - name: Set up Python 3.7
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.7
+    - name: Install wheel and setuptools
+      run: >-
+        python -m
+        pip install
+        wheel
+        setuptools
+        --user
+        --upgrade
+    - name: Build a binary wheel and a source tarball
+      run: >-
+        python3
+        setup.py
+        sdist
+        bdist_wheel
+    - name: Publish distribution 📦 to PyPI
+      uses: pypa/gh-action-pypi-publish@master
+      with:
+        password: ${{ secrets.pypi_password }}

README.md

@@ -7,26 +7,13 @@ Tools to work with manually generated and auto-segmented data in FAFB.
 FAFBseg is on [readthedocs](https://fafbseg-py.readthedocs.io/en/latest/).
 
 ## Quickstart
-```
-pip3 install git+git://github.com/flyconnectome/fafbseg-py.git
-```
+Latest stable version
 
-To update an existing installation run this:
-
-```
-pip3 install git+git://github.com/flyconnectome/fafbseg-py.git --upgrade
+```bash
+pip3 install fafbseg
 ```
 
-## Requirements
-You will need brainmaps API access. See [brainmappy](https://github.com/schlegelp/brainmappy)
-for details.
-
-On top of that, you need to install below dependencies.
-
-### Dependencies
-Make sure you have the *most_recent* version of the following libraries:
-
-- [pymaid](https://pymaid.readthedocs.io/en/latest/): `pip3 install git+git://github.com/schlegelp/pymaid@master`
-- [brainmappy](https://github.com/schlegelp/brainmappy): `pip3 install git+git://github.com/schlegelp/brainmappy@master`
-- [inquirer](https://magmax.org/python-inquirer/index.html): `pip3 install inquirer`
-- [iPython](https://ipython.org/install.html): `pip3 install ipython`
+Install from Github
+```bash
+pip3 install git+git://github.com/flyconnectome/fafbseg-py.git
+```

docs/api.rst

@@ -3,48 +3,73 @@
 API Documentation
 =================
 
-Mapping
--------
-These functions will let you map between autoseg and manual CATMAID.
+FAFBseg is divided into separate modules dedicated to a single
+data source/type or functionality:
 
+  - ``fafbseg.flywire`` for FlyWire-related functions
+  - ``fafbseg.google`` for Google segmentation-related functions
+  - ``fafbseg.synapses`` for querying Buhmann et al. synapse predictions
+  - ``fafbseg.xform`` for transforming spatial data between FAFB14 and flywire's FAFB14.1
+  - ``fafbseg.move`` for moving/merging data between data sets
+
+See below for a by-module breakdown:
+
+Flywire segmentation
+--------------------
 .. autosummary::
     :toctree: generated/
 
-    ~fafbseg.segments_to_skids
-    ~fafbseg.segments_to_neuron
-    ~fafbseg.neuron_to_segments
-    ~fafbseg.find_autoseg_fragments
-    ~fafbseg.find_fragments
-    ~fafbseg.get_seg_ids
-
-Merge
------
-These functions let you merge autoseg neurons into manual.
+    fafbseg.flywire.locs_to_segments
+    fafbseg.flywire.encode_url
+    fafbseg.flywire.decode_url
+    fafbseg.flywire.fetch_edit_history
+    fafbseg.flywire.locs_to_supervoxels
+    fafbseg.flywire.skid_to_id
+    fafbseg.flywire.update_ids
+    fafbseg.flywire.get_mesh_neuron
+    fafbseg.flywire.skeletonize_neuron
+    fafbseg.flywire.generate_open_ends_url
+    fafbseg.flywire.merge_flywire_neuron
 
+Google segmentation
+-------------------
 .. autosummary::
     :toctree: generated/
 
-    ~fafbseg.merge_neuron
-    ~fafbseg.find_missed_branches
-
-Review
-------
-These functions let you review (autoseg) tracings
+    fafbseg.google.locs_to_segments
+    fafbseg.google.segments_to_neuron
+    fafbseg.google.segments_to_skids
+    fafbseg.google.neuron_to_segments
+    fafbseg.google.find_autoseg_fragments
+    fafbseg.google.find_fragments
+    fafbseg.google.find_missed_branches
+    fafbseg.google.get_mesh
+    fafbseg.google.autoreview_edges
+    fafbseg.google.test_edges
 
+Buhmann synapse predictions
+---------------------------
 .. autosummary::
     :toctree: generated/
 
-    ~fafbseg.autoreview_edges
-    ~fafbseg.test_edges
+    fafbseg.synapses.locs_to_segments
+    fafbseg.synapses.query_synapses
+    fafbseg.synapses.query_connections
+    fafbseg.synapses.get_neuron_synapses
+    fafbseg.synapses.get_neuron_synapses
+    fafbseg.synapses.assign_connectors
+
+Spatial transformation
+----------------------
+.. autosummary::
+    :toctree: generated/
 
-Setup
------
-Functions for setting up segmentation data source.
+    fafbseg.xform.flywire_to_fafb14
+    fafbseg.xform.fafb14_to_flywire
 
+Merging/combining data
+----------------------
 .. autosummary::
     :toctree: generated/
 
-    ~fafbseg.use_brainmaps
-    ~fafbseg.use_google_storage
-    ~fafbseg.use_remote_service
-    ~fafbseg.use_local_data
+    fafbseg.move.merge_into_catmaid

docs/conf.py

@@ -8,13 +8,19 @@
 import sys
 import os
 
-MOCK_MODULES = ['numpy', 'pandas',
-                'pymaid',
+MOCK_MODULES = ['pymaid',
+                'navis',
+                'trimesh',
                 'cloudvolume',
                 'cloudvolume.frontends.precomputed.CloudVolumePrecomputed',
-                'networkx', 'tqdm',
-                'pyoctree', 'pyoctree.pyoctree',
-                'mcubes',
+                'networkx',
+                'ncollpyde',
+                'PyMCubes', 'mcubes',
+                'pyperclip',
+                'requests_futures', 'requests_futures.sessions',
+                'scipy', 'scipy.spatial',
+                'sqlite3',
+                'tqdm', 'tqdm.auto',
                 'inquirer', 'inquirer.themes', 'inquirer.themes.GreenPassion']
 for mod_name in MOCK_MODULES:
     sys.modules[mod_name] = mock.Mock()

docs/examples/autocomplete.rst

@@ -3,32 +3,51 @@
 Autocomplete Neuron
 ===================
 
-This examples will illustrate how to use the ``auto-seg`` to autocomplete
+This examples will illustrate how to use the ``autoseg`` to autocomplete
 a partially reconstructed neuron.
 
 .. note::
 
     For this to work, you need to have CATMAID API write and import access
 
-Please first see and follow the instructions in the
-:doc:`General Setup<general_setup>`.
+First initialize connections to the ``manual`` and ``autoseg`` CATMAID instances
+set up - make sure to replace ``HTTP_USER``, ``HTTP_PW`` and ``API_TOKEN`` with
+your corresponding credentials:
 
-OK now that we're all set, we can start the process:
+.. code-block:: python
+
+  import pymaid
+  manual = pymaid.CatmaidInstance('https://neuropil.janelia.org/tracing/fafb/v14',
+                                  api_token='API_TOKEN',
+                                  http_user='HTTP_USER',
+                                  http_password='HTTP_PW',
+                                  caching=False,
+                                  max_threads=20)
+
+.. code-block:: python
+
+  auto = pymaid.CatmaidInstance('https://spine.janelia.org/catmaid/fafb-v14-seg-li-200412.0',
+                                api_token='API_TOKEN',
+                                http_user='HTTP_USER',
+                                http_password='HTTP_PW',
+                                caching=False,
+                                max_threads=20)
 
-First get the neuron to autocomplete (exchange the ``16`` for the skeleton ID
+Next get the neuron to autocomplete (exchange the ``16`` for the skeleton ID
 of your neuron).
 
 .. code-block:: python
 
   x = pymaid.get_neuron(16, remote_instance=manual)
 
 
-Now get everything that overlaps with this neuron in ``auto-seg``. See
+Now get everything that overlaps with this neuron in ``autoseg``. See
 :func:`fafbseg.find_autoseg_fragments` on how to fine tune this step.
 
 .. code-block:: python
 
-  ol = fafbseg.find_autoseg_fragments(x, autoseg_instance=auto)
+  import fafbseg
+  ol = fafbseg.google.find_autoseg_fragments(x, autoseg_instance=auto)
 
 
 Now visualise the large overlapping fragments alongside the manual tracings:
@@ -50,36 +69,36 @@ and then run this:
 .. code-block:: python
 
   import numpy as np
-  visible = ol[np.isin(ol.skeleton_id, pymaid.get_viewer().visible)]
+  visible = ol[np.isin(ol.skeleton_id, navis.get_viewer().visible)]
 
-Before we can start the merge process, we have to stitch all ``auto-seg``
+Before we can start the merge process, we have to stitch all ``autoseg``
 fragments to form a single virtual neuron for upload:
 
 .. code-block:: python
 
-  y = pymaid.stitch_neurons(visible, method='NONE')
+  y = navis.stitch_neurons(visible, method='NONE')
 
 If you want to have a final look this is how you can co-visualize the manual
-tracings and the to-be-merged ``auto-seg`` fragments:
+tracings and the to-be-merged ``autoseg`` fragments:
 
 .. code-block:: python
 
   x.plot3d(color='w', clear=True)
   y.plot3d(color='r')
 
 Once you are ready start the upload process as described in
-:doc:`Merging<merge_neuron>`. (see also :func:`fafbseg.merge_neuron` for
+:doc:`Merging<merge_neuron>`. (see also :func:`fafbseg.move.merge_neuron` for
 additional parameters):
 
 .. code-block:: python
 
-  resp = fafbseg.merge_neuron(y, target_instance=manual, tag='YOURTAG')
+  resp = fafbseg.move.merge_neuron(y, target_instance=manual, tag='YOURTAG')
 
 
 Gotchas
 -------
 
-When looking for overlapping ``auto-seg`` fragments, you can end up finding the
+When looking for overlapping ``autoseg`` fragments, you can end up finding the
 autoseg version of your original neuron - ``x`` in above example. This happens
 if somebody has merged a Google skeleton into ``x``.
 

docs/examples/find_missing.rst

@@ -3,36 +3,48 @@
 Finding missed branches
 =======================
 
-This examples will illustrate how to use the ``auto-seg`` to find potential
-missed branches.
+This examples will illustrate how to use the Google segmentation to find
+potential missed branches.
 
-.. note::
+We will take a single, manually traced neuron and try finding potentially missed
+branches by comparing it to the same Google skeleton.
 
-    For this to work, you need to have CATMAID API write access.
+First initialize connections to the ``manual`` and ``autoseg`` CATMAID instances
+set up - make sure to replace ``HTTP_USER``, ``HTTP_PW`` and ``API_TOKEN`` with
+your corresponding credentials:
 
-Please first see and follow the instructions in the
-:doc:`General Setup<general_setup>`
+.. code-block:: python
+
+  manual = pymaid.CatmaidInstance('https://neuropil.janelia.org/tracing/fafb/v14',
+                                  api_token='API_TOKEN',
+                                  http_user='HTTP_USER',
+                                  http_password='HTTP_PW',
+                                  caching=False,
+                                  max_threads=20)
 
-OK now that we're all set, we can start the actual merging process:
+.. code-block:: python
 
-In this example, we will take a single manually traced neuron and try
-finding potentially missed branches using the ``auto-seg``.
+  auto = pymaid.CatmaidInstance('https://spine.janelia.org/catmaid/fafb-v14-seg-li-200412.0',
+                                api_token='API_TOKEN',
+                                http_user='HTTP_USER',
+                                http_password='HTTP_PW',
+                                caching=False,
+                                max_threads=20)
 
-First fetch the neuron (exchange the ``16`` for the skeleton ID
-of your neuron):
+Then fetch the neuron (exchange the ``16`` for the skeleton ID of your neuron):
 
 .. code-block:: python
 
   x = pymaid.get_neuron(16, remote_instance=manual)
 
-Now find and tag missed branches (see:func:`fafbseg.find_missed_branches` for
+Now find and tag missed branches (see:func:`fafbseg.google.find_missed_branches` for
 additional parameters):
 
 .. code-block:: python
 
   (summary,
    fragments,
-   branches) = fafbseg.find_missed_branches(x, autoseg_instance=auto)
+   branches) = fafbseg.google.find_missed_branches(x, autoseg_instance=auto)
 
 Show summary of missed branches: