0.4.0dev merge to main (#375)

* pin README links

* initial cli skeleton

* first-pass docs

* first-pass pydantic models

* improve typing and testing of settings.py

* full prototype of `recorder calculate_transfer_function`

* complete cli draft

* buttons work via cli

* connect background corrections

* update LC_DEVICE_NAME

* 20220920 -> 20230418

* document check for "PC software version"

* three -> two

* mention changed name

* remove remaining converter util function

Signed-off-by: Ziwen Liu <[email protected]>

* fix syntax error

* improve software+firmware instructions

* device adapter -> driver

* direct download link

* add regularization settings

* connect recon params to cli

* no kwargs

* invert 2d phase to match

* more background integration

* style cleanup

* WIP draw overlay with a hook
use a dask array as overlay layer data

* eager mode for now

* fix typo

* move large overlay info banner

* reduce info banner size limit

* avoid repeated updates

* only use event information

* bump mm version

* update instructions

* remove acquisition mode suffix from birefringence

* detect orientation layer and set colormap

* log

* increase log level

* remove copy and test that input is not modified

* wait until the birefringence layers are spawn

* "changed" -> "moved" hook

* Hot fix for flip + reflection convention

* reuse variable

* match repetitive birefringence layers

* pass overlay name

* use names with suffices

* set colormap for multiple orientation layers

* fix key

* remove unused function

* sort once

* pin upgraded pm

* update to version compatible with multithreading

* update waveorder dependency

* `illumination_wavelength` -> `wavelength_illumination`

* added tests for compute-transfer-function and separated function logic

* added documentation for compute-transfer-function

* test

* `anisotropic_thin` -> `oriented_thick`

* changed function names and removed returns

* rename output test and got output-path option working

* added a possible template for testing different configurations

* typo

* removed test file

* matched function name and fixed typo

* remove unused reconstruction settings class

* added unit tests, fixed typos

* added tests, fixed docstring, added fixtures

* cleaned up tests

* revised settings with examples

* initial revision of cli

* document and test `model_to_yaml` and `yaml_to_model`

* update compute_transfer_function

* update apply_inverse_transfer_function

* doc strings

* improved names

* `-h` for help

* first pass `reconstruct` cli

* rename to prepare for merge

* remove `or "None"`

* fixed docstring parameters

* cleaned imports and fixtures

* complete reconstruct draft

* add 3d fluorescence compute-tf

* fluorescence apply_inverse_transfer_function

* add `axial_flip` parameter to phase

* abstracted fixture

* first-pass acquisition_workers.py

* fix bugs and tests

* simplify tests

* typo

* test fluorescence and refactor compute_tf tests

* update default settings

* typo

* apply_inverse_transfer_function tests

* pass birefringence tests

* whoops

* clean imports

* fix tests

* don't accept empty config

* `options` -> `option`

* validate fov path

* unused import

* simplify

* remove redundant "scheme" parameter

* fix docs

* basic test of reconstruct

* check input channel names

* index into input dataset

* document automatic configuration file generation

* improved instructions

* better default to match example

* default 4 states

* revamped examples folder

* remove `compute` module

* simplify examples folder

* remove unused imports

* remove old reconstructors

* remove listener vestiges

* bf fixes

* integration fixes for phase recon

* birefringence integration fixes

* background acquisition integration

* tidy background recons

* bkg correction integration fixes

* Enable phase w/o calibration

* Add `cli_structure` documentation

* fix doc....specifiy input.zarr/0/0/0

* default 4-state pol settings

* fix input channels in tests

* black format tests

* 5 channel pol + BF test input

* test on all PRs (#383)

* update `waveorder` dependency

* support additional channels in input dataset

* Update docs to example settings

* convert from np.uint16 to np.int32 then torch.float32

* np.uint16 to mimic MM datasets

* formatting fix

* make stack traces visible when tests fail

* replace string concatenation with list of args

* replace assertion with ValueError

* deprecate unused function

---------

Signed-off-by: Ziwen Liu <[email protected]>
Co-authored-by: Ziwen Liu <[email protected]>
Co-authored-by: Clinton Huynh <[email protected]>
Co-authored-by: Clinton Huynh <[email protected]>
Co-authored-by: clinton-huynh <[email protected]>

docs/microscope-installation-guide.md

@@ -7,6 +7,22 @@ This guide will walk through a complete recOrder installation consisting of:
 
 Before you start you will need a machine with Windows 10, a Meadowlark DS5020 connected to a liquid crystal device, and a microscope system compatible with `Micromanager`. 
 
+## Install Meadowlark DS5020 and liquid crystals
+
+Start by installing the Meadowlark DS5020 and liquid crystals using the software on the USB stick provided by Meadowlark. You will need to install the USB drivers and CellDrive5000.
+
+**Check your installation versions** by opening CellDrive5000 and double clicking the Meadowlark Optics logo. **We have tested `recOrder == 0.4.0` with "PC software version 1.08" and "Controller firmware version 1.04",** and you will need to upgrade if your software and firmware versions are older. 
+
+To upgrade your "PC software version" use these steps:
+
+- From "Add and remove programs", remove CellDrive5000 and "National Instruments Software".
+- From "Device manager", open the "Meadowlark Optics" group, right click `mlousb`, click "Uninstall device", check "Delete the driver software for this device", and click "Uninstall". Uninstall `Meadowlark Optics D5020 LC Driver` following the same steps.
+- Using the USB stick provided by Meadowlark, reinstall the USB drivers and CellDrive5000. 
+- Confirm that "PC software version" == 1.08
+- **Upgrading users:** you will need to reinstall the Meadowlark device to your micromanager configuration file, because the device driver's name has changed to `MeadowlarkLC`. 
+
+To upgrade your DS5020's firmware, use Meadowlark's "Firmware Updater".
+
 ## Install recOrder software
 
 (Optional but recommended) install [anaconda](https://www.anaconda.com/products/distribution) and create a virtual environment  
@@ -27,11 +43,11 @@ should launch napari (may take 15 seconds on a fresh installation) with the recO
 
 ## Install and configure `Micromanager`
 
-Install `Micromanager 2.0` nightly build `20220920` (https://micro-manager.org/Micro-Manager_Nightly_Builds). 
+Download and install [`Micromanager 2.0` nightly build `20230426` (~150 MB link).](https://download.micro-manager.org/nightly/2.0/Windows/MMSetup_64bit_2.0.1_20230426.exe)
 
-**Note:** We have tested recOrder with `20220920`, but most features will work with newer builds. We recommend testing a minimal installation with `20220920` before testing with a different nightly build or additional device drivers. 
+**Note:** We have tested recOrder with `20230426`, but most features will work with newer builds. We recommend testing a minimal installation with `20230426` before testing with a different nightly build or additional device drivers. 
 
-Before launching `Micromanager`, download the Meadowlark device adapters and calibration files from the [release page](https://github.com/mehta-lab/recOrder/releases/) and place these three unzipped files into your `Micromanager` folder (likely `C:\Program Files\Micro-Manager` or similar). 
+Before launching `Micromanager`, download the USB driver dll from the [release page](https://github.com/mehta-lab/recOrder/releases/) and place this pair of unzipped files into your `Micromanager` folder (likely `C:\Program Files\Micro-Manager` or similar). 
 
 Launch `Micromanager`, open `Devices > Hardware Configuration Wizard...`, and add the `MeadowlarkLcOpenSource` device to your configuration. Confirm your installation by opening `Devices > Device Property Browser...` and confirming that `MeadowlarkLCOpenSource` properties appear. 
 

docs/reconstruction-guide.md

@@ -1,13 +1,25 @@
 # Automating reconstructions
 
-`recOrder` is undergoing changes to the way that it handles automated reconstructions.
+`recOrder` uses a configuration-file-based command-line interface (CLI) to perform all reconstructions.
 
-`recOrder==0.2.0` had an offline mode with `config.yml` files that were used to configure reconstructions. These config files could be generated and saved via the GUI, and the reconstructions could be run via GUI or CLI. 
+## How can I use `recOrder`'s CLI to perform reconstructions?
+`recOrder`'s CLI is summarized in the following figure:
+<img src="./images/cli_structure.png" align="center">
 
-Although the offline mode had many valuable and convenient features, we found that it had diverged from the online mode and it was difficult to recreate results between online and offline modes. These design limitations led us to the following plan for our upcoming releases:
+The main command `reconstruct` command is composed of two subcommands: `compute-tf` and `apply-inv-tf`. 
 
-`recOrder==0.3.0` (release candidate in the coming weeks) will remove the offline mode and use a set of scripts instead. We recommend modifying the reconstruction scripts in `recOrder/examples/` to recreate and automate reconstructions. 
+A reconstruction can be performed with a single `reconstruct` call. For example:
+```
+recorder reconstruct data.zarr/0/0/0 -c config.yml -o reconstruction.zarr
+```
+Equivalently, a reconstruction can be performed with a `compute-tf` call followed by an `apply-inv-tf` call. For example:
+```
+recorder compute-tf data.zarr/0/0/0 -c config.yml -o tf.zarr
+recorder apply-inv-tf data.zarr/0/0/0 tf.zarr -c config.yml -o reconstruction.zarr
+```
+Computing the transfer function is typically the most expensive part of the reconstruction, so saving a transfer function then applying it to many datasets can save time. 
 
-Although these scripts are not as user friendly as a GUI+CLI solution, we are preparing for a much cleaner solution in `1.0.0`, and we appreciate your patience as we go through this change. We ask users who are having any difficulty with the scripts to [open an issue](https://github.com/mehta-lab/recOrder/issues/new/choose) or [send us an email](mailto:[email protected],[email protected]).
+## What types of reconstructions are supported?
+See `/recOrder/examples/` for a list of example configuration files. 
 
-`recOrder==1.0.0` will use a single mode to acquire and reconstruct the data. We are currently planning a refactor that will enable an "acquire once, quickly iterate your reconstruction" workflow, an "acquire now, reconstruct later" workflow, and a "live acquisition, live reconstruction" workflow, among others.  
+TODO: Expand this documentation...need docs for each reconstruction type and parameter.