DEEP-IMAGE-MATCHING

SIFT	DISK	IMAGES ORIENTATION	DENSE WITH ROMA

SIFT	SUPERGLUE

Multivew matcher for SfM software. Support both deep-learning based and hand-crafted local features and matchers and export keypoints and matches directly in a COLMAP database or to Agisoft Metashape by importing the reconstruction in Bundler format. It supports both CLI and GUI. Feel free to collaborate!

Check the documentation at Docs.

Please, note that deep-image-matching is under active development and it is still in an experimental stage. If you find any bug, please open an issue.

Key features:

• Multiview
• Large format images
• SOTA deep-learning and hand-crafted features
• Full compatibility with COLMAP
• Support for image rotations
• Compatibility with Agisoft Metashape (only on Linux and MacOS by using pycolmap)
• Support image retrieval with deep-learning local features

Supported Extractors	Supported Matchers
✓ SuperPoint	✓ Lightglue (with Superpoint, Disk, and ALIKED)
✓ DISK	✓ SuperGlue (with Superpoint)
✓ ALIKE	✓ LoFTR
✓ ALIKED	☐ SE2-LoFTR
☐ Superpoint free	✓ Nearest neighbor (with KORNIA Descriptor Matcher)
✓ KeyNet + OriNet + HardNet8	✓ RoMa
✓ ORB (opencv)	☐ GlueStick
✓ SIFT (opencv)
✓ DeDoDe

Supported SfM software
✓ COLMAP
☐ OpenMVG
✓ Agisoft Metashape
✓ Software that supports bundler format

Colab demo

Want to run on a sample dataset? ➡️

Want to run on your images? ➡️

Installation

For installing deep-image-matching, first create a conda environment:

conda create -n deep-image-matching python=3.9
conda activate deep-image-matching
pip install --upgrade pip

Clone the repository and install deep-image-matching in editable mode:

git clone https://github.com/3DOM-FBK/deep-image-matching.git
cd deep-image-matching
pip install -e .

Install pycolmap:

pip install pycolmap

As pycolmap was released on PyPi only for Linux and macOS (up to version 0.4.0), it is not installed by default with deep-image-matching. From version 0.5.0, pycolmap can be installed on Windows too. However, it needs some testing before including in dependencies of deep-image-matching, as there are some errors on Windows that are blocking deep_image_matching pipeline (while it works completely fine on Linux).

For more information, check the documentation.

Usage instructions

You can run deep-image-matching from the command line or from the GUI.

Use the following command to see all the available options from the CLI:

python main.py --help

For example, to run the matching with SuperPoint and LightGlue on a dataset, you can use the following command:

python main.py --dir assets/example_cyprus --pipeline superpoint+lightglue

For all the usage instructions and configurations, refer to the documenation at https://3dom-fbk.github.io/deep-image-matching/ or check the example notebooks.

To run the GUI, you can use the following command:

python main.py --gui

Advanced usage

For advanced usage, you can check the scripts directory.

Merging databases with different local features

To run the matching with different local features and/or matchers and marging together the results, you can use scripts in the ./scripts directory for merging the COLMAP databases.

python ./join_databases.py --help
python ./join_databases.py --input assets/to_be_joined --output docs/assets/to_be_joined

Exporting the solution to Metashape

To export the solution to Metashape, you can export the COLMAP database to Bundler format and then import it into Metashape. This can be done from Metashape GUI, by first importing the images and then use the function Import Cameras (File -> Import -> Import Cameras) to select Bundler file (e.g., bundler.out) and the image list file (e.g., bundler_list.txt).

Alternatevely, you can use the export_to_metashape.py script to automatically create a Metashape project from a reconstruction saved in Bundler format. The script export_to_metashape.py takes as input the solution in Bundler format and the images and it exports the solution to Metashape. It requires to install Metashape as a Python module in your environment and to have a valid license. Please, refer to the instructions at https://github.com/franioli/metashape.

How to contribute

Any contribution to this repo is really welcome! If you want to contribute to the project, please, check the contributing guidelines.

TODO:

• Tile processing for high resolution images
• Manage image rotations
• Add image retrieval with global descriptors
• add GUI
• Add pycolmap compatibility
• Add exporting to Bundler format ready for importing into Metashape (only on Linux and MacOS by using pycolmap)
• Add visualization for extracted features and matches
• Improve speed
• Autoselect tiling grid in order to fit images in GPU memory
• Add tests, documentation and examples (e.g. colab, ..)
• Apply masks during feature extraction
• Integrate support for Pix4D Open Photogrammetry Format
• Work with submodules
• Automatically download weights for all the models
• Cleanup repository to removed large files from Git history
• Update README CLI options

References

If you find the repository useful for your work consider citing the papers:

@article{morelli2022photogrammetry,
  title={PHOTOGRAMMETRY NOW AND THEN--FROM HAND-CRAFTED TO DEEP-LEARNING TIE POINTS--},
  author={Morelli, Luca and Bellavia, Fabio and Menna, Fabio and Remondino, Fabio},
  journal={The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
  volume={48},
  pages={163--170},
  year={2022},
  publisher={Copernicus GmbH}
}

@article{ioli2023replicable,
  title={A Replicable Open-Source Multi-Camera System for Low-Cost 4d Glacier Monitoring},
  author={Ioli, F and Bruno, E and Calzolari, D and Galbiati, M and Mannocchi, A and Manzoni, P and Martini, M and Bianchi, A and Cina, A and De Michele, C and others},
  journal={The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
  volume={48},
  pages={137--144},
  year={2023},
  publisher={Copernicus GmbH}
}

Depending on the options used, consider citing the corresponding work of KORNIA, HLOC, and local features.