Initial commit

.gitignore

@@ -0,0 +1,12 @@
+*.pyc
+.vscode
+output
+build
+diff_rasterization/diff_rast.egg-info
+diff_rasterization/dist
+tensorboard_3d
+screenshots
+debug
+wandb
+data
+*.txt

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,109 @@
+<p align="center">
+
+  <h1 align="center">Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting</h1>
+  <p align="center">
+    <a href="https://vladimiryugay.github.io/"><strong>Vladimir Yugay</strong></a>
+    ·
+    <a href="https://unique1i.github.io/"><strong>Yue Li</strong></a>
+    ·
+    <a href="https://staff.fnwi.uva.nl/th.gevers/"><strong>Theo Gevers</strong></a>
+    ·
+    <a href="https://people.inf.ethz.ch/moswald/"><strong>Martin Oswald</strong></a>
+  </p>
+  <h3 align="center"><a href="https://vladimiryugay.github.io/gaussian_slam/index.html">Project Page</a></h3>
+  <div align="center"></div>
+</p>
+
+<p align="center">
+  <a href="">
+    <img src="./assets/gaussian_slam.gif" width="90%">
+  </a>
+</p>
+
+## ⚙️ Setting Things Up
+
+Clone the repo:
+
+```
+git clone https://github.com/VladimirYugay/Gaussian-SLAM
+```
+
+Make sure that gcc and g++ paths on your system are exported:
+
+```
+export CC=<gcc path>
+export CXX=<g++ path>
+```
+
+To find the <i>gcc path</i> and <i>g++ path</i> on your machine you can use <i>which gcc</i>.
+
+
+Then setup environment from the provided conda environment file,
+
+```
+conda env create -f environment.yml
+conda activate gslam
+```
+We tested our code on RTX3090 and RTX A6000 GPUs respectively and Ubuntu22 and CentOS7.5.
+
+## 🔨 Running Gaussian-SLAM
+
+Here we elaborate on how to load the necessary data, configure Gaussian-SLAM for your use-case, debug it, and how to reproduce the results mentioned in the paper.
+
+  <details>
+  <summary><b>Downloading the Data</b></summary>
+  We tested our code on Replica, TUM_RGBD, ScanNet, and ScanNet++ datasets. We also provide scripts for downloading Replica nad TUM_RGBD. <br>
+  For downloading ScanNet, follow the procedure described on <a href="http://www.scan-net.org/">here</a>.<br>
+  For downloading ScanNet++, follow the procedure described on <a href="https://kaldir.vc.in.tum.de/scannetpp/">here</a>.<br>
+  The config files are named after the sequences that we used for our method.
+  </details>
+
+  <details>
+  <summary><b>Running the code</b></summary>
+  Start the system with the command:
+
+  ```
+  python run_slam.py configs/<dataset_name>/<config_name> --input_path <path_to_the_scene> --output_path <output_path>
+  ```
+  For example:
+  ```
+  python run_slam.py configs/Replica/room0.yaml --input_path /home/datasets/Replica/room0 --output_path output/Replica/room0
+  ```  
+  You can also configure input and output paths in the config yaml file.
+  </details> 
+
+  <details>
+  <summary><b>Reproducing Results</b></summary>
+  While we made all parts of our code deterministic, differential rasterizer of Gaussian Splatting is not. The metrics can be slightly different from run to run. In the paper we report average metrics that were computed over three seeds: 0, 1, and 2. 
+
+  You can reproduce the results for a single scene by running:
+
+  ```
+  python run_slam.py configs/<dataset_name>/<config_name> --input_path <path_to_the_scene> --output_path <output_path>
+  ```
+
+  If you are running on a SLURM cluster, you can reproduce the results for all scenes in a dataset by running the script:
+  ```
+  ./scripts/reproduce_sbatch.sh
+  ``` 
+  Please note the evaluation of ```depth_L1``` metric requires reconstruction of the mesh, which in turns requires headless installation of open3d if you are running on a cluster.
+  </details>
+
+  <details>
+  <summary><b>Demo</b></summary>
+  We used the camera path tool in <a href="https://github.com/yzslab/gaussian-splatting-lightning">gaussian-splatting-lightning</a> repo to help make the fly-through video based on the reconstructed scenes. We thank its author for the great work.
+  </details>
+
+## 📌 Citation
+
+If you find our paper and code useful, please cite us:
+
+```bib
+@misc{yugay2023gaussianslam,
+      title={Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting}, 
+      author={Vladimir Yugay and Yue Li and Theo Gevers and Martin R. Oswald},
+      year={2023},
+      eprint={2312.10070},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}

configs/Replica/office0.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: office0
+  input_path: data/Replica-SLAM/Replica/office0/
+  output_path: output/Replica/office0/

configs/Replica/office1.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: office1
+  input_path: data/Replica-SLAM/Replica/office1/
+  output_path: output/Replica/office1/

configs/Replica/office2.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: office2
+  input_path: data/Replica-SLAM/Replica/office2/
+  output_path: output/Replica/office2/

configs/Replica/office3.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: office3
+  input_path: data/Replica-SLAM/Replica/office3/
+  output_path: output/Replica/office3/

configs/Replica/office4.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: office4
+  input_path: data/Replica-SLAM/Replica/office4/
+  output_path: output/Replica/office4/

configs/Replica/replica.yaml

@@ -0,0 +1,42 @@
+project_name: "Gaussian_SLAM_replica"
+dataset_name: "replica"
+checkpoint_path: null
+use_wandb: False
+frame_limit: -1 # for debugging, set to -1 to disable
+seed: 0
+mapping:
+  new_submap_every: 50
+  map_every: 5
+  iterations: 100
+  new_submap_iterations: 1000
+  new_submap_points_num: 600000 
+  new_submap_gradient_points_num: 50000 
+  new_frame_sample_size: -1 
+  new_points_radius: 0.0000001
+  current_view_opt_iterations: 0.4 # What portion of iterations to spend on the current view
+  alpha_thre: 0.6
+  pruning_thre: 0.1 
+  submap_using_motion_heuristic: True
+tracking:
+  gt_camera: False
+  w_color_loss: 0.95
+  iterations: 60
+  cam_rot_lr: 0.0002
+  cam_trans_lr: 0.002
+  odometry_type: "odometer"  # gt, const_speed, odometer
+  help_camera_initialization: False # temp option to help const_init
+  init_err_ratio: 5
+  odometer_method: "point_to_plane" # hybrid or point_to_plane 
+  filter_alpha: False
+  filter_outlier_depth: True
+  alpha_thre: 0.98
+  soft_alpha: True
+  mask_invalid_depth: False
+cam:
+  H: 680
+  W: 1200
+  fx: 600.0
+  fy: 600.0
+  cx: 599.5
+  cy: 339.5
+  depth_scale: 6553.5

configs/Replica/room0.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: room0
+  input_path: data/Replica-SLAM/room0/
+  output_path: output/Replica/room0/

configs/Replica/room1.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: room1
+  input_path: data/Replica-SLAM/Replica/room1/
+  output_path: output/Replica/room1/

configs/Replica/room2.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/Replica/replica.yaml
+data:
+  scene_name: room2
+  input_path: data/Replica-SLAM/Replica/room2/
+  output_path: output/Replica/room2/

configs/ScanNet/scannet.yaml

@@ -0,0 +1,43 @@
+project_name: "Gaussian_SLAM_scannet"
+dataset_name: "scan_net"
+checkpoint_path: null
+use_wandb: False
+frame_limit: -1 # for debugging, set to -1 to disable
+seed: 0
+mapping:
+  new_submap_every: 50
+  map_every: 1
+  iterations: 100
+  new_submap_iterations: 100
+  new_submap_points_num: 100000 
+  new_submap_gradient_points_num: 50000 
+  new_frame_sample_size: 30000 
+  new_points_radius: 0.0001
+  current_view_opt_iterations: 0.4 # What portion of iterations to spend on the current view
+  alpha_thre: 0.6
+  pruning_thre: 0.5
+  submap_using_motion_heuristic: False
+tracking:
+  gt_camera: False
+  w_color_loss: 0.6
+  iterations: 200
+  cam_rot_lr: 0.002
+  cam_trans_lr: 0.01
+  odometry_type: "const_speed"  # gt, const_speed, odometer
+  help_camera_initialization: False # temp option to help const_init
+  init_err_ratio: 5
+  odometer_method: "hybrid" # hybrid or point_to_plane
+  filter_alpha: True
+  filter_outlier_depth: True
+  alpha_thre: 0.98
+  soft_alpha: True
+  mask_invalid_depth: True
+cam: 
+  H: 480 
+  W: 640
+  fx: 577.590698
+  fy: 578.729797
+  cx: 318.905426
+  cy: 242.683609
+  depth_scale: 1000.
+  crop_edge: 12

configs/ScanNet/scene0000_00.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0000_00
+  output_path: output/ScanNet/scene0000
+  scene_name: scene0000_00

configs/ScanNet/scene0059_00.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0059_00
+  output_path: output/ScanNet/scene0059
+  scene_name: scene0059_00

configs/ScanNet/scene0106_00.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0106_00
+  output_path: output/ScanNet/scene0106
+  scene_name: scene0106_00

configs/ScanNet/scene0169_00.yaml

@@ -0,0 +1,10 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0169_00
+  output_path: output/ScanNet/scene0169
+  scene_name: scene0169_00
+cam: 
+  fx: 574.540771
+  fy: 577.583740
+  cx: 322.522827
+  cy: 238.558853

configs/ScanNet/scene0181_00.yaml

@@ -0,0 +1,10 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0181_00
+  output_path: output/ScanNet/scene0181
+  scene_name: scene0181_00
+cam: 
+  fx: 575.547668
+  fy: 577.459778
+  cx: 323.171967
+  cy: 236.417465

configs/ScanNet/scene0207_00.yaml

@@ -0,0 +1,5 @@
+inherit_from: configs/ScanNet/scannet.yaml
+data:
+  input_path: data/scannet/scans/scene0207_00
+  output_path: output/ScanNet/scene0207
+  scene_name: scene0207_00

configs/TUM_RGBD/rgbd_dataset_freiburg1_desk.yaml

@@ -0,0 +1,14 @@
+inherit_from: configs/TUM_RGBD/tum_rgbd.yaml
+data:
+  input_path: data/TUM_RGBD-SLAM/rgbd_dataset_freiburg1_desk
+  output_path: output/TUM_RGBD/rgbd_dataset_freiburg1_desk/
+  scene_name: rgbd_dataset_freiburg1_desk
+cam:  #intrinsic is different per scene in TUM
+  H: 480 
+  W: 640 
+  fx: 517.3
+  fy: 516.5
+  cx: 318.6
+  cy: 255.3
+  crop_edge: 50
+  distortion: [0.2624, -0.9531, -0.0054, 0.0026, 1.1633]

configs/TUM_RGBD/rgbd_dataset_freiburg2_xyz.yaml

@@ -0,0 +1,14 @@
+inherit_from: configs/TUM_RGBD/tum_rgbd.yaml
+data:
+  input_path: data/TUM_RGBD-SLAM/rgbd_dataset_freiburg2_xyz
+  output_path: output/TUM_RGBD/rgbd_dataset_freiburg2_xyz/
+  scene_name: rgbd_dataset_freiburg2_xyz
+cam:  #intrinsic is different per scene in TUM
+  H: 480 
+  W: 640 
+  fx: 520.9
+  fy: 521.0
+  cx: 325.1
+  cy: 249.7
+  crop_edge: 8
+  distortion: [0.2312, -0.7849, -0.0033, -0.0001, 0.9172]

configs/TUM_RGBD/rgbd_dataset_freiburg3_long_office_household.yaml

@@ -0,0 +1,14 @@
+inherit_from: configs/TUM_RGBD/tum_rgbd.yaml
+data:
+  input_path: data/TUM_RGBD-SLAM/rgbd_dataset_freiburg3_long_office_household/
+  output_path: output/TUM_RGBD/rgbd_dataset_freiburg3_long_office_household/
+  scene_name: rgbd_dataset_freiburg3_long_office_household
+cam:  #intrinsic is different per scene in TUM
+  H: 480 
+  W: 640 
+  fx: 517.3
+  fy: 516.5
+  cx: 318.6
+  cy: 255.3
+  crop_edge: 50
+  distortion: [0.2624, -0.9531, -0.0054, 0.0026, 1.1633]