Skip to content

Latest commit

 

History

History
104 lines (77 loc) · 6.6 KB

TRAIN.md

File metadata and controls

104 lines (77 loc) · 6.6 KB

🏋️ Training and Inference

Environment Setup

Follow setup instructions from README.

$ conda activate crossover

Train Instance Baseline

Adjust path parameters in configs/train/train_instance_baseline.yaml and run the following:

$ bash scripts/train/train_instance_baseline.sh

Train Instance Retrieval Pipeline

Adjust path parameters in configs/train/train_instance_crossover.yaml and run the following:

$ bash scripts/train/train_instance_crossover.sh

Train Scene Retrieval Pipeline

Adjust path/configuration parameters in configs/train/train_scene_crossover.yaml. You can also add your customised dataset or choose to train on Scannet & 3RScan & MultiScan or any combination of the same. Run the following:

$ bash scripts/train/train_scene_crossover.sh

The scene retrieval pipeline uses the trained weights from instance retrieval pipeline (for object feature calculation), please ensure to update task:UnifiedTrain:object_enc_ckpt in the config file.

Checkpoint Inventory

We provide all an inventory of all checkpoints on G-Drive here. Detailed descriptions in the table below:

instance_baseline

Description Checkpoint Link
Instance Baseline trained on 3RScan 3RScan
Instance Baseline trained on ScanNet ScanNet
Instance Baseline trained on ScanNet + 3RScan ScanNet+3RScan

instance_crossover

Description Checkpoint Link
Instance CrossOver trained on 3RScan 3RScan
Instance CrossOver trained on ScanNet ScanNet
Instance CrossOver trained on ScanNet + 3RScan ScanNet+3RScan
Instance CrossOver trained on ScanNet + 3RScan + MultiScan ScanNet+3RScan+MultiScan

scene_crossover

Description Checkpoint Link
Unified CrossOver trained on ScanNet + 3RScan ScanNet+3RScan
Unified CrossOver trained on ScanNet + 3RScan + MultiScan ScanNet+3RScan+MultiScan

🛡️ Single Inference

We release script to perform inference (generate scene-level embeddings) on a single scan of 3RScan/Scannet. Detailed usage in the file. Quick instructions below:

$ python single_inference/scene_inference.py

Various configurable parameters:

  • --dataset: dataset name, Scannet/Scan3R/MultiScan
  • --data_dir: data directory (eg: ./datasets/Scannet, assumes similar structure as in preprocess.md).
  • --floorplan_dir: directory consisting of the rasterized floorplans (this can point to the downloaded preprocessed directory), only for Scannet
  • --ckpt: Path to the pre-trained scene crossover model checkpoint (details here), example_path: ./checkpoints/scene_crossover_scannet+scan3r.pth/.
  • --scan_id: the scan id from the dataset you'd like to calculate embeddings for (if not provided, embeddings for all scans are calculated).

The script will output embeddings in the same format as provided here.

📊 Evaluation

Cross-Modal Object Retrieval

Run the following script (refer to the script to run instance baseline/instance crossover) for objects instance + scene retrieval results using the instance-based methods. Detailed usage inside the script.

$ bash scripts/evaluation/eval_instance_retrieval.sh

Running this script for 3RScan dataset will also show point-to-point temporal instance matching results on the RIO category subset.

Cross-Modal Scene Retrieval

Run the following script (for scene crossover). Detailed usage inside the script.

$ bash scripts/evaluation/eval_instance_retrieval.sh