Skip to content

Latest commit

 

History

History
executable file
·
129 lines (96 loc) · 7.91 KB

README.md

File metadata and controls

executable file
·
129 lines (96 loc) · 7.91 KB
Raw

PWC

PWC

PWC

PWC

Table of contents
  1. Installation guide
  2. Data Preparation
  3. Run the code
  4. Acknowledgments

Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance CVPR'24

by Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen

News ⚡

Abstract: We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D environments exhibit diverse shapes, scales, and colors, making precise instance-level identification a challenging task. Recent advancements in Open-Vocabulary scene understanding have made significant strides in this area by employing class-agnostic 3D instance proposal networks for object localization and learning queryable features for each 3D mask. While these methods produce high-quality instance proposals, they struggle with identifying small-scale and geometrically ambiguous objects. The key idea of our method is a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals addressing the above limitations. These are then combined with 3D class-agnostic instance proposals to include a wide range of objects in the real world. To validate our approach, we conducted experiments on three prominent datasets, including Scannet200, S3DIS, and Replica, demonstrating significant performance gains in segmenting objects with diverse categories over the state-of-the-art approaches.

overview

Details of the model architecture and experimental results can be found in our paper:

@inproceedings{nguyen2023open3dis,
        title={Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance}, 
        author={Phuc D. A. Nguyen and Tuan Duc Ngo and Evangelos Kalogerakis and Chuang Gan and Anh Tran and Cuong Pham and Khoi Nguyen},
        year={2024},
        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}
}

Please CITE our paper whenever this repository is used to help produce published results or incorporated into other software.

Features 📣

  • State-of-the-art performance of Open-Vocabulary Instance Segmentation on ScanNet200, S3DIS, and Replica.
  • Support Open-Vocabulary queries: affordances, materials, color, shape, etc.
  • Reproducibility code for ScanNet200, Replica, S3DIS, ArkitScenes and Scannet++ datasets!
  • 2D segmenter: Support RAM++, Grounding DINO, SAM, YOLO-World!
  • 3D segmenter: Support ISBNet!
  • Demo application for scene visualization

Installation guide 🔨

Please refer to installation guide

Data Preparation 📂

Please refer to data preparation guide

Run the code 🏃

Please refer to running the code guide

Acknowledgments

This repo is built upon ISBNet, OpenScene, OpenSUN3D.

Contacts

If you have any questions or suggestions about this repo, please feel free to contact me ([email protected]).

Copyright (c) 2024 VinAI

THE DATA IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE DATA OR THE USE OR OTHER DEALINGS IN THE
DATA.