MsIFT : Multi-source Image Fusion Transformer

This is the official implementation of MsIFT (Remote Sensing), a transformer-based image fusion method for classification and segmantation. For more details, please refer to:

MsIFT : Multi-source Image Fusion Transformer[Paper]
Xin Zhang , Hangzhi Jiang, Nuo Xu, Lei Ni, Chunlei Huo , Chunhong Pan

Our code is based on mmsegmentation and mmclassification.

Usage

Requirements

For classification:

• Python 3.6+
• PyTorch 1.3+
• mmcv 1.1.4+
• torchvision
• timm
• mmcv-full==1.3.17
• mmcls==0.8.0

Getting Started

Preparation

Clone the code

git clone [email protected]:XinZhangNLPR/MsIFT.git

Download the model weight used in the paper:

VAIS dataset

	Accuracy	download
MsIFT	92.3	Google

Put the model to classification/work_dirs/VAIS/

DFC2013 dataset

	Accuracy	download
MsIFT	93.02	Google

Put the model to classification/work_dirs/DFC2013/

SpaceNet6 dataset

	Seg Method	mIoU	Accuarcy	download
MsIFT	PSPNet	67.51	70.49	Google
MsIFT	DANet	67.94	70.82	Google

Put the PSPNet model to segmentation/work_dir/PSPNet/

Put the DANet model to segmentation/work_dir/DANet/

Evaluate

1.Multi-GPUs Test

VAIS:

cd MsIFT/classification
./tools/dist_test.sh work_dirs/VAIS/SOTA_VAIS.py work_dirs/VAIS/VAIS.pth 8 --eval accuracy

DFC2013:

cd MsIFT/classification
./tools/dist_test.sh work_dirs/DFC2013/finetune_9285.py work_dirs/DFC2013/DFC2013.pth 8 --eval accuracy

SpaceNet6:

cd MsIFT/segmentation
./tools/dist_test.sh work_dir/PSPNet/pspnet_r50-d8.py work_dir/PSPNet/PSPNet.pth 8 --eval mIoU

2.Single-GPU Test

VAIS:

cd MsIFT/classification
python tools/test.py work_dirs/VAIS/SOTA_VAIS.py work_dirs/VAIS/VAIS.pth --eval accuracy

DFC2013:

cd MsIFT/classification
python tools/test.py work_dirs/DFC2013/finetune_9285.py work_dirs/DFC2013/DFC2013.pth 8 --eval accuracy 

SpaceNet6:

cd MsIFT/segmentation
python tools/test.py work_dir/PSPNet/pspnet_r50-d8.py work_dir/PSPNet/PSPNet.pth 8 --eval mIoU

Citation

If you find our work relevant to your research, please cite:

@Article{rs14164062,
AUTHOR = {Zhang, Xin and Jiang, Hangzhi and Xu, Nuo and Ni, Lei and Huo, Chunlei and Pan, Chunhong},
TITLE = {MsIFT: Multi-Source Image Fusion Transformer},
JOURNAL = {Remote Sensing},
VOLUME = {14},
YEAR = {2022},
NUMBER = {16},
ARTICLE-NUMBER = {4062},
ISSN = {2072-4292},
DOI = {10.3390/rs14164062}
}