Skip to content

Latest commit

 

History

History
executable file
·
130 lines (86 loc) · 5.2 KB

README.md

File metadata and controls

executable file
·
130 lines (86 loc) · 5.2 KB
Raw

📝NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS '22)

This is the official PyTorch Implementation of "NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS '22)" by Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee.

[ arXiv ] [ Website ]

Installation Guide

  1. Download or clone our repository
  2. Set up a python environment using conda (see below)
  3. Prepare datasets (see below)
  4. Run the code (see below)

Python Environment

We use Conda environment. You can get conda by installing Anaconda first.

We share our python environment that contains all required python packages. Please refer to the ./note.yml file

You can import our environment using conda:

conda env create -f note.yml -n note

Reference: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file

Prepare Datasets

This code reproduces the results in Table 1:

  • CIFAR10-C
  • CIFAR100-C
  • ImageNet-C

To run our codes, you first need to download at least one of the datasets. Run the following commands:

$cd .                           # project root
$. download_cifar10c.sh        # download CIFAR10/CIFAR10-C datasets
$. download_cifar100c.sh       # download CIFAR100/CIFAR100-C datasets
$cat download_imagenetc.md       # instruction for ImageNet/ImageNet-C datasets (please refer to the instruction)

Run

Prepare Source model

"Source model" refers to a model that is trained with the source (clean) data only. Source models are required to all methods to perform test-time adaptation.

We provide pre-trained models:

After extracting log.zip, put this folder to the project root directory, e.g., NOTE/log. For CIFAR10-C and CIFAR100-C, pre-trained models consist of without-IABN layers and with-IABN layers under three random seeds (0,1,2). For ImageNet-C, the pre-trained model is with-IABN layers under one random seed (0) because we use the torchvision-provided ResNet model for the without-IABN case.

Alternatively, you can train source models via:

$. train_src.sh                 #generate source models for CIFAR10 as default.

You can specify which dataset to use in the script file.

Run Test-Time Adaptation (TTA)

Given source models are available, you can run TTA via:

$. tta.sh                       #Run NOTE for CIFAR10 as default.

You can specify which dataset and which method in the script file.

Results on CIFAR-10 CIFAR-100, temporally correlated test stream

CIFAR10-C CIFAR100-C Avg
Source 42.3 ± 1.1 66.6 ± 0.1 54.4
BN Stats 73.4 ± 1.3 65.0 ± 0.3 69.2
ONDA 63.6 ± 1.0 49.6 ± 0.3 56.6
PL 75.4 ± 1.8 66.4 ± 0.4 70.9
TENT 76.4 ± 2.7 66.9 ± 0.6 71.7
LAME 36.2 ± 1.3 63.3 ± 0.3 49.7
CoTTA 75.5 ± 0.7 64.2 ± 0.2 69.8
NOTE 21.1 ± 0.6 47.0 ± 0.1 34.0

Results on CIFAR-10 CIFAR-100, i.i.d. test stream

CIFAR10-C CIFAR100-C Avg
Source 42.3 ± 1.1 66.6 ± 0.1 54.4
BN Stats 21.6 ± 0.4 46.6 ± 0.2 34.1
ONDA 21.7 ± 0.4 46.5 ± 0.1 34.1
PL 21.6 ± 0.2 43.1 ± 0.3 32.3
TENT 18.8 ± 0.2 40.3 ± 0.2 29.6
LAME 44.1 ± 0.5 68.8 ± 0.1 56.4
CoTTA 17.8 ± 0.3 44.3 ± 0.2 31.1
NOTE 20.1 ± 0.5 46.4 ± 0.0 33.2
NOTE* 17.6 ± 0.3 41.0 ± 0.2 29.3

Log

Raw logs

In addition to console outputs, the result will be saved as a log file with the following structure: ./log/{DATASET}/{METHOD}/{TGT}/{LOG_PREFIX}_{SEED}_{DIST}/online_eval.json

Obtaining results

In order to print the classification errors(%) on test set, run the following commands:

$python eval_script.py --dataset cifar10 --method note --seed all    #print the result of the specified condition.
$python eval_script.py --dataset all --method all --seed all         #print the entire results.

Tested Environment

We tested our codes under this environment.

  • OS: Ubuntu 20.04.4 LTS
  • GPU: NVIDIA GeForce RTX 3090
  • GPU Driver Version: 470.74
  • CUDA Version: 11.4

Citation

@inproceedings{gong2022note,
    author = {Gong, Taesik and Jeong, Jongheon and Kim, Taewon and Kim, Yewon and Shin, Jinwoo and Lee, Sung-Ju},
    title = {{NOTE}: Robust Continual Test-time Adaptation Against Temporal Correlation},
    booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
    year = {2022}
}