diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/Readme.md b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/Readme.md new file mode 100644 index 00000000..2c5d7182 --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/Readme.md @@ -0,0 +1,147 @@ + + +--- + +# Quick Start about Single Task Learning Semantic Segmentation + +Welcome to Ianvs! Ianvs is a benchmarking platform designed to evaluate the performance of distributed synergy AI solutions in accordance with recognized standards. This quick start guide will help you test your **Single Task Learning (STL)** algorithm on Ianvs. By following these streamlined steps, you can efficiently develop and benchmark your solution within minutes. + +### **Prerequisites** +Before using Ianvs, ensure that your system meets the following requirements: +- A single machine (a laptop or a virtual machine is sufficient; no cluster is needed) +- At least 2 CPUs +- 4GB+ of free memory (depending on the algorithm and simulation settings) +- 10GB+ of free disk space +- An internet connection for accessing GitHub, pip, etc. +- Python 3.6+ installed + +This guide assumes you are using **Linux** with Python 3.8. If you’re on Windows, most steps will apply, but some commands and package requirements may differ. + +--- + +## Step 1. Ianvs Installation + +### Clone Ianvs +First, set up a workspace and clone Ianvs: +```shell +mkdir /ianvs +cd /ianvs + +mkdir project +cd project +git clone https://github.com/kubeedge/ianvs.git +``` + +### Install Dependencies +Next, install the required third-party dependencies: +```shell +sudo apt-get update +sudo apt-get install libgl1-mesa-glx -y +python -m pip install --upgrade pip + +cd ianvs +python -m pip install ./examples/resources/third_party/* +python -m pip install -r requirements.txt +``` + +### Install Ianvs +Finally, install Ianvs: +```shell +python setup.py install +``` + +--- + +## Step 2. Dataset Preparation + +### Cloud-Robotics Dataset Summary + +The **Cloud-Robotics Dataset** features **annotated real-world images** with **dense semantic and instance segmentation** across **30 classes** in 7 groups (e.g., vehicles, humans, nature, objects). It includes polygonal annotations, diverse daytime scenes, dynamic objects, and varying layouts. Data is provided in a JSON format, making it ideal for pixel-level semantic labeling and benchmarking vision models for robotics. + +Organize the dataset for STL as shown below: + +```plaintext +Dataset/ +├── 1280x760 +│ ├── gtFine +│ │ ├── train +│ │ ├── test +│ │ └── val +│ ├── rgb +│ │ ├── train +│ │ ├── test +│ │ └── val +│ └── viz +│ ├── train +│ ├── test +│ └── val +├── 2048x1024 +│ ├── gtFine +│ │ ├── train +│ │ ├── test +│ │ └── val +│ ├── rgb +│ │ ├── train +│ │ ├── test +│ │ └── val +│ └── viz +│ ├── train +│ ├── test +│ └── val +├── 640x480 + ├── gtFine + │ ├── train + │ ├── test + │ └── val + ├── json + │ ├── train + │ ├── test + │ └── val + ├── rgb + │ ├── train + │ ├── test + │ └── val + └── viz + ├── train + ├── test + └── val +``` + +### Dataset Preparation Command +```shell +mkdir dataset +cd dataset +unzip dataset.zip +``` + +Update the dataset's **URL address** in the `testenv.yaml` configuration file. More details can be found in the [testenv.yaml guide](https://ianvs.readthedocs.io/en/latest/guides/how-to-test-algorithms.html#step-1-test-environment-preparation). + +--- + +## Step 3. Configure Algorithm + + +Update the algorithm's **URL address** in the `algorithm.yaml` file. Refer to the [algorithm.yaml guide](https://ianvs.readthedocs.io/en/latest/guides/how-to-test-algorithms.html#step-1-test-environment-preparation) for detailed instructions. + +--- + +## Step 4. Ianvs Execution and Results + +Run Ianvs for benchmarking: +```shell +cd /ianvs/project +ianvs -f examples/cloud-robotics/single_task_learning/semantic-segmentation/benchmarkingjob.yaml +``` + +## Example Benchmarking Results + +## Explanation of Results: + +- mIoU: The mean intersection over union (mIoU) measures the overall performance by averaging IoU values across all classes. Higher mIoU indicates better segmentation quality. +- Resolution Impact: Results may vary with image resolution, with higher resolutions typically offering more precise segmentation. + +## Final mIoU Summary: + +**Overall mIoU: 72.35%** + +--- diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/benchmarkingjob.yaml b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/benchmarkingjob.yaml new file mode 100644 index 00000000..2e337470 --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/benchmarkingjob.yaml @@ -0,0 +1,54 @@ +benchmarkingjob: + # job name of bechmarking; string type; + name: "benchmarkingjob" + # the url address of job workspace that will reserve the output of tests; string type; + workspace: "/ianvs/single_task_learning/cloud" + + # the url address of test environment configuration file; string type; + # the file format supports yaml/yml; + testenv: "./examples/cloud_robot/single_task_learning/testenv/testenv.yaml" + + # the configuration of test object + test_object: + # test type; string type; + # currently the option of value is "algorithms",the others will be added in succession. + type: "algorithms" + # test algorithm configuration files; list type; + algorithms: + # algorithm name; string type; + - name: "rfnet_singletask_learning" + # the url address of test algorithm configuration file; string type; + # the file format supports yaml/yml + url: "./examples/cloud-robotics/single_task_learning/testalgorithm/rfnet/algorithm.yaml" + + + # selected dataitem configuration + # The user can add his/her interested dataitems in terms of "paradigms", "modules", "hyperparameters" and "metrics", + # so that the selected columns will be shown. + selected_dataitem: + # currently the options of value are as follows: + # 1> "all": select all paradigms in the leaderboard; + # 2> paradigms in the leaderboard, e.g., "singletasklearning" + paradigms: [ "all" ] + # currently the options of value are as follows: + # 1> "all": select all modules in the leaderboard; + # 2> modules in the leaderboard, e.g., "basemodel" + modules: [ "all" ] + # currently the options of value are as follows: + # 1> "all": select all hyperparameters in the leaderboard; + # 2> hyperparameters in the leaderboard, e.g., "momentum" + hyperparameters: [ "all" ] + # currently the options of value are as follows: + # 1> "all": select all metrics in the leaderboard; + # 2> metrics in the leaderboard, e.g., "F1_SCORE" + metrics: [ "accuracy", "task_avg_acc", "BWT", "FWT"] + + # model of save selected and all dataitems in workspace `./rank` ; string type; + # currently the options of value are as follows: + # 1> "selected_and_all": save selected and all dataitems; + # 2> "selected_only": save selected dataitems; + save_mode: "selected_and_all" + + + + diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/algorithm.yaml b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/algorithm.yaml new file mode 100644 index 00000000..b306de89 --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/algorithm.yaml @@ -0,0 +1,22 @@ +algorithm: + paradigm_type: "singletasklearning" + initial_model_url: "./models/model.pth" + + modules: + - type: "basemodel" + name: "RFNet" + url: "./examples/cloudrobotics/singletask_learning_bench/testalgorithms/rfnet/basemodel.py" + + hyperparameters: + - momentum: + values: + - 0.95 + - 0.5 + - learning_rate: + values: + - 0.1 + - epochs: + values: + - 2 + + \ No newline at end of file diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/basemofel.py b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/basemofel.py new file mode 100644 index 00000000..9de7631f --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testalgorithms/basemofel.py @@ -0,0 +1,174 @@ +# Copyright 2022 The KubeEdge Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import absolute_import, division, print_function + +import os +import zipfile +import logging + +import numpy as np +from PIL import Image +import torch +from torch.utils.data import DataLoader +from torchvision import transforms +from sedna.common.class_factory import ClassType, ClassFactory +from sedna.common.config import Context +from sedna.common.file_ops import FileOps +from RFNet.dataloaders import make_data_loader +from RFNet.dataloaders import custom_transforms as tr +from RFNet.utils.lr_scheduler import LR_Scheduler +from RFNet.train import Trainer +from RFNet.eval import Validator +from RFNet.utils.saver import Saver +from RFNet.eval import load_my_state_dict +import RFNet.train_config as train_cfgs +import RFNet.eval_config as valid_cfgs + +logging.disable(logging.WARNING) + +__all__ = ["BaseModel"] + +os.environ['BACKEND_TYPE'] = 'TORCH' + + +@ClassFactory.register(ClassType.GENERAL, alias="RFNet") +class BaseModel: + + def __init__(self, **kwargs): + self.train_cfgs = train_cfgs + self.valid_cfgs = valid_cfgs + + self.train_cfgs.lr = kwargs.get("learning_rate", 0.0001) + self.train_cfgs.momentum = kwargs.get("momentum", 0.9) + self.train_cfgs.epochs = kwargs.get("epochs", 2) + self.train_cfgs.checkname = '' + self.train_cfgs.batch_size = 4 + self.train_cfgs.depth = True + + os.environ["MODEL_NAME"] = "model_best_mapi_only.pth" + + self.train_cfgs.gpu_ids = ['cuda:0'] + self.valid_cfgs.gpu_ids = ['cuda:0'] + self.valid_cfgs.depth = True + + self.validator = Validator(self.valid_cfgs) + self.trainer = Trainer(self.train_cfgs) + self.trainer.saver = Saver(self.train_cfgs) + self.checkpoint_path = self.load( + Context.get_parameters("base_model_url")) + + def train(self, train_data, valid_data=None, **kwargs): + if train_data is None or train_data.x is None or train_data.y is None: + raise Exception("Train data is None.") + + self.trainer.train_loader, self.trainer.val_loader, _, _ = \ + make_data_loader( + self.trainer.args, train_data=train_data, valid_data=valid_data, **kwargs) + + # Define lr scheduler + self.trainer.scheduler = LR_Scheduler(self.trainer.args.lr_scheduler, self.trainer.args.lr, + self.trainer.args.epochs, len(self.trainer.train_loader)) + print("Total epoches:", self.trainer.args.epochs) + + for epoch in range(self.trainer.args.start_epoch, self.trainer.args.epochs): + if epoch == 0 and self.trainer.val_loader: + self.trainer.validation(epoch) + self.trainer.training(epoch) + + if self.trainer.args.no_val and \ + (epoch % self.trainer.args.eval_interval == (self.trainer.args.eval_interval - 1) + or epoch == self.trainer.args.epochs - 1): + # save checkpoint when it meets eval_interval or the training finished + is_best = False + self.checkpoint_path = self.trainer.saver.save_checkpoint({ + 'epoch': epoch + 1, + 'state_dict': self.trainer.model.state_dict(), + 'optimizer': self.trainer.optimizer.state_dict(), + 'best_pred': self.trainer.best_pred, + }, is_best) + + # if not self.trainer.args.no_val and \ + # epoch % self.train_args.eval_interval == (self.train_args.eval_interval - 1) \ + # and self.trainer.val_loader: + # self.trainer.validation(epoch) + # self.checkpoint_path = os.path.join(self.temp_dir, '{}_'.format( + # cfgs.DATASET_NAME) + str(_global_step) + "_" + str(time.time()) + '_model.ckpt') + # saver.save(sess, self.checkpoint_path) + # print('Weights have been saved to {}.'.format(self.checkpoint_path)) + + self.trainer.writer.close() + + return self.checkpoint_path + + def save(self, model_path): + if not model_path: + raise Exception("model path is None.") + + model_dir, model_name = os.path.split(self.checkpoint_path) + models = [model for model in os.listdir( + model_dir) if model_name in model] + + if os.path.splitext(model_path)[-1] != ".zip": + model_path = os.path.join(model_path, "model.zip") + + if not os.path.isdir(os.path.dirname(model_path)): + os.makedirs(os.path.dirname(model_path)) + + with zipfile.ZipFile(model_path, "w") as f: + for model_file in models: + model_file_path = os.path.join(model_dir, model_file) + f.write(model_file_path, model_file, + compress_type=zipfile.ZIP_DEFLATED) + + return model_path + + def predict(self, data, **kwargs): + if not isinstance(data[0][0], dict): + data = self._preprocess(data) + + if type(data) is np.ndarray: + data = data.tolist() + + self.validator.test_loader = DataLoader( + data, batch_size=self.validator.args.test_batch_size, shuffle=False, pin_memory=True) + return self.validator.validate() + + def load(self, model_url): + model_url = '/home/wxc/dev/ianvs/models/model_best_mapi_only.pth' + if FileOps.exists(model_url): + self.validator.new_state_dict = torch.load( + model_url, map_location=torch.device("cpu")) + else: + raise Exception("model url does not exist.") + self.validator.model = load_my_state_dict( + self.validator.model, self.validator.new_state_dict['state_dict']) + return model_url + + def _preprocess(self, image_urls): + transformed_images = [] + for img_path, depth_path in image_urls: + _img = Image.open(img_path).convert('RGB') + _depth = Image.open(depth_path) + + sample = {'image': _img, 'depth': _depth, 'label': _img} + composed_transforms = transforms.Compose([ + + tr.Normalize(mean=(0.485, 0.456, 0.406), + std=(0.229, 0.224, 0.225)), + tr.ToTensor()]) + + transformed_images.append((composed_transforms(sample), img_path)) + + return transformed_images \ No newline at end of file diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/accuracy.py b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/accuracy.py new file mode 100644 index 00000000..33c883e4 --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/accuracy.py @@ -0,0 +1,68 @@ +# Copyright 2022 The KubeEdge Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from tqdm import tqdm + +from sedna.common.class_factory import ClassType, ClassFactory + +from RFNet.dataloaders import make_data_loader +from RFNet.utils.metrics import Evaluator +from RFNet.utils.args import ValArgs + +__all__ = ('accuracy', 'compute_cpa', 'compute_miou', 'compute_fwiou') + +logger = logging.getLogger(__name__) + +@ClassFactory.register(ClassType.GENERAL, alias="accuracy") +def accuracy(y_true, y_pred, **kwargs): + args = ValArgs() + _, _, test_loader, num_class = make_data_loader(args, test_data=y_true) + evaluator = Evaluator(num_class) + tbar = tqdm(test_loader, desc='\r') + for i, (sample, img_path) in enumerate(tbar): + if args.depth: + image, depth, target = sample['image'], sample['depth'], sample['label'] + else: + image, target = sample['image'], sample['label'] + if args.cuda: + image, target = image.cuda(args.gpu_ids), target.cuda(args.gpu_ids) + if args.depth: + depth = depth.cuda(args.gpu_ids) + + target[target > evaluator.num_class-1] = 255 + target = target.cpu().numpy() + evaluator.add_batch(target, y_pred[i]) + + CPA = compute_cpa(evaluator) + mIoU = compute_miou(evaluator) + FWIoU = compute_fwiou(evaluator) + + logger.info("CPA:{}, mIoU:{}, fwIoU: {}".format(CPA, mIoU, FWIoU)) + return mIoU + +def compute_cpa(evaluator): + CPA = evaluator.Pixel_Accuracy_Class() + logger.info(f"CPA: {CPA}") + return CPA + +def compute_miou(evaluator): + mIoU = evaluator.Mean_Intersection_over_Union() + logger.info(f"mIoU: {mIoU}") + return mIoU + +def compute_fwiou(evaluator): + FWIoU = evaluator.Frequency_Weighted_Intersection_over_Union() + logger.info(f"FWIoU: {FWIoU}") + return FWIoU diff --git a/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/testenv.yaml b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/testenv.yaml new file mode 100644 index 00000000..2f0263e0 --- /dev/null +++ b/examples/Cloud_Robotics/singletask_learning_bench/Semantic_Segmentation/testenv/testenv.yaml @@ -0,0 +1,35 @@ +testenv: + # dataset configuration + dataset: + # the url address of train dataset index; string type; + train_url: "/home/aryan/ianvs/dataset/train-index.txt" + # the url address of test dataset index; string type; + test_url: "/home/aryan/ianvs/dataset/test-index.txt" + + # model eval configuration of incremental learning; + model_eval: + # metric used for model evaluation + model_metric: + # metric name; string type; + name: "accuracy" + # the url address of python file + url: "./examples/cloud-robotics/single_task_learing/testenv/accuracy.py" + mode: "no-inference" + + # condition of triggering inference model to update + # threshold of the condition; types are float/int + threshold: 0 + # operator of the condition; string type; + # values are ">=", ">", "<=", "<" and "="; + operator: "<" + + # metrics configuration for test case's evaluation; list type; + metrics: + # metric name; string type; + - name: "accuracy" + # the url address of python file + url: "./examples/cloud_robotics/sinlge_task_learning/testenv/accuracy.py" + - name: "samples_transfer_ratio" + + # incremental rounds setting; int type; default value is 2; + incremental_rounds: 5 \ No newline at end of file