sim2real-install-guide.md

1. Host operation

1.1 Docker

If docker local installation hasn't been done, switch to the docker_server folder first:

cd ./ICRA-RM-Sim2Real/docker_server

Execution:

./docker_install.sh  

Evaluation

docker --version

Reference for docker installation on Ubuntu:

• docker install

If the shell script cannot be run, check if there is permission for the script. Otherwise change the mode with chmod

1.2 Nvidia driver

Check the version of host GPU driver before creating the docker and container, carefully keeping the same with the version inside docker. Currently the NVIDIA driver version inside the docker repos is 470.86.

In Ubuntu, Software & Updates > Additional Drivers is recommanded to update the Nvidia driver.

Open the terminal, input nvidia-smi and press enter to get the driver version:

Know issue:

• If your OS is Ubuntu21.04 or later, please refer to issue 18 to fix.

1.3 Install the nvidia-docker2

1.3.1 Main stages for docker installation reference

sudo systemctl --now enable docker

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker

# test
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

Reference link for docker installation:

• nvidia-docker2

1.4 docker login

Register the dockerhub account:

• dockerhub

And get access token for late use:

• https://docs.docker.com/docker-hub/access-tokens/

login the docker account

sudo docker login

1.5 Download the docker image

Download the docker image:

sudo docker pull rmus2022/server:v1.0.0

Four image sources are provided for users:

• rmus2022/server:v1.0.0
• rmus2022a/server:v1.0.0
• rmus2022b/server:v1.0.0
• rmus2022c/server:v1.0.0

Note:

• please pay attention to the official docker host for further update.
  • ⁣https://hub.docker.com/r/rmus2022/
• Campus networks might limit the pull docker speed in the mainland

Due to the size of the image files, waiting for minutes to more than an hour to download the files is typical.

1.6 Create the docker container

cd ./ICRA-RM-Sim2Real/docker_server

Please confirm the right tag and repository name in create_container_server.sh

./create_container_server.sh

An Error will be reported while there is no sim2real_server container. No need to worry.

Changes without docker commit will be deleted after each time the script run.

2. Docker Server operation

2.1 To start the docker

Run this line again after reset

sudo docker start sim2real_server 

cd ./ICRA-RM-Sim2Real/docker_server

password: 123

./exec_server.sh

Enter the docker

2.2 Start the Habitat sim

cd ~/habitat-sim/

./build/viewer ./data/scene_datasets/habitat-test-scenes/van-gogh-room.glb 

There should be a window created and scene showed in the window, use W, A, S, D to control agent move.

2.3 Start the server simulator

Start a new terminal.

cd ./ICRA-RM-Sim2Real/docker_server

./exec_server.sh

roscore

Create a new terminal

cd ./ICRA-RM-Sim2Real/docker_server

./exec_server.sh

cd ~/ros_x_habitat_ws/src/ros_x_habitat/

python3 src/scripts/roam_with_joy.py --hab-env-config-path ./configs/roam_configs/pointnav_rgbd_roam_mp3d_test_scenes.yaml

The displays of the RGB/depth/third_rgb should be right.

If there is error, start it again

2.4 Control the movement via the keyboard

Create a new terminal

cd ./ICRA-RM-Sim2Real/docker_server

./exec_server.sh

rosrun teleop_twist_keyboard teleop_twist_keyboard.py

Mouse click to activate the terminal of keyboard control

Press q, z, increase or decrease the speed of the robot.

• Press i， j， ， ， l， to control forward/backward/rotation.
• Press I， J， <， L， to control the horizontal movement.
• Press k, stop the robot moving.
• Press 1, to move the robotic arm to the place to grab up.
• Press 2， to move the robotic arm to the place to place down.
• Press 3， to pick the ore.
• Press 4， to place the ore.

3. Docker Client operation

3.1 Download the docker image

Download the image (according to the latest released version).

sudo docker pull rmus2022/client:v1.0.0

Four image sources are provided for users:

• rmus2022/client:v1.0.0
• rmus2022a/client:v1.0.0
• rmus2022b/client:v1.0.0
• rmus2022c/client:v1.0.0

Note:

• please pay attention to the official docker host for further update.
  • ⁣https://hub.docker.com/r/rmus2022/
• Campus networks might limit the pull docker speed in the mainland

Please confirm the right tag and repository name in create_container_server.sh

3.2 Creator the client container

cd ./ICRA-RM-Sim2Real/docker_client

Confirm the tag in create_container_client is the right version.

Change the CPU and RAM parameter from the create_container_algo.sh according to the host machine, to meet the performance of robot on board NUC unit.

Formula to compute CPU scaling coefficient

cpu = (Freq of single CPU core in NUC * cores of NUC CPU)/(Freq of CPU in the host)

For example:

the host CPU: Intel® Xeon(R) W-2125 CPU @ 4.00GHz * 8
NUC onboard cpu: 11th Gen Intel® Core i7-1165G7 @ 2.80GHz * 8

then: cpu = (2.8 * 8) / 4 = 5.6

NUC onboard RAM: 8GB

then: M=8192M

./create_container_client.sh

for the first time of execution there will be "Error: No such container: sim2real_algo", no need to worry.

Changes without docker commit will be deleted after each time the script run.

3.3 rtab navigation

3.3.1 Restart the server (follow the step 3 in docker server)

The RGB, depth, third_rgb monitor should be correct.

If there is error, start it again.

3.3.2 Start the client

Running again after restart.

sudo docker start sim2real_client

Start a new terminal

cd ICRA-RM-Sim2Real/docker_client

./exec_client.sh

cd ~

roslaunch rtab_navigation rtab_navigation.launch

Send 2D Nav Goal through rviz

3.4 cartographer navigation

3.4.1 Restart the server (follow the step 3 in docker server)

The RGB, depth, third_rgb monitor should be correct.

If there is error, start it again.

3.4.2 Start the client

Start a new terminal with the client docker:

cd ./ICRA-RM-Sim2Real/docker_client

./exec_client.sh

cd ~

roslaunch carto_navigation navigation.launch

Send 2D Nav Goal through rviz

3.5 Pick the mineral ore

Here we will introduce the basic process of how pickup/place the ore, and the demonstration of the usage of the automatic detector.

Note: Here comes the the Vision Markers on the Cube, feel free to build your own detector.

3.5.1 Start the red marker detector

1. Remote control the robot to the front of the ore
2. Start a new terminal with the client docker for the commands below:

roscd ep_detect_and_grasp

python3 detect_cube.py

3.5.2 Start the mineral ore pick demonstration

Start a new terminal with the client docker for the commands below:

python3 grasp_cube.py

3.6 Place the mineral ore with the demonstration

Start a new terminal with the client docker for the commands below:

roscd ep_detect_and_grasp

python3 place_cube.py

1. Remote control the robot to the front of the Exchange Stations
2. Ensure detect_cube.py is started
3. Start a new terminal for the following command:

python3 place_cube.py

4. The robot will get the O between the BOX three Exchange Stations to place

4. Update the client image

The challenge submit the player algorithm with https://hub.docker.com/

4.1 Create a new privtate repo

Create a new private repo for the challenge with the name rmus2022

4.2 Push the client image to the private repo just created

Package the downloaded client image with tag (could be user defined), with the dockerhub_name from the name of your dockerhub account:

sudo docker tag rmus2022/client:v.0.1.0 dockerhub_name/rmus2022:tag

Push the new client image with your tag to your private repo:

sudo docker push dockerhub_name/rmus2022:tag

4.3 Develop your algorithm

According to your private repo and the name of your tag, change the image name in create_container_client.sh:

then:

1. Start the create_client.sh, create a new container
2. Start the exec_client.sh enter the client image to develop

git tool is recommanded to manage the code version.

vscode editor is available to develop in the docker

code ~/ep_ws

4.4 Docker commit

Save the edited docker locally, using the original tag will overwrite the content of the previous tag version.

sudo docker commit sim2real_client dockerhub_name/rmus2022:new_tag

4.5 Docker push

Push the docker to private repo to save the current docker image to dockerhub

sudo docker push dockerhub_name/rmus2022:tag

4.6 Get the access token

Reference link to docker token

• https://docs.docker.com/docker-hub/access-tokens/

While submit the evaluation version to the challenge, the account name and token of dockerhub will submit via the online system.