Port HIL-SERL

[michel-aractingi]

What this does

Adds HIL-SERL to the policies of LeRobot in lerobot/common/policies/hilserl/.

What this PR contains so far

1. The ability to assign binary rewards during recording datasets in lerobot/scripts/control_robot.py -> record function.
2. Reward classifier,
   • Code to define and train a reward classifier model to detect successful tasks in lerobot/common/policies/hilserl/classifier.
   • Script to train the reward classifier lerobot/scripts/train_hilserl_classifier.py
3. Rollout on the real robot and human intervention, in script lerobot/scripts/eval_on_robot.py we added the ability to do policy rollouts on the real robot. Moreover, you also have the ability to stop the policy actions bieng rolled-out and take over if you have a leader arm.

How to test:

1. Annotate episodes with reward during recordings:

python lerobot/scripts/control_robot.py record \
    --robot-path lerobot/configs/robot/moss.yaml \
    --fps 30 \
    --root data \
    --repo-id ${HF_USER}/moss_test \
    --tags moss tutorial \
    --warmup-time-s 5 \
    --episode-time-s 40 \
    --reset-time-s 10 \
    --num-episodes 2 \
    --push-to-hub 1 \
    --assign_rewards 1

2. Train reward classifier:

python lerobot/scripts/train_hilserl_classifier.py --config-name policy/reward_classifier.yaml

3. Run an example of eval on the robot and test human interventions

python lerobot/scripts/eval_on_robot.py --robot-path lerobot/configs/robot/koch.yaml

References:

1. HIL-SERL implementation https://github.com/rail-berkeley/hil-serl/tree/main
2. reward assignment PR Reward assignment during recording  #518 and classifier Reward classifier and training #528 by @ChorntonYoel
3. human Interventions Add human intervention mechanism and eval_robot script to evaluate policy on the robot #541

[mydhui]

@michel-aractingi Hi, Can you elaborate more on how to test hil-serl ?

During step 1, for instance a cube grasping task, should we record failure samples on purpose or reward transition from 0 to 1 after successful grasping is enough ?

What should be the expected behavior in step 3 ("eval on the robot and test human interventions"), would this algorithm perform better than ACT ?

Thanks.