Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable control of the follower arm with PS4 joystick #516

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

IliaLarchenko
Copy link

What this does

You can control the follower arm using a PlayStation 4 controller, which can be useful for people who don't have a leader arm or for anybody who wants to try an alternative way of teleoperation.

  • Control all action of the follower arm with the PS4 joystick
  • Can be used for dataset collection
  • Assign fixed positions to any buttons for quick movements
  • Has vibration and red color indication when motor position limits are reached
  • Can even control gripper position using controller IMU (don't think this one is useful, just fun)

Limitations:
The controller should be connected via USB (Bluetooth can require some extra fixes)

I added a new type of device - controllers. I think it makes sense in the long run to have them as a separate device type.

How it was tested

I tested it on Mac and Ubuntu and only with SO-ARM100
You can see the results here: https://www.youtube.com/shorts/Ini2DATLawU

How to checkout & try? (for the reviewer)

It requires hid to run, I didn't add it to the dependencies because it is not needed for most of the users who don't have PS4. So, if you want to try you will need to install it yourself.

I created a new config so100_ps4, adjust the port of the follower arm, connect PS4 joystick via USB and run:

python lerobot/scripts/control_robot.py teleoperate \
    --robot-path lerobot/configs/robot/so100_ps4.yaml \
    --robot-overrides '~cameras' \
    --display-cameras 0

@Cadene
Copy link
Collaborator

Cadene commented Nov 22, 2024

Interesting design! Thanks for your PR. We would like to include it after our improvements of the robot devices API.

@mydhui
Copy link

mydhui commented Dec 11, 2024

@IliaLarchenko Impressive!May I ask some questions,

  1. If I only have ps5 controllers available, can it seamlessly be applied?
  2. Does it support bluetooth connection?
  3. Is it efficient in this way for teleoperation (in terms of data quality)

@IliaLarchenko
Copy link
Author

@mydhui
Unfortunately, it is not so easy.

  1. The PS5 controller has a different structure of packets it sends/receives. It shouldn't work out of the box. But I don't have a PS5 controller and haven't tested it. My guess is it is straightforward to adapt it to PS5, but I didn't try it.
  2. For some reason the data structure when the controller is connected via Bluetooth is not the same as for USB. But it is very similar and if you comment out the accelerometer and feedback part it will work (though when I tested it I saw much higher latency and I would recommend using a USB connection).
  3. You can teleoperate the arm using a controller and do pretty much anything, but it feels very different from teleoperation using the leader's arm. In some cases, it is even a bit more convenient, but generally, it looks more artificial when you use the controller. But anyway, the data is stored in the same format, you can train some policy on top of it and so on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants