Remote Teleoperation over WebRTC

[chenosaurus]

Remote Teleoperation Over WebRTC

This PR adds the capability for the leader & follower to be in different locations, streaming data between using LiveKit WebRTC server. All actions/observations/videos are streamed over a single connection, enabling easy remote teleoperation from anywhere.

• Added RemoteRobot and RemoteTeleoperator classes
• Added examples of SO100RemoteLeader and SO100RemoteTeleoperator

RemoteRobot - an instance of a robot that's not physically connected to computer. The send_action() method will send the desired action over LiveKit as a data message.

RemoteTeleoperator - an instance of a teleoperator that's not physically connected to computer. It will subscribe to action data topic and return the received action command in the get_action() method. It also has a new method publish_observation which will publish both the robot state & video to LiveKit.

When running the local leader with display_data=true the received video streams and observations will be rendered in Rerun.

To test, you can use either the OSS LiveKit server or use a free account on LiveKit cloud:

• generate 2 tokens, 1 for the leader and 1 for the follower.

Run local leader with RemoteRobot:

python -m lerobot.teleoperate \
    --robot.type=so100_remote_follower \
    --robot.livekit_url=<livekit server host> \
    --robot.livekit_token=<livekit follower token> \
    --robot.id=white \
    --teleop.type=so100_leader \
    --teleop.port=/dev/tty.usbmodem58FA1025471 \
    --teleop.id=black --display_data=true

Run local follower with RemoteTeleoperator:

python -m lerobot.teleoperate \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ gripper: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \
    --robot.id=white \
    --teleop.type=so100_remote_leader \
    --teleop.livekit_url=<livekit server host> \
    --teleop.livekit_token=<livekit leader token> \
    --teleop.id=black

[Copilot]

Pull Request Overview

This PR adds support for remote teleoperation over WebRTC via a LiveKit server, enabling leader and follower roles to run in separate locations.

• Introduces LiveKitService to manage WebRTC connections and data/video channels.
• Implements RemoteRobot and RemoteTeleoperator classes for sending actions and publishing observations remotely.
• Updates the main teleoperation loop to send observations back to remote teleoperators when applicable.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
lerobot/teleoperate.py	Hook into RemoteTeleoperator to publish observations
lerobot/common/transport/livekit_service.py	Added LiveKitService for WebRTC connection management
lerobot/common/teleoperators/utils.py	Registered so100_remote_leader in factory
lerobot/common/teleoperators/so100_leader/so100_remote_leader.py	New remote leader subclass
lerobot/common/teleoperators/so100_leader/config_so100_leader.py	Config subclass for remote leader
lerobot/common/teleoperators/so100_leader/init.py	Exposed SO100RemoteLeader in package
lerobot/common/teleoperators/remote_teleoperator.py	Core RemoteTeleoperator implementation
lerobot/common/teleoperators/config.py	Added RemoteTeleoperatorConfig
lerobot/common/robots/utils.py	Registered so100_remote_follower in factory
lerobot/common/robots/so100_follower/so100_remote_follower.py	New remote follower subclass
lerobot/common/robots/so100_follower/config_so100_follower.py	Config subclass for remote follower
lerobot/common/robots/so100_follower/init.py	Exposed SO100RemoteFollower in package
lerobot/common/robots/remote_robot.py	Core RemoteRobot implementation
lerobot/common/robots/config.py	Added RemoteRobotConfig

Comments suppressed due to low confidence (1)

lerobot/common/teleoperators/remote_teleoperator.py:200

• The new publish_observation method introduces complex JSON and video-publishing logic without accompanying unit or integration tests; consider adding tests to cover serialization, topic dispatching, and video-track handling.

    def publish_observation(self, observation: dict[str, Any]) -> None:

[Copilot]

Commented-out camera initialization suggests incomplete implementation; either integrate make_cameras_from_configs(config.cameras) or remove the dead code to clarify intent and avoid confusion.

Suggested change

	self.cameras = {} #make_cameras_from_configs(config.cameras)
	self.cameras = make_cameras_from_configs(config.cameras)

[Copilot]

Calling publish_observation (which uses blocking sync methods) inside the main loop can introduce latency and drop below target FPS; consider offloading observation publishing to a separate thread or using an asynchronous approach.

Suggested change

	teleop.publish_observation(observation)
	observation_queue.put(observation)