diff --git a/assets/contributors.csv b/assets/contributors.csv index 533b863724..5c10e1609e 100644 --- a/assets/contributors.csv +++ b/assets/contributors.csv @@ -72,7 +72,6 @@ Kieran Hejmadi,Arm,kieranhejmadi01,kieran-hejmadi-88920815b,, Alex Su,,,,, Chaodong Gong,,,,, Owen Wu,Arm,,,, -Koki Mitsunami,,,,, Nikhil Gupta,,,,, Nobel Chowdary Mandepudi,Arm,,,, Ravi Malhotra,Arm,,,, diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/1-introduction.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/1-introduction.md new file mode 100644 index 0000000000..02a8d98bae --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/1-introduction.md @@ -0,0 +1,68 @@ +--- +title: Understand the SmolVLA fine-tuning workflow +description: Review the LeRobot workflow for recording an SO-101 dataset, fine-tuning SmolVLA, and evaluating the result. +weight: 2 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Overview + +You will teach an SO-101 robot arm to pick up a vial and place it in a rack. First, you will guide the robot through the task and record examples. You will then adapt an AI model called SmolVLA with those examples and use it to control the robot. The software workflow runs on an Arm-based NVIDIA DGX Spark. + +![SO-101 robot arm beside a vial and four-hole yellow rack. The black task mat keeps the pickup and placement area visually distinct for data collection.#center](images/1-so101-workspace.jpg "SO-101 pick-and-place workspace") + +## What is the SO-101? + +The [SO-101 robot arm](https://huggingface.co/docs/lerobot/so101) is an open-source robotic arm designed for learning from demonstrations. This setup uses two arms with matching joints: + +- You move the *leader* arm by hand to demonstrate the task. +- The powered *follower* arm mirrors the leader during data collection and later executes the learned behavior autonomously. + +The leader has the operator handle used to demonstrate arm and gripper motion. The follower has the task gripper and wrist-mounted camera used for recording and autonomous control. + +| SO-101 leader | SO-101 follower | +| --- | --- | +| ![SO-101 leader arm with the operator handle used to demonstrate joint and gripper movements. The leader sends target positions to the follower during teleoperation and data collection.#center](images/1-so101-leader.jpg "SO-101 leader arm") | ![SO-101 follower arm with its task gripper and wrist-mounted camera. The follower mirrors demonstrations during recording and is the arm controlled by the fine-tuned model.#center](images/1-so101-follower.jpg "SO-101 follower arm") | + +Each arm has motors for shoulder, elbow, wrist, and gripper motion. Calibration maps their physical ranges to compatible position values so a demonstration from the leader becomes a meaningful target for the follower. + +The leader is needed only while you demonstrate and record the task. The final setup uses only the follower arm. The fine-tuned AI model reads the follower's cameras and joint state, then sends actions to the follower motors. + +## What is LeRobot? + +[LeRobot](https://huggingface.co/docs/lerobot) is an open-source robotics framework from Hugging Face. It provides command-line tools and Python components for data collection, model training, and model evaluation. + +- During data collection, it reads the cameras and follower joint state, receives target joint positions from the leader, and stores these observations and actions in a standard dataset. +- During training, it loads the same features, applies the model's preprocessing and postprocessing, and saves the fine-tuned model with its configuration. +- During evaluation, LeRobot connects the saved model to the follower and runs the control loop. It captures a new observation, asks the model for actions, sends those actions to the follower, and repeats. + +## What are VLA models and SmolVLA? + +A useful way to understand a vision-language-action (VLA) model is as a progression from language models to models that can act in the physical world: + +1. A large language model (LLM) receives text tokens and produces text tokens. It can follow written instructions, but it doesn't directly see the robot's environment. +2. A vision-language model (VLM) adds images to the text context. It can describe a scene or answer questions about it, but its normal output is still language. +3. A VLA adds robot state and an action-generation component. Instead of producing only text, it produces numerical actions that can control a robot. + +A VLA receives one or more camera images, a natural-language task instruction, and the robot's current joint state. For the SO-101, the state includes the measured joint positions. The output is a continuous robot action containing targets for the shoulder, elbow, wrist, and gripper joints. The model repeatedly observes the updated scene and state, predicts the next actions, and forms a closed control loop with the robot. + +[SmolVLA](https://huggingface.co/docs/lerobot/smolvla) is Hugging Face's lightweight foundation model for robot control. Its pretrained representations provide a starting point for understanding images and instructions, but you fine-tune it on your own demonstrations so it learns the robot geometry, camera viewpoints, and task behavior. + +For this Learning Path, SmolVLA receives the gripper-camera image, workspace-camera image, pick-and-place instruction, and current SO-101 follower state. It outputs joint targets that LeRobot sends to the follower. The fine-tuning page explains the SmolVLA architecture and action-generation process in more detail. + +## Review the workflow + +The Learning Path follows this sequence: + +1. Install LeRobot and verify CUDA and the robot command-line tools. +2. Connect the SO-101 leader, follower, and two cameras, then identify their device paths. +3. Calibrate both arms and verify leader-to-follower teleoperation. +4. Record and inspect the demonstrations, then optionally upload the dataset to your Hugging Face account. +5. Fine-tune SmolVLA from the local dataset on the DGX Spark GPU. +6. Run controlled physical trials and record each outcome. + +## What you've learned + +You now know how the SO-101, LeRobot, and SmolVLA fit into the workflow. Next, create the LeRobot environment and verify that CUDA and all required command-line tools are available. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/2-install-lerobot.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/2-install-lerobot.md new file mode 100644 index 0000000000..91f6522fa0 --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/2-install-lerobot.md @@ -0,0 +1,78 @@ +--- +title: Install LeRobot +description: Prepare and verify a LeRobot environment for SO-101 data collection and SmolVLA fine-tuning. +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Prepare the software environment + +The tested workflow uses LeRobot v0.6.0, Python 3.12.3, and PyTorch 2.11.0 with CUDA 12.8. + +Check that `git`, `ffmpeg`, and `uv` are installed: + +```bash +git --version +ffmpeg -version +uv --version +``` + +Use the [LeRobot installation documentation](https://huggingface.co/docs/lerobot/installation) if one of these tools is missing. + +## Create an isolated environment + +Clone the official LeRobot repository and check out the tested revision. + +```bash +git clone https://github.com/huggingface/lerobot.git +cd lerobot +git checkout 30da8e687a6dfc617fcd94afc367ac7071c376ce + +uv venv --python 3.12 .venv +source .venv/bin/activate +uv pip install -e ".[core_scripts,training,feetech,smolvla]" +``` + +The extras install the robot command-line tools, training dependencies, Feetech motor support, and SmolVLA dependencies. + +## Verify LeRobot and CUDA + +Print the core versions and confirm that PyTorch can see CUDA: + +```bash +python -c "import platform, torch; from importlib.metadata import version; print({'python': platform.python_version(), 'lerobot': version('lerobot'), 'torch': torch.__version__, 'cuda': torch.cuda.is_available()})" +``` + +The output from the DGX Spark should be similar to: + +```output +{'python': '3.12.3', 'lerobot': '0.6.0', 'torch': '2.11.0+cu128', 'cuda': True} +``` + +Verify the commands used later: + +```bash +for cli in lerobot-calibrate lerobot-find-cameras lerobot-find-port \ + lerobot-record lerobot-rollout lerobot-teleoperate lerobot-train; do + command -v "$cli" > /dev/null || exit 1 +done +``` + +The loop exits without output when all commands are available. + +## Authenticate with Hugging Face + +Sign into Hugging Face interactively: + +```bash +hf auth login +hf auth whoami +``` + +Follow the browser or terminal prompt. + +## What you've accomplished + +You now have a Python environment with CUDA, LeRobot's SO-101 tools, and SmolVLA dependencies. Keep this terminal active. Next, connect the two arms and cameras to the DGX Spark and identify their device paths. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/3-prepare-so101.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/3-prepare-so101.md new file mode 100644 index 0000000000..3f842e1f7f --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/3-prepare-so101.md @@ -0,0 +1,139 @@ +--- +title: Connect the SO-101 and cameras +description: Connect and identify the SO-101 leader, follower, gripper camera, and workspace camera for LeRobot. +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Connect the SO-101 hardware + +Connect the SO-101 leader, SO-101 follower, gripper camera, and workspace camera to the DGX Spark over USB. + +The connections are: + +```text +SO-101 leader ------- USB ---\ +SO-101 follower ----- USB ----+-- DGX Spark +Gripper camera ------ USB ----| (LeRobot host) +Workspace camera ---- USB ---/ +``` + +Mount the gripper camera on the follower for a close view of grasping. Keep the workspace camera fixed and frame the follower, pickup area, rack, and full range of arm movement. + +{{% notice Note %}} +On Ubuntu, the robot controller appears as a serial device. Membership in the `dialout` group lets your user account open that device without running every LeRobot command with `sudo`. Add your account to the group: + +```bash +sudo usermod -aG dialout "$USER" +``` + +Log out and back in. Return to the cloned LeRobot directory, reactivate the environment, and verify group membership: + +```bash +source .venv/bin/activate +id -nG | tr ' ' '\n' | grep '^dialout$' +``` + +The expected output is: + +```output +dialout +``` +{{% /notice %}} + +Hardware commands in this and later pages use environment variables named `ROBOT_PORT`, `LEADER_PORT`, `GRIPPER_CAMERA_ID`, and `WORKSPACE_CAMERA_ID`. Set them to the device paths reported by LeRobot's discovery tools. + +## Identify the robot arm ports + +Run the port finder: + +```bash +lerobot-find-port +``` + +The output lists all serial ports on the host. The relevant lines are similar to: + +```output +Finding all available ports for the MotorsBus. +Ports before disconnecting: [..., '/dev/ttyACM0', '/dev/ttyACM1'] +Remove the USB cable from your MotorsBus and press Enter when done. +The port of this MotorsBus is '/dev/ttyACM0' +Reconnect the USB cable. +``` + +Device paths can differ on your system. Identify the two roles in this order: + +1. On the first run, disconnect only the follower USB data cable. The detected path is the value for `ROBOT_PORT`. Reconnect the follower and wait for the device to reappear. +2. Run `lerobot-find-port` again and disconnect only the leader USB data cable. The detected path is the value for `LEADER_PORT`. Reconnect the leader before continuing. + +Save the complete device paths in environment variables for the current terminal. For example, if the two discovery runs reported `/dev/ttyACM0` for the follower and `/dev/ttyACM1` for the leader, run: + +```bash +export ROBOT_PORT="/dev/ttyACM0" +export LEADER_PORT="/dev/ttyACM1" +``` + +## Discover the cameras + +{{% notice Note %}} +Close browsers, video-conferencing software, and desktop camera applications before discovery so OpenCV can open both camera streams. +{{% /notice %}} + +Discover the OpenCV streams and save a diagnostic image from each camera: + +```bash +lerobot-find-cameras opencv --output-dir outputs/captured_images +``` + +An abbreviated output looks like: + +```output +--- Detected Cameras --- +Camera #0: + Name: OpenCV Camera @ /dev/video0 + Type: OpenCV + Id: /dev/video0 + Backend api: V4L2 + Default stream profile: + ... + Fourcc: YUYV + Width: 640 + Height: 480 + Fps: 30.0 +-------------------- +Camera #1: + Name: OpenCV Camera @ /dev/video2 + Type: OpenCV + Id: /dev/video2 + Backend api: V4L2 + Default stream profile: + ... + Fourcc: YUYV + Width: 640 + Height: 480 + Fps: 30.0 +-------------------- +``` + +Open the images in `outputs/captured_images/` and match each `Id` to its view. + +| Gripper camera view | Workspace camera view | +| --- | --- | +| ![LeRobot camera-discovery frame showing the orange gripper tips and pickup surface. Use this view for the `gripper_cam` role.#center](images/3-lerobot-gripper-camera.png "Gripper camera role") | ![LeRobot camera-discovery frame showing the follower, vial, yellow rack, and full range of arm movement. Use this view for the `workspace_cam` role.#center](images/3-lerobot-workspace-camera.png "Workspace camera role") | + +Camera numbers and device paths can differ on your host. Use each camera's `Id` value, not `Camera #0` or `Camera #1`. Assign the camera showing the gripper to `GRIPPER_CAMERA_ID`, and assign the camera covering the complete task to `WORKSPACE_CAMERA_ID`. + +The tested setup used the following mapping, but your device paths might differ: + +```bash +export GRIPPER_CAMERA_ID="/dev/video0" +export WORKSPACE_CAMERA_ID="/dev/video2" +``` + +Device paths can change after reconnection, so don't assume that a previous device path still identifies the same arm or camera. + +## What you've accomplished + +You connected and identified both SO-101 arms and assigned the gripper and workspace camera roles. Keep the hardware connected and the camera mounts fixed. Next, calibrate both arms and verify teleoperation before recording demonstrations. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/4-calibrate-teleoperate-so101.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/4-calibrate-teleoperate-so101.md new file mode 100644 index 0000000000..41b145b3dc --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/4-calibrate-teleoperate-so101.md @@ -0,0 +1,78 @@ +--- +title: Calibrate and teleoperate the SO-101 +description: Calibrate the SO-101 leader and follower, then verify teleoperation before recording demonstrations. +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Calibrate the leader and follower + +Continue in the terminal where you set `ROBOT_PORT`, `LEADER_PORT`, `GRIPPER_CAMERA_ID`, and `WORKSPACE_CAMERA_ID`. If you opened a new terminal or reconnected a USB device, repeat discovery and export the current device paths before calibration. + +Calibration maps each joint's encoder readings to its usable motion range so LeRobot can reproduce the leader's movements on the follower consistently. Start each arm with its joints near the middle of their usable ranges, then move each requested joint slowly through its safe range. Support the arm and stop before reaching a mechanical limit. + +LeRobot 0.6.0 asks you to sweep shoulder pan, shoulder lift, elbow flex, wrist flex, and gripper. It assigns the complete encoder range to `wrist_roll`, so that joint isn't included in the recorded sweep. + +Calibrate the leader first: + +```bash +lerobot-calibrate \ + --teleop.type=so101_leader \ + --teleop.port="$LEADER_PORT" \ + --teleop.id=smolvla_leader +``` + +Place the leader in its middle-range pose before pressing Enter. Each joint should have room to move in both directions. The image shows the starting pose used in this setup. + +![Leader SO-101 supported in the middle-range starting pose before calibration begins. Its shoulder, elbow, wrist, and gripper are positioned away from their motion limits.#center](images/4-lerobot-leader-calibration.jpg "Leader calibration starting pose") + +After pressing Enter, support the leader and move each requested joint through its full safe range. The following animation is sped up to show the complete sequence. Perform the movements slowly on your own hardware. + +![Sped-up leader calibration showing the operator supporting the SO-101 and moving the shoulder, elbow, wrist, and gripper through their usable ranges one joint at a time.#center](images/4-lerobot-leader-calibration.gif "Leader joint-range calibration") + +Calibrate the follower with the same process: + +```bash +lerobot-calibrate \ + --robot.type=so101_follower \ + --robot.port="$ROBOT_PORT" \ + --robot.id=smolvla_follower +``` + +Place the follower in the middle-range pose shown in the image, then press Enter and perform the same safe joint sweep used for the leader. + +![Follower SO-101 supported in the middle-range starting pose before calibration begins. The shoulder, elbow, wrist, and gripper all have room to move in both directions.#center](images/4-lerobot-follower-calibration.png "Follower calibration starting pose") + +If LeRobot finds an existing calibration, follow the prompt to reuse it or press `c` to recalibrate. Here, `c` belongs to the calibration prompt; it isn't a dataset-recording control. + +## Verify teleoperation + +Teleoperation uses the leader arm to control the follower in real time. + +Place both arms in similar stable poses and clear the follower workspace. Run a 60-second test: + +```bash +lerobot-teleoperate \ + --robot.type=so101_follower \ + --robot.port="$ROBOT_PORT" \ + --robot.id=smolvla_follower \ + --robot.cameras="{gripper_cam: {type: opencv, index_or_path: $GRIPPER_CAMERA_ID, width: 640, height: 480, fps: 30}, workspace_cam: {type: opencv, index_or_path: $WORKSPACE_CAMERA_ID, width: 640, height: 480, fps: 30}}" \ + --teleop.type=so101_leader \ + --teleop.port="$LEADER_PORT" \ + --teleop.id=smolvla_leader \ + --fps=30 \ + --teleop_time_s=60 \ + --display_data=false +``` + +Move one joint at a time at first. Confirm that the follower mirrors each movement correctly and that its gripper opens and closes. Stop the test if the follower moves unexpectedly. + +The animation shows the follower mirroring the leader’s arm pose and gripper movement. + +![SO-101 teleoperation showing the operator moving the leader while the follower mirrors its arm and gripper motion across the workspace.#center](images/4-lerobot-teleoperation.gif "Leader-to-follower teleoperation") + +## What you've accomplished + +You calibrated both arms and confirmed that the follower mirrors the leader during teleoperation. Next, record the pick-and-place demonstrations. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/5-record-pick-and-place-dataset.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/5-record-pick-and-place-dataset.md new file mode 100644 index 0000000000..39c131c909 --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/5-record-pick-and-place-dataset.md @@ -0,0 +1,176 @@ +--- +title: Record and validate a pick-and-place dataset +description: Record and inspect SO-101 pick-and-place demonstrations for SmolVLA fine-tuning, then optionally upload the dataset to Hugging Face. +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Prepare the recording workspace + +After calibrating and verifying teleoperation, record demonstrations that provide the camera images, robot state, and actions used to fine-tune SmolVLA. You will then inspect the recorded episodes and optionally upload the validated dataset to Hugging Face. + +Use the two-camera workspace prepared during hardware setup. Lay the vial on the black mat and place the rack beside it, within the follower's calibrated reach. The mat provides a consistent task area and visual contrast. Leave enough space between the vial and rack for the follower to grasp and move the vial without colliding with the rack. + +![SO-101 follower workspace with one camera mounted near the gripper and a second camera fixed above the table. The fixed camera covers the vial, yellow rack, follower, and motion area while the gripper camera provides the close manipulation view.#center](images/5-so101-workspace-2cameras.jpg "Two-camera SO-101 workspace") + +## Define a consistent demonstration + +Use this task instruction for every episode: + +```text +Pick up the vial and place it in the yellow rack +``` + +Start each episode with the vial lying on the pickup surface, the rack within the follower's safe reach, and both objects visible in the workspace camera. + +Move the leader smoothly, grasp the vial, place it into the rack, release it, and return to a stable pose. + +Vary the vial and rack positions or orientations slightly between episodes, but stay inside the camera views and calibrated motion range. + +Use these acceptance rules: + +- The gripper closes around the vial rather than pushing it. +- The vial finishes inside the intended rack hole and is released. +- The full task remains visible in both camera streams. +- No person, cable, or unrelated object enters the arm’s range of movement. +- Each accepted episode follows the same task order and lasts for the configured interval. + +## Configure the recording session + +Choose a repository identifier (ID) owned by your Hugging Face account, even when recording locally. Use the form `/` for ``. Choose an empty local path for ``. + +The following command records 50 episodes, each followed by a 15-second reset period: + +```bash +test ! -e && \ +lerobot-record \ + --robot.type=so101_follower \ + --robot.port="$ROBOT_PORT" \ + --robot.id=smolvla_follower \ + --robot.cameras="{gripper_cam: {type: opencv, index_or_path: $GRIPPER_CAMERA_ID, width: 640, height: 480, fps: 30}, workspace_cam: {type: opencv, index_or_path: $WORKSPACE_CAMERA_ID, width: 640, height: 480, fps: 30}}" \ + --teleop.type=so101_leader \ + --teleop.port="$LEADER_PORT" \ + --teleop.id=smolvla_leader \ + --dataset.repo_id= \ + --dataset.root= \ + --dataset.single_task="Pick up the vial and place it in the yellow rack" \ + --dataset.num_episodes=50 \ + --dataset.episode_time_s=30 \ + --dataset.reset_time_s=15 \ + --dataset.fps=30 \ + --dataset.video=true \ + --dataset.streaming_encoding=false \ + --dataset.push_to_hub=false \ + --display_data=false \ + --play_sounds=true \ + --resume=false +``` + +{{% notice Note %}} +LeRobot provides keyboard controls to end an episode or reset period early, or to cancel the current episode. These controls can be useful when you need to discard a demonstration. In the tested LeRobot 0.6.0 setup, however, using them occasionally caused the recording process to stop with an error. Behavior might differ with other versions or systems. To reduce the risk of an interrupted session, let the configured episode and reset timers expire whenever possible. If you use a keyboard control, confirm that the recorder remains active before starting the next episode. +{{% /notice %}} + +After the recorder exits, read the finalized summary: + +```bash +python - <<'PY' +import json +from pathlib import Path + +info = json.loads((Path("") / "meta/info.json").read_text()) +print({ + "episodes": info["total_episodes"], + "frames": info["total_frames"], + "fps": info["fps"], + "splits": info["splits"], +}) +PY +``` + +For reference, the validated run produced: + +```output +{'episodes': 50, 'frames': 44490, 'fps': 30, 'splits': {'train': '0:50'}} +``` + +This summary confirms that the recorder finalized the requested episodes and train split. + +## Review an episode with Rerun + +[Rerun](https://rerun.io/docs/getting-started/what-is-rerun) is a visualization toolkit for time-aligned multimodal data. It lets you scrub camera frames alongside robot state and action plots, making it easier to spot missing images, timing gaps, or unexpected motion. + +The `lerobot-dataset-viz` command launches Rerun with the recorded camera, state, and action data: + +```bash +lerobot-dataset-viz \ + --repo-id \ + --root \ + --episode-index 15 \ + --mode distant \ + --host 127.0.0.1 \ + --web-port 9090 \ + --grpc-port 9876 \ + --display-compressed-images +``` + +Open `http://127.0.0.1:9090` while the command is running. Use `--display-compressed-images` for image datasets because uncompressed frames can exceed Rerun's distant-mode memory buffer and make early images appear blank. Press `Ctrl+C` after review. + +The animation shows the Rerun view while episode 15 is scrubbed. Both camera panes update with the task, and the action and state plots update as the arm moves. + +![Rerun Viewer animation showing episode 15 being scrubbed with the gripper and workspace camera panes, action and state plots, and image timelines visible together.#center](images/5-lerobot-dataset-viz.gif "Inspecting an episode with LeRobot and Rerun") + +Scrub from the first frame to the last. Check that the approach, grasp, lift, placement, release, and final pose before the reset period are visible. Reject a dataset with missing camera intervals, abrupt unexplained actions, unsafe motion, or inconsistent task order. + +### Understand state and action + +The visualizer plots `state` and `action` alongside the camera images: + +- `state` contains calibrated joint positions measured from the follower. +- `action` contains target joint positions produced by the leader and commanded to the follower. + +For the tested SO-101, vector indices map to joints as follows: + +| Index | Joint | +|---|---| +| `[0]` | Shoulder pan | +| `[1]` | Shoulder lift | +| `[2]` | Elbow flex | +| `[3]` | Wrist flex | +| `[4]` | Wrist roll | +| `[5]` | Gripper | + +LeRobot reads the follower state before sending the current action. The curves don't need to overlap exactly because one is a measured position and the other is a target at a slightly different point in the control loop. Motor response time, load, and mechanical compliance add further delay. + +During fine-tuning, SmolVLA learns to predict actions from camera observations, task text, and robot state. The action is the behavior to reproduce; the state describes the pose from which the model acts. + +## Optionally upload to Hugging Face + +Uploading isn't needed when training on the same machine that stores the dataset. A Hub repository is useful when you want versioned sharing, the hosted dataset viewer, or access from a cloud GPU job. Uploading stores the dataset; it doesn't provision GPU compute. + +Use a repository ID under your own account. Choose a license that covers the demonstrations and video, and decide whether the repository should be public or private. Then run: + +```bash +python - <<'PY' +from pathlib import Path +from lerobot.datasets.lerobot_dataset import LeRobotDataset + +dataset = LeRobotDataset( + repo_id="", + root=Path(""), +) +dataset.push_to_hub( + tags=["lerobot", "so101", "smolvla"], + license="", + private=False, + push_videos=True, +) +PY +``` + +Set `private=True` if the data shouldn't be public. The `push_to_hub()` method creates the repository, uploads the dataset and videos, generates a dataset card, and tags the revision with the LeRobot version. + +## What you've accomplished + +You recorded and validated a pick-and-place dataset with synchronized camera, state, and action features. You can train directly from the local dataset or use the optional Hub upload for sharing and remote access. Next, fine-tune SmolVLA from `` on the DGX Spark GPU. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/6-finetune-smolvla.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/6-finetune-smolvla.md new file mode 100644 index 0000000000..13e4532201 --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/6-finetune-smolvla.md @@ -0,0 +1,133 @@ +--- +title: Fine-tune SmolVLA +description: Fine-tune SmolVLA with the recorded SO-101 dataset, monitor training, and validate the saved model. +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Fine-tune SmolVLA with your dataset + +You will fine-tune the SmolVLA model with the SO-101 demonstrations you recorded. Fine-tuning adapts the model to your two camera views, robot joint layout, and pick-and-place task. Training runs locally on the DGX Spark GPU and produces a model checkpoint for physical evaluation. + +Use the same LeRobot environment as the previous pages. Keep the values of `` and `` from the recording step. + +## Run fine-tuning + +Choose unused paths for `` and ``. The output directory stores checkpoints and training metadata, while the log file stores the terminal output. The command stops instead of overwriting either path if it already exists. + +Replace the placeholders, then run: + +```bash +set -o pipefail +test ! -e && \ +test ! -e && \ +PYTHONUNBUFFERED=1 lerobot-train \ + --policy.path=lerobot/smolvla_base \ + --policy.device=cuda \ + --policy.push_to_hub=false \ + --policy.empty_cameras=1 \ + --rename_map='{"observation.images.gripper_cam":"observation.images.camera1","observation.images.workspace_cam":"observation.images.camera2"}' \ + --dataset.repo_id= \ + --dataset.root= \ + --output_dir= \ + --job_name=smolvla-so101-pick-place \ + --batch_size=64 \ + --steps=20000 \ + --save_checkpoint=true \ + --save_freq=20000 \ + --log_freq=200 \ + --wandb.enable=false \ + 2>&1 | tee -a +``` + +The command loads the local dataset, fine-tunes the model, and saves the final checkpoint. + +The key options are: + +- `--policy.path=lerobot/smolvla_base` selects the pretrained SmolVLA model to fine-tune. In robotics, a policy is a model that converts observations, such as camera images and joint positions, into actions that control the robot. +- `--rename_map` maps the camera feature names in the dataset to the names expected by SmolVLA. The model expects camera inputs named `camera1`, `camera2`, and `camera3`, while this dataset contains `gripper_cam` and `workspace_cam`. The command maps: + - `gripper_cam` to `camera1` + - `workspace_cam` to `camera2` +- `--policy.empty_cameras=1` allows one expected camera input to be absent. The model's third camera input, `camera3`, is left empty. +- `--steps=20000` sets a fine-tuning baseline of 20,000 optimization steps. The SmolVLA documentation uses this value as a starting point. +- `--batch_size=64` follows the documented starting configuration and fits within the memory available on the DGX Spark used for this Learning Path. +- `--save_checkpoint=true` enables checkpoint saving, while `--save_freq=20000` saves a checkpoint after 20,000 steps. Because the training run is also 20,000 steps long, this configuration saves the final checkpoint without creating intermediate checkpoints. +- `--policy.push_to_hub=false` prevents the trained model from being uploaded automatically to the Hugging Face Hub. +- `PYTHONUNBUFFERED=1` makes Python write log messages immediately instead of buffering them. This allows `tee` to display and save the training output in real time. +- `2>&1 | tee -a ` displays both standard output and error messages in the terminal and appends them to the specified log file. + +## Monitor training + +Open another terminal and run: + +```bash +nvidia-smi +tail -f +``` + +`nvidia-smi` confirms that the training process is using the CUDA-enabled GPU. The log shows the current step, loss, learning rate, and timing information. Press `Ctrl+C` to stop following the log; this doesn't stop training in the original terminal. + +After training finishes, set `` to the saved model directory: + +```text +/checkpoints/020000/pretrained_model +``` + +Verify that the required model, configuration, and processor files exist and aren't empty: + +```bash +for file in config.json model.safetensors train_config.json \ + policy_preprocessor.json policy_postprocessor.json; do + test -s "/$file" || { + echo "Missing or empty: $file" >&2 + exit 1 + } +done +echo "Checkpoint files verified." +``` + +The expected output is: + +```output +Checkpoint files verified. +``` + +## Understand how SmolVLA works + +[SmolVLA](https://huggingface.co/docs/lerobot/smolvla) is a compact vision-language-action (VLA) model designed for fine-tuning on robot demonstrations. It combines a pretrained SmolVLM2 vision-language backbone with an action expert: + +1. Camera images, the task instruction, and the current joint state provide context. +2. The vision-language backbone encodes the images and text, while a projection encodes the robot state. +3. A flow-matching action expert generates a chunk of continuous future joint commands. +4. During evaluation, LeRobot executes commands from the chunk and requests updated chunks as new observations arrive. + +An *action chunk* is a short sequence of commands rather than one command. + +- During training, flow matching adds noise to demonstrated action chunks and learns how to recover the demonstrated actions. +- During evaluation, the model starts from noise and iteratively produces a chunk conditioned on the live observations and instruction. + +The [SmolVLA research paper](https://arxiv.org/abs/2506.01844) describes the model and its asynchronous inference design. + +Fine-tuning adapts the model to the SO-101 geometry, camera viewpoints, and demonstrated behavior. With this configuration, the vision encoder remains frozen while the action expert and projection layers learn from the dataset. + +## Review results from the example experiment + +The following results come from the experiment performed while creating this Learning Path. They show what happened with this dataset on the DGX Spark; they aren't expected values for every training run. + +The example run completed 20,000 steps in about 30 hours. Mean loss decreased across successive 4,000-step windows: + +| Steps | Mean training loss | +|---|---:| +| 200–4,000 | 0.1031 | +| 4,200–8,000 | 0.0472 | +| 8,200–12,000 | 0.0287 | +| 12,200–16,000 | 0.0204 | +| 16,200–20,000 | 0.0175 | + +The final logged loss was 0.017, and the last 2,000 steps stayed between 0.017 and 0.018. These values show stable optimization for this example. These values don’t measure physical task success. + +## What you've accomplished + +You fine-tuned SmolVLA with the recorded SO-101 dataset. Next, connect the follower and cameras to run controlled physical evaluations. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/7-evaluate-model.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/7-evaluate-model.md new file mode 100644 index 0000000000..5690527299 --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/7-evaluate-model.md @@ -0,0 +1,117 @@ +--- +title: Evaluate the fine-tuned model +description: Load the fine-tuned SmolVLA model, run an SO-101 trial, and evaluate the task outcomes. +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Evaluate the fine-tuned model + +You will load the model checkpoint from the previous page and use it to control the SO-101 follower for one pick-and-place attempt. + +During evaluation, the leader arm is not used. The model receives: + +- Images from the gripper and workspace cameras +- The current joint positions of the follower +- The task instruction + +## Prepare the evaluation + +Reconnect the follower and both cameras. + +Confirm that the following environment variables still identify the correct devices: + +- `$ROBOT_PORT` +- `$GRIPPER_CAMERA_ID` +- `$WORKSPACE_CAMERA_ID` + +USB device identifiers can change when a device is disconnected or the computer is restarted. If a device is no longer detected correctly, repeat the device-discovery steps from the hardware setup section. + +Keep the following settings consistent with data collection: + +- Camera positions and mounting angles +- Task instruction +- Vial and rack layout used during data collection + +Place the vial and rack within the range of positions represented in the training dataset. A fine-tuned model is less likely to succeed when the starting layout is significantly different from its demonstrations. + +Set `` to the checkpoint created during fine-tuning. + +## Run the model + +{{% notice Warning %}} +The follower may begin moving as soon as model initialization finishes. Keep your hands and other objects outside the robot workspace, and keep the emergency stop within reach. +{{% /notice %}} + +In robotics, one complete execution of a trained model is called a rollout. + +Run the following command: + +```bash +lerobot-rollout \ + --strategy.type=base \ + --inference.type=rtc \ + --policy.path= \ + --device=cuda \ + --robot.type=so101_follower \ + --robot.port="$ROBOT_PORT" \ + --robot.id=smolvla_follower \ + --robot.cameras="{gripper_cam: {type: opencv, index_or_path: $GRIPPER_CAMERA_ID, width: 640, height: 480, fps: 30}, workspace_cam: {type: opencv, index_or_path: $WORKSPACE_CAMERA_ID, width: 640, height: 480, fps: 30}}" \ + --rename_map="{observation.images.gripper_cam: observation.images.camera1, observation.images.workspace_cam: observation.images.camera2}" \ + --task="Pick up the vial and place it in the yellow rack" \ + --fps=30 \ + --duration=30 \ + --display_data=false \ + --play_sounds=true \ + --return_to_initial_position=false +``` + +The command runs the model for up to 30 seconds. During this time, observe how the follower approaches the vial, attempts to grasp it, and moves it toward the rack. + +The key options are: + +- `--policy.path=` loads the fine-tuned model checkpoint. +- `--robot.type=so101_follower` selects the SO-101 follower implementation. +- `--robot.id=smolvla_follower` selects the calibration associated with this robot identifier. +- `--rename_map` maps the live camera feature names to the input names expected by the model. This mapping must match the mapping used during fine-tuning: + - `gripper_cam` becomes `camera1` + - `workspace_cam` becomes `camera2` +- `--task` provides the language instruction to the model. Use the same wording that was used when recording the dataset. +- `--inference.type=rtc` enables Real-Time Chunking. RTC allows the model to update its planned action sequence when new observations become available while the robot-control loop continues running. +- `--fps=30` sets the target robot-control frequency to 30 updates per second. +- `--duration=30` stops model control after 30 seconds. +- `--return_to_initial_position=false` prevents the follower from automatically moving back to its initial pose after the rollout. + +When the command finishes, the follower remains in its final pose. Wait until all motion has stopped before approaching the robot or resetting the workspace. + +## Observe the result + +The animation shows an example rollout of the fine-tuned model. + +Watch how the follower: + +1. Moves toward the vial +2. Positions the gripper around it +3. Closes the gripper +4. Moves toward the rack +5. Releases the vial + +![Animation of an SO-101 follower running a SmolVLA evaluation. The arm approaches and grasps the vial, then moves it toward the rack, showing the autonomous motion to observe during a physical trial.#center](images/7-smolvla-model-evaluation.gif "SmolVLA pick-and-place evaluation on the SO-101 follower") + +A successful rollout should complete the requested task without human intervention. For this example, that means the follower grasps the vial, moves it to the rack, places it inside a rack hole, and releases it safely. + +A rollout is unsuccessful if: + +- The follower does not grasp the vial +- The vial is dropped +- The vial is released outside a rack hole +- The task is not completed before the time limit +- You need to stop or assist the robot + +A failed rollout does not necessarily indicate a problem with the command. It may indicate that the starting layout is outside the demonstrated range, the demonstrations were inconsistent, or the model needs additional training. + +## What you've accomplished + +You loaded a fine-tuned SmolVLA checkpoint and used it to control the SO-101 follower. You also observed the model's behavior during a physical pick-and-place attempt. diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_index.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_index.md new file mode 100644 index 0000000000..88badde22a --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_index.md @@ -0,0 +1,84 @@ +--- +title: Fine-tune SmolVLA for an SO-101 pick-and-place task + +draft: true +cascade: + draft: true + +description: Record SO-101 demonstrations, fine-tune SmolVLA on an Arm-based NVIDIA DGX Spark, and evaluate the model with LeRobot. + +minutes_to_complete: 120 + +who_is_this_for: This is an advanced topic for robotics and AI developers who want to train a vision-language-action model from their own SO-101 demonstrations. + +learning_objectives: + - Set up a LeRobot environment for SO-101 data collection and SmolVLA training. + - Connect, calibrate, and teleoperate an SO-101 leader-follower pair with cameras. + - Record and inspect a pick-and-place dataset, then optionally upload it. + - Fine-tune and physically evaluate a SmolVLA model on a NVIDIA DGX Spark. + +prerequisites: + - A NVIDIA DGX Spark with at least 30 GB of free storage. + - An assembled SO-101 leader and follower, two USB cameras, and an unobstructed workspace. + - A vial or similar graspable object and a stable rack for the placement target. + - A [Hugging Face account](https://huggingface.co/join) if you want to upload the dataset. + +author: Koki Mitsunami + +# New Learning Paths are opted in for the next manual generated summary/FAQ run. +# The generator resets this to false after a successful write. +generate_summary_faq: true + +# Optional one-shot controls: set either field to true to regenerate just that +# generated section the next time the summary/FAQ tool runs. The tool resets +# them to false after a successful write. +rerun_summary: false +rerun_faqs: false + +### Tags +skilllevels: Advanced +subjects: ML +armips: + - Cortex-X + - Cortex-A +tools_software_languages: + - Python + - PyTorch + - LeRobot + - SmolVLA + - Hugging Face +operatingsystems: + - Linux + +further_reading: + - resource: + title: LeRobot installation documentation + link: https://huggingface.co/docs/lerobot/installation + type: documentation + - resource: + title: LeRobot SO-101 documentation + link: https://huggingface.co/docs/lerobot/so101 + type: documentation + - resource: + title: SmolVLA documentation + link: https://huggingface.co/docs/lerobot/smolvla + type: documentation + - resource: + title: SmolVLA base model card + link: https://huggingface.co/lerobot/smolvla_base + type: documentation + - resource: + title: SmolVLA research paper + link: https://arxiv.org/abs/2506.01844 + type: website + - resource: + title: Train an SO-101 Robot From Sim-to-Real With NVIDIA Isaac + link: https://docs.nvidia.com/learning/physical-ai/sim-to-real-so-101/latest/index.html + type: documentation + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_next-steps.md b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-follower.jpg b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-follower.jpg new file mode 100644 index 0000000000..88389d9ddc Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-follower.jpg differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-leader.jpg b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-leader.jpg new file mode 100644 index 0000000000..f5f02d4498 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-leader.jpg differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-workspace.jpg b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-workspace.jpg new file mode 100644 index 0000000000..1ab7d206d3 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/1-so101-workspace.jpg differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-gripper-camera.png b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-gripper-camera.png new file mode 100644 index 0000000000..8b846faa30 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-gripper-camera.png differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-workspace-camera.png b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-workspace-camera.png new file mode 100644 index 0000000000..e9cee21d64 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/3-lerobot-workspace-camera.png differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-follower-calibration.png b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-follower-calibration.png new file mode 100644 index 0000000000..dd45df9dd1 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-follower-calibration.png differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.gif b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.gif new file mode 100644 index 0000000000..3a68089f12 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.gif differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.jpg b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.jpg new file mode 100644 index 0000000000..4b2930098c Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-leader-calibration.jpg differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-teleoperation.gif b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-teleoperation.gif new file mode 100644 index 0000000000..e1963ac9de Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/4-lerobot-teleoperation.gif differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-lerobot-dataset-viz.gif b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-lerobot-dataset-viz.gif new file mode 100644 index 0000000000..b145e829b3 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-lerobot-dataset-viz.gif differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-so101-workspace-2cameras.jpg b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-so101-workspace-2cameras.jpg new file mode 100644 index 0000000000..c25633282e Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/5-so101-workspace-2cameras.jpg differ diff --git a/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/7-smolvla-model-evaluation.gif b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/7-smolvla-model-evaluation.gif new file mode 100644 index 0000000000..4e8c0f1331 Binary files /dev/null and b/content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/images/7-smolvla-model-evaluation.gif differ