Add a Learning Path for teaching an SO-101 robot arm a pick-and-place task

assets/contributors.csv

@@ -72,7 +72,6 @@ Kieran Hejmadi,Arm,kieranhejmadi01,kieran-hejmadi-88920815b,,
 Alex Su,,,,,
 Chaodong Gong,,,,,
 Owen Wu,Arm,,,,
-Koki Mitsunami,,,,,
 Nikhil Gupta,,,,,
 Nobel Chowdary Mandepudi,Arm,,,,
 Ravi Malhotra,Arm,,,,

content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/1-introduction.md

@@ -0,0 +1,68 @@
+---
+title: Understand the SmolVLA fine-tuning workflow
+description: Review the LeRobot workflow for recording an SO-101 dataset, fine-tuning SmolVLA, and evaluating the result.
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview
+
+You will teach an SO-101 robot arm to pick up a vial and place it in a rack. First, you will guide the robot through the task and record examples. You will then adapt an AI model called SmolVLA with those examples and use it to control the robot. The software workflow runs on an Arm-based NVIDIA DGX Spark.
+
+![SO-101 robot arm beside a vial and four-hole yellow rack. The black task mat keeps the pickup and placement area visually distinct for data collection.#center](images/1-so101-workspace.jpg "SO-101 pick-and-place workspace")
+
+## What is the SO-101?
+
+The [SO-101 robot arm](https://huggingface.co/docs/lerobot/so101) is an open-source robotic arm designed for learning from demonstrations. This setup uses two arms with matching joints:
+
+- You move the *leader* arm by hand to demonstrate the task.
+- The powered *follower* arm mirrors the leader during data collection and later executes the learned behavior autonomously.
+
+The leader has the operator handle used to demonstrate arm and gripper motion. The follower has the task gripper and wrist-mounted camera used for recording and autonomous control.
+
+| SO-101 leader | SO-101 follower |
+| --- | --- |
+| ![SO-101 leader arm with the operator handle used to demonstrate joint and gripper movements. The leader sends target positions to the follower during teleoperation and data collection.#center](images/1-so101-leader.jpg "SO-101 leader arm") | ![SO-101 follower arm with its task gripper and wrist-mounted camera. The follower mirrors demonstrations during recording and is the arm controlled by the fine-tuned model.#center](images/1-so101-follower.jpg "SO-101 follower arm") |
+
+Each arm has motors for shoulder, elbow, wrist, and gripper motion. Calibration maps their physical ranges to compatible position values so a demonstration from the leader becomes a meaningful target for the follower.
+
+The leader is needed only while you demonstrate and record the task. The final setup uses only the follower arm. The fine-tuned AI model reads the follower's cameras and joint state, then sends actions to the follower motors.
+
+## What is LeRobot?
+
+[LeRobot](https://huggingface.co/docs/lerobot) is an open-source robotics framework from Hugging Face. It provides command-line tools and Python components for data collection, model training, and model evaluation.
+
+- During data collection, it reads the cameras and follower joint state, receives target joint positions from the leader, and stores these observations and actions in a standard dataset.
+- During training, it loads the same features, applies the model's preprocessing and postprocessing, and saves the fine-tuned model with its configuration.
+- During evaluation, LeRobot connects the saved model to the follower and runs the control loop. It captures a new observation, asks the model for actions, sends those actions to the follower, and repeats.
+
+## What are VLA models and SmolVLA?
+
+A useful way to understand a vision-language-action (VLA) model is as a progression from language models to models that can act in the physical world:
+
+1. A large language model (LLM) receives text tokens and produces text tokens. It can follow written instructions, but it doesn't directly see the robot's environment.
+2. A vision-language model (VLM) adds images to the text context. It can describe a scene or answer questions about it, but its normal output is still language.
+3. A VLA adds robot state and an action-generation component. Instead of producing only text, it produces numerical actions that can control a robot.
+
+A VLA receives one or more camera images, a natural-language task instruction, and the robot's current joint state. For the SO-101, the state includes the measured joint positions. The output is a continuous robot action containing targets for the shoulder, elbow, wrist, and gripper joints. The model repeatedly observes the updated scene and state, predicts the next actions, and forms a closed control loop with the robot.
+
+[SmolVLA](https://huggingface.co/docs/lerobot/smolvla) is Hugging Face's lightweight foundation model for robot control. Its pretrained representations provide a starting point for understanding images and instructions, but you fine-tune it on your own demonstrations so it learns the robot geometry, camera viewpoints, and task behavior.
+
+For this Learning Path, SmolVLA receives the gripper-camera image, workspace-camera image, pick-and-place instruction, and current SO-101 follower state. It outputs joint targets that LeRobot sends to the follower. The fine-tuning page explains the SmolVLA architecture and action-generation process in more detail.
+
+## Review the workflow
+
+The Learning Path follows this sequence:
+
+1. Install LeRobot and verify CUDA and the robot command-line tools.
+2. Connect the SO-101 leader, follower, and two cameras, then identify their device paths.
+3. Calibrate both arms and verify leader-to-follower teleoperation.
+4. Record and inspect the demonstrations, then optionally upload the dataset to your Hugging Face account.
+5. Fine-tune SmolVLA from the local dataset on the DGX Spark GPU.
+6. Run controlled physical trials and record each outcome.
+
+## What you've learned
+
+You now know how the SO-101, LeRobot, and SmolVLA fit into the workflow. Next, create the LeRobot environment and verify that CUDA and all required command-line tools are available.

content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/2-install-lerobot.md

@@ -0,0 +1,78 @@
+---
+title: Install LeRobot
+description: Prepare and verify a LeRobot environment for SO-101 data collection and SmolVLA fine-tuning.
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Prepare the software environment
+
+The tested workflow uses LeRobot v0.6.0, Python 3.12.3, and PyTorch 2.11.0 with CUDA 12.8.
+
+Check that `git`, `ffmpeg`, and `uv` are installed:
+
+```bash
+git --version
+ffmpeg -version
+uv --version
+```
+
+Use the [LeRobot installation documentation](https://huggingface.co/docs/lerobot/installation) if one of these tools is missing.
+
+## Create an isolated environment
+
+Clone the official LeRobot repository and check out the tested revision.
+
+```bash
+git clone https://github.com/huggingface/lerobot.git
+cd lerobot
+git checkout 30da8e687a6dfc617fcd94afc367ac7071c376ce
+
+uv venv --python 3.12 .venv
+source .venv/bin/activate
+uv pip install -e ".[core_scripts,training,feetech,smolvla]"
+```
+
+The extras install the robot command-line tools, training dependencies, Feetech motor support, and SmolVLA dependencies.
+
+## Verify LeRobot and CUDA
+
+Print the core versions and confirm that PyTorch can see CUDA:
+
+```bash
+python -c "import platform, torch; from importlib.metadata import version; print({'python': platform.python_version(), 'lerobot': version('lerobot'), 'torch': torch.__version__, 'cuda': torch.cuda.is_available()})"
+```
+
+The output from the DGX Spark should be similar to:
+
+```output
+{'python': '3.12.3', 'lerobot': '0.6.0', 'torch': '2.11.0+cu128', 'cuda': True}
+```
+
+Verify the commands used later:
+
+```bash
+for cli in lerobot-calibrate lerobot-find-cameras lerobot-find-port \
+           lerobot-record lerobot-rollout lerobot-teleoperate lerobot-train; do
+    command -v "$cli" > /dev/null || exit 1
+done
+```
+
+The loop exits without output when all commands are available.
+
+## Authenticate with Hugging Face
+
+Sign into Hugging Face interactively:
+
+```bash
+hf auth login
+hf auth whoami
+```
+
+Follow the browser or terminal prompt.
+
+## What you've accomplished
+
+You now have a Python environment with CUDA, LeRobot's SO-101 tools, and SmolVLA dependencies. Keep this terminal active. Next, connect the two arms and cameras to the DGX Spark and identify their device paths.

content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/3-prepare-so101.md

@@ -0,0 +1,139 @@
+---
+title: Connect the SO-101 and cameras
+description: Connect and identify the SO-101 leader, follower, gripper camera, and workspace camera for LeRobot.
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Connect the SO-101 hardware
+
+Connect the SO-101 leader, SO-101 follower, gripper camera, and workspace camera to the DGX Spark over USB.
+
+The connections are:
+
+```text
+SO-101 leader ------- USB ---\
+SO-101 follower ----- USB ----+-- DGX Spark
+Gripper camera ------ USB ----|   (LeRobot host)
+Workspace camera ---- USB ---/
+```
+
+Mount the gripper camera on the follower for a close view of grasping. Keep the workspace camera fixed and frame the follower, pickup area, rack, and full range of arm movement.
+
+{{% notice Note %}}
+On Ubuntu, the robot controller appears as a serial device. Membership in the `dialout` group lets your user account open that device without running every LeRobot command with `sudo`. Add your account to the group:
+
+```bash
+sudo usermod -aG dialout "$USER"
+```
+
+Log out and back in. Return to the cloned LeRobot directory, reactivate the environment, and verify group membership:
+
+```bash
+source .venv/bin/activate
+id -nG | tr ' ' '\n' | grep '^dialout$'
+```
+
+The expected output is:
+
+```output
+dialout
+```
+{{% /notice %}}
+
+Hardware commands in this and later pages use environment variables named `ROBOT_PORT`, `LEADER_PORT`, `GRIPPER_CAMERA_ID`, and `WORKSPACE_CAMERA_ID`. Set them to the device paths reported by LeRobot's discovery tools.
+
+## Identify the robot arm ports
+
+Run the port finder:
+
+```bash
+lerobot-find-port
+```
+
+The output lists all serial ports on the host. The relevant lines are similar to:
+
+```output
+Finding all available ports for the MotorsBus.
+Ports before disconnecting: [..., '/dev/ttyACM0', '/dev/ttyACM1']
+Remove the USB cable from your MotorsBus and press Enter when done.
+The port of this MotorsBus is '/dev/ttyACM0'
+Reconnect the USB cable.
+```
+
+Device paths can differ on your system. Identify the two roles in this order:
+
+1. On the first run, disconnect only the follower USB data cable. The detected path is the value for `ROBOT_PORT`. Reconnect the follower and wait for the device to reappear.
+2. Run `lerobot-find-port` again and disconnect only the leader USB data cable. The detected path is the value for `LEADER_PORT`. Reconnect the leader before continuing.
+
+Save the complete device paths in environment variables for the current terminal. For example, if the two discovery runs reported `/dev/ttyACM0` for the follower and `/dev/ttyACM1` for the leader, run:
+
+```bash
+export ROBOT_PORT="/dev/ttyACM0"
+export LEADER_PORT="/dev/ttyACM1"
+```
+
+## Discover the cameras
+
+{{% notice Note %}}
+Close browsers, video-conferencing software, and desktop camera applications before discovery so OpenCV can open both camera streams.
+{{% /notice %}}
+
+Discover the OpenCV streams and save a diagnostic image from each camera:
+
+```bash
+lerobot-find-cameras opencv --output-dir outputs/captured_images
+```
+
+An abbreviated output looks like:
+
+```output
+--- Detected Cameras ---
+Camera #0:
+  Name: OpenCV Camera @ /dev/video0
+  Type: OpenCV
+  Id: /dev/video0
+  Backend api: V4L2
+  Default stream profile:
+    ...
+    Fourcc: YUYV
+    Width: 640
+    Height: 480
+    Fps: 30.0
+--------------------
+Camera #1:
+  Name: OpenCV Camera @ /dev/video2
+  Type: OpenCV
+  Id: /dev/video2
+  Backend api: V4L2
+  Default stream profile:
+    ...
+    Fourcc: YUYV
+    Width: 640
+    Height: 480
+    Fps: 30.0
+--------------------
+```
+
+Open the images in `outputs/captured_images/` and match each `Id` to its view.
+
+| Gripper camera view | Workspace camera view |
+| --- | --- |
+| ![LeRobot camera-discovery frame showing the orange gripper tips and pickup surface. Use this view for the `gripper_cam` role.#center](images/3-lerobot-gripper-camera.png "Gripper camera role") | ![LeRobot camera-discovery frame showing the follower, vial, yellow rack, and full range of arm movement. Use this view for the `workspace_cam` role.#center](images/3-lerobot-workspace-camera.png "Workspace camera role") |
+
+Camera numbers and device paths can differ on your host. Use each camera's `Id` value, not `Camera #0` or `Camera #1`. Assign the camera showing the gripper to `GRIPPER_CAMERA_ID`, and assign the camera covering the complete task to `WORKSPACE_CAMERA_ID`.
+
+The tested setup used the following mapping, but your device paths might differ:
+
+```bash
+export GRIPPER_CAMERA_ID="/dev/video0"
+export WORKSPACE_CAMERA_ID="/dev/video2"
+```
+
+Device paths can change after reconnection, so don't assume that a previous device path still identifies the same arm or camera.
+
+## What you've accomplished
+
+You connected and identified both SO-101 arms and assigned the gripper and workspace camera roles. Keep the hardware connected and the camera mounts fixed. Next, calibrate both arms and verify teleoperation before recording demonstrations.

content/learning-paths/laptops-and-desktops/finetune-smolvla-lerobot/4-calibrate-teleoperate-so101.md

@@ -0,0 +1,78 @@
+---
+title: Calibrate and teleoperate the SO-101
+description: Calibrate the SO-101 leader and follower, then verify teleoperation before recording demonstrations.
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Calibrate the leader and follower
+
+Continue in the terminal where you set `ROBOT_PORT`, `LEADER_PORT`, `GRIPPER_CAMERA_ID`, and `WORKSPACE_CAMERA_ID`. If you opened a new terminal or reconnected a USB device, repeat discovery and export the current device paths before calibration.
+
+Calibration maps each joint's encoder readings to its usable motion range so LeRobot can reproduce the leader's movements on the follower consistently. Start each arm with its joints near the middle of their usable ranges, then move each requested joint slowly through its safe range. Support the arm and stop before reaching a mechanical limit.
+
+LeRobot 0.6.0 asks you to sweep shoulder pan, shoulder lift, elbow flex, wrist flex, and gripper. It assigns the complete encoder range to `wrist_roll`, so that joint isn't included in the recorded sweep.
+
+Calibrate the leader first:
+
+```bash
+lerobot-calibrate \
+  --teleop.type=so101_leader \
+  --teleop.port="$LEADER_PORT" \
+  --teleop.id=smolvla_leader
+```
+
+Place the leader in its middle-range pose before pressing Enter. Each joint should have room to move in both directions. The image shows the starting pose used in this setup.
+
+![Leader SO-101 supported in the middle-range starting pose before calibration begins. Its shoulder, elbow, wrist, and gripper are positioned away from their motion limits.#center](images/4-lerobot-leader-calibration.jpg "Leader calibration starting pose")
+
+After pressing Enter, support the leader and move each requested joint through its full safe range. The following animation is sped up to show the complete sequence. Perform the movements slowly on your own hardware.
+
+![Sped-up leader calibration showing the operator supporting the SO-101 and moving the shoulder, elbow, wrist, and gripper through their usable ranges one joint at a time.#center](images/4-lerobot-leader-calibration.gif "Leader joint-range calibration")
+
+Calibrate the follower with the same process:
+
+```bash
+lerobot-calibrate \
+  --robot.type=so101_follower \
+  --robot.port="$ROBOT_PORT" \
+  --robot.id=smolvla_follower
+```
+
+Place the follower in the middle-range pose shown in the image, then press Enter and perform the same safe joint sweep used for the leader.
+
+![Follower SO-101 supported in the middle-range starting pose before calibration begins. The shoulder, elbow, wrist, and gripper all have room to move in both directions.#center](images/4-lerobot-follower-calibration.png "Follower calibration starting pose")
+
+If LeRobot finds an existing calibration, follow the prompt to reuse it or press `c` to recalibrate. Here, `c` belongs to the calibration prompt; it isn't a dataset-recording control.
+
+## Verify teleoperation
+
+Teleoperation uses the leader arm to control the follower in real time.
+
+Place both arms in similar stable poses and clear the follower workspace. Run a 60-second test:
+
+```bash
+lerobot-teleoperate \
+  --robot.type=so101_follower \
+  --robot.port="$ROBOT_PORT" \
+  --robot.id=smolvla_follower \
+  --robot.cameras="{gripper_cam: {type: opencv, index_or_path: $GRIPPER_CAMERA_ID, width: 640, height: 480, fps: 30}, workspace_cam: {type: opencv, index_or_path: $WORKSPACE_CAMERA_ID, width: 640, height: 480, fps: 30}}" \
+  --teleop.type=so101_leader \
+  --teleop.port="$LEADER_PORT" \
+  --teleop.id=smolvla_leader \
+  --fps=30 \
+  --teleop_time_s=60 \
+  --display_data=false
+```
+
+Move one joint at a time at first. Confirm that the follower mirrors each movement correctly and that its gripper opens and closes. Stop the test if the follower moves unexpectedly.
+
+The animation shows the follower mirroring the leader’s arm pose and gripper movement.
+
+![SO-101 teleoperation showing the operator moving the leader while the follower mirrors its arm and gripper motion across the workspace.#center](images/4-lerobot-teleoperation.gif "Leader-to-follower teleoperation")
+
+## What you've accomplished
+
+You calibrated both arms and confirmed that the follower mirrors the leader during teleoperation. Next, record the pick-and-place demonstrations.