LeRobot SO-101 Dual-Arm Platform - Generative Design & RL Simulation Case Study
Development of a lightweight dual-arm robotic system featuring generative design in Fusion360 and Jetson Nano integration, with NVIDIA Isaac Sim for RL training planned as a future workstream.
Project Overview
The Challenge: Building a capable dual-arm robotic platform from the ground up involves significant mechanical design complexity, control challenges, and the need for simulated environments to train reinforcement learning policies without risking hardware damage. Off-the-shelf robotic arms are either prohibitively expensive or lack the lightweight, optimized structure needed for high-speed teleoperation.
The Goal: Design and build a lightweight dual-arm manipulation platform capable of low-latency teleoperation with real-time visual feedback, then integrate physics-accurate simulation to train RL policies for diverse manipulation tasks.
The Outcome: A fully operational LeRobot SO-101 platform with a custom generative-designed arm structure and Jetson Nano-powered control stack. Isaac Sim simulation and RL training are planned as the next major workstream.
Technical Approach
This project was executed in two primary workstreams: mechanical design and fabrication, and embedded systems and control. A third workstream for simulation and reinforcement learning is planned for future development.
Development Philosophy
- Weight-Optimized: Every gram of the arm structure was justified through generative design analysis
- Latency-Critical: All components were selected and configured with end-to-end latency as a primary constraint
- Iterative CAD: Mechanical design went through 14 iterations, each informed by FEA results and assembly testing
- Simulation-First (planned): Isaac Sim will be used to validate and train policies once the URDF model is finalized
Key Modifications
Generative-Designed Dual-Arm Structure in Fusion360
Why This Upgrade
Traditional robotic arms are over-engineered for hobbyist and research applications, carrying unnecessary mass that limits speed and increases motor torque requirements. Generative design in Fusion360 allows the algorithm to propose organic, weight-optimized structures that meet specified load and stiffness constraints while minimizing material usage.
Implementation Challenge
Defining accurate boundary conditions, load cases, and manufacturing constraints required deep understanding of both FEA and the physical assembly. The generative solver ran for 6-8 hours per iteration on a workstation, exploring thousands of possible geometries. Each result needed manual cleanup and validation before export to STEP for manufacturing.
Measured Impact
- Arm mass reduced from an estimated 650g (conventional design) to 420g (35% reduction)
- First natural frequency increased from 18Hz to 32Hz (78% improvement)
- Peak stress within safe limits at 150% of expected operating loads
- Print time per arm: 22 hours on modified Ender 3 with Klipper in PETG
Technical Details
- Software: Autodesk Fusion360 with Generative Design extension
- Material: PETG
- Constraints: Max deflection <0.5mm under 2kg end-effector load, safety factor >2.5
- Manufacturing: FDM printed on modified Ender 3 with Klipper with 0.4mm nozzle, 0.12mm layer height
Learning: Generative design is not a “set and forget” tool. The quality of output depends entirely on how well you define the problem space. Spend 80% of the time on boundary conditions and load cases.
LeRobot SO-101 Platform Assembly & Jetson Nano Integration
Why This Upgrade
The LeRobot SO-101 provides an open-source, standardized dual-arm platform designed for imitation learning and teleoperation research. Pairing it with a Jetson Nano provides the computational horsepower needed for real-time pose estimation, control loop execution, and visual feedback processing at the edge.
Implementation Challenge
The Jetson Nano ran the standard LeRobot Python SDK for motor control and teleoperation, with USB cameras providing visual feedback. The teleoperation latency budget was managed through standard USB polling and the LeRobot communication protocol.
Measured Impact
- End-to-end teleoperation latency: 12-18ms (well under 50ms target)
- Real-time visual feedback at 30 FPS with 640x480 resolution
- Motor control loop running at 60Hz via standard LeRobot Python SDK
- Stable operation for 4+ hours continuous teleoperation sessions
Technical Details
- Hardware: LeRobot SO-101 dual-arm kit, NVIDIA Jetson Nano (4GB), Raspberry Pi Camera Module v2
- Software: Ubuntu 20.04 L4T, LeRobot Python SDK
- Communication: USB for teleoperation master device and motor control via LeRobot SDK
- Control Rate: 60Hz teleoperation and motor control loop
- Power: 5V/4A dedicated supply for Jetson, 12V/5A for servo motors
Troubleshooting Note: Initial USB latency issues were traced to the Jetson’s USB controller being in low-power mode. Forcing USB to “on” mode in the device tree configuration resolved intermittent 50-80ms latency spikes.
NVIDIA Isaac Sim & Isaac Lab for Physics Simulation (In Progress)
Status: In Progress — implementation is pending a redesigned URDF model for the custom arm geometry. The section below outlines the planned approach based on published community benchmarks and research.
Planned Approach
- Complete redesigned URDF model for the custom arm structure
- Import URDF into Isaac Sim and validate the digital twin
- Configure Isaac Lab for PPO-based training across multiple task environments
- Apply domain randomization for sim-to-real transfer
Expected Performance (based on community benchmarks from similar LeRobot SO-101 setups)
- 12+ synthetic environments for task training (pick-and-place, peg-in-hole, assembly, etc.)
- 45-60 FPS simulation speed on RTX 4070-class GPU
- ~87% policy transfer success rate with proper domain randomization
- Training time approximately 12 hours in simulation vs ~200 hours real-world
Technical Details (planned)
- Software: NVIDIA Isaac Sim, Isaac Lab (Orbit framework), PyTorch
- Hardware: Desktop PC with RTX 5060ti 16GB, 64GB RAM, Intel i7-12700KF
- RL Algorithm: Proximal Policy Optimization (PPO) with domain randomization
Real-Time Visual Feedback & Teleoperation Pipeline
Why This Upgrade
Effective teleoperation requires the operator to see what the robot sees with minimal delay. A camera feed integrated with the teleoperation interface enables precise remote manipulation.
Implementation Challenge
Setting up the camera feed was straightforward using the standard LeRobot teleoperation script, which handles video capture and display out of the box. A basic USB camera was used for testing.
Measured Impact
- Single USB camera providing real-time visual feedback through the LeRobot teleoperation interface
- Stable operation with no additional configuration beyond the standard library defaults
Technical Details
- Camera: USB camera (Amazon)
- Software: LeRobot teleoperation script (standard library)
- Interface: Built-in LeRobot teleoperation viewer
Key Learnings & Takeaways
What Worked Well
- Generative design for robotics: The weight savings directly translated to higher allowable speeds and reduced motor strain
- Latency measurement as a first-class concern: Instrumenting every stage of the pipeline revealed bottlenecks that would have been invisible otherwise
- LeRobot SDK simplicity: The standard Python SDK made integration straightforward with minimal custom code
What I’d Do Differently
- More compute for planned simulation work: A higher-end GPU will allow more parallel environments and faster training
- Better thermal management for Jetson: Under sustained load, the Nano throttles after ~45 minutes; active cooling is essential
- Standardized connector system: Using different connectors for cameras vs. motors caused confusion during assembly
Transferable Skills Gained
- Generative design workflow for structural optimization
- Real-time control system design and latency analysis
- Planned physics simulation pipeline and RL training workflow
- Sim-to-real transfer techniques and domain randomization (research phase)
- Embedded systems integration (Jetson Nano, camera, LeRobot SDK)
Conclusion
This project delivered a lightweight dual-arm manipulation platform based on the LeRobot SO-101 architecture. Generative-designed mechanical components were fabricated using FDM printing on a modified Ender 3, and the Jetson Nano provides real-time control via the standard LeRobot Python SDK. Isaac Sim simulation and RL training are planned as the next major workstream following a redesigned URDF model.
Most Impactful Workstreams (in order):
- Generative design optimization - Enabled the lightweight arm structure
- Jetson Nano integration - Provided the compute needed for real-time control
The total investment was $291.57 in parts ($231.87 + $59.70, the latter including an extra servo for a planned tool changer) and ~180 hours of work across mechanical design and embedded development.