A learned policy on the MJAK-UMI robot gripper

Each cell shows diffusion policy success first and behaviour cloning second, over 50 evaluation episodes. Below that is how many of the 60 demonstration attempts survived each system’s data pipeline. Rows shaded warm are ours.

The clearest gap is the 82 mm box. No gripper with an 80 mm stroke can even demonstrate it so those policies never get data. On the other four tasks the systems are close and most differences sit inside the intervals. SLAM based pipelines lose about four in ten demonstrations which costs them training data. Our clean encoder width labels help the SLAM version of MJAK-UMI hold up regardless.

We also trained image based diffusion policies on a GPU. Each one sees a 96 by 96 wrist camera view with its own system’s fingers. With 20 demonstration attempts and 3000 training steps per cell none of them learned the task for any system including ours. So this run only proves the pipeline works end to end. A real comparison needs about 200 demonstrations and 50 thousand training steps per cell and is scheduled for the next GPU session.

The benchmark policies on film

These clips are the trained policies from the table above running in closed loop. We recorded the same demonstrations, trained the same diffusion policy and filmed fresh test seeds. Nothing in them is scripted: every motion comes from the policy. Each clip shows its result at the end of every attempt.

These benchmark policies get the object position directly, as described in the method section. The camera based policy is the one in section 3.

We run UMI’s own ORB-SLAM3 fork, compiled from source with the same code as their public container, and feed it MJAK-UMI data through mjakumi slam. The only change we tested is a one line patch for plain scenes, described below and kept in the repository. The camera model is UMI’s published GoPro Max Lens calibration at 960 by 720 and 60 fps. The camera to IMU transform comes straight from our CAD and the self mask is rendered from the CAD with the jaws fully open and fully closed. Every sequence runs twice: once with a GoPro class IMU at 200 Hz as UMI uses and once with the ISM330DHCX we fit at 833 Hz. Each run repeats three times because ORB-SLAM3 is not deterministic.

What we found

On textured scenes every one of the 48 runs initialised and tracked about 97% of frames. With the IMU noise model taken straight from the datasheet our sensor came out slightly worse than the GoPro class IMU at 11.9 mm against 9.6 mm. That is a settings problem, not a hardware one: ORB-SLAM3 trusted the datasheet numbers too much. With the noise model inflated five times, which is standard practice, our IMU reaches 10.0 mm, the lowest rotation error of any setup at 0.056 deg and the lowest drift per second. So on position the two sensors are level and on rotation and drift ours is slightly ahead.

Plain scenes are where UMI is known to fail and our benchmark reproduces it: none of the 12 runs initialised with either IMU. We measured why. A plain scene gives about 250 corners per frame instead of 1250 and only about 90 survive a geometric check between frames. ORB-SLAM3 needs to triangulate a first map from two views within a short window and never gets enough good matches to do it. We tried three fixes on the same footage. Raising the feature count to 3000 with lower corner thresholds did not help. Putting UMI’s own 16 cm ArUco init tag in the starting view got 1 of 12 runs started and that track was poor at 82 mm. Widening the initialisation window from 0.3 s to 1 s in a patched build did not help either. The honest conclusion is that camera based SLAM cannot start on blank surfaces whatever IMU is fitted. The fix is a pose source that does not need texture. MJAK-UMI carries a mast for a Vive Ultimate tracker for exactly this case. UMI has no such option.

ATE is the RMS position error of the camera after aligning the track to ground truth. SE3 alignment fits no scale, which is the fair test for a visual inertial system. UMI reports 6.1 mm and 3.5 deg for its own pipeline against motion capture on real hardware. Simulated video has no rolling shutter or motion blur so these figures are a floor. Real hardware numbers come next.

Full video: media/videos/mjak_umi_slam.mp4

This version of the report is about one question: does data recorded with the MJAK-UMI handheld train a camera based policy that works on the MJAK-UMI robot gripper? The first attempt placed the cube in 0% of trials. After three fixes it places it in 80% of 30 trials with no scripted motion anywhere. Every move in the clips below comes from the trained network looking through the wrist fisheye camera.

The task and the setup

We recorded 150 handheld demonstrations of picking a 40 mm cube and placing it in a bin, each through the production path: device log, fisheye video, LED sync code and pose. The objects move up to 4 cm between episodes. That gave 11,143 training windows. The policy is a diffusion policy with a ResNet18 image encoder and a conditional 1D UNet, the recipe UMI uses. It sees the last two wrist images and the last two gripper poses and widths, and predicts the next 16 poses and widths relative to where the gripper is now. It then runs on the robot gripper in MuJoCo at 10 Hz on test layouts it has never seen.

What was wrong and how we found it

• fixedEvery demonstration starts and ends with about a second of holding still while the LED flashes its sync code. That taught the policy to wait. We now cut the still frames at both ends before training.
• fixedThe diffusion sampler did not clip its estimate of the clean action. In the first denoising steps that estimate is divided by a number close to zero so it exploded. Clipping it as the standard implementation does took the error on training frames from 84 mm to 7 mm with the same weights.
• fixedOn the robot the controller stored a new observation only once per chunk of 8 actions so the previous frame the policy saw was 0.8 s old instead of the 0.1 s it was trained on. It now records one every 0.1 s.
• fixedCommitting to 12 actions per policy call instead of 8 or 4 stopped the policy from switching plans halfway. The sweep below shows the effect.

The sweep runs used 12 trials each to choose the settings. The highlighted bars are separate 30 trial runs on the chosen setting, so the headline number does not come from the runs used to pick it.

The learned policy on film

Where it still fails

It missed 6 of 30 layouts: 9016, 9017, 9022, 9024, 9028, 9029. The same layouts fail at every checkpoint so this is not bad luck. In each one the gripper flies to the bin first with empty open jaws and waits there. That pose is the last thing every demonstration shows so the policy believes the job is done. More demonstrations with the cube near the bin, a short look around before moving and a progress signal in the observation are the next things to try.

The handheld firmware compiles cleanly with PlatformIO for the Seeed XIAO ESP32 S3. It uses 8.8% of RAM (28,740 bytes) and 10.8% of flash (362,521 bytes), which leaves plenty of room. The build took 85 seconds on the evaluation machine.

Both fingers get the same 30 N per jaw. The gripper lifts a 66 mm can 15 cm and shakes it sideways at about 1 g. The object’s friction governs the contact and the arm is stiff so the finger is the only variable.

The two fingers hold the same mass. Finger shape does not change payload. Grip force does and that is where our robot gripper has the edge: it is rated at 30 N per jaw with a 45 N current limit ceiling and the drivetrain is checked for it below.

We grasped 250 random objects per variant between 10 and 80 mm wide and 20 to 600 g in mass with friction from 0.4 to 1.0 and up to 8 mm of placement error. Each was lifted and shaken at 1 g. The robot variant held 99.6% and the handheld held 99.6%. The encoder read the object width within 0.77 mm on average.

Every printed tooth is checked with the Lewis bending equation at the rated load. The release gate fails if any safety factor drops below 2 at rated force or 1.2 at the current limit.

drive layer m1.25 z12 face 9.5 mm; jaw layer m1.25 z16 face 9.0 mm

handheld rated: 18.8 N per jaw
  OK  jaw pinion                4.8 MPa  SF 10.4 (need 2.0)
  OK  jaw rack                  3.5 MPa  SF 14.4 (need 2.0)
  OK  drive pinion             13.3 MPa  SF  3.8 (need 2.0)
  OK  drive rack (slider)       8.8 MPa  SF  5.7 (need 2.0)
  OK  servo gear               13.3 MPa  SF  3.8 (need 2.0)

robot rated: 30.0 N per jaw
  OK  jaw pinion                7.7 MPa  SF  6.5 (need 2.0)
  OK  jaw rack                  5.6 MPa  SF  9.0 (need 2.0)
  OK  drive pinion             21.3 MPa  SF  2.4 (need 2.0)
  OK  drive rack (slider)      14.0 MPa  SF  3.6 (need 2.0)
  OK  servo gear               21.3 MPa  SF  2.4 (need 2.0)

robot current limit ceiling: 45.0 N per jaw
  OK  jaw pinion               11.6 MPa  SF  4.3 (need 1.2)
  OK  jaw rack                  8.3 MPa  SF  6.0 (need 1.2)
  OK  drive pinion             31.9 MPa  SF  1.6 (need 1.2)
  OK  drive rack (slider)      21.1 MPa  SF  2.4 (need 1.2)
  OK  servo gear               31.9 MPa  SF  1.6 (need 1.2)

handheld: friction limited static payload ~3.1 kg (mu 0.8), 1.5 kg with 1 g dynamic margin

robot: friction limited static payload ~4.9 kg (mu 0.8), 2.4 kg with 1 g dynamic margin

trigger: 50 N at the spring hard stop, slider travel 32.2 mm
PASS

Rendered from the released CAD meshes. Every part flies into place, the jaws run the full 0 to 86 mm stroke and the robot gripper picks and places objects in MuJoCo. Video: media/videos/mjak_umi_assembly.mp4

One jaw core serves both units. A single MGN9 rail carries two jaws driven by one pinion so they always move together. A 12 bit magnetic encoder on the pinion shaft reads the opening to 0.03 mm. The handheld adds a spring loaded trigger that measures grip force. The robot unit swaps the trigger for a Feetech STS3215 servo and an ISO 9409 flange. Both carry the camera at the same point so the robot sees what the handheld recorded.

Every system uses the same simulated jaw, the same 30 N robot side grip force, a force controlled gripper executor and the same tasks seeds and 10 Hz relative pose actions that UMI style policies use. Only the published hardware differences change: finger shape, stroke, how gripper width is measured during collection, how much pose error the tracker adds to the labels and how many demonstrations survive processing. For UMI we use its real soft finger STL from the MIT licensed actuated UMI release.

Sources are the UMI, FastUMI and exUMI papers. The ArUco noise and dropout rates are our own estimates and are written down in sim/compare/systems.py. Fingers are simulated as rigid shapes with soft contact so Fin Ray bending is not modelled.

• fixedThe printed drive gear ran at a safety factor of 1.2 under rated grip. It is now module 1.25 with a 9.5 mm face and every tooth is at 2.4 or better.
• fixedThe finger was 75 mm against UMI’s 123 mm. It is now a 100 mm Fin Ray with a 24 mm deep base and 2 mm walls.
• fixedNine assembly problems in the CAD including rail screws into an untapped rail, a camera foot too thin for its screws, unreachable shell screws and a flange too thin under the M6 heads.
• fixedThe camera saddle hid the sync LED and the finger bases from the lens. The simulator caught it.
• fixedThe firmware now compiles for the XIAO ESP32 S3 and uses under 11% of flash and 9% of RAM.
• fixedWith datasheet noise our IMU tracked slightly worse than a GoPro class IMU. A tuned noise model brings it level on position and ahead on rotation and drift.
• fixedA training and robot deployment stack now exists in learning/ and runs end to end on a GPU.
• fixedThe camera based policy trained on handheld data now places the cube in 80% of 30 robot trials in simulation.
• open6 of 30 test layouts still fail the same way: the gripper goes to the bin empty and waits.
• openCamera SLAM cannot start on plain surfaces with any IMU. Use the tracker mast there.
• openPose accuracy on real hardware against motion capture.
• openReal task success rates and the usable data fraction on printed units.
• openFinger adapters for Robotiq, WSG 50 and xArm grippers, fingertip tactile sensing and two handed collection.

Everything needed to print and build a pair lives in the repository. 25 print ready STL files sit in cad/out/stl with STEP files for each part and four coloured STEP assemblies. There is a drawing sheet per part, a bill of materials, a print plan for the K1, assembly and QC procedures, firmware for the ESP32 S3 and the mjakumi host software. No file is released unless the interference sweep, the watertight STL check and the strength check all pass.