Benchmarking In-Hand Manipulation

The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability.

arxiv/PDF

Benchmark Template

Protocol Template

Cite this work using the following Bib:

@Article{cruciani-ral2020-benchmarking-in-hand,
author = {Silvia Cruciani* and Balakumar Sundaralingam* and Kaiyu Hang and Vikash Kumar and Tucker Hermans and Danica Kragic},
title = {{Benchmarking In-Hand Manipulation (*equal contribution)}},
journal = {IEEE Robotics and Automation Letters (Special Issue: Benchmarking Protocols for Robotic Manipulation)},
year = {2020},
url = "https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation"}

Benchmark Dataset

Software

Benchmark Results

The initial, goal and reached states are available in this repo for comparison.

Real world

Level Method Group / Researchers Robot Results
I Relaxed-Rigidity University of Utah LL4MA Lab / B. Sundaralingam & T. Hermans Allegro hand PDF

Simulation/Planning

Level Method Group / Researchers Robot Results
III Dexterous Manipulation Graphs KTH Royal Institute of Technology / S. Cruciani, C. Smith, D. Kragic & Yale University / K. Hang YuMi PDF