The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability.

arxiv/PDF

Benchmark Template

Protocol Template

Cite this work using the following Bib:

@Article{cruciani-ral2020-benchmarking-in-hand,
author = {Silvia Cruciani* and Balakumar Sundaralingam* and Kaiyu Hang and Vikash Kumar and Tucker Hermans and Danica Kragic},
title = {{Benchmarking In-Hand Manipulation (*equal contribution)}},
journal = {IEEE Robotics and Automation Letters (Special Issue: Benchmarking Protocols for Robotic Manipulation)},
year = {2020},
url = "https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation"}

• Hand poses: bih_dataset
• YCB object meshes: ycb_dataset
• Initial & Goal Contact regions for YCB objects: Gdrive

• To compute the mesh error metrics: ll4ma_collision_wrapper
• To get box plot: statistical_plot_scripts

The initial, goal and reached states are available in this repo for comparison.