~~NOTOC~~
====== Benchmarking In-Hand Manipulation ======
The purpose of this benchmark is to evaluate the planning and control aspects
of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. 
Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states for various in-hand manipulation tasks. 
We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability.

[[https://arxiv.org/abs/2001.03070|arxiv/PDF]]

{{ :project:bih_benchmark.pdf | Benchmark Template}}

{{ :project:bih_protocol.pdf | Protocol Template}}

Cite this work using the following Bib: 

  @Article{cruciani-ral2020-benchmarking-in-hand,
  author = {Silvia Cruciani* and Balakumar Sundaralingam* and Kaiyu Hang and Vikash Kumar and Tucker Hermans and Danica Kragic},
  title = {{Benchmarking In-Hand Manipulation (*equal contribution)}},
  journal = {IEEE Robotics and Automation Letters (Special Issue: Benchmarking Protocols for Robotic Manipulation)},
  year = {2020},
  url = "https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation"}
==== Benchmark Dataset ====
  * Hand poses: [[https://github.com/balakumar-s/bih_dataset|bih_dataset]]
  * YCB object meshes: [[http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/|ycb_dataset]]
  * Initial & Goal Contact regions for YCB objects: [[https://drive.google.com/drive/folders/1RO-bakB1seXpv_XpRiXmp95AsRjgKfmE?usp=sharing| Gdrive]]

==== Software ====
  * To compute the mesh error metrics: [[https://bitbucket.org/robot-learning/ll4ma_collision_wrapper|ll4ma_collision_wrapper]]
  * To get box plot: [[https://bitbucket.org/robot-learning/statistical_plot_scripts/src/master/|statistical_plot_scripts]]
===== Benchmark Results =====
The initial, goal and reached states are available in this [[https://github.com/balakumar-s/bih_dataset/tree/master/results|repo]] for comparison.
=== Real world ===
^ Level ^ Method ^ Group / Researchers ^ Robot ^ Results ^
| I | Relaxed-Rigidity |University of Utah LL4MA Lab / B. Sundaralingam & T. Hermans | Allegro hand | {{ :project:bih_demo_lvl1.pdf |PDF}} |
=== Simulation/Planning ===
^ Level ^ Method ^ Group / Researchers ^ Robot ^ Results ^ 
| III | Dexterous Manipulation Graphs | KTH Royal Institute of Technology / S. Cruciani, C. Smith, D. Kragic & Yale University /  K. Hang  | YuMi | {{ :project:bih_demo_lvl3.pdf |PDF}} |