Benchmarking Initiative

Successful Benchmarks

Defining a benchmark — even a sound valid benchmark — could be an easy task, if it is just taken as an academic exercise. Defining a successful benchmark is something completely different. A benchmark can be considered as successful if the community extensively uses it in publications, conferences and reports as a way of measuring and comparing results. To put it in a few words: a benchmark is successful if and only if it is widely accepted by the community at which it is targeted.

This kind of success is somehow difficult to predict but some the following considerations may help:

• its success is related to its quality, i.e. it really must serve its purpose
• benchmarks should focus on particular subdomains, such as visual servoing, grasping, motion planning, ...
• it is easier for a benchmark to be successful within a scientific (sub) community if it arises from the community itself instead of having it defined outside this community.