Benchmarking Initiative

Enabling comparative research


Current practice in publishing research results in robotics makes it extremely difficult not only to compare results of different approaches, but also to assess the quality of the research presented by the authors.

The long-term benefits of systematic benchmarking practice will not only foster the overall quality of research results but will also improve publication opportunities and lead to more rapid adoption of new research results by application developers and the robotics industry.

[ More on the rationale ... ]

Successful benchmarks


A benchmark can only be considered successful if the target community accepts it and uses it extensively in publications, conferences and reports as a way of measuring and comparing results. The most successful benchmarks existing today are probably those used in robot competitions.

[ More on measuring success ... ]

Current status

  1. An exhaustive survey of existing efforts in comparative research and a discussion of the lessons learned, is approximately completed. The extensive website produced includes information on relevant workshops, robot competitions and benchmarking initiatives.
  2. Various meetings have been organised to increase awareness of the initiative and to encourage experts in the different subfields to get involved in the process. This is still on-going.
  3. A set of four core robotics subareas in which it is possible to define benchmarks has been identified, and the process of developing benchmarks begun. The four core areas are Manipulation and Grasping, Motion Planning , Networked Robotics, and Visual Servoing.
[ More on current work ... ]

The future


So far it is too early to design new successful benchmarks, but this is an on-going effort involving more and more researchers with the aim of ensuring wide acceptance in the community at large. The generation of the benchmark surveys will no doubt focus attention on comparative research and it is hoped that on-going work will result in the design of comprehensive benchmarks both in terms of methodology and of standard data sets that will become well-used references for robotics research.

Getting Involved


We hope that you will get involved in many of EURON's activities and encourage others to do so as well.

Attend Meetings


Attending meetings is an excellent way of making contacts and friends within the community. These contacts are normally interesting, and may also be useful!

EURON Schools are aimed at PhD students so are particularly good for people wishing to deepen an already existing interest. [ More details  ... ]

Contribute to the Initiatives


Your knowledge and resources are valuable — please consider sharing them with others by adding items to the databases, contributing your opinions to the roadmap and benchmarking initiatives, and your information to this website.

Especially, please contribute to the Video Collection for Students asap.

[ More details  ... ]

Use the Resources


Make use of the resources that EURON offers. Much of the information is freely available to all, but for other resources you have to be a EURON member. Apply for the grants, use the databases and the contacts, apply for the awards, use the mailing lists.

Webmaster :  Last update :  Thursday 28 August, 2008 (Technical Details) © Copyright EURON 2008
 Graphic design :  Maibritt Popp Stuckert Jørgensen Structural design :  Bridget Hallam