Benchmarking Initiative

Enabling comparative research


Current practice in publishing research results in robotics makes it extremely difficult not only to compare results of different approaches, but also to assess the quality of the research presented by the authors.

The long-term benefits of systematic benchmarking practice will not only foster the overall quality of research results but will also improve publication opportunities and lead to more rapid adoption of new research results by application developers and the robotics industry.

[ More on the rationale ... ]

Successful benchmarks


A benchmark can only be considered successful if the target community accepts it and uses it extensively in publications, conferences and reports as a way of measuring and comparing results. The most successful benchmarks existing today are probably those used in robot competitions.

[ More on measuring success ... ]

Current status

  1. An exhaustive survey of existing efforts in comparative research and a discussion of the lessons learned, is approximately completed. The extensive website produced includes information on relevant workshops, robot competitions and benchmarking initiatives.
  2. Various meetings have been organised to increase awareness of the initiative and to encourage experts in the different subfields to get involved in the process. This is still on-going.
  3. A set of four core robotics subareas in which it is possible to define benchmarks has been identified, and the process of developing benchmarks begun. The four core areas are Manipulation and Grasping, Motion Planning , Networked Robotics, and Visual Servoing.
[ More on current work ... ]

The future


So far it is too early to design new successful benchmarks, but this is an on-going effort involving more and more researchers with the aim of ensuring wide acceptance in the community at large. The generation of the benchmark surveys will no doubt focus attention on comparative research and it is hoped that on-going work will result in the design of comprehensive benchmarks both in terms of methodology and of standard data sets that will become well-used references for robotics research.

 

Getting Involved


We hope that you will get involved in many of EURON's activities and encourage others to do so as well.


Attend Meetings


Attending meetings is an excellent way of making contacts and friends within the community. The Annual Meetings are the major instrument for this purpose.


Communicate


A lot of things are happening all the time in the European robotics community: conferences and workshops are being organised, projects are starting, job offers are announced, etc. The EURON mailing list is the place-to-be for all your communications around these topics.
Webmaster :  Last update :  Thursday 28 August, 2008 (Technical Details) © Copyright EURON 2009
 Graphic design :  Maibritt Popp Stuckert Jørgensen Structural design :  Bridget Hallam