Benchmarking Initiative

Plan of Work

Reaching consensus does take time: proposing good benchmarks for the community to accept is a long process that requires agreement between many people in many subcommunities within robotics. Consequently the most appropriate role of EURON in this context does not seem to be that of itself defining benchmarks but rather of supporting grass-roots efforts by making different robotics sub-communities aware of the problem, encouraging them to make efforts to define benchmarks in their areas, following their work, and helping in disseminating their proposals so that they become publicly known. EURON can also help by disseminating information about other efforts in the field, organizing meetings and discussions, etc.

Since this work is being carried out in the framework of "Beyond Robotics", we have been most concerned with non-industrial scenarios. This report builds on previous work developed in EURON I [Dillamnn 04] in which benchmarks in industry were discussed. In general this seems to be a different situation, since industry can provide the resources to measure whatever features they desire in a robot. In this sense they can not only develop their own benchmarks, but also they have even organized competitions: a famous example is the one held in March 1996, when Ford U.S.A. organized a competition for an order of 400 welding robots with the result that the KUKA robots could solve the benchmark problems considerably faster and smoother than the robots of the main competitor. KUKA won this contract and since then all Ford European plants become equipped with KUKA robots exclusively [Ford Competition].

Service robots integrate a very large quantity of cutting-edge technologies. Integration and maintenance of such systems is therefore a key challenge that has to be addressed for successful R&D in this field. In this context there have been some still immature proposals regarding the installation of some EU Robotics Centers that will integrate and maintain the most sophisticated platforms and offer them for experimentation to the European robotics research community. A simple way of ensuring similar conditions for comparing research could be that these centers would offer as a service to the community the possibility of using their platforms for benchmarking. However this possibility arises some controversy and requires further elaboration.

In order to attain the above-mentioned goals, namely successful robotics benchmarks in the medium term, the following actions have been identified:

1. Exhaustive survey: The first step is to make accessible to the community the state of the art in existing efforts in comparative research such as competitions, benchmarks, and challenges, followed by a discussion phase about lessons learned.
2. Increase awareness by organizing and participating in meetings, discussions, workshops, ... and encourage experts in the different subfields to get involved in the process of benchmark definition
3. Identify a set of core robotics subareas, in which benchmarks could be defined. Initially these sub-areas are selected by taking advantage of existing Special Interest Groups in EURON-I, later on other areas will be selected as suggested by the research roadmap, the launched IPs, and the research KA committee. These benchmarks should be operational and described in sufficient detail, as well as their associated metrics, and an independent measurement procedure, keeping always in mind that the fundamental goal is that they are widely accepted in the community at large.

More concretely, we planned a series of discussions and refinements in some steps, similar to the procedure described in [Burke et al. 04] as a continuous process of convergence towards consensus in order to ensure community wide acceptance:

1. Identification of some core areas
2. Establish working groups (WG): one per core area, plus one per IP
3. Discussion phase: on-line and physical within WGs
4. First EURON Workshop on benchmarks at European event followed by analysis and synthesis phase
5. First consensus on detailed benchmarks, their metrics and measurement procedure
6. Dissemination through a report and the web site
7. Discussion phase: online and physical open to the community at large
8. Workshop on benchmarks at international event followed by analysis and synthesis phase.

The results of action (a) above is an exhaustive, detailed survey and inventory of current existing efforts in comparative research. This survey focuses on competitions benchmarks and challenges such as RoboCup, FIRA, Cleaning Robot Contest, DARPA Grand Challenge, etc. This has been the result of a long process of information gathering, either obtained from different sources or kindly provided by a number of persons. Most of this information was previously unavailable, scattered in different sources and merged with irrelevant issues. It has now been gathered together and is available to the community from the website http://www.robot.uji.es/benchmark Detailed descriptions of the rules, metrics and procedures for current robot competitions and benchmarks have included in this website for its interest for future benchmark development.

Actions (b) and (c) have been partially addressed with many meetings and discussions. Progress has varied in the different groups, but in some of them, such as motion planning and visual servoing, some web-based initiatives have already been launched.