Benchmarking Initiative

Rationale

The current practice of publishing research results in robotics makes it extremely difficult not only to compare results from different approaches, but also to assess the quality of the research presented by the authors. Though for pure theoretical articles this may not be the case, typically when researchers claim that their particular algorithm or system is capable of achieving some performance, those claims are intrinsically unverifiable, either because it is their unique system or just because a lack of experimental details, including working hypothesis. Often papers published in robotics journals and generally considered as good would not meet the minimum requirements in domains in which good practice calls for the inclusion of a detailed section describing the materials and experimental methods that support the authors' claims. This is, of course, partly due to the very nature of robotics research: reported results are tested by solving a limited set of specific examples on different types of scenarios, using different underlying software libraries, incompatible problem representations, and implemented by different people using different hardware, including computers, sensors, arms, grippers...

This state of affairs cannot be changed in the short term, but some steps can be taken in the right direction by studying the ways in which research results in robotics can be assessed and compared. In this context EURON can play an important role. The long-term benefits of systematic benchmarking practice are evident: not only will it foster the overall quality of research results but it will also improve publication opportunities for EU-based research, thereby increasing international visibility of European research and lead to rapid adoption of new research results by application developers and the robotics industry.

Curiously enough, some of the most popular organized events in robotics are related to comparative research: the different successful robot competitions that exist today are a way of comparing the performance of the competing systems by means of very well-defined rules and metrics. The organization of these scientific competitions has proven a quick way to attract substantial research efforts and rapid to produce high-quality working solutions.

What can possibly be the role of EURON in relation to these issues? Given the limited available resources, trying to set up a task force to define a set of gold standards in robotics by itself does not seem the right approach. Proposing grand challenges as long-term goals could be interesting, but grand challenges are usually described only vaguely and so are not very useful for measuring progress or comparing results. Nevertheless, benchmarks could be conceived as stepping stones towards a grand challenge.