Good Experimental Methodology (GEM)Coordinator: Fabio BonsignorioWe have created a google group for members. It will be reachable from the coming GEM SIG web site. Some 80/90 members of Euron are currently backing this initiative. If you are working on Good Experimental Methodology and Benchmarking in Robotics and/or related topics and wish to contribute to the SIG, then do join us. All science proceeds from experiment, which motivates the creation of new theory and establishes the limits and validity of the existing theoretical basis. Individual branches of science conduct experiments differently, depending on the topic of investigation, but all have in common a body of knowledge concerning experimental methodology that specifies how to design and conduct `good' experiments in that discipline. If robotics aims to be serious science, serious attention must be paid to experimental method. We can all point to published papers that report poor experimental work: few or no replications or tests; no, or dubious, comparison between algorithms; lack of suitable quantification of performance and its variability; conclusions which, while perhaps correct, are unjustified by the reported experimental work. It is time to address this. This Special Interest Group focusses on increasing the quality of experimental methodology practiced in robotics. We believe this general aim can be achieved, for instance, by sharing good practice via educational workshops, summer schools, email discussion and web presentation; by providing assistance to journal and conference reviewers and editors concerning what constitutes experimental robotics and good practice in that sub-discipline; by encouraging the principled replication and comparison of results; and by encouraging the development and use of appropriate systems benchmarks and standard evaluation procedures. This SIG has a small but active Board who will maintain momentum and attempt to engage the large, interested, but busy community in improving the quality of our experimental methodology. We see benchmarking as an important part of good experimental method and will encourage that activity within the SIG. Recommended reading: Aeronautical Design Standard Performance Specification on Handling Qualities Requirements For Military Rotorcraft. This document provides guidelines on the what and how of performance assessment of helicopters. Even though our robots demonstrate very different tasks in a multiplicity of environments, the above document could serve as a source of inspiration to write together a living online document specifying a graded series of tasks and how to evaluate them. A roboticist would then report which task numbers were achieved with his/her robot and what levels of performance were attained. If helicopter builders can do it and if, to some extent, computer vision researchers can do it, then we can do it too. SIG Deliverables:Concretely, we suggest forming aSpecial Interest Group with the particular aim of promoting good experimental science in robotics. Such a group could have a significant effect. We shall: D1) make a document enumerating recommended quality criteria for Robotics Journal/Conference reviews, including a one page quickguide summary, that the community expects a proper high-quality experimental robotics paper have to satisfy. This could be formulated as a reviewer checklist and circulated to Journal Editors and Conference Programme Committees. We could also use the criteria when reviewing papers ourselves. For instance, EURON might adopt of these rules in EUROS08. D2) Develop a SIG Web site as part of the EURON web facility that includes information on planned meetings and SIG activities, educational material on good experimental practices, recommendations and outcomes from SIG/community discussion, and a compendium of other efforts to promote good experimentation and quantitative methods in robotics, both in the EU and internationally. D3) Organize/support workshops on GEM and Benchmarking at major robotics conferences We will organize 8 or more open meetings preferably coinciding with the major robotics conferences. Other anticipated outcomes of the SIG will be:
Joining this GEM SIGMembership is free of charge and is open to everyone working in the area worldwide. However, funding, when applicable, is reserved for Euron members. To apply, please send the following information to one of the contact people given below:
|
Getting InvolvedWe hope that you will get involved in many of EURON's activities and encourage others to do so as well. Attend MeetingsAttending meetings is an excellent way of making contacts and friends within the community. The Annual Meetings are the major instrument for this purpose. CommunicateA lot of things are happening all the time in the European robotics community: conferences and workshops are being organised, projects are starting, job offers are announced, etc. The EURON mailing list is the place-to-be for all your communications around these topics. |
|||||||||||||||||||
| Webmaster : | webmaster AT euron.org | Last update : | Thursday 04 September, 2008 | (Technical Details) | © Copyright EURON 2009 |
| Graphic design : | Maibritt Popp Stuckert Jørgensen | Structural design : | Bridget Hallam | ||
