Not logged in.
Quick Search - Contribution
Contribution Details
| Type | Conference or Workshop Paper |
| Scope | Discipline-based scholarship |
| Published in Proceedings | Yes |
| Title | CHEVAL: Chur Evaluation Laboratory |
| Organization Unit | |
| Authors |
|
| Presentation Type | paper |
| Item Subtype | Original Work |
| Refereed | Yes |
| Status | Published in final form |
| Language |
|
| ISBN | 978-1905305544 |
| Page Range | 393 - 400 |
| Event Title | European Conference on Information Management and Evaluation (ECIME) |
| Event Type | conference |
| Event Location | Montpellier |
| Event Start Date | September 20 - 2007 |
| Event End Date | September 21 - 2007 |
| Place of Publication | Montpellier |
| Publisher | Academic Conferences Ltd |
| Abstract Text | Incorporating novel approaches like visual components, semantic web ideas or Web 2.0 concepts into information retrieval systems poses new challenges for their systematic evaluation. Currently, the development of valid evaluation settings cannot keep up with the development of new search engines and innovative information retrieval concepts. Therefore, the Swiss Institute for Information Research (SII), is currently developing a testbed called CHEVAL (Chur Evaluation Laboratory) to tackle this problem. The vision of CHEVAL is to design an integrated, multi-level and multi-methodological web-based system and framework to support different kinds of evaluation types (e.g. usability tests, IR efficiency measurement, benchmarking studies etc.) of several types of information retrieval systems. In the context of CHEVAL, an evaluation can have multiple dimensions regarding the type of the evaluation (long-term or short-term test phase, comparative or non-comparative evaluation, field or laboratory test environment) and the methods used for the evaluation, which can either be from IR efficiency measurement or usability testing as well as a combination of both. The paper will give an overview of some well-known and widely accepted evaluation initiatives. This also includes background information about the history of these initiatives. Furthermore the strengths and weaknesses of the described evaluation initiatives will be presented and discussed. Based on the deficiencies of current approaches for evaluating information retrieval systems with visual or semantic components the vision and the goals of the Chur Evaluation Laboratory will be explained. Following, the architecture of the testbed will be introduced. An example will illustrate how the system is intended to be used and what advantages CHEVAL will give to evaluators of information retrieval systems. Finally, the paper will present the success factors and a short roadmap for the further development of the Chur Evaluation Laboratory. |
| Export |
BibTeX
EP3 XML (ZORA)
|