Conference and Labs of the Evaluation Forum

The Conference and Labs of the Evaluation Forum (formerly Cross-Language Evaluation Forum), or CLEF, is an organization promoting research in multilingual information access (currently focusing on European languages). Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards. The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the US, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.

Conference and Labs of the Evaluation Forum
Unlocking Information Access
Abbreviation	CLEF
Discipline	information retrieval
Publication details
Publisher	Springer and CEUR Workshop Proceedings
History	2000 (2000) (formerly known as Cross-Language Evaluation Forum from 2000 to 2009)
Frequency	annual
Website	www.clef-initiative.eu

CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book on the lessons learned in 20 years of evaluation activities.

Structure of CLEF

Before 2010, CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs.

Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas.,

In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images. In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.

List of CLEF workshops and conferences

Year	Location	Note	Proceedings	Editors
2000	Lisbon	Constituting workshop	Cross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000)	Carol Ann Peters
2001	Darmstadt		Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross—Language Evaluation Forum (CLEF 2001) Revised Papers	Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2002	Rome		Advances in Cross-Language Information Retrieval: Third Workshop of the Cross—Language Evaluation Forum (CLEF 2002) Revised Papers	Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2003	Trondheim		Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross—Language Evaluation Forum (CLEF 2003) Revised Selected Papers	Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2004	Bath		Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross—Language Evaluation Forum (CLEF 2004) Revised Selected Papers	Carol Ann Peters and Paul Clough and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini
2005	Vienna		Accessing Multilingual Information Repositories: Sixth Workshop of the Cross—Language Evaluation Forum (CLEF 2005). Revised Selected Papers	Carol Ann Peters and Fred C Gey and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini and Henning Müller and Maarten de Rijke
2006	Alicante		Evaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross—Language Evaluation Forum (CLEF 2006). Revised Selected Papers	Carol Ann Peters and Clough, P. and Fred C Gey and Jussi Karlgren and Bernardo Magnini and Doug Oard and Maarten de Rijke and M Stempfhuber
2007	Budapest		Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross—Language Evaluation Forum (CLEF 2007). Revised Selected Papers	Carol Ann Peters and Jijkoun, V. and Mandl, T. and Henning Müller and Oard, D. W. and Peñas, A. and Petras, V. and Santos, D.
2008	Århus		Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross—Language Evaluation Forum (CLEF 2008). Revised Selected Papers	Carol Ann Peters and Deselaers, T. and Nicola Ferro and Julio Gonzalo and Gareth J F Jones and Kurimo, M. and Mandl, T. and Peñas, A.
2009	Corfu		Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments—Tenth Workshop of the Cross—Language Evaluation Forum (CLEF 2009). Revised Selected Papers	Carol Ann Peters and Di Nunzio, G. M. and Kurimo, M. and Mandl, T. and Mostefa, D. and Peñas, A. and Roda, G.
2010	Padua	First conference	Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010	Maristella Agosti and Nicola Ferro and Carol Ann Peters and Maarten de Rijke and Alan Smeaton
2011	Amsterdam		Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011)	Forner, P. and Julio Gonzalo and Kekäläinen, J. and Lalmas, M. and Maarten de Rijke
2012	Rome		Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012)	Catarci, T. and Forner, P. and Hiemstra, D. and Peñas, A. and Santucci, G.
2013	Valencia		Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013)	Forner, P. and Henning Müller and Paredes, R. and Rosso, P. and Stein, B.
2014	Sheffield		Information Access Evaluation—Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014)	Kanoulas, E. and Lupu, M. and Clough, P. and Sanderson, M. and Hall, M. and Hanbury, A. and Toms, E.
2015	Toulouse		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015)	Mothe, J. and Savoy, J. and Jaap Kamps and Pinel-Sauvagnat, K. and Gareth J F Jones and SanJuan, E. and Cappellato, L. and Nicola Ferro
2016	Evora		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016)	Fuhr, N. and Quaresma, P. and Goncalves, T. and Larsen, B. and Balog, K. and Macdonald, C. and Cappellato, L. and Nicola Ferro
2017	Dublin		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017)	Gareth J F Jones and Lawless, S. and Julio Gonzalo and Kelly, L. and Lorraine Goeuriot and Thomas Mandl and Cappellato, L. and Nicola Ferro
2018	Avignone		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018)	Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian-Yun Nie, Laure Soulier, Eric SanJuan, Linda Cappellato, and Nicola Ferro
2019	Lugano		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019)	Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David Losada, Gundula Heinatz, Linda Cappellato, and Nicola Ferro
2020	Thessaloniki (virtual due to COVID-19)		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020)	Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, A., Linda Cappellato, and Nicola Ferro
2021	Bucharest (virtual due to COVID-19)		Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021)	K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro

External links

• CLEF homepage
• CLEF history before 2010