MATTERS

MATlab Toolkit for Evaluation of information Retrieval Systems

Find Out More

MATTERS is a software toolkit written in Matlab for information retrieval evaluation.

Potential users of MATTERS are researchers and developers of new information retrieval systems and components, organizers of evaluation activities and researchers on new evaluation methodologies and measures.

It can be used both to evaluate the effectiveness of information retrieval systems in different settings and tasks and to conduct meta-analises on evaluation measures and evaluation tasks.

Main Features

TREC Format

It is complaint with the TREC-format and thus runs and pools from CLEF, FIRE, NTCIR, and TREC can be straightforwardly imported by the toolkit.

Measures

It implements widely-used measures (for binary and graded qrels) such as Average Precision, (D)CG, bpref, Q-Measure, RBP.

Analysis

It calculates descriptive statistics of a measure set and implements methods for measure evaluation (discriminative power, pool downsampling).

Plots

It produces ready-to-use plots such as the discriminative power at different pool samples or the correlation between a measure on the pool and on its samples.

Get the software!

You can browse the software at 

http://ims-svn.dei.unipd.it/repos/matters/
.

Username: guest - Password: guest

You can check it out using Subversion 

$ svn checkout --username guest --password guest 
 http://ims-svn.dei.unipd.it/repos/matters/ matters

Documentation

Consult the API

Papers using MATTERS

If you use this toolkit in a paper please cite it as:
MATTERS (http://www.matters.dei.unipd.it/) is developed and maintained
by N. Ferro and G. Silvello, University of Padua, Italy.

Please let us know if you used MATTERS in one of your papers so that we can add your contribution to the following list:

Paper Data
Ferro, N., and Silvello, S. A General Linear Mixed Models Approach to Study System Component Effects. In 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016). In press. ACM Press, New USA, 2016
Ferro, N., Silvello, G., Keskustalo, H., Pirkola, A., and Järvelin, K. The Twist Measure for IR Evaluation: Taking User’s Effort into Account. Journal of the Association for Information Science and Technology (JASIST). John Wiley & Sons. vol. 67, num. 3, pp. 620-648, March 2016.
Angelini, M., Ferro, N., Santucci, G. and Silvello, S. A Visual Analytics Approach for What-If Analysis of Information Retrieval Systems. In 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016). In press. ACM Press, New USA, 2016.
Di Nunzio, G. M., and Silvello, S. A Graphical View of Distance Between Rankings: The Point and Area Measures. In Proceedings of the 6th Italian Information Retrieval Workshop, IIR 2015. CEUR Workshop Proceedings, Volume 1404, 2015.
Ferro, N. and Silvello, G. Rank-Biased Precision Reloaded: Reproducibility and Generalization. In N. Fuhr, A. Rauber, G. Kazai and A. Hanbury, eds. Proc. of the 37th European Conference on Information Retrieval (ECIR 2015), Lecture Notes in Computer Science (LNCS) 9022, pp. 768-780. Springer International Publishing Switzerland, 2015.
Ferrante, M., Ferro, N., and Maistro, M. Injecting User Models and Time into Precision via Markov Chains. In Geva, S., Trotman, A., Bruza, P., Clarke, C. L. A., and Järvelin, K., editors, Proc. 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014). ACM Press, New USA.
Ferrante, M., Ferro, N., and Maistro, M. Rethinking How to Extend Average Precision to Graded Relevance. Information Access Evaluation meets Multilinguality, Multimodality, and Interaction (CLEF 2014). 15 - 18 September 2014, Sheffield - UK, pp. 19-30. In Lecture Notes in Computer Science 8685, Springer International Publishing Switzerland.
Ferro, N., and Silvello, G. CLEF 15th Birthday: What can we Learn From Ad Hoc Retrieval?. Information Access Evaluation meets Multilinguality, Multimodality, and Interaction (CLEF 2014). 15 - 18 September 2014, Sheffield - UK, pp. 32-44. In Lecture Notes in Computer Science 8685, Springer International Publishing Switzerland.

Credits and License

MATTERS is developed and maintained by:
Nicola Ferro and Gianmaria Silvello
Information Management Systems Research Group,
Department of Information Engineering, University of Padua.

MATTERS is released under Apache License, Version 2.0