Potential users of MATTERS are researchers and developers of new information retrieval systems and components, organizers of evaluation activities and researchers on new evaluation methodologies and measures.
It can be used both to evaluate the effectiveness of information retrieval systems in different settings and tasks and to conduct meta-analises on evaluation measures and evaluation tasks.
It implements widely-used measures (for binary and graded qrels) such as Average Precision, (D)CG, bpref, Q-Measure, RBP.
It calculates descriptive statistics of a measure set and implements methods for measure evaluation (discriminative power, pool downsampling).
It produces ready-to-use plots such as the discriminative power at different pool samples or the correlation between a measure on the pool and on its samples.
You can browse the software at
Username: guest - Password: guest
You can check it out using Subversion
$ svn checkout --username guest --password guest
If you use this toolkit in a paper please cite it as:
MATTERS (http://www.matters.dei.unipd.it/) is developed and maintained
by N. Ferro and G. Silvello, University of Padua, Italy.
Please let us know if you used MATTERS in one of your papers so that we can add your contribution to the following list:
|Ferro, N., and Silvello, S. A General Linear Mixed Models Approach to Study System Component Effects. In 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016). In press. ACM Press, New USA, 2016|
|Ferro, N., Silvello, G., Keskustalo, H., Pirkola, A., and Järvelin, K. The Twist Measure for IR Evaluation: Taking User’s Effort into Account. Journal of the Association for Information Science and Technology (JASIST). John Wiley & Sons. vol. 67, num. 3, pp. 620-648, March 2016.|
|Angelini, M., Ferro, N., Santucci, G. and Silvello, S. A Visual Analytics Approach for What-If Analysis of Information Retrieval Systems. In 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016). In press. ACM Press, New USA, 2016.|
|Di Nunzio, G. M., and Silvello, S. A Graphical View of Distance Between Rankings: The Point and Area Measures. In Proceedings of the 6th Italian Information Retrieval Workshop, IIR 2015. CEUR Workshop Proceedings, Volume 1404, 2015.|
|Ferro, N. and Silvello, G. Rank-Biased Precision Reloaded: Reproducibility and Generalization. In N. Fuhr, A. Rauber, G. Kazai and A. Hanbury, eds. Proc. of the 37th European Conference on Information Retrieval (ECIR 2015), Lecture Notes in Computer Science (LNCS) 9022, pp. 768-780. Springer International Publishing Switzerland, 2015.|
|Ferrante, M., Ferro, N., and Maistro, M. Injecting User Models and Time into Precision via Markov Chains. In Geva, S., Trotman, A., Bruza, P., Clarke, C. L. A., and Järvelin, K., editors, Proc. 37th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2014). ACM Press, New USA.|
|Ferrante, M., Ferro, N., and Maistro, M. Rethinking How to Extend Average Precision to Graded Relevance. Information Access Evaluation meets Multilinguality, Multimodality, and Interaction (CLEF 2014). 15 - 18 September 2014, Sheffield - UK, pp. 19-30. In Lecture Notes in Computer Science 8685, Springer International Publishing Switzerland.|
|Ferro, N., and Silvello, G. CLEF 15th Birthday: What can we Learn From Ad Hoc Retrieval?. Information Access Evaluation meets Multilinguality, Multimodality, and Interaction (CLEF 2014). 15 - 18 September 2014, Sheffield - UK, pp. 32-44. In Lecture Notes in Computer Science 8685, Springer International Publishing Switzerland.|