Zoekmachines - Final Project
Organized by uva_zoekmachines - Current server time: May 3, 2025, 12:26 a.m. UTC
First phase
Milestone 01 Retrieval and EvaluationSept. 20, 2024, midnight UTC
End
Competition EndsOct. 31, 2024, midnight UTC
Zoekmachines Final Project (2024)
Overall Goal
This project aims to let students get a better understanding of information retrieval systems by implementing an information system that can return relevant passages given users’ queries.
Task
Given a query from a user and a collection of passages, the information retrieval system conducts ranking and returns a top 100 ranked list of passages.
Leaderboard designed by Zahra Abbasiantaeb. Questions? z.abbasiantaeb@uva.nl
Evaluation
Your submissions would be evaluated using the following metrics:
- MRR@10 (validation set)
- MRR@10 (test set)
- P@10 (test set)
- nDCG@10 (test set)
- Recall@100 (test set)
The submission that do not include the result for all test and train queries will get score -1.
Rules
Submissions for each Milestone must be made before the end of each corresponding phase. You may submit 10 submissions every day and 100 in total per phase.
Milestone 01 Retrieval and Evaluation
Start: Sept. 20, 2024, midnight
Description: The focus of this milestone is on ranking models. In this milestone, you will learn how to use ElasticSearch (ES) to implement an inverted index of the collection. You will learn how to use and implement different ranking models and how to evaluate the performance of your ranking model. In addition, you will do an in-depth analysis of the effect of different parameters of the ranking models and text pre-processing techniques. For evaluation, you will learn to work with the 'pytrec_eval' library.
Milestone 02 Re-ranking
Start: Oct. 15, 2024, midnight
Description: This is the last milestone where you will focus on improving your results even better. Now your pipeline is complete, and it’s time to investigate where you can reach better improvement. There are various options you can explore, such as: - working with various lexical and semantic features and train a learning-to-rank model; - testing different sets of features for your learning-to-rank model; - tuning the hyper-parameters of your learning to rank model; - test the performance of your model with respect to different metrics. At the end of this milestone, you will make the final submission.
Competition Ends
Oct. 31, 2024, midnight
You must be logged in to participate in competitions.
Sign In