Start: Sept. 19, 2022, midnight
Description: For this milestone, we ask each team to implement the inverted index in memory and adapt the current Starting Kit to work with their implemented inverted index. Furthermore, we ask each group to implement additional metrics such as P@K, nDCG@K, and MAP and report the performance of their model on the validation set in terms of these metrics.
Start: Oct. 1, 2022, midnight
Description: In this milestone, we ask you to pre-process the documents (e.g., stemming, stopping, etc.) and rebuild your inverted index. First, we want to see the effect of pre-processing on the current ranking method and implement more advanced lexical ranking algorithms such as TF-IDF, BM25, QL, etc. Each team should implement at least two new ranking methods and compare the results with each other. Also, experiments on the effect of model parameters is required.
Start: Oct. 9, 2022, midnight
Description: For this milestone, you would take the ranking of the previous milestone and re-rank for better performance. To do this, there are various options to choose from. Teams can focus on better feature selection/engineering and use the RankNet implementation in the starting kit. Furthermore, teams can investigate the performance of more advanced learning-to-rank models such as ListNet and LambdaMart. Also, you can test the model’s performance with different values of hyper-parameters.
Start: Oct. 17, 2022, midnight
Description: This is the last milestone where you will focus on improving your results even better. Now your pipeline is complete, and it’s time to investigate where you can reach better improvement. There are various options you can explore, such as: - testing different pre-processing techniques and checking the result; - tuning the hyper-parameters of your lexical rankers; - testing different sets of features for your learning to rank model; - tuning the hyper-parameters of your learning to rank model; - test the performance of your model with respect to different metrics. At the end of this milestone, you will make the final submission.
Oct. 31, 2022, 1 a.m.
You must be logged in to participate in competitions.Sign In