MS MARCO Submission Instructions
Question Answering
We have made the official evaluation script along with a sample output file on the dev set available on github. [ evaluation script] The evaluation script takes as inputs a reference and candidate output file. You can execute the evaluation script to evaluate your models as follows:
./run.sh <path to reference json file> <path to candidate json file>
To evaluate with on NLGen task you will have to process your files to be in the same format as the Q&A task. There is a script in the github that does this for you.
Passage Reranking
We have made the official evaluation script along with a sample output file on the dev set available for download as well so that you can evaluate your models. [evaluation script] The evaluation script takes as inputs a reference relevance labels(qrels) and candidate output file. You can execute the evaluation script to evaluate your models as follows:
python3 <reference qrels file> <path to candidate tsv file>
Submitting Your Model for the Challenge
Once you have built a model that meets your expectations on evaluation with the dev set, you can submit your test results to get official evaluation on the test set. To ensure the integrity of the official test results, we do not release the correct answers for test set to the public. To submit your model for official evaluation on the test set, follow the below steps:
- Run the evaluation script on the test/dev set and generate the output results file for submission.
- Submit the following information by
- Individual/Team Name: Name of the individual or the team to appear in the leaderboard [Required]
- Individual/Team Institution: Name of the institution of the individual or the team to appear in the leaderboard [Optional]
- Model code: Training code for the model [Recommended]
- Model information: Name of the model/technique to appear in the leaderboard [Required]
- Paper Information: Name, Citation, URL of the paper if model is from a published work to appear in the leaderboard [Optional]
To avoid "P-hacking" we discourage too many submissions from the same group in a short period of time. Because submissions don't require the final trained model we also retain the right to request a model to validate the results being submitted
