MS MARCO Submission Instructions

Getting Started

It is easy to get started using the dataset and participate in the challenge using the resources that we have built for you.

Download the train and dev sets in order to train your model and iterate on it as you desire.

We have made the official evaluation script along with a sample output file on the dev set available for download as well so that you can evaluate your models. [Download the evaluation scripts] The evaluation script takes as inputs a reference and candidate output file. You can execute the evaluation script to evaluate your models as follows:

./run.sh <path to reference json file> <path to candidate json file>

Submitting Your Model for the Challenge

Once you have built a model that meets your expectations on evaluation with the dev set, you can submit your test results to get official evaluation on the test set. To ensure the integrity of the official test results, we do not release the correct answers for test set to the public. To submit your model for official evaluation on the test set, follow the below steps:

Run the evaluation script on the test set and generate the output results file for submission
Submit the following information by

Individual/Team Name: Name of the individual or the team to appear in the leaderboard [Required]
Individual/Team Institution: Name of the institution of the individual or the team to appear in the leaderboard [Optional]
Model information: Name of the model/technique to appear in the leaderboard [Required]
Paper Information: Name, Citation, URL of the paper if model is from a published work to appear in the leaderboard [Optional]

To avoid "P-hacking" we discourage too many submissions from the same group in a short period of time. Because submissions don't require the final trained model we also retain the right to request a model to validate the results being submitted