MS MARCO DataSets

First released at NIPS 2016 the MS MARCO dataset was an ambitious, real-world Machine Reading Comprehension Dataset. Since then we have been slowly improving the existing QA datasets and releasing new datasets. How does your model perform?

1. Given a document select the 3 most salient keyphrases.

2. Given a session with 2-n queries with one query being masked, predict the masked query(Conversational Search).

3. Given a query and a corpus of 8.8m passages, rank the passages by relevance. Use either the full corpus or start with BM25s top 1000(Ranking)

4. Given a query and 10 passages provide the best answer avaible based(Q&A). The target answers in the dataset files are 'answers'

5. Given a query and 10 passages provide the best answer avaible in natural language that could be used by a smart device/digital assistant(Q&A + Natural Language Generation). The target answers in the dataset files are 'wellFormedAnswers'



KeyPhrase Extraction(10/18/2019) ranked by F1 @3 on Eval

Rank Model Submission Date Precision @1,@3,@5 Recall @1,@3,@5 F1 @1,@3,@5
1 BERT (Base) Sequence Tagging Baseline Si Sun (Tsinghua University), Chenyan Xiong (MSR AI), Zhiyuan Liu (Tsinghua University) [Code] November 5th, 2019 0.484, 0.312, 0.227 0.255, 0.469, 0.563 0.321, 0.361, 0.314
2 LLbeBack Rodrigo Nogueira (Epistemic AI), Jimmy Lin (University of Waterloo) November 19th, 2019 0.519, 0.297, 0.178 0.281, 0.428, 0.438 0.349, 0.341, 0.246
3 Baseline finetuned on Bing Queries MSMARCO Team [Xiong, et al. '19<] October 19th, 2019 0.397, 0.249, 0.149 0.215, 0.391, 0.391 0.267, 0.292, 0.209
4 Baseline MSMARCO Team [Xiong, et al. '19<]< October 19th, 2019 0.365, 0.237, 0.142 0.196, 0.367, 0.367 0.244, 0.277, 0.198

Passage Retrieval(10/26/2018-Present) ranked by MRR on Eval

Rank Model Ranking Style Submission Date MRR@10 On Eval MRR@10 On Dev
1 Enriched BERT base + AOA index + CAS Ming Yan of Alibaba Damo NLP Full Ranking August 20th, 2019 0.393 0.408
2 W-Index retrieval + BERT-F re-rank Zhuyun Dai of Carnegie Mellon University Full Ranking September 12th,2019 0.388 0.394
3 Enriched BERT base + AOA index V1 Ming Yan of Alibaba Damo NLP Full Ranking May 13th, 2019 0.383 0.397
4 BERTter pretraining (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) Full Ranking May 21st, 2019 0.383 0.395
5 Enriched BERT base + AOA index V2 Ming Yan of Alibaba Damo NLP Full Ranking May 13th, 2019 0.380 0.389
6 BM25 + monoBERT + duoBERT + TCP (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira, et al. '19] Full Ranking June 26th, 2019 0.379 0.390
7 BERT^2 (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) Full Ranking May 13th, 2019 0.375 0.386
8 Enriched BERT base + AOA index Ming Yan of Alibaba Damo NLP Full Ranking May 6th, 2019 0.373 0.387
9 BM25 + monoBERT + duoBERT (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira, et al. '19] Full Ranking June 26th, 2019 0.370 0.382
10 ReinforcedQGen+BERTRank Rajarshee Mitra of Microsoft STCI Full Ranking August 5th, 2019 0.369 -
11 BERTter Indexing (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira et al. '19] and [Code] Full Ranking April 8th, 2019 0.368 0.375
12 Enriched BERT base + AOA index Ming Yan of Alibaba Damo NLP ReRanking May 6th, 2019 0.368 0.373
13 BM25 + monoBERT (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira, et al. '19] Full Ranking June 26th, 2019 0.365 0.372
14 BERT base + attention ranking anonymous ReRanking August 26th, 2019 0.364 0.377
15 SAN + BERT base Yu Wang, Xiaodong Liu, Jianfeng Gao - Deep Learning Group, Microsoft Research AI [Xiaodong, et al. '18] ReRanking January 22th, 2019 0.359 0.370
16 BERT + Small Training Rodrigo Nogueira(1) and Kyunghyun Cho(2) - New York University(1,2), Facebook AI Research(2) [Nogueira, et al. '19] and [Code] ReRanking January 7th, 2019 0.359 0.365
17 BERT base + L2R Ming Yan of Alibaba Damo NLP ReRanking March 16th,2019 0.356 0.364
18 BERT + Projected Matching Yifan Qiao(1), Chenyan Xiong(2), Zhenghao Liu(3), Zhiyuan Lui(4) - Tsinghua University(1,3,4), Microsoft Research(2) [ Qiao et al. '19] ReRanking February 7th,2019 0.356 -
19 BERT base + attention ranking anonymous ReRanking March 1st, 2019 0.347 0.317
20 BERT + Small Training Xue-He Wang, Chia-Hung Yuan, Bing-Han Chiang, Dong-Ze Wu, Lu-Dan Ruan, Shan-Hung Wu of National Tsing Hua University ReRanking June 20th, 2019 0.347 0.361
21 BERT-base +ranking loss + horovod Milk&Cereal ReRanking May 6th, 2019 0.346 0.352
22 BERT-base fine-tune ICT-NLU ReRanking May 23rd, 2019 0.346 0.349
23 BERT base + attention ranking anonymous ReRanking March 11th, 2019 0.344 -
24 BERT base + attention ranking anonymous ReRanking March 4th, 2019 0.343 -
25 Bert-base + hinge ranking loss Milk&Cereral ReRanking April 24th, 2019 0.342 0.345
26 BERT + L2R ICT-NLU ReRanking June 11th, 2019 0.342 0.348
27 BERT+ENA Di Zhao, Hui Fang, UD Infolab ReRanking August 11th, 2019 0.339 -
28 BERT Base + Highway+Cross Entropy Loss + Axioms Di Zhao, Hui Fang, UD Infolab ReRanking August 11th, 2019 0.336 -
29 BERT Base + Highway + Cross Entropy Loss + Axioms Di Zhao, Hui Fang, UD Infolab ReRanking August 9th, 2019 0.336 0.340
30 BERT base + attention ranking anonymous ReRanking March 2nd, 2019 0.335 -
31 BERT + CNN Chia-Hung Yuan, Bing-Han Chiang, Xue-He Wang, Dong-Ze Wu, Lu-Dan Ruan, Shan-Hung Wu of National Tsing Hua University ReRanking June 15th, 2019 0.333 0.346
32 BERT + Multilayer Interaction Yifan Qiao(1), Chenyan Xiong(2), Zhenghao Liu(3), Zhiyuan Lui(4) - Tsinghua University(1,3,4), Microsoft Research(2) [ Qiao et al. '19] ReRanking February 19th,2019 0.329 0.311
33 BERT base + ranking Yifan Qiao(1), Chenyan Xiong(2), Zhenghao Liu(3), Zhiyuan Lui(4) - Tsinghua University(1,3,4), Microsoft Research(2) [ Qiao et al. '19] ReRanking February 8th, 2019 0.326 0.316
34 BERT Base + Highway + Ranking Loss Di Zhao, Hui Fang, UD Infolab ReRanking August 9th, 2019 0.323 -
35 FastText + Conv-KNRM (Ensemble) Sebastian Hofstätter (1), Navid Rekabsaz (2), Carsten Eickhoff (3), and Allan Hanbury (1) - TU Wien(1), Idiap Research Institute(2), Brown University(3) [ Hofstätter et al. '19] and [Code] ReRanking May 8th, 2019 0.309 0.318
36 IRNet (Deep CNN/IR Hybrid Network) Dave DeBarr, Navendu Jain, Robert Sim, Justin Wang, Nirupama Chandrasekaran – Microsoft ReRanking January 2nd, 2019 0.281 0.278
37 FastText + Conv-KNRM (Single)Sebastian Hofstätter (1), Navid Rekabsaz (2), Carsten Eickhoff (3), and Allan Hanbury (1) - TU Wien(1), Idiap Research Institute(2), Brown University(3) [ Hofstätter et al. '19] and [Code] ReRanking May 8th, 2019 0.277 0.290
38 docTTTTTquery Rodrigo Nogueira (Epistemic AI), Jimmy Lin (University of Waterloo) [Paper] and [Code] Full Ranking November 27th, 2019 0.272 0.277
39 Neural Kernel Match IR (Conv-KNRM) (Ensembled)(1)Yifan Qiao, (2)Chenyan Xiong, (3)Zhenghao Liu, (4)Zhiyuan Liu-Tsinghua University(1, 3, 4); Microsoft Research AI(2) [Dai et al. '18] ReRanking Novmeber 28th, 2018 0.271 0.290
40 Axiom-Regularized Conv-KNRM Corby Rosset, Bhaskar Mitra, Chenyan Xiong, Nick Craswell, Xia Song, Saurabh Tiwary - Microsoft AI & Research[Rosset et al. '19] ReRanking February 19, 2019 0.263 0.262
41 [Official Baseline] Duet V2 (Ensembled) Bhaskar Mitra, Fernando Diaz, Nick Craswell - Microsoft AI & Research [Mitra et al. '19] and [Code] ReRanking February 19, 2019 0.253 0.252
42 Duet with query term independence assumption (Single) Bhaskar Mitra (1, 2), Corby Rosset (1), David Hawking (1), Nick Craswell (1), Fernando Diaz (1), and Emine Yilmaz (2) of (1) Microsoft & (2) UCL Paper ReRanking March 14th, 2019 0.252 0.254
43 Neural Kernel Match IR (Conv-KNRM) (Single)(1)Yifan Qiao, (2)Chenyan Xiong, (3)Zhenghao Liu, (4)Zhiyuan Liu-Tsinghua University(1, 3, 4); Microsoft Research AI(2) [Dai et al. '18] ReRanking February 19, 2019 0.247 0.247
44 [Official Baseline] Duet V2 (Single) Bhaskar Mitra, Fernando Diaz, Nick Craswell - Microsoft AI & Research [Mitra et al. '19s] and [Code] ReRanking February 20, 2019 0.245 0.243
45 DW Index + BM25 anonymous Full Ranking April 29th, 2019 0.239 0.243
46 BERT Base + Highway + Cross Entropy Loss + Axioms Di Zhao, Hui Fang, UD Infolab ReRanking August 5th, 2019 0.223 0.340
47 BERT Base + Highway + Ranking Loss Di Zhao, Hui Fang, UD Infolab ReRanking August 5th, 2019 0.222 0.340
48 BM25 (Anserini) + doc2query (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira et al. '19] and [Code] Full Ranking April 10th, 2019 0.218 0.215
49 Neural Kernel Match IR (Conv-KNRM) (Ensembled)(1)Yifan Qiao, (2)Chenyan Xiong, (3)Zhenghao Liu, (4)Zhiyuan Liu-Tsinghua University(1, 3, 4); Microsoft Research AI(2) [Dai et al. '18] ReRanking Novmeber 26th, 2018 0.199 0.199
50 Neural Kernel Match IR (KNRM) ((1)Yifan Qiao, (2)Chenyan Xiong, (3)Zhenghao Liu, (4)Zhiyuan Liu-Tsinghua University(1, 3, 4); Microsoft Research AI(2) [ Xiong et al. '17] ReRanking December 10th, 2018 0.198 0.218
51 Feature-based LeToR: simple-feature based RankSVM(1)Yifan Qiao, (2)Chenyan Xiong, (3)Zhenghao Liu, (4)Zhiyuan Liu-Tsinghua University(1, 3, 4); Microsoft Research AI(2) ReRanking December 10th, 2018 0.191 0.195
52 BM25 (Lucene8, tuned) (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira, et al. '19] Full Ranking June 26th, 2019 0.190 0.187
53 BM25 (Anserini) (1)Rodrigo Nogueira, (2)Wei Yang, (3)Jimmy Lin, (4)Kyunghyun Cho - New York University(1,4), University of Waterloo(2,3), Facebook AI Research(4) [Nogueira et al. '19] and [Code] Full Ranking April 10th, 2019 0.186 0.184
54 Unnamed Hongyin Zhu ReRanking June 26th, 2019 0.174 -
55 [Official Baseline]BM25 Stephen E. Robertson; Steve Walker; Susan Jones; Micheline Hancock-Beaulieu & Mike Gatford (Implemented by MSMARCO Team) [ Robertson et al. '94] Full Ranking Novmeber 1st, 2018 0.165 0.167
56 BERT Represenatation Yifan Qiao(1), Chenyan Xiong(2), Zhenghao Liu(3), Zhiyuan Lui(4) - Tsinghua University(1,3,4), Microsoft Research(2) [Qiao et al. '19] ReRanking February 19th,2019 0.015 0.043

Q&A Task(03/01/2018-Present)

Rank Model Submission Date Rouge-L Bleu-1
1 Multi-doc Enriched BERT Ming Yan of Alibaba Damo NLP June 20th, 2019 0.540 0.565
2 Human Performance April 23th, 2018 0.539 0.485
3 BERT Encoded T-Net Y. Zhang, C. Wang, X.L. Chen August 5th, 2019 0.526 0.539
4 Selector+Combine-Content-Generator QA Model Shengjie Qian of Caiyun xiaoyi AI and BUPT March 19th, 2019 0.525 0.544
5 LM+Generator Alibaba Damo NLP November 25th,2019 0.522 0.516
6 Masque Q&A Style NTT Media Intelligence Laboratories [Nishida et al. '19] January 3rd, 2019 0.522 0.437
7 Deep Cascade QA Ming Yan of Alibaba Damo NLP [Yan et al. '18] December 12th, 2018 0.520 0.546
8 Unnamed anonymous December 9th,2019 0.518 0.507
9 PALM Alibaba Damo NLP December 9th,2019 0.518 0.507
10 VNET Baidu NLP [Wang et al. '18] November 8th, 2018 0.516 0.543
11 MultiLM QnA Model anonymous December 2nd, 2019 0.514 0.498
12 BERT Encoded T-NET Y. Zhang, C. Wang, X.L. Chen July 12th, 2019 0.506 0.525
13 MultiLM QnA Model anonymous December 5th, 2019 0.499 0.430
14 BERT+ Multi-Pointer-Generator Tongjun Li of the ColorfulClouds Tech and BUPT June 11th, 2019 0.498 0.525
15 Selector+Combine-Content-Generator NL Model Shengjie Qian of Caiyun xiaoyi AI and BUPT March 11th, 2019 0.496 0.535
16 CompLM Alibaba Damo NLP December 2nd, 2019 0.495 0.516
17 LM+Generator anonymous November 21st,2019 0.494 0.529
18 PALM Alibaba Damo NLP December 9th,2019 0.492 0.510
19 LNET S.L. Liu of the NEUKG Nov 19th, 2019 0.491 0.530
20 BERT+ Multi-Pointer-Generator Tongjun Li of the ColorfulClouds Tech and BUPT May 21st, 2019 0.491 0.520
21 CompLM Alibaba Damo NLP December 3rd, 2019 0.490 0.502
22 Masque NLGEN Style NTT Media Intelligence Laboratories [Nishida et al. '19] January 3rd, 2019 0.489 0.488
23 Communicating BERT Xuan Liang of RIDLL from the University of Technology Sydney October 4th, 2019 0.483 0.506
24 MultiLM NLGen Model anonymous December 2nd, 2019 0.482 0.514
25 LM+Generator anonymous November 19th,2019 0.478 0.481
26 MultiLM NLGen Model anonymous December 5th, 2019 0.475 0.479
27 BERT + Transfer anonymous October 16th, 2019 0.474 0.499
28 Bert Based Multi-taskZhangY & WangC June 26th, 2019 0.471 0.512
29 SNET + CES2S Bo Shao of SYSU University July 24th, 2018 0.450 0.464
30 ranking+nlg anonymous October 9th, 2019 0.449 0.468
31 ranker-reader RCZoo of UCAS May 15th, 2019 0.441 0.371
32 Extraction-net zlsh80826 October 20th, 2018 0.437 0.444
33 SNET JY Zhao August 30th, 2018 0.436 0.463
34 BIDAF+ELMo+SofterMax Wang Changbao November 16th, 2018 0.436 0.459
35 ranking+nlg anonymous August 12th, 2019 0.434 0.411
36 DNET QA Geeks August 1st, 2018 0.432 0.479
37 KIGN-QA Chenliang Li April 22nd, 2019 0.426 0.404
38 Reader-Writer Microsoft Business Applications Group AI Research September 16th, 2018 0.421 0.436
39 BERT+Multi-Loss S.L. Liu of NEUKG November 4th, 2019 0.413 0.422
40 SNET+seq2seq Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS June 1st, 2018 0.398 0.423
41 lightNLP+BiDAF Enliple AI February 1st, 2019 0.298 0.156
42 BIDAF+seq2seq Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS May 29th, 2018 0.276 0.288
43 BiDaF Baseline(Implemented By MSMARCO Team)
Allen Institute for AI & University of Washington [Seo et al. '16]
April 23th, 2018 0.240 0.106
44 TrioNLP + BiDAF Trio.AI of the CCNU September 23rd, 2018 0.205 0.232
45 BiDAF + LSTM Meefly January 15th,2019 0.153 0.120

Q&A + Natural Language Generation Task(03/01/2018-Present)

Rank Model Submission Date Rouge-L Bleu-1
1 Human Performance April 23th, 2018 0.632 0.530
2 Masque NLGEN Style NTT Media Intelligence Laboratories [Nishida et al. '19] January 3rd, 2019 0.496 0.501
3 CompLM Alibaba Damo NLP December 3rd, 2019 0.496 0.489
4 PALM Alibaba Damo NLP December 9th,2019 0.496 0.484
5 BERT+ Multi-Pointer-Generator Tongjun Li of the ColorfulClouds Tech and BUPT June 11th,2019 0.495 0.476
6 CompLM Alibaba Damo NLP November 19th,2019 0.495 0.470
7 CompLM Alibaba Damo NLP December 2nd, 2019 0.493 0.475
8 BERT+ Multi-Pointer-Generator Tongjun Li of the ColorfulClouds Tech and BUPT May 21st,2019 0.491 0.474
9 CompLM Alibaba Damo NLP November 19th,2019 0.488 0.485
10 BERT+ Multi-Pointer-Generator Tongjun Li of the ColorfulClouds Tech and BUPT March 26th,2019 0.487 0.465
11 Selector+Combine-Content-Generator NLGEN Model Shengjie Qian of Caiyun xiaoyi AI and BUPT March 11th, 2019 0.487 0.449
12 VNET Baidu NLP [Wang et al. '18] November 8th, 2018 0.484 0.468
13 BERT+ Multi-Pointer-Generator (Single) Tongjun Li of the ColorfulClouds Tech and BUPT March 19th,2019 0.484 0.459
14 Communicating BERT Xuan Liang of RIDLL from the University of Technology Sydney October 4th, 2019 0.483 0.472
15 MultiLM NLGen Model anonymous December 2nd, 2019 0.483 0.461
16 ranking+nlg anonymous October 9th, 2019 0.481 0.468
17 MultiLM NLGen Model anonymous December 5th, 2019 0.478 0.481
18 ranking+nlg anonymous October 9th, 2019 0.462 0.451
19 SNET + CES2S Bo Shao of SYSU University July 24th, 2018 0.450 0.406
20 KIGN-QA Chenliang Li April 22nd, 2019 0.441 0.462
21 Reader-Writer Microsoft Business Applications Group AI Research September 16th, 2018 0.439 0.426
22 ranking+nlg anonymous August 12th, 2019 0.439 0.411
23 ConZNet Samsung Research [Indurthi et al. '18] July 16th, 2018 0.421 0.386
24 Anonymous Anonymous November 21st, 2019 0.412 0.410
25 Bayes QA Bin Bi of Alibaba NLP June 14st, 2018 0.411 0.435
26 SNET+seq2seq Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS June 1st, 2018 0.401 0.375
27 BPG-NET Zhijie Sang of the Center for Intelligence Science and Technology Research(CIST) of the Beijing University of Posts and Telecommunications (BUPT) August 1st, 2018 0.382 0.347
28 GUM anonymous from anonymous September 4th, 2019 0.375 0.438
29 Deep Cascade QA Ming Yan of Alibaba Damo NLP October 25th, 2018 0.351 0.374
30 AE + ReRanking + Bert Based Multi-task ZhangY & WangC July 12th, 2019 0.331 0.376
31 BERT Encoded T-Net Y. Zhang, C. Wang, X.L. Chen August 5th, 2019 0.329 0.373
32 Multi-doc Enriched BERT Ming Yan of Alibaba Damo NLP June 20th, 2019 0.325 0.377
33 BIDAF+seq2seq Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS May 29th, 2018 0.322 0.283
34 BERT Encoded T-Net Y. Zhang, C. Wang, X.L. Chen July 12th, 2019 0.320 0.361
35 Unnamed Anonymous December 9th,2019 0.318 0.384
36 LM+Generator anonymous November 25th,2019 0.299 0.372
37 Masque Q&A Style NTT Media Intelligence Laboratories [Nishida et al. '19] January 3rd, 2019 0.285 0.399
38 Bert Based Multi-taskZhangY & WangC June 26th, 2019 0.284 0.349
39 Selector+Combine-Content-Generator QA Model Shengjie Qian of Caiyun xiaoyi AI and BUPT March 11th, 2019 0.281 0.337
40 DNET QA Geeks August 1st, 2018 0.275 0.332
41 ranker-reader RCZoo of UCAS May 15th, 2019 0.271 0.382
42 BIDAF+ELMo+SofterMax Wang Changbao November 16th, 2018 0.268 0.346
43 BERT+Multi-Loss S.L. Liu of NEUKG November 4th, 2019 0.413 0.422
44 LNET S.L. Liu of the NEUKG Nov 19th, 2019 0.284 0.339
45 MultiLM QnA Model anonymous December 2nd, 2019 0.266 0.340
46 MultiLM NLGen Model anonymous December 5th, 2019 0.257 0.360
47 SNET JY Zhao May 29th, 2018 0.247 0.308
48 Extraction-net zlsh80826 August 14th, 2018 0.247 0.321
49 lightNLP+BiDAF Enliple AI February 1st, 2019 0.210 0.108
50 BiDaF Baseline(Implemented By MSMARCO Team)
Allen Institute for AI & University of Washington [Seo et al. '16]
April 23th, 2018 0.169 0.093
51 TrioNLP + BiDAF Trio.AI of the CCNU September 23rd, 2018 0.142 0.160
52 BiDAF + LSTM Meefly January 15th,2019 0.119 0.173

MS MARCO V1 Leaderboard(12/01/2016-03/31/2018)

Rank Model Submission Date Rouge-L Bleu-1
1 MARS
YUANFUDAO research NLP
March 26th, 2018 0.497 0.480
2 Human Performance
December 2016 0.470 0.460
3 V-Net
Baidu NLP [Wang et al '18]
February 15th, 2018 0.462 0.445
4 S-Net
Microsoft AI and Research [Tan et al. '17]
June 2017 0.452 0.438
5 R-Net
Microsoft AI and Research [Wei et al. '16]
May 2017 0.429 0.422
6 HieAttnNet
Akaitsuki
March 26th, 2018 0.423 0.448
7 BiAttentionFlow+
ShanghaiTech University GeekPie_HPC team
March 11th, 2018 0.415 0.381
8 ReasoNet
Microsoft AI and Research [Shen et al. '16]
April 28th, 2017 0.388 0.399
9 Prediction
Singapore Management University [Wang et al. '16]
March 2017 0.373 0.407
10 FastQA_Ext
DFKI German Research Center for AI [Weissenborn et al. '17]
March 2017 0.337 0.339
11 FastQA
DFKI German Research Center for AI [Weissenborn et al. '17]
March 2017 0.321 0.340
12 Flypaper Model
ZhengZhou University
March 14th, 2018 0.317 0.342
13 DCNMarcoNet
Flying Riddlers @ Carnegie Mellon University
March 31st, 2018 0.313 0.238
14 BiDaF Baseline for V2 (Implemented By MSMARCO Team)
Allen Institute for AI & University of Washington [Seo et al. '16]
April 23th, 2018 0.268 0.129
15 ReasoNet Baseline
Trained on SQuAd, Microsoft AI & Research [Shen et al. '16]
December 2016 0.192 0.148