| Dataset | 11 |Domain | 12 |#Question | 13 |#Answer | 14 |
|---|---|---|---|
| TRECQA | 19 |Open-domain | 20 |1,229 | 21 |5,3417 | 22 |
| WikiQA | 25 |Open-domain | 26 |3,047 | 27 |29,258 | 28 |
| InsuranceQA | 31 |Insurance | 32 |12,889 | 33 |21,325 | 34 |
| FiQA | 37 |Financial | 38 |6,648 | 39 |57,641 | 40 |
| Yahoo! Answers | 43 |Open-domain | 44 |50,112 | 45 |253,440 | 46 |
| SemEval-2015 Task 3 | 49 |Open-domain | 50 |2,600 | 51 |16,541 | 52 |
| SemEval-2016 Task 3 | 55 |Open-domain | 56 |4,879 | 57 |36,198 | 58 |
| SemEval-2017 Task 3 | 61 |Open-domain | 62 |4,879 | 63 |36,198 | 64 |
| Dataset | 11 |# sentence pair | 12 |
|---|---|
| SNLI | 17 |570K | 18 |
| MultiNLI | 21 |433K | 22 |
| SciTail | 25 |27K | 26 |
| Dataset | 22 |pairs of sentence | 23 |
|---|---|
| MRPC | 28 |5800 | 29 |
| STS | 32 |1750 | 33 |
| SICK-R | 36 |9840 | 37 |
| SICK-E | 40 |9840 | 41 |
| Quora Question Pair | 44 |404290 | 45 |
2 | 
3 |
4 |
6 |
3 | 4 |
Awesome Neural Models for Semantic Match
5 |7 |
8 | A collection of papers maintained by MatchZoo Team.
9 |
10 | Checkout our open source toolkit MatchZoo for more information!
11 |
13 | 14 | Text matching is a core component in many natural language processing tasks, where many task can be viewed as a matching between two texts input. 15 | 16 |
17 | 
18 |
19 |
20 | Where **s** and **t** are source text input and target text input, respectively. The **psi** and **phi** are representation function for input **s** and **t**, respectively. The **f** is the interaction function, and **g** is the aggregation function. More detailed explaination about this formula can be found on [A Deep Look into Neural Ranking Models for Information Retrieval](https://arxiv.org/abs/1903.06902). The representative matching tasks are as follows:
21 |
22 | | **Tasks** | **Source Text** | **Target Text** |
23 | | :------------------------------------------------------------------------------------------: | :-------------: | :----------------------: |
24 | | [Ad-hoc Information Retrieval](Ad-hoc-Information-Retrieval/Ad-hoc-Information-Retrieval.md) | query | document (title/content) |
25 | | [Community Question Answering](Community-Question-Answering/Community-Question-Answering.md) | question | question/answer |
26 | | [Paraphrase Identification](Paraphrase-Identification/Paraphrase-Identification.md) | string1 | string2 |
27 | | [Natural Language Inference](Natural-Language-Inference/Natural-Language-Inference.md) | premise | hypothesis |
28 | | [Response Retrieval](Response-Retrieval/Response-Retrieval.md) | context/utterances | response |
29 | | [Long Form Question Answering](LFQA/LFQA.md) | question+document | answer |
30 |
31 | ### Healthcheck
32 |
33 | ```python
34 | pip3 install -r requirements.txt
35 | python3 healthcheck.py
36 | ```
37 |
--------------------------------------------------------------------------------
/Response-Retrieval/Response-Retrieval.md:
--------------------------------------------------------------------------------
1 | ## Response retrieval
2 |
3 | **Response retrieval**/selection aims to rank/select a proper response from a dialog repository.
4 | Automatic conversation (AC) aims to create an automatic human-computer dialog process for the purpose of question answering, task completion, and social chat (i.e., chit-chat). In general, AC could be formulated either as an IR problem that aims to rank/select a proper response from a dialog repository or a generation problem that aims to generate an appropriate response with respect to the input utterance. Here, we refer response retrieval as the IR-based way to do AC.
5 | Example:
6 | 
7 |
8 | ## Classic Datasets
9 |
10 | |Dataset |Partition | #Context Response pair | #Candidate per Context | Positive:Negative |Avg #turns per context|
11 | |---- | ---- | ---- |---- | ---- | ---- |
12 | |UDC| train/validation/test| 1M/500k/500k| 2/10/10| 1:1/1:9/1:9| 10.13/10.11/10.11|
13 | |[**Douban**](https://github.com/MarkWuNLP/MultiTurnResponseSelection)| train/validation/test| 1M/50k/10k |2/2/10|1:1/1:1/1.18:8.82|6.69/6.75/6.45|
14 | |[**MSDialog**](https://ciir.cs.umass.edu/downloads/msdialog/)| train/validation/test| 173k/37k/35k | 10/10/10|1:9/1:9/1:9|5.0/4.9/4.4|
15 | |[**EDC**](https://github.com/cooelf/DeepUtteranceAggregation)| train/validation/test| 1M/10k/10k| 2/2/10| 1:1/1:1/1:9| 5.51/5.48/5.64|
16 | |Persona-Chat dataset| 8939/1000/968 | 20/20/20 | 1:19/1:19/1:19 | 7.35/7.80/7.76 |
17 | |CMUDoG dataset| 2881/196/537 | 20/20/20 | 1:19/1:19/1:19 | 12.55/12.37/12.36 |
18 |
19 | - Ubuntu Dialog Corpus (UDC) contains multi-turn dialogues collected from chat logs of the Ubuntu Forum. The data set consists of 1 million context-response pairs for training, 0.5 million pairs for validation, and 0.5 million pairs for testing. Positive responses are true responses from humans, and negative ones are randomly sampled. The ratio of the positive and the negative is 1:1 in training, and 1:9 in validation and testing.
20 | - [**Douban Conversation Corpus**](https://github.com/MarkWuNLP/MultiTurnResponseSelection) is an open domain dataset constructed from Douban group (a popular social networking service in China). The data set consists of 1 million context-response pairs for training, 50k pairs for validation, and 10k pairs for testing, corresponding to 2, 2, and 10 response candidates per context respectively. Response candidates on the test set, retrieved from Sina Weibo (the largest microblogging service in China), are labeled by human judges.
21 | - [**MSDialog**](https://ciir.cs.umass.edu/downloads/msdialog/) is a labeled dialog dataset of question answering (QA) interactions between information seekers and answer providers from an online forum on Microsoft products (Microsoft Community). The dataset contains more than 2,000 multi-turn information-seeking conversations with 10,000 utterances that are annotated with user intent on the utterance level.
22 | - [**E-commerce Dialogue Corpus**](https://github.com/cooelf/DeepUtteranceAggregation) contains over 5 types of conversations (e.g. commodity consultation, logistics express, recommendation, negotiation and chitchat) based on over 20 commodities. The ratio of the positive and the negative
23 | is 1:1 in training and validation, and 1:9 in testing.
24 |
25 | $R_n@k$: recall at position $k$ in $n$ candidates.
26 |
27 | ## Performance
28 |
29 | ### Ubuntu Corpus
30 |
31 | | Model | Code|MAP|$R_2@1$|$R_{10}@1$|$R_{10}@2$|$R_{10}@5$|Paper| type |
32 | | ---- | ---- | ---- | --- | --- | --- | --- | ---- | ---- |
33 | | Multi-View (Zhou et al. 2016)| N/A | — | 0.908 | 0.662 | 0.801 | 0.951 | [Multi-view Response Selection for Human-Computer Conversation, ACL 2016](https://www.aclweb.org/anthology/D16-1036.pdf) | multi-turn |
34 | | DL2R (Yan, Song and Wu 2016)| N/A | — | 0.899 | 0.626 | 0.783 | 0.944|[Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System, SIGIR 2016](http://www.ruiyan.me/pubs/SIGIR2016.pdf) | multi-turn|
35 | | SMN (Wu et al. 2017) | [](https://github.com/MarkWuNLP/MultiTurnResponseSelection) | 0.7327 | 0.927 | 0.726 |0.847| 0.962 |[Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots, ACL 2017](https://www.aclweb.org/anthology/P17-1046.pdf)| Multi-turn|
36 | |DAM(Zhou et al. 2018) | [](https://github.com/baidu/Dialogue/tree/master/DAM) | — | 0.938 | 0.767 | 0.874 | 0.969 |[Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network, ACL 2018](https://www.aclweb.org/anthology/P18-1103.pdf) | multi-turn|
37 | |DUA (Zhang et al. 2018)|[](https://github.com/cooelf/DeepUtteranceAggregation)| — | — | 0.752 | 0.868 | 0.962 |[Modeling Multi-turn Conversation with Deep Utterance Aggregation, arXiv 2018](https://arxiv.org/pdf/1806.09102.pdf)|multi-turn|
38 | | DMN (Yang et al. 2018)| [](https://github.com/yangliuy/NeuralResponseRanking) | 0.7719 | — | — | — | — |[Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems, arXiv 2018](https://arxiv.org/pdf/1805.00188.pdf) |multi-turn |
39 | |U2U-IMN(Gu et al. 2019 a)|[](https://github.com/JasonForJoy/U2U-IMN) | **0.866** | 0.945 | 0.790 | 0.886 | 0.973 |[Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1911.06940v1.pdf)|multi-turn|
40 | |TripleNet(Ma et al. 2019)|[](https://github.com/wtma/TripleNet.)|— | 0.943 | 0.79 | 0.885 | 0.97 |[TripleNet: Triple Attention Network for Multi-Turn Response Selection in Retrieval-based Chatbots, arXiv 2019](https://arxiv.org/pdf/1909.10666v2.pdf)|multi-turn|
41 | |IMN(Gu et al. 2019 b)|[](https://github.com/JasonForJoy/IMN)| — | 0.946 | 0.794 | 0.889 | 0.974|[Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1901.01824v2.pdf)|multi-turn|
42 | |IOI-local(Tao et al. 2019)|[](https://github.com/chongyangtao/IOI)| — | 0.947 | 0.796 | 0.894 | 0.974 | [One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues, ACL 2019](https://www.aclweb.org/anthology/P19-1001.pdf) |multi-turn|
43 | |MSN(Yuan et al. 2019)|[](https://github.com/chunyuanY/Dialogue)|— |— | 0.8 | 0.899 | 0.978 | [Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots, ACL 2019](https://www.aclweb.org/anthology/D19-1011.pdf) |multi-turn|
44 | |SA-BERT (Gu et al. 2020)|[](https://github.com/JasonForJoy/SA-BERT)| — | **0.965** | **0.855** | **0.928** | **0.983** |[Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2020](https://arxiv.org/pdf/2004.03588v1.pdf)|multi-turn|
45 | |RoBERTaBASE-SS-DA (Lu et al. 2020)|[](https://github.com/CSLujunyu/Improving-Contextual-Language-Modelsfor-Response-Retrieval-in-Multi-Turn-Conversation) | - |0.955 |0.826 |0.909 |0.978 | [Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation, SIGIR 2020](https://dl.acm.org/doi/pdf/10.1145/3397271.3401255) | multi-turn|
46 | |SMN + ECMo (Tao et al. 2020)| N/A | - |0.934 |0.756 |0.867 |0.966 |[Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection, SIGIR 2020](https://dl.acm.org/doi/pdf/10.1145/3397271.3401290) | multi-turn|
47 |
48 |
49 | ### Douban Conversation Corpus
50 |
51 | | Model | Code | MAP | MRR | P@1| $R_{10}@1$ |$R_{10}@2$ |$R_{10}@5$|Paper| type |
52 | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
53 | | Multi-View (Zhou et al. 2016)| N/A | 0.505 | 0.543 | 0.342 | 0.202 | 0.350 | 0.729 | [Multi-view Response Selection for Human-Computer Conversation, ACL 2016](https://www.aclweb.org/anthology/D16-1036.pdf) | multi-turn |
54 | | DL2R (Yan, Song and Wu 2016)| N/A |0.488 | 0.527 | 0.33 | 0.193 | 0.342 | 0.705 |[Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System, SIGIR 2016](http://www.ruiyan.me/pubs/SIGIR2016.pdf) | multi-turn|
55 | | SMN (Wu et al. 2017) | [](https://github.com/MarkWuNLP/MultiTurnResponseSelection) | 0.529 | 0.572 | 0.397 | 0.236 | 0.396 | 0.734 |[Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots, ACL 2017](https://www.aclweb.org/anthology/P17-1046.pdf)| Multi-turn|
56 | |DAM(Zhou et al. 2018) | [](https://github.com/baidu/Dialogue/tree/master/DAM) | 0.55 | 0.601 | 0.427 | 0.254 | 0.410 | 0.757 |[Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network, ACL 2018](https://www.aclweb.org/anthology/P18-1103.pdf) | multi-turn|
57 | |DUA (Zhang et al. 2018)|[](https://github.com/cooelf/DeepUtteranceAggregation)| 0.551 | 0.599 | 0.421 | 0.243 | 0.421 | 0.780 |[Modeling Multi-turn Conversation with Deep Utterance Aggregation, arXiv 2018](https://arxiv.org/pdf/1806.09102.pdf)|multi-turn|
58 | |U2U-IMN(Gu et al. 2019 a)|[](https://github.com/JasonForJoy/U2U-IMN) |0.564 | 0.611 | 0.429 | 0.259 | 0.43 | 0.791 |[Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1911.06940v1.pdf)|multi-turn|
59 | |TripleNet(Ma et al. 2019)|[](https://github.com/wtma/TripleNet.)|0.564 | 0.618 | 0.447 | 0.268 | 0.426 | 0.778 |[TripleNet: Triple Attention Network for Multi-Turn Response Selection in Retrieval-based Chatbots, arXiv 2019](https://arxiv.org/pdf/1909.10666v2.pdf)|multi-turn|
60 | |IMN(Gu et al. 2019 b)|[](https://github.com/JasonForJoy/IMN)|0.570 | 0.615 | 0.433 | 0.262 | 0.452 | 0.789 |[Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1901.01824v2.pdf)|multi-turn|
61 | |IOI-local(Tao et al. 2019)|[](https://github.com/chongyangtao/IOI)| 0.573 | 0.621 | 0.444 | 0.269 | 0.451 | 0.786 |[One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues, ACL 2019](https://www.aclweb.org/anthology/P19-1001.pdf)|multi-turn|
62 | |MSN(Yuan et al. 2019)|[](https://github.com/chunyuanY/Dialogue)| 0.587 | 0.632 | 0.470 | 0.295 | 0.452 | 0.788 |[Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots, ACL 2019](https://www.aclweb.org/anthology/D19-1011.pdf)|multi-turn|
63 | |SA-BERT(Gu et al. 2020)|[](https://github.com/JasonForJoy/SA-BERT)| **0.619** | **0.659** | **0.496** | **0.313** | **0.481** | **0.847** |[Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2020](https://arxiv.org/pdf/2004.03588v1.pdf)|multi-turn|
64 | |RoBERTaBASE-SS-DA (Lu et al. 2020)|[](https://github.com/CSLujunyu/Improving-Contextual-Language-Modelsfor-Response-Retrieval-in-Multi-Turn-Conversation) | 0.602 | 0.646 | 0.460 | 0.280 | 0.495 | 0.847 | [Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation, SIGIR 2020](https://dl.acm.org/doi/pdf/10.1145/3397271.3401255) | multi-turn|
65 | |SMN + ECMo (Tao et al. 2020)| N/A | 0.549 | 0.593 | 0.409 | 0.247 | 0.416 | 0.774 |[Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection, SIGIR 2020](https://dl.acm.org/doi/pdf/10.1145/3397271.3401290) | multi-turn|
66 |
67 |
68 | ### MSDialog
69 |
70 | | Model | Code | MAP | Recall@5| Recall@1| Recall@2|Paper| type |
71 | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
72 | | DMN (Yang et al. 2018)|[](https://github.com/yangliuy/NeuralResponseRanking)| 0.6792 | 0.9356 | 0.5021 | 0.7122 |[Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems, arXiv 2018](https://arxiv.org/pdf/1805.00188.pdf) |multi-turn |
73 |
74 | ### E-commerce Corpus
75 |
76 | | Model | Code | MAP | $R_{10}@1$ | $R_{10}@2$ | $R_{10}@5$ | Paper| type |
77 | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
78 | | Multi-View (Zhou et al. 2016)| N/A | — | 0.421 | 0.601 | 0.861 | [Multi-view Response Selection for Human-Computer Conversation, ACL 2016](https://www.aclweb.org/anthology/D16-1036.pdf) | multi-turn |
79 | | DL2R (Yan, Song and Wu 2016)| N/A | — |0.399 | 0.571 | 0.842 |[Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System, SIGIR 2016](http://www.ruiyan.me/pubs/SIGIR2016.pdf) | multi-turn|
80 | | SMN (Wu et al. 2017) | [](https://github.com/MarkWuNLP/MultiTurnResponseSelection) | — | 0.453 | 0.654 | 0.886 | [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots, ACL 2017](https://www.aclweb.org/anthology/P17-1046.pdf) | Multi-turn|
81 | |DAM(Zhou et al. 2018) | [](https://github.com/baidu/Dialogue/tree/master/DAM) | — | 0.526 | 0.727 | 0.933 |[Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network, ACL 2018](https://www.aclweb.org/anthology/P18-1103.pdf) | multi-turn|
82 | |DUA (Zhang et al. 2018)|[](https://github.com/cooelf/DeepUtteranceAggregation)| — | 0.501 | 0.700 | 0.921 |[Modeling Multi-turn Conversation with Deep Utterance Aggregation, arXiv 2018](https://arxiv.org/pdf/1806.09102.pdf)|multi-turn|
83 | |U2U-IMN(Gu et al. 2019 a)|[](https://github.com/JasonForJoy/U2U-IMN) |**0.759** | 0.616 | 0.806 | 0.966 |[Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1911.06940v1.pdf)|multi-turn|
84 | |IMN(Gu et al. 2019 b)|[](https://github.com/JasonForJoy/IMN)| — | 0.621 | 0.797 | 0.964 |[Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2019](https://arxiv.org/pdf/1901.01824v2.pdf)|multi-turn|
85 | |IOI-local(Tao et al. 2019)|[](https://github.com/chongyangtao/IOI)|— | 0.563 | 0.768 | 0.950 |[One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues, ACL 2019](https://www.aclweb.org/anthology/P19-1001.pdf)|multi-turn|
86 | |MSN(Yuan et al. 2019)|[](https://github.com/chunyuanY/Dialogue)| — | 0.606 | 0.770 | 0.937 |[Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots, ACL 2019](https://www.aclweb.org/anthology/D19-1011.pdf)|multi-turn|
87 | |SA-BERT(Gu et al. 2020)|[](https://github.com/JasonForJoy/SA-BERT)| — | 0.704 | 0.879 | **0.985** |[Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots, arXiv 2020](https://arxiv.org/pdf/2004.03588v1.pdf)|multi-turn|
88 | |RoBERTaBASE-SS-DA (Lu et al. 2020)|[](https://github.com/CSLujunyu/Improving-Contextual-Language-Modelsfor-Response-Retrieval-in-Multi-Turn-Conversation) | - | **0.800** | **0.910** | 0.972 | [Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation, SIGIR 2020](https://dl.acm.org/doi/pdf/10.1145/3397271.3401255) | multi-turn|
89 |
90 | ### Persona-Chat dataset
91 |
92 | Orinigal Persona
93 | | Model | Code | $R_{20}@1$ | $R_{20}@2$ | $R_{20}@5$ | Paper | type |
94 | | ---- | ---- | ---- | ----| ---- | ---- | ---- |
95 | | RSM-DCK (Hua et al. 2020) | N/A | 0.7965 | 0.9021 | 0.9747 | [Learning to Detect Relevant Contexts and Knowledge for Response Selection in Retrieval-based Dialogue Systems, CIKM 2020](https://dl.acm.org/doi/pdf/10.1145/3340531.3411967) | multi-turn |
96 |
97 | Revised Persona
98 | | Model | Code | $R_{20}@1$ | $R_{20}@2$ | $R_{20}@5$ | Paper | type |
99 | | ---- | ---- | ---- | ----| ---- | ---- | ---- |
100 | | RSM-DCK (Hua et al. 2020) | N/A | 0.7185 | 0.8494 | 0.9550 | [Learning to Detect Relevant Contexts and Knowledge for Response Selection in Retrieval-based Dialogue Systems, CIKM 2020](https://dl.acm.org/doi/pdf/10.1145/3340531.3411967) | multi-turn |
101 | ### CMUDoG dataset
102 | | Model | Code | $R_{20}@1$ | $R_{20}@2$ | $R_{20}@5$ | Paper | type |
103 | | ---- | ---- | ---- | ----| ---- | ---- | ---- |
104 | | RSM-DCK (Hua et al. 2020) | N/A | 0.7925 | 0.8884 | 0.9666 | [Learning to Detect Relevant Contexts and Knowledge for Response Selection in Retrieval-based Dialogue Systems, CIKM 2020](https://dl.acm.org/doi/pdf/10.1145/3340531.3411967) | multi-turn |
105 |
106 |
--------------------------------------------------------------------------------
/_config.yml:
--------------------------------------------------------------------------------
1 | theme: jekyll-theme-slate
--------------------------------------------------------------------------------
/_includes/chart.html:
--------------------------------------------------------------------------------
1 |
20 |
21 |
18 |
22 | {% for result in include.results %}
23 | {% assign score = result[include.score] %}
24 |
30 |
--------------------------------------------------------------------------------
/_includes/table.html:
--------------------------------------------------------------------------------
1 | {% assign scores = include.scores | split: "," %}
2 |
3 |
25 |
28 | {% endfor %}
29 | {{ result.authors }} ({{ result.year }})
26 |{{ score }}
27 || Model | 7 | {% for score in scores %} 8 |{{ score }} | 9 | {% endfor %} 10 |Paper / Source | 11 |Code | 12 |
|---|---|---|---|
| {% if result.model %} {{ result.model }} by {% endif %} {{ result.authors }} ({{ result.year }}) | 18 | {% for score in scores %} 19 |{{ result[score] }} | 20 | {% endfor %} 21 |{{ result.paper }} | 22 |23 | {% for el in result.code %} 24 | {{ el.name }} 25 | {% endfor %} 26 | | 27 |