├── README.md └── Copy.md /README.md: -------------------------------------------------------------------------------- 1 | ### Link Prediction Problem: Experiment Results on Benchmark Datasets 2 | 3 | In this thread I have collected recently published and well-aged approaches for knowledge graph completion (link prediction) problems. The included methods can be roughly divided into groups: Rule-based Methods, Embedding Methods, Graph Neural Network Methods and Hybird Methods (relational reinforcement learning, differentiable reasoning, etc.). Experiments results are collected from original and follow-up papers, and best effort has been attempted to assure results of the same method are carried out with similar configurations, but do expect results under different settings being put around each other. 4 | 5 | #### Type Markers: 6 | - **E** as embedding methods 7 | - **R** as rule-based methods 8 | - **NN** as graph neural nets 9 | - Hybird Methods: 10 | - **E+NN** 11 | - **R+NN** 12 | - **R+E** 13 | - **R+E+NN** 14 | - **R+RL** : Rule + reinforcement learning 15 | 16 | Approach Format: 17 | - (Approach Type)-(Method Name)-(Published Year)-(Reference Index) 18 | 19 | #### Datasets 20 | - Large 21 | - FB15K 22 | - FB15K-237 23 | - NELL-995 24 | - WN18 25 | - WN18RR 26 | - YAGO3-10 27 | - YAGO37 28 | - Small/Medium 29 | - Kinship 30 | - UMLS 31 | - Citeseer 32 | - Cora 33 | - Pubmed 34 | - UWCSE 35 | 36 | #### Experiments Results 37 | 38 | > Filter Mean Reciprocal Rank (Filter MRR) 39 | 40 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 41 | |--|--|--|--|--|--|--|--| 42 | |**E**-TransE-2013[9]|0.401|0.38|0.294|0.455|0.226|0.068|- 43 | |**E**-DistMult-2015-[6]|0.485|0.639|0.281|0.813|0.444|0.516|0.878 44 | |**E**-ANALOGY-2017-[8]|-|0.725|0.219|0.942|-|-|- 45 | |**E**-SimpleE-2018-[1]|-|0.727|-|0.942|-|-|-| 46 | |**E**-ConvE-2018-[2]|0.491|0.745|0.316|0.942|0.46|0.833|0.797 47 | |**E**-ComplEx-N3-2018-[3]|0.482|**0.86**|0.37|**0.95**|**0.48**|0.823|0.838| 48 | |**E**-CrossE-2019-[7]|-|0.728|0.299|0.83|-|-|- 49 | |**E**-Rotate-2019-[23]|-|0.799|0.338|0.949|0.476|-|- 50 | |**E**-IterE-2019-[26]|-|0.628|0.247|0.913|0.274|-|- 51 | |**R**-Node+LinkFeat-2015-[17]|-|0.821|0.237|0.94|-|-|- 52 | |**R**-AMIE-2015[10]|-|-|-|-|-|-|-| 53 | |**R**-Gaifman-2016-[24]|-|-|-|-|-|-|- 54 | |**R**-RuleN-2018-[11]|-|-|-|-|-|-|-| 55 | |**R**-RUGE-2018-[13]|-|0.768|-|-|-|-|- 56 | |**R**-AnyBURL-2019-[12]|-|0.83|0.31|**0.95**|**0.48**|-|-| 57 | |**R+E**-RLvLR-2018-[15]|-|-|0.24|-|-|-|- 58 | |**R+E**-RPJE-2020-[29]|0.361|0.816|0.47|0.946|-|-|- 59 | |**NN**-R-GCN-2017-[5]|-|0.651|0.248|0.814|-|-|- 60 | |**NN**-SACN-2019-[27]|-|-|0.35|-|0.47|-|-|- 61 | |**E+NN**-R-GCN+\-2018-[4]|-|0.696|0.249|0.819|-|-|-| 62 | |**E+NN**-ConvKB-2018-[16]|0.43|-|0.396|-|0.248|0.033|- 63 | |**E+NN**-AttentionE-2019-[18]|0.530|-|**0.518**|-|0.44|**0.904**|- 64 | |**R+NN**-Neural LP-2017-[14]|-|0.76|0.24|0.94|0.435|0.619|0.778 65 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.793|0.912 66 | |**R+NN**-DRUM-2019-[22]|-|-|0.343|-|**0.486**|0.61|0.81 67 | |**R+RL**-MINERVA-2018-[19]|0.725|-|0.293|-|0.448|0.720|0.825 68 | |**R+RL**-Multi-Hop-2018-[21]|**0.727**|-|0.407|-|0.472|0.878|**0.94** 69 | 70 | > Hits@1 71 | > 72 | 73 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 74 | |--|--|--|--|--|--|--|--| 75 | |**E**-TransE-2013[9]|0.344|0.231|0.198|0.089|0.0427|0.009|- 76 | |**E**-DistMult-2015-[6]|0.401|0.522|0.199|0.701|0.412|0.367|**0.916** 77 | |**E**-ANALOGY-2017-[8]|-|0.646|0.131|0.939|-|-|- 78 | |**E**-SimpleE-2018-[1]|-|0.660|-|0.939|-|-|- 79 | |**E**-ConvE-2018-[2]|0.403|0.67|0.239|0.935|0.39|0.738|0.894| 80 | |**E**-ComplEx-N3-2018-[3]|0.399|0.599|0.132|0.936|0.409|0.733|0.823| 81 | |**E**-CrossE-2019-[7]|-|0.634|0.211|0.741|-|-|- 82 | |**E**-Rotate-2019-[23]|-|0.75|0.241|0.944|0.428|-|-| 83 | |**E**-IterE-2019-[26]|-|0.551|0.179|0.891|0.254|-|-| 84 | |**R**-Node+LinkFeat-2015-[17]|-|-|-|-|-|-|-| 85 | |**R**-AMIE-2015[10]|-|0.647|0.174|0.872|0.358|-|- 86 | |**R**-Gaifman-2016-[24]|-|0.692|-|0.767|-|-|-| 87 | |**R**-RuleN-2018-[11]|-|0.772|0.182|0.945|0.427|-|- 88 | |**R**-RUGE-2018-[13]|-|0.703|-|-|-|-|- 89 | |**R**-AnyBURL-2019-[12]|-|**0.808**|0.233|**0.946**|**0.446**|-|- 90 | |**R+E**-RLvLR-2018-[15]|-|-|-|-|-|-|-| 91 | |**NN**-R-GCN-2017-[5]|-|0.541|0.153|0.686|-|-|- 92 | |**NN**-SACN-2019-[27]|-|-|0.26|-|0.43|-|-|- 93 | |**E+NN**-R-GCN+\-2018-[4]|-|0.601|0.151|0.697|-|-|- 94 | |**E+NN**-ConvKB-2018-[16]|0.37|-|0.198|-|0.0562|0.4362|-| 95 | |**E+NN**-AttentionE-2019-[18]|0.447|-|0.46|-|0.361|**0.859**|- 96 | |**R+NN**-Neural LP-2017-[14]|-|-|0.166|-|0.376|0.475|0.643| 97 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.759|0.843| 98 | |**R+NN**-DRUM-2019-[22]|-|-|0.255|-|0.425|0.46|0.67 99 | |**R+RL**-MINERVA-2018-[19]|**0.663**|-|0.217|-|0.413|0.605|0.728 100 | |**R+RL**-Multi-Hop-2018-[21]|0.656|-|**0.329**|-|0.437|0.811|0.902 101 | 102 | > Hits@3 103 | > 104 | 105 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 106 | |--|--|--|--|--|--|--|--| 107 | |**E**-TransE-2013[9]|0.472|0.472|0.376|0.823|0.441|0.643|- 108 | |**E**-DistMult-2015-[6]|0.524|0.718|0.301|0.921|0.47|0.581|0.967 109 | |**E**-ANALOGY-2017-[8]|-|0.785|0.240|0.944|-|-|-| 110 | |**E**-SimpleE-2018-[1]|-|0.773|-|0.944|-|-|- 111 | |**E**-ConvE-2018-[2]|0.531|0.801|0.35|0.947|0.43|0.917|0.964| 112 | |**E**-ComplEx-N3-2018-[3]|0.528|0.759|0.244|0.945|0.469|0.899|0.962| 113 | |**E**-CrossE-2019-[7]|-|0.802|0.331|0.931|-|-|- 114 | |**E**-Rotate-2019-[23]|-|**0.83**|0.375|**0.952**|0.492|-|-| 115 | |**E**-IterE-2019-[26]|-|0.673|0.262|0.935|0.281|-|-| 116 | |**R**-Node+LinkFeat-2015-[17]|-|-|-|-|-|-|-| 117 | |**R**-AMIE-2015[10]|-|-|-|-|-|-|-| 118 | |**R**-Gaifman-2016-[24]|-|-|-|-|-|-|- 119 | |**R**-RuleN-2018-[11]|-|-|-|-|-|-|-| 120 | |**R**-RUGE-2018-[13]|-|0.815|-|-|-|-|- 121 | |**R**-AnyBURL-2019-[12]|-|-|-|-|-|-|-| 122 | |**R+E**-RLvLR-2018-[15]|-|-|-|-|-|-|-| 123 | |**NN**-R-GCN-2017-[5]|-|0.736|0.258|0.928|-|-|- 124 | |**NN**-SACN-2019-[27]|-|-|0.39|-|0.48|-|-|- 125 | |**E+NN**-R-GCN+\-2018-[4]|0.126|0.760|0.264|0.929|0.137|0.088|- 126 | |**E+NN**-ConvKB-2018-[16]|0.47|-|0.324|-|0.445|0.755|-| 127 | |**E+NN**-AttentionE-2019-[18]|0.564|-|**0.54**|-|0.483|**0.941**|- 128 | |**R+NN**-Neural LP-2017-[14]|-|-|0.248|-|0.468|0.707|0.869| 129 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.798|**0.983** 130 | |**R+NN**-DRUM-2019-[22]|-|-|0.378|-|**0.513**|0.71|0.94 131 | |**R+RL**-MINERVA-2018-[19]|**0.773**|-|0.329|-|0.456|0.812|0.9| 132 | |**R+RL**-Multi-Hop-2018-[21]|-|-|-|-|-|-|-| 133 | 134 | > Hits@10 135 | > 136 | 137 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 138 | |--|--|--|--|--|--|--|--| 139 | |**E**-TransE-2013[9]|0.501|0.539|0.465|0.909|0.501|0.841|- 140 | |**E**-DistMult-2015-[6]|0.61|0.814|0.408|0.943|0.49|0.867|0.992 141 | |**E**-ANALOGY-2017-[8]|-|0.854|0.405|0.947|-|-|- 142 | |**E**-SimpleE-2018-[1]|-|0.838|-|0.947|-|-|- 143 | |**E**-ConvE-2018-[2]|0.613|0.873|0.491|0.955|0.48|0.9814|0.992| 144 | |**E**-ComplEx-N3-2018-[3]|0.606|**0.91**|0.56|0.96|0.57|0.9711|0.995 145 | |**E**-CrossE-2019-[7]|-|-|0.474|-|-|-|- 146 | |**E**-Rotate-2019-[23]|-|0.884|0.533|0.959|0.571|-|-| 147 | |**E**-IterE-2019-[26]|-|0.771|0.392|0.948|0.314|-|-| 148 | |**R**-Node+LinkFeat-2015-[17]|-|0.87|0.36|0.943|-|-|- 149 | |**R**-AMIE-2015[10]|-|0.858|0.409|0.948|0.388|-|- 150 | |**R**-Gaifman-2016-[24]|-|0.842|-|0.939|-|-|-|- 151 | |**R**-RuleN-2018-[11]|-|0.87|0.42|0.958|0.536|-|- 152 | |**R**-RUGE-2018-[13]|-|0.865|-|-|-|-|-| 153 | |**R**-AnyBURL-2019-[12]|-|0.89|0.486|0.959|0.555|-|- 154 | |**R+E**-RLvLR-2018-[15]|-|-|0.393|-|-|-|- 155 | |**R+E**-RPJE-2020-[29]|0.501|0.903|0.625|0.951|-|-|- 156 | |**NN**-R-GCN-2017-[5]|-|0.825|0.414|0.955|-|-|-| 157 | |**NN**-SACN-2019-[27]|-|-|0.54|-|0.54|-|-|- 158 | |**E+NN**-R-GCN+\-2018-[4]|0.188|0.842|0.417|**0.964**|0.08|0.239|-| 159 | |**E+NN**-ConvKB-2018-[16]|0.545|-|0.517|-|0.525|0.953|- 160 | |**E+NN**-AttentionE-2019-[18]|0.695|-|**0.626**|-|0.581|0.98|-| 161 | |**R+NN**-Neural LP-2017-[14]|-|0.837|0.362|0.945|0.566|0.912|0.962 162 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.878|**1.0** 163 | |**R+NN**-DRUM-2019-[22]|-|-|0.516|-|**0.586**|0.92|0.98 164 | |**R+RL**-MINERVA-2018-[19]|0.831|-|0.456|-|0.513|0.924|0.968 165 | |**R+RL**-Multi-Hop-2018-[21]|**0.844**|-|0.564|-|0.542|**0.982**|0.992 166 | 167 | ### References 168 | [1] Kazemi, S. M., & Poole, D. (2018). SimplE Embedding for Link Prediction in Knowledge Graphs. NIPS, 1–12. 169 | 170 | [2] Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2018). Convolutional 2D Knowledge Graph Embeddings. AAAI, 1811–1818. 171 | 172 | [3] Lacroix, T., Usunier, N., & Obozinski, G. (2018). Canonical Tensor Decomposition for Knowledge Base Completion. ICML. 173 | 174 | [4] Schlichtkrull, M., Kipf, T. N., Bloem, P., Titov, I., & Welling, M. (2018). Modeling Relational Data with Graph Convolutional Networks. ESWC, 593–607. 175 | 176 | [5] Kipf, T. N., & Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. ICLR, 1–14. Retrieved from http://arxiv.org/abs/1609.02907 177 | 178 | [6] Yang, B., Yih, W., He, X., Gao, J., & Deng, L. (2015). Embedding entities and relations for learning and inference in knowledge bases. ICLR, 1–13. 179 | 180 | [7] Zhang, W., Paudel, B., Zhang, W., Bernstein, A., & Chen, H. (2019). Interaction Embeddings for Prediction and Explanation in Knowledge Graphs. WSDM. 181 | 182 | [8] Liu, H., Wu, Y., & Yang, Y. (2017). Analogical Inference for Multi-relational Embeddings. ICML. 183 | 184 | [9] Bordes, A., Usunier, N., Weston, J., & Yakhnenko, O. (2013). Translating Embeddings for Modeling Multi-Relational Data. NIPS, 2787–2795. https://doi.org/10.1007/s13398-014-0173-7.2 185 | 186 | [10] Galárraga, L., Teflioudi, C., Hose, K., & Suchanek, F. M. (2015). Fast rule mining in ontological knowledge bases with AMIE+. VLDB Journal, 24(6), 707–730. https://doi.org/10.1007/s00778-015-0394-1 187 | 188 | [11] Meilicke, C., Fink, M., Wang, Y., Ruffinelli, D., Gemulla, R., & Stuckenschmidt, H. (2018). Fine-grained evaluation of rule- and embedding-based systems for knowledge graph completion. ISWC, 3–20. https://doi.org/10.1007/978-3-030-00671-6_1 189 | 190 | [12] Meilicke, C., Chekol, M. W., Ruffinelli, D., & Stuckenschmidt, H. (2019). Anytime Bottom-Up Rule Learning for Knowledge Graph Completion. IJCAI, 3137–3143. 191 | 192 | [13] Guo, S., Wang, Q., Wang, L., Wang, B., & Guo, L. (2018). Knowledge Graph Embedding with Iterative Guidance from Soft Rules. AAAI, 4816–4823. 193 | 194 | [14] Yang, F., & Cohen, W. W. (2017). Differentiable Learning of Logical Rules for Knowledge Base Reasoning. NIPS. 195 | 196 | [15] Omran, P. G., Wang, K., & Wang, Z. (2018). Scalable rule learning via learning representation. IJCAI, 2149–2155. 197 | 198 | [16] Nguyen, D. Q., Nguyen, T. D., Nguyen, D. Q., & Phung, D. (2018). A Novel Embedding Model for Knowledge Base Completion. NAACL. 199 | 200 | [17] Toutanova, K., & Chen, D. (2015). Observed versus latent features for knowledge base and text inference. Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality, 57–66. https://doi.org/10.18653/v1/w15-4007 201 | 202 | [18] Nathani, D., Chauhan, J., Sharma, C., & Kaul, M. (2019). Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs. ACL. Retrieved from http://arxiv.org/abs/1906.01195 203 | 204 | [19] Das, R., Dhuliawala, S., Zaheer, M., Vilnis, L., Durugkar, I., Krishnamurthy, A., … Mccallum, A. (2018). Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. ICLR. 205 | 206 | [20] Rocktäschel, T., & Riedel, S. (2017). End-to-End Differentiable Proving. NIPS. Retrieved from http://arxiv.org/abs/1705.11040 207 | 208 | [21] Lin, X. V., Richard, S., & Caiming, X. (2018). Multi-Hop Knowledge Graph Reasoning with Reward Shaping. ACL, 3243–3253. 209 | 210 | [22] Sadeghian, A., Armandpour, M., Ding, P., & Wang, D. Z. (2019). DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs. NIPS, 1–13. Retrieved from http://arxiv.org/abs/1911.00055 211 | 212 | [23] Sun, Z., Deng, Z., Nie, J., & Tang, J. (2019). Rotate: Knowledge graph embedding by relational rotation in complex space. ICLR, 1–18. 213 | 214 | [24] Niepert, M. (2016). Discriminative gaifman models. NIPS, 3413–3421. 215 | 216 | [26] Zhang, W., Paudel, B., Wang, L., Chen, J., Zhu, H., Zhang, W., … Chen, H. (2019). Iteratively Learning Embeddings and Rules for Knowledge Graph Reasoning. WWW. 217 | 218 | [27] Shang, C., Tang, Y., Huang, J., Bi, J., He, X., & Zhou, B. (2019). End-to-End Structure-Aware Convolutional Networks for Knowledge Base Completion. AAAI, 3060–3067. 219 | 220 | [29] Niu, G., Zhang, Y., Li, B., Cui, P., Liu, S., Li, J., & Zhang, X. (2020). Rule-Guided Compositional Representation Learning on Knowledge Graphs. AAAI. 221 | 222 | ### Planned Addition 223 | 224 | [25] Zhu, Y., Liu, H., Wu, Z., Song, Y., & Zhang, T. (2019). Representation Learning with Ordered Relation Paths for Knowledge Graph Completion. EMNLP. 225 | [28] Niu, G., Zhang, Y., Li, B., Cui, P., Liu, S., Li, J., & Zhang, X. (2019). Rule-Guided Compositional Representation Learning on Knowledge Graphs. Arxiv. 226 | -------------------------------------------------------------------------------- /Copy.md: -------------------------------------------------------------------------------- 1 | ### Link Prediction Problem: Experiment Results on Benchmark Datasets 2 | 3 | In this thread I have collected recently published and well-aged approaches for knowledge graph completion (link prediction) problems. The included methods can be roughly divided into groups: Rule-based Methods, Embedding Methods, Graph Neural Network Methods and Hybird Methods (relational reinforcement learning, differentiable reasoning, etc.). Experiments results are collected from original and follow-up papers, and best effort has been attempted to assure results of the same method are carried out with similar configurations, but do expect results under different settings being put around each other. 4 | 5 | #### Type Markers: 6 | - **E** as embedding methods 7 | - **R** as rule-based methods 8 | - **NN** as graph neural nets 9 | - Hybird Methods: 10 | - **E+NN** 11 | - **R+NN** 12 | - **R+E** 13 | - **R+E+NN** 14 | - **R+RL** : Rule + reinforcement learning 15 | 16 | Approach Format: 17 | - (Approach Type)-(Method Name)-(Published Year)-(Reference Index) 18 | 19 | #### Datasets 20 | - Large 21 | - FB15K 22 | - FB15K-237 23 | - NELL-995 24 | - WN18 25 | - WN18RR 26 | - YAGO3-10 27 | - YAGO37 28 | - Small/Medium 29 | - Kinship 30 | - UMLS 31 | - Citeseer 32 | - Cora 33 | - Pubmed 34 | - UWCSE 35 | 36 | #### Experiments Results 37 | 38 | > Mean Rank (MR) 39 | > 40 | 41 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 42 | |--|--|--|--|--|--|--|--| 43 | |**E**-TransE-2013-[9]|2100|-|323|-|2300|6.8|-| 44 | |**E**-DistMult-2015-[6]|4213|-|512|-|7000|5.26|-| 45 | |**E**-ANALOGY-2017-[8]|-|-|-|-|-|-|-| 46 | |**E**-SimpleE-2018-[1]|-|-|-|-|-|-|-| 47 | |**E**-ConvE-2018-[2]|3560|64|245|504|4464|2.03|- 48 | |**E**-ComplEx-N3-2018-[3]|4600|-|546|-|7882|2.48|-| 49 | |**E**-CrossE-2019-[7]|-|-|-|-|-|-|-| 50 | |**E**-Rotate-2019-[23]|-|**40**|**177**|**254**|2923|-|- 51 | |**R**-Node+LinkFeat-2015-[17]|-|-|-|-|-|-|-| 52 | |**R**-AMIE-2015-[10]|-|-|-|-|-|-|-| 53 | |**R**-Gaifman-2016-[24]|-|75|-|298|-|-|- 54 | |**R**-RuleN-2018-[11]|-|-|-|-|-|-|-| 55 | |**R**-RUGE-2018-[13]|-|-|-|-|-|-|-| 56 | |**R**-AnyBURL-2019-[12]|-|-|-|-|-|-|-| 57 | |**R+E**-RLvLR-2018-[15]|-|-|-|-|-|-|-| 58 | |**NN**-R-GCN-2017-[5]|-|-|-|-|-|-|-| 59 | |**E+NN**-R-GCN+\-2018-[4]|7600|-|600|-|6700|25.92|-| 60 | |**E+NN**-ConvKB-2018-[16]|**600**|-|216|-|**1295**|3.3|- 61 | |**E+NN**-AttentionE-2019-[18]|965|-|210|-|1940|**1.94**|- 62 | |**R+NN**-Neural LP-2017-[14]|-|-|-|-|-|-|-| 63 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|-|-| 64 | |**R+NN**-DRUM-2019-[22]|-|-|-|-|-|-|-| 65 | |**R+RL**-MINERVA-2018-[19]|-|-|-|-|-|-|-| 66 | |**R+RL**-Multi-Hop-2018-[21]|-|-|-|-|-|-|-| 67 | 68 | > Filter Mean Reciprocal Rank (Filter MRR) 69 | > 70 | 71 | The filtered MRR is concerned about the problem that when evaluating embedding methods, positive instances are used to generate negative instances (or testing sets) by corrupting its head and tail, however, in raw MRR, there is no guarantee that the generated instances never overlap with existing positive insatnces, thus introduce unfairness to the evaluation. Filter MRR simply remove any generated instances that are known as positive facts. This problem won't appear in the rule-based methods (or question-answering systems), so the MRR evaluation on rule-based methods is filter MRR by default. 72 | 73 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 74 | |--|--|--|--|--|--|--|--| 75 | |**E**-TransE-2013[9]|0.401|0.38|0.294|0.455|0.226|0.068|- 76 | |**E**-DistMult-2015-[6]|0.485|0.639|0.281|0.813|0.444|0.516|0.878 77 | |**E**-ANALOGY-2017-[8]|-|0.725|0.219|0.942|-|-|- 78 | |**E**-SimpleE-2018-[1]|-|0.727|-|0.942|-|-|-| 79 | |**E**-ConvE-2018-[2]|0.491|0.745|0.316|0.942|0.46|0.833|0.797 80 | |**E**-ComplEx-N3-2018-[3]|0.482|**0.86**|0.37|**0.95**|**0.48**|0.823|0.838| 81 | |**E**-CrossE-2019-[7]|-|0.728|0.299|0.83|-|-|- 82 | |**E**-Rotate-2019-[23]|-|0.799|0.338|0.949|0.476|-|- 83 | |**E**-IterE-2019-[26]|-|0.628|0.247|0.913|0.274|-|- 84 | |**R**-Node+LinkFeat-2015-[17]|-|0.821|0.237|0.94|-|-|- 85 | |**R**-AMIE-2015[10]|-|-|-|-|-|-|-| 86 | |**R**-Gaifman-2016-[24]|-|-|-|-|-|-|- 87 | |**R**-RuleN-2018-[11]|-|-|-|-|-|-|-| 88 | |**R**-RUGE-2018-[13]|-|0.768|-|-|-|-|- 89 | |**R**-AnyBURL-2019-[12]|-|0.83|0.31|**0.95**|**0.48**|-|-| 90 | |**R+E**-RLvLR-2018-[15]|-|-|0.24|-|-|-|- 91 | |**NN**-R-GCN-2017-[5]|-|0.651|0.248|0.814|-|-|- 92 | |**E+NN**-R-GCN+\-2018-[4]|-|0.696|0.249|0.819|-|-|-| 93 | |**E+NN**-ConvKB-2018-[16]|0.43|-|0.396|-|0.248|0.033|- 94 | |**E+NN**-AttentionE-2019-[18]|0.530|-|**0.518**|-|0.44|**0.904**|- 95 | |**R+NN**-Neural LP-2017-[14]|-|0.76|0.24|0.94|0.435|0.619|0.778 96 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.793|0.912 97 | |**R+NN**-DRUM-2019-[22]|-|-|0.343|-|**0.486**|0.61|0.81 98 | |**R+RL**-MINERVA-2018-[19]|0.725|-|0.293|-|0.448|0.720|0.825 99 | |**R+RL**-Multi-Hop-2018-[21]|**0.727**|-|0.407|-|0.472|0.878|**0.94** 100 | 101 | > Hits@1 102 | > 103 | 104 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 105 | |--|--|--|--|--|--|--|--| 106 | |**E**-TransE-2013[9]|0.344|0.231|0.198|0.089|0.0427|0.009|- 107 | |**E**-DistMult-2015-[6]|0.401|0.522|0.199|0.701|0.412|0.367|**0.916** 108 | |**E**-ANALOGY-2017-[8]|-|0.646|0.131|0.939|-|-|- 109 | |**E**-SimpleE-2018-[1]|-|0.660|-|0.939|-|-|- 110 | |**E**-ConvE-2018-[2]|0.403|0.67|0.239|0.935|0.39|0.738|0.894| 111 | |**E**-ComplEx-N3-2018-[3]|0.399|0.599|0.132|0.936|0.409|0.733|0.823| 112 | |**E**-CrossE-2019-[7]|-|0.634|0.211|0.741|-|-|- 113 | |**E**-Rotate-2019-[23]|-|0.75|0.241|0.944|0.428|-|-| 114 | |**E**-IterE-2019-[26]|-|0.551|0.179|0.891|0.254|-|-| 115 | |**R**-Node+LinkFeat-2015-[17]|-|-|-|-|-|-|-| 116 | |**R**-AMIE-2015[10]|-|0.647|0.174|0.872|0.358|-|- 117 | |**R**-Gaifman-2016-[24]|-|0.692|-|0.767|-|-|-| 118 | |**R**-RuleN-2018-[11]|-|0.772|0.182|0.945|0.427|-|- 119 | |**R**-RUGE-2018-[13]|-|0.703|-|-|-|-|- 120 | |**R**-AnyBURL-2019-[12]|-|**0.808**|0.233|**0.946**|**0.446**|-|- 121 | |**R+E**-RLvLR-2018-[15]|-|-|-|-|-|-|-| 122 | |**NN**-R-GCN-2017-[5]|-|0.541|0.153|0.686|-|-|- 123 | |**E+NN**-R-GCN+\-2018-[4]|-|0.601|0.151|0.697|-|-|- 124 | |**E+NN**-ConvKB-2018-[16]|0.37|-|0.198|-|0.0562|0.4362|-| 125 | |**E+NN**-AttentionE-2019-[18]|0.447|-|0.46|-|0.361|**0.859**|- 126 | |**R+NN**-Neural LP-2017-[14]|-|-|0.166|-|0.376|0.475|0.643| 127 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.759|0.843| 128 | |**R+NN**-DRUM-2019-[22]|-|-|0.255|-|0.425|0.46|0.67 129 | |**R+RL**-MINERVA-2018-[19]|**0.663**|-|0.217|-|0.413|0.605|0.728 130 | |**R+RL**-Multi-Hop-2018-[21]|0.656|-|**0.329**|-|0.437|0.811|0.902 131 | 132 | > Hits@3 133 | > 134 | 135 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 136 | |--|--|--|--|--|--|--|--| 137 | |**E**-TransE-2013[9]|0.472|0.472|0.376|0.823|0.441|0.643|- 138 | |**E**-DistMult-2015-[6]|0.524|0.718|0.301|0.921|0.47|0.581|0.967 139 | |**E**-ANALOGY-2017-[8]|-|0.785|0.240|0.944|-|-|-| 140 | |**E**-SimpleE-2018-[1]|-|0.773|-|0.944|-|-|- 141 | |**E**-ConvE-2018-[2]|0.531|0.801|0.35|0.947|0.43|0.917|0.964| 142 | |**E**-ComplEx-N3-2018-[3]|0.528|0.759|0.244|0.945|0.469|0.899|0.962| 143 | |**E**-CrossE-2019-[7]|-|0.802|0.331|0.931|-|-|- 144 | |**E**-Rotate-2019-[23]|-|**0.83**|0.375|**0.952**|0.492|-|-| 145 | |**E**-IterE-2019-[26]|-|0.673|0.262|0.935|0.281|-|-| 146 | |**R**-Node+LinkFeat-2015-[17]|-|-|-|-|-|-|-| 147 | |**R**-AMIE-2015[10]|-|-|-|-|-|-|-| 148 | |**R**-Gaifman-2016-[24]|-|-|-|-|-|-|- 149 | |**R**-RuleN-2018-[11]|-|-|-|-|-|-|-| 150 | |**R**-RUGE-2018-[13]|-|0.815|-|-|-|-|- 151 | |**R**-AnyBURL-2019-[12]|-|-|-|-|-|-|-| 152 | |**R+E**-RLvLR-2018-[15]|-|-|-|-|-|-|-| 153 | |**NN**-R-GCN-2017-[5]|-|0.736|0.258|0.928|-|-|- 154 | |**E+NN**-R-GCN+\-2018-[4]|0.126|0.760|0.264|0.929|0.137|0.088|- 155 | |**E+NN**-ConvKB-2018-[16]|0.47|-|0.324|-|0.445|0.755|-| 156 | |**E+NN**-AttentionE-2019-[18]|0.564|-|**0.54**|-|0.483|**0.941**|- 157 | |**R+NN**-Neural LP-2017-[14]|-|-|0.248|-|0.468|0.707|0.869| 158 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.798|**0.983** 159 | |**R+NN**-DRUM-2019-[22]|-|-|0.378|-|**0.513**|0.71|0.94 160 | |**R+RL**-MINERVA-2018-[19]|**0.773**|-|0.329|-|0.456|0.812|0.9| 161 | |**R+RL**-Multi-Hop-2018-[21]|-|-|-|-|-|-|-| 162 | 163 | > Hits@10 164 | > 165 | 166 | |Approach|NELL-995|FB15K|FB15K-237|WN18|WN18RR|Kinship|UMLS 167 | |--|--|--|--|--|--|--|--| 168 | |**E**-TransE-2013[9]|0.501|0.539|0.465|0.909|0.501|0.841|- 169 | |**E**-DistMult-2015-[6]|0.61|0.814|0.408|0.943|0.49|0.867|0.992 170 | |**E**-ANALOGY-2017-[8]|-|0.854|0.405|0.947|-|-|- 171 | |**E**-SimpleE-2018-[1]|-|0.838|-|0.947|-|-|- 172 | |**E**-ConvE-2018-[2]|0.613|0.873|0.491|0.955|0.48|0.9814|0.992| 173 | |**E**-ComplEx-N3-2018-[3]|0.606|0.91|0.56|0.96|0.57|0.9711|0.995 174 | |**E**-CrossE-2019-[7]|-|-|0.474|-|-|-|- 175 | |**E**-Rotate-2019-[23]|-|0.884|0.533|0.959|0.571|-|-| 176 | |**E**-IterE-2019-[26]|-|0.771|0.392|0.948|0.314|-|-| 177 | |**R**-Node+LinkFeat-2015-[17]|-|0.87|0.36|0.943|-|-|- 178 | |**R**-AMIE-2015[10]|-|0.858|0.409|0.948|0.388|-|- 179 | |**R**-Gaifman-2016-[24]|-|0.842|-|0.939|-|-|-|- 180 | |**R**-RuleN-2018-[11]|-|0.87|0.42|0.958|0.536|-|- 181 | |**R**-RUGE-2018-[13]|-|0.865|-|-|-|-|-| 182 | |**R**-AnyBURL-2019-[12]|-|**0.89**|0.486|0.959|0.555|-|- 183 | |**R+E**-RLvLR-2018-[15]|-|-|0.393|-|-|-|- 184 | |**NN**-R-GCN-2017-[5]|-|0.825|0.414|0.955|-|-|-| 185 | |**E+NN**-R-GCN+\-2018-[4]|0.188|0.842|0.417|**0.964**|0.08|0.239|-| 186 | |**E+NN**-ConvKB-2018-[16]|0.545|-|0.517|-|0.525|0.953|- 187 | |**E+NN**-AttentionE-2019-[18]|0.695|-|**0.626**|-|0.581|0.98|-| 188 | |**R+NN**-Neural LP-2017-[14]|-|0.837|0.362|0.945|0.566|0.912|0.962 189 | |**R+NN**-NTP lambda-2017-[20]|-|-|-|-|-|0.878|**1.0** 190 | |**R+NN**-DRUM-2019-[22]|-|-|0.516|-|**0.586**|0.92|0.98 191 | |**R+RL**-MINERVA-2018-[19]|0.831|-|0.456|-|0.513|0.924|0.968 192 | |**R+RL**-Multi-Hop-2018-[21]|**0.844**|-|0.564|-|0.542|**0.982**|0.992 193 | 194 | ### References 195 | [1] Kazemi, S. M., & Poole, D. (2018). SimplE Embedding for Link Prediction in Knowledge Graphs. NIPS, 1–12. 196 | 197 | [2] Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2018). Convolutional 2D Knowledge Graph Embeddings. AAAI, 1811–1818. 198 | 199 | [3] Lacroix, T., Usunier, N., & Obozinski, G. (2018). Canonical Tensor Decomposition for Knowledge Base Completion. ICML. 200 | 201 | [4] Schlichtkrull, M., Kipf, T. N., Bloem, P., Titov, I., & Welling, M. (2018). Modeling Relational Data with Graph Convolutional Networks. ESWC, 593–607. 202 | 203 | [5] Kipf, T. N., & Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. ICLR, 1–14. Retrieved from http://arxiv.org/abs/1609.02907 204 | 205 | [6] Yang, B., Yih, W., He, X., Gao, J., & Deng, L. (2015). Embedding entities and relations for learning and inference in knowledge bases. ICLR, 1–13. 206 | 207 | [7] Zhang, W., Paudel, B., Zhang, W., Bernstein, A., & Chen, H. (2019). Interaction Embeddings for Prediction and Explanation in Knowledge Graphs. WSDM. 208 | 209 | [8] Liu, H., Wu, Y., & Yang, Y. (2017). Analogical Inference for Multi-relational Embeddings. ICML. 210 | 211 | [9] Bordes, A., Usunier, N., Weston, J., & Yakhnenko, O. (2013). Translating Embeddings for Modeling Multi-Relational Data. NIPS, 2787–2795. https://doi.org/10.1007/s13398-014-0173-7.2 212 | 213 | [10] Galárraga, L., Teflioudi, C., Hose, K., & Suchanek, F. M. (2015). Fast rule mining in ontological knowledge bases with AMIE+. VLDB Journal, 24(6), 707–730. https://doi.org/10.1007/s00778-015-0394-1 214 | 215 | [11] Meilicke, C., Fink, M., Wang, Y., Ruffinelli, D., Gemulla, R., & Stuckenschmidt, H. (2018). Fine-grained evaluation of rule- and embedding-based systems for knowledge graph completion. ISWC, 3–20. https://doi.org/10.1007/978-3-030-00671-6_1 216 | 217 | [12] Meilicke, C., Chekol, M. W., Ruffinelli, D., & Stuckenschmidt, H. (2019). Anytime Bottom-Up Rule Learning for Knowledge Graph Completion. IJCAI, 3137–3143. 218 | 219 | [13] Guo, S., Wang, Q., Wang, L., Wang, B., & Guo, L. (2018). Knowledge Graph Embedding with Iterative Guidance from Soft Rules. AAAI, 4816–4823. 220 | 221 | [14] Yang, F., & Cohen, W. W. (2017). Differentiable Learning of Logical Rules for Knowledge Base Reasoning. NIPS. 222 | 223 | [15] Omran, P. G., Wang, K., & Wang, Z. (2018). Scalable rule learning via learning representation. IJCAI, 2149–2155. 224 | 225 | [16] Nguyen, D. Q., Nguyen, T. D., Nguyen, D. Q., & Phung, D. (2018). A Novel Embedding Model for Knowledge Base Completion. NAACL. 226 | 227 | [17] Toutanova, K., & Chen, D. (2015). Observed versus latent features for knowledge base and text inference. Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality, 57–66. https://doi.org/10.18653/v1/w15-4007 228 | 229 | [18] Nathani, D., Chauhan, J., Sharma, C., & Kaul, M. (2019). Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs. ACL. Retrieved from http://arxiv.org/abs/1906.01195 230 | 231 | [19] Das, R., Dhuliawala, S., Zaheer, M., Vilnis, L., Durugkar, I., Krishnamurthy, A., … Mccallum, A. (2018). Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. ICLR. 232 | 233 | [20] Rocktäschel, T., & Riedel, S. (2017). End-to-End Differentiable Proving. NIPS. Retrieved from http://arxiv.org/abs/1705.11040 234 | 235 | [21] Lin, X. V., Richard, S., & Caiming, X. (2018). Multi-Hop Knowledge Graph Reasoning with Reward Shaping. ACL, 3243–3253. 236 | 237 | [22] Sadeghian, A., Armandpour, M., Ding, P., & Wang, D. Z. (2019). DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs. NIPS, 1–13. Retrieved from http://arxiv.org/abs/1911.00055 238 | 239 | [23] Sun, Z., Deng, Z., Nie, J., & Tang, J. (2019). Rotate: Knowledge graph embedding by relational rotation in complex space. ICLR, 1–18. 240 | 241 | [24] Niepert, M. (2016). Discriminative gaifman models. NIPS, 3413–3421. 242 | 243 | [26] Zhang, W., Paudel, B., Wang, L., Chen, J., Zhu, H., Zhang, W., … Chen, H. (2019). Iteratively Learning Embeddings and Rules for Knowledge Graph Reasoning. WWW. 244 | 245 | ### Planned Addition 246 | 247 | [25] Zhu, Y., Liu, H., Wu, Z., Song, Y., & Zhang, T. (2019). Representation Learning with Ordered Relation Paths for Knowledge Graph Completion. EMNLP. 248 | --------------------------------------------------------------------------------