├── LICENSE ├── README.md └── notebooks ├── Duet.ipynb ├── data └── ngraphs.txt └── img └── duet-architecture.png /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Bhaskar Mitra 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # NDRM 2 | A repository for Neural Document Ranking Models. 3 | -------------------------------------------------------------------------------- /notebooks/Duet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# A Duet Neural Document Ranking Model\n", 8 | "\n", 9 | "This is a tutorial for implementing the Duet neural document ranking model using the [Microsoft Cognitive Tookit](https://www.microsoft.com/en-us/cognitive-toolkit/) as described in the following papers:\n", 10 | "\n", 11 | "1. [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://arxiv.org/abs/1610.08136). Bhaskar Mitra, Fernando Diaz, and Nick Craswell. In Proceedings of the 26th International Conference on World Wide Web, WWW – World Wide Web Consortium (W3C). April, 2017.\n", 12 | "\n", 13 | "2. [Benchmark for Complex Answer Retrieval](https://arxiv.org/abs/1705.04803). Federico Nanni, Bhaskar Mitra, Matt Magnusson, and Laura Dietz. In Proceedings of the 3rd International Conference on the Theory of Information Retrieval (ICTIR), ACM. October, 2017.\n", 14 | "\n", 15 | "Duet is a document ranking model composed of two separate deep neural networks, one that matches the query and the document using a _local representation_, and another that matches the query and the document using learned [_distributed representations_](http://web.stanford.edu/class/psych209a/ReadingsByDate/02_01/HintonMcCRumelhart86DistribRep.pdf). The two networks are jointly trained as part of a single neural network.\n", 16 | "\n", 17 | "**Note**: The Bing document ranking dataset used for training/evaluating the duet model in the [original paper](https://arxiv.org/abs/1610.08136) is a proprietary dataset that I **can not** share because of privacy considerations. However, the [TREC CAR dataset](http://trec-car.cs.unh.edu/) used in the [following paper](https://arxiv.org/abs/1705.04803) _is_ publicly available that you may want to consider if you are looking for a large labelled dataset for DNN benchmarking.\n", 18 | "\n", 19 | "\n", 20 | "### Change log\n", 21 | "**Jan 25, 2017**: The original Duet model was implemented on CNTK using [BrainScript](https://docs.microsoft.com/en-us/cognitive-toolkit/Using-CNTK-with-BrainScript) and [a custom CNTK reader](https://github.com/Microsoft/CNTK/tree/bmitra/NDRMReader/Source/Readers/NDRMReader). I have re-implemented the original model on CNTK v2 using the new [python API](https://www.cntk.ai/pythondocs/).\n", 22 | "\n", 23 | "**Nov 07, 2017**: Updated the script for [CNTK v2.2](https://docs.microsoft.com/en-us/cognitive-toolkit/releasenotes/cntk_2_2_release_notes).\n", 24 | "\n", 25 | "## Let's start\n", 26 | "\n", 27 | "We assume you have already installed CNTK on your machine following the steps outlined [here](https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-CNTK-on-your-machine). So we start begin by importing all the relevant modules that we will need later." 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": { 34 | "collapsed": false 35 | }, 36 | "outputs": [], 37 | "source": [ 38 | "from __future__ import print_function\n", 39 | "import sys\n", 40 | "import os\n", 41 | "import csv\n", 42 | "import re\n", 43 | "import time\n", 44 | "import math\n", 45 | "import numpy as np\n", 46 | "import cntk as C\n", 47 | "\n", 48 | "# uncomment the following if you want to run on cpu\n", 49 | "#C.try_set_default_device(C.cpu(), acquire_device_lock=False)\n", 50 | "C.cntk_py.set_fixed_random_seed(1)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "Using the `set_default_device` method we can set the code to run on either CPU or GPU.\n", 58 | "\n", 59 | "## Data reader \n", 60 | "Our Duet implementation accepts data in the format of a tab-separated text file, where the first few columns (optionally) contains some meta-information about each sample (e.g., query or document ID), followed by a column containing query text, and finally some fixed number of columns containing (body) text from the different candidate documents to be ranked for that query. \n", 61 | "\n", 62 | "For example, a training data file may look like the following,\n", 63 | "\n", 64 | "\n", 65 | " \n", 66 | " \n", 67 | " \n", 68 | "
some queryrelevant document body textnon-relevant document body text
another queryanother relevant document body textanother non-relevant document body text
yet another queryyet another relevant document body textyet another non-relevant document body text
\n", 69 | "\n", 70 | "At test time, we will need to deal with a variable number of documents per query. So it may be easier to put a single query-document pair per line along with corresponding query / document IDs. For example,\n", 71 | "\n", 72 | "\n", 73 | " \n", 74 | " \n", 75 | " \n", 76 | " \n", 77 | " \n", 78 | "
first query IDID for document Arating for document Afirst querydocument A body text
first query IDID for document Brating for document Bfirst querydocument B body text
first query IDID for document Crating for document Cfirst querydocument C body text
second query IDID for document Drating for document Dsecond querydocument D body text
second query IDID for document Erating for document Esecond querydocument E body text
\n", 79 | "\n", 80 | "Note that the `DataReader` below returns input features for both the _local_ and the _distributed_ subnetworks of the Duet model." 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": false 88 | }, 89 | "outputs": [], 90 | "source": [ 91 | "class Sample:\n", 92 | " \n", 93 | " def __init__(self):\n", 94 | " self.query = \"\"\n", 95 | " self.docs = []\n", 96 | " \n", 97 | "class DataReader:\n", 98 | " max_query_words = 10\n", 99 | " max_doc_words = 1000\n", 100 | " \n", 101 | " def __init__(self, data_file, ngraphs_file, num_docs, num_meta_cols, multi_pass):\n", 102 | " self.__load_ngraphs(ngraphs_file)\n", 103 | " self.data_file = open(data_file, mode='r')\n", 104 | " self.num_docs = num_docs\n", 105 | " self.num_meta_cols = num_meta_cols\n", 106 | " self.multi_pass = multi_pass\n", 107 | " \n", 108 | " def __load_ngraphs(self, filename):\n", 109 | " self.ngraphs = {}\n", 110 | " self.max_ngraph_len = 0\n", 111 | " with open(filename, mode='r') as f:\n", 112 | " reader = csv.reader(f, delimiter='\\t')\n", 113 | " for row in reader:\n", 114 | " self.ngraphs[row[0]] = int(row[1]) - 1\n", 115 | " self.max_ngraph_len = max(self.max_ngraph_len, len(row[0]))\n", 116 | " self.num_ngraphs = len(self.ngraphs)\n", 117 | "\n", 118 | " def __read_samples(self, num_samples):\n", 119 | " labels = np.zeros((num_samples, self.num_docs), dtype=np.float32)\n", 120 | " samples = []\n", 121 | " meta = []\n", 122 | " mb_size = 0\n", 123 | " for i in range(num_samples):\n", 124 | " row = self.data_file.readline()\n", 125 | " if row == \"\":\n", 126 | " if self.multi_pass:\n", 127 | " self.data_file.seek(0)\n", 128 | " row = self.data_file.readline()\n", 129 | " else:\n", 130 | " break\n", 131 | " cols = row.split('\\t')\n", 132 | " curr_sample = Sample()\n", 133 | " curr_sample.query = re.sub('[^0-9a-z\\t]+', ' ', cols[self.num_meta_cols].lower()).strip()\n", 134 | " for j in range(self.num_meta_cols+1, min(self.num_meta_cols+self.num_docs+1, len(cols))):\n", 135 | " curr_sample.docs.append(re.sub('[^0-9a-z\\t]+', ' ', cols[j].lower()).strip())\n", 136 | " samples.append(curr_sample)\n", 137 | " labels[i][0] = np.float32(1)\n", 138 | " meta.append([cols[i] for i in range(0, self.num_meta_cols)])\n", 139 | " mb_size += 1\n", 140 | " return samples, labels, meta, mb_size\n", 141 | " \n", 142 | " def __get_interaction_features(self, samples):\n", 143 | " features = np.zeros((len(samples), self.num_docs, self.max_query_words, self.max_doc_words), dtype=np.float32)\n", 144 | " for sample_idx, sample in enumerate(samples):\n", 145 | " for doc_idx, doc in enumerate(sample.docs):\n", 146 | " for qw_idx, qword in enumerate(sample.query.split()):\n", 147 | " if qw_idx == self.max_query_words:\n", 148 | " break\n", 149 | " for dw_idx, dword in enumerate(doc.split()):\n", 150 | " if dw_idx == self.max_doc_words:\n", 151 | " break\n", 152 | " if qword == dword:\n", 153 | " features[sample_idx, doc_idx, qw_idx, dw_idx] = np.float32(1)\n", 154 | " return features\n", 155 | " \n", 156 | " def __get_ngraph_features(self, samples):\n", 157 | " features_query = np.zeros((len(samples), self.num_ngraphs, self.max_query_words), dtype=np.float32)\n", 158 | " features_docs = np.zeros((len(samples), self.num_docs, self.num_ngraphs, self.max_doc_words), dtype=np.float32)\n", 159 | " for sample_idx, sample in enumerate(samples):\n", 160 | " # loop over query and docs -- doc_idx = 0 corresponds to query \n", 161 | " for doc_idx in range(len(sample.docs)+1):\n", 162 | " doc = sample.query if doc_idx == 0 else sample.docs[doc_idx-1]\n", 163 | " max_words = self.max_query_words if doc_idx == 0 else self.max_doc_words\n", 164 | " for w_idx, word in enumerate(doc.split()):\n", 165 | " if w_idx == max_words:\n", 166 | " break\n", 167 | " token = '#' + word + '#'\n", 168 | " token_len = len(token)\n", 169 | " for i in range(token_len):\n", 170 | " for j in range(1, self.max_ngraph_len + 1):\n", 171 | " if i+j < token_len:\n", 172 | " ngraph_idx = self.ngraphs.get(token[i:i+j])\n", 173 | " if ngraph_idx != None:\n", 174 | " if doc_idx == 0:\n", 175 | " features_query[sample_idx, ngraph_idx, w_idx] += 1\n", 176 | " else:\n", 177 | " features_docs[sample_idx, doc_idx-1, ngraph_idx, w_idx] += 1\n", 178 | " return features_query, features_docs\n", 179 | "\n", 180 | " def get_minibatch(self, num_samples):\n", 181 | " samples, labels, meta, mb_size = self.__read_samples(num_samples)\n", 182 | " features_local = self.__get_interaction_features(samples)\n", 183 | " features_distrib_query, features_distrib_docs = self.__get_ngraph_features(samples)\n", 184 | " return features_local, features_distrib_query, features_distrib_docs, labels, meta, mb_size" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "## The model\n", 192 | "\n", 193 | "The figure below provides a detailed schematic view of the duet architecture. The distributed model projects the query and the document text into an embedding space before matching, while the local model operates over an interaction matrix comparing every query term to every document term. The final score under the duet setup is the sum of scores from the local and the distributed networks,\n", 194 | "\n", 195 | "$$f(Q, D) = f_l(Q, D) + f_d(Q, D)$$\n", 196 | "\n", 197 | "Where both the query and the document are considered as ordered list of terms. Each query term q and document term d is a m X 1 vector where m is the input representation of the text (e.g. the number of terms in the vocabulary for the local model).\n", 198 | "\n", 199 | "![Duet architecture](img/duet-architecture.png)" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "metadata": { 206 | "collapsed": false 207 | }, 208 | "outputs": [], 209 | "source": [ 210 | "def duet(features_local, features_distrib_query, features_distrib_docs, num_ngraphs, words_per_query, words_per_doc, num_docs):\n", 211 | " num_hidden_nodes = 300\n", 212 | " word_window_size = 3\n", 213 | " pooling_kernel_width_query = words_per_query - word_window_size + 1 # = 8\n", 214 | " pooling_kernel_width_doc = 100\n", 215 | " num_pooling_windows_doc = ((words_per_doc - word_window_size + 1) - pooling_kernel_width_doc) + 1 # = 899\n", 216 | " \n", 217 | " duet_local = C.layers.Sequential ([\n", 218 | " C.layers.Convolution((1, words_per_doc), num_hidden_nodes, activation=C.tanh, strides=(1, 1), pad=False),\n", 219 | " C.layers.Dense(num_hidden_nodes, activation=C.tanh),\n", 220 | " C.layers.Dense(num_hidden_nodes, activation=C.tanh),\n", 221 | " C.layers.Dropout(0.2),\n", 222 | " C.layers.Dense(1, activation=C.tanh)])\n", 223 | " \n", 224 | " duet_embed_q = C.layers.Sequential ([\n", 225 | " C.layers.Convolution((word_window_size, 1), num_hidden_nodes, activation=C.tanh, strides=(1, 1), pad=False),\n", 226 | " C.layers.MaxPooling((pooling_kernel_width_query, 1), strides=(1, 1), pad=False),\n", 227 | " C.layers.Dense(num_hidden_nodes, activation=C.tanh)])\n", 228 | " \n", 229 | " duet_embed_d = C.layers.Sequential ([\n", 230 | " C.layers.Convolution((word_window_size, 1), num_hidden_nodes, activation=C.tanh, strides=(1, 1), pad=False),\n", 231 | " C.layers.MaxPooling((pooling_kernel_width_doc, 1), strides=(1, 1), pad=False),\n", 232 | " C.layers.Convolution((1, 1), num_hidden_nodes, activation=C.tanh, strides=(1, 1), pad=False)])\n", 233 | " \n", 234 | " duet_distrib = C.layers.Sequential ([\n", 235 | " C.layers.Dense(num_hidden_nodes, activation=C.tanh),\n", 236 | " C.layers.Dense(num_hidden_nodes, activation=C.tanh),\n", 237 | " C.layers.Dropout(0.2),\n", 238 | " C.layers.Dense(1, activation=C.tanh)])\n", 239 | " \n", 240 | " net_local = [C.slice(features_local, 0, idx, idx+1) for idx in range(0, num_docs)]\n", 241 | " net_local = [C.reshape(d, (1, words_per_query, words_per_doc)) for d in net_local]\n", 242 | " net_local = [duet_local(d) for d in net_local]\n", 243 | " net_local = [C.reshape(d, (1, 1)) for d in net_local]\n", 244 | " net_local = C.splice(*net_local)\n", 245 | " \n", 246 | " net_distrib_q = C.reshape(features_distrib_query, (num_ngraphs, words_per_query, 1))\n", 247 | " net_distrib_q = duet_embed_q(net_distrib_q)\n", 248 | " net_distrib_q = [net_distrib_q for idx in range(0, num_pooling_windows_doc)]\n", 249 | " net_distrib_q = C.splice(*net_distrib_q)\n", 250 | " net_distrib_q = C.reshape(net_distrib_q, (num_hidden_nodes * num_pooling_windows_doc, 1))\n", 251 | " \n", 252 | " net_distrib_d = [C.slice(features_distrib_docs, 0, idx, idx+1) for idx in range(0, num_docs)]\n", 253 | " net_distrib_d = [C.reshape(d, (num_ngraphs, words_per_doc, 1)) for d in net_distrib_d]\n", 254 | " net_distrib_d = [duet_embed_d(d) for d in net_distrib_d]\n", 255 | " net_distrib_d = [C.reshape(d, (num_hidden_nodes * num_pooling_windows_doc, 1)) for d in net_distrib_d]\n", 256 | "\n", 257 | " net_distrib = [C.element_times(net_distrib_q, d) for d in net_distrib_d]\n", 258 | " net_distrib = [duet_distrib(d) for d in net_distrib]\n", 259 | " net_distrib = [C.reshape(d, (1, 1)) for d in net_distrib]\n", 260 | " net_distrib = C.splice(*net_distrib)\n", 261 | " \n", 262 | " net = C.plus(net_local, net_distrib)\n", 263 | " \n", 264 | " return net" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "## Train and evaluate\n", 272 | "\n", 273 | "Next, we define the train and eval methods. This includes specifying the loss function for training and other hyper-parameters such as minibatch size and learning rate. Remember to try a few different scales for the learning rate to make sure it's not too large or too small (I typically run with 0.5, 0.1, 0.01, 0.001, and so on)." 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": { 280 | "collapsed": false 281 | }, 282 | "outputs": [], 283 | "source": [ 284 | "def train(train_file, ngraphs_file, num_docs, num_meta_cols):\n", 285 | " \n", 286 | " # initialize train data readers\n", 287 | " reader_train = DataReader(train_file, ngraphs_file, num_docs, num_meta_cols, True)\n", 288 | " \n", 289 | " # input variables denoting the features and label data\n", 290 | " features_local = C.input_variable((reader_train.num_docs, reader_train.max_query_words, reader_train.max_doc_words), np.float32)\n", 291 | " features_distrib_query = C.input_variable((reader_train.num_ngraphs, reader_train.max_query_words), np.float32)\n", 292 | " features_distrib_docs = C.input_variable((reader_train.num_docs, reader_train.num_ngraphs, reader_train.max_doc_words), np.float32)\n", 293 | " labels = C.input_variable((reader_train.num_docs), np.float32)\n", 294 | "\n", 295 | " # Instantiate the Duet neural document ranking model and specify loss function\n", 296 | " z = duet(features_local, features_distrib_query, features_distrib_docs, reader_train.num_ngraphs, reader_train.max_query_words, reader_train.max_doc_words, reader_train.num_docs)\n", 297 | " ce = C.cross_entropy_with_softmax(z, labels)\n", 298 | " pe = C.classification_error(z, labels)\n", 299 | "\n", 300 | " # Instantiate the trainer object to drive the model training\n", 301 | " lr_per_minibatch = C.learning_rate_schedule(0.001, C.UnitType.minibatch)\n", 302 | " progress_printers = [C.logging.ProgressPrinter(freq=100, tag='Training', gen_heartbeat=False)]\n", 303 | " trainer = C.Trainer(z, (ce, pe), [C.sgd(z.parameters, lr=lr_per_minibatch)], progress_printers)\n", 304 | "\n", 305 | " # Get minibatches of training data and perform model training\n", 306 | " minibatch_size = 64\n", 307 | " minibatches_per_epoch = 32\n", 308 | " epochs = 4\n", 309 | " \n", 310 | " C.logging.log_number_of_parameters(ce)\n", 311 | " print()\n", 312 | " \n", 313 | " for i in range(epochs):\n", 314 | " for j in range(minibatches_per_epoch):\n", 315 | " train_features_local, train_features_distrib_query, train_features_distrib_docs, train_labels, train_meta, actual_mb_size = reader_train.get_minibatch(minibatch_size)\n", 316 | " trainer.train_minibatch({features_local : train_features_local, features_distrib_query : train_features_distrib_query, features_distrib_docs : train_features_distrib_docs, labels : train_labels})\n", 317 | " trainer.summarize_training_progress()\n", 318 | "\n", 319 | " return z\n", 320 | "\n", 321 | "def eval(model, test_file, ngraphs_file, num_docs, num_meta_cols, score_file):\n", 322 | " \n", 323 | " minibatch_size = 64\n", 324 | " actual_mb_size = minibatch_size\n", 325 | " \n", 326 | " # initialize test data readers\n", 327 | " reader_test = DataReader(test_file, ngraphs_file, num_docs, num_meta_cols, False)\n", 328 | "\n", 329 | " with open(score_file, mode='w') as f:\n", 330 | " while(actual_mb_size == minibatch_size):\n", 331 | " test_features_local, test_features_distrib_query, test_features_distrib_docs, test_labels, test_meta, actual_mb_size = reader_test.get_minibatch(minibatch_size)\n", 332 | " if actual_mb_size > 0:\n", 333 | " result = model.eval({model.arguments[0] : test_features_local, model.arguments[1] : test_features_distrib_query, model.arguments[2] : test_features_distrib_docs})\n", 334 | " result = result.reshape((actual_mb_size, num_docs))\n", 335 | " result = [row[0] for row in result]\n", 336 | " for idx in range(actual_mb_size):\n", 337 | " f.write(\"{}\\t{}\\t{}\\t{}\\n\".format(test_meta[idx][0], test_meta[idx][1], test_meta[idx][2], result[idx]))" 338 | ] 339 | }, 340 | { 341 | "cell_type": "markdown", 342 | "metadata": {}, 343 | "source": [ 344 | "## Compute metric\n", 345 | "\n", 346 | "After we run the eval step to score all the query-document pairs in the test data we need to compute our metric-of-choice over the eval output. We use NDCG for this tutorial." 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": null, 352 | "metadata": { 353 | "collapsed": true 354 | }, 355 | "outputs": [], 356 | "source": [ 357 | "def ComputeDCG(sorted_ranks):\n", 358 | " dcg = 0\n", 359 | " for pos, rating in enumerate(sorted_ranks):\n", 360 | " dcg += ((2^rating - 1)/math.log2(pos + 2))\n", 361 | " return dcg\n", 362 | "\n", 363 | "def ComputeNDCGPerQuery(ideal_ratings, scored_ratings):\n", 364 | " ideal_ranks = sorted(ideal_ratings, reverse=True)\n", 365 | " model_ranks = [pair[1] for pair in sorted(scored_ratings, key=lambda tup: tup[0], reverse=True)]\n", 366 | " ideal_dcg = ComputeDCG(ideal_ranks)\n", 367 | " model_dcg = ComputeDCG(model_ranks)\n", 368 | " return model_dcg / ideal_dcg if ideal_dcg > 0 else 0\n", 369 | "\n", 370 | "def ComputeNDCG(score_file, ndcg_pos):\n", 371 | " ndcg = 0\n", 372 | " curr_qid = -1\n", 373 | " ideal_ratings = []\n", 374 | " scored_ratings = []\n", 375 | " q_count = 0\n", 376 | " \n", 377 | " with open(score_file, mode='r') as f:\n", 378 | " reader = csv.reader(f, delimiter='\\t')\n", 379 | " for row in reader:\n", 380 | " q_id = row[0]\n", 381 | " doc_id = row[1]\n", 382 | " rating = int(row[1])\n", 383 | " score = float(row[1])\n", 384 | " \n", 385 | " if q_id != curr_qid:\n", 386 | " ndcg += ComputeNDCGPerQuery(ideal_ratings, scored_ratings)\n", 387 | " q_count += 1\n", 388 | " ideal_ratings = []\n", 389 | " scored_ratings = []\n", 390 | " \n", 391 | " curr_qid = q_id\n", 392 | " ideal_ratings.append(rating)\n", 393 | " scored_ratings.append((score, rating))\n", 394 | " \n", 395 | " ndcg += ComputeNDCGPerQuery(ideal_ratings, scored_ratings)\n", 396 | " q_count += 1\n", 397 | " \n", 398 | " return ndcg / q_count" 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "metadata": {}, 404 | "source": [ 405 | "## Run\n", 406 | "Finally, we are ready to put all the pieces of the puzzle together and train and evaluate our Duet model. Woohoo!" 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": { 413 | "collapsed": false, 414 | "scrolled": true 415 | }, 416 | "outputs": [], 417 | "source": [ 418 | "ngraphs_file = \"data\\\\ngraphs.txt\"\n", 419 | "train_file = \"data\\\\train.txt\"\n", 420 | "test_file = \"data\\\\test.txt\"\n", 421 | "score_file = \"data\\\\score.txt\"\n", 422 | "\n", 423 | "model = train(train_file, ngraphs_file, 2, 0)\n", 424 | "eval(model, test_file, ngraphs_file, 2, 3, score_file)\n", 425 | "ndcg = ComputeNDCG(score_file, 10)\n", 426 | "print(\"test ndcg = {}\".format(ndcg))" 427 | ] 428 | } 429 | ], 430 | "metadata": { 431 | "anaconda-cloud": {}, 432 | "kernelspec": { 433 | "display_name": "Python [cntk-py34]", 434 | "language": "python", 435 | "name": "Python [cntk-py34]" 436 | }, 437 | "language_info": { 438 | "codemirror_mode": { 439 | "name": "ipython", 440 | "version": 3 441 | }, 442 | "file_extension": ".py", 443 | "mimetype": "text/x-python", 444 | "name": "python", 445 | "nbconvert_exporter": "python", 446 | "pygments_lexer": "ipython3", 447 | "version": "3.4.4" 448 | } 449 | }, 450 | "nbformat": 4, 451 | "nbformat_minor": 0 452 | } 453 | -------------------------------------------------------------------------------- /notebooks/data/ngraphs.txt: -------------------------------------------------------------------------------- 1 | a 1 2 | e 2 3 | i 3 4 | r 4 5 | s 5 6 | n 6 7 | o 7 8 | t 8 9 | l 9 10 | c 10 11 | d 11 12 | m 12 13 | u 13 14 | h 14 15 | p 15 16 | b 16 17 | g 17 18 | 1 18 19 | 0 19 20 | k 20 21 | 2 21 22 | y 22 23 | f 23 24 | 3 24 25 | er 25 26 | 5 26 27 | s# 27 28 | 4 28 29 | v 29 30 | w 30 31 | 6 31 32 | 8 32 33 | 7 33 34 | an 34 35 | 9 35 36 | e# 36 37 | in 37 38 | #s 38 39 | ar 39 40 | on 40 41 | re 41 42 | en 42 43 | n# 43 44 | a# 44 45 | es 45 46 | #a 46 47 | te 47 48 | ra 48 49 | #c 49 50 | al 50 51 | st 51 52 | #m 52 53 | j 53 54 | z 54 55 | le 55 56 | at 56 57 | or 57 58 | ri 58 59 | x 59 60 | #b 60 61 | ma 61 62 | ti 62 63 | #p 63 64 | r# 64 65 | li 65 66 | ne 66 67 | la 67 68 | #t 68 69 | ro 69 70 | t# 70 71 | ch 71 72 | el 72 73 | ta 73 74 | #d 74 75 | de 75 76 | is 76 77 | #1 77 78 | nt 78 79 | ic 79 80 | as 80 81 | ng 81 82 | he 82 83 | ha 83 84 | 00 84 85 | it 85 86 | to 86 87 | co 87 88 | me 88 89 | ll 89 90 | th 90 91 | y# 91 92 | #r 92 93 | ca 93 94 | se 94 95 | na 95 96 | et 96 97 | d# 97 98 | nd 98 99 | am 99 100 | i# 100 101 | 0# 101 102 | ea 102 103 | il 103 104 | ni 104 105 | ac 105 106 | o# 106 107 | lo 107 108 | l# 108 109 | #l 109 110 | #f 110 111 | ed 111 112 | #g 112 113 | ol 113 114 | #h 114 115 | #e 115 116 | tr 116 117 | si 117 118 | #k 118 119 | ie 119 120 | q 120 121 | di 121 122 | io 122 123 | ad 123 124 | 1# 124 125 | ve 125 126 | ho 126 127 | ia 127 128 | sh 128 129 | #n 129 130 | ur 130 131 | sa 131 132 | ns 132 133 | da 133 134 | us 134 135 | er# 135 136 | 10 136 137 | #2 137 138 | mi 138 139 | ec 139 140 | g# 140 141 | ge 141 142 | ou 142 143 | #i 143 144 | hi 144 145 | pa 145 146 | om 146 147 | un 147 148 | rt 148 149 | be 149 150 | ba 150 151 | os 151 152 | ce 152 153 | 2# 153 154 | ss 154 155 | #w 155 156 | pe 156 157 | rs 157 158 | mo 158 159 | 01 159 160 | ing 160 161 | 20 161 162 | ee 162 163 | no 163 164 | m# 164 165 | vi 165 166 | ot 166 167 | em 167 168 | 3# 168 169 | so 169 170 | 5# 170 171 | h# 171 172 | ai 172 173 | es# 173 174 | ke 174 175 | id 175 176 | 4# 176 177 | oo 177 178 | ga 178 179 | ir 179 180 | pr 180 181 | k# 181 182 | sc 182 183 | c# 183 184 | po 184 185 | do 185 186 | ab 186 187 | 6# 187 188 | #j 188 189 | ag 189 190 | 7# 190 191 | ts 191 192 | ck 192 193 | #o 193 194 | 12 194 195 | 8# 195 196 | #v 196 197 | bo 197 198 | ka 198 199 | 9# 199 200 | tt 200 201 | ap 201 202 | ter 202 203 | im 203 204 | nc 204 205 | on# 205 206 | ion 206 207 | #0 207 208 | ki 208 209 | ig 209 210 | ut 210 211 | op 211 212 | ci 212 213 | 11 213 214 | #3 214 215 | 50 215 216 | ul 216 217 | wa 217 218 | sp 218 219 | #ma 219 220 | bi 220 221 | fi 221 222 | 02 222 223 | su 223 224 | oc 224 225 | ent 225 226 | od 226 227 | rd 227 228 | and 228 229 | 30 229 230 | ng# 230 231 | 15 231 232 | ct 232 233 | ay 233 234 | va 234 235 | ak 235 236 | pi 236 237 | tu 237 238 | 13 238 239 | the 239 240 | ow 240 241 | 14 241 242 | an# 242 243 | ru 243 244 | au 244 245 | br 245 246 | gr 246 247 | bl 247 248 | 03 248 249 | nn 249 250 | we 250 251 | 16 251 252 | ei 252 253 | #4 253 254 | lu 254 255 | fo 255 256 | 40 256 257 | #co 257 258 | 21 258 259 | fa 259 260 | ly 260 261 | #u 261 262 | 60 262 263 | ep 263 264 | 80 264 265 | p# 265 266 | gi 266 267 | um 267 268 | ers 268 269 | cr 269 270 | 05 270 271 | 19 271 272 | 04 272 273 | go 273 274 | wi 274 275 | fe 275 276 | 25 276 277 | 18 277 278 | eb 278 279 | ry 279 280 | pl 280 281 | tio 281 282 | 23 282 283 | av 283 284 | ati 284 285 | og 285 286 | u# 286 287 | #5 287 288 | bu 288 289 | 06 289 290 | rm 290 291 | 22 291 292 | ev 292 293 | ls 293 294 | 08 294 295 | ov 295 296 | b# 296 297 | 07 297 298 | eg 298 299 | ty 299 300 | rr 300 301 | mp 301 302 | tion 302 303 | 24 303 304 | dr 304 305 | 09 305 306 | iv 306 307 | rc 307 308 | 17 308 309 | 70 309 310 | en# 310 311 | x# 311 312 | 31 312 313 | #6 313 314 | 90 314 315 | ing# 315 316 | ph 316 317 | #re 317 318 | #ca 318 319 | rn 319 320 | ine 320 321 | ew 321 322 | cl 322 323 | lt 323 324 | 32 324 325 | ld 325 326 | ds 326 327 | of 327 328 | ya 328 329 | #7 329 330 | ey 330 331 | man 331 332 | ob 332 333 | est 333 334 | sta 334 335 | ko 335 336 | #9 336 337 | lin 337 338 | ste 338 339 | ex 339 340 | sk 340 341 | 41 341 342 | ip 342 343 | le# 343 344 | 51 344 345 | 35 345 346 | ate 346 347 | ja 347 348 | #8 348 349 | 26 349 350 | ah 350 351 | 27 351 352 | ue 352 353 | all 353 354 | rl 354 355 | ne# 355 356 | 52 356 357 | 28 357 358 | 36 358 359 | 000 359 360 | 61 360 361 | #ch 361 362 | ed# 362 363 | mu 363 364 | 45 364 365 | f# 365 366 | 33 366 367 | 42 367 368 | 34 368 369 | ik 369 370 | ef 370 371 | tra 371 372 | cu 372 373 | fr 373 374 | uc 374 375 | #th 375 376 | gu 376 377 | ver 377 378 | #de 378 379 | sm 379 380 | rg 380 381 | ill 381 382 | 29 382 383 | gh 383 384 | w# 384 385 | #st 385 386 | aa 386 387 | ys 387 388 | ell 388 389 | wo 389 390 | hu 390 391 | rk 391 392 | af 392 393 | 81 393 394 | res 394 395 | ui 395 396 | ist 396 397 | qu 397 398 | 65 398 399 | 53 399 400 | 55 400 401 | #pa 401 402 | in# 402 403 | 97 403 404 | te# 404 405 | if 405 406 | du 406 407 | ran 407 408 | #ba 408 409 | 43 409 410 | 78 410 411 | z# 411 412 | 00# 412 413 | 72 413 414 | 91 414 415 | rs# 415 416 | 62 416 417 | 37 417 418 | che 418 419 | mb 419 420 | 71 420 421 | 56 421 422 | nk 422 423 | eo 423 424 | 46 424 425 | 38 425 426 | #pr 426 427 | ant 427 428 | 63 428 429 | per 429 430 | cha 430 431 | 54 431 432 | #in 432 433 | art 433 434 | 44 434 435 | 75 435 436 | up 436 437 | 82 437 438 | al# 438 439 | ess 439 440 | 39 440 441 | pp 441 442 | ff 442 443 | 85 443 444 | eri 444 445 | 99 445 446 | 48 446 447 | 64 447 448 | #sa 448 449 | 47 449 450 | tor 450 451 | mm 451 452 | cc 452 453 | 57 453 454 | ok 454 455 | #z 455 456 | ms 456 457 | 92 457 458 | pro 458 459 | con 459 460 | fl 460 461 | her 461 462 | ts# 462 463 | 73 463 464 | 74 464 465 | lan 465 466 | ub 466 467 | aw 467 468 | der 468 469 | ze 469 470 | ua 470 471 | tc 471 472 | 58 472 473 | 98 473 474 | 76 474 475 | ud 475 476 | re# 476 477 | mar 477 478 | 49 478 479 | 95 479 480 | #be 480 481 | 59 481 482 | lle 482 483 | 67 483 484 | ns# 484 485 | 66 485 486 | nte 486 487 | gl 487 488 | 83 488 489 | 96 489 490 | 88 490 491 | 86 491 492 | ib 492 493 | 69 493 494 | #mi 494 495 | ks 495 496 | 93 496 497 | 77 497 498 | 68 498 499 | ek 499 500 | za 500 501 | ps 501 502 | 84 502 503 | sl 503 504 | #me 504 505 | 87 505 506 | tin 506 507 | #mo 507 508 | one 508 509 | iz 509 510 | 89 510 511 | ht 511 512 | ons 512 513 | #y 513 514 | tl 514 515 | oa 515 516 | ion# 516 517 | 94 517 518 | v# 518 519 | rb 519 520 | oi 520 521 | ny 521 522 | #ha 522 523 | 79 523 524 | men 524 525 | str 525 526 | hr 526 527 | ae 527 528 | yo 528 529 | nu 529 530 | #se 530 531 | ast 531 532 | #sh 532 533 | #an 533 534 | ia# 534 535 | ari 535 536 | han 536 537 | jo 537 538 | ali 538 539 | ang 539 540 | int 540 541 | pu 541 542 | us# 542 543 | rin 543 544 | for 544 545 | az 545 546 | #al 546 547 | ara 547 548 | je 548 549 | sto 549 550 | gs 550 551 | rea 551 552 | ica 552 553 | chi 553 554 | ort 554 555 | ana 555 556 | #su 556 557 | ere 557 558 | #di 558 559 | #la 559 560 | are 560 561 | car 561 562 | ber 562 563 | les 563 564 | era 564 565 | #tr 565 566 | ar# 566 567 | dd 567 568 | ide 568 569 | ye 569 570 | atio 570 571 | st# 571 572 | ard 572 573 | vo 573 574 | ian 574 575 | is# 575 576 | nde 576 577 | rp 577 578 | eu 578 579 | ft 579 580 | #10 580 581 | pt 581 582 | ame 582 583 | sti 583 584 | ani 584 585 | eh 585 586 | my 586 587 | har 587 588 | ation 588 589 | #po 589 590 | #ka 590 591 | nt# 591 592 | yl 592 593 | #the 593 594 | ug 594 595 | nes 595 596 | sch 596 597 | fu 597 598 | min 598 599 | #da 599 600 | ina 600 601 | as# 601 602 | ic# 602 603 | com 603 604 | zi 604 605 | #ra 605 606 | ale 606 607 | ton 607 608 | ite 608 609 | lm 609 610 | #te 610 611 | cs 611 612 | #li 612 613 | 100 613 614 | son 614 615 | ric 615 616 | nf 616 617 | ers# 617 618 | nl 618 619 | oe 619 620 | ku 620 621 | ive 621 622 | mer 622 623 | #pe 623 624 | #x 624 625 | hy 625 626 | tri 626 627 | #br 627 628 | #bo 628 629 | tic 629 630 | par 630 631 | sha 631 632 | ele 632 633 | ch# 633 634 | rat 634 635 | ert 635 636 | lc 636 637 | ren 637 638 | na# 638 639 | ner 639 640 | ass 640 641 | yc 641 642 | sb 642 643 | ore 643 644 | rv 644 645 | yn 645 646 | #ta 646 647 | sw 647 648 | end 648 649 | #ho 649 650 | ris 650 651 | se# 651 652 | mc 652 653 | #ro 653 654 | ce# 654 655 | sy 655 656 | ree 656 657 | age 657 658 | ss# 658 659 | ect 659 660 | ind 660 661 | ala 661 662 | ser 662 663 | #20 663 664 | #so 664 665 | tal 665 666 | ler 666 667 | ach 667 668 | tion# 668 669 | 200 669 670 | #ar 670 671 | #vi 671 672 | et# 672 673 | ins 673 674 | os# 674 675 | nb 675 676 | dy 676 677 | #he 677 678 | ans 678 679 | #na 679 680 | #ne 680 681 | ear 681 682 | ws 682 683 | ont 683 684 | #sc 684 685 | nce 685 686 | gra 686 687 | cal 687 688 | bs 688 689 | anc 689 690 | ack 690 691 | ll# 691 692 | mon 692 693 | lli 693 694 | lla 694 695 | #q 695 696 | #sp 696 697 | #to 697 698 | tro 698 699 | ene 699 700 | ra# 700 701 | ama 701 702 | mat 702 703 | tar 703 704 | el# 704 705 | ome 705 706 | or# 706 707 | gn 707 708 | rf 708 709 | tan 709 710 | bb 710 711 | ain 711 712 | ly# 712 713 | ii 713 714 | ies 714 715 | eli 715 716 | #si 716 717 | tm 717 718 | shi 718 719 | dc 719 720 | sn 720 721 | oh 721 722 | ens 722 723 | #le 723 724 | ann 724 725 | #gr 725 726 | #bi 726 727 | ini 727 728 | #fa 728 729 | dl 729 730 | nta 730 731 | pre 731 732 | ju 732 733 | ter# 733 734 | ten 734 735 | ym 735 736 | ick 736 737 | ger 737 738 | ta# 738 739 | ona 739 740 | oy 740 741 | lb 741 742 | nge 742 743 | ove 743 744 | nti 744 745 | #lo 745 746 | tw 746 747 | ron 747 748 | ry# 748 749 | our 749 750 | las 750 751 | oni 751 752 | ble 752 753 | 001 753 754 | ck# 754 755 | tte 755 756 | 01# 756 757 | lp 757 758 | gg 758 759 | #ga 759 760 | lis 760 761 | den 761 762 | la# 762 763 | yt 763 764 | nd# 764 765 | sin 765 766 | des 766 767 | kin 767 768 | #do 768 769 | ath 769 770 | tre 770 771 | por 771 772 | ett 772 773 | lf 773 774 | edi 774 775 | hl 775 776 | tes 776 777 | rac 777 778 | ade 778 779 | nm 779 780 | gen 780 781 | ich 781 782 | 201 782 783 | de# 783 784 | sho 784 785 | ori 785 786 | san 786 787 | ita 787 788 | sd 788 789 | #ge 789 790 | #no 790 791 | hin 791 792 | uk 792 793 | kr 793 794 | pla 794 795 | ie# 795 796 | kl 796 797 | lit 797 798 | dg 798 799 | yp 799 800 | #wa 800 801 | ost 801 802 | aj 802 803 | wh 803 804 | ez 804 805 | zo 805 806 | #ja 806 807 | ane 807 808 | ky 808 809 | cy 809 810 | #bu 810 811 | cti 811 812 | ment 812 813 | ker 813 814 | ous 814 815 | am# 815 816 | nat 816 817 | igh 817 818 | #cr 818 819 | tur 819 820 | oun 820 821 | hea 821 822 | db 822 823 | len 823 824 | #fr 824 825 | sf 825 826 | ire 826 827 | ster 827 828 | ria 828 829 | nh 829 830 | #ac 830 831 | ox 831 832 | ax 832 833 | red 833 834 | und 834 835 | ty# 835 836 | wn 836 837 | hn 837 838 | att 838 839 | tb 839 840 | net 840 841 | j# 841 842 | #wi 842 843 | din 843 844 | ash 844 845 | rec 845 846 | nne 846 847 | yb 847 848 | b0 848 849 | #go 849 850 | ure 850 851 | ero 851 852 | eve 852 853 | cat 853 854 | can 854 855 | nda 855 856 | #fi 856 857 | ata 857 858 | ami 858 859 | oli 859 860 | ato 860 861 | rt# 861 862 | ice 862 863 | dm 863 864 | eat 864 865 | nic 865 866 | ine# 866 867 | #11 867 868 | ds# 868 869 | ros 869 870 | #fo 870 871 | iti 871 872 | ls# 872 873 | lat 873 874 | rie 874 875 | wor 875 876 | ech 876 877 | hs 877 878 | olo 878 879 | th# 879 880 | #00 880 881 | ish 881 882 | by 882 883 | ave 883 884 | cho 884 885 | dt 885 886 | ili 886 887 | lk 887 888 | 10# 888 889 | ge# 889 890 | #we 890 891 | log 891 892 | ji 892 893 | ema 893 894 | nin 894 895 | sr 895 896 | dh 896 897 | iu 897 898 | tha 898 899 | nal 899 900 | oll 900 901 | #un 901 902 | at# 902 903 | lar 903 904 | ile 904 905 | tat 905 906 | #12 906 907 | tn 907 908 | ust 908 909 | ram 909 910 | rit 910 911 | tv 911 912 | col 912 913 | yd 913 914 | ity 914 915 | lic 915 916 | nr 916 917 | ark 917 918 | rh 918 919 | nv 919 920 | eme 920 921 | eco 921 922 | kh 922 923 | nter 923 924 | ock 924 925 | rma 925 926 | ord 926 927 | np 927 928 | yr 928 929 | me# 929 930 | ssi 930 931 | ey# 931 932 | #am 932 933 | dis 933 934 | ay# 934 935 | q# 935 936 | tp 936 937 | rad 937 938 | ano 938 939 | #bl 939 940 | #mu 940 941 | ace 941 942 | erm 942 943 | tz 943 944 | sen 944 945 | cor 945 946 | #ki 946 947 | ier 947 948 | #ad 948 949 | ix 949 950 | ted 950 951 | #pi 951 952 | xi 952 953 | err 953 954 | ora 954 955 | dia 955 956 | #pro 956 957 | bar 957 958 | sco 958 959 | ern 959 960 | ven 960 961 | hm 961 962 | she 962 963 | eal 963 964 | #as 964 965 | #cl 965 966 | rd# 966 967 | oma 967 968 | tel 968 969 | out 969 970 | ght 970 971 | pri 971 972 | gt 972 973 | enc 973 974 | met 974 975 | sse 975 976 | e1 976 977 | arr 977 978 | ndi 978 979 | ada 979 980 | lv 980 981 | mes 981 982 | ni# 982 983 | it# 983 984 | #con 984 985 | to# 985 986 | bra 986 987 | #ve 987 988 | #ko 988 989 | #va 989 990 | ete 990 991 | ong 991 992 | mas 992 993 | ood 993 994 | use 994 995 | mr 995 996 | hel 996 997 | hen 997 998 | #ab 998 999 | ve# 999 1000 | ake 1000 1001 | 0000 1001 1002 | ond 1002 1003 | ela 1003 1004 | bc 1004 1005 | ail 1005 1006 | ial 1006 1007 | ntr 1007 1008 | ao 1008 1009 | act 1009 1010 | nst 1010 1011 | kt 1011 1012 | tch 1012 1013 | ting 1013 1014 | #en 1014 1015 | ee# 1015 1016 | ions 1016 1017 | thi 1017 1018 | nj 1018 1019 | tf 1019 1020 | unt 1020 1021 | ons# 1021 1022 | df 1022 1023 | ngs 1023 1024 | del 1024 1025 | #mar 1025 1026 | dw 1026 1027 | let 1027 1028 | inc 1028 1029 | ien 1029 1030 | 002 1030 1031 | #14 1031 1032 | s1 1032 1033 | ral 1033 1034 | nz 1034 1035 | 11# 1035 1036 | #ke 1036 1037 | arc 1037 1038 | zz 1038 1039 | ent# 1039 1040 | um# 1040 1041 | cd 1041 1042 | adi 1042 1043 | pc 1043 1044 | ima 1044 1045 | ret 1045 1046 | tec 1046 1047 | eni 1047 1048 | esi 1048 1049 | lea 1049 1050 | port 1050 1051 | ime 1051 1052 | ndo 1052 1053 | gy 1053 1054 | da# 1054 1055 | cke 1055 1056 | #jo 1056 1057 | abl 1057 1058 | #ri 1058 1059 | app 1059 1060 | eth 1060 1061 | #fl 1061 1062 | #ti 1062 1063 | ein 1063 1064 | ese 1064 1065 | ras 1065 1066 | ura 1066 1067 | war 1067 1068 | a1 1068 1069 | ban 1069 1070 | ena 1070 1071 | mt 1071 1072 | dan 1072 1073 | alt 1073 1074 | win 1074 1075 | erg 1075 1076 | xt 1076 1077 | cen 1077 1078 | ther 1078 1079 | een 1079 1080 | 500 1080 1081 | que 1081 1082 | air 1082 1083 | #ni 1083 1084 | ight 1084 1085 | ies# 1085 1086 | nit 1086 1087 | ise 1087 1088 | 20# 1088 1089 | 12# 1089 1090 | #el 1090 1091 | uf 1091 1092 | cm 1092 1093 | fre 1093 1094 | oto 1094 1095 | rai 1095 1096 | 010 1096 1097 | ih 1097 1098 | 101 1098 1099 | #dr 1099 1100 | cb 1100 1101 | med 1101 1102 | ino 1102 1103 | lor 1103 1104 | mor 1104 1105 | iss 1105 1106 | dn 1106 1107 | bal 1107 1108 | ase 1108 1109 | yi 1109 1110 | erc 1110 1111 | rw 1111 1112 | hal 1112 1113 | ess# 1113 1114 | man# 1114 1115 | nis 1115 1116 | eta 1116 1117 | 000# 1117 1118 | rom 1118 1119 | cre 1119 1120 | llo 1120 1121 | lg 1121 1122 | ro# 1122 1123 | ski 1123 1124 | spe 1124 1125 | mil 1125 1126 | ead 1126 1127 | #15 1127 1128 | new 1128 1129 | ces 1129 1130 | 800 1130 1131 | nan 1131 1132 | nder 1132 1133 | spo 1133 1134 | tho 1134 1135 | ri# 1135 1136 | sg 1136 1137 | cou 1137 1138 | pan 1138 1139 | roc 1139 1140 | gar 1140 1141 | orm 1141 1142 | gre 1142 1143 | ors 1143 1144 | ad# 1144 1145 | 2f 1145 1146 | ma# 1146 1147 | line 1147 1148 | ml 1148 1149 | ado 1149 1150 | ki# 1150 1151 | #13 1151 1152 | pen 1152 1153 | td 1153 1154 | tiv 1154 1155 | mal 1155 1156 | xp 1156 1157 | pin 1157 1158 | ej 1158 1159 | rti 1159 1160 | #cha 1160 1161 | 003 1161 1162 | 300 1162 1163 | #my 1163 1164 | #au 1164 1165 | ean 1165 1166 | lac 1166 1167 | ars 1167 1168 | sal 1168 1169 | ij 1169 1170 | ote 1170 1171 | hd 1171 1172 | rch 1172 1173 | 02# 1173 1174 | nts 1174 1175 | oz 1175 1176 | #16 1176 1177 | fs 1177 1178 | 50# 1178 1179 | kn 1179 1180 | mic 1180 1181 | lia 1181 1182 | c1 1182 1183 | ti# 1183 1184 | cia 1184 1185 | bri 1185 1186 | ks# 1186 1187 | no# 1187 1188 | ham 1188 1189 | nw 1189 1190 | ith 1190 1191 | fc 1191 1192 | ola 1192 1193 | rid 1193 1194 | itt 1194 1195 | dp 1195 1196 | als 1196 1197 | ici 1197 1198 | usi 1198 1199 | nto 1199 1200 | rap 1200 1201 | #hi 1201 1202 | bro 1202 1203 | ook 1203 1204 | bil 1204 1205 | #18 1205 1206 | lly 1206 1207 | emo 1207 1208 | rou 1208 1209 | ln 1209 1210 | wr 1210 1211 | ka# 1211 1212 | iy 1212 1213 | pho 1213 1214 | inte 1214 1215 | e2 1215 1216 | gan 1216 1217 | etr 1217 1218 | ode 1218 1219 | erv 1219 1220 | eng 1220 1221 | ate# 1221 1222 | r1 1222 1223 | aut 1223 1224 | 2b 1224 1225 | lr 1225 1226 | hol 1226 1227 | hor 1227 1228 | eti 1228 1229 | land 1229 1230 | py 1230 1231 | hil 1231 1232 | sio 1232 1233 | hc 1233 1234 | 15# 1234 1235 | lon 1235 1236 | fer 1236 1237 | cas 1237 1238 | ole 1238 1239 | om# 1239 1240 | eas 1240 1241 | rop 1241 1242 | van 1242 1243 | li# 1243 1244 | rth 1244 1245 | #ap 1245 1246 | gm 1246 1247 | cra 1247 1248 | io# 1248 1249 | ple 1249 1250 | val 1250 1251 | amp 1251 1252 | tim 1252 1253 | qui 1253 1254 | evi 1254 1255 | uz 1255 1256 | #ex 1256 1257 | ope 1257 1258 | #pl 1258 1259 | #wo 1259 1260 | mis 1260 1261 | arm 1261 1262 | #ph 1262 1263 | 13# 1263 1264 | bel 1264 1265 | mus 1265 1266 | ida 1266 1267 | ede 1267 1268 | sh# 1268 1269 | avi 1269 1270 | mn 1270 1271 | ula 1271 1272 | lie 1272 1273 | alo 1273 1274 | lec 1274 1275 | aro 1275 1276 | #ce 1276 1277 | #fe 1277 1278 | sv 1278 1279 | #car 1279 1280 | 0a 1280 1281 | sa# 1281 1282 | tions 1282 1283 | lh 1283 1284 | sit 1284 1285 | cro 1285 1286 | uni 1286 1287 | asi 1287 1288 | nga 1288 1289 | tis 1289 1290 | tom 1290 1291 | pm 1291 1292 | ms# 1292 1293 | ink 1293 1294 | mag 1294 1295 | ose 1295 1296 | b00 1296 1297 | fin 1297 1298 | les# 1298 1299 | nth 1299 1300 | isi 1300 1301 | bea 1301 1302 | ult 1302 1303 | bh 1303 1304 | 16# 1304 1305 | nch 1305 1306 | ank 1306 1307 | gam 1307 1308 | yu 1308 1309 | #b0 1309 1310 | vin 1310 1311 | ung 1311 1312 | omp 1312 1313 | bd 1313 1314 | gb 1314 1315 | cla 1315 1316 | ille 1316 1317 | sel 1317 1318 | xa 1318 1319 | gin 1319 1320 | tg 1320 1321 | pat 1321 1322 | hat 1322 1323 | vie 1323 1324 | md 1324 1325 | yw 1325 1326 | sic 1326 1327 | vil 1327 1328 | hb 1328 1329 | #gu 1329 1330 | dv 1330 1331 | dre 1331 1332 | sca 1332 1333 | ld# 1333 1334 | wer 1334 1335 | ool 1335 1336 | rol 1336 1337 | 03# 1337 1338 | esc 1338 1339 | hes 1339 1340 | ke# 1340 1341 | ott 1341 1342 | nar 1342 1343 | own 1343 1344 | 23# 1344 1345 | don 1345 1346 | mac 1346 1347 | #com 1347 1348 | #je 1348 1349 | emi 1349 1350 | spa 1350 1351 | #tra 1351 1352 | 005 1352 1353 | sma 1353 1354 | hou 1354 1355 | dar 1355 1356 | isc 1356 1357 | rts 1357 1358 | oth 1358 1359 | pol 1359 1360 | rel 1360 1361 | you 1361 1362 | il# 1362 1363 | #19 1363 1364 | rem 1364 1365 | ling 1365 1366 | vid 1366 1367 | able 1367 1368 | 400 1368 1369 | rre 1369 1370 | tle 1370 1371 | 110 1371 1372 | dra 1372 1373 | 05# 1373 1374 | hot 1374 1375 | elo 1375 1376 | hon 1376 1377 | sup 1377 1378 | uri 1378 1379 | s2 1379 1380 | ring 1380 1381 | ega 1381 1382 | iel 1382 1383 | yg 1383 1384 | tru 1384 1385 | kk 1385 1386 | zy 1386 1387 | cf 1387 1388 | ca# 1388 1389 | rte 1389 1390 | cp 1390 1391 | tem 1391 1392 | #pu 1392 1393 | n1 1393 1394 | nia 1394 1395 | tie 1395 1396 | 004 1396 1397 | ssa 1397 1398 | id# 1398 1399 | 120 1399 1400 | #sta 1400 1401 | 30# 1401 1402 | rz 1402 1403 | gc 1403 1404 | #ci 1404 1405 | d1 1405 1406 | #cu 1406 1407 | 14# 1407 1408 | dat 1408 1409 | xe 1409 1410 | hp 1410 1411 | stu 1411 1412 | one# 1412 1413 | bur 1413 1414 | ctio 1414 1415 | 006 1415 1416 | rod 1416 1417 | vel 1417 1418 | ings 1418 1419 | 0c 1419 1420 | c2 1420 1421 | do# 1421 1422 | 04# 1422 1423 | 21# 1423 1424 | kar 1424 1425 | a2 1425 1426 | ance 1426 1427 | aba 1427 1428 | ndr 1428 1429 | ono 1429 1430 | cle 1430 1431 | gs# 1431 1432 | ico 1432 1433 | cs# 1433 1434 | erb 1434 1435 | erl 1435 1436 | boo 1436 1437 | ol# 1437 1438 | 600 1438 1439 | rep 1439 1440 | old 1440 1441 | bla 1441 1442 | #b00 1442 1443 | uy 1443 1444 | kb 1444 1445 | ary 1445 1446 | ovi 1446 1447 | and# 1447 1448 | rst 1448 1449 | oj 1449 1450 | ork 1450 1451 | ses 1451 1452 | ction 1452 1453 | ions# 1453 1454 | mit 1454 1455 | ken 1455 1456 | acc 1456 1457 | 07# 1457 1458 | #du 1458 1459 | aga 1459 1460 | #or 1460 1461 | cer 1461 1462 | yf 1462 1463 | cn 1463 1464 | orn 1464 1465 | esh 1465 1466 | off 1466 1467 | nam 1467 1468 | ah# 1468 1469 | sou 1469 1470 | lam 1470 1471 | enn 1471 1472 | tes# 1472 1473 | hom 1473 1474 | ha# 1474 1475 | 25# 1475 1476 | ley 1476 1477 | sion 1477 1478 | 08# 1478 1479 | ics 1479 1480 | t1 1480 1481 | 011 1481 1482 | ow# 1482 1483 | #lu 1483 1484 | 40# 1484 1485 | #pre 1485 1486 | 123 1486 1487 | #17 1487 1488 | top 1488 1489 | der# 1489 1490 | 007 1490 1491 | 22# 1491 1492 | pd 1492 1493 | ira 1493 1494 | #sha 1494 1495 | mg 1495 1496 | bt 1496 1497 | rri 1497 1498 | ava 1498 1499 | rse 1499 1500 | vic 1500 1501 | bm 1501 1502 | ila 1502 1503 | owe 1503 1504 | lig 1504 1505 | tas 1505 1506 | km 1506 1507 | nel 1507 1508 | bre 1508 1509 | rot 1509 1510 | #qu 1510 1511 | b2 1511 1512 | hw 1512 1513 | nds 1513 1514 | 09# 1514 1515 | #ju 1515 1516 | loc 1516 1517 | 06# 1517 1518 | uo 1518 1519 | ps# 1519 1520 | nie 1520 1521 | over 1521 1522 | b1 1522 1523 | #fu 1523 1524 | ente 1524 1525 | sea 1525 1526 | #21 1526 1527 | nor 1527 1528 | vis 1528 1529 | gd 1529 1530 | low 1530 1531 | co# 1531 1532 | vs 1532 1533 | wl 1533 1534 | non 1534 1535 | sur 1535 1536 | lw 1536 1537 | arl 1537 1538 | ets 1538 1539 | 24# 1539 1540 | rus 1540 1541 | blo 1541 1542 | 012 1542 1543 | sis 1543 1544 | #gi 1544 1545 | amo 1545 1546 | cks 1546 1547 | 60# 1547 1548 | sar 1548 1549 | fil 1549 1550 | ben 1550 1551 | abi 1551 1552 | a0 1552 1553 | d2 1553 1554 | ign 1554 1555 | #at 1555 1556 | day 1556 1557 | rio 1557 1558 | lov 1558 1559 | els 1559 1560 | #30 1560 1561 | ery 1561 1562 | #25 1562 1563 | ity# 1563 1564 | uto 1564 1565 | sz 1565 1566 | sol 1566 1567 | nse 1567 1568 | son# 1568 1569 | gro 1569 1570 | rne 1570 1571 | ot# 1571 1572 | tai 1572 1573 | yan 1573 1574 | #is 1574 1575 | kan 1575 1576 | ler# 1576 1577 | cin 1577 1578 | mai 1578 1579 | pos 1579 1580 | kc 1580 1581 | 700 1581 1582 | roo 1582 1583 | hoo 1583 1584 | his 1584 1585 | eq 1585 1586 | lif 1586 1587 | onn 1587 1588 | asa 1588 1589 | ike 1589 1590 | ness 1590 1591 | #im 1591 1592 | eno 1592 1593 | #par 1593 1594 | #hu 1594 1595 | 150 1595 1596 | #tu 1596 1597 | hem 1597 1598 | 0b 1598 1599 | tti 1599 1600 | nas 1600 1601 | #es 1601 1602 | ters 1602 1603 | inf 1603 1604 | omm 1604 1605 | ur# 1605 1606 | isa 1606 1607 | rge 1607 1608 | gal 1608 1609 | nna 1609 1610 | hh 1610 1611 | x1 1611 1612 | chr 1612 1613 | #int 1613 1614 | nsi 1614 1615 | tea 1615 1616 | omi 1616 1617 | aci 1617 1618 | bas 1618 1619 | fd 1619 1620 | lem 1620 1621 | uti 1621 1622 | 160 1622 1623 | ian# 1623 1624 | amb 1624 1625 | sq 1625 1626 | ews 1626 1627 | fic 1627 1628 | cto 1628 1629 | stor 1629 1630 | #ru 1630 1631 | zu 1631 1632 | ton# 1632 1633 | spi 1633 1634 | lay 1634 1635 | dj 1635 1636 | #01 1636 1637 | ted# 1637 1638 | tab 1638 1639 | abe 1639 1640 | ull 1640 1641 | 900 1641 1642 | fan 1642 1643 | 0x 1643 1644 | qua 1644 1645 | star 1645 1646 | #sch 1646 1647 | rde 1647 1648 | nli 1648 1649 | mf 1649 1650 | erf 1650 1651 | gf 1651 1652 | 99# 1652 1653 | eed 1653 1654 | eet 1654 1655 | ya# 1655 1656 | inn 1656 1657 | ange 1657 1658 | ket 1658 1659 | eso 1659 1660 | nco 1660 1661 | cam 1661 1662 | set 1662 1663 | ito 1663 1664 | ning 1664 1665 | onl 1665 1666 | esa 1666 1667 | iq 1667 1668 | rag 1668 1669 | stra 1669 1670 | 015 1670 1671 | #22 1671 1672 | king 1672 1673 | cent 1673 1674 | mel 1674 1675 | aca 1675 1676 | dal 1676 1677 | tk 1677 1678 | erd 1678 1679 | #sk 1679 1680 | cel 1680 1681 | tter 1681 1682 | 80# 1682 1683 | uv 1683 1684 | los 1684 1685 | atu 1685 1686 | xc 1686 1687 | fb 1687 1688 | #mc 1688 1689 | reg 1689 1690 | he# 1690 1691 | chan 1691 1692 | ape 1692 1693 | 102 1693 1694 | aq 1694 1695 | cg 1695 1696 | 17# 1696 1697 | #gl 1697 1698 | wal 1698 1699 | eam 1699 1700 | lle# 1700 1701 | 008 1701 1702 | tta 1702 1703 | dri 1703 1704 | #sl 1704 1705 | ize 1705 1706 | ves 1706 1707 | dic 1707 1708 | uh 1708 1709 | lo# 1709 1710 | 020 1710 1711 | onc 1711 1712 | sig 1712 1713 | apa 1713 1714 | gp 1714 1715 | 18# 1715 1716 | phi 1716 1717 | rob 1717 1718 | uck 1718 1719 | comp 1719 1720 | eld 1720 1721 | bac 1721 1722 | eto 1722 1723 | ak# 1723 1724 | nni 1724 1725 | dio 1725 1726 | xx 1726 1727 | yh 1727 1728 | ner# 1728 1729 | get 1729 1730 | rev 1730 1731 | oss 1731 1732 | ard# 1732 1733 | rna 1733 1734 | eca 1734 1735 | eda 1735 1736 | web 1736 1737 | bf 1737 1738 | 013 1738 1739 | ble# 1739 1740 | #ku 1740 1741 | aka 1741 1742 | wit 1742 1743 | pac 1743 1744 | mme 1744 1745 | pb 1745 1746 | esp 1746 1747 | 00000 1747 1748 | arn 1748 1749 | 33# 1749 1750 | hop 1750 1751 | oro 1751 1752 | rav 1752 1753 | r2 1753 1754 | sam 1754 1755 | nce# 1755 1756 | led 1756 1757 | c0 1757 1758 | iona 1758 1759 | #kr 1759 1760 | ting# 1760 1761 | gle 1761 1762 | ue# 1762 1763 | #man 1763 1764 | dor 1764 1765 | 90# 1765 1766 | m1 1766 1767 | dro 1767 1768 | #wh 1768 1769 | aco 1769 1770 | ys# 1770 1771 | bor 1771 1772 | od# 1772 1773 | 009 1773 1774 | tit 1774 1775 | #24 1775 1776 | ife 1776 1777 | rta 1777 1778 | 1a 1778 1779 | inter 1779 1780 | ical 1780 1781 | fm 1781 1782 | lad 1782 1783 | not 1783 1784 | yst 1784 1785 | est# 1785 1786 | row 1786 1787 | #per 1787 1788 | are# 1788 1789 | rei 1789 1790 | 016 1790 1791 | 77# 1791 1792 | rvi 1792 1793 | mot 1793 1794 | elle 1794 1795 | wt 1795 1796 | em# 1796 1797 | t2 1797 1798 | gel 1798 1799 | ir# 1799 1800 | #che 1800 1801 | ger# 1801 1802 | 19# 1802 1803 | uth 1803 1804 | tech 1804 1805 | aste 1805 1806 | oca 1806 1807 | atr 1807 1808 | ebe 1808 1809 | wil 1809 1810 | cit 1810 1811 | ea# 1811 1812 | enti 1812 1813 | ivi 1813 1814 | oti 1814 1815 | raf 1815 1816 | rra 1816 1817 | rig 1817 1818 | 32# 1818 1819 | vers 1819 1820 | eck 1820 1821 | pet 1821 1822 | #97 1822 1823 | ding 1823 1824 | lab 1824 1825 | 55# 1825 1826 | cap 1826 1827 | f1 1827 1828 | ute 1828 1829 | 45# 1829 1830 | 70# 1830 1831 | scr 1831 1832 | ges 1832 1833 | rim 1833 1834 | n2 1834 1835 | e3 1835 1836 | dge 1836 1837 | its 1837 1838 | mbe 1838 1839 | fra 1839 1840 | mh 1840 1841 | pal 1841 1842 | oot 1842 1843 | aya 1843 1844 | #sm 1844 1845 | tac 1845 1846 | sso 1846 1847 | p1 1847 1848 | 88# 1848 1849 | ct# 1849 1850 | ress 1850 1851 | ek# 1851 1852 | dea 1852 1853 | iva 1853 1854 | 1c 1854 1855 | nes# 1855 1856 | #em 1856 1857 | #us 1857 1858 | c3 1858 1859 | epa 1859 1860 | awa 1860 1861 | ai# 1861 1862 | #dis 1862 1863 | dit 1863 1864 | #ag 1864 1865 | b3 1865 1866 | enta 1866 1867 | tam 1867 1868 | yk 1868 1869 | ats 1869 1870 | leg 1870 1871 | wat 1871 1872 | chu 1872 1873 | 0d 1873 1874 | pf 1874 1875 | 180 1875 1876 | cont 1876 1877 | kw 1877 1878 | cri 1878 1879 | #on 1879 1880 | ite# 1880 1881 | tt# 1881 1882 | cz 1882 1883 | vill 1883 1884 | ux 1884 1885 | 31# 1885 1886 | l1 1886 1887 | arb 1887 1888 | bg 1888 1889 | ft# 1889 1890 | #chi 1890 1891 | run 1891 1892 | wy 1892 1893 | 28# 1893 1894 | iw 1894 1895 | wc 1895 1896 | ount 1896 1897 | eer 1897 1898 | osi 1898 1899 | 111 1899 1900 | abo 1900 1901 | pha 1901 1902 | ture 1902 1903 | tos 1903 1904 | 44# 1904 1905 | pn 1905 1906 | a4 1906 1907 | 41# 1907 1908 | pres 1908 1909 | duc 1909 1910 | nec 1910 1911 | pic 1911 1912 | #for 1912 1913 | aus 1913 1914 | tto 1914 1915 | #yo 1915 1916 | bp 1916 1917 | 27# 1917 1918 | vr 1918 1919 | arg 1919 1920 | #ste 1920 1921 | tia 1921 1922 | iat 1922 1923 | wb 1923 1924 | uch 1924 1925 | ist# 1925 1926 | bin 1926 1927 | oin 1927 1928 | ism 1928 1929 | lee 1929 1930 | char 1930 1931 | ppe 1931 1932 | sof 1932 1933 | org 1933 1934 | 35# 1934 1935 | hi# 1935 1936 | ild 1936 1937 | die 1937 1938 | kat 1938 1939 | 26# 1939 1940 | 2a 1940 1941 | erp 1941 1942 | tive 1942 1943 | a3 1943 1944 | 014 1944 1945 | ww 1945 1946 | ator 1946 1947 | lls 1947 1948 | tor# 1948 1949 | och 1949 1950 | e4 1950 1951 | form 1951 1952 | #50 1952 1953 | alle 1953 1954 | hee 1954 1955 | ny# 1955 1956 | imi 1956 1957 | ory 1957 1958 | #er 1958 1959 | 978 1959 1960 | lde 1960 1961 | nen 1961 1962 | 34# 1962 1963 | omo 1963 1964 | serv 1964 1965 | va# 1965 1966 | 65# 1966 1967 | imp 1967 1968 | ifi 1968 1969 | #aa 1969 1970 | e0 1970 1971 | lus 1971 1972 | mad 1972 1973 | ues 1973 1974 | #40 1974 1975 | 103 1975 1976 | rce 1976 1977 | ngl 1977 1978 | woo 1978 1979 | #sw 1979 1980 | 51# 1980 1981 | orl 1981 1982 | ota 1982 1983 | hit 1983 1984 | hk 1984 1985 | ive# 1985 1986 | stat 1986 1987 | wd 1987 1988 | riv 1988 1989 | hic 1989 1990 | lj 1990 1991 | cli 1991 1992 | d0 1992 1993 | obi 1993 1994 | mpl 1994 1995 | oft 1995 1996 | #bar 1996 1997 | cur 1997 1998 | dz 1998 1999 | 0s 1999 2000 | 0m 2000 2001 | -------------------------------------------------------------------------------- /notebooks/img/duet-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bhaskar-mitra/NDRM/ff57cfe27290e65592449743b917b8d61f1e8116/notebooks/img/duet-architecture.png --------------------------------------------------------------------------------