├── README.md ├── datasets.py ├── evaluation.py ├── homogeneous_data.py ├── model.py ├── pre_transforms.py ├── query_dump.py ├── server.py ├── static ├── dataset │ └── arch │ │ └── annotations │ │ └── dump_data_pair.py └── web │ ├── bootstrap.min.css │ ├── index.css │ ├── index.js │ ├── jquery.min.js │ └── jumbotron-narrow.css ├── templates └── index.html ├── test.py ├── tools.py ├── train.py ├── utils.py └── vocab.py /README.md: -------------------------------------------------------------------------------- 1 | 2 | ### 如何运行代码？ 3 | 4 | 5 | ##### 1. 安装环境 6 | 7 | ```Python 8 | pip install http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl 9 | pip install torchvision 10 | pip install gensim 11 | pip install hyperboard 12 | ``` 13 | 14 | 15 | ##### 2. 运行项目代码 16 | 17 | 18 | ```Python 19 | python dump_data_pair.py 20 | python pre_transforms.py 21 | hyperboard-run --port 5020 22 | python test.py 23 | python query_dumpy.py 24 | python server.py 25 | 26 | ``` 27 | 28 | 29 | 30 | ### 对代码的详细解释 31 | 32 | `dump_data_pair.py` 从原始数据 jianzhu_tag.json 中抽取特征 title, detail, image, (url)，同时对数据做 shuffle. 33 | 34 | `pre_transforms.py` 将图片预处理成 2048 维的向量。文本使用 one-hot 编码。训练集得到图片向量 images_train 和图片描述 captions_train。验证集用于 Recall@K 的计算，需要去除重复的文本，并存储 caps_obj_id 和 imgs_obj_id 来判断图片和文本是否匹配。 caps_url, imgs_url, imgs_path 主要用于做补充信息的展示。测试集是训练集和验证集的合并，用于用户查询和寻找 good case。寻找 good case 时做模型效果验证（图片和文本是否匹配），所以也需要存储 caps_obj_id 和 imgs_obj_id。**预处理后需要对得到的文件做一些移动。**train 和 dev 数据全部放到 data 目录下相应位置，test 的 caps.txt 和 imgs.npy 放在 data 目录下相应位置，用来给 load_dataset 读取数据。 test 的 caps_url.json、imgs_url.json 和 imgs_path.json 放到 vse 目录下对应的 server 子目录中，用来做查询后的展示。 35 | 36 | `test.py` 训练好的模型和对应的超参数会被保存下来。这一步包含数据的读取和词典构造、处理不同长度的句子、计算 pairwise ranking loss、计算 Recall@K 等，这些会在后面的文档中详细进行说明。 37 | 38 | `query_dump.py` 读取当前最好的训练模型和对应的超参数，将数据集中对应的图片和文本转换成图片向量和文本向量，并保存在 vse 目录下对应的 server 子目录中，以备查询之用。这里重新保存了训练模型和超参数，表示图片向量和文本向量是使用这个模型和超参数得到的。 39 | 40 | `server.py` 搭建图文互搜网站，供用户输入建筑描述或建筑图片，返回相应的查询结果 41 | 42 | 43 | 这里我们重点关注 train.py，这份代码是图文互搜项目的核心代码。 44 | 45 | ##### 1. 处理不同长度的句子 46 | 47 | 读取训练集和验证集的数据，并利用两者的 caption 构造字典。build_dictionary 返回的是 worddict 和 wordcount。两者都按照word出现的次数做了排序。worddict 是单词以及它们的 id。wordcount是单词以及它们出现的次数。 48 | 49 | 50 | HomogeneousData 返回 Batch，**且每个 Batch 的文本长度都相同**。`prepare()` 统计每个长度下有多少句子以及每个句子的位置。`reset()` 对句子长度做乱序，对同一句子长度中的句子顺序做乱序。len_curr_counts 存储着每个长度下还有多少句子没有被使用。与之对应的是，len_indices_pos 存储着每个长度下访问到了哪个句子。 51 | 52 | `next()` 如果当前句子长度是否还有句子没有被访问，那么跳出 while 循环去访问该句子长度下还未被访问的句子，然后跳到下一个句子长度 (注意这里不是把某个长度下的所有句子都访问完后，再访问下一个句子长度的句子)。否则，查看下一个句子长度是否还有句子未被访问，然后继续上面的操作。如果所有句子长度下的所有句子都被访问了。那么一个 epoch 就结束了，调用 reset 重置相关变量。 53 | 54 | 在访问某句子长度下还未被访问的句子时，首先通过 len_curr_counts 确定该长度下还有多少句子未被访问。然后和 batch_size 取一个较小值，并命名为 curr_batch_size。然后通过 len_indices_pos 得到当前长度访问到了哪个句子，从该句子开始访问 curr_batch_size 个句子，并通过 len_indices 得到这些句子的位置。更新 len_indices_pos 和 len_curr_counts。最后返回对应位置的句子和图片。 55 | 56 | `prepare_data()` 输入的是 batch，里面包含文本 caps 和图片特征 features。对 caps 做分词并通过worddict 转换为单词 id。抛弃到长度大于 maxlen 的句子。然后将文本向量和图片特征向量从 list 转成 numpy。 57 | 58 | `encode_sentences()` 对验证集中的句子进行向量编码。ds 是一个可以按照长度访问句子的字典。[minibatch::numbatches] 的意思是从 minibatch 开始，每 numbatches 个取一个。然后将单词转换为单词 id，最后将沿着 f_senc ==> build_sentence_encoder 得到文本向量。在构造batch的时候句子的序号是按照相同长度被打乱的，但是到 features 的时候又根据句子的 id 进行重新排位，这时图片和文本又能够对应上了。 59 | 60 | 61 | ##### 2. PairwiseRankingLoss 的计算 62 | 63 | PairWiseRanking 的输入是图片 (batch_size, dim) 和文本 (batch_size, dim)。将图片矩乘以文本矩阵得到相似度矩阵。相似度矩阵的对角线是图文对的相似度。每一行是图片和其它文本 (包括匹配文本)的相似度，每一列是文本和其它图片 (包括匹配图片)的相似度。有了这些之后，我们就可以计算 pairwise ranking loss 了!!!! 64 | 65 | #### 3. Recall@K 的计算 66 | 67 | arch 数据集没有公开数据集那么规范，每张图片都有5个对应描述，然后使用天然的 index 得到图片和文本是否匹配。Recall@K 只用到验证集的数据，对训练集没有影响。在作验证集数据处理的时候，读入的每个样例包含图文对以及它们所属的 obj_id。然后将所有文本做去重处理，得到 captions_dev。而与之对应的数组 caps_obj_id ，标识了每个 caption 对应的 obj_id。类似的，images_dev 也有 imgs_obj_id 来标识每个 image 对应的 obj_id。因此，在计算 Recall@K 时，我们通过图片和文本的 obj_id 来判断两者是否匹配。 68 | 69 | i2t_arch() 计算以图搜文的 Recall@K。输入是全部的图片和全部文本向量。之前我们已经提到，在文本到文本向量的转换中，其顺序并没有改变。图片到图片向量的过程也是。遍历所有图片，依次和所有文本计算相似度。inds 按相似度排序的文本的序号。因为之前的处理中文本顺序并没有改变，所以我们可以通过序号直接找到其对应的 obj_id。caps_obj_id[inds] 得到按相似度排序的文本的 obj_id，numpy.where 得到与图片对应的 obj_id 在 caps_obj_id[inds] 中出现的位置。t2i_arch() 的过程也是类似的，这里不再赘述。 70 | 71 | -------------------------------------------------------------------------------- /datasets.py: -------------------------------------------------------------------------------- 1 | """ 2 | Dataset loading 3 | """ 4 | import numpy 5 | 6 | path_to_data = 'data/' 7 | 8 | def load_dataset(name='f8k', load_test=False): 9 | """ 10 | Load captions and image features 11 | """ 12 | loc = path_to_data + name + '/' 13 | 14 | if load_test: 15 | # Captions 16 | test_caps = [] 17 | with open(loc+name+'_test_caps.txt', 'rb') as f: 18 | for line in f: 19 | test_caps.append(line.strip()) 20 | # Image features 21 | test_ims = numpy.load(loc+name+'_test_ims.npy') 22 | return (test_caps, test_ims) 23 | else: 24 | # Captions 25 | train_caps, dev_caps = [], [] 26 | with open(loc+name+'_train_caps.txt', 'rb') as f: 27 | for line in f: 28 | train_caps.append(line.strip()) 29 | 30 | with open(loc+name+'_dev_caps.txt', 'rb') as f: 31 | for line in f: 32 | dev_caps.append(line.strip()) 33 | 34 | # Image features 35 | train_ims = numpy.load(loc+name+'_train_ims.npy') 36 | dev_ims = numpy.load(loc+name+'_dev_ims.npy') 37 | 38 | return (train_caps, train_ims), (dev_caps, dev_ims) -------------------------------------------------------------------------------- /evaluation.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | import torch 3 | from datasets import load_dataset 4 | from tools import encode_sentences, encode_images 5 | import json 6 | 7 | def evalrank(model, data, split='dev'): 8 | """ 9 | Evaluate a trained model on either dev ortest 10 | """ 11 | 12 | print 'Loading dataset' 13 | if split == 'dev': 14 | X = load_dataset(data)[1] 15 | else: 16 | X = load_dataset(data, load_test=True) 17 | 18 | 19 | print 'Computing results...' 20 | ls = encode_sentences(model, X[0]) 21 | lim = encode_images(model, X[1]) 22 | 23 | if data == 'arch': 24 | # Find the good case in test dataset 25 | (r1, r5, r10, medr) = i2t_arch_case(lim, ls, X[0]) 26 | print "Image to text: %.1f, %.1f, %.1f, %.1f" % (r1, r5, r10, medr) 27 | (r1i, r5i, r10i, medri) = t2i_arch_case(lim, ls, X[0]) 28 | print "Text to image: %.1f, %.1f, %.1f, %.1f" % (r1i, r5i, r10i, medri) 29 | else: 30 | (r1, r5, r10, medr) = i2t(lim, ls) 31 | print "Image to text: %.1f, %.1f, %.1f, %.1f" % (r1, r5, r10, medr) 32 | (r1i, r5i, r10i, medri) = t2i(lim, ls) 33 | print "Text to image: %.1f, %.1f, %.1f, %.1f" % (r1i, r5i, r10i, medri) 34 | 35 | 36 | def i2t(images, captions, npts=None): 37 | """ 38 | Images->Text (Image Annotation) 39 | Images: (5N, K) matrix of images 40 | Captions: (5N, K) matrix of captions 41 | """ 42 | if npts == None: 43 | npts = images.size()[0] / 5 44 | 45 | ranks = numpy.zeros(npts) 46 | for index in range(npts): 47 | 48 | # Get query image 49 | im = images[5 * index].unsqueeze(0) 50 | 51 | # Compute scores 52 | d = torch.mm(im, captions.t()) 53 | d_sorted, inds = torch.sort(d, descending=True) 54 | inds = inds.data.squeeze(0).cpu().numpy() 55 | 56 | # Score 57 | rank = 1e20 58 | # find the highest ranking 59 | for i in range(5*index, 5*index + 5, 1): 60 | tmp = numpy.where(inds == i)[0][0] 61 | if tmp < rank: 62 | rank = tmp 63 | ranks[index] = rank 64 | 65 | # Compute metrics 66 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 67 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 68 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 69 | medr = numpy.floor(numpy.median(ranks)) + 1 70 | return (r1, r5, r10, medr) 71 | 72 | 73 | def t2i(images, captions, npts=None, data='f8k'): 74 | """ 75 | Text->Images (Image Search) 76 | Images: (5N, K) matrix of images 77 | Captions: (5N, K) matrix of captions 78 | """ 79 | if npts == None: 80 | npts = images.size()[0] / 5 81 | 82 | ims = torch.cat([images[i].unsqueeze(0) for i in range(0, len(images), 5)]) 83 | 84 | ranks = numpy.zeros(5 * npts) 85 | for index in range(npts): 86 | 87 | # Get query captions 88 | queries = captions[5*index : 5*index + 5] 89 | 90 | # Compute scores 91 | d = torch.mm(queries, ims.t()) 92 | for i in range(d.size()[0]): 93 | d_sorted, inds = torch.sort(d[i], descending=True) 94 | inds = inds.data.squeeze(0).cpu().numpy() 95 | ranks[5 * index + i] = numpy.where(inds == index)[0][0] 96 | 97 | # Compute metrics 98 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 99 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 100 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 101 | medr = numpy.floor(numpy.median(ranks)) + 1 102 | return (r1, r5, r10, medr) 103 | 104 | 105 | def i2t_arch(images, captions): 106 | npts = images.size()[0] 107 | ranks = numpy.zeros(npts) 108 | caps_obj_id = numpy.load(open('data/arch/arch_dev_caps_id.npy')) 109 | imgs_obj_id = numpy.load(open('data/arch/arch_dev_imgs_id.npy')) 110 | for index in range(npts): 111 | # Get query image 112 | im = images[index:index+1] 113 | # Compute scores 114 | d = torch.mm(im, captions.t()) 115 | d_sorted, inds = torch.sort(d, descending=True) 116 | inds = inds.data.squeeze(0).cpu().numpy() 117 | ranks[index] = numpy.where(caps_obj_id[inds] == imgs_obj_id[index])[0][0] 118 | 119 | # Compute metrics 120 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 121 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 122 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 123 | medr = numpy.floor(numpy.median(ranks)) + 1 124 | return (r1, r5, r10, medr) 125 | 126 | 127 | def t2i_arch(images, captions): 128 | npts = captions.size()[0] 129 | ranks = numpy.zeros(npts) 130 | caps_obj_id = numpy.load(open('data/arch/arch_dev_caps_id.npy')) 131 | imgs_obj_id = numpy.load(open('data/arch/arch_dev_imgs_id.npy')) 132 | for index in range(npts): 133 | # Get query caption 134 | cap = captions[index:index+1] 135 | # Compute scores 136 | d = torch.mm(cap, images.t()) 137 | d_sorted, inds = torch.sort(d, descending=True) 138 | inds = inds.data.squeeze(0).cpu().numpy() 139 | ranks[index] = numpy.where(imgs_obj_id[inds] == caps_obj_id[index])[0][0] 140 | 141 | # Compute metrics 142 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 143 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 144 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 145 | medr = numpy.floor(numpy.median(ranks)) + 1 146 | return (r1, r5, r10, medr) 147 | 148 | def i2t_arch_case(images, captions, caps_orig): 149 | npts = images.size()[0] 150 | ranks = numpy.zeros(npts) 151 | caps_obj_id = numpy.load(open('data/arch/arch_test_caps_id.npy')) 152 | imgs_obj_id = numpy.load(open('data/arch/arch_test_imgs_id.npy')) 153 | imgs_url = json.load(open('data/arch/arch_test_imgs_url.json')) 154 | 155 | print_num = 10 156 | for index in range(npts): 157 | # Get query image 158 | im = images[index:index+1] 159 | # Compute scores 160 | d = torch.mm(im, captions.t()) 161 | d_sorted, inds = torch.sort(d, descending=True) 162 | inds = inds.data.squeeze(0).cpu().numpy() 163 | ranks[index] = numpy.where(caps_obj_id[inds] == imgs_obj_id[index])[0][0] 164 | temp_rank = int(ranks[index]) 165 | if temp_rank == 0 and print_num > 0: 166 | print 'i2t: %d' %(10-print_num) 167 | print 'image_url: ', imgs_url[index] 168 | print 'captions ', caps_orig[inds[0]] 169 | print '\n\n' 170 | print_num -= 1 171 | 172 | # Compute metrics 173 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 174 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 175 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 176 | medr = numpy.floor(numpy.median(ranks)) + 1 177 | return (r1, r5, r10, medr) 178 | 179 | 180 | def t2i_arch_case(images, captions, caps_orig): 181 | npts = captions.size()[0] 182 | ranks = numpy.zeros(npts) 183 | caps_obj_id = numpy.load(open('data/arch/arch_test_caps_id.npy')) 184 | imgs_obj_id = numpy.load(open('data/arch/arch_test_imgs_id.npy')) 185 | imgs_url = json.load(open('data/arch/arch_test_imgs_url.json')) 186 | print_num = 10 187 | for index in range(npts): 188 | # Get query caption 189 | cap = captions[index:index+1] 190 | # Compute scores 191 | d = torch.mm(cap, images.t()) 192 | d_sorted, inds = torch.sort(d, descending=True) 193 | inds = inds.data.squeeze(0).cpu().numpy() 194 | ranks[index] = numpy.where(imgs_obj_id[inds] == caps_obj_id[index])[0][0] 195 | temp_rank = int(ranks[index]) 196 | if temp_rank == 0 and print_num > 0: 197 | print 't2i: %d' %(10-print_num) 198 | print 'caption: ', caps_orig[index] 199 | print 'img_url: ', imgs_url[inds[0]] 200 | print '\n\n' 201 | print_num -= 1 202 | 203 | # Compute metrics 204 | r1 = 100.0 * len(numpy.where(ranks < 1)[0]) / len(ranks) 205 | r5 = 100.0 * len(numpy.where(ranks < 5)[0]) / len(ranks) 206 | r10 = 100.0 * len(numpy.where(ranks < 10)[0]) / len(ranks) 207 | medr = numpy.floor(numpy.median(ranks)) + 1 208 | return (r1, r5, r10, medr) 209 | -------------------------------------------------------------------------------- /homogeneous_data.py: -------------------------------------------------------------------------------- 1 | import numpy 2 | import copy 3 | import sys 4 | 5 | 6 | class HomogeneousData(): 7 | 8 | def __init__(self, data, batch_size=128, maxlen=None): 9 | self.data = data 10 | self.batch_size = batch_size 11 | self.maxlen = maxlen 12 | 13 | self.prepare() 14 | self.reset() 15 | 16 | def prepare(self): 17 | # self.caps = [cap[:self.maxlen] for cap in self.data[0]] 18 | self.caps = self.data[0] 19 | self.feats = self.data[1] 20 | 21 | # find the unique lengths 22 | self.lengths = [len(cc.split()) for cc in self.caps] 23 | self.len_unique = numpy.unique(self.lengths) 24 | # remove any overly long sentences 25 | if self.maxlen: 26 | self.len_unique = [ll for ll in self.len_unique if ll <= self.maxlen] 27 | 28 | # indices of unique lengths 29 | self.len_indices = dict() 30 | self.len_counts = dict() 31 | for ll in self.len_unique: 32 | self.len_indices[ll] = numpy.where(self.lengths == ll)[0] 33 | self.len_counts[ll] = len(self.len_indices[ll]) 34 | 35 | # current counter 36 | self.len_curr_counts = copy.copy(self.len_counts) 37 | 38 | def reset(self): 39 | self.len_curr_counts = copy.copy(self.len_counts) 40 | self.len_unique = numpy.random.permutation(self.len_unique) 41 | self.len_indices_pos = dict() 42 | for ll in self.len_unique: 43 | self.len_indices_pos[ll] = 0 44 | self.len_indices[ll] = numpy.random.permutation(self.len_indices[ll]) 45 | self.len_idx = -1 46 | 47 | def next(self): 48 | count = 0 49 | while True: 50 | self.len_idx = numpy.mod(self.len_idx+1, len(self.len_unique)) 51 | if self.len_curr_counts[self.len_unique[self.len_idx]] > 0: 52 | break 53 | count += 1 54 | if count >= len(self.len_unique): 55 | break 56 | if count >= len(self.len_unique): 57 | self.reset() 58 | raise StopIteration() 59 | 60 | # get the batch size 61 | curr_batch_size = numpy.minimum(self.batch_size, self.len_curr_counts[self.len_unique[self.len_idx]]) 62 | curr_pos = self.len_indices_pos[self.len_unique[self.len_idx]] 63 | # get the indices for the current batch 64 | curr_indices = self.len_indices[self.len_unique[self.len_idx]][curr_pos:curr_pos+curr_batch_size] 65 | self.len_indices_pos[self.len_unique[self.len_idx]] += curr_batch_size 66 | self.len_curr_counts[self.len_unique[self.len_idx]] -= curr_batch_size 67 | 68 | caps = [self.caps[ii] for ii in curr_indices] 69 | feats = [self.feats[ii] for ii in curr_indices] 70 | 71 | return caps, feats 72 | 73 | def __iter__(self): 74 | return self 75 | 76 | 77 | def prepare_data(caps, features, worddict, maxlen=None, n_words=10000): 78 | """ 79 | Put data into format useable by the model 80 | """ 81 | seqs = [] 82 | feat_list = [] 83 | for i, cc in enumerate(caps): 84 | seqs.append([worddict[w] if worddict[w] < n_words else 1 for w in cc.split()]) 85 | feat_list.append(features[i]) 86 | 87 | lengths = [len(s) for s in seqs] 88 | 89 | if maxlen != None and numpy.max(lengths) >= maxlen: 90 | new_seqs = [] 91 | new_feat_list = [] 92 | new_lengths = [] 93 | for l, s, y in zip(lengths, seqs, feat_list): 94 | if l < maxlen: 95 | new_seqs.append(s) 96 | new_feat_list.append(y) 97 | new_lengths.append(l) 98 | lengths = new_lengths 99 | feat_list = new_feat_list 100 | seqs = new_seqs 101 | 102 | if len(lengths) < 1: 103 | return None, None 104 | 105 | # Why not use the following code? 106 | # y_np = numpy.asarray(feat_list, dtype=numpy.float32) 107 | y = numpy.zeros((len(feat_list), len(feat_list[0]))).astype('float32') 108 | for idx, ff in enumerate(feat_list): 109 | y[idx,:] = ff 110 | 111 | n_samples = len(seqs) 112 | maxlen = numpy.max(lengths)+1 113 | 114 | x = numpy.zeros((maxlen, n_samples)).astype('int64') 115 | for idx, s in enumerate(seqs): 116 | x[:lengths[idx],idx] = s 117 | 118 | return x, y 119 | -------------------------------------------------------------------------------- /model.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | import torch 3 | from utils import l2norm, xavier_weight 4 | from torch.autograd import Variable 5 | import torch.nn.init as init 6 | from gensim.models.word2vec import Word2Vec 7 | import numpy 8 | 9 | wvModel = Word2Vec.load('static/word2vec-chi/word2vec_news.model') 10 | 11 | class ImgSenRanking(torch.nn.Module): 12 | def __init__(self, model_options): 13 | super(ImgSenRanking, self).__init__() 14 | self.linear = torch.nn.Linear(model_options['dim_image'], model_options['dim']) 15 | self.lstm = torch.nn.LSTM(model_options['dim_word'], model_options['dim'], 1) 16 | self.embedding = torch.nn.Embedding(model_options['n_words'], model_options['dim_word']) 17 | self.model_options = model_options 18 | self.init_weights() 19 | 20 | def init_weights(self): 21 | xavier_weight(self.linear.weight) 22 | # init.xavier_normal(self.linear.weight) 23 | self.linear.bias.data.fill_(0) 24 | 25 | def forward(self, x_id, im, x): 26 | x_id_emb = self.embedding(x_id) 27 | im = self.linear(im) 28 | 29 | x_w2v = torch.zeros(*x_id_emb.size()) 30 | x_cat = None 31 | if self.model_options['concat']: 32 | for i, text in enumerate(x): 33 | for j, word in enumerate(text.split()): 34 | try: 35 | x_w2v[j, i] = torch.from_numpy(wvModel[word.decode('utf8')]) 36 | except KeyError: 37 | pass 38 | x_w2v = Variable(x_w2v.cuda()) 39 | x_cat = torch.cat([x_id_emb, x_w2v]) 40 | else: 41 | x_cat = x_id_emb 42 | 43 | 44 | if self.model_options['encoder'] == 'bow': 45 | x_cat = x_cat.sum(0).squeeze(0) 46 | else: 47 | _, (x_cat, _) = self.lstm(x_cat) 48 | x_cat = x_cat.squeeze(0) 49 | 50 | return l2norm(x_cat), l2norm(im) 51 | 52 | def forward_sens(self, x_id, x): 53 | x_id_emb = self.embedding(x_id) 54 | 55 | x_w2v = torch.zeros(*x_id_emb.size()) 56 | x_cat = None 57 | if self.model_options['concat']: 58 | for i, text in enumerate(x): 59 | for j, word in enumerate(text): 60 | try: 61 | x_w2v[j, i] = torch.from_numpy(wvModel[word.decode('utf8')]) 62 | except KeyError: 63 | pass 64 | 65 | x_w2v = Variable(x_w2v.cuda()) 66 | x_cat = torch.cat([x_id_emb, x_w2v]) 67 | else: 68 | x_cat = x_id_emb 69 | 70 | if self.model_options['encoder'] == 'bow': 71 | x_cat = x_cat.sum(0).squeeze(0) 72 | else: 73 | _, (x_cat, _) = self.lstm(x_cat) 74 | x_cat = x_cat.squeeze(0) 75 | return l2norm(x_cat) 76 | 77 | def forward_imgs(self, im): 78 | im = self.linear(im) 79 | return l2norm(im) 80 | 81 | class PairwiseRankingLoss(torch.nn.Module): 82 | 83 | def __init__(self, margin=1.0): 84 | super(PairwiseRankingLoss, self).__init__() 85 | self.margin = margin 86 | 87 | def forward(self, im, s): 88 | margin = self.margin 89 | # compute image-sentence score matrix 90 | scores = torch.mm(im, s.transpose(1, 0)) 91 | diagonal = scores.diag() 92 | 93 | # compare every diagonal score to scores in its column (i.e, all contrastive images for each sentence) 94 | cost_s = torch.max(Variable(torch.zeros(scores.size()[0], scores.size()[1]).cuda()), (margin-diagonal).expand_as(scores)+scores) 95 | # compare every diagonal score to scores in its row (i.e, all contrastive sentences for each image) 96 | cost_im = torch.max(Variable(torch.zeros(scores.size()[0], scores.size()[1]).cuda()), (margin-diagonal).expand_as(scores).transpose(1, 0)+scores) 97 | 98 | for i in xrange(scores.size()[0]): 99 | cost_s[i, i] = 0 100 | cost_im[i, i] = 0 101 | 102 | return cost_s.sum() + cost_im.sum() 103 | -------------------------------------------------------------------------------- /pre_transforms.py: -------------------------------------------------------------------------------- 1 | 2 | # coding: utf-8 3 | 4 | # In[1]: 5 | 6 | import jieba.analyse 7 | jieba.analyse.set_stop_words('static/dataset/stopwords.txt') 8 | 9 | from torchvision import transforms 10 | import json, os 11 | import numpy as np 12 | import torch 13 | from PIL import Image, ImageFile 14 | import torchvision.models as models 15 | from torch.autograd import Variable 16 | 17 | dataset_dir = 'static/dataset/arch' 18 | max_num = 1000 19 | 20 | # In[2]: 21 | 22 | image_transform = transforms.Compose([ 23 | transforms.Scale([224, 224]), 24 | transforms.ToTensor(), 25 | transforms.Normalize(mean = [ 0.485, 0.456, 0.406 ], 26 | std = [ 0.229, 0.224, 0.225 ]), 27 | ]) 28 | 29 | resnet = models.resnet152(pretrained=True) 30 | resnet.fc = torch.nn.Dropout(p=0) 31 | resnet = resnet.eval() 32 | resnet = resnet.cuda() 33 | 34 | 35 | def normalize(v): 36 | norm=np.linalg.norm(v) 37 | if norm==0: 38 | return v 39 | return v/norm 40 | 41 | 42 | def pre_transforms(): 43 | 44 | ImageFile.LOAD_TRUNCATED_IMAGES = True 45 | 46 | print 'Pre-transforming train ...' 47 | 48 | json_file = 'annotations/data_pair_train.json' 49 | train_data_pair = json.load(open(os.path.join(dataset_dir, json_file), 'r')) 50 | 51 | captions_train, images_train = [], [] 52 | for k, item in enumerate(train_data_pair): 53 | if k % 100 == 0: 54 | print 'Processing %d/%d' %(k, len(train_data_pair)) 55 | caption = ' '.join(jieba.analyse.extract_tags(item['caption'], topK=20, withWeight=False, allowPOS=())) 56 | if len(caption) == 0: 57 | continue 58 | 59 | captions_train.append(caption.encode('utf8')+'\n') 60 | 61 | img_vec = image_transform(Image.open(item['img_path']).convert('RGB')).unsqueeze(0) 62 | img_vec = resnet(Variable(img_vec.cuda())).data.squeeze(0).cpu().numpy() 63 | images_train.append(img_vec) 64 | 65 | if k > max_num: 66 | break 67 | 68 | with open('arch_train_caps.txt', 'w') as f_write: 69 | f_write.writelines(captions_train) 70 | 71 | images_train = np.asarray(images_train, dtype=np.float32) 72 | images_train = normalize(images_train) 73 | np.save('arch_train_ims.npy', images_train) 74 | 75 | print 'Pre-transforming train Done' 76 | 77 | print 'Pre-transforming dev ...' 78 | 79 | json_file = 'annotations/data_pair_val.json' 80 | dev_data_pair = json.load(open(os.path.join(dataset_dir, json_file), 'r')) 81 | 82 | captions_dev, images_dev = [], [] 83 | caps_obj_id, imgs_obj_id = [], [] 84 | caps_url, imgs_url, imgs_path = [], [], [] 85 | for k, item in enumerate(dev_data_pair): 86 | if k % 100 == 0: 87 | print 'Processing %d/%d' %(k, len(dev_data_pair)) 88 | if item['obj_id'] not in caps_obj_id: 89 | caption = ' '.join(jieba.analyse.extract_tags(item['caption'], topK=20, withWeight=False, allowPOS=())) 90 | if len(caption) == 0: 91 | continue 92 | 93 | captions_dev.append(caption.encode('utf8')+'\n') 94 | caps_obj_id.append(item['obj_id']) 95 | caps_url.append(item['url']) 96 | 97 | img_vec = image_transform(Image.open(item['img_path']).convert('RGB')).unsqueeze(0) 98 | img_vec = resnet(Variable(img_vec.cuda())).data.squeeze(0).cpu().numpy() 99 | images_dev.append(img_vec) 100 | imgs_obj_id.append(item['obj_id']) 101 | imgs_url.append(item['url']) 102 | imgs_path.append(item['img_path']) 103 | 104 | if k > max_num: 105 | break 106 | 107 | with open('arch_dev_caps.txt', 'w') as f_write: 108 | f_write.writelines(captions_dev) 109 | 110 | json.dump(caps_url, open('arch_dev_caps_url.json', 'w')) 111 | json.dump(imgs_url, open('arch_dev_imgs_url.json', 'w')) 112 | json.dump(imgs_path, open('arch_dev_imgs_path.json', 'w')) 113 | 114 | images_dev = np.asarray(images_dev, dtype=np.float32) 115 | images_dev = normalize(images_dev) 116 | np.save('arch_dev_ims.npy', images_dev) 117 | 118 | caps_obj_id = np.asarray(caps_obj_id, dtype=np.float32) 119 | imgs_obj_id = np.asarray(imgs_obj_id, dtype=np.float32) 120 | np.save('arch_dev_caps_id.npy', caps_obj_id) 121 | np.save('arch_dev_imgs_id.npy', imgs_obj_id) 122 | 123 | print 'Pre-transforming dev Done' 124 | 125 | print 'Pre-transforming test ...' 126 | 127 | test_data_pair = train_data_pair + dev_data_pair 128 | 129 | captions_test, images_test = [], [] 130 | caps_obj_id_test, imgs_obj_id_test = [], [] 131 | caps_url_test, imgs_url_test, imgs_path_test = [], [], [] 132 | for k, item in enumerate(test_data_pair): 133 | if k % 100 == 0: 134 | print 'Processing %d/%d' %(k, len(test_data_pair)) 135 | if item['obj_id'] not in caps_obj_id_test: 136 | caption = ' '.join(jieba.analyse.extract_tags(item['caption'], topK=20, withWeight=False, allowPOS=())) 137 | if len(caption) == 0: 138 | continue 139 | 140 | captions_test.append(caption.encode('utf8')+'\n') 141 | caps_obj_id_test.append(item['obj_id']) 142 | caps_url_test.append(item['url']) 143 | 144 | img_vec = image_transform(Image.open(item['img_path']).convert('RGB')).unsqueeze(0) 145 | img_vec = resnet(Variable(img_vec.cuda())).data.squeeze(0).cpu().numpy() 146 | images_test.append(img_vec) 147 | imgs_obj_id_test.append(item['obj_id']) 148 | imgs_url_test.append(item['url']) 149 | imgs_path_test.append(item['img_path']) 150 | 151 | if k > max_num: 152 | break 153 | 154 | with open('arch_test_caps.txt', 'w') as f_write: 155 | f_write.writelines(captions_test) 156 | 157 | json.dump(caps_url_test, open('arch_test_caps_url.json', 'w')) 158 | json.dump(imgs_url_test, open('arch_test_imgs_url.json', 'w')) 159 | json.dump(imgs_path_test, open('arch_test_imgs_path.json', 'w')) 160 | 161 | images_test = np.asarray(images_test, dtype=np.float32) 162 | images_test = normalize(images_test) 163 | np.save('arch_test_ims.npy', images_test) 164 | 165 | caps_obj_id_test = np.asarray(caps_obj_id_test, dtype=np.float32) 166 | imgs_obj_id_test = np.asarray(imgs_obj_id_test, dtype=np.float32) 167 | np.save('arch_test_caps_id.npy', caps_obj_id_test) 168 | np.save('arch_test_imgs_id.npy', imgs_obj_id_test) 169 | 170 | print 'Pre-transforming test Done' 171 | 172 | if __name__ == "__main__": 173 | pre_transforms() 174 | -------------------------------------------------------------------------------- /query_dump.py: -------------------------------------------------------------------------------- 1 | from model import ImgSenRanking 2 | import cPickle as pkl 3 | import torch 4 | from datasets import load_dataset 5 | import numpy as np 6 | from tools import encode_sentences, encode_images 7 | import json 8 | 9 | data = 'arch' 10 | loadfrom = 'vse/' + data 11 | saveto = 'vse/%s_server/%s' %(data, data) 12 | hyper_params = '%s_params.pkl' % loadfrom 13 | model_params = '%s_model.pkl' % loadfrom 14 | 15 | print 'Building model ... ', 16 | model_options = pkl.load(open(hyper_params, 'r')) 17 | model = ImgSenRanking(model_options).cuda() 18 | model.load_state_dict(torch.load(model_params)) 19 | print 'Done' 20 | 21 | test = load_dataset(data, load_test=True) 22 | 23 | print 'Dumping data ... ' 24 | 25 | curr_model = {} 26 | curr_model['options'] = model_options 27 | curr_model['worddict'] = model_options['worddict'] 28 | curr_model['word_idict'] = model_options['word_idict'] 29 | curr_model['img_sen_model'] = model 30 | 31 | ls, lim = encode_sentences(curr_model, test[0]), encode_images(curr_model, test[1]) 32 | 33 | # save the using params and model when dumping data 34 | torch.save(ls, '%s_ls.pkl'%saveto) 35 | torch.save(lim, '%s_lim.pkl'%saveto) 36 | pkl.dump(model_options, open('%s_params_dump.pkl'%saveto, 'wb')) 37 | torch.save(model.state_dict(), '%s_model_dump.pkl'%saveto) 38 | json.dump(test[0], open('%s_caps.json'%saveto, 'w')) 39 | 40 | print 'ls: ', ls.data.size() 41 | print 'lim: ', lim.data.size() 42 | 43 | -------------------------------------------------------------------------------- /server.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import json 4 | import os 5 | import numpy as np 6 | import torch.nn 7 | from torch.autograd import Variable 8 | from model import ImgSenRanking 9 | from PIL import Image, ImageFile 10 | from flask import Flask, request, render_template, jsonify 11 | from tools import encode_sentences, encode_images 12 | from pre_transforms import image_transform, resnet 13 | import cPickle as pkl 14 | import torch 15 | # TODO: Defind text_transforms in pre_transforms.py 16 | import jieba.analyse 17 | jieba.analyse.set_stop_words('static/dataset/stopwords.txt') 18 | 19 | app = Flask(__name__) 20 | 21 | 22 | ImageFile.LOAD_TRUNCATED_IMAGES = True 23 | 24 | UPLOAD_FOLDER = 'static/upload/' 25 | dump_path = 'vse/arch_server/' 26 | 27 | app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER 28 | 29 | print 'loading image_dump.json' 30 | images_dump = torch.load(os.path.join(dump_path, 'arch_lim.pkl')) 31 | images_path = json.load(open(os.path.join(dump_path, 'arch_test_imgs_path.json'))) 32 | images_url = json.load(open(os.path.join(dump_path, 'arch_test_imgs_url.json'))) 33 | 34 | print 'loading text_dump.json' 35 | texts_dump = torch.load(os.path.join(dump_path, 'arch_ls.pkl')) 36 | texts_orig = json.load(open(os.path.join(dump_path, 'arch_caps.json'))) 37 | texts_url = json.load(open(os.path.join(dump_path, 'arch_test_caps_url.json'))) 38 | 39 | print 'loading jianzhu model' 40 | model_options = pkl.load(open(os.path.join(dump_path, 'arch_params_dump.pkl'))) 41 | model = ImgSenRanking(model_options).cuda() 42 | model.load_state_dict(torch.load(os.path.join(dump_path, 'arch_model_dump.pkl'))) 43 | 44 | curr_model = {} 45 | curr_model['options'] = model_options 46 | curr_model['worddict'] = model_options['worddict'] 47 | curr_model['word_idict'] = model_options['word_idict'] 48 | curr_model['img_sen_model'] = model 49 | 50 | 51 | @app.route('/') 52 | def index(): 53 | return render_template('index.html') 54 | 55 | @app.route('/query', methods=['POST']) 56 | def query(): 57 | query_sen = request.form.get('query_sentence', '') 58 | k_input = int(request.form.get('k_input', '')) 59 | query_img = request.files['query_image'] 60 | img_name = query_img.filename 61 | upload_img = os.path.join(app.config['UPLOAD_FOLDER'], img_name) 62 | sim_images, sim_images_url = [], [] 63 | sim_texts, sim_texts_url = [], [] 64 | if img_name: 65 | query_img.save(upload_img) 66 | img_vec = image_transform(Image.open(upload_img).convert('RGB')).unsqueeze(0) 67 | image_emb = encode_images(curr_model, resnet(Variable(img_vec.cuda())).data.cpu().numpy()) 68 | d = torch.mm(image_emb, texts_dump.t()) 69 | d_sorted, inds = torch.sort(d, descending=True) 70 | inds = inds.data.squeeze(0).cpu().numpy() 71 | # sim_text_degree = 1-distance[0][:k_input]/distance[0][-1] 72 | sim_texts = np.array(texts_orig)[inds[:k_input]] 73 | sim_texts_url = np.array(texts_url)[inds[:k_input]] 74 | # sim_texts, sim_text_degree = sim_texts.tolist(), sim_text_degree.tolist() 75 | sim_texts, sim_texts_url = sim_texts.tolist(), sim_texts_url.tolist() 76 | if query_sen: 77 | query_sen = ' '.join(jieba.analyse.extract_tags(query_sen, topK=100, withWeight=False, allowPOS=())) 78 | query_sen = [query_sen.encode('utf8')] 79 | sentence = encode_sentences(curr_model, query_sen) 80 | d = torch.mm(sentence, images_dump.t()) 81 | d_sorted, inds = torch.sort(d, descending=True) 82 | inds = inds.data.squeeze(0).cpu().numpy() 83 | # sim_image_degree = 1-distance[0][:k_input]/distance[0][-1] 84 | sim_images = np.array(images_path)[inds[:k_input]] 85 | sim_images_url = np.array(images_url)[inds[:k_input]] 86 | # sim_images, sim_image_degree = sim_images.tolist(), sim_image_degree.tolist() 87 | sim_images, sim_images_url = sim_images.tolist(), sim_images_url.tolist() 88 | 89 | upload_img = upload_img if img_name else 'no_upload_img' 90 | return jsonify(sim_images=sim_images, sim_images_url=sim_images_url, 91 | upload_img=upload_img, sim_texts=sim_texts, sim_texts_url=sim_texts_url) 92 | 93 | 94 | if __name__ == "__main__": 95 | app.run(host='0.0.0.0', port=2333) 96 | -------------------------------------------------------------------------------- /static/dataset/arch/annotations/dump_data_pair.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import os 3 | import json 4 | import random 5 | 6 | dev_num = 10000 7 | 8 | data_pair = [] 9 | 10 | # Keep pace with coco dataset in 11 | # - json field : caption, img_path, obj_id 12 | # - dataset split : train/val=2/1 13 | 14 | # Feel free to choose title or detail as caption 15 | 16 | with open('jianzhu_tag.json') as f_read: 17 | source_data = map(json.loads, f_read.readlines()) 18 | 19 | for k, line in enumerate(source_data): 20 | if k % 1000 == 0: 21 | print 'Processing %d / %d' %(k, len(source_data)) 22 | detail = line['detail'].strip() 23 | title, tag = line['title'].strip(), line['tag'].strip() 24 | title = title if title != '' else tag 25 | url = line['other']['url'] if line['other'].has_key('url') else '' 26 | poster = line['poster'].replace('/data/crawler/', '') 27 | if poster == '': 28 | continue 29 | poster = os.path.join('../../', poster) 30 | 31 | if os.path.exists(poster): 32 | for img_name in os.listdir(poster): 33 | poster = poster.replace('../../', 'static/dataset/') 34 | img_path = os.path.join(poster, img_name) 35 | data_pair.append({'caption': detail, 'img_path': img_path, 36 | 'obj_id': k, 'url': url}) 37 | 38 | random.shuffle(data_pair) 39 | print 'Remaining data_pair: ', len(data_pair) 40 | 41 | json.dump(data_pair[:dev_num], open('data_pair_val.json', 'w')) 42 | json.dump(data_pair[dev_num:], open('data_pair_train.json', 'w')) 43 | -------------------------------------------------------------------------------- /static/web/index.css: -------------------------------------------------------------------------------- 1 | div.header { 2 | text-align: center; 3 | } 4 | 5 | @media (min-width: 768px) { 6 | .container { 7 | max-width: 100000px; 8 | } 9 | } 10 | 11 | div.query_input { 12 | overflow: hidden; 13 | margin-bottom: 50px; 14 | } 15 | 16 | 17 | div.left { 18 | float: left; 19 | width: 50%; 20 | padding-right: 80px; 21 | } 22 | 23 | div.left textarea#query_sentence { 24 | float: right; 25 | } 26 | 27 | div.right { 28 | float: right; 29 | width: 50%; 30 | padding-left: 80px; 31 | } 32 | 33 | #query_sentence { 34 | padding: 20px; 35 | } 36 | 37 | img { 38 | height: 200px; 39 | margin: 10px; 40 | } 41 | 42 | div.query_other { 43 | text-align: center; 44 | } 45 | 46 | div.query_other input { 47 | width: 80px; 48 | padding: 3px; 49 | margin-right: 20px; 50 | } 51 | 52 | div.query_other button { 53 | margin-right: 20px; 54 | } 55 | 56 | form { 57 | margin-bottom: 50px; 58 | } 59 | 60 | div#img_result { 61 | margin-bottom: 50px; 62 | } 63 | 64 | div#text_result { 65 | margin-bottom: 50px; 66 | } 67 | 68 | div#text_result div { 69 | margin: 10px; 70 | overflow: hidden; 71 | } 72 | 73 | div#text_result div a { 74 | float: left; 75 | margin: 10px; 76 | } 77 | 78 | -------------------------------------------------------------------------------- /static/web/index.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Created by lindayong on 17-5-9. 3 | */ 4 | 5 | $(function () { 6 | $('#query_button').click(query); 7 | $('#reset_button').click(function () { 8 | $('form')[0].reset(); 9 | $('#upload_img').remove(); 10 | }); 11 | 12 | function query() { 13 | // var formData = new FormData(document.querySelector('form')); 14 | var formData = new FormData($('form')[0]); 15 | var query_button = $('#query_button'); 16 | query_button.attr('disabled', 'true'); 17 | query_button.text('正在查询中，请稍后 ...'); 18 | $.ajax({ 19 | url: '/query', 20 | data: formData, 21 | type: 'POST', 22 | processData: false, 23 | contentType: false, 24 | success: function (response) { 25 | $('#img_result').empty(); 26 | $('#upload_img').remove(); 27 | $('#text_result').empty(); 28 | query_button.removeAttr('disabled'); 29 | query_button.text('查询'); 30 | console.log(response); 31 | var imgs_dir = response['sim_images']; 32 | var upload_img = response['upload_img']; 33 | var texts = response['sim_texts']; 34 | var imgs_url = response['sim_images_url']; 35 | var texts_url = response['sim_texts_url']; 36 | if (upload_img !== 'no_upload_img') { 37 | upload_img = 'upload_img'; 38 | $('#query_image').after(upload_img) 39 | } 40 | for (num in imgs_dir) { 41 | var img_dir = '../'+imgs_dir[num].replace(/%/g, '%25'); 42 | var img_html = ''; 43 | var img_url = '' + img_html + ''; 44 | $('#img_result').append(img_url); 45 | } 46 | for (num in texts) { 47 | var text_html = '
' + texts[num] + '
'; 48 | var text_url = ' url '; 49 | var text_div = '
' + text_url + text_html + '
'; 50 | $('#text_result').append(text_div); 51 | } 52 | }, 53 | error: function (error) { 54 | console.log(error); 55 | } 56 | }); 57 | } 58 | }); 59 | -------------------------------------------------------------------------------- /static/web/jquery.min.js: -------------------------------------------------------------------------------- 1 | /*! jQuery v3.1.1 | (c) jQuery Foundation | jquery.org/license */ 2 | !function(a,b){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){"use strict";var c=[],d=a.document,e=Object.getPrototypeOf,f=c.slice,g=c.concat,h=c.push,i=c.indexOf,j={},k=j.toString,l=j.hasOwnProperty,m=l.toString,n=m.call(Object),o={};function p(a,b){b=b||d;var c=b.createElement("script");c.text=a,b.head.appendChild(c).parentNode.removeChild(c)}var q="3.1.1",r=function(a,b){return new r.fn.init(a,b)},s=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,t=/^-ms-/,u=/-([a-z])/g,v=function(a,b){return b.toUpperCase()};r.fn=r.prototype={jquery:q,constructor:r,length:0,toArray:function(){return f.call(this)},get:function(a){return null==a?f.call(this):a<0?this[a+this.length]:this[a]},pushStack:function(a){var b=r.merge(this.constructor(),a);return b.prevObject=this,b},each:function(a){return r.each(this,a)},map:function(a){return this.pushStack(r.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(f.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(a<0?b:0);return this.pushStack(c>=0&&c0&&b-1 in a)}var x=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+1*new Date,v=a.document,w=0,x=0,y=ha(),z=ha(),A=ha(),B=function(a,b){return a===b&&(l=!0),0},C={}.hasOwnProperty,D=[],E=D.pop,F=D.push,G=D.push,H=D.slice,I=function(a,b){for(var c=0,d=a.length;c+~]|"+K+")"+K+"*"),S=new RegExp("="+K+"*([^\\]'\"]*?)"+K+"*\\]","g"),T=new RegExp(N),U=new RegExp("^"+L+"$"),V={ID:new RegExp("^#("+L+")"),CLASS:new RegExp("^\\.("+L+")"),TAG:new RegExp("^("+L+"|[*])"),ATTR:new RegExp("^"+M),PSEUDO:new RegExp("^"+N),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+K+"*(even|odd|(([+-]|)(\\d*)n|)"+K+"*(?:([+-]|)"+K+"*(\\d+)|))"+K+"*\\)|)","i"),bool:new RegExp("^(?:"+J+")$","i"),needsContext:new RegExp("^"+K+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+K+"*((?:-\\d)?\\d*)"+K+"*\\)|)(?=[^-]|$)","i")},W=/^(?:input|select|textarea|button)$/i,X=/^h\d$/i,Y=/^[^{]+\{\s*\[native \w/,Z=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,$=/[+~]/,_=new RegExp("\\\\([\\da-f]{1,6}"+K+"?|("+K+")|.)","ig"),aa=function(a,b,c){var d="0x"+b-65536;return d!==d||c?b:d<0?String.fromCharCode(d+65536):String.fromCharCode(d>>10|55296,1023&d|56320)},ba=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ca=function(a,b){return b?"\0"===a?"\ufffd":a.slice(0,-1)+"\\"+a.charCodeAt(a.length-1).toString(16)+" ":"\\"+a},da=function(){m()},ea=ta(function(a){return a.disabled===!0&&("form"in a||"label"in a)},{dir:"parentNode",next:"legend"});try{G.apply(D=H.call(v.childNodes),v.childNodes),D[v.childNodes.length].nodeType}catch(fa){G={apply:D.length?function(a,b){F.apply(a,H.call(b))}:function(a,b){var c=a.length,d=0;while(a[c++]=b[d++]);a.length=c-1}}}function ga(a,b,d,e){var f,h,j,k,l,o,r,s=b&&b.ownerDocument,w=b?b.nodeType:9;if(d=d||[],"string"!=typeof a||!a||1!==w&&9!==w&&11!==w)return d;if(!e&&((b?b.ownerDocument||b:v)!==n&&m(b),b=b||n,p)){if(11!==w&&(l=Z.exec(a)))if(f=l[1]){if(9===w){if(!(j=b.getElementById(f)))return d;if(j.id===f)return d.push(j),d}else if(s&&(j=s.getElementById(f))&&t(b,j)&&j.id===f)return d.push(j),d}else{if(l[2])return G.apply(d,b.getElementsByTagName(a)),d;if((f=l[3])&&c.getElementsByClassName&&b.getElementsByClassName)return G.apply(d,b.getElementsByClassName(f)),d}if(c.qsa&&!A[a+" "]&&(!q||!q.test(a))){if(1!==w)s=b,r=a;else if("object"!==b.nodeName.toLowerCase()){(k=b.getAttribute("id"))?k=k.replace(ba,ca):b.setAttribute("id",k=u),o=g(a),h=o.length;while(h--)o[h]="#"+k+" "+sa(o[h]);r=o.join(","),s=$.test(a)&&qa(b.parentNode)||b}if(r)try{return G.apply(d,s.querySelectorAll(r)),d}catch(x){}finally{k===u&&b.removeAttribute("id")}}}return i(a.replace(P,"$1"),b,d,e)}function ha(){var a=[];function b(c,e){return a.push(c+" ")>d.cacheLength&&delete b[a.shift()],b[c+" "]=e}return b}function ia(a){return a[u]=!0,a}function ja(a){var b=n.createElement("fieldset");try{return!!a(b)}catch(c){return!1}finally{b.parentNode&&b.parentNode.removeChild(b),b=null}}function ka(a,b){var c=a.split("|"),e=c.length;while(e--)d.attrHandle[c[e]]=b}function la(a,b){var c=b&&a,d=c&&1===a.nodeType&&1===b.nodeType&&a.sourceIndex-b.sourceIndex;if(d)return d;if(c)while(c=c.nextSibling)if(c===b)return-1;return a?1:-1}function ma(a){return function(b){var c=b.nodeName.toLowerCase();return"input"===c&&b.type===a}}function na(a){return function(b){var c=b.nodeName.toLowerCase();return("input"===c||"button"===c)&&b.type===a}}function oa(a){return function(b){return"form"in b?b.parentNode&&b.disabled===!1?"label"in b?"label"in b.parentNode?b.parentNode.disabled===a:b.disabled===a:b.isDisabled===a||b.isDisabled!==!a&&ea(b)===a:b.disabled===a:"label"in b&&b.disabled===a}}function pa(a){return ia(function(b){return b=+b,ia(function(c,d){var e,f=a([],c.length,b),g=f.length;while(g--)c[e=f[g]]&&(c[e]=!(d[e]=c[e]))})})}function qa(a){return a&&"undefined"!=typeof a.getElementsByTagName&&a}c=ga.support={},f=ga.isXML=function(a){var b=a&&(a.ownerDocument||a).documentElement;return!!b&&"HTML"!==b.nodeName},m=ga.setDocument=function(a){var b,e,g=a?a.ownerDocument||a:v;return g!==n&&9===g.nodeType&&g.documentElement?(n=g,o=n.documentElement,p=!f(n),v!==n&&(e=n.defaultView)&&e.top!==e&&(e.addEventListener?e.addEventListener("unload",da,!1):e.attachEvent&&e.attachEvent("onunload",da)),c.attributes=ja(function(a){return a.className="i",!a.getAttribute("className")}),c.getElementsByTagName=ja(function(a){return a.appendChild(n.createComment("")),!a.getElementsByTagName("*").length}),c.getElementsByClassName=Y.test(n.getElementsByClassName),c.getById=ja(function(a){return o.appendChild(a).id=u,!n.getElementsByName||!n.getElementsByName(u).length}),c.getById?(d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){return a.getAttribute("id")===b}},d.find.ID=function(a,b){if("undefined"!=typeof b.getElementById&&p){var c=b.getElementById(a);return c?[c]:[]}}):(d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){var c="undefined"!=typeof a.getAttributeNode&&a.getAttributeNode("id");return c&&c.value===b}},d.find.ID=function(a,b){if("undefined"!=typeof b.getElementById&&p){var c,d,e,f=b.getElementById(a);if(f){if(c=f.getAttributeNode("id"),c&&c.value===a)return[f];e=b.getElementsByName(a),d=0;while(f=e[d++])if(c=f.getAttributeNode("id"),c&&c.value===a)return[f]}return[]}}),d.find.TAG=c.getElementsByTagName?function(a,b){return"undefined"!=typeof b.getElementsByTagName?b.getElementsByTagName(a):c.qsa?b.querySelectorAll(a):void 0}:function(a,b){var c,d=[],e=0,f=b.getElementsByTagName(a);if("*"===a){while(c=f[e++])1===c.nodeType&&d.push(c);return d}return f},d.find.CLASS=c.getElementsByClassName&&function(a,b){if("undefined"!=typeof b.getElementsByClassName&&p)return b.getElementsByClassName(a)},r=[],q=[],(c.qsa=Y.test(n.querySelectorAll))&&(ja(function(a){o.appendChild(a).innerHTML="",a.querySelectorAll("[msallowcapture^='']").length&&q.push("[*^$]="+K+"*(?:''|\"\")"),a.querySelectorAll("[selected]").length||q.push("\\["+K+"*(?:value|"+J+")"),a.querySelectorAll("[id~="+u+"-]").length||q.push("~="),a.querySelectorAll(":checked").length||q.push(":checked"),a.querySelectorAll("a#"+u+"+*").length||q.push(".#.+[+~]")}),ja(function(a){a.innerHTML="";var b=n.createElement("input");b.setAttribute("type","hidden"),a.appendChild(b).setAttribute("name","D"),a.querySelectorAll("[name=d]").length&&q.push("name"+K+"*[*^$|!~]?="),2!==a.querySelectorAll(":enabled").length&&q.push(":enabled",":disabled"),o.appendChild(a).disabled=!0,2!==a.querySelectorAll(":disabled").length&&q.push(":enabled",":disabled"),a.querySelectorAll("*,:x"),q.push(",.*:")})),(c.matchesSelector=Y.test(s=o.matches||o.webkitMatchesSelector||o.mozMatchesSelector||o.oMatchesSelector||o.msMatchesSelector))&&ja(function(a){c.disconnectedMatch=s.call(a,"*"),s.call(a,"[s!='']:x"),r.push("!=",N)}),q=q.length&&new RegExp(q.join("|")),r=r.length&&new RegExp(r.join("|")),b=Y.test(o.compareDocumentPosition),t=b||Y.test(o.contains)?function(a,b){var c=9===a.nodeType?a.documentElement:a,d=b&&b.parentNode;return a===d||!(!d||1!==d.nodeType||!(c.contains?c.contains(d):a.compareDocumentPosition&&16&a.compareDocumentPosition(d)))}:function(a,b){if(b)while(b=b.parentNode)if(b===a)return!0;return!1},B=b?function(a,b){if(a===b)return l=!0,0;var d=!a.compareDocumentPosition-!b.compareDocumentPosition;return d?d:(d=(a.ownerDocument||a)===(b.ownerDocument||b)?a.compareDocumentPosition(b):1,1&d||!c.sortDetached&&b.compareDocumentPosition(a)===d?a===n||a.ownerDocument===v&&t(v,a)?-1:b===n||b.ownerDocument===v&&t(v,b)?1:k?I(k,a)-I(k,b):0:4&d?-1:1)}:function(a,b){if(a===b)return l=!0,0;var c,d=0,e=a.parentNode,f=b.parentNode,g=[a],h=[b];if(!e||!f)return a===n?-1:b===n?1:e?-1:f?1:k?I(k,a)-I(k,b):0;if(e===f)return la(a,b);c=a;while(c=c.parentNode)g.unshift(c);c=b;while(c=c.parentNode)h.unshift(c);while(g[d]===h[d])d++;return d?la(g[d],h[d]):g[d]===v?-1:h[d]===v?1:0},n):n},ga.matches=function(a,b){return ga(a,null,null,b)},ga.matchesSelector=function(a,b){if((a.ownerDocument||a)!==n&&m(a),b=b.replace(S,"='$1']"),c.matchesSelector&&p&&!A[b+" "]&&(!r||!r.test(b))&&(!q||!q.test(b)))try{var d=s.call(a,b);if(d||c.disconnectedMatch||a.document&&11!==a.document.nodeType)return d}catch(e){}return ga(b,n,null,[a]).length>0},ga.contains=function(a,b){return(a.ownerDocument||a)!==n&&m(a),t(a,b)},ga.attr=function(a,b){(a.ownerDocument||a)!==n&&m(a);var e=d.attrHandle[b.toLowerCase()],f=e&&C.call(d.attrHandle,b.toLowerCase())?e(a,b,!p):void 0;return void 0!==f?f:c.attributes||!p?a.getAttribute(b):(f=a.getAttributeNode(b))&&f.specified?f.value:null},ga.escape=function(a){return(a+"").replace(ba,ca)},ga.error=function(a){throw new Error("Syntax error, unrecognized expression: "+a)},ga.uniqueSort=function(a){var b,d=[],e=0,f=0;if(l=!c.detectDuplicates,k=!c.sortStable&&a.slice(0),a.sort(B),l){while(b=a[f++])b===a[f]&&(e=d.push(f));while(e--)a.splice(d[e],1)}return k=null,a},e=ga.getText=function(a){var b,c="",d=0,f=a.nodeType;if(f){if(1===f||9===f||11===f){if("string"==typeof a.textContent)return a.textContent;for(a=a.firstChild;a;a=a.nextSibling)c+=e(a)}else if(3===f||4===f)return a.nodeValue}else while(b=a[d++])c+=e(b);return c},d=ga.selectors={cacheLength:50,createPseudo:ia,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(a){return a[1]=a[1].replace(_,aa),a[3]=(a[3]||a[4]||a[5]||"").replace(_,aa),"~="===a[2]&&(a[3]=" "+a[3]+" "),a.slice(0,4)},CHILD:function(a){return a[1]=a[1].toLowerCase(),"nth"===a[1].slice(0,3)?(a[3]||ga.error(a[0]),a[4]=+(a[4]?a[5]+(a[6]||1):2*("even"===a[3]||"odd"===a[3])),a[5]=+(a[7]+a[8]||"odd"===a[3])):a[3]&&ga.error(a[0]),a},PSEUDO:function(a){var b,c=!a[6]&&a[2];return V.CHILD.test(a[0])?null:(a[3]?a[2]=a[4]||a[5]||"":c&&T.test(c)&&(b=g(c,!0))&&(b=c.indexOf(")",c.length-b)-c.length)&&(a[0]=a[0].slice(0,b),a[2]=c.slice(0,b)),a.slice(0,3))}},filter:{TAG:function(a){var b=a.replace(_,aa).toLowerCase();return"*"===a?function(){return!0}:function(a){return a.nodeName&&a.nodeName.toLowerCase()===b}},CLASS:function(a){var b=y[a+" "];return b||(b=new RegExp("(^|"+K+")"+a+"("+K+"|$)"))&&y(a,function(a){return b.test("string"==typeof a.className&&a.className||"undefined"!=typeof a.getAttribute&&a.getAttribute("class")||"")})},ATTR:function(a,b,c){return function(d){var e=ga.attr(d,a);return null==e?"!="===b:!b||(e+="","="===b?e===c:"!="===b?e!==c:"^="===b?c&&0===e.indexOf(c):"*="===b?c&&e.indexOf(c)>-1:"$="===b?c&&e.slice(-c.length)===c:"~="===b?(" "+e.replace(O," ")+" ").indexOf(c)>-1:"|="===b&&(e===c||e.slice(0,c.length+1)===c+"-"))}},CHILD:function(a,b,c,d,e){var f="nth"!==a.slice(0,3),g="last"!==a.slice(-4),h="of-type"===b;return 1===d&&0===e?function(a){return!!a.parentNode}:function(b,c,i){var j,k,l,m,n,o,p=f!==g?"nextSibling":"previousSibling",q=b.parentNode,r=h&&b.nodeName.toLowerCase(),s=!i&&!h,t=!1;if(q){if(f){while(p){m=b;while(m=m[p])if(h?m.nodeName.toLowerCase()===r:1===m.nodeType)return!1;o=p="only"===a&&!o&&"nextSibling"}return!0}if(o=[g?q.firstChild:q.lastChild],g&&s){m=q,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n&&j[2],m=n&&q.childNodes[n];while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if(1===m.nodeType&&++t&&m===b){k[a]=[w,n,t];break}}else if(s&&(m=b,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n),t===!1)while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if((h?m.nodeName.toLowerCase()===r:1===m.nodeType)&&++t&&(s&&(l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),k[a]=[w,t]),m===b))break;return t-=e,t===d||t%d===0&&t/d>=0}}},PSEUDO:function(a,b){var c,e=d.pseudos[a]||d.setFilters[a.toLowerCase()]||ga.error("unsupported pseudo: "+a);return e[u]?e(b):e.length>1?(c=[a,a,"",b],d.setFilters.hasOwnProperty(a.toLowerCase())?ia(function(a,c){var d,f=e(a,b),g=f.length;while(g--)d=I(a,f[g]),a[d]=!(c[d]=f[g])}):function(a){return e(a,0,c)}):e}},pseudos:{not:ia(function(a){var b=[],c=[],d=h(a.replace(P,"$1"));return d[u]?ia(function(a,b,c,e){var f,g=d(a,null,e,[]),h=a.length;while(h--)(f=g[h])&&(a[h]=!(b[h]=f))}):function(a,e,f){return b[0]=a,d(b,null,f,c),b[0]=null,!c.pop()}}),has:ia(function(a){return function(b){return ga(a,b).length>0}}),contains:ia(function(a){return a=a.replace(_,aa),function(b){return(b.textContent||b.innerText||e(b)).indexOf(a)>-1}}),lang:ia(function(a){return U.test(a||"")||ga.error("unsupported lang: "+a),a=a.replace(_,aa).toLowerCase(),function(b){var c;do if(c=p?b.lang:b.getAttribute("xml:lang")||b.getAttribute("lang"))return c=c.toLowerCase(),c===a||0===c.indexOf(a+"-");while((b=b.parentNode)&&1===b.nodeType);return!1}}),target:function(b){var c=a.location&&a.location.hash;return c&&c.slice(1)===b.id},root:function(a){return a===o},focus:function(a){return a===n.activeElement&&(!n.hasFocus||n.hasFocus())&&!!(a.type||a.href||~a.tabIndex)},enabled:oa(!1),disabled:oa(!0),checked:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&!!a.checked||"option"===b&&!!a.selected},selected:function(a){return a.parentNode&&a.parentNode.selectedIndex,a.selected===!0},empty:function(a){for(a=a.firstChild;a;a=a.nextSibling)if(a.nodeType<6)return!1;return!0},parent:function(a){return!d.pseudos.empty(a)},header:function(a){return X.test(a.nodeName)},input:function(a){return W.test(a.nodeName)},button:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&"button"===a.type||"button"===b},text:function(a){var b;return"input"===a.nodeName.toLowerCase()&&"text"===a.type&&(null==(b=a.getAttribute("type"))||"text"===b.toLowerCase())},first:pa(function(){return[0]}),last:pa(function(a,b){return[b-1]}),eq:pa(function(a,b,c){return[c<0?c+b:c]}),even:pa(function(a,b){for(var c=0;c=0;)a.push(d);return a}),gt:pa(function(a,b,c){for(var d=c<0?c+b:c;++d1?function(b,c,d){var e=a.length;while(e--)if(!a[e](b,c,d))return!1;return!0}:a[0]}function va(a,b,c){for(var d=0,e=b.length;d-1&&(f[j]=!(g[j]=l))}}else r=wa(r===g?r.splice(o,r.length):r),e?e(null,g,r,i):G.apply(g,r)})}function ya(a){for(var b,c,e,f=a.length,g=d.relative[a[0].type],h=g||d.relative[" "],i=g?1:0,k=ta(function(a){return a===b},h,!0),l=ta(function(a){return I(b,a)>-1},h,!0),m=[function(a,c,d){var e=!g&&(d||c!==j)||((b=c).nodeType?k(a,c,d):l(a,c,d));return b=null,e}];i1&&ua(m),i>1&&sa(a.slice(0,i-1).concat({value:" "===a[i-2].type?"*":""})).replace(P,"$1"),c,i0,e=a.length>0,f=function(f,g,h,i,k){var l,o,q,r=0,s="0",t=f&&[],u=[],v=j,x=f||e&&d.find.TAG("*",k),y=w+=null==v?1:Math.random()||.1,z=x.length;for(k&&(j=g===n||g||k);s!==z&&null!=(l=x[s]);s++){if(e&&l){o=0,g||l.ownerDocument===n||(m(l),h=!p);while(q=a[o++])if(q(l,g||n,h)){i.push(l);break}k&&(w=y)}c&&((l=!q&&l)&&r--,f&&t.push(l))}if(r+=s,c&&s!==r){o=0;while(q=b[o++])q(t,u,g,h);if(f){if(r>0)while(s--)t[s]||u[s]||(u[s]=E.call(i));u=wa(u)}G.apply(i,u),k&&!f&&u.length>0&&r+b.length>1&&ga.uniqueSort(i)}return k&&(w=y,j=v),t};return c?ia(f):f}return h=ga.compile=function(a,b){var c,d=[],e=[],f=A[a+" "];if(!f){b||(b=g(a)),c=b.length;while(c--)f=ya(b[c]),f[u]?d.push(f):e.push(f);f=A(a,za(e,d)),f.selector=a}return f},i=ga.select=function(a,b,c,e){var f,i,j,k,l,m="function"==typeof a&&a,n=!e&&g(a=m.selector||a);if(c=c||[],1===n.length){if(i=n[0]=n[0].slice(0),i.length>2&&"ID"===(j=i[0]).type&&9===b.nodeType&&p&&d.relative[i[1].type]){if(b=(d.find.ID(j.matches[0].replace(_,aa),b)||[])[0],!b)return c;m&&(b=b.parentNode),a=a.slice(i.shift().value.length)}f=V.needsContext.test(a)?0:i.length;while(f--){if(j=i[f],d.relative[k=j.type])break;if((l=d.find[k])&&(e=l(j.matches[0].replace(_,aa),$.test(i[0].type)&&qa(b.parentNode)||b))){if(i.splice(f,1),a=e.length&&sa(i),!a)return G.apply(c,e),c;break}}}return(m||h(a,n))(e,b,!p,c,!b||$.test(a)&&qa(b.parentNode)||b),c},c.sortStable=u.split("").sort(B).join("")===u,c.detectDuplicates=!!l,m(),c.sortDetached=ja(function(a){return 1&a.compareDocumentPosition(n.createElement("fieldset"))}),ja(function(a){return a.innerHTML="","#"===a.firstChild.getAttribute("href")})||ka("type|href|height|width",function(a,b,c){if(!c)return a.getAttribute(b,"type"===b.toLowerCase()?1:2)}),c.attributes&&ja(function(a){return a.innerHTML="",a.firstChild.setAttribute("value",""),""===a.firstChild.getAttribute("value")})||ka("value",function(a,b,c){if(!c&&"input"===a.nodeName.toLowerCase())return a.defaultValue}),ja(function(a){return null==a.getAttribute("disabled")})||ka(J,function(a,b,c){var d;if(!c)return a[b]===!0?b.toLowerCase():(d=a.getAttributeNode(b))&&d.specified?d.value:null}),ga}(a);r.find=x,r.expr=x.selectors,r.expr[":"]=r.expr.pseudos,r.uniqueSort=r.unique=x.uniqueSort,r.text=x.getText,r.isXMLDoc=x.isXML,r.contains=x.contains,r.escapeSelector=x.escape;var y=function(a,b,c){var d=[],e=void 0!==c;while((a=a[b])&&9!==a.nodeType)if(1===a.nodeType){if(e&&r(a).is(c))break;d.push(a)}return d},z=function(a,b){for(var c=[];a;a=a.nextSibling)1===a.nodeType&&a!==b&&c.push(a);return c},A=r.expr.match.needsContext,B=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i,C=/^.[^:#\[\.,]*$/;function D(a,b,c){return r.isFunction(b)?r.grep(a,function(a,d){return!!b.call(a,d,a)!==c}):b.nodeType?r.grep(a,function(a){return a===b!==c}):"string"!=typeof b?r.grep(a,function(a){return i.call(b,a)>-1!==c}):C.test(b)?r.filter(b,a,c):(b=r.filter(b,a),r.grep(a,function(a){return i.call(b,a)>-1!==c&&1===a.nodeType}))}r.filter=function(a,b,c){var d=b[0];return c&&(a=":not("+a+")"),1===b.length&&1===d.nodeType?r.find.matchesSelector(d,a)?[d]:[]:r.find.matches(a,r.grep(b,function(a){return 1===a.nodeType}))},r.fn.extend({find:function(a){var b,c,d=this.length,e=this;if("string"!=typeof a)return this.pushStack(r(a).filter(function(){for(b=0;b1?r.uniqueSort(c):c},filter:function(a){return this.pushStack(D(this,a||[],!1))},not:function(a){return this.pushStack(D(this,a||[],!0))},is:function(a){return!!D(this,"string"==typeof a&&A.test(a)?r(a):a||[],!1).length}});var E,F=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/,G=r.fn.init=function(a,b,c){var e,f;if(!a)return this;if(c=c||E,"string"==typeof a){if(e="<"===a[0]&&">"===a[a.length-1]&&a.length>=3?[null,a,null]:F.exec(a),!e||!e[1]&&b)return!b||b.jquery?(b||c).find(a):this.constructor(b).find(a);if(e[1]){if(b=b instanceof r?b[0]:b,r.merge(this,r.parseHTML(e[1],b&&b.nodeType?b.ownerDocument||b:d,!0)),B.test(e[1])&&r.isPlainObject(b))for(e in b)r.isFunction(this[e])?this[e](b[e]):this.attr(e,b[e]);return this}return f=d.getElementById(e[2]),f&&(this[0]=f,this.length=1),this}return a.nodeType?(this[0]=a,this.length=1,this):r.isFunction(a)?void 0!==c.ready?c.ready(a):a(r):r.makeArray(a,this)};G.prototype=r.fn,E=r(d);var H=/^(?:parents|prev(?:Until|All))/,I={children:!0,contents:!0,next:!0,prev:!0};r.fn.extend({has:function(a){var b=r(a,this),c=b.length;return this.filter(function(){for(var a=0;a-1:1===c.nodeType&&r.find.matchesSelector(c,a))){f.push(c);break}return this.pushStack(f.length>1?r.uniqueSort(f):f)},index:function(a){return a?"string"==typeof a?i.call(r(a),this[0]):i.call(this,a.jquery?a[0]:a):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(a,b){return this.pushStack(r.uniqueSort(r.merge(this.get(),r(a,b))))},addBack:function(a){return this.add(null==a?this.prevObject:this.prevObject.filter(a))}});function J(a,b){while((a=a[b])&&1!==a.nodeType);return a}r.each({parent:function(a){var b=a.parentNode;return b&&11!==b.nodeType?b:null},parents:function(a){return y(a,"parentNode")},parentsUntil:function(a,b,c){return y(a,"parentNode",c)},next:function(a){return J(a,"nextSibling")},prev:function(a){return J(a,"previousSibling")},nextAll:function(a){return y(a,"nextSibling")},prevAll:function(a){return y(a,"previousSibling")},nextUntil:function(a,b,c){return y(a,"nextSibling",c)},prevUntil:function(a,b,c){return y(a,"previousSibling",c)},siblings:function(a){return z((a.parentNode||{}).firstChild,a)},children:function(a){return z(a.firstChild)},contents:function(a){return a.contentDocument||r.merge([],a.childNodes)}},function(a,b){r.fn[a]=function(c,d){var e=r.map(this,b,c);return"Until"!==a.slice(-5)&&(d=c),d&&"string"==typeof d&&(e=r.filter(d,e)),this.length>1&&(I[a]||r.uniqueSort(e),H.test(a)&&e.reverse()),this.pushStack(e)}});var K=/[^\x20\t\r\n\f]+/g;function L(a){var b={};return r.each(a.match(K)||[],function(a,c){b[c]=!0}),b}r.Callbacks=function(a){a="string"==typeof a?L(a):r.extend({},a);var b,c,d,e,f=[],g=[],h=-1,i=function(){for(e=a.once,d=b=!0;g.length;h=-1){c=g.shift();while(++h-1)f.splice(c,1),c<=h&&h--}),this},has:function(a){return a?r.inArray(a,f)>-1:f.length>0},empty:function(){return f&&(f=[]),this},disable:function(){return e=g=[],f=c="",this},disabled:function(){return!f},lock:function(){return e=g=[],c||b||(f=c=""),this},locked:function(){return!!e},fireWith:function(a,c){return e||(c=c||[],c=[a,c.slice?c.slice():c],g.push(c),b||i()),this},fire:function(){return j.fireWith(this,arguments),this},fired:function(){return!!d}};return j};function M(a){return a}function N(a){throw a}function O(a,b,c){var d;try{a&&r.isFunction(d=a.promise)?d.call(a).done(b).fail(c):a&&r.isFunction(d=a.then)?d.call(a,b,c):b.call(void 0,a)}catch(a){c.call(void 0,a)}}r.extend({Deferred:function(b){var c=[["notify","progress",r.Callbacks("memory"),r.Callbacks("memory"),2],["resolve","done",r.Callbacks("once memory"),r.Callbacks("once memory"),0,"resolved"],["reject","fail",r.Callbacks("once memory"),r.Callbacks("once memory"),1,"rejected"]],d="pending",e={state:function(){return d},always:function(){return f.done(arguments).fail(arguments),this},"catch":function(a){return e.then(null,a)},pipe:function(){var a=arguments;return r.Deferred(function(b){r.each(c,function(c,d){var e=r.isFunction(a[d[4]])&&a[d[4]];f[d[1]](function(){var a=e&&e.apply(this,arguments);a&&r.isFunction(a.promise)?a.promise().progress(b.notify).done(b.resolve).fail(b.reject):b[d[0]+"With"](this,e?[a]:arguments)})}),a=null}).promise()},then:function(b,d,e){var f=0;function g(b,c,d,e){return function(){var h=this,i=arguments,j=function(){var a,j;if(!(b=f&&(d!==N&&(h=void 0,i=[a]),c.rejectWith(h,i))}};b?k():(r.Deferred.getStackHook&&(k.stackTrace=r.Deferred.getStackHook()),a.setTimeout(k))}}return r.Deferred(function(a){c[0][3].add(g(0,a,r.isFunction(e)?e:M,a.notifyWith)),c[1][3].add(g(0,a,r.isFunction(b)?b:M)),c[2][3].add(g(0,a,r.isFunction(d)?d:N))}).promise()},promise:function(a){return null!=a?r.extend(a,e):e}},f={};return r.each(c,function(a,b){var g=b[2],h=b[5];e[b[1]]=g.add,h&&g.add(function(){d=h},c[3-a][2].disable,c[0][2].lock),g.add(b[3].fire),f[b[0]]=function(){return f[b[0]+"With"](this===f?void 0:this,arguments),this},f[b[0]+"With"]=g.fireWith}),e.promise(f),b&&b.call(f,f),f},when:function(a){var b=arguments.length,c=b,d=Array(c),e=f.call(arguments),g=r.Deferred(),h=function(a){return function(c){d[a]=this,e[a]=arguments.length>1?f.call(arguments):c,--b||g.resolveWith(d,e)}};if(b<=1&&(O(a,g.done(h(c)).resolve,g.reject),"pending"===g.state()||r.isFunction(e[c]&&e[c].then)))return g.then();while(c--)O(e[c],h(c),g.reject);return g.promise()}});var P=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;r.Deferred.exceptionHook=function(b,c){a.console&&a.console.warn&&b&&P.test(b.name)&&a.console.warn("jQuery.Deferred exception: "+b.message,b.stack,c)},r.readyException=function(b){a.setTimeout(function(){throw b})};var Q=r.Deferred();r.fn.ready=function(a){return Q.then(a)["catch"](function(a){r.readyException(a)}),this},r.extend({isReady:!1,readyWait:1,holdReady:function(a){a?r.readyWait++:r.ready(!0)},ready:function(a){(a===!0?--r.readyWait:r.isReady)||(r.isReady=!0,a!==!0&&--r.readyWait>0||Q.resolveWith(d,[r]))}}),r.ready.then=Q.then;function R(){d.removeEventListener("DOMContentLoaded",R), 3 | a.removeEventListener("load",R),r.ready()}"complete"===d.readyState||"loading"!==d.readyState&&!d.documentElement.doScroll?a.setTimeout(r.ready):(d.addEventListener("DOMContentLoaded",R),a.addEventListener("load",R));var S=function(a,b,c,d,e,f,g){var h=0,i=a.length,j=null==c;if("object"===r.type(c)){e=!0;for(h in c)S(a,b,h,c[h],!0,f,g)}else if(void 0!==d&&(e=!0,r.isFunction(d)||(g=!0),j&&(g?(b.call(a,d),b=null):(j=b,b=function(a,b,c){return j.call(r(a),c)})),b))for(;h1,null,!0)},removeData:function(a){return this.each(function(){W.remove(this,a)})}}),r.extend({queue:function(a,b,c){var d;if(a)return b=(b||"fx")+"queue",d=V.get(a,b),c&&(!d||r.isArray(c)?d=V.access(a,b,r.makeArray(c)):d.push(c)),d||[]},dequeue:function(a,b){b=b||"fx";var c=r.queue(a,b),d=c.length,e=c.shift(),f=r._queueHooks(a,b),g=function(){r.dequeue(a,b)};"inprogress"===e&&(e=c.shift(),d--),e&&("fx"===b&&c.unshift("inprogress"),delete f.stop,e.call(a,g,f)),!d&&f&&f.empty.fire()},_queueHooks:function(a,b){var c=b+"queueHooks";return V.get(a,c)||V.access(a,c,{empty:r.Callbacks("once memory").add(function(){V.remove(a,[b+"queue",c])})})}}),r.fn.extend({queue:function(a,b){var c=2;return"string"!=typeof a&&(b=a,a="fx",c--),arguments.length\x20\t\r\n\f]+)/i,ka=/^$|\/(?:java|ecma)script/i,la={option:[1,""],thead:[1,"","
"],col:[2,"","
"],tr:[2,"","
"],td:[3,"","
"],_default:[0,"",""]};la.optgroup=la.option,la.tbody=la.tfoot=la.colgroup=la.caption=la.thead,la.th=la.td;function ma(a,b){var c;return c="undefined"!=typeof a.getElementsByTagName?a.getElementsByTagName(b||"*"):"undefined"!=typeof a.querySelectorAll?a.querySelectorAll(b||"*"):[],void 0===b||b&&r.nodeName(a,b)?r.merge([a],c):c}function na(a,b){for(var c=0,d=a.length;c-1)e&&e.push(f);else if(j=r.contains(f.ownerDocument,f),g=ma(l.appendChild(f),"script"),j&&na(g),c){k=0;while(f=g[k++])ka.test(f.type||"")&&c.push(f)}return l}!function(){var a=d.createDocumentFragment(),b=a.appendChild(d.createElement("div")),c=d.createElement("input");c.setAttribute("type","radio"),c.setAttribute("checked","checked"),c.setAttribute("name","t"),b.appendChild(c),o.checkClone=b.cloneNode(!0).cloneNode(!0).lastChild.checked,b.innerHTML="",o.noCloneChecked=!!b.cloneNode(!0).lastChild.defaultValue}();var qa=d.documentElement,ra=/^key/,sa=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,ta=/^([^.]*)(?:\.(.+)|)/;function ua(){return!0}function va(){return!1}function wa(){try{return d.activeElement}catch(a){}}function xa(a,b,c,d,e,f){var g,h;if("object"==typeof b){"string"!=typeof c&&(d=d||c,c=void 0);for(h in b)xa(a,h,c,d,b[h],f);return a}if(null==d&&null==e?(e=c,d=c=void 0):null==e&&("string"==typeof c?(e=d,d=void 0):(e=d,d=c,c=void 0)),e===!1)e=va;else if(!e)return a;return 1===f&&(g=e,e=function(a){return r().off(a),g.apply(this,arguments)},e.guid=g.guid||(g.guid=r.guid++)),a.each(function(){r.event.add(this,b,e,d,c)})}r.event={global:{},add:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=V.get(a);if(q){c.handler&&(f=c,c=f.handler,e=f.selector),e&&r.find.matchesSelector(qa,e),c.guid||(c.guid=r.guid++),(i=q.events)||(i=q.events={}),(g=q.handle)||(g=q.handle=function(b){return"undefined"!=typeof r&&r.event.triggered!==b.type?r.event.dispatch.apply(a,arguments):void 0}),b=(b||"").match(K)||[""],j=b.length;while(j--)h=ta.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n&&(l=r.event.special[n]||{},n=(e?l.delegateType:l.bindType)||n,l=r.event.special[n]||{},k=r.extend({type:n,origType:p,data:d,handler:c,guid:c.guid,selector:e,needsContext:e&&r.expr.match.needsContext.test(e),namespace:o.join(".")},f),(m=i[n])||(m=i[n]=[],m.delegateCount=0,l.setup&&l.setup.call(a,d,o,g)!==!1||a.addEventListener&&a.addEventListener(n,g)),l.add&&(l.add.call(a,k),k.handler.guid||(k.handler.guid=c.guid)),e?m.splice(m.delegateCount++,0,k):m.push(k),r.event.global[n]=!0)}},remove:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=V.hasData(a)&&V.get(a);if(q&&(i=q.events)){b=(b||"").match(K)||[""],j=b.length;while(j--)if(h=ta.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n){l=r.event.special[n]||{},n=(d?l.delegateType:l.bindType)||n,m=i[n]||[],h=h[2]&&new RegExp("(^|\\.)"+o.join("\\.(?:.*\\.|)")+"(\\.|$)"),g=f=m.length;while(f--)k=m[f],!e&&p!==k.origType||c&&c.guid!==k.guid||h&&!h.test(k.namespace)||d&&d!==k.selector&&("**"!==d||!k.selector)||(m.splice(f,1),k.selector&&m.delegateCount--,l.remove&&l.remove.call(a,k));g&&!m.length&&(l.teardown&&l.teardown.call(a,o,q.handle)!==!1||r.removeEvent(a,n,q.handle),delete i[n])}else for(n in i)r.event.remove(a,n+b[j],c,d,!0);r.isEmptyObject(i)&&V.remove(a,"handle events")}},dispatch:function(a){var b=r.event.fix(a),c,d,e,f,g,h,i=new Array(arguments.length),j=(V.get(this,"events")||{})[b.type]||[],k=r.event.special[b.type]||{};for(i[0]=b,c=1;c=1))for(;j!==this;j=j.parentNode||this)if(1===j.nodeType&&("click"!==a.type||j.disabled!==!0)){for(f=[],g={},c=0;c-1:r.find(e,this,null,[j]).length),g[e]&&f.push(d);f.length&&h.push({elem:j,handlers:f})}return j=this,i\x20\t\r\n\f]*)[^>]*)\/>/gi,za=/\s*$/g;function Da(a,b){return r.nodeName(a,"table")&&r.nodeName(11!==b.nodeType?b:b.firstChild,"tr")?a.getElementsByTagName("tbody")[0]||a:a}function Ea(a){return a.type=(null!==a.getAttribute("type"))+"/"+a.type,a}function Fa(a){var b=Ba.exec(a.type);return b?a.type=b[1]:a.removeAttribute("type"),a}function Ga(a,b){var c,d,e,f,g,h,i,j;if(1===b.nodeType){if(V.hasData(a)&&(f=V.access(a),g=V.set(b,f),j=f.events)){delete g.handle,g.events={};for(e in j)for(c=0,d=j[e].length;c1&&"string"==typeof q&&!o.checkClone&&Aa.test(q))return a.each(function(e){var f=a.eq(e);s&&(b[0]=q.call(this,e,f.html())),Ia(f,b,c,d)});if(m&&(e=pa(b,a[0].ownerDocument,!1,a,d),f=e.firstChild,1===e.childNodes.length&&(e=f),f||d)){for(h=r.map(ma(e,"script"),Ea),i=h.length;l")},clone:function(a,b,c){var d,e,f,g,h=a.cloneNode(!0),i=r.contains(a.ownerDocument,a);if(!(o.noCloneChecked||1!==a.nodeType&&11!==a.nodeType||r.isXMLDoc(a)))for(g=ma(h),f=ma(a),d=0,e=f.length;d0&&na(g,!i&&ma(a,"script")),h},cleanData:function(a){for(var b,c,d,e=r.event.special,f=0;void 0!==(c=a[f]);f++)if(T(c)){if(b=c[V.expando]){if(b.events)for(d in b.events)e[d]?r.event.remove(c,d):r.removeEvent(c,d,b.handle);c[V.expando]=void 0}c[W.expando]&&(c[W.expando]=void 0)}}}),r.fn.extend({detach:function(a){return Ja(this,a,!0)},remove:function(a){return Ja(this,a)},text:function(a){return S(this,function(a){return void 0===a?r.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=a)})},null,a,arguments.length)},append:function(){return Ia(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Da(this,a);b.appendChild(a)}})},prepend:function(){return Ia(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Da(this,a);b.insertBefore(a,b.firstChild)}})},before:function(){return Ia(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this)})},after:function(){return Ia(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this.nextSibling)})},empty:function(){for(var a,b=0;null!=(a=this[b]);b++)1===a.nodeType&&(r.cleanData(ma(a,!1)),a.textContent="");return this},clone:function(a,b){return a=null!=a&&a,b=null==b?a:b,this.map(function(){return r.clone(this,a,b)})},html:function(a){return S(this,function(a){var b=this[0]||{},c=0,d=this.length;if(void 0===a&&1===b.nodeType)return b.innerHTML;if("string"==typeof a&&!za.test(a)&&!la[(ja.exec(a)||["",""])[1].toLowerCase()]){a=r.htmlPrefilter(a);try{for(;c1)}});function Ya(a,b,c,d,e){return new Ya.prototype.init(a,b,c,d,e)}r.Tween=Ya,Ya.prototype={constructor:Ya,init:function(a,b,c,d,e,f){this.elem=a,this.prop=c,this.easing=e||r.easing._default,this.options=b,this.start=this.now=this.cur(),this.end=d,this.unit=f||(r.cssNumber[c]?"":"px")},cur:function(){var a=Ya.propHooks[this.prop];return a&&a.get?a.get(this):Ya.propHooks._default.get(this)},run:function(a){var b,c=Ya.propHooks[this.prop];return this.options.duration?this.pos=b=r.easing[this.easing](a,this.options.duration*a,0,1,this.options.duration):this.pos=b=a,this.now=(this.end-this.start)*b+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),c&&c.set?c.set(this):Ya.propHooks._default.set(this),this}},Ya.prototype.init.prototype=Ya.prototype,Ya.propHooks={_default:{get:function(a){var b;return 1!==a.elem.nodeType||null!=a.elem[a.prop]&&null==a.elem.style[a.prop]?a.elem[a.prop]:(b=r.css(a.elem,a.prop,""),b&&"auto"!==b?b:0)},set:function(a){r.fx.step[a.prop]?r.fx.step[a.prop](a):1!==a.elem.nodeType||null==a.elem.style[r.cssProps[a.prop]]&&!r.cssHooks[a.prop]?a.elem[a.prop]=a.now:r.style(a.elem,a.prop,a.now+a.unit)}}},Ya.propHooks.scrollTop=Ya.propHooks.scrollLeft={set:function(a){a.elem.nodeType&&a.elem.parentNode&&(a.elem[a.prop]=a.now)}},r.easing={linear:function(a){return a},swing:function(a){return.5-Math.cos(a*Math.PI)/2},_default:"swing"},r.fx=Ya.prototype.init,r.fx.step={};var Za,$a,_a=/^(?:toggle|show|hide)$/,ab=/queueHooks$/;function bb(){$a&&(a.requestAnimationFrame(bb),r.fx.tick())}function cb(){return a.setTimeout(function(){Za=void 0}),Za=r.now()}function db(a,b){var c,d=0,e={height:a};for(b=b?1:0;d<4;d+=2-b)c=ba[d],e["margin"+c]=e["padding"+c]=a;return b&&(e.opacity=e.width=a),e}function eb(a,b,c){for(var d,e=(hb.tweeners[b]||[]).concat(hb.tweeners["*"]),f=0,g=e.length;f1)},removeAttr:function(a){return this.each(function(){r.removeAttr(this,a)})}}),r.extend({attr:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return"undefined"==typeof a.getAttribute?r.prop(a,b,c):(1===f&&r.isXMLDoc(a)||(e=r.attrHooks[b.toLowerCase()]||(r.expr.match.bool.test(b)?ib:void 0)), 4 | void 0!==c?null===c?void r.removeAttr(a,b):e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:(a.setAttribute(b,c+""),c):e&&"get"in e&&null!==(d=e.get(a,b))?d:(d=r.find.attr(a,b),null==d?void 0:d))},attrHooks:{type:{set:function(a,b){if(!o.radioValue&&"radio"===b&&r.nodeName(a,"input")){var c=a.value;return a.setAttribute("type",b),c&&(a.value=c),b}}}},removeAttr:function(a,b){var c,d=0,e=b&&b.match(K);if(e&&1===a.nodeType)while(c=e[d++])a.removeAttribute(c)}}),ib={set:function(a,b,c){return b===!1?r.removeAttr(a,c):a.setAttribute(c,c),c}},r.each(r.expr.match.bool.source.match(/\w+/g),function(a,b){var c=jb[b]||r.find.attr;jb[b]=function(a,b,d){var e,f,g=b.toLowerCase();return d||(f=jb[g],jb[g]=e,e=null!=c(a,b,d)?g:null,jb[g]=f),e}});var kb=/^(?:input|select|textarea|button)$/i,lb=/^(?:a|area)$/i;r.fn.extend({prop:function(a,b){return S(this,r.prop,a,b,arguments.length>1)},removeProp:function(a){return this.each(function(){delete this[r.propFix[a]||a]})}}),r.extend({prop:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return 1===f&&r.isXMLDoc(a)||(b=r.propFix[b]||b,e=r.propHooks[b]),void 0!==c?e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:a[b]=c:e&&"get"in e&&null!==(d=e.get(a,b))?d:a[b]},propHooks:{tabIndex:{get:function(a){var b=r.find.attr(a,"tabindex");return b?parseInt(b,10):kb.test(a.nodeName)||lb.test(a.nodeName)&&a.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),o.optSelected||(r.propHooks.selected={get:function(a){var b=a.parentNode;return b&&b.parentNode&&b.parentNode.selectedIndex,null},set:function(a){var b=a.parentNode;b&&(b.selectedIndex,b.parentNode&&b.parentNode.selectedIndex)}}),r.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){r.propFix[this.toLowerCase()]=this});function mb(a){var b=a.match(K)||[];return b.join(" ")}function nb(a){return a.getAttribute&&a.getAttribute("class")||""}r.fn.extend({addClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).addClass(a.call(this,b,nb(this)))});if("string"==typeof a&&a){b=a.match(K)||[];while(c=this[i++])if(e=nb(c),d=1===c.nodeType&&" "+mb(e)+" "){g=0;while(f=b[g++])d.indexOf(" "+f+" ")<0&&(d+=f+" ");h=mb(d),e!==h&&c.setAttribute("class",h)}}return this},removeClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).removeClass(a.call(this,b,nb(this)))});if(!arguments.length)return this.attr("class","");if("string"==typeof a&&a){b=a.match(K)||[];while(c=this[i++])if(e=nb(c),d=1===c.nodeType&&" "+mb(e)+" "){g=0;while(f=b[g++])while(d.indexOf(" "+f+" ")>-1)d=d.replace(" "+f+" "," ");h=mb(d),e!==h&&c.setAttribute("class",h)}}return this},toggleClass:function(a,b){var c=typeof a;return"boolean"==typeof b&&"string"===c?b?this.addClass(a):this.removeClass(a):r.isFunction(a)?this.each(function(c){r(this).toggleClass(a.call(this,c,nb(this),b),b)}):this.each(function(){var b,d,e,f;if("string"===c){d=0,e=r(this),f=a.match(K)||[];while(b=f[d++])e.hasClass(b)?e.removeClass(b):e.addClass(b)}else void 0!==a&&"boolean"!==c||(b=nb(this),b&&V.set(this,"__className__",b),this.setAttribute&&this.setAttribute("class",b||a===!1?"":V.get(this,"__className__")||""))})},hasClass:function(a){var b,c,d=0;b=" "+a+" ";while(c=this[d++])if(1===c.nodeType&&(" "+mb(nb(c))+" ").indexOf(b)>-1)return!0;return!1}});var ob=/\r/g;r.fn.extend({val:function(a){var b,c,d,e=this[0];{if(arguments.length)return d=r.isFunction(a),this.each(function(c){var e;1===this.nodeType&&(e=d?a.call(this,c,r(this).val()):a,null==e?e="":"number"==typeof e?e+="":r.isArray(e)&&(e=r.map(e,function(a){return null==a?"":a+""})),b=r.valHooks[this.type]||r.valHooks[this.nodeName.toLowerCase()],b&&"set"in b&&void 0!==b.set(this,e,"value")||(this.value=e))});if(e)return b=r.valHooks[e.type]||r.valHooks[e.nodeName.toLowerCase()],b&&"get"in b&&void 0!==(c=b.get(e,"value"))?c:(c=e.value,"string"==typeof c?c.replace(ob,""):null==c?"":c)}}}),r.extend({valHooks:{option:{get:function(a){var b=r.find.attr(a,"value");return null!=b?b:mb(r.text(a))}},select:{get:function(a){var b,c,d,e=a.options,f=a.selectedIndex,g="select-one"===a.type,h=g?null:[],i=g?f+1:e.length;for(d=f<0?i:g?f:0;d-1)&&(c=!0);return c||(a.selectedIndex=-1),f}}}}),r.each(["radio","checkbox"],function(){r.valHooks[this]={set:function(a,b){if(r.isArray(b))return a.checked=r.inArray(r(a).val(),b)>-1}},o.checkOn||(r.valHooks[this].get=function(a){return null===a.getAttribute("value")?"on":a.value})});var pb=/^(?:focusinfocus|focusoutblur)$/;r.extend(r.event,{trigger:function(b,c,e,f){var g,h,i,j,k,m,n,o=[e||d],p=l.call(b,"type")?b.type:b,q=l.call(b,"namespace")?b.namespace.split("."):[];if(h=i=e=e||d,3!==e.nodeType&&8!==e.nodeType&&!pb.test(p+r.event.triggered)&&(p.indexOf(".")>-1&&(q=p.split("."),p=q.shift(),q.sort()),k=p.indexOf(":")<0&&"on"+p,b=b[r.expando]?b:new r.Event(p,"object"==typeof b&&b),b.isTrigger=f?2:3,b.namespace=q.join("."),b.rnamespace=b.namespace?new RegExp("(^|\\.)"+q.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,b.result=void 0,b.target||(b.target=e),c=null==c?[b]:r.makeArray(c,[b]),n=r.event.special[p]||{},f||!n.trigger||n.trigger.apply(e,c)!==!1)){if(!f&&!n.noBubble&&!r.isWindow(e)){for(j=n.delegateType||p,pb.test(j+p)||(h=h.parentNode);h;h=h.parentNode)o.push(h),i=h;i===(e.ownerDocument||d)&&o.push(i.defaultView||i.parentWindow||a)}g=0;while((h=o[g++])&&!b.isPropagationStopped())b.type=g>1?j:n.bindType||p,m=(V.get(h,"events")||{})[b.type]&&V.get(h,"handle"),m&&m.apply(h,c),m=k&&h[k],m&&m.apply&&T(h)&&(b.result=m.apply(h,c),b.result===!1&&b.preventDefault());return b.type=p,f||b.isDefaultPrevented()||n._default&&n._default.apply(o.pop(),c)!==!1||!T(e)||k&&r.isFunction(e[p])&&!r.isWindow(e)&&(i=e[k],i&&(e[k]=null),r.event.triggered=p,e[p](),r.event.triggered=void 0,i&&(e[k]=i)),b.result}},simulate:function(a,b,c){var d=r.extend(new r.Event,c,{type:a,isSimulated:!0});r.event.trigger(d,null,b)}}),r.fn.extend({trigger:function(a,b){return this.each(function(){r.event.trigger(a,b,this)})},triggerHandler:function(a,b){var c=this[0];if(c)return r.event.trigger(a,b,c,!0)}}),r.each("blur focus focusin focusout resize scroll click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup contextmenu".split(" "),function(a,b){r.fn[b]=function(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),r.fn.extend({hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)}}),o.focusin="onfocusin"in a,o.focusin||r.each({focus:"focusin",blur:"focusout"},function(a,b){var c=function(a){r.event.simulate(b,a.target,r.event.fix(a))};r.event.special[b]={setup:function(){var d=this.ownerDocument||this,e=V.access(d,b);e||d.addEventListener(a,c,!0),V.access(d,b,(e||0)+1)},teardown:function(){var d=this.ownerDocument||this,e=V.access(d,b)-1;e?V.access(d,b,e):(d.removeEventListener(a,c,!0),V.remove(d,b))}}});var qb=a.location,rb=r.now(),sb=/\?/;r.parseXML=function(b){var c;if(!b||"string"!=typeof b)return null;try{c=(new a.DOMParser).parseFromString(b,"text/xml")}catch(d){c=void 0}return c&&!c.getElementsByTagName("parsererror").length||r.error("Invalid XML: "+b),c};var tb=/\[\]$/,ub=/\r?\n/g,vb=/^(?:submit|button|image|reset|file)$/i,wb=/^(?:input|select|textarea|keygen)/i;function xb(a,b,c,d){var e;if(r.isArray(b))r.each(b,function(b,e){c||tb.test(a)?d(a,e):xb(a+"["+("object"==typeof e&&null!=e?b:"")+"]",e,c,d)});else if(c||"object"!==r.type(b))d(a,b);else for(e in b)xb(a+"["+e+"]",b[e],c,d)}r.param=function(a,b){var c,d=[],e=function(a,b){var c=r.isFunction(b)?b():b;d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(null==c?"":c)};if(r.isArray(a)||a.jquery&&!r.isPlainObject(a))r.each(a,function(){e(this.name,this.value)});else for(c in a)xb(c,a[c],b,e);return d.join("&")},r.fn.extend({serialize:function(){return r.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var a=r.prop(this,"elements");return a?r.makeArray(a):this}).filter(function(){var a=this.type;return this.name&&!r(this).is(":disabled")&&wb.test(this.nodeName)&&!vb.test(a)&&(this.checked||!ia.test(a))}).map(function(a,b){var c=r(this).val();return null==c?null:r.isArray(c)?r.map(c,function(a){return{name:b.name,value:a.replace(ub,"\r\n")}}):{name:b.name,value:c.replace(ub,"\r\n")}}).get()}});var yb=/%20/g,zb=/#.*$/,Ab=/([?&])_=[^&]*/,Bb=/^(.*?):[ \t]*([^\r\n]*)$/gm,Cb=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Db=/^(?:GET|HEAD)$/,Eb=/^\/\//,Fb={},Gb={},Hb="*/".concat("*"),Ib=d.createElement("a");Ib.href=qb.href;function Jb(a){return function(b,c){"string"!=typeof b&&(c=b,b="*");var d,e=0,f=b.toLowerCase().match(K)||[];if(r.isFunction(c))while(d=f[e++])"+"===d[0]?(d=d.slice(1)||"*",(a[d]=a[d]||[]).unshift(c)):(a[d]=a[d]||[]).push(c)}}function Kb(a,b,c,d){var e={},f=a===Gb;function g(h){var i;return e[h]=!0,r.each(a[h]||[],function(a,h){var j=h(b,c,d);return"string"!=typeof j||f||e[j]?f?!(i=j):void 0:(b.dataTypes.unshift(j),g(j),!1)}),i}return g(b.dataTypes[0])||!e["*"]&&g("*")}function Lb(a,b){var c,d,e=r.ajaxSettings.flatOptions||{};for(c in b)void 0!==b[c]&&((e[c]?a:d||(d={}))[c]=b[c]);return d&&r.extend(!0,a,d),a}function Mb(a,b,c){var d,e,f,g,h=a.contents,i=a.dataTypes;while("*"===i[0])i.shift(),void 0===d&&(d=a.mimeType||b.getResponseHeader("Content-Type"));if(d)for(e in h)if(h[e]&&h[e].test(d)){i.unshift(e);break}if(i[0]in c)f=i[0];else{for(e in c){if(!i[0]||a.converters[e+" "+i[0]]){f=e;break}g||(g=e)}f=f||g}if(f)return f!==i[0]&&i.unshift(f),c[f]}function Nb(a,b,c,d){var e,f,g,h,i,j={},k=a.dataTypes.slice();if(k[1])for(g in a.converters)j[g.toLowerCase()]=a.converters[g];f=k.shift();while(f)if(a.responseFields[f]&&(c[a.responseFields[f]]=b),!i&&d&&a.dataFilter&&(b=a.dataFilter(b,a.dataType)),i=f,f=k.shift())if("*"===f)f=i;else if("*"!==i&&i!==f){if(g=j[i+" "+f]||j["* "+f],!g)for(e in j)if(h=e.split(" "),h[1]===f&&(g=j[i+" "+h[0]]||j["* "+h[0]])){g===!0?g=j[e]:j[e]!==!0&&(f=h[0],k.unshift(h[1]));break}if(g!==!0)if(g&&a["throws"])b=g(b);else try{b=g(b)}catch(l){return{state:"parsererror",error:g?l:"No conversion from "+i+" to "+f}}}return{state:"success",data:b}}r.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:qb.href,type:"GET",isLocal:Cb.test(qb.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":Hb,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":r.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(a,b){return b?Lb(Lb(a,r.ajaxSettings),b):Lb(r.ajaxSettings,a)},ajaxPrefilter:Jb(Fb),ajaxTransport:Jb(Gb),ajax:function(b,c){"object"==typeof b&&(c=b,b=void 0),c=c||{};var e,f,g,h,i,j,k,l,m,n,o=r.ajaxSetup({},c),p=o.context||o,q=o.context&&(p.nodeType||p.jquery)?r(p):r.event,s=r.Deferred(),t=r.Callbacks("once memory"),u=o.statusCode||{},v={},w={},x="canceled",y={readyState:0,getResponseHeader:function(a){var b;if(k){if(!h){h={};while(b=Bb.exec(g))h[b[1].toLowerCase()]=b[2]}b=h[a.toLowerCase()]}return null==b?null:b},getAllResponseHeaders:function(){return k?g:null},setRequestHeader:function(a,b){return null==k&&(a=w[a.toLowerCase()]=w[a.toLowerCase()]||a,v[a]=b),this},overrideMimeType:function(a){return null==k&&(o.mimeType=a),this},statusCode:function(a){var b;if(a)if(k)y.always(a[y.status]);else for(b in a)u[b]=[u[b],a[b]];return this},abort:function(a){var b=a||x;return e&&e.abort(b),A(0,b),this}};if(s.promise(y),o.url=((b||o.url||qb.href)+"").replace(Eb,qb.protocol+"//"),o.type=c.method||c.type||o.method||o.type,o.dataTypes=(o.dataType||"*").toLowerCase().match(K)||[""],null==o.crossDomain){j=d.createElement("a");try{j.href=o.url,j.href=j.href,o.crossDomain=Ib.protocol+"//"+Ib.host!=j.protocol+"//"+j.host}catch(z){o.crossDomain=!0}}if(o.data&&o.processData&&"string"!=typeof o.data&&(o.data=r.param(o.data,o.traditional)),Kb(Fb,o,c,y),k)return y;l=r.event&&o.global,l&&0===r.active++&&r.event.trigger("ajaxStart"),o.type=o.type.toUpperCase(),o.hasContent=!Db.test(o.type),f=o.url.replace(zb,""),o.hasContent?o.data&&o.processData&&0===(o.contentType||"").indexOf("application/x-www-form-urlencoded")&&(o.data=o.data.replace(yb,"+")):(n=o.url.slice(f.length),o.data&&(f+=(sb.test(f)?"&":"?")+o.data,delete o.data),o.cache===!1&&(f=f.replace(Ab,"$1"),n=(sb.test(f)?"&":"?")+"_="+rb++ +n),o.url=f+n),o.ifModified&&(r.lastModified[f]&&y.setRequestHeader("If-Modified-Since",r.lastModified[f]),r.etag[f]&&y.setRequestHeader("If-None-Match",r.etag[f])),(o.data&&o.hasContent&&o.contentType!==!1||c.contentType)&&y.setRequestHeader("Content-Type",o.contentType),y.setRequestHeader("Accept",o.dataTypes[0]&&o.accepts[o.dataTypes[0]]?o.accepts[o.dataTypes[0]]+("*"!==o.dataTypes[0]?", "+Hb+"; q=0.01":""):o.accepts["*"]);for(m in o.headers)y.setRequestHeader(m,o.headers[m]);if(o.beforeSend&&(o.beforeSend.call(p,y,o)===!1||k))return y.abort();if(x="abort",t.add(o.complete),y.done(o.success),y.fail(o.error),e=Kb(Gb,o,c,y)){if(y.readyState=1,l&&q.trigger("ajaxSend",[y,o]),k)return y;o.async&&o.timeout>0&&(i=a.setTimeout(function(){y.abort("timeout")},o.timeout));try{k=!1,e.send(v,A)}catch(z){if(k)throw z;A(-1,z)}}else A(-1,"No Transport");function A(b,c,d,h){var j,m,n,v,w,x=c;k||(k=!0,i&&a.clearTimeout(i),e=void 0,g=h||"",y.readyState=b>0?4:0,j=b>=200&&b<300||304===b,d&&(v=Mb(o,y,d)),v=Nb(o,v,y,j),j?(o.ifModified&&(w=y.getResponseHeader("Last-Modified"),w&&(r.lastModified[f]=w),w=y.getResponseHeader("etag"),w&&(r.etag[f]=w)),204===b||"HEAD"===o.type?x="nocontent":304===b?x="notmodified":(x=v.state,m=v.data,n=v.error,j=!n)):(n=x,!b&&x||(x="error",b<0&&(b=0))),y.status=b,y.statusText=(c||x)+"",j?s.resolveWith(p,[m,x,y]):s.rejectWith(p,[y,x,n]),y.statusCode(u),u=void 0,l&&q.trigger(j?"ajaxSuccess":"ajaxError",[y,o,j?m:n]),t.fireWith(p,[y,x]),l&&(q.trigger("ajaxComplete",[y,o]),--r.active||r.event.trigger("ajaxStop")))}return y},getJSON:function(a,b,c){return r.get(a,b,c,"json")},getScript:function(a,b){return r.get(a,void 0,b,"script")}}),r.each(["get","post"],function(a,b){r[b]=function(a,c,d,e){return r.isFunction(c)&&(e=e||d,d=c,c=void 0),r.ajax(r.extend({url:a,type:b,dataType:e,data:c,success:d},r.isPlainObject(a)&&a))}}),r._evalUrl=function(a){return r.ajax({url:a,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},r.fn.extend({wrapAll:function(a){var b;return this[0]&&(r.isFunction(a)&&(a=a.call(this[0])),b=r(a,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstElementChild)a=a.firstElementChild;return a}).append(this)),this},wrapInner:function(a){return r.isFunction(a)?this.each(function(b){r(this).wrapInner(a.call(this,b))}):this.each(function(){var b=r(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=r.isFunction(a);return this.each(function(c){r(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(a){return this.parent(a).not("body").each(function(){r(this).replaceWith(this.childNodes)}),this}}),r.expr.pseudos.hidden=function(a){return!r.expr.pseudos.visible(a)},r.expr.pseudos.visible=function(a){return!!(a.offsetWidth||a.offsetHeight||a.getClientRects().length)},r.ajaxSettings.xhr=function(){try{return new a.XMLHttpRequest}catch(b){}};var Ob={0:200,1223:204},Pb=r.ajaxSettings.xhr();o.cors=!!Pb&&"withCredentials"in Pb,o.ajax=Pb=!!Pb,r.ajaxTransport(function(b){var c,d;if(o.cors||Pb&&!b.crossDomain)return{send:function(e,f){var g,h=b.xhr();if(h.open(b.type,b.url,b.async,b.username,b.password),b.xhrFields)for(g in b.xhrFields)h[g]=b.xhrFields[g];b.mimeType&&h.overrideMimeType&&h.overrideMimeType(b.mimeType),b.crossDomain||e["X-Requested-With"]||(e["X-Requested-With"]="XMLHttpRequest");for(g in e)h.setRequestHeader(g,e[g]);c=function(a){return function(){c&&(c=d=h.onload=h.onerror=h.onabort=h.onreadystatechange=null,"abort"===a?h.abort():"error"===a?"number"!=typeof h.status?f(0,"error"):f(h.status,h.statusText):f(Ob[h.status]||h.status,h.statusText,"text"!==(h.responseType||"text")||"string"!=typeof h.responseText?{binary:h.response}:{text:h.responseText},h.getAllResponseHeaders()))}},h.onload=c(),d=h.onerror=c("error"),void 0!==h.onabort?h.onabort=d:h.onreadystatechange=function(){4===h.readyState&&a.setTimeout(function(){c&&d()})},c=c("abort");try{h.send(b.hasContent&&b.data||null)}catch(i){if(c)throw i}},abort:function(){c&&c()}}}),r.ajaxPrefilter(function(a){a.crossDomain&&(a.contents.script=!1)}),r.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(a){return r.globalEval(a),a}}}),r.ajaxPrefilter("script",function(a){void 0===a.cache&&(a.cache=!1),a.crossDomain&&(a.type="GET")}),r.ajaxTransport("script",function(a){if(a.crossDomain){var b,c;return{send:function(e,f){b=r(" 11 | 12 | 13 | 14 | 15 | 16 |
17 |
18 |
19 |
20 |

相似建筑搜索

21 |
22 |
23 |
24 |
25 | 26 |
27 |
28 | 29 | 30 |
31 |
32 |
33 | 34 | 35 | 36 | 37 |
38 |
39 |
40 |
41 |
42 | 43 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | import train 2 | 3 | data = 'arch' 4 | saveto = 'vse/%s' %data 5 | dim_image = 2048 if data == 'arch' or data == 'arch_small' else 4096 6 | 7 | 8 | train.trainer(data=data, dim_image=dim_image, lrate=0.01, encoder='bow', max_epochs=100000, 9 | dim=300, maxlen_w=150, dispFreq=10, validFreq=10000, concat=False, saveto=saveto) -------------------------------------------------------------------------------- /tools.py: -------------------------------------------------------------------------------- 1 | 2 | # coding: utf-8 3 | 4 | import numpy 5 | from collections import defaultdict 6 | import torch 7 | from torch.autograd import Variable 8 | 9 | def encode_sentences(model, X, verbose=False, batch_size=128): 10 | """ 11 | Encode sentences into the joint embedding space 12 | """ 13 | # features = numpy.zeros((len(X), model['options']['dim']), dtype='float32') 14 | features = Variable(torch.zeros(len(X), model['options']['dim']).cuda()) 15 | 16 | # length dictionary 17 | ds = defaultdict(list) 18 | captions = [s.split() for s in X] 19 | for i,s in enumerate(captions): 20 | ds[len(s)].append(i) 21 | 22 | 23 | # quick check if a word is in the dictionary 24 | d = defaultdict(lambda : 0) 25 | for w in model['worddict'].keys(): 26 | d[w] = 1 27 | 28 | # Get features. This encodes by length, in order to avoid wasting computation 29 | for k in ds.keys(): 30 | if verbose: 31 | print k 32 | numbatches = len(ds[k]) / batch_size + 1 33 | for minibatch in range(numbatches): 34 | caps = ds[k][minibatch::numbatches] 35 | caption = [captions[c] for c in caps] 36 | 37 | seqs = [] 38 | for i, cc in enumerate(caption): 39 | seqs.append([model['worddict'][w] if d[w] > 0 and model['worddict'][w] < model['options']['n_words'] else 1 for w in cc]) 40 | x = numpy.zeros((k+1, len(caption))).astype('int64') 41 | for idx, s in enumerate(seqs): 42 | x[:k,idx] = s 43 | 44 | x = Variable(torch.from_numpy(x).cuda()) 45 | ff = model['img_sen_model'].forward_sens(x, caption) 46 | 47 | for ind, c in enumerate(caps): 48 | # features[c] = ff[ind].data.cpu().numpy() 49 | features[c] = ff[ind] 50 | 51 | return features 52 | 53 | def encode_images(model, IM): 54 | """ 55 | Encode images into the joint embedding space 56 | """ 57 | IM = Variable(torch.from_numpy(IM).cuda()) 58 | images = model['img_sen_model'].forward_imgs(IM) 59 | # images = images.data.cpu().numpy() 60 | return images 61 | 62 | -------------------------------------------------------------------------------- /train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import torch 4 | import os 5 | import cPickle as pkl 6 | from datasets import load_dataset 7 | from vocab import build_dictionary 8 | import homogeneous_data 9 | from torch.autograd import Variable 10 | import time 11 | from model import ImgSenRanking, PairwiseRankingLoss 12 | import numpy 13 | from tools import encode_sentences, encode_images 14 | from evaluation import i2t, t2i, i2t_arch, t2i_arch 15 | 16 | from hyperboard import Agent 17 | 18 | cur_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time())) 19 | 20 | # @profile 21 | def trainer(data='coco', 22 | margin=0.2, 23 | dim=1024, 24 | dim_image=4096, 25 | dim_word=300, 26 | encoder='gru', 27 | max_epochs=15, 28 | dispFreq=10, 29 | decay_c=0.0, 30 | grad_clip=2.0, 31 | maxlen_w=150, 32 | batch_size=128, 33 | saveto='vse/coco', 34 | validFreq=100, 35 | lrate=0.0002, 36 | concat=True, 37 | reload_=False): 38 | 39 | 40 | hyper_params = { 41 | 'data': data, 42 | 'encoder': encoder, 43 | 'batch_size': batch_size, 44 | 'time': cur_time, 45 | 'lrate': lrate, 46 | 'concat': concat, 47 | } 48 | 49 | i2t_r1 = dict([('i2t_recall', 'r1')]+hyper_params.items()) 50 | i2t_r5 = dict([('i2t_recall', 'r5')]+hyper_params.items()) 51 | i2t_r10 = dict([('i2t_recall', 'r10')]+hyper_params.items()) 52 | t2i_r1 = dict([('t2i_recall', 'r1')]+hyper_params.items()) 53 | t2i_r5 = dict([('t2i_recall', 'r5')]+hyper_params.items()) 54 | t2i_r10 = dict([('t2i_recall', 'r10')]+hyper_params.items()) 55 | 56 | i2t_med = dict([('i2t_med', 'i2t_med')]+hyper_params.items()) 57 | t2i_med = dict([('t2i_med', 't2i_med')]+hyper_params.items()) 58 | 59 | agent = Agent(port=5020) 60 | i2t_r1_agent = agent.register(i2t_r1, 'recall', overwrite=True) 61 | i2t_r5_agent = agent.register(i2t_r5, 'recall', overwrite=True) 62 | i2t_r10_agent = agent.register(i2t_r10, 'recall', overwrite=True) 63 | t2i_r1_agent = agent.register(t2i_r1, 'recall', overwrite=True) 64 | t2i_r5_agent = agent.register(t2i_r5, 'recall', overwrite=True) 65 | t2i_r10_agent = agent.register(t2i_r10, 'recall', overwrite=True) 66 | 67 | i2t_med_agent = agent.register(i2t_med, 'median', overwrite=True) 68 | t2i_med_agent = agent.register(t2i_med, 'median', overwrite=True) 69 | 70 | 71 | # Model options 72 | model_options = {} 73 | model_options['data'] = data 74 | model_options['margin'] = margin 75 | model_options['dim'] = dim 76 | model_options['dim_image'] = dim_image 77 | model_options['dim_word'] = dim_word 78 | model_options['encoder'] = encoder 79 | model_options['max_epochs'] = max_epochs 80 | model_options['dispFreq'] = dispFreq 81 | model_options['decay_c'] = decay_c 82 | model_options['grad_clip'] = grad_clip 83 | model_options['maxlen_w'] = maxlen_w 84 | model_options['batch_size'] = batch_size 85 | model_options['saveto'] = saveto 86 | model_options['validFreq'] = validFreq 87 | model_options['lrate'] = lrate 88 | model_options['reload_'] = reload_ 89 | model_options['concat'] = concat 90 | 91 | print model_options 92 | 93 | # reload options 94 | if reload_ and os.path.exists(saveto): 95 | print 'reloading...' + saveto 96 | with open('%s.pkl'%saveto, 'rb') as f: 97 | model_options = pkl.load(f) 98 | 99 | # Load training and development sets 100 | print 'loading dataset' 101 | train, dev = load_dataset(data)[:2] 102 | 103 | # Create and save dictionary 104 | print 'Create dictionary' 105 | worddict = build_dictionary(train[0]+dev[0])[0] 106 | n_words = len(worddict) 107 | model_options['n_words'] = n_words 108 | print 'Dictionary size: ' + str(n_words) 109 | with open('%s.dictionary.pkl'%saveto, 'wb') as f: 110 | pkl.dump(worddict, f) 111 | 112 | 113 | # Inverse dictionary 114 | word_idict = dict() 115 | for kk, vv in worddict.iteritems(): 116 | word_idict[vv] = kk 117 | word_idict[0] = '' 118 | word_idict[1] = 'UNK' 119 | 120 | model_options['worddict'] = worddict 121 | model_options['word_idict'] = word_idict 122 | 123 | # Each sentence in the minibatch have same length (for encoder) 124 | train_iter = homogeneous_data.HomogeneousData([train[0], train[1]], batch_size=batch_size, maxlen=maxlen_w) 125 | 126 | img_sen_model = ImgSenRanking(model_options) 127 | img_sen_model = img_sen_model.cuda() 128 | 129 | loss_fn = PairwiseRankingLoss(margin=margin) 130 | loss_fn = loss_fn.cuda() 131 | 132 | params = filter(lambda p: p.requires_grad, img_sen_model.parameters()) 133 | optimizer = torch.optim.Adam(params, lrate) 134 | 135 | uidx = 0 136 | curr = 0.0 137 | n_samples = 0 138 | 139 | for eidx in xrange(max_epochs): 140 | 141 | print 'Epoch ', eidx 142 | 143 | for x, im in train_iter: 144 | n_samples += len(x) 145 | uidx += 1 146 | 147 | x_id, im = homogeneous_data.prepare_data(x, im, worddict, maxlen=maxlen_w, n_words=n_words) 148 | 149 | if x_id is None: 150 | print 'Minibatch with zero sample under length ', maxlen_w 151 | uidx -= 1 152 | continue 153 | 154 | x_id = Variable(torch.from_numpy(x_id).cuda()) 155 | im = Variable(torch.from_numpy(im).cuda()) 156 | # Update 157 | ud_start = time.time() 158 | x, im = img_sen_model(x_id, im, x) 159 | cost = loss_fn(im, x) 160 | optimizer.zero_grad() 161 | cost.backward() 162 | torch.nn.utils.clip_grad_norm(params, grad_clip) 163 | optimizer.step() 164 | ud = time.time() - ud_start 165 | 166 | if numpy.mod(uidx, dispFreq) == 0: 167 | print 'Epoch ', eidx, 'Update ', uidx, 'Cost ', cost.data.cpu().numpy()[0], 'UD ', ud 168 | 169 | if numpy.mod(uidx, validFreq) == 0: 170 | 171 | print 'Computing results...' 172 | curr_model = {} 173 | curr_model['options'] = model_options 174 | curr_model['worddict'] = worddict 175 | curr_model['word_idict'] = word_idict 176 | curr_model['img_sen_model'] = img_sen_model 177 | 178 | ls, lim = encode_sentences(curr_model, dev[0]), encode_images(curr_model, dev[1]) 179 | 180 | r1, r5, r10, medr = 0.0, 0.0, 0.0, 0 181 | r1i, r5i, r10i, medri = 0.0, 0.0, 0.0, 0 182 | r_time = time.time() 183 | if data == 'arch' or data == 'arch_small': 184 | (r1, r5, r10, medr) = i2t_arch(lim, ls) 185 | print "Image to text: %.1f, %.1f, %.1f, %.1f" % (r1, r5, r10, medr) 186 | (r1i, r5i, r10i, medri) = t2i_arch(lim, ls) 187 | print "Text to image: %.1f, %.1f, %.1f, %.1f" % (r1i, r5i, r10i, medri) 188 | else: 189 | (r1, r5, r10, medr) = i2t(lim, ls) 190 | print "Image to text: %.1f, %.1f, %.1f, %.1f" % (r1, r5, r10, medr) 191 | (r1i, r5i, r10i, medri) = t2i(lim, ls) 192 | print "Text to image: %.1f, %.1f, %.1f, %.1f" % (r1i, r5i, r10i, medri) 193 | 194 | print "Cal Recall@K using %ss" %(time.time()-r_time) 195 | 196 | record_num = uidx / validFreq 197 | agent.append(i2t_r1_agent, record_num, r1) 198 | agent.append(i2t_r5_agent, record_num, r5) 199 | agent.append(i2t_r10_agent, record_num, r10) 200 | agent.append(t2i_r1_agent, record_num, r1i) 201 | agent.append(t2i_r5_agent, record_num, r5i) 202 | agent.append(t2i_r10_agent, record_num, r10i) 203 | 204 | agent.append(i2t_med_agent, record_num, medr) 205 | agent.append(t2i_med_agent, record_num, medri) 206 | 207 | currscore = r1 + r5 + r10 + r1i + r5i + r10i 208 | if currscore > curr: 209 | curr = currscore 210 | 211 | # Save model 212 | print 'Saving model...', 213 | pkl.dump(model_options, open('%s_params_%s.pkl'%(saveto, encoder), 'wb')) 214 | torch.save(img_sen_model.state_dict(), '%s_model_%s.pkl'%(saveto, encoder)) 215 | print 'Done' 216 | 217 | print 'Seen %d samples'%n_samples 218 | 219 | if __name__ == '__main__': 220 | pass 221 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import numpy 3 | from torch.autograd import Variable 4 | 5 | 6 | def l2norm(input, p=2.0, dim=1, eps=1e-12): 7 | """ 8 | Compute L2 norm, row-wise 9 | """ 10 | return input / input.norm(p, dim).clamp(min=eps).expand_as(input) 11 | 12 | 13 | def xavier_weight(tensor): 14 | if isinstance(tensor, Variable): 15 | xavier_weight(tensor.data) 16 | return tensor 17 | 18 | nin, nout = tensor.size()[0], tensor.size()[1] 19 | r = numpy.sqrt(6.) / numpy.sqrt(nin + nout) 20 | return tensor.normal_(0, r) 21 | 22 | -------------------------------------------------------------------------------- /vocab.py: -------------------------------------------------------------------------------- 1 | """ 2 | Constructing and loading dictionaries 3 | """ 4 | import cPickle as pkl 5 | import numpy 6 | from collections import OrderedDict 7 | 8 | def build_dictionary(text): 9 | """ 10 | Build a dictionary 11 | text: list of sentences (pre-tokenized) 12 | """ 13 | wordcount = OrderedDict() 14 | for cc in text: 15 | words = cc.split() 16 | for w in words: 17 | if w not in wordcount: 18 | wordcount[w] = 0 19 | wordcount[w] += 1 20 | words = wordcount.keys() 21 | freqs = wordcount.values() 22 | sorted_idx = numpy.argsort(freqs)[::-1] 23 | 24 | worddict = OrderedDict() 25 | for idx, sidx in enumerate(sorted_idx): 26 | worddict[words[sidx]] = idx + 2 # 0: , 1: 27 | 28 | return worddict, wordcount --------------------------------------------------------------------------------