├── AFM ├── master.sh ├── data_generator.py ├── train_AFM_model.py ├── preprocess.py ├── AFM_Model.py ├── README.md ├── data.py └── AFMLayer.py ├── BST ├── bst_model.png ├── din.py ├── bst_model.py ├── README.md └── transformer.py ├── BilinearFFM ├── master.sh ├── data_generator.py ├── train_BilinearFFM_model.py ├── preprocess.py ├── BilinearFFM.py ├── BilinearInteraction.py ├── README.md └── data.py ├── MIND ├── master.sh ├── mind_model.png ├── train_mind.py ├── predict.py ├── preprocess.py ├── data_generator.py ├── data.py ├── mind.py ├── CapsuleLayer.py └── README.md ├── Word2vec ├── word2vec ├── README.md └── train_w2v_model.sh ├── MMoE └── README.md ├── YouTubeNet ├── YouTubeNet_model.png ├── master.sh ├── train_YouTubeNet_model.py ├── preprocess.py ├── data_generator.py ├── SequencePoolingLayer.py ├── load_YouTubeNet_model_to_predict.py ├── data.py ├── YouTubeNet.py └── README.md ├── Wide&Deep ├── wide_and_deep_multi_model.png ├── wide_and_deep_binary_model.png ├── wide_and_deep.py └── README.md ├── ESMM ├── master.sh ├── run_train_esmm_model.sh ├── split_train_val.sh ├── split_train_val.py ├── tar_model.py ├── write_tfrecord.py ├── README.md └── train_esmm_finetune.py ├── vgg16_figure_search_annoy ├── threadings_download_txt.py ├── README.md ├── extract_figure_feature.py ├── build_figure_ann.py ├── search_topN_figure.py └── download_jd_figures.py ├── README.md ├── DeepFM └── README.md └── NFM └── README.md /AFM/master.sh: -------------------------------------------------------------------------------- 1 | python data.py 2 | 3 | python train_AFM_model.py 4 | -------------------------------------------------------------------------------- /BST/bst_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/BST/bst_model.png -------------------------------------------------------------------------------- /BilinearFFM/master.sh: -------------------------------------------------------------------------------- 1 | python data.py 2 | 3 | python train_BilinearFFM_model.py 4 | -------------------------------------------------------------------------------- /MIND/master.sh: -------------------------------------------------------------------------------- 1 | python data.py 2 | 3 | python train_mind.py 4 | 5 | python predict.py -------------------------------------------------------------------------------- /MIND/mind_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/MIND/mind_model.png -------------------------------------------------------------------------------- /Word2vec/word2vec: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/Word2vec/word2vec -------------------------------------------------------------------------------- /MMoE/README.md: -------------------------------------------------------------------------------- 1 | # 参考 2 | 3 | [https://github.com/drawbridge/keras-mmoe](https://github.com/drawbridge/keras-mmoe) -------------------------------------------------------------------------------- /YouTubeNet/YouTubeNet_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/YouTubeNet/YouTubeNet_model.png -------------------------------------------------------------------------------- /Wide&Deep/wide_and_deep_multi_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/Wide&Deep/wide_and_deep_multi_model.png -------------------------------------------------------------------------------- /Wide&Deep/wide_and_deep_binary_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wziji/deep_ctr/HEAD/Wide&Deep/wide_and_deep_binary_model.png -------------------------------------------------------------------------------- /ESMM/master.sh: -------------------------------------------------------------------------------- 1 | today=`date '+%Y-%m-%d %H:%M:%S'` 2 | echo $today 3 | 4 | 5 | 6 | sh split_train_val.sh 7 | 8 | python write_tfrecord.py 9 | 10 | sh run_train_esmm_model.sh 11 | -------------------------------------------------------------------------------- /Word2vec/README.md: -------------------------------------------------------------------------------- 1 | # 添加权限 2 | 3 | ```shell 4 | chmod +x train_w2v_model.sh 5 | 6 | chmod +x word2vec 7 | 8 | ``` 9 | 10 | # 运行 11 | 12 | ```shell 13 | 14 | sh train_w2v_model.sh 15 | ``` 16 | 17 | -------------------------------------------------------------------------------- /YouTubeNet/master.sh: -------------------------------------------------------------------------------- 1 | 2 | # 1. data preprocession 3 | python data.py 4 | 5 | 6 | # 2. train mdoel 7 | python train_YouTubeNet_model.py 8 | 9 | 10 | # 3. load model to predict 11 | python load_YouTubeNet_model_to_predict.py 12 | -------------------------------------------------------------------------------- /ESMM/run_train_esmm_model.sh: -------------------------------------------------------------------------------- 1 | train_data_path=data/train_data.tfrecord 2 | val_data_path=data/val_data.tfrecord 3 | model_path=__model/esmm_finetune.model 4 | train_val_summary_path=data/train_val_summary 5 | 6 | python train_esmm_finetune.py $train_data_path $val_data_path $model_path $train_val_summary_path 7 | 8 | 9 | # tar model 10 | model_dir=__model 11 | python tar_model.py $model_dir 12 | -------------------------------------------------------------------------------- /ESMM/split_train_val.sh: -------------------------------------------------------------------------------- 1 | raw_sample_data_path=data/esmm_raw_sample_data 2 | sample_data_path=data/esmm_sample_data 3 | 4 | # shuffle 5 | echo `date` 6 | echo 'Start to shuffle sample data ...' 7 | shuf $raw_sample_data_path > $sample_data_path 8 | echo `date` 9 | echo 'Shuffle Done!' 10 | 11 | 12 | 13 | # split train and val 14 | train_path=data/train_data 15 | val_path=data/val_data 16 | summary_path=data/train_val_summary 17 | train_percent=0.75 # percent in (0, 1) 18 | 19 | echo 20 | echo `date` 21 | echo 'Start to split train and validate data ...' 22 | python split_train_val.py $sample_data_path $train_path $val_path $summary_path $train_percent 23 | echo `date` 24 | echo 'Split train and validate data Done!' 25 | -------------------------------------------------------------------------------- /Word2vec/train_w2v_model.sh: -------------------------------------------------------------------------------- 1 | # trainData: skus, split by space 2 | 3 | trainData=data/w2v_order_seq 4 | outputData=data/w2v_order_item_vec_128dim 5 | 6 | 7 | 8 | size=128 9 | window=5 10 | iter=20 11 | classes=0 12 | binary=0 13 | threads=22 14 | min_count=8 15 | negative=25 16 | 17 | 18 | if [ -s $trainData ]; then 19 | 20 | # train model 21 | ./word2vec -train $trainData -output $outputData -size $size -window $window -sample 1e-4 -negative $negative -hs 0 -binary $binary -cbow 0 -iter $iter -threads $threads -classes $classes -min_count $min_count 22 | 23 | # Delete the top 2 lines 24 | sed -i 1,2d $outputData 25 | sed -i "s/ /\t/g" $outputData 26 | fi 27 | 28 | 29 | # cbow (-cbow 1) 30 | # skip (-cbow 0) -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/threadings_download_txt.py: -------------------------------------------------------------------------------- 1 | # http://www.uml.org.cn/python/201901221.asp 2 | 3 | 4 | import threading 5 | import urllib.request 6 | import time 7 | 8 | 9 | def download_image(url, filename): 10 | print("download txt from {}".format(url)) 11 | 12 | urllib.request.urlretrieve(url, filename) 13 | print("download done!") 14 | 15 | 16 | def execute_thread(i): 17 | textname = "temp/jiari-{}.txt".format(i) 18 | 19 | download_image("http://tool.bitefu.net/jiari/data/2017.txt", textname) 20 | 21 | 22 | def main(): 23 | threads = [] 24 | 25 | for i in range(10): 26 | thread = threading.Thread(target=execute_thread, args=(i,)) 27 | threads.append(thread) 28 | thread.start() 29 | 30 | for i in threads: 31 | i.join() 32 | 33 | 34 | main() -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/README.md: -------------------------------------------------------------------------------- 1 | # vgg16 model extract figure feature, Annoy to search similar figures 2 | 3 | ### 推荐阅读：[Annoy最近邻检索技术之 “图片检索”](https://zhuanlan.zhihu.com/p/148819536) 4 | 5 | 6 | ### 1. 创建一个文件夹用于保存图片 7 | 8 | > mkdir figures 9 | 10 | 11 | ### 2. 下载数据 12 | 13 | > python download_jd_figure.py 14 | 15 | 16 | ### 3. 提取图片特征 17 | 18 | [vgg16 官方下载](https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5) 19 | 20 | [百度网盘中下载](https://pan.baidu.com/s/1Exa8g_q9hVmqOU9SBrIxrg) ，提取码：qtsb 21 | 22 | 将 vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5 模型放入 ~/.keras/models 路径中即可。 23 | 24 | > python extract_figure_feature.py 25 | 26 | 27 | ### 4. 基于图片向量特征，构建ANN树 28 | > python build_figure_ann.py 29 | 30 | 31 | ### 5. 搜索topN相似图片 32 | > python search_topN_figure.py 33 | -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/extract_figure_feature.py: -------------------------------------------------------------------------------- 1 | from tensorflow.keras.applications.vgg16 import VGG16 2 | from tensorflow.keras.preprocessing import image 3 | from tensorflow.keras.applications.vgg16 import preprocess_input 4 | import numpy as np 5 | from tqdm import tqdm 6 | import pickle 7 | import os 8 | 9 | # 1. 加载vgg16模型 10 | model = VGG16(weights='imagenet', include_top=False) 11 | #print(model.summary()) 12 | 13 | 14 | # 2. 提取图片特征 15 | img_path = "figures/" 16 | 17 | img_name_list = [] 18 | img_feature_list = [] 19 | 20 | for file in tqdm(os.listdir(img_path)): 21 | img_name_list.append(file) 22 | file_path = img_path + file 23 | 24 | img = image.load_img(file_path, target_size=(224, 224)) 25 | x = image.img_to_array(img) 26 | x = np.expand_dims(x, axis=0) 27 | x = preprocess_input(x) 28 | 29 | features = model.predict(x) 30 | img_feature_list.append(features.reshape((7*7*512,))) 31 | 32 | 33 | # 3. 将图片名称和图片特征保存为pkl格式 34 | f = open("img_feature_list.pkl", 'wb') 35 | pickle.dump(img_feature_list, f) 36 | f.close() 37 | 38 | g = open("img_name_list.pkl", 'wb') 39 | pickle.dump(img_name_list, g) 40 | g.close() -------------------------------------------------------------------------------- /ESMM/split_train_val.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import numpy as np 3 | 4 | 5 | def split_train_val(sample_path, train_path, val_path, summary_path, train_percent = None): 6 | 7 | fr = open(sample_path, 'r') 8 | 9 | fw_train = open(train_path, 'w') 10 | fw_val = open(val_path, 'w') 11 | 12 | fw_summary = open(summary_path, 'w') 13 | 14 | n_train = 0 15 | n_val = 0 16 | for line in fr: 17 | r = np.random.random() 18 | if r < train_percent: 19 | fw_train.write(line) 20 | n_train += 1 21 | else: 22 | fw_val.write(line) 23 | n_val +=1 24 | 25 | summary = 'n_train=' + str(n_train) + ',' + 'n_val=' + str(n_val) 26 | fw_summary.write(summary + '\n') 27 | 28 | 29 | fr.close() 30 | fw_train.close() 31 | fw_val.close() 32 | fw_summary.close() 33 | 34 | print('Split train and val done!') 35 | print('Write summary done!') 36 | 37 | 38 | 39 | if __name__ == '__main__': 40 | sample_path = sys.argv[1] 41 | 42 | train_path = sys.argv[2] 43 | val_path = sys.argv[3] 44 | summary_path = sys.argv[4] 45 | train_percent = float(sys.argv[5]) 46 | 47 | split_train_val(sample_path, train_path, val_path, summary_path, train_percent = train_percent) 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /ESMM/tar_model.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import datetime 3 | import tarfile 4 | from hashlib import md5 5 | import time 6 | import os 7 | 8 | 9 | def tar(input_paths, output_path): 10 | t = tarfile.open(output_path + ".tar.gz", "w:gz") 11 | 12 | 13 | for input_path in input_paths: 14 | t.add(input_path, arcname = input_path.split('/')[-1]) 15 | t.close() 16 | 17 | 18 | def md5_file(name): 19 | m = md5() 20 | a_file = open(name, 'rb') 21 | m.update(a_file.read()) 22 | a_file.close() 23 | return m.hexdigest() 24 | 25 | 26 | if __name__ == '__main__': 27 | model_dir = sys.argv[1] 28 | 29 | input_paths = [] 30 | 31 | for model_path in os.listdir(model_dir): 32 | if model_path.endswith('.tar.gz') or model_path.endswith('.md5'): 33 | continue 34 | input_paths.append(model_dir + '/' + model_path) 35 | 36 | 37 | now_time = datetime.datetime.now().strftime('%Y%m%d%H%m') 38 | 39 | tar_model_path = './__model/' + now_time 40 | md5_path = tar_model_path + '.md5' 41 | 42 | # tar 43 | print("Staring to tar %s" % input_paths) 44 | start = time.time() 45 | tar(input_paths, tar_model_path) 46 | print("Tar to %s done! Lasts %.2fs" % (tar_model_path, time.time() - start)) 47 | 48 | # generate md5 49 | start = time.time() 50 | md5_str = md5_file(tar_model_path + '.tar.gz') 51 | with open(md5_path, 'w', encoding='utf-8') as f: 52 | f.write(md5_str) 53 | print("Write md5 to %s done! Lasts %.2fs" % (md5_path, time.time() - start)) 54 | 55 | -------------------------------------------------------------------------------- /AFM/data_generator.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import numpy as np 4 | 5 | 6 | def init_output(): 7 | user_id = [] 8 | gender = [] 9 | age = [] 10 | occupation = [] 11 | zip = [] 12 | movie_id = [] 13 | label = [] 14 | 15 | 16 | return user_id, gender, age, occupation, zip, movie_id, label 17 | 18 | 19 | def file_generator(input_path, batch_size): 20 | 21 | user_id, gender, age, occupation, zip, movie_id, label = init_output() 22 | 23 | cnt = 0 24 | 25 | num_lines = sum([1 for line in open(input_path)]) 26 | 27 | while True: 28 | 29 | with open(input_path, 'r') as f: 30 | for line in f.readlines(): 31 | 32 | buf = line.strip().split('\t') 33 | 34 | user_id.append(int(buf[0])) 35 | gender.append(int(buf[1])) 36 | age.append(int(buf[2])) 37 | occupation.append(int(buf[3])) 38 | zip.append(int(buf[4])) 39 | movie_id.append(int(buf[5])) 40 | label.append(int(buf[6])) 41 | 42 | cnt += 1 43 | 44 | if cnt % batch_size == 0 or cnt == num_lines: 45 | user_id = np.array(user_id, dtype='int32') 46 | gender = np.array(gender, dtype='int32') 47 | age = np.array(age, dtype='int32') 48 | occupation = np.array(occupation, dtype='int32') 49 | zip = np.array(zip, dtype='int32') 50 | movie_id = np.array(movie_id, dtype='int32') 51 | 52 | label = np.array(label, dtype='int32') 53 | 54 | yield [user_id, gender, age, occupation, zip, movie_id], label 55 | 56 | user_id, gender, age, occupation, zip, movie_id, label = init_output() 57 | 58 | -------------------------------------------------------------------------------- /BilinearFFM/data_generator.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import numpy as np 4 | 5 | 6 | def init_output(): 7 | user_id = [] 8 | gender = [] 9 | age = [] 10 | occupation = [] 11 | zip = [] 12 | movie_id = [] 13 | label = [] 14 | 15 | 16 | return user_id, gender, age, occupation, zip, movie_id, label 17 | 18 | 19 | def file_generator(input_path, batch_size): 20 | 21 | user_id, gender, age, occupation, zip, movie_id, label = init_output() 22 | 23 | cnt = 0 24 | 25 | num_lines = sum([1 for line in open(input_path)]) 26 | 27 | while True: 28 | 29 | with open(input_path, 'r') as f: 30 | for line in f.readlines(): 31 | 32 | buf = line.strip().split('\t') 33 | 34 | user_id.append(int(buf[0])) 35 | gender.append(int(buf[1])) 36 | age.append(int(buf[2])) 37 | occupation.append(int(buf[3])) 38 | zip.append(int(buf[4])) 39 | movie_id.append(int(buf[5])) 40 | label.append(int(buf[6])) 41 | 42 | cnt += 1 43 | 44 | if cnt % batch_size == 0 or cnt == num_lines: 45 | user_id = np.array(user_id, dtype='int32') 46 | gender = np.array(gender, dtype='int32') 47 | age = np.array(age, dtype='int32') 48 | occupation = np.array(occupation, dtype='int32') 49 | zip = np.array(zip, dtype='int32') 50 | movie_id = np.array(movie_id, dtype='int32') 51 | 52 | label = np.array(label, dtype='int32') 53 | 54 | yield [user_id, gender, age, occupation, zip, movie_id], label 55 | 56 | user_id, gender, age, occupation, zip, movie_id, label = init_output() 57 | 58 | -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/build_figure_ann.py: -------------------------------------------------------------------------------- 1 | # encoding:utf-8 2 | from annoy import AnnoyIndex 3 | import pickle 4 | import numpy as np 5 | np.random.seed(20200601) 6 | import sys, time 7 | from tqdm import tqdm 8 | 9 | def build_ann(name_path=None, \ 10 | vec_path=None, \ 11 | index_to_name_dict_path=None, \ 12 | ann_path=None, \ 13 | dim=64, \ 14 | n_trees=10): 15 | 16 | name_path = open(name_path, 'rb') 17 | vec_path = open(vec_path, 'rb') 18 | img_name_list = pickle.load(name_path) 19 | img_vec_list = pickle.load(vec_path) 20 | 21 | 22 | ann = AnnoyIndex(dim) 23 | idx = 0 24 | batch_size = 100 * 10000 25 | index_to_name_dict = {} 26 | 27 | 28 | for name, vec in tqdm(zip(img_name_list, img_vec_list)): 29 | ann.add_item(idx, vec) 30 | index_to_name_dict[idx] = name 31 | 32 | idx += 1 33 | if idx % batch_size == 0: 34 | print("%s00w" % (int(idx/batch_size))) 35 | 36 | print("Add items Done!\nStart building trees") 37 | 38 | ann.build(n_trees) 39 | print("Build Trees Done!") 40 | 41 | ann.save(ann_path) 42 | print("Save ann to %s Done!" % (ann_path)) 43 | 44 | fd = open(index_to_name_dict_path, 'wb') 45 | pickle.dump(index_to_name_dict, fd) 46 | fd.close() 47 | print("Saving index_to_name mapping Done!") 48 | 49 | 50 | if __name__ == '__main__': 51 | name_path = "img_name_list.pkl" 52 | vec_path = "img_feature_list.pkl" 53 | index_to_name_dict_path = "index_to_name_dict.pkl" 54 | ann_path = "img_feature_list.ann" 55 | dim = 25088 56 | n_trees = 10 57 | 58 | build_ann(name_path=name_path, \ 59 | vec_path=vec_path, \ 60 | index_to_name_dict_path=index_to_name_dict_path, \ 61 | ann_path=ann_path, \ 62 | dim=dim, \ 63 | n_trees=n_trees) -------------------------------------------------------------------------------- /AFM/train_AFM_model.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import tensorflow as tf 4 | from tensorflow.keras.optimizers import Adam 5 | 6 | from data_generator import file_generator 7 | from AFM_Model import AFM 8 | 9 | 10 | 11 | 12 | # 1. Load data 13 | 14 | train_path = "train.txt" 15 | val_path = "test.txt" 16 | batch_size = 1000 17 | 18 | n_train = sum([1 for i in open(train_path)]) 19 | n_val = sum([1 for i in open(val_path)]) 20 | 21 | train_steps = n_train / batch_size 22 | train_steps_ = n_train // batch_size 23 | validation_steps = n_val / batch_size 24 | validation_steps_ = n_val // batch_size 25 | 26 | 27 | train_generator = file_generator(train_path, batch_size) 28 | val_generator = file_generator(val_path, batch_size) 29 | 30 | steps_per_epoch = train_steps_ if train_steps==train_steps_ else train_steps_ + 1 31 | validation_steps = validation_steps_ if validation_steps==validation_steps_ else validation_steps_ + 1 32 | 33 | print("n_train: ", n_train) 34 | print("n_val: ", n_val) 35 | 36 | print("steps_per_epoch: ", steps_per_epoch) 37 | print("validation_steps: ", validation_steps) 38 | 39 | 40 | 41 | 42 | # 2. Train model 43 | 44 | early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) 45 | callbacks = [early_stopping_cb] 46 | 47 | 48 | model = AFM() 49 | print(model.summary()) 50 | tf.keras.utils.plot_model(model, to_file='AFM_model.png', show_shapes=True) 51 | 52 | 53 | model.compile(loss='binary_crossentropy', \ 54 | optimizer=Adam(lr=1e-3), \ 55 | metrics=['accuracy']) 56 | 57 | 58 | history = model.fit(train_generator, \ 59 | epochs=1, \ 60 | steps_per_epoch = steps_per_epoch, \ 61 | callbacks = callbacks, 62 | validation_data = val_generator, \ 63 | validation_steps = validation_steps, \ 64 | shuffle=True 65 | ) 66 | 67 | 68 | 69 | model.save_weights('AFM_model.h5') 70 | -------------------------------------------------------------------------------- /MIND/train_mind.py: -------------------------------------------------------------------------------- 1 | import os 2 | import tensorflow as tf 3 | from data_generator import file_generator 4 | from mind import mind 5 | 6 | from tensorflow.keras.optimizers import Adam 7 | 8 | 9 | 10 | # 1. Load data 11 | 12 | train_path = "train.txt" 13 | val_path = "test.txt" 14 | batch_size = 1000 15 | 16 | n_train = sum([1 for i in open(train_path)]) 17 | n_val = sum([1 for i in open(val_path)]) 18 | 19 | train_steps = n_train / batch_size 20 | train_steps_ = n_train // batch_size 21 | validation_steps = n_val / batch_size 22 | validation_steps_ = n_val // batch_size 23 | 24 | 25 | train_generator = file_generator(train_path, batch_size) 26 | val_generator = file_generator(val_path, batch_size) 27 | 28 | steps_per_epoch = train_steps_ if train_steps==train_steps_ else train_steps_ + 1 29 | validation_steps = validation_steps_ if validation_steps==validation_steps_ else validation_steps_ + 1 30 | 31 | print("n_train: ", n_train) 32 | print("n_val: ", n_val) 33 | 34 | print("steps_per_epoch: ", steps_per_epoch) 35 | print("validation_steps: ", validation_steps) 36 | 37 | 38 | 39 | # 2. Train model 40 | 41 | 42 | early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) 43 | callbacks = [early_stopping_cb] 44 | 45 | 46 | model = mind() 47 | 48 | model.compile(loss='sparse_categorical_crossentropy', \ 49 | optimizer=Adam(lr=1e-3), \ 50 | metrics=['sparse_categorical_accuracy']) 51 | 52 | # loss="sparse_categorical_accuracy"的应用方式参见：https://mp.weixin.qq.com/s/H4ET0bO_xPm8TNqltMt3Fg 53 | 54 | 55 | 56 | history = model.fit(train_generator, \ 57 | epochs=2, \ 58 | steps_per_epoch = steps_per_epoch, \ 59 | callbacks = callbacks, 60 | validation_data = val_generator, \ 61 | validation_steps = validation_steps, \ 62 | shuffle=True 63 | ) 64 | 65 | 66 | model.save_weights('mind_model.h5') 67 | -------------------------------------------------------------------------------- /BilinearFFM/train_BilinearFFM_model.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import tensorflow as tf 4 | from tensorflow.keras.optimizers import Adam 5 | 6 | from data_generator import file_generator 7 | from BilinearFFM import BilinearFFM 8 | 9 | 10 | 11 | 12 | # 1. Load data 13 | 14 | train_path = "train.txt" 15 | val_path = "test.txt" 16 | batch_size = 1000 17 | 18 | n_train = sum([1 for i in open(train_path)]) 19 | n_val = sum([1 for i in open(val_path)]) 20 | 21 | train_steps = n_train / batch_size 22 | train_steps_ = n_train // batch_size 23 | validation_steps = n_val / batch_size 24 | validation_steps_ = n_val // batch_size 25 | 26 | 27 | train_generator = file_generator(train_path, batch_size) 28 | val_generator = file_generator(val_path, batch_size) 29 | 30 | steps_per_epoch = train_steps_ if train_steps==train_steps_ else train_steps_ + 1 31 | validation_steps = validation_steps_ if validation_steps==validation_steps_ else validation_steps_ + 1 32 | 33 | print("n_train: ", n_train) 34 | print("n_val: ", n_val) 35 | 36 | print("steps_per_epoch: ", steps_per_epoch) 37 | print("validation_steps: ", validation_steps) 38 | 39 | 40 | 41 | 42 | # 2. Train model 43 | 44 | early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) 45 | callbacks = [early_stopping_cb] 46 | 47 | 48 | model = BilinearFFM() 49 | print(model.summary()) 50 | tf.keras.utils.plot_model(model, to_file='BilinearFFM_model.png', show_shapes=True) 51 | 52 | 53 | model.compile(loss='binary_crossentropy', \ 54 | optimizer=Adam(lr=1e-3), \ 55 | metrics=['accuracy']) 56 | 57 | 58 | history = model.fit(train_generator, \ 59 | epochs=2, \ 60 | steps_per_epoch = steps_per_epoch, \ 61 | callbacks = callbacks, 62 | validation_data = val_generator, \ 63 | validation_steps = validation_steps, \ 64 | shuffle=True 65 | ) 66 | 67 | 68 | 69 | model.save_weights('BilinearFFM_model.h5') 70 | -------------------------------------------------------------------------------- /YouTubeNet/train_YouTubeNet_model.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import tensorflow as tf 4 | from tensorflow.keras.optimizers import Adam 5 | 6 | from data_generator import file_generator 7 | from YouTubeNet import YouTubeNet 8 | 9 | 10 | 11 | 12 | # 1. Load data 13 | 14 | train_path = "train.txt" 15 | val_path = "test.txt" 16 | batch_size = 1000 17 | 18 | n_train = sum([1 for i in open(train_path)]) 19 | n_val = sum([1 for i in open(val_path)]) 20 | 21 | train_steps = n_train / batch_size 22 | train_steps_ = n_train // batch_size 23 | validation_steps = n_val / batch_size 24 | validation_steps_ = n_val // batch_size 25 | 26 | 27 | train_generator = file_generator(train_path, batch_size) 28 | val_generator = file_generator(val_path, batch_size) 29 | 30 | steps_per_epoch = train_steps_ if train_steps==train_steps_ else train_steps_ + 1 31 | validation_steps = validation_steps_ if validation_steps==validation_steps_ else validation_steps_ + 1 32 | 33 | print("n_train: ", n_train) 34 | print("n_val: ", n_val) 35 | 36 | print("steps_per_epoch: ", steps_per_epoch) 37 | print("validation_steps: ", validation_steps) 38 | 39 | 40 | 41 | 42 | # 2. Train model 43 | 44 | early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) 45 | callbacks = [early_stopping_cb] 46 | 47 | 48 | model = YouTubeNet() 49 | 50 | model.compile(loss='sparse_categorical_crossentropy', \ 51 | optimizer=Adam(lr=1e-3), \ 52 | metrics=['sparse_categorical_accuracy']) 53 | # loss="sparse_categorical_accuracy"的应用方式参见：https://mp.weixin.qq.com/s/H4ET0bO_xPm8TNqltMt3Fg 54 | 55 | 56 | 57 | history = model.fit(train_generator, \ 58 | epochs=2, \ 59 | steps_per_epoch = steps_per_epoch, \ 60 | callbacks = callbacks, 61 | validation_data = val_generator, \ 62 | validation_steps = validation_steps, \ 63 | shuffle=True 64 | ) 65 | 66 | 67 | 68 | 69 | 70 | model.save_weights('YouTubeNet_model.h5') 71 | -------------------------------------------------------------------------------- /MIND/predict.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | from tensorflow.keras.models import Model 4 | 5 | from mind import mind 6 | from data_generator import init_output 7 | 8 | 9 | 10 | 11 | # 1. Load model 12 | 13 | re_model = mind() 14 | re_model.load_weights('mind_model.h5') 15 | 16 | print(re_model.summary()) 17 | 18 | 19 | 20 | # 2. Load data 21 | 22 | user_id, gender, age, occupation, zip, \ 23 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 24 | 25 | with open("test.txt", 'r') as f: 26 | for line in f.readlines(): 27 | 28 | buf = line.strip().split('\t') 29 | 30 | user_id.append(int(buf[0])) 31 | gender.append(int(buf[1])) 32 | age.append(int(buf[2])) 33 | occupation.append(int(buf[3])) 34 | zip.append(int(buf[4])) 35 | hist_movie_id.append(np.array([int(i) for i in buf[5].strip().split(",")])) 36 | hist_len.append(int(buf[6])) 37 | pos_movie_id.append(int(buf[7])) 38 | 39 | 40 | user_id = np.array(user_id, dtype='int32') 41 | gender = np.array(gender, dtype='int32') 42 | age = np.array(age, dtype='int32') 43 | occupation = np.array(occupation, dtype='int32') 44 | zip = np.array(zip, dtype='int32') 45 | hist_movie_id = np.array(hist_movie_id, dtype='int32') 46 | hist_len = np.array(hist_len, dtype='int32') 47 | pos_movie_id = np.array(pos_movie_id, dtype='int32') 48 | 49 | 50 | 51 | 52 | # 3. Generate user features for testing and full item features for retrieval 53 | 54 | test_user_model_input = [user_id, gender, age, occupation, zip, hist_movie_id, hist_len] 55 | all_item_model_input = list(range(0, 3706+1)) 56 | 57 | user_embedding_model = Model(inputs=re_model.user_input, outputs=re_model.user_embedding) 58 | item_embedding_model = Model(inputs=re_model.item_input, outputs=re_model.item_embedding) 59 | 60 | user_embs = user_embedding_model.predict(test_user_model_input) 61 | item_embs = item_embedding_model.predict(all_item_model_input, batch_size=2 ** 12) 62 | 63 | print(user_embs.shape) 64 | print(item_embs.shape) 65 | 66 | 67 | user_embs = np.reshape(user_embs, (-1, 64)) 68 | item_embs = np.reshape(item_embs, (-1, 64)) 69 | 70 | print(user_embs[:2]) 71 | print(item_embs.shape) -------------------------------------------------------------------------------- /AFM/preprocess.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | from tqdm import tqdm 4 | from tensorflow.python.keras.preprocessing.sequence import pad_sequences 5 | 6 | def gen_data_set(data, negsample=0): 7 | 8 | data.sort_values("timestamp", inplace=True) 9 | item_ids = data['movie_id'].unique() 10 | 11 | train_set = [] 12 | test_set = [] 13 | for reviewerID, hist in tqdm(data.groupby('user_id')): 14 | pos_list = hist['movie_id'].tolist() 15 | rating_list = hist['rating'].tolist() 16 | 17 | if negsample > 0: 18 | candidate_set = list(set(item_ids) - set(pos_list)) 19 | neg_list = np.random.choice(candidate_set,size=len(pos_list)*negsample,replace=True) 20 | for i in range(1, len(pos_list)): 21 | hist = pos_list[:i] 22 | if i != len(pos_list) - 1: 23 | train_set.append((reviewerID, hist[::-1], pos_list[i], 1,len(hist[::-1]),rating_list[i])) 24 | for negi in range(negsample): 25 | train_set.append((reviewerID, hist[::-1], neg_list[i*negsample+negi], 0,len(hist[::-1]))) 26 | else: 27 | test_set.append((reviewerID, hist[::-1], pos_list[i],1,len(hist[::-1]),rating_list[i])) 28 | 29 | random.shuffle(train_set) 30 | random.shuffle(test_set) 31 | 32 | print(len(train_set[0]),len(test_set[0])) 33 | 34 | return train_set,test_set 35 | 36 | def gen_model_input(train_set,user_profile,seq_max_len): 37 | 38 | train_uid = np.array([line[0] for line in train_set]) 39 | train_seq = [line[1] for line in train_set] 40 | train_iid = np.array([line[2] for line in train_set]) 41 | train_label = np.array([line[3] for line in train_set]) 42 | train_hist_len = np.array([line[4] for line in train_set]) 43 | 44 | train_seq_pad = pad_sequences(train_seq, maxlen=seq_max_len, padding='post', value=0) 45 | train_model_input = {"user_id": train_uid, "movie_id": train_iid, "hist_movie_id": train_seq_pad, 46 | "hist_len": train_hist_len} 47 | 48 | for key in ["gender", "age", "occupation", "zip"]: 49 | train_model_input[key] = user_profile.loc[train_model_input['user_id']][key].values 50 | 51 | return train_model_input,train_label 52 | -------------------------------------------------------------------------------- /BilinearFFM/preprocess.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | from tqdm import tqdm 4 | from tensorflow.python.keras.preprocessing.sequence import pad_sequences 5 | 6 | def gen_data_set(data, negsample=0): 7 | 8 | data.sort_values("timestamp", inplace=True) 9 | item_ids = data['movie_id'].unique() 10 | 11 | train_set = [] 12 | test_set = [] 13 | for reviewerID, hist in tqdm(data.groupby('user_id')): 14 | pos_list = hist['movie_id'].tolist() 15 | rating_list = hist['rating'].tolist() 16 | 17 | if negsample > 0: 18 | candidate_set = list(set(item_ids) - set(pos_list)) 19 | neg_list = np.random.choice(candidate_set,size=len(pos_list)*negsample,replace=True) 20 | for i in range(1, len(pos_list)): 21 | hist = pos_list[:i] 22 | if i != len(pos_list) - 1: 23 | train_set.append((reviewerID, hist[::-1], pos_list[i], 1,len(hist[::-1]),rating_list[i])) 24 | for negi in range(negsample): 25 | train_set.append((reviewerID, hist[::-1], neg_list[i*negsample+negi], 0,len(hist[::-1]))) 26 | else: 27 | test_set.append((reviewerID, hist[::-1], pos_list[i],1,len(hist[::-1]),rating_list[i])) 28 | 29 | random.shuffle(train_set) 30 | random.shuffle(test_set) 31 | 32 | print(len(train_set[0]),len(test_set[0])) 33 | 34 | return train_set,test_set 35 | 36 | def gen_model_input(train_set,user_profile,seq_max_len): 37 | 38 | train_uid = np.array([line[0] for line in train_set]) 39 | train_seq = [line[1] for line in train_set] 40 | train_iid = np.array([line[2] for line in train_set]) 41 | train_label = np.array([line[3] for line in train_set]) 42 | train_hist_len = np.array([line[4] for line in train_set]) 43 | 44 | train_seq_pad = pad_sequences(train_seq, maxlen=seq_max_len, padding='post', value=0) 45 | train_model_input = {"user_id": train_uid, "movie_id": train_iid, "hist_movie_id": train_seq_pad, 46 | "hist_len": train_hist_len} 47 | 48 | for key in ["gender", "age", "occupation", "zip"]: 49 | train_model_input[key] = user_profile.loc[train_model_input['user_id']][key].values 50 | 51 | return train_model_input,train_label 52 | -------------------------------------------------------------------------------- /MIND/preprocess.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | from tqdm import tqdm 4 | from tensorflow.python.keras.preprocessing.sequence import pad_sequences 5 | 6 | def gen_data_set(data, negsample=0): 7 | 8 | data.sort_values("timestamp", inplace=True) 9 | item_ids = data['movie_id'].unique() 10 | 11 | train_set = [] 12 | test_set = [] 13 | for reviewerID, hist in tqdm(data.groupby('user_id')): 14 | pos_list = hist['movie_id'].tolist() 15 | rating_list = hist['rating'].tolist() 16 | 17 | if negsample > 0: 18 | candidate_set = list(set(item_ids) - set(pos_list)) 19 | neg_list = np.random.choice(candidate_set,size=len(pos_list)*negsample,replace=True) 20 | for i in range(1, len(pos_list)): 21 | hist = pos_list[:i] 22 | if i != len(pos_list) - 1: 23 | train_set.append((reviewerID, hist[::-1], pos_list[i], 1,len(hist[::-1]),rating_list[i])) 24 | for negi in range(negsample): 25 | train_set.append((reviewerID, hist[::-1], neg_list[i*negsample+negi], 0,len(hist[::-1]))) 26 | else: 27 | test_set.append((reviewerID, hist[::-1], pos_list[i],1,len(hist[::-1]),rating_list[i])) 28 | 29 | random.shuffle(train_set) 30 | random.shuffle(test_set) 31 | 32 | print(len(train_set[0]),len(test_set[0])) 33 | 34 | return train_set,test_set 35 | 36 | 37 | def gen_model_input(train_set,user_profile,seq_max_len): 38 | 39 | train_uid = np.array([line[0] for line in train_set]) 40 | train_seq = [line[1] for line in train_set] 41 | train_iid = np.array([line[2] for line in train_set]) 42 | train_label = np.array([line[3] for line in train_set]) 43 | train_hist_len = np.array([line[4] for line in train_set]) 44 | 45 | train_seq_pad = pad_sequences(train_seq, maxlen=seq_max_len, padding='post', value=0) 46 | train_model_input = {"user_id": train_uid, "movie_id": train_iid, "hist_movie_id": train_seq_pad, 47 | "hist_len": train_hist_len} 48 | 49 | for key in ["gender", "age", "occupation", "zip"]: 50 | train_model_input[key] = user_profile.loc[train_model_input['user_id']][key].values 51 | 52 | return train_model_input,train_label 53 | -------------------------------------------------------------------------------- /YouTubeNet/preprocess.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | from tqdm import tqdm 4 | from tensorflow.python.keras.preprocessing.sequence import pad_sequences 5 | 6 | def gen_data_set(data, negsample=0): 7 | 8 | data.sort_values("timestamp", inplace=True) 9 | item_ids = data['movie_id'].unique() 10 | 11 | train_set = [] 12 | test_set = [] 13 | for reviewerID, hist in tqdm(data.groupby('user_id')): 14 | pos_list = hist['movie_id'].tolist() 15 | rating_list = hist['rating'].tolist() 16 | 17 | if negsample > 0: 18 | candidate_set = list(set(item_ids) - set(pos_list)) 19 | neg_list = np.random.choice(candidate_set,size=len(pos_list)*negsample,replace=True) 20 | for i in range(1, len(pos_list)): 21 | hist = pos_list[:i] 22 | if i != len(pos_list) - 1: 23 | train_set.append((reviewerID, hist[::-1], pos_list[i], 1,len(hist[::-1]),rating_list[i])) 24 | for negi in range(negsample): 25 | train_set.append((reviewerID, hist[::-1], neg_list[i*negsample+negi], 0,len(hist[::-1]))) 26 | else: 27 | test_set.append((reviewerID, hist[::-1], pos_list[i],1,len(hist[::-1]),rating_list[i])) 28 | 29 | random.shuffle(train_set) 30 | random.shuffle(test_set) 31 | 32 | print(len(train_set[0]),len(test_set[0])) 33 | 34 | return train_set,test_set 35 | 36 | def gen_model_input(train_set,user_profile,seq_max_len): 37 | 38 | train_uid = np.array([line[0] for line in train_set]) 39 | train_seq = [line[1] for line in train_set] 40 | train_iid = np.array([line[2] for line in train_set]) 41 | train_label = np.array([line[3] for line in train_set]) 42 | train_hist_len = np.array([line[4] for line in train_set]) 43 | 44 | train_seq_pad = pad_sequences(train_seq, maxlen=seq_max_len, padding='post', value=0) 45 | train_model_input = {"user_id": train_uid, "movie_id": train_iid, "hist_movie_id": train_seq_pad, 46 | "hist_len": train_hist_len} 47 | 48 | for key in ["gender", "age", "occupation", "zip"]: 49 | train_model_input[key] = user_profile.loc[train_model_input['user_id']][key].values 50 | 51 | return train_model_input,train_label 52 | -------------------------------------------------------------------------------- /ESMM/write_tfrecord.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | 4 | 5 | def _parse_line(line): 6 | buf = line.split(',') 7 | ctr_label = int(buf[0]) 8 | cvr_label = int(buf[1]) 9 | pin_vec = np.array(buf[2: 128+2], dtype=np.float32) 10 | sku_vec = np.array(buf[128+2: 128+128+2], dtype=np.float32) 11 | 12 | return pin_vec, sku_vec, ctr_label, cvr_label 13 | 14 | 15 | def _to_float_feature(value_list): 16 | return tf.train.Feature(float_list=tf.train.FloatList(value=value_list)) 17 | 18 | 19 | def _to_int64_feature(value): 20 | return tf.train.Feature(int64_list=tf.train.Int64List(value=[value])) 21 | 22 | 23 | def serialize_example(pin_vec, sku_vec, ctr_label, cvr_label): 24 | feature = { 25 | 'pin_vec': _to_float_feature(pin_vec), 26 | 'sku_vec': _to_float_feature(sku_vec), 27 | 'ctr_label': _to_int64_feature(ctr_label), 28 | 'ctcvr_label': _to_int64_feature(cvr_label) 29 | } 30 | 31 | example_proto = tf.train.Example(features=tf.train.Features( 32 | feature=feature)) 33 | 34 | example = example_proto.SerializeToString() 35 | 36 | return example 37 | 38 | 39 | def make_tfrecord(input_path, tf_path): 40 | writer = tf.data.experimental.TFRecordWriter(tf_path, compression_type="ZLIB") 41 | 42 | def generator(): 43 | with open(input_path, 'r') as f: 44 | cnt = 0 45 | for line in f: 46 | cnt += 1 47 | if cnt % 10000 == 0: 48 | print("write %s lines." % cnt) 49 | 50 | pin_vec, sku_vec, ctr_label, cvr_label = _parse_line(line[:-1]) 51 | 52 | yield serialize_example(pin_vec, sku_vec, ctr_label, cvr_label) 53 | 54 | 55 | serialized_dataset = tf.data.Dataset.from_generator( 56 | generator, output_types=tf.string, output_shapes=()) 57 | 58 | writer.write(serialized_dataset) 59 | 60 | print("Write tf_path=%s done!" % tf_path) 61 | 62 | 63 | if __name__ == '__main__': 64 | dir_path = './' 65 | train_path = dir_path + 'data/train_data' 66 | tf_train_path = dir_path + 'data/train_data.tfrecord' 67 | make_tfrecord(train_path, tf_train_path) 68 | 69 | val_path = dir_path + 'data/val_data' 70 | tf_val_path = dir_path + 'data/val_data.tfrecord' 71 | make_tfrecord(val_path, tf_val_path) 72 | -------------------------------------------------------------------------------- /MIND/data_generator.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import numpy as np 4 | 5 | 6 | def init_output(): 7 | user_id = [] 8 | gender = [] 9 | age = [] 10 | occupation = [] 11 | zip = [] 12 | hist_movie_id = [] 13 | hist_len = [] 14 | pos_movie_id = [] 15 | neg_movie_id = [] 16 | 17 | 18 | return user_id, gender, age, occupation, zip, \ 19 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id 20 | 21 | 22 | def file_generator(input_path, batch_size): 23 | 24 | user_id, gender, age, occupation, zip, \ 25 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 26 | 27 | cnt = 0 28 | 29 | num_lines = sum([1 for line in open(input_path)]) 30 | 31 | while True: 32 | 33 | with open(input_path, 'r') as f: 34 | for line in f.readlines(): 35 | 36 | buf = line.strip().split('\t') 37 | 38 | user_id.append(int(buf[0])) 39 | gender.append(int(buf[1])) 40 | age.append(int(buf[2])) 41 | occupation.append(int(buf[3])) 42 | zip.append(int(buf[4])) 43 | hist_movie_id.append(np.array([int(i) for i in buf[5].strip().split(",")])) 44 | hist_len.append(int(buf[6])) 45 | pos_movie_id.append(int(buf[7])) 46 | neg_movie_id.append(np.array([int(i) for i in buf[8].strip().split(",")])) 47 | 48 | cnt += 1 49 | 50 | if cnt % batch_size == 0 or cnt == num_lines: 51 | user_id = np.array(user_id, dtype='int32') 52 | gender = np.array(gender, dtype='int32') 53 | age = np.array(age, dtype='int32') 54 | occupation = np.array(occupation, dtype='int32') 55 | zip = np.array(zip, dtype='int32') 56 | hist_movie_id = np.array(hist_movie_id, dtype='int32') 57 | hist_len = np.array(hist_len, dtype='int32') 58 | pos_movie_id = np.array(pos_movie_id, dtype='int32') 59 | neg_movie_id = np.array(neg_movie_id, dtype='int32') 60 | 61 | label = np.zeros(len(user_id)) # 正样本的index位置为0, 10个负样本的索引位置为[1-10] 62 | 63 | yield [user_id, gender, age, occupation, zip, hist_movie_id, hist_len, pos_movie_id, neg_movie_id], label 64 | 65 | user_id, gender, age, occupation, zip, hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 66 | 67 | -------------------------------------------------------------------------------- /YouTubeNet/data_generator.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import numpy as np 4 | 5 | 6 | def init_output(): 7 | user_id = [] 8 | gender = [] 9 | age = [] 10 | occupation = [] 11 | zip = [] 12 | hist_movie_id = [] 13 | hist_len = [] 14 | pos_movie_id = [] 15 | neg_movie_id = [] 16 | 17 | 18 | return user_id, gender, age, occupation, zip, \ 19 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id 20 | 21 | 22 | def file_generator(input_path, batch_size): 23 | 24 | user_id, gender, age, occupation, zip, \ 25 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 26 | 27 | cnt = 0 28 | 29 | num_lines = sum([1 for line in open(input_path)]) 30 | 31 | while True: 32 | 33 | with open(input_path, 'r') as f: 34 | for line in f.readlines(): 35 | 36 | buf = line.strip().split('\t') 37 | 38 | user_id.append(int(buf[0])) 39 | gender.append(int(buf[1])) 40 | age.append(int(buf[2])) 41 | occupation.append(int(buf[3])) 42 | zip.append(int(buf[4])) 43 | hist_movie_id.append(np.array([int(i) for i in buf[5].strip().split(",")])) 44 | hist_len.append(int(buf[6])) 45 | pos_movie_id.append(int(buf[7])) 46 | neg_movie_id.append(np.array([int(i) for i in buf[8].strip().split(",")])) 47 | 48 | cnt += 1 49 | 50 | if cnt % batch_size == 0 or cnt == num_lines: 51 | user_id = np.array(user_id, dtype='int32') 52 | gender = np.array(gender, dtype='int32') 53 | age = np.array(age, dtype='int32') 54 | occupation = np.array(occupation, dtype='int32') 55 | zip = np.array(zip, dtype='int32') 56 | hist_movie_id = np.array(hist_movie_id, dtype='int32') 57 | hist_len = np.array(hist_len, dtype='int32') 58 | pos_movie_id = np.array(pos_movie_id, dtype='int32') 59 | neg_movie_id = np.array(neg_movie_id, dtype='int32') 60 | 61 | label = np.zeros(len(user_id)) # 正样本的index位置为0, 10个负样本的索引位置为[1-10] 62 | 63 | yield [user_id, gender, age, occupation, zip, hist_movie_id, hist_len, pos_movie_id, neg_movie_id], label 64 | 65 | user_id, gender, age, occupation, zip, hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 66 | 67 | -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/search_topN_figure.py: -------------------------------------------------------------------------------- 1 | # encoding:utf-8 2 | from annoy import AnnoyIndex 3 | import numpy as np 4 | np.random.seed(20200601) 5 | import pickle 6 | import sys 7 | from matplotlib import image as mpimg 8 | from matplotlib import pyplot as plt 9 | 10 | def load_ann(ann_path=None, index_to_name_dict_path=None, dim=64): 11 | ann = AnnoyIndex(dim) 12 | ann.load(ann_path) 13 | 14 | with open(index_to_name_dict_path, 'rb') as f: 15 | index_to_name_dict = pickle.load(f) 16 | return ann, index_to_name_dict 17 | 18 | 19 | def query_ann(ann=None, index_to_name_dict=None, query_vec=None, topN=None): 20 | topN_item_idx_list = ann.get_nns_by_vector(query_vec, topN) 21 | 22 | topN_item_id_list = [] 23 | 24 | for idx in topN_item_idx_list: 25 | item_id = index_to_name_dict[idx] 26 | topN_item_id_list.append(item_id) 27 | 28 | return topN_item_id_list 29 | 30 | 31 | if __name__ == '__main__': 32 | index_to_name_dict_path = "index_to_name_dict.pkl" 33 | ann_path = "img_feature_list.ann" 34 | name_path = "img_name_list.pkl" 35 | vec_path = "img_feature_list.pkl" 36 | dim = 25088 37 | topN = 9 38 | 39 | name_path = open(name_path, 'rb') 40 | vec_path = open(vec_path, 'rb') 41 | img_name_list = pickle.load(name_path) 42 | img_vec_list = pickle.load(vec_path) 43 | 44 | idx = 126 45 | query_name = img_name_list[idx] 46 | query_vec = img_vec_list[idx] 47 | 48 | ann, index_to_name_dict = load_ann(ann_path=ann_path, \ 49 | index_to_name_dict_path=index_to_name_dict_path, \ 50 | dim=dim) 51 | 52 | topN_item_list = query_ann(ann=ann, \ 53 | index_to_name_dict=index_to_name_dict, \ 54 | query_vec=query_vec, \ 55 | topN=topN) 56 | 57 | # query 商品图片 58 | print("query_image: \n") 59 | fig, axes = plt.subplots(1, 1) 60 | query_image = mpimg.imread("figures/" + query_name) 61 | axes.imshow(query_image/255) 62 | axes.axis('off') 63 | axes.axis('off') 64 | axes.set_title('%s' % query_name, fontsize=8, color='r') 65 | 66 | # Top-9 相似商品 67 | fig, axes = plt.subplots(3, 3) 68 | for idx, img_path in enumerate(topN_item_list): 69 | 70 | i = idx % 3 # Get subplot row 71 | j = idx // 3 # Get subplot column 72 | image = mpimg.imread("figures/" + img_path) 73 | axes[i, j].imshow(image/255) 74 | axes[i, j].axis('off') 75 | axes[i, j].axis('off') 76 | 77 | axes[i, j].set_title('%s' % img_path, fontsize=8, color='b') -------------------------------------------------------------------------------- /BilinearFFM/BilinearFFM.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | 4 | import tensorflow as tf 5 | from tensorflow.keras.layers import Input, Embedding, concatenate, Dense, Dropout 6 | 7 | from tensorflow.keras.models import Model 8 | from tensorflow.keras.optimizers import Adam 9 | from tensorflow.keras.callbacks import EarlyStopping 10 | from BilinearInteraction import BilinearInteraction 11 | 12 | 13 | def BilinearFFM( 14 | sparse_input_length=1, 15 | embedding_dim = 64 16 | ): 17 | 18 | # 1. Input layer 19 | user_id_input_layer = Input(shape=(sparse_input_length, ), name="user_id_input_layer") 20 | gender_input_layer = Input(shape=(sparse_input_length, ), name="gender_input_layer") 21 | age_input_layer = Input(shape=(sparse_input_length, ), name="age_input_layer") 22 | occupation_input_layer = Input(shape=(sparse_input_length, ), name="occupation_input_layer") 23 | zip_input_layer = Input(shape=(sparse_input_length, ), name="zip_input_layer") 24 | item_input_layer = Input(shape=(sparse_input_length, ), name="item_input_layer") 25 | 26 | 27 | # 2. Embedding layer 28 | user_id_embedding_layer = Embedding(6040+1, embedding_dim, mask_zero=True, name='user_id_embedding_layer')(user_id_input_layer) 29 | gender_embedding_layer = Embedding(2+1, embedding_dim, mask_zero=True, name='gender_embedding_layer')(gender_input_layer) 30 | age_embedding_layer = Embedding(7+1, embedding_dim, mask_zero=True, name='age_embedding_layer')(age_input_layer) 31 | occupation_embedding_layer = Embedding(21+1, embedding_dim, mask_zero=True, name='occupation_embedding_layer')(occupation_input_layer) 32 | zip_embedding_layer = Embedding(3439+1, embedding_dim, mask_zero=True, name='zip_embedding_layer')(zip_input_layer) 33 | item_id_embedding_layer = Embedding(3706+1, embedding_dim, mask_zero=True, name='item_id_embedding_layer')(item_input_layer) 34 | 35 | 36 | sparse_embedding_list = [user_id_embedding_layer, gender_embedding_layer, age_embedding_layer, \ 37 | occupation_embedding_layer, zip_embedding_layer, item_id_embedding_layer] 38 | 39 | 40 | 41 | # 3. Bilinear FFM 42 | bilinear_out = BilinearInteraction()(sparse_embedding_list) 43 | 44 | 45 | # Output 46 | dot_output = tf.nn.sigmoid(tf.reduce_sum(bilinear_out, axis=-1)) 47 | 48 | sparse_input_list = [user_id_input_layer, gender_input_layer, age_input_layer, occupation_input_layer, zip_input_layer, item_input_layer] 49 | model = Model(inputs = sparse_input_list, 50 | outputs = dot_output) 51 | 52 | 53 | 54 | return model -------------------------------------------------------------------------------- /AFM/AFM_Model.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | 4 | import tensorflow as tf 5 | from tensorflow.keras.layers import Input, Embedding, concatenate, Dense, Dropout 6 | 7 | from tensorflow.keras.models import Model 8 | from tensorflow.keras.optimizers import Adam 9 | from tensorflow.keras.callbacks import EarlyStopping 10 | from AFMLayer import AFMLayer 11 | 12 | 13 | def AFM( 14 | sparse_input_length=1, 15 | embedding_dim = 64 16 | ): 17 | 18 | # 1. Input layer 19 | user_id_input_layer = Input(shape=(sparse_input_length, ), name="user_id_input_layer") 20 | gender_input_layer = Input(shape=(sparse_input_length, ), name="gender_input_layer") 21 | age_input_layer = Input(shape=(sparse_input_length, ), name="age_input_layer") 22 | occupation_input_layer = Input(shape=(sparse_input_length, ), name="occupation_input_layer") 23 | zip_input_layer = Input(shape=(sparse_input_length, ), name="zip_input_layer") 24 | item_input_layer = Input(shape=(sparse_input_length, ), name="item_input_layer") 25 | 26 | 27 | # 2. Embedding layer 28 | user_id_embedding_layer = Embedding(6040+1, embedding_dim, mask_zero=True, name='user_id_embedding_layer')(user_id_input_layer) 29 | gender_embedding_layer = Embedding(2+1, embedding_dim, mask_zero=True, name='gender_embedding_layer')(gender_input_layer) 30 | age_embedding_layer = Embedding(7+1, embedding_dim, mask_zero=True, name='age_embedding_layer')(age_input_layer) 31 | occupation_embedding_layer = Embedding(21+1, embedding_dim, mask_zero=True, name='occupation_embedding_layer')(occupation_input_layer) 32 | zip_embedding_layer = Embedding(3439+1, embedding_dim, mask_zero=True, name='zip_embedding_layer')(zip_input_layer) 33 | item_id_embedding_layer = Embedding(3706+1, embedding_dim, mask_zero=True, name='item_id_embedding_layer')(item_input_layer) 34 | 35 | 36 | sparse_embedding_list = [user_id_embedding_layer, gender_embedding_layer, age_embedding_layer, \ 37 | occupation_embedding_layer, zip_embedding_layer, item_id_embedding_layer] 38 | 39 | 40 | 41 | # 3. AFM 42 | attention_factor = int(len(sparse_embedding_list) * (len(sparse_embedding_list)-1) / 2) 43 | afm_out = AFMLayer(attention_factor=attention_factor, l2_reg_w=1e-5, dropout_rate=0, seed=2020)\ 44 | (sparse_embedding_list) 45 | print("\n"*3) 46 | print("afm_out: ", afm_out) 47 | 48 | 49 | # Output 50 | output = tf.nn.sigmoid(tf.reduce_sum(afm_out, axis=-1)) 51 | 52 | sparse_input_list = [user_id_input_layer, gender_input_layer, age_input_layer, occupation_input_layer, zip_input_layer, item_input_layer] 53 | model = Model(inputs = sparse_input_list, 54 | outputs = output) 55 | 56 | 57 | return model -------------------------------------------------------------------------------- /vgg16_figure_search_annoy/download_jd_figures.py: -------------------------------------------------------------------------------- 1 | # encoding="utf-8" 2 | 3 | from requests_html import HTMLSession 4 | import re 5 | import os 6 | import time 7 | import threading 8 | import urllib.request 9 | 10 | 11 | item_english_list = ["Mobile-phone", "T-shirt", "Milk", "Mask", "Headset", \ 12 | "Wine", "Helmet", "Fan", "Sneaker", "Cup", \ 13 | "Glasses", "Backpack", "UAV", "Sofa", "Bicycle", \ 14 | "Cleanser", "Paper", "Bread", "Sausage", "Toilet", \ 15 | "Book", "Tire", "Clock", "Mango", "Shrimp", \ 16 | "Stroller", "Necklace", "Baby-bottle", "Yuba", "Pot"] 17 | 18 | 19 | item_chinese_list = ["手机", "T恤", "牛奶", "口罩", "耳机", \ 20 | "酒", "头盔", "风扇", "运动鞋", "杯子", \ 21 | "眼镜", "背包", "无人机", "沙发", "自行车", \ 22 | "洗面奶", "抽纸", "面包", "香肠", "马桶", \ 23 | "书", "轮胎", "钟表", "芒果", "虾", \ 24 | "童车", "项链", "奶瓶", "浴霸", "锅"] 25 | 26 | 27 | session = HTMLSession() 28 | 29 | 30 | def download_images(inx, key): 31 | 32 | for j in range(1, 10): 33 | 34 | time.sleep(2) 35 | url = 'https://search.jd.com/Search?keyword=%s&wq=%s&page=%s&s=90&click=0' % \ 36 | (key, key, str(j)) 37 | 38 | r = session.get(url) 39 | 40 | for i in range(1, 20): 41 | try: 42 | contain_pic_url = str(r.html.find('#J_goodsList > ul > li:nth-child('+str(i)+') > div > div > div.gl-i-tab-content > div.tab-content-item.tab-cnt-i-selected > div.p-img > a > img')) 43 | src_start = re.search('src',contain_pic_url).end() + 2 44 | src_end = int(re.search("'",contain_pic_url[src_start:]).start()) 45 | pic_url = 'https:'+contain_pic_url[src_start:src_start + src_end] 46 | 47 | os.chdir('C:\\Users\\Desktop\\figures') 48 | pic = session.get(pic_url) 49 | open(item_english_list[inx]+'_page_'+str(j)+'_NO_'+str(i)+'.jpg','wb').write(pic.content) 50 | 51 | except: 52 | try: 53 | contain_pic_url = str(r.html.find('#J_goodsList > ul > li:nth-child('+str(i)+') > div > div.p-img > a > img')) 54 | src_start = re.search('src',contain_pic_url).end() + 2 55 | src_end = int(re.search("'",contain_pic_url[src_start:]).start()) 56 | pic_url = 'https:'+contain_pic_url[src_start:src_start + src_end] 57 | 58 | os.chdir('C:\\Users\\Desktop\\figures') 59 | pic = session.get(pic_url) 60 | open(item_english_list[inx]+'_page_'+str(j)+'_NO_'+str(i)+'.jpg','wb').write(pic.content) 61 | 62 | except: 63 | pass 64 | 65 | print("Download %s done !!!" % item_english_list[inx]) 66 | 67 | 68 | 69 | def main(): 70 | threads = [] 71 | 72 | for inx, key in enumerate(item_chinese_list): 73 | thread = threading.Thread(target=download_images, args=(inx, key)) 74 | threads.append(thread) 75 | thread.start() 76 | 77 | for i in threads: 78 | i.join() 79 | 80 | 81 | 82 | start_time = time.time() 83 | main() 84 | 85 | print("use time is: ", time.time()-start_time) -------------------------------------------------------------------------------- /YouTubeNet/SequencePoolingLayer.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | 4 | import tensorflow as tf 5 | from tensorflow.keras.layers import Lambda, Layer 6 | 7 | 8 | class SequencePoolingLayer(Layer): 9 | 10 | def __init__(self, mode="mean", support_mask=True, sequence_mask_length=50, **kwargs): 11 | 12 | if mode not in ["mean", "max", "sum"]: 13 | raise ValueError("mode must be `mean`, `max` or `sum` !") 14 | 15 | self.mode = mode 16 | self.eps = tf.constant(1e-8, tf.float32) 17 | self.support_mask = support_mask 18 | self.sequence_mask_length = sequence_mask_length 19 | 20 | super(SequencePoolingLayer, self).__init__(**kwargs) 21 | 22 | 23 | def build(self, input_shape): 24 | super(SequencePoolingLayer, self).build(input_shape) 25 | 26 | 27 | def call(self, input_hist_seq_list, **kwargs): 28 | 29 | hist_user_embedding_list, hist_user_behavior_length = input_hist_seq_list 30 | 31 | if not self.support_mask: 32 | 33 | if self.mode == "max": 34 | return tf.reduce_max(hist_user_embedding_list, 1, keepdims=True) 35 | 36 | mode_sum = tf.reduce_sum(hist_user_embedding_list, 1, keepdims=True) 37 | 38 | if self.mode == "sum": 39 | return mode_sum 40 | 41 | if self.mode == "mean": 42 | return tf.divide(mode_sum, self.sequence_mask_length + self.eps) 43 | 44 | 45 | if self.support_mask: 46 | 47 | # mask matrix 48 | mask_list = tf.sequence_mask(hist_user_behavior_length, self.sequence_mask_length, dtype=tf.float32) 49 | 50 | # transpose mask matrix 51 | mask_transpose_list = tf.transpose(mask_list, (0, 2, 1)) 52 | embedding_dim = hist_user_embedding_list.shape[-1] 53 | 54 | # expand mask matrix 55 | mask_tile_list = tf.tile(mask_transpose_list, [1, 1, embedding_dim]) 56 | 57 | 58 | # max 59 | if self.mode == "max": 60 | hist = hist_user_embedding_list - (1-mask_tile_list) * 1e9 61 | return tf.reduce_max(hist, 1, keepdims=True) 62 | 63 | 64 | mode_sum = tf.reduce_sum(hist_user_embedding_list * mask_tile_list, 1, keepdims=True) 65 | 66 | # sum 67 | if self.mode == "sum": 68 | return mode_sum 69 | 70 | # mean 71 | if self.mode == "mean": 72 | hist_user_behavior_length = tf.reduce_sum(mask_list, axis=-1, keepdims=True) 73 | 74 | return tf.divide(mode_sum, \ 75 | tf.cast(hist_user_behavior_length, tf.float32) + self.eps) 76 | 77 | 78 | def compute_output_shape(self, input_shape): 79 | return (None, 1, input_shape[0][-1]) 80 | 81 | 82 | def config(self): 83 | config = {"mode": self.mode, "support_mask": self.support_mask, \ 84 | "sequence_mask_length": self.sequence_mask_length} 85 | 86 | base_config = super(SequencePoolingLayer, self).get_config() 87 | 88 | return dict(list(base_config.items()) + list(config.items)) -------------------------------------------------------------------------------- /ESMM/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | # Data format : 4 | 5 | ```python 6 | 说明： 7 | 8 | （1）第1列：ctr 的值，0表示曝光后未点击，1表示曝光后点击； 9 | （2）第2列：ctcvr 的值，0表示（曝光后未点击、或曝光后点击）后未购买，1表示（曝光、点击）后购买； 10 | 11 | （3）第3-130列：用户的预训练Embedding； 12 | （4）第131-258列：商品的预训练Embedding； 13 | 14 | ``` 15 | 16 | 17 | # run model 18 | ```shell 19 | sh master.sh 20 | 21 | ``` 22 | 23 | 24 | # 模型 summary 25 | ```python 26 | __________________________________________________________________________________________________ 27 | Layer (type) Output Shape Param # Connected to 28 | ================================================================================================== 29 | input_1 (InputLayer) [(None, 128)] 0 30 | __________________________________________________________________________________________________ 31 | input_2 (InputLayer) [(None, 128)] 0 32 | __________________________________________________________________________________________________ 33 | dense (Dense) (None, 64) 8256 input_1[0][0] 34 | __________________________________________________________________________________________________ 35 | dense_1 (Dense) (None, 64) 8256 input_2[0][0] 36 | __________________________________________________________________________________________________ 37 | dense_2 (Dense) (None, 64) 8256 input_1[0][0] 38 | __________________________________________________________________________________________________ 39 | dense_3 (Dense) (None, 64) 8256 input_2[0][0] 40 | __________________________________________________________________________________________________ 41 | multiply (Multiply) (None, 64) 0 dense[0][0] 42 | dense_1[0][0] 43 | __________________________________________________________________________________________________ 44 | multiply_1 (Multiply) (None, 64) 0 dense_2[0][0] 45 | dense_3[0][0] 46 | __________________________________________________________________________________________________ 47 | ctr_prob (Dense) (None, 1) 65 multiply[0][0] 48 | __________________________________________________________________________________________________ 49 | cvr_prob (Dense) (None, 1) 65 multiply_1[0][0] 50 | __________________________________________________________________________________________________ 51 | ctcvr_prob (Multiply) (None, 1) 0 ctr_prob[0][0] 52 | cvr_prob[0][0] 53 | ================================================================================================== 54 | Total params: 33,154 55 | Trainable params: 33,154 56 | Non-trainable params: 0 57 | __________________________________________________________________________________________________ 58 | ``` 59 | 60 | 61 | # 参考 62 | 63 | ```python 64 | 1. https://mp.weixin.qq.com/s/J3GH1G1xuzEaPdxLzKDGOA 65 | ``` -------------------------------------------------------------------------------- /BilinearFFM/BilinearInteraction.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.keras.layers import Layer 3 | from tensorflow.keras import backend as K 4 | import itertools 5 | from tensorflow.keras.initializers import (Zeros, glorot_normal, glorot_uniform) 6 | from tensorflow.keras.layers import Concatenate 7 | 8 | 9 | class BilinearInteraction(Layer): 10 | """ 11 | Input shape 12 | - A list of 3D tensor with shape: ``(batch_size,1,embedding_size)``. 13 | 14 | Output shape 15 | - 3D tensor with shape: ``(batch_size,1,embedding_size)``. 16 | 17 | Arguments 18 | - **str** : String, types of bilinear functions used in this layer. 19 | 20 | - **seed** : A Python integer to use as random seed. 21 | """ 22 | 23 | def __init__(self, bilinear_type="each", seed=1024, **kwargs): 24 | self.bilinear_type = bilinear_type 25 | self.seed = seed 26 | 27 | super(BilinearInteraction, self).__init__(**kwargs) 28 | 29 | def build(self, input_shape): 30 | 31 | if not isinstance(input_shape, list) or len(input_shape) < 2: 32 | raise ValueError('A `AttentionalFM` layer should be called on a list of at least 2 inputs') 33 | embedding_size = int(input_shape[0][-1]) 34 | 35 | if self.bilinear_type == "all": 36 | self.W = self.add_weight(shape=(embedding_size, embedding_size), initializer=glorot_normal( 37 | seed=self.seed), name="bilinear_weight") 38 | elif self.bilinear_type == "each": 39 | self.W_list = [self.add_weight(shape=(embedding_size, embedding_size), initializer=glorot_normal( 40 | seed=self.seed), name="bilinear_weight" + str(i)) for i in range(len(input_shape) - 1)] 41 | elif self.bilinear_type == "interaction": 42 | self.W_list = [self.add_weight(shape=(embedding_size, embedding_size), initializer=glorot_normal( 43 | seed=self.seed), name="bilinear_weight" + str(i) + '_' + str(j)) for i, j in 44 | itertools.combinations(range(len(input_shape)), 2)] 45 | else: 46 | raise NotImplementedError 47 | 48 | super(BilinearInteraction, self).build( 49 | input_shape) # Be sure to call this somewhere! 50 | 51 | def call(self, inputs, **kwargs): 52 | 53 | if K.ndim(inputs[0]) != 3: 54 | raise ValueError( 55 | "Unexpected inputs dimensions %d, expect to be 2 dimensions" % (K.ndim(inputs))) 56 | 57 | if self.bilinear_type == "all": 58 | p = [tf.multiply(tf.tensordot(v_i, self.W, axes=(-1, 0)), v_j) 59 | for v_i, v_j in itertools.combinations(inputs, 2)] 60 | elif self.bilinear_type == "each": 61 | p = [tf.multiply(tf.tensordot(inputs[i], self.W_list[i], axes=(-1, 0)), inputs[j]) 62 | for i, j in itertools.combinations(range(len(inputs)), 2)] 63 | elif self.bilinear_type == "interaction": 64 | p = [tf.multiply(tf.tensordot(v[0], w, axes=(-1, 0)), v[1]) 65 | for v, w in zip(itertools.combinations(inputs, 2), self.W_list)] 66 | else: 67 | raise NotImplementedError 68 | 69 | return Concatenate(axis=-1)(p) 70 | 71 | 72 | def compute_output_shape(self, input_shape): 73 | filed_size = len(input_shape) 74 | embedding_size = input_shape[0][-1] 75 | 76 | return (None, 1, filed_size * (filed_size - 1) // 2 * embedding_size) 77 | 78 | def get_config(self, ): 79 | config = {'bilinear_type': self.bilinear_type, 'seed': self.seed} 80 | base_config = super(BilinearInteraction, self).get_config() 81 | return dict(list(base_config.items()) + list(config.items())) 82 | -------------------------------------------------------------------------------- /YouTubeNet/load_YouTubeNet_model_to_predict.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | from tensorflow.keras.models import Model 6 | 7 | from YouTubeNet import YouTubeNet 8 | from data_generator import init_output 9 | 10 | 11 | # 1. Load model 12 | 13 | re_model = YouTubeNet() 14 | re_model.load_weights('YouTubeNet_model.h5') 15 | 16 | 17 | 18 | 19 | # 2. Load data 20 | 21 | user_id, gender, age, occupation, zip, \ 22 | hist_movie_id, hist_len, pos_movie_id, neg_movie_id = init_output() 23 | 24 | with open("test.txt", 'r') as f: 25 | for line in f.readlines(): 26 | 27 | buf = line.strip().split('\t') 28 | 29 | user_id.append(int(buf[0])) 30 | gender.append(int(buf[1])) 31 | age.append(int(buf[2])) 32 | occupation.append(int(buf[3])) 33 | zip.append(int(buf[4])) 34 | hist_movie_id.append(np.array([int(i) for i in buf[5].strip().split(",")])) 35 | hist_len.append(int(buf[6])) 36 | pos_movie_id.append(int(buf[7])) 37 | 38 | 39 | user_id = np.array(user_id, dtype='int32') 40 | gender = np.array(gender, dtype='int32') 41 | age = np.array(age, dtype='int32') 42 | occupation = np.array(occupation, dtype='int32') 43 | zip = np.array(zip, dtype='int32') 44 | hist_movie_id = np.array(hist_movie_id, dtype='int32') 45 | hist_len = np.array(hist_len, dtype='int32') 46 | pos_movie_id = np.array(pos_movie_id, dtype='int32') 47 | 48 | 49 | 50 | # 3. Generate user features for testing and full item features for retrieval 51 | 52 | test_user_model_input = [user_id, gender, age, occupation, zip, hist_movie_id, hist_len] 53 | all_item_model_input = list(range(0, 3706+1)) 54 | 55 | user_embedding_model = Model(inputs=re_model.user_input, outputs=re_model.user_embedding) 56 | item_embedding_model = Model(inputs=re_model.item_input, outputs=re_model.item_embedding) 57 | 58 | user_embs = user_embedding_model.predict(test_user_model_input) 59 | item_embs = item_embedding_model.predict(all_item_model_input, batch_size=2 ** 12) 60 | 61 | print(user_embs.shape) 62 | print(item_embs.shape) 63 | 64 | 65 | user_embs = np.reshape(user_embs, (-1, 64)) 66 | item_embs = np.reshape(item_embs, (-1, 64)) 67 | 68 | print(user_embs[:2]) 69 | 70 | 71 | 72 | """ 73 | (6040, 1, 64) 74 | (3707, 1, 64) 75 | 76 | [[0. 0.84161407 0.5913373 1.4273984 0.3627409 0.3708319 77 | 0. 0. 1.1993251 2.023305 0. 0. 78 | 0. 0. 1.7670951 0.558543 1.0881244 1.7819335 79 | 0.6492757 2.6123888 0.3125449 0.36506268 0. 1.1256831 80 | 4.410721 1.7535956 0.52042466 1.4845431 0.4248005 0. 81 | 2.1689777 1.296214 1.1852415 0. 0. 0.43460703 82 | 1.927466 5.7313547 0. 0. 0. 0.36566824 83 | 2.012046 0. 0. 1.5223947 3.8016186 0. 84 | 0.34814402 1.909086 1.8206354 0.39664558 1.0465539 0. 85 | 1.8064818 0. 1.3177121 0.5385138 0. 2.6539533 86 | 0. 0. 0. 0. ] 87 | [0.8107976 1.1632944 0. 0.53690577 1.0428483 1.2018232 88 | 3.4726145 2.21235 0. 0.1572555 0.97843236 0. 89 | 0. 0.99380946 0.76257807 0.05231025 1.6611706 0.0405544 90 | 0.9629851 1.3969578 1.9982753 0. 0.1676663 0. 91 | 0. 0.07090688 2.1441605 0.5842841 0.09379 0. 92 | 0. 0. 0.49283475 2.134187 0. 0.8167961 93 | 0. 0. 1.8054122 0. 0. 1.266642 94 | 2.730833 0. 0. 0.5958151 0. 1.2587492 95 | 0.08325796 0. 0.22326717 0.6559374 0.54102665 0. 96 | 1.0489423 0. 0.5308376 0.62447524 0. 0. 97 | 2.3295872 0. 2.5632188 1.3600256 ]] 98 | (3707, 64) 99 | 100 | 101 | """ -------------------------------------------------------------------------------- /BST/din.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | from tensorflow.keras import backend as K 4 | 5 | 6 | 7 | def din_padding_mask(seq): 8 | # 获取为 0的padding项 9 | seq = tf.cast(tf.math.equal(seq, 0), tf.float32) 10 | 11 | # 扩充维度用于attention矩阵 12 | return seq[:, np.newaxis, :] # (batch_size, 1, seq_len) 13 | 14 | 15 | 16 | class LocalActivationUnit(tf.keras.layers.Layer): 17 | 18 | def __init__(self, d_model, middle_units, dropout_rate, **kwargs): 19 | self.d_model = d_model 20 | self.middle_units = middle_units 21 | self.dropout_rate = dropout_rate 22 | 23 | super(LocalActivationUnit, self).__init__(**kwargs) 24 | 25 | 26 | def build(self, input_shape): 27 | 28 | self.dnn = tf.keras.Sequential([ 29 | tf.keras.layers.Dense(self.middle_units, activation='relu'), 30 | tf.keras.layers.Dense(self.d_model, activation='relu') 31 | ]) 32 | 33 | super(LocalActivationUnit, self).build(input_shape) 34 | 35 | 36 | 37 | def call(self, inputs, training=None, **kwargs): 38 | 39 | query, keys = inputs 40 | keys_len = keys.get_shape()[1] 41 | queries = K.repeat_elements(query, keys_len, 1) 42 | 43 | att_input = tf.concat([queries, keys, queries - keys, queries * keys], axis=-1) 44 | att_out = self.dnn(att_input) 45 | attention_score = tf.keras.layers.Dense(1)(att_out) 46 | 47 | return attention_score 48 | 49 | 50 | def compute_output_shape(self, input_shape): 51 | return input_shape[1][:2] + (1,) 52 | 53 | 54 | def get_config(self): 55 | config = {'d_model': self.d_model, 'middle_units': self.middle_units, 'dropout_rate': self.dropout_rate} 56 | base_config = super(LocalActivationUnit, self).get_config() 57 | 58 | return dict(list(base_config.items()) + list(config.items())) 59 | 60 | 61 | 62 | # 构造 Din Attention Layer 层 63 | 64 | class DinAttentionLayer(tf.keras.layers.Layer): 65 | def __init__(self, d_model, middle_units, dropout_rate, **kwargs): 66 | super(DinAttentionLayer, self).__init__(**kwargs) 67 | 68 | self.d_model = d_model 69 | self.middle_units = middle_units 70 | self.dropout_rate = dropout_rate 71 | 72 | self.local_activation_unit = LocalActivationUnit(d_model, middle_units, dropout_rate) 73 | 74 | 75 | def call(self, inputs, **kwargs): 76 | query, keys, values, mask = inputs 77 | 78 | scaled_attention_logits = self.local_activation_unit([query, keys]) 79 | scaled_attention_logits = tf.transpose(scaled_attention_logits, perm=[0, 2, 1]) 80 | 81 | if mask is not None: 82 | scaled_attention_logits += (mask * -1e9) 83 | 84 | attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1) 85 | output = tf.matmul(attention_weights, values) 86 | 87 | return output 88 | 89 | 90 | def compute_output_shape(self, input_shape): 91 | return (input_shape[1][0], 1, input_shape[1][2]) 92 | 93 | 94 | def get_config(self): 95 | config = {'d_model': self.d_model, 'use_bias': self.middle_units, 'dropout_rate': self.dropout_rate} 96 | base_config = super(DinAttentionLayer, self).get_config() 97 | return dict(list(base_config.items()) + list(config.items())) 98 | 99 | 100 | 101 | if __name__ == "__main__": 102 | query = tf.random.uniform((10, 1, 64)) 103 | keys = tf.random.uniform((10, 50, 64)) 104 | vecs = keys 105 | 106 | din_padding_mask_list = din_padding_mask(np.random.randint(0, 15, size=(10, 50))) 107 | print("din_padding_mask_list.shape: ", din_padding_mask_list.shape) 108 | 109 | output = DinAttentionLayer(32, 64, 0.1)([query, keys, vecs, din_padding_mask_list]) 110 | print("output.shape: ", output.shape) 111 | -------------------------------------------------------------------------------- /AFM/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | # Data format : 4 | 5 | ```python 6 | 说明： 7 | 8 | （1）第1列：user id； 9 | （2）第2列：user gender id； 10 | （3）第3列：user age id； 11 | （4）第4列：user occupation id； 12 | （5）第5列：user zip id； 13 | （6）第6列：item id； 14 | （7）第7列：label； 15 | 16 | ``` 17 | 18 | 19 | # run model 20 | ```shell 21 | sh master.sh 22 | 23 | ``` 24 | 25 | 26 | # 模型 summary 27 | ```python 28 | __________________________________________________________________________________________________ 29 | Layer (type) Output Shape Param # Connected to 30 | ================================================================================================== 31 | user_id_input_layer (InputLayer [(None, 1)] 0 32 | __________________________________________________________________________________________________ 33 | gender_input_layer (InputLayer) [(None, 1)] 0 34 | __________________________________________________________________________________________________ 35 | age_input_layer (InputLayer) [(None, 1)] 0 36 | __________________________________________________________________________________________________ 37 | occupation_input_layer (InputLa [(None, 1)] 0 38 | __________________________________________________________________________________________________ 39 | zip_input_layer (InputLayer) [(None, 1)] 0 40 | __________________________________________________________________________________________________ 41 | item_input_layer (InputLayer) [(None, 1)] 0 42 | __________________________________________________________________________________________________ 43 | user_id_embedding_layer (Embedd (None, 1, 64) 386624 user_id_input_layer[0][0] 44 | __________________________________________________________________________________________________ 45 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 46 | __________________________________________________________________________________________________ 47 | age_embedding_layer (Embedding) (None, 1, 64) 512 age_input_layer[0][0] 48 | __________________________________________________________________________________________________ 49 | occupation_embedding_layer (Emb (None, 1, 64) 1408 occupation_input_layer[0][0] 50 | __________________________________________________________________________________________________ 51 | zip_embedding_layer (Embedding) (None, 1, 64) 220160 zip_input_layer[0][0] 52 | __________________________________________________________________________________________________ 53 | item_id_embedding_layer (Embedd (None, 1, 64) 237248 item_input_layer[0][0] 54 | __________________________________________________________________________________________________ 55 | afm_layer (AFMLayer) (None, 1) 1054 user_id_embedding_layer[0][0] 56 | gender_embedding_layer[0][0] 57 | age_embedding_layer[0][0] 58 | occupation_embedding_layer[0][0] 59 | zip_embedding_layer[0][0] 60 | item_id_embedding_layer[0][0] 61 | __________________________________________________________________________________________________ 62 | tf_op_layer_Sum (TensorFlowOpLa [(None,)] 0 afm_layer[0][0] 63 | __________________________________________________________________________________________________ 64 | tf_op_layer_Sigmoid (TensorFlow [(None,)] 0 tf_op_layer_Sum[0][0] 65 | ================================================================================================== 66 | Total params: 847,198 67 | Trainable params: 847,198 68 | Non-trainable params: 0 69 | __________________________________________________________________________________________________ 70 | None 71 | ``` 72 | 73 | 74 | # 参考 75 | 76 | ```python 77 | 78 | 1. 深度学习推进系统，王喆著 79 | 80 | 2. https://github.com/shenweichen/DeepCTR/blob/master/deepctr/layers/interaction.py 81 | ``` -------------------------------------------------------------------------------- /BilinearFFM/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | # Data format : 4 | 5 | ```python 6 | 说明： 7 | 8 | （1）第1列：user id； 9 | （2）第2列：user gender id； 10 | （3）第3列：user age id； 11 | （4）第4列：user occupation id； 12 | （5）第5列：user zip id； 13 | （6）第6列：item id； 14 | （7）第7列：label； 15 | 16 | ``` 17 | 18 | 19 | # run model 20 | ```shell 21 | sh master.sh 22 | 23 | ``` 24 | 25 | 26 | # 模型 summary 27 | ```python 28 | __________________________________________________________________________________________________ 29 | Layer (type) Output Shape Param # Connected to 30 | ================================================================================================== 31 | user_id_input_layer (InputLayer [(None, 1)] 0 32 | __________________________________________________________________________________________________ 33 | gender_input_layer (InputLayer) [(None, 1)] 0 34 | __________________________________________________________________________________________________ 35 | age_input_layer (InputLayer) [(None, 1)] 0 36 | __________________________________________________________________________________________________ 37 | occupation_input_layer (InputLa [(None, 1)] 0 38 | __________________________________________________________________________________________________ 39 | zip_input_layer (InputLayer) [(None, 1)] 0 40 | __________________________________________________________________________________________________ 41 | item_input_layer (InputLayer) [(None, 1)] 0 42 | __________________________________________________________________________________________________ 43 | user_id_embedding_layer (Embedd (None, 1, 64) 386624 user_id_input_layer[0][0] 44 | __________________________________________________________________________________________________ 45 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 46 | __________________________________________________________________________________________________ 47 | age_embedding_layer (Embedding) (None, 1, 64) 512 age_input_layer[0][0] 48 | __________________________________________________________________________________________________ 49 | occupation_embedding_layer (Emb (None, 1, 64) 1408 occupation_input_layer[0][0] 50 | __________________________________________________________________________________________________ 51 | zip_embedding_layer (Embedding) (None, 1, 64) 220160 zip_input_layer[0][0] 52 | __________________________________________________________________________________________________ 53 | item_id_embedding_layer (Embedd (None, 1, 64) 237248 item_input_layer[0][0] 54 | __________________________________________________________________________________________________ 55 | bilinear_interaction (BilinearI (None, 1, 960) 20480 user_id_embedding_layer[0][0] 56 | gender_embedding_layer[0][0] 57 | age_embedding_layer[0][0] 58 | occupation_embedding_layer[0][0] 59 | zip_embedding_layer[0][0] 60 | item_id_embedding_layer[0][0] 61 | __________________________________________________________________________________________________ 62 | tf_op_layer_Sum (TensorFlowOpLa [(None, 1)] 0 bilinear_interaction[0][0] 63 | __________________________________________________________________________________________________ 64 | tf_op_layer_Sigmoid (TensorFlow [(None, 1)] 0 tf_op_layer_Sum[0][0] 65 | ================================================================================================== 66 | Total params: 866,624 67 | Trainable params: 866,624 68 | Non-trainable params: 0 69 | __________________________________________________________________________________________________ 70 | None 71 | 72 | ``` 73 | 74 | 75 | # 参考 76 | 77 | ```python 78 | 79 | 1. [FFM及DeepFFM模型在推荐系统的探索](https://zhuanlan.zhihu.com/p/67795161) 80 | 81 | 2. https://github.com/shenweichen/DeepCTR/blob/master/deepctr/layers/interaction.py 82 | ``` -------------------------------------------------------------------------------- /Wide&Deep/wide_and_deep.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from tensorflow.keras.models import Model 4 | from tensorflow.keras.layers import Input, Embedding, Dense, Dropout, concatenate 5 | from tensorflow.keras.optimizers import Adam 6 | from tensorflow.keras.callbacks import EarlyStopping 7 | 8 | 9 | def wide_and_deep( 10 | sparse_feature_list=[], 11 | sparse_feature_vocabulary_size_list=[], 12 | sparse_input_length=1, 13 | 14 | dense_feature_list=[], 15 | dense_input_length=1, 16 | 17 | embedding_dim = 64, 18 | hidden_unit_list = [128, 64], 19 | classification = 3 20 | ): 21 | 22 | 23 | # 1. Input layer 24 | input_layer_dict = {} 25 | input_layer_list = [] 26 | 27 | 28 | # 1.1 sparse feature 29 | for f in sparse_feature_list: 30 | input_layer_dict[f] = Input(shape=(sparse_input_length, ), name=f+"_input_layer") 31 | input_layer_list.append(input_layer_dict[f]) 32 | 33 | 34 | # 1.2 dense feature 35 | for f in dense_feature_list: 36 | row_dense_input_layer = Input(shape=(dense_input_length, ), name=f+"_input_layer") 37 | input_layer_list.append(row_dense_input_layer) 38 | input_layer_dict[f] = tf.expand_dims(row_dense_input_layer, 1) 39 | 40 | 41 | 42 | # 2. Embedding 43 | embedding_layer_dict = {} 44 | 45 | # 2.1 sparse feature embedding 46 | for f, v in zip(sparse_feature_list, sparse_feature_vocabulary_size_list): 47 | embedding_layer_dict[f] = Embedding(v+1, embedding_dim, mask_zero=True, \ 48 | name=f+'_embedding_layer')(input_layer_dict[f]) 49 | 50 | 51 | # 2.2 concat 52 | concat_layer = concatenate([embedding_layer_dict[i] for i in sparse_feature_list] + \ 53 | [input_layer_dict[j] for j in dense_feature_list], \ 54 | axis=-1) 55 | 56 | 57 | 58 | # 3. Linear part 59 | linear_part = Dense(1, activation='linear')(concat_layer) 60 | 61 | 62 | 63 | # 4. Deep part 64 | deep_part = concat_layer 65 | 66 | for i, u in enumerate(hidden_unit_list): 67 | deep_part = Dense(u, activation="relu", name="FC_{0}".format(i+1))(deep_part) 68 | deep_part = Dropout(0.3)(deep_part) 69 | 70 | 71 | 72 | # Output 73 | output = tf.keras.layers.concatenate([linear_part, deep_part], axis=-1) 74 | 75 | # Multi-classification 76 | if classification > 2: 77 | output = Dense(classification, activation="softmax")(output) 78 | 79 | # Binary-classification 80 | else: 81 | output = Dense(1, activation="sigmoid")(output) 82 | 83 | 84 | model = Model(inputs = input_layer_list, \ 85 | outputs = output) 86 | 87 | 88 | return model 89 | 90 | 91 | 92 | if __name__ == "__main__": 93 | 94 | sparse_feature_list = ["user_id", "gender", "age", "item_id"] 95 | sparse_feature_vocabulary_size_list = [100, 2, 10, 500] 96 | dense_feature_list = ["click_count", "sales_count"] 97 | classification = 2 98 | 99 | model = wide_and_deep(sparse_feature_list=sparse_feature_list, \ 100 | sparse_feature_vocabulary_size_list = sparse_feature_vocabulary_size_list, \ 101 | dense_feature_list = dense_feature_list, \ 102 | classification = classification) 103 | 104 | print(model.summary()) 105 | tf.keras.utils.plot_model(model, to_file='wide_and_deep_model.png', show_shapes=True) 106 | 107 | 108 | # Multi-classification 109 | if classification > 2: 110 | model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=['accuracy']) 111 | 112 | # Binary-classification 113 | else: 114 | model.compile(loss="binary_crossentropy", optimizer="adam", metrics=['accuracy']) 115 | 116 | 117 | early_stopping_cb = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) 118 | callbacks = [early_stopping_cb] 119 | 120 | 121 | """ 122 | model.fit(X_train_input, y_train, batch_size=50, epochs=100, 123 | callbacks = callbacks, 124 | validation_data = (X_val_input, y_val)) 125 | 126 | """ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 点击预估模型 2 | 3 | 4 | ## 1. Recall 5 | 6 | | 算法 | 论文 | 公众号或知乎文章介绍 | 7 | | -------- | ----- | ---- | 8 | | Word2vec | [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781v3) | | 9 | | YouTubeNet | [Deep Neural Networks for YouTube Recommendations](https://www.sci-hub.ren/10.1145/2959100.2959190) | [推荐系统召回模型之YouTubeNet](https://mp.weixin.qq.com/s/hiabDQW0qGfgPwiZdiZ_Mg) | 10 | | DSSM | [Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations](https://www.sci-hub.ren/10.1145/3298689.3346996) | [实践DSSM召回模型](https://zhuanlan.zhihu.com/p/136253355) | 11 | | MIND | [Multi-Interest Network with Dynamic Routing for Recommendation at Tmall](https://arxiv.org/abs/1904.08030v1) | [推荐系统召回模型之MIND用户多兴趣网络实践](https://mp.weixin.qq.com/s/Ys4EZw97ulrcBWFdN1OMyQ) | 12 | 13 | 14 | ## 2. Rank 15 | 16 | | 算法 | 论文 | 公众号文章介绍 | 17 | | -------- | ----- | ---- | 18 | | FFM | [Field-aware Factorization Machines for CTR Prediction](https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf) | [FFM算法原理及Bi-FFM算法实现](https://mp.weixin.qq.com/s/T46HbKC-_9yYzVTgl8Fh8w) | 19 | | Wide & Deep | [Wide & Deep Learning for Recommender Systems](https://arxiv.org/abs/1606.07792) | | 20 | | NFM | [Neural Factorization Machines for Sparse Predictive Analytics](https://arxiv.org/pdf/1708.05027.pdf) | [NFM模型理论与实践](https://mp.weixin.qq.com/s/1sWYlzIydiLAPMBnr-a5sQ) | 21 | | AFM | [Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](https://arxiv.org/pdf/1708.04617.pdf) | [注意力机制在深度推荐算法中的应用之AFM模型](https://mp.weixin.qq.com/s/sj5bxwtgiw-SaIItsjbeew) | 22 | | DeepFM | [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://arxiv.org/abs/1703.04247) | [DeepFM实践](https://zhuanlan.zhihu.com/p/137894818) | 23 | | BST | [Behavior sequence transformer for e-commerce recommendation in Alibaba](https://arxiv.org/pdf/1905.06874.pdf) | [Transformer 在美团搜索排序中的实践](https://zhuanlan.zhihu.com/p/161311198) | 24 | 25 | 26 | 27 | ## 3. Multi-Task 28 | 29 | | 算法 | 论文 | 公众号文章介绍 | 30 | | -------- | ----- | ---- | 31 | | ESMM | [Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://arxiv.org/abs/1804.07931) | [ESMM多任务学习算法在推荐系统中的应用](https://mp.weixin.qq.com/s/x521rMWLf6CLk0e2uXEJng) | 32 | | MMoE | [Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts](https://dl.acm.org/doi/10.1145/3219819.3220007) | [多任务学习之MMOE模型](https://mp.weixin.qq.com/s/cBy0Y5xDtkc6PxhF1HNomg) | 33 | 34 | 35 | ## 4. Recall_ANN 36 | 37 | | 算法 | 开源地址 | 公众号文章介绍 | 38 | | -------- | ----- | ---- | 39 | | Annoy | [https://github.com/spotify/annoy](https://github.com/spotify/annoy) | [Annoy最近邻检索技术之 “图片检索”](https://zhuanlan.zhihu.com/p/148819536) | 40 | |Faiss|[https://github.com/facebookresearch/faiss](https://github.com/facebookresearch/faiss)|| 41 | 42 | 43 | 44 | # 代码参考 45 | 46 | > https://github.com/shenweichen/DeepCTR 47 | 48 | > https://github.com/shenweichen/DeepMatch 49 | 50 | 51 | # 待学习及分享 52 | 53 | ## Recall 54 | 55 | [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://arxiv.org/abs/1905.01969v3) 56 | 57 | [Controllable Multi-Interest Framework for Recommendation](https://static.aminer.cn/storage/pdf/arxiv/20/2005/2005.09347.pdf)， 58 | 代码：[https://github.com/THUDM/ComiRec](https://github.com/THUDM/ComiRec) 59 | 60 | ## Pre-Rank 61 | 62 | [COLD: Towards the Next Generation of Pre-Ranking System](https://arxiv.org/abs/2007.16122) 63 | 64 | 65 | ## Rank 66 | 67 | DIN：[Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/abs/1706.06978) 68 | 69 | DIEN：[Deep Interest Evolution Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1809.03672.pdf)， 70 | 代码： [https://github.com/mouna99/dien](https://github.com/mouna99/dien) 71 | 72 | MIMN：[Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://arxiv.org/pdf/1905.09248.pdf) 73 | 74 | Search-based Interest Model：[Search-based User Interest Modeling with Lifelong Sequential 75 | Behavior Data for Click-Through Rate Prediction](https://arxiv.org/pdf/2006.05639.pdf) 76 | 77 | 78 | ## Multi-Task 79 | YouTube，2019: Recommending What Video to Watch Next-A Multitask Ranking System 80 | -------------------------------------------------------------------------------- /AFM/data.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | # https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb 4 | 5 | 6 | #! wget http://files.grouplens.org/datasets/movielens/ml-1m.zip -O ./ml-1m.zip 7 | #! wget https://raw.githubusercontent.com/shenweichen/DeepMatch/master/examples/preprocess.py -O preprocess.py 8 | #! unzip -o ml-1m.zip 9 | 10 | 11 | import pandas as pd 12 | import numpy as np 13 | 14 | from preprocess import gen_data_set, gen_model_input 15 | from sklearn.preprocessing import LabelEncoder 16 | from tensorflow.python.keras import backend as K 17 | from tensorflow.python.keras.models import Model 18 | 19 | 20 | 21 | # 1. Load data 22 | unames = ['user_id','gender','age','occupation','zip'] 23 | user = pd.read_csv('ml-1m/users.dat',sep='::',header=None,names=unames) 24 | 25 | rnames = ['user_id','movie_id','rating','timestamp'] 26 | ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header=None,names=rnames) 27 | 28 | mnames = ['movie_id','title','genres'] 29 | movies = pd.read_csv('ml-1m/movies.dat',sep='::',header=None,names=mnames) 30 | 31 | data = pd.merge(pd.merge(ratings, movies), user) 32 | 33 | 34 | 35 | print(data.shape) 36 | # (1000209, 10) 37 | 38 | 39 | 40 | # 2. Label Encoding for sparse features, 41 | # and process sequence features with `gen_date_set` and `gen_model_input` 42 | 43 | sparse_features = ["movie_id", "user_id", "gender", "age", "occupation", "zip"] 44 | SEQ_LEN = 50 45 | negsample = 0 46 | 47 | 48 | features = ['user_id', 'movie_id', 'gender', 'age', 'occupation', 'zip'] 49 | feature_max_idx = {} 50 | 51 | for feature in features: 52 | lbe = LabelEncoder() 53 | data[feature] = lbe.fit_transform(data[feature]) + 1 54 | feature_max_idx[feature] = data[feature].max() + 1 55 | 56 | user_profile = data[["user_id", "gender", "age", "occupation", "zip"]].drop_duplicates('user_id') 57 | item_profile = data[["movie_id"]].drop_duplicates('movie_id') 58 | 59 | user_profile.set_index("user_id", inplace=True) 60 | user_item_list = data.groupby("user_id")['movie_id'].apply(list) 61 | 62 | train_set, test_set = gen_data_set(data, negsample) 63 | train_model_input, train_label = gen_model_input(train_set, user_profile, SEQ_LEN) 64 | test_model_input, test_label = gen_model_input(test_set, user_profile, SEQ_LEN) 65 | 66 | 67 | 68 | 69 | # 3. Create neg samples 70 | 71 | import random 72 | from tqdm import tqdm 73 | 74 | train_neg_sample_list = [] 75 | test_neg_sample_list = [] 76 | all_movie_list = set(data['movie_id']) 77 | neg_sample_num = 10 78 | 79 | for i in tqdm(range(len(train_label))): 80 | a = set(train_model_input['hist_movie_id'][i] + train_model_input['movie_id'][i]) 81 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 82 | train_neg_sample_list.append(np.array(neg_list)) 83 | 84 | for i in tqdm(range(len(test_label))): 85 | a = set(test_model_input['hist_movie_id'][i] + test_model_input['movie_id'][i]) 86 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 87 | test_neg_sample_list.append(np.array(neg_list)) 88 | 89 | 90 | 91 | 92 | # 4. Write to .txt 93 | 94 | train = open("train.txt", "w") 95 | 96 | for i in range(len(train_label)): 97 | a = train_model_input["user_id"][i] 98 | b = train_model_input["gender"][i] 99 | c = train_model_input["age"][i] 100 | d = train_model_input["occupation"][i] 101 | e = train_model_input["zip"][i] 102 | 103 | h = train_model_input["movie_id"][i] 104 | m = train_neg_sample_list[i] 105 | 106 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t1\n"\ 107 | %(str(a), str(b), str(c), str(d), str(e), str(h))) 108 | 109 | for x_i in m: 110 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t0\n"\ 111 | %(str(a), str(b), str(c), str(d), str(e), str(x_i))) 112 | 113 | train.close() 114 | 115 | 116 | 117 | test = open("test.txt", "w") 118 | 119 | for i in range(len(test_label)): 120 | a = test_model_input["user_id"][i] 121 | b = test_model_input["gender"][i] 122 | c = test_model_input["age"][i] 123 | d = test_model_input["occupation"][i] 124 | e = test_model_input["zip"][i] 125 | 126 | h = test_model_input["movie_id"][i] 127 | m = test_neg_sample_list[i] 128 | 129 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t1\n"\ 130 | %(str(a), str(b), str(c), str(d), str(e), str(h))) 131 | 132 | for x_i in m: 133 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t0\n"\ 134 | %(str(a), str(b), str(c), str(d), str(e), str(x_i))) 135 | test.close() 136 | -------------------------------------------------------------------------------- /BilinearFFM/data.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | # https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb 4 | 5 | 6 | #! wget http://files.grouplens.org/datasets/movielens/ml-1m.zip -O ./ml-1m.zip 7 | #! wget https://raw.githubusercontent.com/shenweichen/DeepMatch/master/examples/preprocess.py -O preprocess.py 8 | #! unzip -o ml-1m.zip 9 | 10 | 11 | import pandas as pd 12 | import numpy as np 13 | 14 | from preprocess import gen_data_set, gen_model_input 15 | from sklearn.preprocessing import LabelEncoder 16 | from tensorflow.python.keras import backend as K 17 | from tensorflow.python.keras.models import Model 18 | 19 | 20 | 21 | # 1. Load data 22 | unames = ['user_id','gender','age','occupation','zip'] 23 | user = pd.read_csv('ml-1m/users.dat',sep='::',header=None,names=unames) 24 | 25 | rnames = ['user_id','movie_id','rating','timestamp'] 26 | ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header=None,names=rnames) 27 | 28 | mnames = ['movie_id','title','genres'] 29 | movies = pd.read_csv('ml-1m/movies.dat',sep='::',header=None,names=mnames) 30 | 31 | data = pd.merge(pd.merge(ratings, movies), user) 32 | 33 | 34 | 35 | print(data.shape) 36 | # (1000209, 10) 37 | 38 | 39 | 40 | # 2. Label Encoding for sparse features, 41 | # and process sequence features with `gen_date_set` and `gen_model_input` 42 | 43 | sparse_features = ["movie_id", "user_id", "gender", "age", "occupation", "zip"] 44 | SEQ_LEN = 50 45 | negsample = 0 46 | 47 | 48 | features = ['user_id', 'movie_id', 'gender', 'age', 'occupation', 'zip'] 49 | feature_max_idx = {} 50 | 51 | for feature in features: 52 | lbe = LabelEncoder() 53 | data[feature] = lbe.fit_transform(data[feature]) + 1 54 | feature_max_idx[feature] = data[feature].max() + 1 55 | 56 | user_profile = data[["user_id", "gender", "age", "occupation", "zip"]].drop_duplicates('user_id') 57 | item_profile = data[["movie_id"]].drop_duplicates('movie_id') 58 | 59 | user_profile.set_index("user_id", inplace=True) 60 | user_item_list = data.groupby("user_id")['movie_id'].apply(list) 61 | 62 | train_set, test_set = gen_data_set(data, negsample) 63 | train_model_input, train_label = gen_model_input(train_set, user_profile, SEQ_LEN) 64 | test_model_input, test_label = gen_model_input(test_set, user_profile, SEQ_LEN) 65 | 66 | 67 | 68 | 69 | # 3. Create neg samples 70 | 71 | import random 72 | from tqdm import tqdm 73 | 74 | train_neg_sample_list = [] 75 | test_neg_sample_list = [] 76 | all_movie_list = set(data['movie_id']) 77 | neg_sample_num = 10 78 | 79 | for i in tqdm(range(len(train_label))): 80 | a = set(train_model_input['hist_movie_id'][i] + train_model_input['movie_id'][i]) 81 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 82 | train_neg_sample_list.append(np.array(neg_list)) 83 | 84 | for i in tqdm(range(len(test_label))): 85 | a = set(test_model_input['hist_movie_id'][i] + test_model_input['movie_id'][i]) 86 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 87 | test_neg_sample_list.append(np.array(neg_list)) 88 | 89 | 90 | 91 | 92 | # 4. Write to .txt 93 | 94 | train = open("train.txt", "w") 95 | 96 | for i in range(len(train_label)): 97 | a = train_model_input["user_id"][i] 98 | b = train_model_input["gender"][i] 99 | c = train_model_input["age"][i] 100 | d = train_model_input["occupation"][i] 101 | e = train_model_input["zip"][i] 102 | 103 | h = train_model_input["movie_id"][i] 104 | m = train_neg_sample_list[i] 105 | 106 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t1\n"\ 107 | %(str(a), str(b), str(c), str(d), str(e), str(h))) 108 | 109 | for x_i in m: 110 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t0\n"\ 111 | %(str(a), str(b), str(c), str(d), str(e), str(x_i))) 112 | 113 | train.close() 114 | 115 | 116 | 117 | test = open("test.txt", "w") 118 | 119 | for i in range(len(test_label)): 120 | a = test_model_input["user_id"][i] 121 | b = test_model_input["gender"][i] 122 | c = test_model_input["age"][i] 123 | d = test_model_input["occupation"][i] 124 | e = test_model_input["zip"][i] 125 | 126 | h = test_model_input["movie_id"][i] 127 | m = test_neg_sample_list[i] 128 | 129 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t1\n"\ 130 | %(str(a), str(b), str(c), str(d), str(e), str(h))) 131 | 132 | for x_i in m: 133 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t0\n"\ 134 | %(str(a), str(b), str(c), str(d), str(e), str(x_i))) 135 | test.close() 136 | -------------------------------------------------------------------------------- /MIND/data.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | # https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb 4 | 5 | 6 | #! wget http://files.grouplens.org/datasets/movielens/ml-1m.zip -O ./ml-1m.zip 7 | #! wget https://raw.githubusercontent.com/shenweichen/DeepMatch/master/examples/preprocess.py -O preprocess.py 8 | #! unzip -o ml-1m.zip 9 | 10 | 11 | import pandas as pd 12 | import numpy as np 13 | 14 | from preprocess import gen_data_set, gen_model_input 15 | from sklearn.preprocessing import LabelEncoder 16 | from tensorflow.python.keras import backend as K 17 | from tensorflow.python.keras.models import Model 18 | 19 | 20 | 21 | # 1. Load data 22 | unames = ['user_id','gender','age','occupation','zip'] 23 | user = pd.read_csv('ml-1m/users.dat',sep='::',header=None,names=unames) 24 | 25 | rnames = ['user_id','movie_id','rating','timestamp'] 26 | ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header=None,names=rnames) 27 | 28 | mnames = ['movie_id','title','genres'] 29 | movies = pd.read_csv('ml-1m/movies.dat',sep='::',header=None,names=mnames) 30 | 31 | data = pd.merge(pd.merge(ratings, movies), user) 32 | 33 | 34 | 35 | print(data.shape) 36 | # (1000209, 10) 37 | 38 | 39 | 40 | # 2. Label Encoding for sparse features, 41 | # and process sequence features with `gen_date_set` and `gen_model_input` 42 | 43 | sparse_features = ["movie_id", "user_id", "gender", "age", "occupation", "zip"] 44 | SEQ_LEN = 50 45 | negsample = 0 46 | 47 | 48 | features = ['user_id', 'movie_id', 'gender', 'age', 'occupation', 'zip'] 49 | feature_max_idx = {} 50 | 51 | for feature in features: 52 | lbe = LabelEncoder() 53 | data[feature] = lbe.fit_transform(data[feature]) + 1 54 | feature_max_idx[feature] = data[feature].max() + 1 55 | 56 | user_profile = data[["user_id", "gender", "age", "occupation", "zip"]].drop_duplicates('user_id') 57 | item_profile = data[["movie_id"]].drop_duplicates('movie_id') 58 | 59 | user_profile.set_index("user_id", inplace=True) 60 | user_item_list = data.groupby("user_id")['movie_id'].apply(list) 61 | 62 | train_set, test_set = gen_data_set(data, negsample) 63 | train_model_input, train_label = gen_model_input(train_set, user_profile, SEQ_LEN) 64 | test_model_input, test_label = gen_model_input(test_set, user_profile, SEQ_LEN) 65 | 66 | 67 | 68 | 69 | # 3. Create neg samples 70 | 71 | import random 72 | from tqdm import tqdm 73 | 74 | train_neg_sample_list = [] 75 | test_neg_sample_list = [] 76 | all_movie_list = set(data['movie_id']) 77 | neg_sample_num = 10 78 | 79 | for i in tqdm(range(len(train_label))): 80 | a = set(train_model_input['hist_movie_id'][i] + train_model_input['movie_id'][i]) 81 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 82 | train_neg_sample_list.append(np.array(neg_list)) 83 | 84 | for i in tqdm(range(len(test_label))): 85 | a = set(test_model_input['hist_movie_id'][i] + test_model_input['movie_id'][i]) 86 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 87 | test_neg_sample_list.append(np.array(neg_list)) 88 | 89 | 90 | 91 | 92 | # 4. Write to .txt 93 | 94 | train = open("train.txt", "w") 95 | 96 | for i in range(len(train_label)): 97 | a = train_model_input["user_id"][i] 98 | b = train_model_input["gender"][i] 99 | c = train_model_input["age"][i] 100 | d = train_model_input["occupation"][i] 101 | e = train_model_input["zip"][i] 102 | f = train_model_input["hist_movie_id"][i] 103 | g = train_model_input["hist_len"][i] 104 | 105 | h = train_model_input["movie_id"][i] 106 | m = train_neg_sample_list[i] 107 | 108 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n"\ 109 | %(str(a), str(b), str(c), str(d), str(e), ','.join([str(ii) for ii in f]), str(g), str(h), ','.join([str(ii) for ii in m]))) 110 | 111 | train.close() 112 | 113 | 114 | 115 | test = open("test.txt", "w") 116 | 117 | for i in range(len(test_label)): 118 | a = test_model_input["user_id"][i] 119 | b = test_model_input["gender"][i] 120 | c = test_model_input["age"][i] 121 | d = test_model_input["occupation"][i] 122 | e = test_model_input["zip"][i] 123 | f = test_model_input["hist_movie_id"][i] 124 | g = test_model_input["hist_len"][i] 125 | 126 | h = test_model_input["movie_id"][i] 127 | m = test_neg_sample_list[i] 128 | 129 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n"\ 130 | %(str(a), str(b), str(c), str(d), str(e), ','.join([str(ii) for ii in f]), str(g), str(h), ','.join([str(ii) for ii in m]))) 131 | 132 | test.close() 133 | -------------------------------------------------------------------------------- /YouTubeNet/data.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | # https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb 4 | 5 | 6 | #! wget http://files.grouplens.org/datasets/movielens/ml-1m.zip -O ./ml-1m.zip 7 | #! wget https://raw.githubusercontent.com/shenweichen/DeepMatch/master/examples/preprocess.py -O preprocess.py 8 | #! unzip -o ml-1m.zip 9 | 10 | 11 | import pandas as pd 12 | import numpy as np 13 | 14 | from preprocess import gen_data_set, gen_model_input 15 | from sklearn.preprocessing import LabelEncoder 16 | from tensorflow.python.keras import backend as K 17 | from tensorflow.python.keras.models import Model 18 | 19 | 20 | 21 | # 1. Load data 22 | unames = ['user_id','gender','age','occupation','zip'] 23 | user = pd.read_csv('ml-1m/users.dat',sep='::',header=None,names=unames) 24 | 25 | rnames = ['user_id','movie_id','rating','timestamp'] 26 | ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header=None,names=rnames) 27 | 28 | mnames = ['movie_id','title','genres'] 29 | movies = pd.read_csv('ml-1m/movies.dat',sep='::',header=None,names=mnames) 30 | 31 | data = pd.merge(pd.merge(ratings, movies), user) 32 | 33 | 34 | 35 | print(data.shape) 36 | # (1000209, 10) 37 | 38 | 39 | 40 | # 2. Label Encoding for sparse features, 41 | # and process sequence features with `gen_date_set` and `gen_model_input` 42 | 43 | sparse_features = ["movie_id", "user_id", "gender", "age", "occupation", "zip"] 44 | SEQ_LEN = 50 45 | negsample = 0 46 | 47 | 48 | features = ['user_id', 'movie_id', 'gender', 'age', 'occupation', 'zip'] 49 | feature_max_idx = {} 50 | 51 | for feature in features: 52 | lbe = LabelEncoder() 53 | data[feature] = lbe.fit_transform(data[feature]) + 1 54 | feature_max_idx[feature] = data[feature].max() + 1 55 | 56 | user_profile = data[["user_id", "gender", "age", "occupation", "zip"]].drop_duplicates('user_id') 57 | item_profile = data[["movie_id"]].drop_duplicates('movie_id') 58 | 59 | user_profile.set_index("user_id", inplace=True) 60 | user_item_list = data.groupby("user_id")['movie_id'].apply(list) 61 | 62 | train_set, test_set = gen_data_set(data, negsample) 63 | train_model_input, train_label = gen_model_input(train_set, user_profile, SEQ_LEN) 64 | test_model_input, test_label = gen_model_input(test_set, user_profile, SEQ_LEN) 65 | 66 | 67 | 68 | 69 | # 3. Create neg samples 70 | 71 | import random 72 | from tqdm import tqdm 73 | 74 | train_neg_sample_list = [] 75 | test_neg_sample_list = [] 76 | all_movie_list = set(data['movie_id']) 77 | neg_sample_num = 10 78 | 79 | for i in tqdm(range(len(train_label))): 80 | a = set(train_model_input['hist_movie_id'][i] + train_model_input['movie_id'][i]) 81 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 82 | train_neg_sample_list.append(np.array(neg_list)) 83 | 84 | for i in tqdm(range(len(test_label))): 85 | a = set(test_model_input['hist_movie_id'][i] + test_model_input['movie_id'][i]) 86 | neg_list = random.sample(list(all_movie_list - a), neg_sample_num) 87 | test_neg_sample_list.append(np.array(neg_list)) 88 | 89 | 90 | 91 | 92 | # 4. Write to .txt 93 | 94 | train = open("train.txt", "w") 95 | 96 | for i in range(len(train_label)): 97 | a = train_model_input["user_id"][i] 98 | b = train_model_input["gender"][i] 99 | c = train_model_input["age"][i] 100 | d = train_model_input["occupation"][i] 101 | e = train_model_input["zip"][i] 102 | f = train_model_input["hist_movie_id"][i] 103 | g = train_model_input["hist_len"][i] 104 | 105 | h = train_model_input["movie_id"][i] 106 | m = train_neg_sample_list[i] 107 | 108 | train.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n"\ 109 | %(str(a), str(b), str(c), str(d), str(e), ','.join([str(ii) for ii in f]), str(g), str(h), ','.join([str(ii) for ii in m]))) 110 | 111 | train.close() 112 | 113 | 114 | 115 | test = open("test.txt", "w") 116 | 117 | for i in range(len(test_label)): 118 | a = test_model_input["user_id"][i] 119 | b = test_model_input["gender"][i] 120 | c = test_model_input["age"][i] 121 | d = test_model_input["occupation"][i] 122 | e = test_model_input["zip"][i] 123 | f = test_model_input["hist_movie_id"][i] 124 | g = test_model_input["hist_len"][i] 125 | 126 | h = test_model_input["movie_id"][i] 127 | m = test_neg_sample_list[i] 128 | 129 | test.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n"\ 130 | %(str(a), str(b), str(c), str(d), str(e), ','.join([str(ii) for ii in f]), str(g), str(h), ','.join([str(ii) for ii in m]))) 131 | 132 | test.close() 133 | -------------------------------------------------------------------------------- /ESMM/train_esmm_finetune.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import time 3 | import numpy as np 4 | import tensorflow as tf 5 | from tensorflow.keras.layers import Input, Dense, Lambda, Activation, Multiply, Dot 6 | from tensorflow.keras.models import Model 7 | from tensorflow.keras.callbacks import EarlyStopping, TensorBoard 8 | 9 | 10 | def parse_example(proto): 11 | desc = {\ 12 | 'pin_vec' : tf.io.FixedLenFeature([128], tf.float32), \ 13 | 'sku_vec' : tf.io.FixedLenFeature([128], tf.float32), \ 14 | 'ctr_label' : tf.io.FixedLenFeature([], tf.int64), \ 15 | 'ctcvr_label' : tf.io.FixedLenFeature([], tf.int64), \ 16 | } 17 | 18 | example = tf.io.parse_single_example(proto, desc) 19 | 20 | pin_vec = example['pin_vec'] 21 | sku_vec = example['sku_vec'] 22 | ctr_label = example['ctr_label'] 23 | ctcvr_label = example['ctcvr_label'] 24 | 25 | return (pin_vec, sku_vec), (ctr_label, ctcvr_label) 26 | 27 | 28 | def get_tfrecord_dataset(tf_path, batch_size = None, num_parallel_calls = None): 29 | dataset = tf.data.TFRecordDataset(tf_path, compression_type = "ZLIB") 30 | print('type(dataset)', type(dataset)) 31 | #dataset = dataset.repeat.shuffle(buffer_size = 1024)\ 32 | dataset = dataset.repeat()\ 33 | .map(parse_example, num_parallel_calls = num_parallel_calls)\ 34 | .batch(batch_size) 35 | 36 | return dataset 37 | 38 | 39 | 40 | def train_finetune(train_path, val_path, model_path, \ 41 | n_train = None, \ 42 | n_val = None): 43 | 44 | model = build_model() 45 | print(model.summary()) 46 | 47 | 48 | batch_size = 128 49 | epochs = 100 50 | 51 | train_steps_per_epoch = int(n_train / batch_size) 52 | val_steps_per_epoch = int(n_val / batch_size) 53 | 54 | num_parallel_calls = 4 55 | 56 | train_tfrecord_dataset = get_tfrecord_dataset(\ 57 | train_path, \ 58 | batch_size = batch_size, \ 59 | num_parallel_calls = num_parallel_calls) 60 | 61 | val_tfrecord_dataset = get_tfrecord_dataset(\ 62 | val_path, \ 63 | batch_size = batch_size, \ 64 | num_parallel_calls = num_parallel_calls) 65 | 66 | 67 | early_stopping_cb = EarlyStopping(monitor = 'val_loss', patience = 10, restore_best_weights = True) 68 | 69 | tensorboard_cb = TensorBoard(\ 70 | log_dir = './logs', \ 71 | histogram_freq = 0, \ 72 | write_graph = True, \ 73 | write_grads = True, \ 74 | write_images = True) 75 | 76 | 77 | 78 | callbacks = [early_stopping_cb, tensorboard_cb] 79 | 80 | 81 | start = time.time() 82 | 83 | history = model.fit(\ 84 | train_tfrecord_dataset, \ 85 | steps_per_epoch = train_steps_per_epoch, \ 86 | epochs = epochs, \ 87 | verbose = 1, \ 88 | callbacks = callbacks, \ 89 | validation_data = val_tfrecord_dataset, \ 90 | validation_steps = val_steps_per_epoch, \ 91 | max_queue_size = 10, \ 92 | workers = 1, \ 93 | use_multiprocessing = False, \ 94 | shuffle = True, \ 95 | initial_epoch = 0) 96 | 97 | model.save_weights(model_path) 98 | 99 | last = time.time() - start 100 | print("Train model to %s done! Lasts %.2fs" % (model_path, last)) 101 | 102 | 103 | 104 | 105 | def build_model(): 106 | n_pin_vec = 128 107 | n_sku_vec = 128 108 | 109 | pin_vec = Input(shape=(n_pin_vec, ), dtype = 'float32') 110 | sku_vec = Input(shape=(n_sku_vec, ), dtype = 'float32') 111 | 112 | # ctr_part 113 | ctr_pin_part = Dense(64, activation='relu')(pin_vec) 114 | ctr_sku_part = Dense(64, activation='relu')(sku_vec) 115 | 116 | ctr_prod = Multiply()([ctr_pin_part, ctr_sku_part]) 117 | ctr_prob = Dense(1, activation='sigmoid', name='ctr_prob')(ctr_prod) 118 | 119 | 120 | # ctcvr_part 121 | cvr_pin_part = Dense(64, activation='relu')(pin_vec) 122 | cvr_sku_part = Dense(64, activation='relu')(sku_vec) 123 | 124 | cvr_prod = Multiply()([cvr_pin_part, cvr_sku_part]) 125 | cvr_prob = Dense(1, activation='sigmoid', name='cvr_prob')(cvr_prod) 126 | 127 | #ctcvr_prob = ctr_prob * cvr_prob 128 | ctcvr_prob = Multiply(name = 'ctcvr_prob')([ctr_prob, cvr_prob]) 129 | 130 | model = Model(inputs = [pin_vec, sku_vec], outputs = [ctr_prob, ctcvr_prob]) 131 | 132 | 133 | model.compile(optimizer = 'adam', \ 134 | loss = {'ctr_prob' : 'binary_crossentropy', 'ctcvr_prob' : 'binary_crossentropy'}, \ 135 | #metrics = {'ctr_prob' : 'accuracy', 'ctcvr_prob' : 'accuracy'}) 136 | metrics = ['accuracy']) 137 | 138 | return model 139 | 140 | 141 | 142 | if __name__ == '__main__': 143 | 144 | train_path = sys.argv[1] 145 | val_path = sys.argv[2] 146 | model_path = sys.argv[3] 147 | 148 | train_val_summary_path = sys.argv[4] 149 | 150 | n_train = 0 151 | n_val = 0 152 | fr = open(train_val_summary_path, 'r') 153 | for line in fr: 154 | buf = line[:-1].split(',') 155 | n_train = int(buf[0].split('=')[1]) 156 | n_val = int(buf[1].split('=')[1]) 157 | break 158 | fr.close() 159 | 160 | train_finetune(train_path, val_path, model_path, \ 161 | n_train = n_train, \ 162 | n_val = n_val) 163 | 164 | 165 | -------------------------------------------------------------------------------- /YouTubeNet/YouTubeNet.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | 4 | import tensorflow as tf 5 | from tensorflow.keras.layers import Input, Embedding, concatenate, Dense, Dropout 6 | 7 | from tensorflow.keras.models import Model 8 | from tensorflow.keras.optimizers import Adam 9 | from tensorflow.keras.callbacks import EarlyStopping 10 | from SequencePoolingLayer import SequencePoolingLayer 11 | 12 | 13 | def YouTubeNet( 14 | sparse_input_length=1, 15 | dense_input_length=1, 16 | sparse_seq_input_length=50, 17 | 18 | embedding_dim = 64, 19 | neg_sample_num = 10, 20 | user_hidden_unit_list = [128, 64] 21 | ): 22 | 23 | # 1. Input layer 24 | user_id_input_layer = Input(shape=(sparse_input_length, ), name="user_id_input_layer") 25 | gender_input_layer = Input(shape=(sparse_input_length, ), name="gender_input_layer") 26 | age_input_layer = Input(shape=(sparse_input_length, ), name="age_input_layer") 27 | occupation_input_layer = Input(shape=(sparse_input_length, ), name="occupation_input_layer") 28 | zip_input_layer = Input(shape=(sparse_input_length, ), name="zip_input_layer") 29 | 30 | 31 | user_click_item_seq_input_layer = Input(shape=(sparse_seq_input_length, ), name="user_click_item_seq_input_layer") 32 | user_click_item_seq_length_input_layer = Input(shape=(sparse_input_length, ), name="user_click_item_seq_length_input_layer") 33 | 34 | 35 | pos_item_sample_input_layer = Input(shape=(sparse_input_length, ), name="pos_item_sample_input_layer") 36 | neg_item_sample_input_layer = Input(shape=(neg_sample_num, ), name="neg_item_sample_input_layer") 37 | 38 | 39 | 40 | # 2. Embedding layer 41 | user_id_embedding_layer = Embedding(6040+1, embedding_dim, mask_zero=True, name='user_id_embedding_layer')(user_id_input_layer) 42 | gender_embedding_layer = Embedding(2+1, embedding_dim, mask_zero=True, name='gender_embedding_layer')(gender_input_layer) 43 | age_embedding_layer = Embedding(7+1, embedding_dim, mask_zero=True, name='age_embedding_layer')(age_input_layer) 44 | occupation_embedding_layer = Embedding(21+1, embedding_dim, mask_zero=True, name='occupation_embedding_layer')(occupation_input_layer) 45 | zip_embedding_layer = Embedding(3439+1, embedding_dim, mask_zero=True, name='zip_embedding_layer')(zip_input_layer) 46 | 47 | item_id_embedding_layer = Embedding(3706+1, embedding_dim, mask_zero=True, name='item_id_embedding_layer') 48 | pos_item_sample_embedding_layer = item_id_embedding_layer(pos_item_sample_input_layer) 49 | neg_item_sample_embedding_layer = item_id_embedding_layer(neg_item_sample_input_layer) 50 | 51 | user_click_item_seq_embedding_layer = item_id_embedding_layer(user_click_item_seq_input_layer) 52 | #user_click_item_seq_embedding_layer = SequencePoolingLayer(sequence_mask_length=sparse_seq_input_length)\ 53 | # ([user_click_item_seq_embedding_layer, user_click_item_seq_length_input_layer]) 54 | 55 | user_click_item_seq_embedding_layer = tf.reduce_mean(user_click_item_seq_embedding_layer, 1, keepdims=True) 56 | 57 | 58 | 59 | ### ********** ### 60 | # user part 61 | ### ********** ### 62 | 63 | # 3. Concat "sparse" embedding & "sparse_seq" embedding 64 | user_embedding_layer = concatenate([user_id_embedding_layer, gender_embedding_layer, age_embedding_layer, 65 | occupation_embedding_layer, zip_embedding_layer, user_click_item_seq_embedding_layer], 66 | axis=-1) 67 | 68 | 69 | for i, u in enumerate(user_hidden_unit_list): 70 | user_embedding_layer = Dense(u, activation="relu", name="FC_{0}".format(i+1))(user_embedding_layer) 71 | #user_embedding_layer = Dropout(0.3)(user_embedding_layer) 72 | 73 | 74 | ### ********** ### 75 | # item part 76 | ### ********** ### 77 | 78 | item_embedding_layer = concatenate([pos_item_sample_embedding_layer, neg_item_sample_embedding_layer], \ 79 | axis=1) 80 | 81 | item_embedding_layer = tf.transpose(item_embedding_layer, [0,2,1]) 82 | 83 | 84 | 85 | # Output 86 | dot_output = tf.matmul(user_embedding_layer, item_embedding_layer) 87 | dot_output = tf.nn.softmax(dot_output) # 输出11个值，index为0的值是正样本，负样本的索引位置为[1-10] 88 | 89 | user_inputs_list = [user_id_input_layer, gender_input_layer, age_input_layer, \ 90 | occupation_input_layer, zip_input_layer, \ 91 | user_click_item_seq_input_layer, user_click_item_seq_length_input_layer] 92 | 93 | item_inputs_list = [pos_item_sample_input_layer, neg_item_sample_input_layer] 94 | 95 | model = Model(inputs = user_inputs_list + item_inputs_list, 96 | outputs = dot_output) 97 | 98 | #print(model.summary()) 99 | #tf.keras.utils.plot_model(model, to_file='YouTubeNet_model.png', show_shapes=True) 100 | 101 | 102 | 103 | model.__setattr__("user_input", user_inputs_list) 104 | model.__setattr__("user_embedding", user_embedding_layer) 105 | 106 | model.__setattr__("item_input", pos_item_sample_input_layer) 107 | model.__setattr__("item_embedding", pos_item_sample_embedding_layer) 108 | 109 | return model -------------------------------------------------------------------------------- /AFM/AFMLayer.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.keras.layers import Layer 3 | from tensorflow.keras import backend as K 4 | import itertools 5 | from tensorflow.keras.initializers import (Zeros, glorot_normal, glorot_uniform) 6 | from tensorflow.keras.layers import Concatenate 7 | from tensorflow.keras.regularizers import l2 8 | 9 | 10 | 11 | class AFMLayer(Layer): 12 | """Attentonal Factorization Machine models pairwise (order-2) feature 13 | interactions without linear term and bias. 14 | 15 | Input shape 16 | - A list of 3D tensor with shape: ``(batch_size,1,embedding_size)``. 17 | 18 | Output shape 19 | - 2D tensor with shape: ``(batch_size, 1)``. 20 | 21 | Arguments 22 | - **attention_factor** : Positive integer, dimensionality of the 23 | attention network output space. 24 | 25 | - **l2_reg_w** : float between 0 and 1. L2 regularizer strength 26 | applied to attention network. 27 | 28 | - **dropout_rate** : float between in [0,1). Fraction of the attention net output units to dropout. 29 | 30 | - **seed** : A Python integer to use as random seed. 31 | 32 | References 33 | - [Attentional Factorization Machines : Learning the Weight of Feature 34 | Interactions via Attention Networks](https://arxiv.org/pdf/1708.04617.pdf) 35 | """ 36 | 37 | def __init__(self, attention_factor=4, l2_reg_w=0, dropout_rate=0, seed=1024, **kwargs): 38 | self.attention_factor = attention_factor 39 | self.l2_reg_w = l2_reg_w 40 | self.dropout_rate = dropout_rate 41 | self.seed = seed 42 | super(AFMLayer, self).__init__(**kwargs) 43 | 44 | def build(self, input_shape): 45 | 46 | if not isinstance(input_shape, list) or len(input_shape) < 2: 47 | raise ValueError('A `AttentionalFM` layer should be called ' 48 | 'on a list of at least 2 inputs') 49 | 50 | shape_set = set() 51 | reduced_input_shape = [shape.as_list() for shape in input_shape] 52 | for i in range(len(input_shape)): 53 | shape_set.add(tuple(reduced_input_shape[i])) 54 | 55 | if len(shape_set) > 1: 56 | raise ValueError('A `AttentionalFM` layer requires ' 57 | 'inputs with same shapes ' 58 | 'Got different shapes: %s' % (shape_set)) 59 | 60 | if len(input_shape[0]) != 3 or input_shape[0][1] != 1: 61 | raise ValueError('A `AttentionalFM` layer requires ' 62 | 'inputs of a list with same shape tensor like\ 63 | (None, 1, embedding_size)' 64 | 'Got different shapes: %s' % (input_shape[0])) 65 | 66 | embedding_size = int(input_shape[0][-1]) 67 | 68 | self.attention_W = self.add_weight(shape=(embedding_size, 69 | self.attention_factor), initializer=glorot_normal(seed=self.seed), 70 | regularizer=l2(self.l2_reg_w), name="attention_W") 71 | self.attention_b = self.add_weight( 72 | shape=(self.attention_factor,), initializer=Zeros(), name="attention_b") 73 | self.projection_h = self.add_weight(shape=(self.attention_factor, 1), 74 | initializer=glorot_normal(seed=self.seed), name="projection_h") 75 | self.projection_p = self.add_weight(shape=( 76 | embedding_size, 1), initializer=glorot_normal(seed=self.seed), name="projection_p") 77 | self.dropout = tf.keras.layers.Dropout( 78 | self.dropout_rate, seed=self.seed) 79 | 80 | self.tensordot = tf.keras.layers.Lambda( 81 | lambda x: tf.tensordot(x[0], x[1], axes=(-1, 0))) 82 | 83 | # Be sure to call this somewhere! 84 | super(AFMLayer, self).build(input_shape) 85 | 86 | def call(self, inputs, training=None, **kwargs): 87 | 88 | if K.ndim(inputs[0]) != 3: 89 | raise ValueError( 90 | "Unexpected inputs dimensions %d, expect to be 3 dimensions" % (K.ndim(inputs))) 91 | 92 | embeds_vec_list = inputs 93 | row = [] 94 | col = [] 95 | 96 | for r, c in itertools.combinations(embeds_vec_list, 2): 97 | row.append(r) 98 | col.append(c) 99 | 100 | p = tf.concat(row, axis=1) 101 | q = tf.concat(col, axis=1) 102 | inner_product = p * q 103 | 104 | bi_interaction = inner_product 105 | attention_temp = tf.nn.relu(tf.nn.bias_add(tf.tensordot( 106 | bi_interaction, self.attention_W, axes=(-1, 0)), self.attention_b)) 107 | 108 | self.normalized_att_score = tf.nn.softmax(tf.tensordot( 109 | attention_temp, self.projection_h, axes=(-1, 0))) 110 | attention_output = tf.reduce_sum( 111 | self.normalized_att_score * bi_interaction, axis=1) 112 | 113 | attention_output = self.dropout(attention_output) # training 114 | 115 | afm_out = self.tensordot([attention_output, self.projection_p]) 116 | return afm_out 117 | 118 | def compute_output_shape(self, input_shape): 119 | 120 | if not isinstance(input_shape, list): 121 | raise ValueError('A `AFMLayer` layer should be called ' 122 | 'on a list of inputs.') 123 | return (None, 1) 124 | 125 | def get_config(self, ): 126 | config = {'attention_factor': self.attention_factor, 127 | 'l2_reg_w': self.l2_reg_w, 'dropout_rate': self.dropout_rate, 'seed': self.seed} 128 | base_config = super(AFMLayer, self).get_config() 129 | return dict(list(base_config.items()) + list(config.items())) 130 | 131 | -------------------------------------------------------------------------------- /BST/bst_model.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.keras.layers import Input, Embedding, concatenate, Flatten, Dense, Dropout 3 | 4 | from tensorflow.keras.models import Model 5 | from tensorflow.keras.optimizers import Adam 6 | from tensorflow.keras.callbacks import EarlyStopping 7 | from tensorflow.keras.utils import plot_model 8 | 9 | from transformer import Encoder, padding_mask 10 | from din import DinAttentionLayer, din_padding_mask 11 | 12 | 13 | 14 | def bst_model(sparse_input_length = 1, \ 15 | max_seq_length = 50, \ 16 | vocab_size_dict = None, \ 17 | embedding_dim = 512, \ 18 | dnn_unit_list = [512, 128, 32], \ 19 | activation = 'relu', \ 20 | dropout_rate = 0.2, \ 21 | n_layers = 2, \ 22 | num_heads = 8, \ 23 | middle_units = 1024, \ 24 | training = False 25 | ): 26 | 27 | 28 | # 1. Input layer 29 | 30 | # 1.1 user 31 | user_id_input_layer = Input(shape=(sparse_input_length, ), name="user_id_input_layer") 32 | gender_input_layer = Input(shape=(sparse_input_length, ), name="gender_input_layer") 33 | age_input_layer = Input(shape=(sparse_input_length, ), name="age_input_layer") 34 | 35 | 36 | user_click_item_seq_input_layer = Input(shape=(max_seq_length, ), name="user_click_item_seq_input_layer") 37 | user_click_cate_seq_input_layer = Input(shape=(max_seq_length, ), name="user_click_cate_seq_input_layer") 38 | 39 | 40 | # 1.2 item 41 | item_input_layer = Input(shape=(sparse_input_length, ), name="item_input_layer") 42 | cate_input_layer = Input(shape=(sparse_input_length, ), name="cate_input_layer") 43 | 44 | 45 | 46 | # 2. Embedding layer 47 | 48 | # 2.1 user 49 | user_id_embedding_layer = Embedding(vocab_size_dict["user_id"]+1, embedding_dim, \ 50 | mask_zero=True, name='user_id_embedding_layer')(user_id_input_layer) 51 | gender_embedding_layer = Embedding(vocab_size_dict["gender"]+1, embedding_dim, \ 52 | mask_zero=True, name='gender_embedding_layer')(gender_input_layer) 53 | age_embedding_layer = Embedding(vocab_size_dict["age"]+1, embedding_dim, \ 54 | mask_zero=True, name='age_embedding_layer')(age_input_layer) 55 | 56 | 57 | item_id_embedding = Embedding(vocab_size_dict["item_id"]+1, embedding_dim, \ 58 | mask_zero=True, name='item_id_embedding') 59 | cate_id_embedding = Embedding(vocab_size_dict["cate_id"]+1, embedding_dim, \ 60 | mask_zero=True, name='cate_id_embedding') 61 | 62 | user_click_item_seq_embedding_layer = item_id_embedding(user_click_item_seq_input_layer) 63 | user_click_cate_seq_embedding_layer = cate_id_embedding(user_click_cate_seq_input_layer) 64 | 65 | 66 | # 2.2 item 67 | target_item_embedding_layer = item_id_embedding(item_input_layer) 68 | target_cate_embedding_layer = cate_id_embedding(cate_input_layer) 69 | 70 | 71 | 72 | # 3. Concat layer 73 | 74 | # 3.1 user: other features 75 | other_features_concat_layer = concatenate([user_id_embedding_layer, gender_embedding_layer, \ 76 | age_embedding_layer], axis=-1) 77 | 78 | 79 | # 3.1 user: sequence features 80 | input_transformer_layer = concatenate([user_click_item_seq_embedding_layer, \ 81 | user_click_cate_seq_embedding_layer], axis=-1) 82 | 83 | 84 | # 3.2 item 85 | input_din_layer_query = concatenate([target_item_embedding_layer, \ 86 | target_cate_embedding_layer], axis=-1) 87 | 88 | 89 | # 4. Transformer layer 90 | 91 | d_model = input_transformer_layer.shape[-1] 92 | padding_mask_list = padding_mask(user_click_item_seq_input_layer) 93 | #print("padding_mask_list.shape: ", padding_mask_list.shape) 94 | 95 | output_tranformer_layer = Encoder(n_layers, d_model, num_heads, 96 | middle_units, max_seq_length, training)([input_transformer_layer, padding_mask_list]) 97 | 98 | #print("output_tranformer_layer.shape: ", output_tranformer_layer.shape) 99 | 100 | 101 | 102 | # 5. Din attention layer 103 | 104 | query = input_din_layer_query 105 | keys = output_tranformer_layer 106 | vecs = output_tranformer_layer 107 | 108 | din_padding_mask_list = din_padding_mask(user_click_item_seq_input_layer) 109 | #print("din_padding_mask_list.shape: ", din_padding_mask_list.shape) 110 | 111 | output_din_layer = DinAttentionLayer(d_model, middle_units, dropout_rate)([query, keys, vecs, din_padding_mask_list]) 112 | #print("output_din_layer.shape: ", output_din_layer.shape) 113 | 114 | 115 | 116 | # 6. DNN layer 117 | input_dnn_layer = concatenate([other_features_concat_layer, output_din_layer], \ 118 | axis=-1) 119 | 120 | input_dnn_layer = tf.squeeze(input=input_dnn_layer, axis=[1]) 121 | 122 | 123 | for inx in range(len(dnn_unit_list)): 124 | input_dnn_layer = Dense(dnn_unit_list[inx], activation=activation, \ 125 | name="FC_{0}".format(inx+1))(input_dnn_layer) 126 | 127 | input_dnn_layer = Dropout(dropout_rate, name="dropout_{0}".format(inx+1))(input_dnn_layer) 128 | 129 | 130 | output = Dense(1, activation='sigmoid', \ 131 | name='Sigmoid_output_layer')(input_dnn_layer) 132 | 133 | 134 | 135 | # Output model 136 | 137 | inputs_list = [user_id_input_layer, gender_input_layer, age_input_layer, \ 138 | user_click_item_seq_input_layer, user_click_cate_seq_input_layer, \ 139 | item_input_layer, cate_input_layer] 140 | 141 | model = Model(inputs = inputs_list, outputs = output) 142 | 143 | 144 | return model 145 | 146 | 147 | 148 | if __name__ == "__main__": 149 | vocab_size_dict = { 150 | "user_id": 300, 151 | "gender": 2, 152 | "age": 10, 153 | "item_id": 5000, 154 | "cate_id": 213} 155 | 156 | bst_model = bst_model(vocab_size_dict=vocab_size_dict) 157 | 158 | print(bst_model.summary()) 159 | 160 | plot_model(bst_model, to_file='bst_model.png') 161 | 162 | -------------------------------------------------------------------------------- /DeepFM/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | 4 | # run model 5 | ```python 6 | DeepFM.ipynb 7 | 8 | ``` 9 | 10 | 11 | # 模型 summary 12 | ```python 13 | __________________________________________________________________________________________________ 14 | Layer (type) Output Shape Param # Connected to 15 | ================================================================================================== 16 | Pclass (InputLayer) [(None, 1)] 0 17 | __________________________________________________________________________________________________ 18 | Sex (InputLayer) [(None, 1)] 0 19 | __________________________________________________________________________________________________ 20 | Cabin (InputLayer) [(None, 1)] 0 21 | __________________________________________________________________________________________________ 22 | Embarked (InputLayer) [(None, 1)] 0 23 | __________________________________________________________________________________________________ 24 | embedding (Embedding) (None, 1, 64) 256 Pclass[0][0] 25 | __________________________________________________________________________________________________ 26 | embedding_1 (Embedding) (None, 1, 64) 192 Sex[0][0] 27 | __________________________________________________________________________________________________ 28 | embedding_2 (Embedding) (None, 1, 64) 9536 Cabin[0][0] 29 | __________________________________________________________________________________________________ 30 | embedding_3 (Embedding) (None, 1, 64) 320 Embarked[0][0] 31 | __________________________________________________________________________________________________ 32 | concatenate (Concatenate) (None, 1, 256) 0 embedding[0][0] 33 | embedding_1[0][0] 34 | embedding_2[0][0] 35 | embedding_3[0][0] 36 | __________________________________________________________________________________________________ 37 | Age (InputLayer) [(None, 1)] 0 38 | __________________________________________________________________________________________________ 39 | SibSp (InputLayer) [(None, 1)] 0 40 | __________________________________________________________________________________________________ 41 | Parch (InputLayer) [(None, 1)] 0 42 | __________________________________________________________________________________________________ 43 | Fare (InputLayer) [(None, 1)] 0 44 | __________________________________________________________________________________________________ 45 | concatenate_1 (Concatenate) (None, 4) 0 Age[0][0] 46 | SibSp[0][0] 47 | Parch[0][0] 48 | Fare[0][0] 49 | __________________________________________________________________________________________________ 50 | flatten (Flatten) (None, 256) 0 concatenate[0][0] 51 | __________________________________________________________________________________________________ 52 | concatenate_3 (Concatenate) (None, 260) 0 flatten[0][0] 53 | concatenate_1[0][0] 54 | __________________________________________________________________________________________________ 55 | concatenate_2 (Concatenate) (None, 4, 64) 0 embedding[0][0] 56 | embedding_1[0][0] 57 | embedding_2[0][0] 58 | embedding_3[0][0] 59 | __________________________________________________________________________________________________ 60 | dnn (DNN) (None, 128) 49920 concatenate_3[0][0] 61 | __________________________________________________________________________________________________ 62 | linear (Linear) (None, 1) 5 concatenate[0][0] 63 | concatenate_1[0][0] 64 | __________________________________________________________________________________________________ 65 | fm (FM) (None, 1) 0 concatenate_2[0][0] 66 | __________________________________________________________________________________________________ 67 | dense (Dense) (None, 1) 128 dnn[0][0] 68 | __________________________________________________________________________________________________ 69 | add (Add) (None, 1) 0 linear[0][0] 70 | fm[0][0] 71 | dense[0][0] 72 | __________________________________________________________________________________________________ 73 | dense_1 (Dense) (None, 1) 2 add[0][0] 74 | ================================================================================================== 75 | Total params: 60,359 76 | Trainable params: 60,359 77 | Non-trainable params: 0 78 | __________________________________________________________________________________________________ 79 | None 80 | 81 | ``` 82 | 83 | 84 | # 参考 85 | 86 | ```python 87 | 88 | 1. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction 89 | 90 | 2. https://github.com/shenweichen/DeepCTR 91 | ``` -------------------------------------------------------------------------------- /NFM/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | 4 | # run model 5 | ```python 6 | NFM.ipynb 7 | 8 | ``` 9 | 10 | 11 | # 模型 summary 12 | ```python 13 | __________________________________________________________________________________________________ 14 | Layer (type) Output Shape Param # Connected to 15 | ================================================================================================== 16 | Pclass (InputLayer) [(None, 1)] 0 17 | __________________________________________________________________________________________________ 18 | Sex (InputLayer) [(None, 1)] 0 19 | __________________________________________________________________________________________________ 20 | Cabin (InputLayer) [(None, 1)] 0 21 | __________________________________________________________________________________________________ 22 | Embarked (InputLayer) [(None, 1)] 0 23 | __________________________________________________________________________________________________ 24 | embedding_16 (Embedding) (None, 1, 64) 256 Pclass[0][0] 25 | __________________________________________________________________________________________________ 26 | embedding_17 (Embedding) (None, 1, 64) 192 Sex[0][0] 27 | __________________________________________________________________________________________________ 28 | embedding_18 (Embedding) (None, 1, 64) 9536 Cabin[0][0] 29 | __________________________________________________________________________________________________ 30 | embedding_19 (Embedding) (None, 1, 64) 320 Embarked[0][0] 31 | __________________________________________________________________________________________________ 32 | concatenate_15 (Concatenate) (None, 4, 64) 0 embedding_16[0][0] 33 | embedding_17[0][0] 34 | embedding_18[0][0] 35 | embedding_19[0][0] 36 | __________________________________________________________________________________________________ 37 | bi_interaction_pooling_3 (BiInt (None, 1, 64) 0 concatenate_15[0][0] 38 | __________________________________________________________________________________________________ 39 | Age (InputLayer) [(None, 1)] 0 40 | __________________________________________________________________________________________________ 41 | SibSp (InputLayer) [(None, 1)] 0 42 | __________________________________________________________________________________________________ 43 | Parch (InputLayer) [(None, 1)] 0 44 | __________________________________________________________________________________________________ 45 | Fare (InputLayer) [(None, 1)] 0 46 | __________________________________________________________________________________________________ 47 | dropout_2 (Dropout) (None, 1, 64) 0 bi_interaction_pooling_3[0][0] 48 | __________________________________________________________________________________________________ 49 | concatenate_14 (Concatenate) (None, 4) 0 Age[0][0] 50 | SibSp[0][0] 51 | Parch[0][0] 52 | Fare[0][0] 53 | __________________________________________________________________________________________________ 54 | flatten_2 (Flatten) (None, 64) 0 dropout_2[0][0] 55 | __________________________________________________________________________________________________ 56 | concatenate_16 (Concatenate) (None, 68) 0 flatten_2[0][0] 57 | concatenate_14[0][0] 58 | __________________________________________________________________________________________________ 59 | concatenate_13 (Concatenate) (None, 1, 256) 0 embedding_16[0][0] 60 | embedding_17[0][0] 61 | embedding_18[0][0] 62 | embedding_19[0][0] 63 | __________________________________________________________________________________________________ 64 | dnn_2 (DNN) (None, 128) 25344 concatenate_16[0][0] 65 | __________________________________________________________________________________________________ 66 | linear_4 (Linear) (None, 1) 5 concatenate_13[0][0] 67 | concatenate_14[0][0] 68 | __________________________________________________________________________________________________ 69 | dense_2 (Dense) (None, 1) 128 dnn_2[0][0] 70 | __________________________________________________________________________________________________ 71 | add_2 (Add) (None, 1) 0 linear_4[0][0] 72 | dense_2[0][0] 73 | __________________________________________________________________________________________________ 74 | prediction_layer_2 (PredictionL (None, 1) 1 add_2[0][0] 75 | ================================================================================================== 76 | Total params: 35,782 77 | Trainable params: 35,782 78 | Non-trainable params: 0 79 | __________________________________________________________________________________________________ 80 | None 81 | 82 | ``` 83 | 84 | 85 | # 参考 86 | 87 | ```python 88 | 89 | 1. Neural Factorization Machines for Sparse Predictive Analytics 90 | 91 | 2. https://github.com/shenweichen/DeepCTR 92 | ``` -------------------------------------------------------------------------------- /YouTubeNet/README.md: -------------------------------------------------------------------------------- 1 | # tf.__version__ == '2.1.0' 2 | 3 | 4 | # 运行 5 | ``` shell 6 | sh master.sh 7 | 8 | ``` 9 | 10 | 11 | # YouTubeNet's summary 12 | 13 | ```python 14 | 15 | Model: "model_1" 16 | __________________________________________________________________________________________________ 17 | Layer (type) Output Shape Param # Connected to 18 | ================================================================================================== 19 | user_click_item_seq_input_layer [(None, 50)] 0 20 | __________________________________________________________________________________________________ 21 | user_id_input_layer (InputLayer [(None, 1)] 0 22 | __________________________________________________________________________________________________ 23 | gender_input_layer (InputLayer) [(None, 1)] 0 24 | __________________________________________________________________________________________________ 25 | age_input_layer (InputLayer) [(None, 1)] 0 26 | __________________________________________________________________________________________________ 27 | occupation_input_layer (InputLa [(None, 1)] 0 28 | __________________________________________________________________________________________________ 29 | zip_input_layer (InputLayer) [(None, 1)] 0 30 | __________________________________________________________________________________________________ 31 | item_id_embedding_layer (Embedd multiple 237248 pos_item_sample_input_layer[0][0] 32 | neg_item_sample_input_layer[0][0] 33 | user_click_item_seq_input_layer[0 34 | __________________________________________________________________________________________________ 35 | user_click_item_seq_length_inpu [(None, 1)] 0 36 | __________________________________________________________________________________________________ 37 | user_id_embedding_layer (Embedd (None, 1, 64) 386624 user_id_input_layer[0][0] 38 | __________________________________________________________________________________________________ 39 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 40 | __________________________________________________________________________________________________ 41 | age_embedding_layer (Embedding) (None, 1, 64) 512 age_input_layer[0][0] 42 | __________________________________________________________________________________________________ 43 | occupation_embedding_layer (Emb (None, 1, 64) 1408 occupation_input_layer[0][0] 44 | __________________________________________________________________________________________________ 45 | zip_embedding_layer (Embedding) (None, 1, 64) 220160 zip_input_layer[0][0] 46 | __________________________________________________________________________________________________ 47 | sequence_pooling_layer_2 (Seque (None, 1, 64) 0 item_id_embedding_layer[2][0] 48 | user_click_item_seq_length_input_ 49 | __________________________________________________________________________________________________ 50 | pos_item_sample_input_layer (In [(None, 1)] 0 51 | __________________________________________________________________________________________________ 52 | neg_item_sample_input_layer (In [(None, 10)] 0 53 | __________________________________________________________________________________________________ 54 | concatenate_4 (Concatenate) (None, 1, 384) 0 user_id_embedding_layer[0][0] 55 | gender_embedding_layer[0][0] 56 | age_embedding_layer[0][0] 57 | occupation_embedding_layer[0][0] 58 | zip_embedding_layer[0][0] 59 | sequence_pooling_layer_2[0][0] 60 | __________________________________________________________________________________________________ 61 | FC_1 (Dense) (None, 1, 128) 49280 concatenate_4[0][0] 62 | __________________________________________________________________________________________________ 63 | concatenate_5 (Concatenate) (None, 11, 64) 0 item_id_embedding_layer[0][0] 64 | item_id_embedding_layer[1][0] 65 | __________________________________________________________________________________________________ 66 | FC_2 (Dense) (None, 1, 64) 8256 FC_1[0][0] 67 | __________________________________________________________________________________________________ 68 | tf_op_layer_transpose_2 (Tensor [(None, 64, 11)] 0 concatenate_5[0][0] 69 | __________________________________________________________________________________________________ 70 | tf_op_layer_MatMul_2 (TensorFlo [(None, 1, 11)] 0 FC_2[0][0] 71 | tf_op_layer_transpose_2[0][0] 72 | __________________________________________________________________________________________________ 73 | tf_op_layer_Softmax_2 (TensorFl [(None, 1, 11)] 0 tf_op_layer_MatMul_2[0][0] 74 | ================================================================================================== 75 | Total params: 903,680 76 | Trainable params: 903,680 77 | Non-trainable params: 0 78 | __________________________________________________________________________________________________ 79 | None 80 | 81 | ``` 82 | 83 | 84 | # 运行过程 85 | 86 | ```python 87 | 88 | WARNING:tensorflow:sample_weight modes were coerced from 89 | ... 90 | to 91 | ['...'] 92 | WARNING:tensorflow:sample_weight modes were coerced from 93 | ... 94 | to 95 | ['...'] 96 | Train for 989 steps, validate for 7 steps 97 | Epoch 1/2 98 | 989/989 [==============================] - 62s 63ms/step - loss: 1.7179 - sparse_categorical_accuracy: 0.3706 - val_loss: 1.7052 - val_sparse_categorical_accuracy: 0.3857 99 | Epoch 2/2 100 | 989/989 [==============================] - 58s 58ms/step - loss: 1.5146 - sparse_categorical_accuracy: 0.4323 - val_loss: 1.5995 - val_sparse_categorical_accuracy: 0.4144 101 | ``` 102 | 103 | 104 | # 参考 105 | 106 | [https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb](https://github.com/shenweichen/DeepMatch/blob/master/examples/colab_MovieLen1M_YoutubeDNN.ipynb) 107 | 108 | -------------------------------------------------------------------------------- /MIND/mind.py: -------------------------------------------------------------------------------- 1 | # create model 2 | 3 | import tensorflow as tf 4 | from tensorflow.keras.layers import Input, Embedding, concatenate, Flatten, Dense, Dropout 5 | 6 | from tensorflow.keras.models import Model 7 | from tensorflow.keras.optimizers import Adam 8 | from tensorflow.keras.callbacks import EarlyStopping 9 | 10 | from CapsuleLayer import SequencePoolingLayer, LabelAwareAttention, CapsuleLayer 11 | 12 | 13 | def tile_user_otherfeat(user_other_feature, k_max): 14 | return tf.tile(tf.expand_dims(user_other_feature, -2), [1, k_max, 1]) 15 | 16 | 17 | def mind( 18 | sparse_input_length=1, 19 | dense_input_length=1, 20 | sparse_seq_input_length=50, 21 | 22 | embedding_dim = 64, 23 | neg_sample_num = 10, 24 | user_hidden_unit_list = [128, 64], 25 | k_max = 5, 26 | p = 1, 27 | dynamic_k = True 28 | ): 29 | 30 | 31 | 32 | # 1. Input layer 33 | user_id_input_layer = Input(shape=(sparse_input_length, ), name="user_id_input_layer") 34 | gender_input_layer = Input(shape=(sparse_input_length, ), name="gender_input_layer") 35 | age_input_layer = Input(shape=(sparse_input_length, ), name="age_input_layer") 36 | occupation_input_layer = Input(shape=(sparse_input_length, ), name="occupation_input_layer") 37 | zip_input_layer = Input(shape=(sparse_input_length, ), name="zip_input_layer") 38 | 39 | 40 | user_click_item_seq_input_layer = Input(shape=(sparse_seq_input_length, ), name="user_click_item_seq_input_layer") 41 | user_click_item_seq_length_input_layer = Input(shape=(sparse_input_length, ), name="user_click_item_seq_length_input_layer") 42 | 43 | 44 | pos_item_sample_input_layer = Input(shape=(sparse_input_length, ), name="pos_item_sample_input_layer") 45 | neg_item_sample_input_layer = Input(shape=(neg_sample_num, ), name="neg_item_sample_input_layer") 46 | 47 | 48 | 49 | # 2. Embedding layer 50 | user_id_embedding_layer = Embedding(6040+1, embedding_dim, mask_zero=True, name='user_id_embedding_layer')(user_id_input_layer) 51 | gender_embedding_layer = Embedding(2+1, embedding_dim, mask_zero=True, name='gender_embedding_layer')(gender_input_layer) 52 | age_embedding_layer = Embedding(7+1, embedding_dim, mask_zero=True, name='age_embedding_layer')(age_input_layer) 53 | occupation_embedding_layer = Embedding(21+1, embedding_dim, mask_zero=True, name='occupation_embedding_layer')(occupation_input_layer) 54 | zip_embedding_layer = Embedding(3439+1, embedding_dim, mask_zero=True, name='zip_embedding_layer')(zip_input_layer) 55 | 56 | item_id_embedding_layer = Embedding(3706+1, embedding_dim, mask_zero=True, name='item_id_embedding_layer') 57 | pos_item_sample_embedding_layer = item_id_embedding_layer(pos_item_sample_input_layer) 58 | neg_item_sample_embedding_layer = item_id_embedding_layer(neg_item_sample_input_layer) 59 | 60 | user_click_item_seq_embedding_layer = item_id_embedding_layer(user_click_item_seq_input_layer) 61 | 62 | 63 | 64 | 65 | ### ********** ### 66 | # 3. user part 67 | ### ********** ### 68 | 69 | # 3.1 pooling layer 70 | user_click_item_seq_embedding_layer_pooling = SequencePoolingLayer()\ 71 | ([user_click_item_seq_embedding_layer, user_click_item_seq_length_input_layer]) 72 | 73 | print("user_click_item_seq_embedding_layer_pooling", user_click_item_seq_embedding_layer_pooling) 74 | 75 | 76 | # 3.2 capsule layer 77 | high_capsule = CapsuleLayer(input_units=embedding_dim, 78 | out_units=embedding_dim, max_len=sparse_seq_input_length, 79 | k_max=k_max)\ 80 | ([user_click_item_seq_embedding_layer, user_click_item_seq_length_input_layer]) 81 | 82 | print("high_capsule: ", high_capsule) 83 | 84 | 85 | # 3.3 Concat "sparse" embedding & "sparse_seq" embedding, and tile embedding 86 | other_user_embedding_layer = concatenate([user_id_embedding_layer, gender_embedding_layer, \ 87 | age_embedding_layer, occupation_embedding_layer, \ 88 | zip_embedding_layer, user_click_item_seq_embedding_layer_pooling], 89 | axis=-1) 90 | 91 | 92 | 93 | other_user_embedding_layer = tf.tile(other_user_embedding_layer, [1, k_max, 1]) 94 | 95 | print("other_user_embedding_layer: ", other_user_embedding_layer) 96 | 97 | 98 | 99 | # 3.4 user dnn part 100 | user_deep_input = concatenate([other_user_embedding_layer, high_capsule], axis=-1) 101 | print("user_deep_input: ", user_deep_input) 102 | 103 | 104 | for i, u in enumerate(user_hidden_unit_list): 105 | user_deep_input = Dense(u, activation="relu", name="FC_{0}".format(i+1))(user_deep_input) 106 | #user_deep_input = Dropout(0.3)(user_deep_input) 107 | 108 | print("user_deep_input: ", user_deep_input) 109 | 110 | 111 | if dynamic_k: 112 | user_embedding_final = LabelAwareAttention(k_max=k_max, pow_p=p, )(\ 113 | [user_deep_input, pos_item_sample_embedding_layer, user_click_item_seq_length_input_layer]) 114 | else: 115 | user_embedding_final = LabelAwareAttention(k_max=k_max, pow_p=p, )(\ 116 | [user_deep_input, pos_item_sample_embedding_layer]) 117 | 118 | 119 | user_embedding_final = tf.expand_dims(user_embedding_final, 1) 120 | print("user_embedding_final: ", user_embedding_final) 121 | 122 | 123 | 124 | ### ********** ### 125 | # 4. item part 126 | ### ********** ### 127 | 128 | item_embedding_layer = concatenate([pos_item_sample_embedding_layer, neg_item_sample_embedding_layer], \ 129 | axis=1) 130 | 131 | item_embedding_layer = tf.transpose(item_embedding_layer, [0,2,1]) 132 | 133 | print("item_embedding_layer: ", item_embedding_layer) 134 | 135 | 136 | 137 | 138 | ### ********** ### 139 | # 5. Output 140 | ### ********** ### 141 | 142 | dot_output = tf.matmul(user_embedding_final, item_embedding_layer) 143 | dot_output = tf.nn.softmax(dot_output) # 输出11个值，index为0的值是正样本，负样本的索引位置为[1-10] 144 | 145 | print(dot_output) 146 | 147 | user_inputs_list = [user_id_input_layer, gender_input_layer, age_input_layer, \ 148 | occupation_input_layer, zip_input_layer, \ 149 | user_click_item_seq_input_layer, user_click_item_seq_length_input_layer] 150 | 151 | item_inputs_list = [pos_item_sample_input_layer, neg_item_sample_input_layer] 152 | 153 | model = Model(inputs = user_inputs_list + item_inputs_list, 154 | outputs = dot_output) 155 | 156 | 157 | #print(model.summary()) 158 | #tf.keras.utils.plot_model(model, to_file='MIND_model.png', show_shapes=True) 159 | 160 | 161 | model.__setattr__("user_input", user_inputs_list) 162 | model.__setattr__("user_embedding", user_deep_input) 163 | 164 | model.__setattr__("item_input", pos_item_sample_input_layer) 165 | model.__setattr__("item_embedding", pos_item_sample_embedding_layer) 166 | 167 | return model -------------------------------------------------------------------------------- /BST/README.md: -------------------------------------------------------------------------------- 1 | # tf.version == '2.1.0' 2 | 3 | # 运行 4 | 5 | + python bst_model.py 6 | 7 | 8 | # 模型 summary 9 | 10 | ```python 11 | __________________________________________________________________________________________________ 12 | Layer (type) Output Shape Param # Connected to 13 | ================================================================================================== 14 | user_click_item_seq_input_layer [(None, 50)] 0 15 | __________________________________________________________________________________________________ 16 | item_input_layer (InputLayer) [(None, 1)] 0 17 | __________________________________________________________________________________________________ 18 | cate_input_layer (InputLayer) [(None, 1)] 0 19 | __________________________________________________________________________________________________ 20 | user_click_cate_seq_input_layer [(None, 50)] 0 21 | __________________________________________________________________________________________________ 22 | tf_op_layer_Equal (TensorFlowOp [(None, 50)] 0 user_click_item_seq_input_layer[0 23 | __________________________________________________________________________________________________ 24 | item_id_embedding (Embedding) multiple 2560512 user_click_item_seq_input_layer[0 25 | item_input_layer[0][0] 26 | __________________________________________________________________________________________________ 27 | cate_id_embedding (Embedding) multiple 109568 user_click_cate_seq_input_layer[0 28 | cate_input_layer[0][0] 29 | __________________________________________________________________________________________________ 30 | tf_op_layer_Cast (TensorFlowOpL [(None, 50)] 0 tf_op_layer_Equal[0][0] 31 | __________________________________________________________________________________________________ 32 | tf_op_layer_Equal_1 (TensorFlow [(None, 50)] 0 user_click_item_seq_input_layer[0 33 | __________________________________________________________________________________________________ 34 | user_id_input_layer (InputLayer [(None, 1)] 0 35 | __________________________________________________________________________________________________ 36 | gender_input_layer (InputLayer) [(None, 1)] 0 37 | __________________________________________________________________________________________________ 38 | age_input_layer (InputLayer) [(None, 1)] 0 39 | __________________________________________________________________________________________________ 40 | concatenate_1 (Concatenate) (None, 50, 1024) 0 item_id_embedding[0][0] 41 | cate_id_embedding[0][0] 42 | __________________________________________________________________________________________________ 43 | tf_op_layer_strided_slice (Tens [(None, 1, 1, 50)] 0 tf_op_layer_Cast[0][0] 44 | __________________________________________________________________________________________________ 45 | tf_op_layer_Cast_1 (TensorFlowO [(None, 50)] 0 tf_op_layer_Equal_1[0][0] 46 | __________________________________________________________________________________________________ 47 | user_id_embedding_layer (Embedd (None, 1, 512) 154112 user_id_input_layer[0][0] 48 | __________________________________________________________________________________________________ 49 | gender_embedding_layer (Embeddi (None, 1, 512) 1536 gender_input_layer[0][0] 50 | __________________________________________________________________________________________________ 51 | age_embedding_layer (Embedding) (None, 1, 512) 5632 age_input_layer[0][0] 52 | __________________________________________________________________________________________________ 53 | concatenate_2 (Concatenate) (None, 1, 1024) 0 item_id_embedding[1][0] 54 | cate_id_embedding[1][0] 55 | __________________________________________________________________________________________________ 56 | encoder (Encoder) (None, 50, 1024) 12603392 concatenate_1[0][0] 57 | tf_op_layer_strided_slice[0][0] 58 | __________________________________________________________________________________________________ 59 | tf_op_layer_strided_slice_1 (Te [(None, 1, 50)] 0 tf_op_layer_Cast_1[0][0] 60 | __________________________________________________________________________________________________ 61 | concatenate (Concatenate) (None, 1, 1536) 0 user_id_embedding_layer[0][0] 62 | gender_embedding_layer[0][0] 63 | age_embedding_layer[0][0] 64 | __________________________________________________________________________________________________ 65 | din_attention_layer (DinAttenti (None, 1, 1024) 5244928 concatenate_2[0][0] 66 | encoder[0][0] 67 | encoder[0][0] 68 | tf_op_layer_strided_slice_1[0][0] 69 | __________________________________________________________________________________________________ 70 | concatenate_3 (Concatenate) (None, 1, 2560) 0 concatenate[0][0] 71 | din_attention_layer[0][0] 72 | __________________________________________________________________________________________________ 73 | tf_op_layer_Squeeze (TensorFlow [(None, 2560)] 0 concatenate_3[0][0] 74 | __________________________________________________________________________________________________ 75 | FC_1 (Dense) (None, 512) 1311232 tf_op_layer_Squeeze[0][0] 76 | __________________________________________________________________________________________________ 77 | dropout_1 (Dropout) (None, 512) 0 FC_1[0][0] 78 | __________________________________________________________________________________________________ 79 | FC_2 (Dense) (None, 128) 65664 dropout_1[0][0] 80 | __________________________________________________________________________________________________ 81 | dropout_2 (Dropout) (None, 128) 0 FC_2[0][0] 82 | __________________________________________________________________________________________________ 83 | FC_3 (Dense) (None, 32) 4128 dropout_2[0][0] 84 | __________________________________________________________________________________________________ 85 | dropout_3 (Dropout) (None, 32) 0 FC_3[0][0] 86 | __________________________________________________________________________________________________ 87 | Sigmoid_output_layer (Dense) (None, 1) 33 dropout_3[0][0] 88 | ================================================================================================== 89 | Total params: 22,060,737 90 | Trainable params: 22,060,737 91 | Non-trainable params: 0 92 | 93 | 94 | ``` 95 | 96 | 97 | # 参考 98 | 99 | ```python 100 | 101 | 1. Behavior sequence transformer for e-commerce recommendation in Alibaba 102 | 103 | 2. https://zhuanlan.zhihu.com/p/161311198 104 | 105 | 3. https://github.com/czy36mengfei/tensorflow2_tutorials_chinese/tree/master/026-Transformer 106 | 107 | 4. https://github.com/shenweichen/DeepCTR/blob/master/deepctr/layers/sequence.py 108 | 109 | ``` -------------------------------------------------------------------------------- /BST/transformer.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | from tensorflow import keras 4 | from tensorflow.keras import backend as K 5 | 6 | 7 | # "Attention is all you need" 中的编码模块 8 | 9 | class Encoder(tf.keras.layers.Layer): 10 | def __init__(self, n_layers, d_model, num_heads, middle_units, 11 | max_seq_len, epsilon=1e-6, dropout_rate=0.1, training=False, **kwargs): 12 | super(Encoder, self).__init__(**kwargs) 13 | 14 | self.n_layers = n_layers 15 | self.d_model = d_model 16 | self.pos_embedding = PositionalEncoding(sequence_len=max_seq_len, embedding_dim=d_model) 17 | 18 | self.encode_layer = [EncoderLayer(d_model=d_model, num_heads=num_heads, 19 | middle_units=middle_units, 20 | epsilon=epsilon, dropout_rate=dropout_rate, 21 | training = training) 22 | for _ in range(n_layers)] 23 | 24 | def call(self, inputs, **kwargs): 25 | emb, mask = inputs 26 | emb = self.pos_embedding(emb) 27 | 28 | for i in range(self.n_layers): 29 | emb = self.encode_layer[i](emb, mask) 30 | 31 | return emb 32 | 33 | 34 | 35 | 36 | # 编码层 37 | class EncoderLayer(tf.keras.layers.Layer): 38 | def __init__(self, d_model, num_heads, middle_units, \ 39 | epsilon=1e-6, dropout_rate=0.1, training=False, **kwargs): 40 | super(EncoderLayer, self).__init__(**kwargs) 41 | 42 | self.mha = MultiHeadAttention(d_model, num_heads) 43 | self.ffn = point_wise_feed_forward_network(d_model, middle_units) 44 | 45 | self.layernorm1 = LayerNormalization() 46 | self.layernorm2 = LayerNormalization() 47 | 48 | self.dropout1 = tf.keras.layers.Dropout(dropout_rate) 49 | self.dropout2 = tf.keras.layers.Dropout(dropout_rate) 50 | 51 | self.training = training 52 | 53 | def call(self, inputs, mask, **kwargs): 54 | 55 | # 多头注意力网络 56 | att_output = self.mha([inputs, inputs, inputs, mask]) 57 | att_output = self.dropout1(att_output, training=self.training) 58 | out1 = self.layernorm1(inputs + att_output) # (batch_size, input_seq_len, d_model) 59 | 60 | # 前向网络 61 | ffn_output = self.ffn(out1) 62 | ffn_output = self.dropout2(ffn_output, training=self.training) 63 | out2 = self.layernorm2(out1 + ffn_output) # (batch_size, input_seq_len, d_model) 64 | 65 | return out2 66 | 67 | 68 | 69 | # 层标准化 70 | class LayerNormalization(tf.keras.layers.Layer): 71 | def __init__(self, epsilon=1e-6, **kwargs): 72 | self.eps = epsilon 73 | super(LayerNormalization, self).__init__(**kwargs) 74 | 75 | def build(self, input_shape): 76 | self.gamma = self.add_weight(name='gamma', shape=input_shape[-1:], 77 | initializer=tf.ones_initializer(), trainable=True) 78 | self.beta = self.add_weight(name='beta', shape=input_shape[-1:], 79 | initializer=tf.zeros_initializer(), trainable=True) 80 | super(LayerNormalization, self).build(input_shape) 81 | 82 | def call(self, x): 83 | mean = tf.keras.backend.mean(x, axis=-1, keepdims=True) 84 | std = tf.keras.backend.std(x, axis=-1, keepdims=True) 85 | return self.gamma * (x - mean) / (std + self.eps) + self.beta 86 | 87 | def compute_output_shape(self, input_shape): 88 | return input_shape 89 | 90 | 91 | 92 | 93 | # 前向网络 94 | def point_wise_feed_forward_network(d_model, middle_units): 95 | 96 | return tf.keras.Sequential([ 97 | tf.keras.layers.Dense(middle_units, activation='relu'), 98 | tf.keras.layers.Dense(d_model, activation='relu')]) 99 | 100 | 101 | 102 | # dot attention 103 | def scaled_dot_product_attention(q, k, v, mask): 104 | 105 | matmul_qk = tf.matmul(q, k, transpose_b=True) 106 | dim_k = tf.cast(tf.shape(k)[-1], tf.float32) 107 | scaled_attention_logits = matmul_qk / tf.math.sqrt(dim_k) 108 | 109 | if mask is not None: 110 | scaled_attention_logits += (mask * -1e9) 111 | 112 | attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1) 113 | output = tf.matmul(attention_weights, v) 114 | 115 | return output 116 | 117 | 118 | 119 | # 构造 multi head attention 层 120 | class MultiHeadAttention(tf.keras.layers.Layer): 121 | def __init__(self, d_model, num_heads, **kwargs): 122 | super(MultiHeadAttention, self).__init__(**kwargs) 123 | self.num_heads = num_heads 124 | self.d_model = d_model 125 | 126 | # d_model 必须可以正确分为各个头 127 | assert d_model % num_heads == 0 128 | 129 | # 分头后的维度 130 | self.depth = d_model // num_heads 131 | 132 | self.wq = tf.keras.layers.Dense(d_model) 133 | self.wk = tf.keras.layers.Dense(d_model) 134 | self.wv = tf.keras.layers.Dense(d_model) 135 | 136 | self.dense = tf.keras.layers.Dense(d_model) 137 | 138 | self.dot_attention = scaled_dot_product_attention 139 | 140 | def split_heads(self, x, batch_size): 141 | # 分头, 将头个数的维度 放到 seq_len 前面 142 | x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth)) 143 | return tf.transpose(x, perm=[0, 2, 1, 3]) 144 | 145 | def call(self, inputs, **kwargs): 146 | q, k, v, mask = inputs 147 | batch_size = tf.shape(q)[0] 148 | 149 | # 分头前的前向网络，获取q、k、v语义 150 | q = self.wq(q) # (batch_size, seq_len, d_model) 151 | k = self.wk(k) 152 | v = self.wv(v) 153 | 154 | # 分头 155 | q = self.split_heads(q, batch_size) # (batch_size, num_heads, seq_len_q, depth) 156 | k = self.split_heads(k, batch_size) # (batch_size, num_heads, seq_len_k, depth) 157 | v = self.split_heads(v, batch_size) # (batch_size, num_heads, seq_len_v, depth) 158 | 159 | # 通过缩放点积注意力层 160 | scaled_attention = self.dot_attention(q, k, v, mask) # (batch_size, num_heads, seq_len_q, depth) 161 | 162 | # “多头维度” 后移 163 | scaled_attention = tf.transpose(scaled_attention, [0, 2, 1, 3]) # (batch_size, seq_len_q, num_heads, depth) 164 | 165 | # 合并 “多头维度” 166 | concat_attention = tf.reshape(scaled_attention, (batch_size, -1, self.d_model)) 167 | 168 | # 全连接层 169 | output = self.dense(concat_attention) 170 | 171 | return output 172 | 173 | 174 | 175 | 176 | # mask功能 177 | def padding_mask(seq): 178 | # 获取为 0的padding项 179 | seq = tf.cast(tf.math.equal(seq, 0), tf.float32) 180 | 181 | # 扩充维度用于attention矩阵 182 | return seq[:, np.newaxis, np.newaxis, :] # (batch_size, 1, 1, seq_len) 183 | 184 | 185 | 186 | 187 | # 位置编码 188 | class PositionalEncoding(tf.keras.layers.Layer): 189 | def __init__(self, sequence_len=None, embedding_dim=None, **kwargs): 190 | self.sequence_len = sequence_len 191 | self.embedding_dim = embedding_dim 192 | super(PositionalEncoding, self).__init__(**kwargs) 193 | 194 | def call(self, inputs): 195 | if self.embedding_dim == None: 196 | self.embedding_dim = int(inputs.shape[-1]) 197 | 198 | position_embedding = np.array([ 199 | [pos / np.power(10000, 2. * i / self.embedding_dim) for i in range(self.embedding_dim)] 200 | for pos in range(self.sequence_len)]) 201 | 202 | position_embedding[:, 0::2] = np.sin(position_embedding[:, 0::2]) # dim 2i 203 | position_embedding[:, 1::2] = np.cos(position_embedding[:, 1::2]) # dim 2i+1 204 | 205 | position_embedding = tf.cast(position_embedding, dtype=tf.float32) 206 | 207 | return position_embedding + inputs 208 | 209 | 210 | def compute_output_shape(self, input_shape): 211 | return input_shape 212 | 213 | 214 | 215 | 216 | if __name__ == "__main__": 217 | 218 | n_layers = 2 219 | d_model = 512 220 | num_heads = 8 221 | middle_units = 1024 222 | max_seq_len = 60 223 | 224 | samples = 10 225 | training = False 226 | 227 | encode_padding_mask_list = padding_mask(np.random.randint(0, 108, size=(samples, max_seq_len))) 228 | input_data = tf.random.uniform((samples, max_seq_len, d_model)) 229 | 230 | sample_encoder = Encoder(n_layers, d_model, num_heads, middle_units, max_seq_len, training) 231 | sample_encoder_output = sample_encoder([input_data, encode_padding_mask_list]) 232 | 233 | print(sample_encoder_output.shape) -------------------------------------------------------------------------------- /MIND/CapsuleLayer.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from tensorflow.keras.layers import Layer 4 | from tensorflow.keras.initializers import RandomNormal, Zeros 5 | 6 | 7 | class SequencePoolingLayer(Layer): 8 | """The SequencePoolingLayer is used to apply pooling operation(sum,mean,max) on variable-length sequence feature/multi-value feature. 9 | 10 | Input shape 11 | - A list of two tensor [seq_value,seq_len] 12 | 13 | - seq_value is a 3D tensor with shape: ``(batch_size, T, embedding_size)`` 14 | 15 | - seq_len is a 2D tensor with shape : ``(batch_size, 1)``,indicate valid length of each sequence. 16 | 17 | Output shape 18 | - 3D tensor with shape: ``(batch_size, 1, embedding_size)``. 19 | 20 | Arguments 21 | - **mode**:str.Pooling operation to be used,can be sum,mean or max. 22 | 23 | - **supports_masking**:If True,the input need to support masking. 24 | """ 25 | 26 | def __init__(self, mode='mean', supports_masking=False, **kwargs): 27 | 28 | if mode not in ['sum', 'mean', 'max']: 29 | raise ValueError("mode must be sum or mean") 30 | self.mode = mode 31 | self.eps = tf.constant(1e-8, tf.float32) 32 | super(SequencePoolingLayer, self).__init__(**kwargs) 33 | 34 | self.supports_masking = supports_masking 35 | 36 | def build(self, input_shape): 37 | if not self.supports_masking: 38 | self.seq_len_max = int(input_shape[0][1]) 39 | super(SequencePoolingLayer, self).build(input_shape) # Be sure to call this somewhere! 40 | 41 | def call(self, seq_value_len_list, mask=None, **kwargs): 42 | if self.supports_masking: 43 | if mask is None: 44 | raise ValueError( 45 | "When supports_masking=True,input must support masking") 46 | uiseq_embed_list = seq_value_len_list 47 | mask = tf.cast(mask, tf.float32) # tf.to_float(mask) 48 | user_behavior_length = tf.reduce_sum(mask, axis=-1, keepdims=True) 49 | mask = tf.expand_dims(mask, axis=2) 50 | else: 51 | uiseq_embed_list, user_behavior_length = seq_value_len_list 52 | 53 | mask = tf.sequence_mask(user_behavior_length, 54 | self.seq_len_max, dtype=tf.float32) 55 | mask = tf.transpose(mask, (0, 2, 1)) 56 | 57 | embedding_size = uiseq_embed_list.shape[-1] 58 | 59 | mask = tf.tile(mask, [1, 1, embedding_size]) 60 | 61 | if self.mode == "max": 62 | hist = uiseq_embed_list - (1-mask) * 1e9 63 | return tf.reduce_max(hist, 1, keepdims=True) 64 | 65 | hist = tf.reduce_sum(uiseq_embed_list * mask, 1, keepdims=False) 66 | 67 | if self.mode == "mean": 68 | hist = tf.divide(hist, tf.cast(user_behavior_length, tf.float32) + self.eps) 69 | 70 | hist = tf.expand_dims(hist, axis=1) 71 | return hist 72 | 73 | def compute_output_shape(self, input_shape): 74 | if self.supports_masking: 75 | return (None, 1, input_shape[-1]) 76 | else: 77 | return (None, 1, input_shape[0][-1]) 78 | 79 | def compute_mask(self, inputs, mask): 80 | return None 81 | 82 | def get_config(self, ): 83 | config = {'mode': self.mode, 'supports_masking': self.supports_masking} 84 | base_config = super(SequencePoolingLayer, self).get_config() 85 | return dict(list(base_config.items()) + list(config.items())) 86 | 87 | 88 | 89 | class LabelAwareAttention(Layer): 90 | def __init__(self, k_max, pow_p=1, **kwargs): 91 | self.k_max = k_max 92 | self.pow_p = pow_p 93 | super(LabelAwareAttention, self).__init__(**kwargs) 94 | 95 | def build(self, input_shape): 96 | # Be sure to call this somewhere! 97 | self.embedding_size = input_shape[0][-1] 98 | super(LabelAwareAttention, self).build(input_shape) 99 | 100 | def call(self, inputs, training=None, **kwargs): 101 | keys = inputs[0] 102 | query = inputs[1] 103 | weight = tf.reduce_sum(keys * query, axis=-1, keepdims=True) 104 | weight = tf.pow(weight, self.pow_p) # [x,k_max,1] 105 | 106 | if len(inputs) == 3: 107 | k_user = tf.cast(tf.maximum( 108 | 1., 109 | tf.minimum( 110 | tf.cast(self.k_max, dtype="float32"), # k_max 111 | tf.math.log1p(tf.cast(inputs[2], dtype="float32")) / tf.math.log(2.) # hist_len 112 | ) 113 | ), dtype="int64") 114 | 115 | seq_mask = tf.transpose(tf.sequence_mask(k_user, self.k_max), [0, 2, 1]) 116 | padding = tf.ones_like(seq_mask, dtype=tf.float32) * (-2 ** 32 + 1) # [x,k_max,1] 117 | weight = tf.where(seq_mask, weight, padding) 118 | 119 | weight = tf.nn.softmax(weight, name="weight") 120 | output = tf.reduce_sum(keys * weight, axis=1) 121 | 122 | return output 123 | 124 | def compute_output_shape(self, input_shape): 125 | return (None, self.embedding_size) 126 | 127 | def get_config(self, ): 128 | config = {'k_max': self.k_max, 'pow_p': self.pow_p} 129 | base_config = super(LabelAwareAttention, self).get_config() 130 | return dict(list(base_config.items()) + list(config.items())) 131 | 132 | 133 | 134 | class CapsuleLayer(Layer): 135 | def __init__(self, input_units, out_units, max_len, k_max, iteration_times=3, 136 | init_std=1.0, **kwargs): 137 | self.input_units = input_units 138 | self.out_units = out_units 139 | self.max_len = max_len 140 | self.k_max = k_max 141 | self.iteration_times = iteration_times 142 | self.init_std = init_std 143 | super(CapsuleLayer, self).__init__(**kwargs) 144 | 145 | def build(self, input_shape): 146 | self.routing_logits = self.add_weight(shape=[1, self.k_max, self.max_len], 147 | initializer=RandomNormal(stddev=self.init_std), 148 | trainable=False, name="B", dtype=tf.float32) 149 | self.bilinear_mapping_matrix = self.add_weight(shape=[self.input_units, self.out_units], 150 | initializer=RandomNormal(stddev=self.init_std), 151 | name="S", dtype=tf.float32) 152 | super(CapsuleLayer, self).build(input_shape) 153 | 154 | def call(self, inputs, **kwargs): 155 | behavior_embddings, seq_len = inputs 156 | batch_size = tf.shape(behavior_embddings)[0] 157 | seq_len_tile = tf.tile(seq_len, [1, self.k_max]) 158 | 159 | for i in range(self.iteration_times): 160 | mask = tf.sequence_mask(seq_len_tile, self.max_len) 161 | pad = tf.ones_like(mask, dtype=tf.float32) * (-2 ** 32 + 1) 162 | routing_logits_with_padding = tf.where(mask, tf.tile(self.routing_logits, [batch_size, 1, 1]), pad) 163 | weight = tf.nn.softmax(routing_logits_with_padding) 164 | behavior_embdding_mapping = tf.tensordot(behavior_embddings, self.bilinear_mapping_matrix, axes=1) 165 | Z = tf.matmul(weight, behavior_embdding_mapping) 166 | interest_capsules = squash(Z) 167 | 168 | delta_routing_logits = tf.reduce_sum( 169 | tf.matmul(interest_capsules, tf.transpose(behavior_embdding_mapping, perm=[0, 2, 1])), 170 | axis=0, keepdims=True 171 | ) 172 | self.routing_logits.assign_add(delta_routing_logits) 173 | 174 | interest_capsules = tf.reshape(interest_capsules, [-1, self.k_max, self.out_units]) 175 | return interest_capsules 176 | 177 | def compute_output_shape(self, input_shape): 178 | return (None, self.k_max, self.out_units) 179 | 180 | def get_config(self, ): 181 | config = {'input_units': self.input_units, 'out_units': self.out_units, 'max_len': self.max_len, 182 | 'k_max': self.k_max, 'iteration_times': self.iteration_times, "init_std": self.init_std} 183 | base_config = super(CapsuleLayer, self).get_config() 184 | return dict(list(base_config.items()) + list(config.items())) 185 | 186 | 187 | 188 | def squash(inputs): 189 | vec_squared_norm = tf.reduce_sum(tf.square(inputs), axis=-1, keepdims=True) 190 | scalar_factor = vec_squared_norm / (1 + vec_squared_norm) / tf.sqrt(vec_squared_norm + 1e-8) 191 | vec_squashed = scalar_factor * inputs 192 | 193 | return vec_squashed -------------------------------------------------------------------------------- /MIND/README.md: -------------------------------------------------------------------------------- 1 | # tf.__version__ == '2.1.0' 2 | 3 | 4 | # 运行 5 | ``` shell 6 | sh master.sh 7 | 8 | ``` 9 | 10 | 11 | # mind's summary 12 | 13 | ```python 14 | 15 | Model: "model_1" 16 | __________________________________________________________________________________________________ 17 | Layer (type) Output Shape Param # Connected to 18 | ================================================================================================== 19 | user_click_item_seq_input_layer [(None, 50)] 0 20 | __________________________________________________________________________________________________ 21 | user_id_input_layer (InputLayer [(None, 1)] 0 22 | __________________________________________________________________________________________________ 23 | gender_input_layer (InputLayer) [(None, 1)] 0 24 | __________________________________________________________________________________________________ 25 | age_input_layer (InputLayer) [(None, 1)] 0 26 | __________________________________________________________________________________________________ 27 | occupation_input_layer (InputLa [(None, 1)] 0 28 | __________________________________________________________________________________________________ 29 | zip_input_layer (InputLayer) [(None, 1)] 0 30 | __________________________________________________________________________________________________ 31 | item_id_embedding_layer (Embedd multiple 237248 pos_item_sample_input_layer[0][0] 32 | neg_item_sample_input_layer[0][0] 33 | user_click_item_seq_input_layer[0 34 | __________________________________________________________________________________________________ 35 | user_click_item_seq_length_inpu [(None, 1)] 0 36 | __________________________________________________________________________________________________ 37 | user_id_embedding_layer (Embedd (None, 1, 64) 386624 user_id_input_layer[0][0] 38 | __________________________________________________________________________________________________ 39 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 40 | __________________________________________________________________________________________________ 41 | age_embedding_layer (Embedding) (None, 1, 64) 512 age_input_layer[0][0] 42 | __________________________________________________________________________________________________ 43 | occupation_embedding_layer (Emb (None, 1, 64) 1408 occupation_input_layer[0][0] 44 | __________________________________________________________________________________________________ 45 | zip_embedding_layer (Embedding) (None, 1, 64) 220160 zip_input_layer[0][0] 46 | __________________________________________________________________________________________________ 47 | sequence_pooling_layer_55 (Sequ (None, 1, 64) 0 item_id_embedding_layer[2][0] 48 | user_click_item_seq_length_input_ 49 | __________________________________________________________________________________________________ 50 | concatenate_133 (Concatenate) (None, 1, 384) 0 user_id_embedding_layer[0][0] 51 | gender_embedding_layer[0][0] 52 | age_embedding_layer[0][0] 53 | occupation_embedding_layer[0][0] 54 | zip_embedding_layer[0][0] 55 | sequence_pooling_layer_55[0][0] 56 | __________________________________________________________________________________________________ 57 | tf_op_layer_Tile_2 (TensorFlowO [(None, 5, 384)] 0 concatenate_133[0][0] 58 | __________________________________________________________________________________________________ 59 | capsule_layer_58 (CapsuleLayer) (None, 5, 64) 4346 item_id_embedding_layer[2][0] 60 | user_click_item_seq_length_input_ 61 | __________________________________________________________________________________________________ 62 | concatenate_134 (Concatenate) (None, 5, 448) 0 tf_op_layer_Tile_2[0][0] 63 | capsule_layer_58[0][0] 64 | __________________________________________________________________________________________________ 65 | FC_1 (Dense) (None, 5, 128) 57472 concatenate_134[0][0] 66 | __________________________________________________________________________________________________ 67 | pos_item_sample_input_layer (In [(None, 1)] 0 68 | __________________________________________________________________________________________________ 69 | neg_item_sample_input_layer (In [(None, 10)] 0 70 | __________________________________________________________________________________________________ 71 | FC_2 (Dense) (None, 5, 64) 8256 FC_1[0][0] 72 | __________________________________________________________________________________________________ 73 | label_aware_attention_43 (Label (None, 64) 0 FC_2[0][0] 74 | item_id_embedding_layer[0][0] 75 | user_click_item_seq_length_input_ 76 | __________________________________________________________________________________________________ 77 | concatenate_135 (Concatenate) (None, 11, 64) 0 item_id_embedding_layer[0][0] 78 | item_id_embedding_layer[1][0] 79 | __________________________________________________________________________________________________ 80 | tf_op_layer_ExpandDims_7 (Tenso [(None, 1, 64)] 0 label_aware_attention_43[0][0] 81 | __________________________________________________________________________________________________ 82 | tf_op_layer_transpose_34 (Tenso [(None, 64, 11)] 0 concatenate_135[0][0] 83 | __________________________________________________________________________________________________ 84 | tf_op_layer_MatMul_29 (TensorFl [(None, 1, 11)] 0 tf_op_layer_ExpandDims_7[0][0] 85 | tf_op_layer_transpose_34[0][0] 86 | __________________________________________________________________________________________________ 87 | tf_op_layer_Softmax_29 (TensorF [(None, 1, 11)] 0 tf_op_layer_MatMul_29[0][0] 88 | ================================================================================================== 89 | Total params: 916,218 90 | Trainable params: 915,968 91 | Non-trainable params: 250 92 | __________________________________________________________________________________________________ 93 | None 94 | 95 | ``` 96 | 97 | 98 | # 运行过程 99 | 100 | ```python 101 | 102 | WARNING:tensorflow:sample_weight modes were coerced from 103 | ... 104 | to 105 | ['...'] 106 | WARNING:tensorflow:sample_weight modes were coerced from 107 | ... 108 | to 109 | ['...'] 110 | Train for 989 steps, validate for 7 steps 111 | Epoch 1/2 112 | 989/989 [==============================] - 137s 139ms/step - loss: 1.6125 - sparse_categorical_accuracy: 0.4041 - val_loss: 1.5422 - val_sparse_categorical_accuracy: 0.4224 113 | Epoch 2/2 114 | 989/989 [==============================] - 131s 133ms/step - loss: 1.3553 - sparse_categorical_accuracy: 0.4910 - val_loss: 1.4716 - val_sparse_categorical_accuracy: 0.4604 115 | ``` 116 | 117 | 118 | # 参考 119 | 120 | [https://github.com/shenweichen/DeepMatch/blob/master/deepmatch/models/mind.py](https://github.com/shenweichen/DeepMatch/blob/master/deepmatch/models/mind.py) 121 | 122 | -------------------------------------------------------------------------------- /Wide&Deep/README.md: -------------------------------------------------------------------------------- 1 | # tf.__version__ == '2.1.0' 2 | 3 | 4 | # Data format: 5 | 6 | ```python 7 | 8 | "user_id","gender","age","item_id","click_count","sales_count","label" 9 | 10 | 1,2,1,10,114,10,1 11 | 4,2,6,7,31,0,0 12 | 13 | ``` 14 | 15 | 16 | # wide_and_deep Binary-classification model.summary() 17 | 18 | ``` python 19 | Model: "model_1" 20 | __________________________________________________________________________________________________ 21 | Layer (type) Output Shape Param # Connected to 22 | ================================================================================================== 23 | user_id_input_layer (InputLayer [(None, 1)] 0 24 | __________________________________________________________________________________________________ 25 | gender_input_layer (InputLayer) [(None, 1)] 0 26 | __________________________________________________________________________________________________ 27 | age_input_layer (InputLayer) [(None, 1)] 0 28 | __________________________________________________________________________________________________ 29 | item_id_input_layer (InputLayer [(None, 1)] 0 30 | __________________________________________________________________________________________________ 31 | click_count_input_layer (InputL [(None, 1)] 0 32 | __________________________________________________________________________________________________ 33 | sales_count_input_layer (InputL [(None, 1)] 0 34 | __________________________________________________________________________________________________ 35 | user_id_embedding_layer (Embedd (None, 1, 64) 6464 user_id_input_layer[0][0] 36 | __________________________________________________________________________________________________ 37 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 38 | __________________________________________________________________________________________________ 39 | age_embedding_layer (Embedding) (None, 1, 64) 704 age_input_layer[0][0] 40 | __________________________________________________________________________________________________ 41 | item_id_embedding_layer (Embedd (None, 1, 64) 32064 item_id_input_layer[0][0] 42 | __________________________________________________________________________________________________ 43 | tf_op_layer_ExpandDims_2 (Tenso [(None, 1, 1)] 0 click_count_input_layer[0][0] 44 | __________________________________________________________________________________________________ 45 | tf_op_layer_ExpandDims_3 (Tenso [(None, 1, 1)] 0 sales_count_input_layer[0][0] 46 | __________________________________________________________________________________________________ 47 | concatenate_2 (Concatenate) (None, 1, 258) 0 user_id_embedding_layer[0][0] 48 | gender_embedding_layer[0][0] 49 | age_embedding_layer[0][0] 50 | item_id_embedding_layer[0][0] 51 | tf_op_layer_ExpandDims_2[0][0] 52 | tf_op_layer_ExpandDims_3[0][0] 53 | __________________________________________________________________________________________________ 54 | FC_1 (Dense) (None, 1, 128) 33152 concatenate_2[0][0] 55 | __________________________________________________________________________________________________ 56 | dropout_2 (Dropout) (None, 1, 128) 0 FC_1[0][0] 57 | __________________________________________________________________________________________________ 58 | FC_2 (Dense) (None, 1, 64) 8256 dropout_2[0][0] 59 | __________________________________________________________________________________________________ 60 | dense_2 (Dense) (None, 1, 1) 259 concatenate_2[0][0] 61 | __________________________________________________________________________________________________ 62 | dropout_3 (Dropout) (None, 1, 64) 0 FC_2[0][0] 63 | __________________________________________________________________________________________________ 64 | concatenate_3 (Concatenate) (None, 1, 65) 0 dense_2[0][0] 65 | dropout_3[0][0] 66 | __________________________________________________________________________________________________ 67 | dense_3 (Dense) (None, 1, 1) 66 concatenate_3[0][0] 68 | ================================================================================================== 69 | Total params: 81,157 70 | Trainable params: 81,157 71 | Non-trainable params: 0 72 | __________________________________________________________________________________________________ 73 | None 74 | ``` 75 | 76 | 77 | 78 | # wide_and_deep Multi-classification model.summary() 79 | 80 | ``` python 81 | 82 | Model: "model_2" 83 | __________________________________________________________________________________________________ 84 | Layer (type) Output Shape Param # Connected to 85 | ================================================================================================== 86 | user_id_input_layer (InputLayer [(None, 1)] 0 87 | __________________________________________________________________________________________________ 88 | gender_input_layer (InputLayer) [(None, 1)] 0 89 | __________________________________________________________________________________________________ 90 | age_input_layer (InputLayer) [(None, 1)] 0 91 | __________________________________________________________________________________________________ 92 | item_id_input_layer (InputLayer [(None, 1)] 0 93 | __________________________________________________________________________________________________ 94 | click_count_input_layer (InputL [(None, 1)] 0 95 | __________________________________________________________________________________________________ 96 | sales_count_input_layer (InputL [(None, 1)] 0 97 | __________________________________________________________________________________________________ 98 | user_id_embedding_layer (Embedd (None, 1, 64) 6464 user_id_input_layer[0][0] 99 | __________________________________________________________________________________________________ 100 | gender_embedding_layer (Embeddi (None, 1, 64) 192 gender_input_layer[0][0] 101 | __________________________________________________________________________________________________ 102 | age_embedding_layer (Embedding) (None, 1, 64) 704 age_input_layer[0][0] 103 | __________________________________________________________________________________________________ 104 | item_id_embedding_layer (Embedd (None, 1, 64) 32064 item_id_input_layer[0][0] 105 | __________________________________________________________________________________________________ 106 | tf_op_layer_ExpandDims_4 (Tenso [(None, 1, 1)] 0 click_count_input_layer[0][0] 107 | __________________________________________________________________________________________________ 108 | tf_op_layer_ExpandDims_5 (Tenso [(None, 1, 1)] 0 sales_count_input_layer[0][0] 109 | __________________________________________________________________________________________________ 110 | concatenate_4 (Concatenate) (None, 1, 258) 0 user_id_embedding_layer[0][0] 111 | gender_embedding_layer[0][0] 112 | age_embedding_layer[0][0] 113 | item_id_embedding_layer[0][0] 114 | tf_op_layer_ExpandDims_4[0][0] 115 | tf_op_layer_ExpandDims_5[0][0] 116 | __________________________________________________________________________________________________ 117 | FC_1 (Dense) (None, 1, 128) 33152 concatenate_4[0][0] 118 | __________________________________________________________________________________________________ 119 | dropout_4 (Dropout) (None, 1, 128) 0 FC_1[0][0] 120 | __________________________________________________________________________________________________ 121 | FC_2 (Dense) (None, 1, 64) 8256 dropout_4[0][0] 122 | __________________________________________________________________________________________________ 123 | dense_4 (Dense) (None, 1, 1) 259 concatenate_4[0][0] 124 | __________________________________________________________________________________________________ 125 | dropout_5 (Dropout) (None, 1, 64) 0 FC_2[0][0] 126 | __________________________________________________________________________________________________ 127 | concatenate_5 (Concatenate) (None, 1, 65) 0 dense_4[0][0] 128 | dropout_5[0][0] 129 | __________________________________________________________________________________________________ 130 | dense_5 (Dense) (None, 1, 10) 660 concatenate_5[0][0] 131 | ================================================================================================== 132 | Total params: 81,751 133 | Trainable params: 81,751 134 | Non-trainable params: 0 135 | __________________________________________________________________________________________________ 136 | None 137 | 138 | ``` --------------------------------------------------------------------------------