16 |
17 | ## Corpus downloads
18 | We used 2 datasets for creating sentence vectors:
19 | 1. word vectors taken from: https://github.com/stanfordnlp/GloVe
20 | 2. word frequencies from: https://github.com/IlyaSemenov/wikipedia-word-frequency/blob/master/results/enwiki-20150602-words-frequency.txt
21 |
22 | ## Configurable parameters (Hyper-Parameters)
23 | We played around with a few configurations to get the best results for short user messages:
24 |
25 | * Confidence Threshold - a number between 0.0 - 1.0 to determine when 2 messages are considered the same
26 | * Distance Function - we used vector dot product
27 | * Normalization - how to deal with words we don't have in our corpus, punctuation marks, non english words
28 | * Vector Size - the longer the vector the higher the accuracy but heavier in memory
29 | * Weight Function - given a word frequency, how to create the vector weights (`the` should weigh less than `camera`)
30 | * Custom Corpus - creating the word vectors and frequencies from real user message data might yield better results
31 | * Random Indices - how many random indices should the client send to the server to mask the original message indices
32 |
33 | ## Running the code
34 | This project includes a single makefile to help with the initialization, dependency installation and corpus download.
35 | You can invoke a help message by running:
36 |
37 | ```
38 | make
39 | ```
40 |
41 | Or you can manually run the server and client apps:
42 |
43 | ### server
44 | In the `server` directory install the pip dependencies in a `virtualenv`:
45 |
46 | ```
47 | pip install -r requirements.txt
48 | ```
49 |
50 | and run the server:
51 | ```
52 | python app.py
53 | ```
54 |
55 | ### web client
56 | To use the web client go into the `webclient` directory in your terminal and then:
57 | ```
58 | npm install
59 | npm run dev
60 | ```
61 |
62 | That should install all dependencies and kick start the project, if it all works you should see something like:
63 | > Project is running at http://localhost:3333/
64 | > webpack output is served from /
65 |
66 | Now load http://localhost:3333/ in your browser
67 |
68 | There are 3 different "view modes" which can be switched using the select box at the top right corner of the page.
69 | The 3 views are:
70 | * Standalone Tester: A textarea in which one can input a message and then either report it as spam or check whether it is classified as spam.
71 | * IM Sender: A textarea in which the user can input a message (or select a message from a bunch of existing ones) and then "send" the message to another client.
72 | * IM Receiver: A view which displays a list of received messages (using the `IM Sender`) and the ability to report each message as spam.
73 |
--------------------------------------------------------------------------------
/clip.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/doodyparizada/word2vec-spam-filter/695898b7e7a3091947626f12fc31c9be4bb03784/clip.gif
--------------------------------------------------------------------------------
/server/Testing.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 142,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import requests\n",
10 | "BASE_URL = 'http://localhost:5000'\n",
11 | "\n",
12 | "def report(msg):\n",
13 | " requests.post(BASE_URL + '/spam/report',\n",
14 | " json={'message': msg}\n",
15 | " ).ok\n",
16 | "\n",
17 | "def is_spam(vector):\n",
18 | " data = requests.get(BASE_URL + '/spam/detect',\n",
19 | " params={'vector': ','.join(str(v) for v in vector)}\n",
20 | " ).json()\n",
21 | " return data['spam'], data['confidence']\n",
22 | "\n",
23 | "def get_word_vectors(indexes):\n",
24 | " data = requests.get(BASE_URL + '/words/vector',\n",
25 | " params={'ids': ','.join(str(i) for i in indexes)}).json()\n",
26 | " return {int(k): v['vector'] for k,v in data['words'].items()}\n",
27 | "\n",
28 | "def get_word_indexes():\n",
29 | " \"\"\"return a dict of word to index.\"\"\"\n",
30 | " content = requests.get(BASE_URL + '/words/list').content\n",
31 | " return {word: i for i, word in enumerate(content.split('\\n'))}"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": 143,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "WORD_INDEXES = get_word_indexes()\n",
41 | "\n",
42 | "\n",
43 | "def calc_message(message):\n",
44 | " import random\n",
45 | " import numpy as np\n",
46 | " \n",
47 | " message = message.lower()\n",
48 | " \n",
49 | " indexes = [WORD_INDEXES[word] \n",
50 | " for word in message.split(' ')\n",
51 | " if word in WORD_INDEXES]\n",
52 | " fakes = [random.randint(0, len(WORD_INDEXES)) for _ in range(20)]\n",
53 | " \n",
54 | " shuffled = indexes + fakes\n",
55 | " random.shuffle(shuffled)\n",
56 | " vectors = get_word_vectors(shuffled)\n",
57 | " \n",
58 | " vec_sum = sum(np.array(vectors[i]) for i in indexes).tolist()\n",
59 | "\n",
60 | " return vec_sum"
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": 144,
66 | "metadata": {},
67 | "outputs": [],
68 | "source": [
69 | "def is_spam_message(msg):\n",
70 | " return is_spam(calc_message(msg))"
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": 145,
76 | "metadata": {},
77 | "outputs": [],
78 | "source": [
79 | "spams = [\n",
80 | " 'Hi are you looking for a sexy cool woman to spend time ,with well I,m lookin for a gentleman who still khowns how to treat a lady, I am here to u.http . G.lovendate.pw code: 605',\n",
81 | " 'I offer you an exchange, you free register on my site http:\\/\\/rachelmel.pro\\nConfirmation email and show me the screenshot. \\nAfter that I\\'ll send you my nude pics',\n",
82 | " '''Hi, do I know u? you just showed up in my kik hmm.. my friends warned me that there are many fake accounts and bots here, no offense, are u a real person? If you are a real person, you won't have any trouble liking my pic, will you;)? the one where I'm wearing a white swimming suit. This way I'll be convinced that you are real''',\n",
83 | " '''This is the first time I will be playing naked on a webcam. Come see my naked body. Reg here, plz:* http:\\/\\/u.to\\/VgdTEA It is completely free'''\n",
84 | " '''go here, click the create private account.. you might have to create a username before you can see me live''',\n",
85 | "]\n",
86 | "\n",
87 | "good = [\n",
88 | " 'Hi I am your mother and I love you',\n",
89 | " 'Did you watch the movie last night?',\n",
90 | " 'Whats up with all the weird clouds?',\n",
91 | "]"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": 146,
97 | "metadata": {},
98 | "outputs": [],
99 | "source": [
100 | "# register spam messages\n",
101 | "for msg in spams:\n",
102 | " for i in range(4):\n",
103 | " report(msg)"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": 115,
109 | "metadata": {},
110 | "outputs": [
111 | {
112 | "data": {
113 | "text/plain": [
114 | "(False, 0.8641516840020381)"
115 | ]
116 | },
117 | "execution_count": 115,
118 | "metadata": {},
119 | "output_type": "execute_result"
120 | }
121 | ],
122 | "source": [
123 | "is_spam_message(good[2])"
124 | ]
125 | },
126 | {
127 | "cell_type": "code",
128 | "execution_count": 117,
129 | "metadata": {},
130 | "outputs": [
131 | {
132 | "data": {
133 | "text/plain": [
134 | "(True, 1.0)"
135 | ]
136 | },
137 | "execution_count": 117,
138 | "metadata": {},
139 | "output_type": "execute_result"
140 | }
141 | ],
142 | "source": [
143 | "is_spam_message(spams[1])"
144 | ]
145 | },
146 | {
147 | "cell_type": "code",
148 | "execution_count": 129,
149 | "metadata": {},
150 | "outputs": [
151 | {
152 | "data": {
153 | "text/plain": [
154 | "(False, 0.6815927618764492)"
155 | ]
156 | },
157 | "execution_count": 129,
158 | "metadata": {},
159 | "output_type": "execute_result"
160 | }
161 | ],
162 | "source": [
163 | "is_spam_message('Hey Doody What is up?')"
164 | ]
165 | },
166 | {
167 | "cell_type": "code",
168 | "execution_count": 141,
169 | "metadata": {},
170 | "outputs": [
171 | {
172 | "name": "stdout",
173 | "output_type": "stream",
174 | "text": [
175 | "['Hi', 'do', 'I', 'know', 'u', 'you', 'just', 'showed', 'up', 'in', 'my', 'kik', 'hmm', 'my', 'friends', 'warned', 'me', 'that', 'there', 'are', 'many', 'fake', 'accounts', 'and', 'bots', 'here', 'no', 'offense', 'are', 'u', 'a', 'real', 'person', 'If', 'you', 'are', 'a', 'real', 'person', 'you', \"won't\", 'have', 'any', 'trouble', 'liking', 'my', 'pic', 'will', 'you', 'the', 'one', 'where', \"I'm\", 'wearing', 'a', 'white', 'swimming', 'suit', 'This', 'way', \"I'll\", 'be', 'convinced', 'that', 'you', 'are', 'real']\n"
176 | ]
177 | }
178 | ],
179 | "source": [
180 | ">>> import re\n",
181 | ">>> string = spams[2]\n",
182 | ">>> pattern = re.compile(\"^\\s+|\\s*[, \\.\\?;\\(\\)]\\s*|\\s+$\")\n",
183 | ">>> print([x for x in pattern.split(string) if x])"
184 | ]
185 | },
186 | {
187 | "cell_type": "code",
188 | "execution_count": null,
189 | "metadata": {},
190 | "outputs": [],
191 | "source": []
192 | }
193 | ],
194 | "metadata": {
195 | "kernelspec": {
196 | "display_name": "Python 2",
197 | "language": "python",
198 | "name": "python2"
199 | },
200 | "language_info": {
201 | "codemirror_mode": {
202 | "name": "ipython",
203 | "version": 2
204 | },
205 | "file_extension": ".py",
206 | "mimetype": "text/x-python",
207 | "name": "python",
208 | "nbconvert_exporter": "python",
209 | "pygments_lexer": "ipython2",
210 | "version": "2.7.14"
211 | }
212 | },
213 | "nbformat": 4,
214 | "nbformat_minor": 2
215 | }
216 |
--------------------------------------------------------------------------------
/server/app.py:
--------------------------------------------------------------------------------
1 | import json
2 | import sys
3 | from math import log
4 |
5 | from flask import request, Flask, jsonify
6 | import numpy as np
7 |
8 | from model import DB
9 | from generate import generate_matrix, normalize_matrix, normalize_vector
10 | from flask_cors import CORS
11 |
12 |
13 | app = Flask(__name__)
14 | CORS(app)
15 |
16 | GLOVE = '../corpus/glove.6B.300d.txt'
17 | FREQ = '../corpus/enwiki-20150602-words-frequency.txt'
18 |
19 | iweights = {}
20 | vocab = {}
21 | ivocab = {}
22 | WORD_LIST = ''
23 | W_norm = None
24 | messages = []
25 |
26 | EPSILON = 0.95
27 | DEFAULT_WEIGHT = 15
28 |
29 |
30 | def init():
31 | """read glove file and generate a word matrix"""
32 | global W_norm, WORD_LIST, vocab, ivocab, iweights
33 | sys.stderr.write('initializing word vectors')
34 | word_vectors = []
35 |
36 | # open and parse word vector file
37 | with open(GLOVE, 'r') as f:
38 | for i, line in enumerate(f):
39 | vals = line.rstrip().split(' ')
40 | vector = [float(x) for x in vals[1:]]
41 | word = vals[0]
42 | word_vectors.append((word, vector))
43 | if i % 10000 == 0:
44 | sys.stderr.write('.')
45 |
46 | WORD_LIST += '\n'.join(w for w, _ in word_vectors)
47 | W, vocab, ivocab = generate_matrix(word_vectors)
48 | W_norm = normalize_matrix(W)
49 |
50 | sys.stderr.write('\ninitializing word weights')
51 | max_freq = None
52 | with open(FREQ, 'r') as f:
53 | for i, line in enumerate(f):
54 | vals = line.rstrip().split(' ')
55 | word = vals[0]
56 | freq = int(vals[1])
57 | max_freq = max_freq or freq # the first iteration will set max_freq. The first line is the highest freq
58 | if word in vocab:
59 | iweights[vocab[word]] = freq_to_weight(freq, max_freq)
60 | if i % 10000 == 0:
61 | sys.stderr.write('.')
62 |
63 | sys.stderr.write('\ndone!\n')
64 |
65 |
66 | def get_vector(idx):
67 | """return the weighted vector for an index."""
68 | return (W_norm[idx, :] * iweights.get(idx, DEFAULT_WEIGHT))
69 |
70 |
71 | def freq_to_weight(freq, max_freq):
72 | """calculate a vector weight for a frequency."""
73 | # taken from https://www.wikiwand.com/en/Word_lists_by_frequency
74 | return 0.5 - log(float(freq)/max_freq, 2)
75 |
76 |
77 | def generate_spam_matrix(report_threashold):
78 | """
79 | put all known spam vectors in a matrix
80 | """
81 | db = DB.load()
82 | word_vectors = [(word, rm.vector)
83 | for word, rm in db.reported_messages.items()
84 | if rm.reports >= report_threashold]
85 | return generate_matrix(word_vectors)
86 |
87 |
88 | def closest_spam(vector, report_threashold=3):
89 | """given a vector, return the closest spam messages and distance."""
90 | W, vocab, ivocab = generate_spam_matrix(report_threashold=report_threashold)
91 |
92 | if not vocab: # means empty db
93 | return '', 0
94 |
95 | vector = normalize_vector(vector)
96 |
97 | dist = np.dot(W, vector.T)
98 |
99 | a = np.argsort(-dist)[:1] # currently returns generator of 3 most closest
100 | for x in a:
101 | return ivocab[x], float(dist[x])
102 |
103 | return '', 0
104 |
105 | def tokenize_message(message):
106 | """return a list of normalized words."""
107 | return (message
108 | .lower()
109 | .replace(".", " .")
110 | .replace(",", " ,")
111 | .replace("?", " ?")
112 | .replace("!", " !")
113 | .replace(":", " :")
114 | .replace("'s", " 's")
115 | .split())
116 |
117 |
118 | def message_to_vector(message):
119 | """sums up all known vectors of a given message."""
120 | vector = np.zeros(W_norm[0, :].shape)
121 | for term in tokenize_message(message):
122 | if term in vocab:
123 | vector += get_vector(vocab[term])
124 | return vector
125 |
126 |
127 | @app.route('/words/list')
128 | def word_list():
129 | """return word list. ordered by indexes."""
130 | return WORD_LIST
131 |
132 |
133 | @app.route('/words/vector')
134 | def word_vectors():
135 | """retrun vectors for the words by given ids."""
136 | ids = {int(i) for i in request.args['ids'].split(',')}
137 |
138 | return jsonify({'words':
139 | {i: {'vector': get_vector(i).tolist()}
140 | for i in ids}})
141 |
142 |
143 | @app.route('/spam/detect')
144 | def detect_spam():
145 | """the given vector should not be normalized. normalization happens on server."""
146 | vector = [float(i) for i in request.args['vector'].split(',')]
147 | msg, dist = closest_spam(vector)
148 | is_spam = dist > EPSILON
149 | return jsonify({'spam': is_spam,
150 | 'confidence': dist,
151 | 'meta': msg})
152 |
153 |
154 | @app.route('/spam/report', methods=['POST'])
155 | def report_spam():
156 | """if spam message already exists or is close to a known message add a report count. else add as new entry in db."""
157 | data = request.get_json()
158 | reported_message = data['message']
159 | vector = message_to_vector(reported_message)
160 |
161 | similar_msg, dist = closest_spam(vector, 0)
162 |
163 | db = DB.load()
164 | if dist > EPSILON:
165 | db.reported_messages[similar_msg].reports += 1
166 | else:
167 | db.add_new_message(reported_message, normalize_vector(vector).tolist())
168 |
169 | db.save()
170 | return jsonify({})
171 |
172 | @app.route('/messages', methods=['POST', 'GET'])
173 | def message_handler():
174 | global messages
175 | if request.method == 'POST':
176 | messages.append(request.get_json()['message'])
177 | return jsonify({})
178 | else:
179 | if messages:
180 | return jsonify({'message': messages.pop(0)})
181 | return jsonify({})
182 |
183 |
184 | if __name__ == '__main__':
185 | init()
186 | app.run()
187 |
--------------------------------------------------------------------------------
/server/generate.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 |
3 |
4 | def normalize_vector(vector):
5 | vector = np.array(vector)
6 | vec_norm = np.zeros(vector.shape)
7 | d = (np.sum(vector ** 2,) ** (0.5))
8 | vec_norm = (vector.T / d).T
9 | return vec_norm
10 |
11 |
12 | # XXX todo see if we can unite vector and matrix normalization
13 | def normalize_matrix(W):
14 | # normalize each word vector to unit variance
15 | W_norm = np.zeros(W.shape)
16 | d = (np.sum(W ** 2, 1) ** (0.5))
17 | W_norm = (W.T / d).T
18 | return W_norm
19 |
20 |
21 | def generate_matrix(word_vectors):
22 | """given a list of word,vector pairs generate matrix and vocab dicts"""
23 | vectors = dict(word_vectors)
24 | words = [w for w, _ in word_vectors]
25 |
26 | vocab = {w: idx for idx, w in enumerate(words)}
27 | ivocab = {idx: w for idx, w in enumerate(words)}
28 |
29 | vocab_size = len(vectors)
30 | vector_dim = len(vectors.values()[0]) if vectors else 0
31 | W = np.zeros((vocab_size, vector_dim))
32 | for word, v in vectors.items():
33 | W[vocab[word], :] = v
34 | return W, vocab, ivocab
35 |
--------------------------------------------------------------------------------
/server/model.py:
--------------------------------------------------------------------------------
1 | import json
2 |
3 | from schematics.models import Model
4 | from schematics.types import StringType, ListType, IntType, FloatType, ModelType, DictType
5 |
6 |
7 | FILENAME = '../database/db.json'
8 |
9 |
10 | class DB(Model):
11 | class ReportedMessage(Model):
12 | reports = IntType()
13 | vector = ListType(FloatType)
14 |
15 | reported_messages = DictType(ModelType(ReportedMessage), default={})
16 |
17 | def add_new_message(self, reported_message, vector):
18 | rm = self.ReportedMessage()
19 | rm.reports = 1
20 | rm.vector = vector
21 | self.reported_messages[reported_message] = rm
22 |
23 | @classmethod
24 | def load(cls):
25 | with open(FILENAME, 'r') as f:
26 | return DB(json.loads(f.read()))
27 |
28 | def save(self):
29 | string = json.dumps(self.to_primitive())
30 | with open(FILENAME, 'w') as f:
31 | f.write(string)
32 |
--------------------------------------------------------------------------------
/server/requirements.txt:
--------------------------------------------------------------------------------
1 | flask==0.12.2
2 | Flask-Cors==3.0.3
3 | requests==2.18.4
4 | schematics==2.0.1
5 | numpy==1.13.3
6 |
--------------------------------------------------------------------------------
/webclient/html/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |