├── .idea
    ├── cmft.iml
    ├── misc.xml
    └── modules.xml
├── README.md
├── arg_parse.py
├── data
    ├── shakespeare
    │   └── shakespeare.txt
    ├── test
    │   └── input.txt
    └── wikipedia
    │   └── input.txt
├── models
    ├── __init__.py
    ├── base.py
    ├── densenet
    │   ├── __init__.py
    │   ├── densenet_decoder.py
    │   ├── densenet_encoder.py
    │   ├── ops.py
    │   └── ops_tests.py
    ├── q1_lstm_baseline.py
    ├── q2_simple_conv.py
    ├── q3_ResidualConvs.py
    ├── q4_DilatedConv.py
    ├── q5_deconvolution_autoencoder.py
    └── small_conv.py
├── prepare_data.py
├── presentation.pdf
├── test_dataLoader.py
└── train.py


/.idea/cmft.iml:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="UTF-8"?>
 2 | <module type="PYTHON_MODULE" version="4">
 3 |   <component name="NewModuleRootManager">
 4 |     <content url="file://$MODULE_DIR$" />
 5 |     <orderEntry type="inheritedJdk" />
 6 |     <orderEntry type="sourceFolder" forTests="false" />
 7 |   </component>
 8 |   <component name="TestRunnerService">
 9 |     <option name="PROJECT_TEST_RUNNER" value="Unittests" />
10 |   </component>
11 | </module>


--------------------------------------------------------------------------------
/.idea/misc.xml:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="UTF-8"?>
 2 | <project version="4">
 3 |   <component name="ProjectLevelVcsManager" settingsEditedManually="false">
 4 |     <OptionsSetting value="true" id="Add" />
 5 |     <OptionsSetting value="true" id="Remove" />
 6 |     <OptionsSetting value="true" id="Checkout" />
 7 |     <OptionsSetting value="true" id="Update" />
 8 |     <OptionsSetting value="true" id="Status" />
 9 |     <OptionsSetting value="true" id="Edit" />
10 |     <ConfirmationsSetting value="0" id="Add" />
11 |     <ConfirmationsSetting value="0" id="Remove" />
12 |   </component>
13 |   <component name="ProjectRootManager" version="2" project-jdk-name="Python 3.5.2 (/usr/bin/python3.5)" project-jdk-type="Python SDK" />
14 | </project>


--------------------------------------------------------------------------------
/.idea/modules.xml:
--------------------------------------------------------------------------------
1 | <?xml version="1.0" encoding="UTF-8"?>
2 | <project version="4">
3 |   <component name="ProjectModuleManager">
4 |     <modules>
5 |       <module fileurl="file://$PROJECT_DIR$/.idea/cmft.iml" filepath="$PROJECT_DIR$/.idea/cmft.iml" />
6 |     </modules>
7 |   </component>
8 | </project>


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Convolutional Methods for Text Workshop 
 2 | ## Excersices for the workshop given at PyCon Israel 2017 
 3 | 
 4 | ## Intro 
 5 | This repository has the excersises for the workshop on convolutional methods for text. You can read about the constructs we'll be using and why you migh want to consider convolutions in my [blog post](https://medium.com/@TalPerry/convolutional-methods-for-text-d5260fd5675f). In a nutshell, convolutions run in parrelell so they are much faster than RNNs. 
 6 | 
 7 | The core task in this repository is to restore missing punctuation and capitalisation in a text, something that we may find ourselves doing when working with speech recognition systems.
 8 | To avoid boilerplate overhead, we are doing this at the charecter level, not word. 
 9 | ## Getting started
10 | Clone this repository
11 | ```
12 | pip3 install tensorflow nltk 
13 | ```
14 | 
15 | ## Contents
16 | `train.py` is your entry point for these tasks. To run a task modify the model import at the top
17 | ```python
18 | from models.q1_lstm_baseline import LSTMBaseline
19 | ```
20 | And then
21 | ```
22 | python3 train.py
23 | ```
24 | 
25 | the *models* directory contains each of the "questions" as well as some boilerplate as follows
26 | 
27 | `base.py` Holds a base class that covers the boiler plate for a NN. Includes setting up the loss and optimiser as well as embedding charecters. You should be able to ignore this during the workshop
28 | `densenet.ops` The densenet folder has all of the ops you may need to implement the various tasks here. I include it for reference but you won't get much from the workshop if you just copy paste it so try not to look here if you are stuck.
29 | ## Tasks
30 | All tasks are described below. You are of course free to do whichever ones you like. They are arranged in order of difficulty and build on one another
31 | ### `q1_lstm_baseline`
32 | In this task you'll use an LSTM or bidirectional LSTM to solve the task. Do this if you are new to Tensorflow/NN or you want to make your own baseline to see how slow LSTMs are compared to convolutions
33 | 
34 | ### `q2_simple_conv` 
35 | In this task we'll implement our own convolution op for 1d seqeunces like text. Once we have the op we'll apply it to our inputs to try and restore missing punctuation and capitalisation. 
36 | 
37 | ### `q3_ResidualConvs` 
38 | This task builds on the previous one. We'll want to increase our receptive field without encountering vanishing gradients. To do that we'll use residual connection a la DenseNet which you will implement. 
39 | 
40 | ### `q4_DilatedConvs`
41 | In this task we'll implement 1d dilated convolutions and use them as another method to increase receptive field without vanishing gradients. Bonus if you build residual dilated convolutions
42 | 
43 | ### `q5_deconvolution_autoencoder`
44 | In this question we'll use the residual convs we built and add 1d pooling and 1d "Deconvolutions" to encode our input as a vector and restore it respectively. 


--------------------------------------------------------------------------------
/arg_parse.py:
--------------------------------------------------------------------------------
 1 | import argparse
 2 | 
 3 | def parse_args(args):
 4 |     parser = argparse.ArgumentParser(...)
 5 |     parser.add_argument('--data_path', type=str,
 6 |                         default='data/wikipedia/input.txt', help="Path to raw txt file")
 7 |     parser.add_argument('--saved_data_path', type=str,
 8 |                         default='data/wikipedia/dataset.npa', help="Where to save/load processed input")
 9 |     parser.add_argument('--batch_size', type=int,
10 |                         default=8, help="Number of sentences in batch")
11 |     parser.add_argument('--lr', type=int,
12 |                         default=0.0001, help="Learning Rate")
13 | 
14 |     # ...Create your parser as you like...
15 |     return parser.parse_args(args)
16 | 


--------------------------------------------------------------------------------
/data/test/input.txt:
--------------------------------------------------------------------------------
 1 | It is the county seat of Alfalfa County .
 2 | Cherokee is a city in Alfalfa County , Oklahoma , United States .
 3 | Skateboard decks are usually between 28 and 33 inches long .
 4 | The underside of the deck can be printed with a design by the manufacturer , blank , or decorated by any other means .
 5 | This was created by two surfers ; Ben Whatson and Jonny Drapper .
 6 | Some of them have special materials that help to keep the deck from breaking : such as fiberglass , bamboo , resin , Kevlar , carbon fiber , aluminum , and plastic .
 7 | `` Old school '' boards -LRB- those made in the 1970s â `` 80s or modern boards that mimic their shape -RRB- are generally wider and often have only one kicktail .
 8 | One of the first deck companies was called `` Drapped '' taken from Jonny 's second name .
 9 | Grip tape , when applied to the top surface of a skateboard , gives a skater 's feet grip on the deck .
10 | Modern decks vary in size , but most are 7 to 10.5 inches wide .
11 | 


--------------------------------------------------------------------------------
/models/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/talolard/CMFT_PyCon2017/7e8db3a3a27baf44e97ee662a6874f42c38fa6ea/models/__init__.py


--------------------------------------------------------------------------------
/models/base.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | import abc
 3 | 
 4 | class ModelBase(metaclass=abc.ABCMeta):
 5 |     def __init__(self,args):
 6 |         self.args = args
 7 |         self.original = tf.placeholder(dtype=tf.int32,shape=[None,None])
 8 |         self.lower = tf.placeholder(dtype=tf.int32, shape=[None, None])
 9 |         self.batch_max_len = self.original.get_shape().as_list()[1]
10 |         self.embedded_source = self._embed_chars()
11 |         logits = self.get_logits()
12 |         self.loss_op = self._loss(logits)
13 |         self.train_op = self._train(self.loss_op)
14 |         self.predictions = self._prediction(logits)
15 | 
16 |     @abc.abstractmethod
17 |     def get_logits(self):
18 |         pass
19 | 
20 |     def _embed_chars(self):
21 |         embedding_size = 4
22 |         embedding_matrix = tf.get_variable("embedding_matrix", shape=[128, embedding_size],
23 |                                            dtype=tf.float32)
24 | 
25 |         return tf.nn.embedding_lookup(embedding_matrix,self.lower)
26 | 
27 | 
28 | 
29 |     def _loss(self,logits):
30 |         lengths = tf.reduce_sum(tf.sign(self.original),axis=1)
31 |         max_len =self.original.get_shape().as_list()[1]
32 |         mask = tf.sequence_mask(lengths,dtype=tf.float32,maxlen=max_len)
33 |         loss = tf.contrib.seq2seq.sequence_loss(logits=logits,
34 |                                                 targets=self.original,
35 |                                                 weights = mask
36 |                                                 )
37 |         return loss
38 | 
39 |     def _train(self,loss):
40 |         #lr = tf.train.exponential_decay(learning_rate=self.args.lr)
41 |         lr = self.args.lr
42 |         opt = tf.train.AdamOptimizer(lr)
43 |         return opt.minimize(loss)
44 | 
45 |     def _prediction(self,logits):
46 |         sm = tf.nn.softmax(logits)
47 |         preds = tf.argmax(sm,axis=2)
48 |         return preds
49 | 
50 | 


--------------------------------------------------------------------------------
/models/densenet/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/talolard/CMFT_PyCon2017/7e8db3a3a27baf44e97ee662a6874f42c38fa6ea/models/densenet/__init__.py


--------------------------------------------------------------------------------
/models/densenet/densenet_decoder.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | from tensorflow.contrib.layers.python.layers.initializers import xavier_initializer
 3 | 
 4 | from models.densenet import ops
 5 | from models.ops import conv1d_transpose
 6 | from arg_getter import FLAGS
 7 | def DenseNetDecoder(_input, layers_per_batch, growth_rate,expansion_rate,final_width=FLAGS.max_len):
 8 |     output = tf.expand_dims(_input,1)
 9 |     block =0
10 |     while output.get_shape().as_list()[1] <final_width:
11 |         with tf.variable_scope("decode_block_{}".format(block),initializer=xavier_initializer()):
12 |             output = conv1d_transpose(output, FLAGS.hidden1, 3, expansion_rate)
13 |             output = ops.makeBlock(output,growth_rate=growth_rate,num_layers=layers_per_batch)
14 | 
15 |             block+=1
16 |     return output
17 | 
18 | 
19 | 


--------------------------------------------------------------------------------
/models/densenet/densenet_encoder.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | from tensorflow.contrib.layers.python.layers.initializers import xavier_initializer
 3 | 
 4 | from models.densenet import ops
 5 | 
 6 | 
 7 | def DenseNetEncoder(_input, num_blocks, layers_per_batch, growth_rate, bottle=False):
 8 |     """Bottle will significantly reduce the amount of computation needed."""
 9 |     tf.logging.info(
10 |         'Building DenseNet Encoder with {} blocks and {} layers per block'.format(num_blocks, layers_per_batch))
11 |     output = _input
12 |     for block in range(num_blocks):
13 |         with tf.variable_scope("block_{}".format(block),initializer=xavier_initializer()):
14 |             output = ops.makeBlock(output, growth_rate=growth_rate, num_layers=layers_per_batch, bottle=bottle)
15 |             print('output after block', block, output)
16 |             if block < num_blocks - 1:
17 |                 output = ops.transition_layer(output)
18 |             else:
19 |                 output = ops.transition_to_vector(output)
20 | 
21 |     #output = tf.squeeze(output)
22 |     return output
23 | 


--------------------------------------------------------------------------------
/models/densenet/ops.py:
--------------------------------------------------------------------------------
 1 | '''
 2 | DenseNet for text with 1dconvolutions
 3 | '''
 4 | import tensorflow as tf
 5 | from tensorflow.contrib.layers import linear, fully_connected,layer_norm as ln
 6 | # from arg_getter import FLAGS
 7 | 
 8 | """as a note, layer normalization usually occurs right after the weight matrix is applied and BEFORE ANY ACTIVATIONS
 9 | """
10 | 
11 | def avg_pool1d(_input, k):
12 |     output = tf.expand_dims(_input,axis=1)
13 |     ksize = [1, 1, k, 1]
14 |     strides = [1, 1, k, 1]
15 |     padding = 'VALID'
16 |     output = tf.nn.avg_pool(output, ksize, strides, padding)
17 |     output = tf.squeeze(output,axis=1)
18 |     return output
19 | 
20 | 
21 | def _conv1d(x, outSize, width):
22 |     inputSize = x.get_shape()[-1]
23 |     filter_ = tf.get_variable("conv_filter", shape=[width, inputSize, outSize])
24 |     convolved = tf.nn.conv1d(x,filters=filter_,stride=1,padding="SAME")
25 |     return convolved
26 | 
27 | def composite_function(_input,outSize,width):
28 |     with tf.variable_scope("composite_function"):
29 |         output = _conv1d(_input, outSize, width=width)
30 |         output = ln(output)
31 |         output = tf.nn.relu(output)
32 | 
33 |     return output
34 | 
35 | def bottleneck(_input, growthRate):
36 |     '''
37 |     Per the paper, each bottlneck outputs 4k feature size where k is the growth rate of the network.
38 |     :return:
39 |     '''
40 | 
41 |     outSize = growthRate * 4
42 |     output = _conv1d(_input,outSize,width=1)
43 |     output = ln(output)
44 |     output = tf.nn.relu(output)
45 | 
46 |     return output
47 | 
48 | 
49 | def addInternalLayer(_input, growth_rate, bottle=False):
50 |     # notice that there is a 3x3 convolution at the beginning
51 |     compOut = composite_function(_input, growth_rate, width=3)
52 |     
53 |     # Concatenation -- think of this as a skip connection
54 |     output = tf.concat([_input,compOut],axis=2) #axis two is correct
55 |     
56 |     # This is a 1x1 convolution to follow up -- we can optionally turn this off
57 |     if bottle:
58 |         output = bottleneck(output,growth_rate) 
59 |     return output
60 | 
61 | def makeBlock(_input,growth_rate,num_layers,bottle=False):
62 |     """Bottle will significantly reduce the amount of computation needed."""
63 |     output = _input
64 |     for layer in range(num_layers):
65 |         with tf.variable_scope("layer_{}".format(layer)):
66 |             output = addInternalLayer(output,growth_rate,bottle=bottle)
67 |             print('output for internal layer', layer, output)
68 |     if bottle:
69 |         output= bottleneck(output, growth_rate)
70 |     return output
71 | 
72 | def transition_layer(_input,reduction=1):
73 |     out_size = int(int(_input.get_shape()[-1]) * reduction)
74 |     output = composite_function(_input,out_size,width=1)
75 |     output = avg_pool1d(output,2)
76 |     return output
77 | 
78 | def transition_to_vector(_input):
79 |     '''
80 |     Transforms the last block into a single vector by avg_pooling
81 |     '''
82 |     output =ln(_input)
83 |     output = tf.nn.relu(output)
84 |     last_pool_kernel = int(output.get_shape()[-2])
85 |     output = avg_pool1d(output,last_pool_kernel)
86 |     output = tf.squeeze(output,axis=1)
87 |     return output
88 | 
89 | 
90 | 
91 | 


--------------------------------------------------------------------------------
/models/densenet/ops_tests.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | from models.densenet import ops
 3 | class OpsTests(tf.test.TestCase):
 4 |     def setUp(self):
 5 |         self.batch_size= 10
 6 |         self.sequence_length = 256
 7 |         self.hidden_size = 8
 8 |         self.data = tf.ones(dtype=tf.float32,shape=[self.batch_size,self.sequence_length,self.hidden_size])
 9 |     def test_avg1d(self):
10 |         with self.test_session():
11 |             output = ops.avg_pool1d(self.data,2)
12 |             self.assertEqual([self.batch_size,self.sequence_length/2,self.hidden_size],output.get_shape().as_list())
13 | 
14 |             output = ops.avg_pool1d(self.data, self.sequence_length)
15 |             self.assertEqual([self.batch_size, 1, self.hidden_size],
16 |                              output.get_shape().as_list())
17 |     def test_composite_function(self):
18 |         with self.test_session():
19 |             with tf.variable_scope("case1"):
20 |                 output = ops.composite_function(self.data,outSize=self.hidden_size,width=3)
21 |                 self.assertEqual([self.batch_size, self.sequence_length , self.hidden_size],
22 |                              output.get_shape().as_list())
23 |         with tf.variable_scope("case2"):
24 |             output = ops.composite_function(self.data,outSize=self.hidden_size,width=5)
25 |             self.assertEqual([self.batch_size, self.sequence_length , self.hidden_size],
26 |                              output.get_shape().as_list())
27 |         with tf.variable_scope("case3"):
28 |             output = ops.composite_function(self.data,outSize=self.hidden_size//2,width=5)
29 |             self.assertEqual([self.batch_size, self.sequence_length , self.hidden_size//2],
30 |                              output.get_shape().as_list())
31 | 
32 |     def test_conv1d(self):
33 |         with self.test_session():
34 |             output = ops.conv1d(self.data, outSize=self.hidden_size, width=3)
35 |             self.assertEqual([self.batch_size, self.sequence_length, self.hidden_size],
36 |                              output.get_shape().as_list())
37 | 
38 |     def test_bottleneck(self):
39 |         with self.test_session():
40 |             with tf.variable_scope("bottlneck_case"):
41 |                 output = ops.bottleneck(self.data,4)
42 |                 self.assertEqual([self.batch_size, self.sequence_length, 16],
43 |                                  output.get_shape().as_list())
44 | 
45 |     def test_makeblock(self):
46 |         with self.test_session():
47 |             with tf.variable_scope("block_layer"):
48 |                 growth_rate = 8
49 |                 num_layers=5
50 |                 output = ops.makeBlock(self.data,growth_rate,num_layers,False)
51 | 
52 |                 self.assertEqual([self.batch_size, self.sequence_length, self.hidden_size+(growth_rate*num_layers)],
53 |                                  output.get_shape().as_list())
54 | 
55 |             with tf.variable_scope("block_layer_bottled"):
56 |                 output = ops.makeBlock(self.data,growth_rate,num_layers,True)
57 |                 self.assertEqual([self.batch_size, self.sequence_length, growth_rate*4],
58 |                                  output.get_shape().as_list())
59 | 
60 | 
61 |     def test_transition_layer(self):
62 |         output = ops.transition_layer(self.data,1)
63 |         self.assertEqual([self.batch_size, self.sequence_length//2, self.hidden_size],
64 |                                  output.get_shape().as_list())
65 |         with tf.variable_scope("compression_case"):
66 |             output = ops.transition_layer(self.data,0.5)
67 |             self.assertEqual([self.batch_size, self.sequence_length//2, self.hidden_size//2],
68 |                                      output.get_shape().as_list())
69 | 
70 | 
71 |     def test_transition_to_vector(self):
72 |         output = ops.transition_to_vector(self.data)
73 |         self.assertEqual([self.batch_size,self.hidden_size],output.get_shape().as_list())
74 | 
75 | if __name__ == '__main__':
76 |     tf.test.main()
77 | 


--------------------------------------------------------------------------------
/models/q1_lstm_baseline.py:
--------------------------------------------------------------------------------
 1 | from .base import ModelBase
 2 | import tensorflow as tf
 3 | class LSTMBaseline(ModelBase):
 4 |     def __init__(self,args):
 5 |         super(LSTMBaseline, self).__init__(args)
 6 |     def get_logits(self):
 7 |         '''
 8 |         1) Fill this function. Use tf.nn.bidirectional_dynamic_rnn with an LSTM or GRU to generate logits for each point
 9 |         in the sequence.
10 | 
11 | 
12 |         2) Bonus infer the sequence length using the sign trick (As done in ModelBase for the weights) to make the LSTM
13 |             run faster.
14 |         3) Time how long it takes for the model to process 100 batches. What loss do you get ?
15 |         :return: logits = tf.contrib.layers.linear(result, num_outputs=128)
16 |         '''
17 | 
18 |         pass
19 | 
20 | 


--------------------------------------------------------------------------------
/models/q2_simple_conv.py:
--------------------------------------------------------------------------------
 1 | from .base import ModelBase
 2 | import tensorflow as tf
 3 | 
 4 | 
 5 | class SimpleConv(ModelBase):
 6 |     def __init__(self, args):
 7 |         super(SimpleConv, self).__init__(args)
 8 | 
 9 |     def conv1d(self,input_):
10 |         '''
11 |             Write your own conv1d implementation such that you can pass the input into tensorflow conv2d function
12 |             Make sure the height of your filters makes sense
13 |             What kind of padding should you use (VALID or SAME) For this task ?
14 |             result = tf.nn.conv2d()
15 | 
16 |         '''
17 | 
18 | 
19 |     def get_logits(self):
20 |         '''
21 |         1) Use the conv1d op you implemented to run a convolution over the input
22 |         2) Don't forget to use an activation function such as tf.nn.relu
23 |         3) Bonus if you stack a few conv layers one atop the other
24 |         4) What is the receptive field of your output layer ?
25 |         :return: logits = tf.contrib.layers.linear(result, num_outputs=128)
26 |         '''
27 | 
28 |         pass
29 | 
30 | 


--------------------------------------------------------------------------------
/models/q3_ResidualConvs.py:
--------------------------------------------------------------------------------
 1 | from .base import ModelBase
 2 | import tensorflow as tf
 3 | 
 4 | 
 5 | class ResidualConv(ModelBase):
 6 |     def __init__(self, args):
 7 |         super(ResidualConv, self).__init__(args)
 8 | 
 9 |     def conv1d(self, input_):
10 |         '''
11 |             Write your own conv1d implementation such that you can pass the input into tensorflow conv2d function
12 |             Make sure the height of your filters makes sense
13 |             What kind of padding should you use (VALID or SAME) For this task ?
14 |             result = tf.nn.conv2d()
15 | 
16 |         '''
17 |     def get_logits(self):
18 |         '''
19 |         1) Use the conv1d op you implemented to run a convolution over the input
20 |         2) This time stack at least 20 layers one on top of the other
21 |         3) Use residual connections to preserve gradients, that is, concatenate the input and output at each layer
22 |         4) What is the receptive field of your model now?
23 |         5) What is the size of the final representation of each "charecter" ?
24 |         :return: logits = tf.contrib.layers.linear(result, num_outputs=128)
25 |         '''
26 | 
27 |         pass
28 | 
29 | 


--------------------------------------------------------------------------------
/models/q4_DilatedConv.py:
--------------------------------------------------------------------------------
 1 | from .base import ModelBase
 2 | import tensorflow as tf
 3 | 
 4 | 
 5 | class DilatedConv(ModelBase):
 6 |     def __init__(self, args):
 7 |         super(DilatedConv, self).__init__(args)
 8 | 
 9 |     def dilated_conv1d(self, input_,dillation_rate):
10 |         '''
11 |             Write your own dilated_conv1d implementation such that you can pass the input into tensorflow dilated_conv2d function
12 | 
13 |         '''
14 |     def get_logits(self):
15 |         '''
16 |         1) Stack the dilated conv1d you wrote above to get a receptive field of 128 characters
17 |         2) Bonus: Concatenate the outputs of successive layers to form residual connections
18 |         :return: logits = tf.contrib.layers.linear(result, num_outputs=128)
19 |         '''
20 | 
21 |         pass
22 | 
23 | 


--------------------------------------------------------------------------------
/models/q5_deconvolution_autoencoder.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | '''
 3 | In this task you'll implemented a convolutional autoencoder for text. This is particularly useful for reducing a document
 4 | to a vector.
 5 | You'll need to implement an encoder function witch applies successive 1d residual convolutions and 1d pooling operations
 6 |  Then you'll implement a decoder which takes the resulting vector and expands it back to the size of the original sequence
 7 | 
 8 | '''
 9 | class DeconvAutoencoder():
10 |     def __init__(self,args):
11 |         self.args = args
12 |         self.original = tf.placeholder(dtype=tf.int32,shape=[None,None])
13 |         self.lower = tf.placeholder(dtype=tf.int32, shape=[None, None])
14 |         self.embedded_source = self._embed_chars()
15 |         logits = self.get_logits()
16 |         self.loss_op = self._loss(logits)
17 |         self.train_op = self._train(self.loss_op)
18 |         self.predictions = self._prediction(logits)
19 |     def get_logits(self):
20 |         encoded = self.encoder(self.embedded_source)
21 |         decoded = self.decoder(encoded)
22 |         logits = tf.contrib.layers.linear(decoded, num_outputs=128)
23 |         return logits
24 |     def avg_pool1d(self,input_):
25 |         '''
26 |         Implement 1d average pooling by wrapping tensorflows avg_pooling function.
27 |         :return:
28 |         '''
29 |     def conv_1d_transpose(self,input_,**kwargs):
30 |         '''
31 |         Implement 1d "deconvolution" by wrapping tensorflows conv2d_transpose function.
32 |         :return:
33 |         '''
34 | 
35 |     def residual_conv_layer_1d(self,input_,**kwargs):
36 |         '''
37 |         Implement a series of stacked 1d convolutions with residual connections (That is concatenate the input and output
38 |         at each layer)
39 |         :return:
40 |         '''
41 |     def encoder(self,input_):
42 |         '''
43 |         Implement DenseNet (more or less). Use successive residual and pooling layers to reduce a sentance to a vector
44 |          representation. Assume the input length is padded/truncated to 512 charecters
45 |         :return:
46 |         '''
47 |     def decoder(self,input_):
48 |         '''
49 |         Implement DenseNet  (this time backward). Use successive residual and deconvolution layers to expand a vector
50 |         to a sentence
51 |         :return:
52 |         '''
53 | 
54 | 
55 |     def _embed_chars(self):
56 |         embedding_size = 4
57 |         embedding_matrix = tf.get_variable("embedding_matrix", shape=[128, embedding_size],
58 |                                            dtype=tf.float32)
59 | 
60 |         return tf.nn.embedding_lookup(embedding_matrix,self.original)
61 | 
62 | 
63 | 
64 |     def _loss(self,logits):
65 |         lengths = tf.reduce_sum(tf.sign(self.original),axis=1)
66 |         maxlen = self.original.get_shape().as_list()[1]
67 |         mask = tf.sequence_mask(lengths,dtype=tf.float32,maxlen=500)
68 |         loss = tf.contrib.seq2seq.sequence_loss(logits=logits,
69 |                                                 targets=self.original,
70 |                                                 weights = mask
71 |                                                 )
72 |         return loss
73 | 
74 |     def _train(self,loss):
75 |         #lr = tf.train.exponential_decay(learning_rate=self.args.lr)
76 |         lr = self.args.lr
77 |         opt = tf.train.AdamOptimizer(lr)
78 |         return opt.minimize(loss)
79 | 
80 |     def _prediction(self,logits):
81 |         sm = tf.nn.softmax(logits)
82 |         preds = tf.argmax(sm,axis=2)
83 |         return preds
84 | 
85 | 


--------------------------------------------------------------------------------
/models/small_conv.py:
--------------------------------------------------------------------------------
 1 | from .base import ModelBase
 2 | import tensorflow as tf
 3 | from .densenet import ops
 4 | class SmallConv(ModelBase):
 5 |     def __init__(self,args):
 6 |         super(SmallConv, self).__init__(args)
 7 |     def get_logits(self):
 8 |         encoded = ops.makeBlock(self.embedded_source, growth_rate=16, num_layers=10)
 9 |         logits = tf.contrib.layers.linear(encoded, num_outputs=128)
10 |         return logits
11 | 
12 | 


--------------------------------------------------------------------------------
/prepare_data.py:
--------------------------------------------------------------------------------
 1 | import codecs
 2 | import nltk
 3 | import string
 4 | import numpy as np
 5 | import os
 6 | import re
 7 | class DataLoader():
 8 |     def __init__(self,args):
 9 |         self.args=args
10 |         self.vocab_size=max(map(lambda x:ord(x),string.printable))
11 |         self.PAD = 0
12 |         self.PAD_CHAR =chr(0)
13 |         self.reg = re.compile(r'[\.,\?!;]')
14 | 
15 |     def replace_punc(self,txt):
16 | 
17 |         return self.reg.sub('',txt)
18 |     def load_txt_data(self):
19 |         with codecs.open(self.args.data_path,encoding='utf8') as f:
20 |             txt = f.read()
21 |             sentances = txt.split('\n')#nltk.sent_tokenize(txt)
22 |             sentances = list(map(lambda x:str(x.encode('ascii',errors='ignore')),sentances))
23 |             max_len = max(map(len,sentances))
24 |             sentances = map(lambda x: x.ljust(max_len,self.PAD_CHAR)[:max_len],sentances)
25 |             org_lower = list(map(lambda x: (x,self.replace_punc(x.lower()).ljust(max_len,self.PAD_CHAR)[:max_len]),sentances))
26 |         return org_lower
27 | 
28 |     @staticmethod
29 |     def is_pure_ascii(sent):
30 |         set(sent).issubset(string.printable)
31 | 
32 | 
33 |     def txt_to_array(self,org_lower):
34 |         temp_org,temp_low =[], []
35 |         for (o,l) in org_lower:
36 |             temp_org.append(np.array(list(map(ord,o))))
37 |             temp_low.append(np.array(list(map(ord, l))))
38 |         org = np.stack(temp_org)
39 |         low = np.stack(temp_low)
40 |         return np.stack([org,low],axis=1) # [num_sent,2,max_len]
41 | 
42 |     def load_data(self):
43 |         if os.path.exists(self.args.saved_data_path):
44 |             self.dataset = np.load(self.args.saved_data_path)
45 | 
46 |         else:
47 |             org_low = self.load_txt_data()
48 |             self.dataset = self.txt_to_array(org_low)
49 |             with open(self.args.saved_data_path,'wb') as f:
50 |                 np.save(f,self.dataset)
51 | 
52 | 
53 |     def get_batch(self):
54 |         start=0
55 |         end = start+self.args.batch_size
56 |         np.random.shuffle(self.dataset)
57 |         while start < len(self.dataset):
58 |             batch = self.dataset[start:end]
59 |             max_len =np.max(np.sum(np.sign(batch[:,0,:]),axis=1))
60 |             original = batch[:,0,: max_len]
61 |             lowered = batch[:, 1,: max_len]
62 |             yield original,lowered
63 |             start+=self.args.batch_size
64 |             end  += self.args.batch_size
65 | 
66 |     def ar_to_str(self,ar):
67 |         return ''.join(map(chr, ar))
68 | 
69 | 
70 | 
71 | 
72 | 
73 | 
74 | 


--------------------------------------------------------------------------------
/presentation.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/talolard/CMFT_PyCon2017/7e8db3a3a27baf44e97ee662a6874f42c38fa6ea/presentation.pdf


--------------------------------------------------------------------------------
/test_dataLoader.py:
--------------------------------------------------------------------------------
 1 | from unittest import TestCase
 2 | import sys
 3 | from arg_parse import parse_args
 4 | from prepare_data import DataLoader
 5 | 
 6 | 
 7 | class TestDataLoader(TestCase):
 8 |     def setUp(self):
 9 |         self.args = parse_args(sys.argv[3:])
10 |         self.args.data_path='data/test/input.txt'
11 |         self.args.saved_data_path = 'data/test/dataset.npa'
12 |         self.DL = DataLoader(self.args)
13 |     def test_load_txt_data(self):
14 |         org_low = self.DL.load_txt_data()
15 | 
16 |     def test_txt_to_array(self):
17 |         org_lower = self.DL.load_txt_data()
18 |         dataset = self.DL.txt_to_array(org_lower)
19 |         self.assertEqual(dataset.shape,(len(org_lower),2,self.DL.max_len))
20 | 
21 |     def test_load_data(self):
22 |         self.DL.load_data()
23 | 
24 |     def test_get_batch(self):
25 |         self.DL.load_data()
26 |         for num,(orig,low) in enumerate(self.DL.get_batch()):
27 |             self.assertLess(num,1000)
28 |         print(''.join(map(chr,orig[0])))
29 |         print(''.join(map(chr, low[0])))
30 | 
31 | 


--------------------------------------------------------------------------------
/train.py:
--------------------------------------------------------------------------------
 1 | import tensorflow as tf
 2 | import sys
 3 | from arg_parse import parse_args
 4 | from models.small_conv import SmallConv as Model
 5 | from prepare_data import DataLoader
 6 | import re
 7 | import numpy as np
 8 | def main(args):
 9 |     args = parse_args(args)
10 |     DL = DataLoader(args)
11 |     print("Loading data")
12 |     DL.load_data()
13 |     print("Done")
14 | 
15 | 
16 |     with tf.Session() as sess:
17 |         print("Loading model")
18 |         model = Model(args)
19 |         init = tf.global_variables_initializer()
20 |         print("Number of trainable vars is {}".format(np.sum([np.prod(v.get_shape().as_list()) for v in tf.trainable_variables()])))
21 |         sess.run(init)
22 |         for epoch in range(10):
23 |             for num,batch in enumerate(DL.get_batch()):
24 |                 orig,lower = batch
25 |                 feed_dict = {model.original:orig,
26 |                              model.lower:lower,
27 | 
28 |                              }
29 |                 fetches = [model.train_op,model.loss_op,model.predictions]
30 |                 _,loss,preds = sess.run(fetches=fetches,feed_dict=feed_dict)
31 |                 if num% 25 ==0:
32 |                     print(loss)
33 |                 if num %100 ==0:
34 |                     print('************************')
35 |                     print(DL.ar_to_str(orig[0]))
36 |                     print(DL.ar_to_str(preds[0]))
37 |                     print(DL.ar_to_str(lower[0]))
38 | if __name__ =='__main__':
39 |     main(sys.argv[1:])
40 | 
41 | 


--------------------------------------------------------------------------------