\n
Initialization
\n
Import necessary libraries and initialize the SparkContext.
\n
\n
Data Preparation and Preprocessing
\n
We first take a look at the unzipped 20news dataset.
\n
\n
Read the texts of each news category as a TextSet.
\n
\n
You can easily call transformation methods of a TextSet one by one to build the text transformation pipeline and do text preprocessing in a distributed fashion.
\n
\n
We split the preprocessed TextSet into training and validation set.
\n
\n
Model Definition and Training
\n
We use built-in TextClassifier model in Analytics Zoo with CNN as encoder. You can try to use LSTM and GRU as well.
\n
\n
After creating the model, we use Keras-Style API to train the model in a distributed fashion.
\n
The training may take several minutes.
\n
\n
Model Evaluation and Inference
\n
We use the validation dataset to evaluate the trained model. You can observe around 85% accuracy after 20 epochs of training.
\n
\n
Finally print out some prediction results to have a look.
\n
\n
In conclusion, this notebook guides you through the steps to build a text classification model using Analytics Zoo. The whole process should be straightforward and the code is neat.
\n
Besides text classification, many more advanced NLP models are supported in Analytics Zoo, including text matching, sequence to sequence, transformer and BERT. If you are interested in these, feel free to find the corresponding documentations on our website: https://analytics-zoo.github.io/
\n
\n
通过Analytics Zoo在Apache Spark上使用Keras/Tensorflow
\n
Keras/Tensorflow是受到广泛使用的机器学习库。在这个Notebook中，我们会看到如何通过Analytics Zoo，在Apache Spark上使用Keras/Tensorflow训练机器学习模型。
\n
准备数据
\n
我们使用MNIST数据。MNIST数据是一个面向初学者的手写数字识别数据集。从MNIST官方网站上下载数据。
\n
上传数据到COS
\n
COS是腾讯云上的对象存储系统。我们将刚才下载的四个文件上传到COS存储桶中，进行下一步的处理。
\n
\n
数据分析
\n
接下来可以对数据进行一些分析，例如不同类别的样本分布、显示出一些图片
\n
数据一览
\n