These objects are imported from other packages. Follow the links 88 | below to see their documentation.
%>%
Serve a TensorFlow Model into a local REST API.
106 | 107 | 108 |serve(model_path, host = "127.0.0.1", port = 8089, browse = interactive())109 | 110 | 111 |
Trains and saves MNIST using TensorFlow
109 | 110 | 111 |mnist_train_save(model_path, overwrite = FALSE)112 | 113 |
| model_path | 118 |Destination path where the model will be saved. |
119 |
|---|
Converts a TensorFlow SavedModel into a other model formats.
100 | 101 | 102 |convert_savedmodel(model_dir = NULL, format = c("tflite"), 103 | target = paste("savedmodel", format, sep = "."), 104 | signature_name = "serving_default", ...)105 | 106 |
| model_dir | 111 |The path to the exported model, as a string. |
112 |
|---|---|
| format | 115 |The target format for the converted model. Valid values
116 | are |
117 |
| target | 120 |The conversion target, currently only |
122 |
| signature_name | 125 |The named entry point to use in the model for prediction. |
126 |
| ... | 129 |Additional arguments. See |
131 |
Serve a TensorFlow SavedModel as a local web api under 88 | http://localhost:8089.
89 | 90 | 91 |serve_savedmodel(model_dir, host = "127.0.0.1", port = 8089, 92 | daemonized = FALSE, browse = !daemonized)93 | 94 |
| model_dir | 99 |The path to the exported model, as a string. |
100 |
|---|---|
| host | 103 |Address to use to serve model, as a string. |
104 |
| port | 107 |Port to use to serve model, as numeric. |
108 |
| daemonized | 111 |Makes 'httpuv' server daemonized so R interactive sessions 112 | are not blocked to handle requests. To terminate a daemonized server, call 113 | 'httpuv::stopDaemonizedServer()' with the handle returned from this call. |
114 |
| browse | 117 |Launch browser with serving landing page? |
118 |
# NOT RUN { 128 | # serve an existing model over a web interface 129 | tfdeploy::serve_savedmodel( 130 | system.file("models/tensorflow-mnist", package = "tfdeploy") 131 | ) 132 | # }133 |
Converts a model to TensorFlow Lite.
100 | 101 | 102 |# S3 method for tflite_conversion 103 | convert_savedmodel(model_dir = NULL, 104 | target = "savedmodel.tflite", signature_name = "serving_default", 105 | inference_type = "FLOAT", quantized_input_stats = NULL, 106 | drop_control_dependency = TRUE, ...)107 | 108 |
| model_dir | 113 |The path to the exported model, as a string. |
114 |
|---|---|
| target | 117 |The conversion target, currently only |
119 |
| signature_name | 122 |The named entry point to use in the model for prediction. |
123 |
| quantized_input_stats | 126 |For each member of input_tensors the mean and
127 | std deviation of training data. Only needed |
129 |
| ... | 132 |Additional arguments. See |
134 |
| drop_control_dependency: | 137 |Drops control dependencies silently. This is 138 | due to tf lite not supporting control dependencies. |
139 |
Performs a prediction using a Web API providing a SavedModel.
88 | 89 | 90 |# S3 method for webapi_prediction 91 | predict_savedmodel(instances, model, ...)92 | 93 |
| instances | 98 |A list of prediction instances to be passed as input tensors 99 | to the service. Even for single predictions, a list with one entry is expected. |
100 |
|---|---|
| model | 103 |The model as a local path, a REST url, CloudML name or graph object. 104 |A local path can be exported using Notice that predicting over a CloudML model requires a A |
113 |
| ... | 116 |See #' @section Implementations:
|
126 |
Performs a prediction using a CloudML model.
88 | 89 | 90 |# S3 method for cloudml_prediction 91 | predict_savedmodel(instances, model, 92 | version = NULL, ...)93 | 94 |
| instances | 99 |A list of prediction instances to be passed as input tensors 100 | to the service. Even for single predictions, a list with one entry is expected. |
101 |
|---|---|
| model | 104 |The model as a local path, a REST url, CloudML name or graph object. 105 |A local path can be exported using Notice that predicting over a CloudML model requires a A |
114 |
| version | 117 |The version of the CloudML model. |
118 |
| ... | 121 |See #' @section Implementations:
|
131 |
Performs a prediction using a locally exported SavedModel.
88 | 89 | 90 |# S3 method for export_prediction 91 | predict_savedmodel(instances, model, 92 | signature_name = "serving_default", ...)93 | 94 |
| instances | 99 |A list of prediction instances to be passed as input tensors 100 | to the service. Even for single predictions, a list with one entry is expected. |
101 |
|---|---|
| model | 104 |The model as a local path, a REST url, CloudML name or graph object. 105 |A local path can be exported using Notice that predicting over a CloudML model requires a A |
114 |
| signature_name | 117 |The named entry point to use in the model for prediction. |
118 |
| ... | 121 |See #' @section Implementations:
|
131 |
Loads a SavedModel using the given TensorFlow session and 88 | returns the model's graph.
89 | 90 | 91 |load_savedmodel(sess, model_dir = NULL)92 | 93 |
| sess | 98 |The TensorFlow session. |
99 |
|---|---|
| model_dir | 102 |The path to the exported model, as a string. Defaults to 103 | a "savedmodel" path or the latest training run. |
104 |
Loading a model improves performance over multiple predict_savedmodel()
110 | calls.
export_savedmodel(), predict_savedmodel()
# NOT RUN { 119 | # start session 120 | sess <- tensorflow::tf$Session() 121 | 122 | # preload an existing model into a TensorFlow session 123 | graph <- tfdeploy::load_savedmodel( 124 | sess, 125 | system.file("models/tensorflow-mnist", package = "tfdeploy") 126 | ) 127 | 128 | # perform prediction based on a pre-loaded model 129 | tfdeploy::predict_savedmodel( 130 | list(rep(9, 784)), 131 | graph 132 | ) 133 | 134 | # close session 135 | sess$close() 136 | # }138 |137 |
Performs a prediction using a SavedModel model already loaded using
88 | load_savedmodel().
# S3 method for graph_prediction 92 | predict_savedmodel(instances, model, sess, 93 | signature_name = "serving_default", ...)94 | 95 |
| instances | 100 |A list of prediction instances to be passed as input tensors 101 | to the service. Even for single predictions, a list with one entry is expected. |
102 |
|---|---|
| model | 105 |The model as a local path, a REST url, CloudML name or graph object. 106 |A local path can be exported using Notice that predicting over a CloudML model requires a A |
115 |
| sess | 118 |The active TensorFlow session. |
119 |
| signature_name | 122 |The named entry point to use in the model for prediction. |
123 |
| ... | 126 |See #' @section Implementations:
|
136 |
Runs a prediction over a saved model file, local service or cloudml model.
88 | 89 | 90 |predict_savedmodel(instances, model, ...)91 | 92 |
| instances | 97 |A list of prediction instances to be passed as input tensors 98 | to the service. Even for single predictions, a list with one entry is expected. |
99 |
|---|---|
| model | 102 |The model as a local path, a REST url, CloudML name or graph object. 103 |A local path can be exported using Notice that predicting over a CloudML model requires a A |
112 |
| ... | 115 |See #' @section Implementations: |
125 |
export_savedmodel(), serve_savedmodel(), load_savedmodel()
# NOT RUN { 135 | # perform prediction based on an existing model 136 | tfdeploy::predict_savedmodel( 137 | list(rep(9, 784)), 138 | system.file("models/tensorflow-mnist", package = "tfdeploy") 139 | ) 140 | # }142 |141 |
The first step towards deploying a trained model is to export this model from TensorFlow. This guide demostrates how to export SavedModels from: TensorFlow, tfestimators and keras using export_savedmodel()
Exporting a SavedModel using the core TensorFlow API requires input and output tensors to be passed to export_savedmodel().
For instance, one can train MNIST as described by MNIST For ML Beginners, track the model’s inputs and outputs named x and y under that particular article and call export_savedmodel() as follows:
export_savedmodel(
86 | sess,
87 | "tensorflow-mnist",
88 | inputs = list(images = x),
89 | outputs = list(scores = y)
90 | )For convinience, a sample training script is included in tfdeploy to train and export this MNIST model as follows:
tfruns::training_run(
93 | system.file("models/tensorflow-mnist.R", package = "tfdeploy")
94 | )A sample model using the mtcars data frame is trained using tfestimators. The resulting model is the one that will be saved to disk. To train, we can follow the TensorFlow estimators Quick Start followed by running:
export_savedmodel(
101 | model,
102 | "tfestimators-mtcars"
103 | )For convinience, a sample training script is included in tfdeploy to train and export this mtcars model as follows:
tfruns::training_run(
106 | system.file("models/tfestimators-mtcars.R", package = "tfdeploy")
107 | )To export from Keras, first train a keras model as described under R interface to Keras, set backend()$set_learning_phase(TRUE) before training and then export this model as follows:
export_savedmodel(
114 | model,
115 | "keras-mnist"
116 | )For convinience, a sample training script is included in tfdeploy to train and export this MNIST model as follows:
tfruns::training_run(
119 | system.file("models/keras-mnist.R", package = "tfdeploy")
120 | )Tensorflow Serving provides support to serve efficiently and at scale TensorFlow model. This guide provides an overview of installation and serving the models exported in the ‘Exporting Models’ section.
83 |See Tensorflow Serving Setup, but in general, from Linux, first install prereqs:
88 |sudo apt-get update && sudo apt-get install -y build-essential curl libcurl3-dev git libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python-dev python-numpy python-pip software-properties-common swig zip zlib1g-devThen install Tensorflow Serving:
90 |echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
91 |
92 | curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
93 |
94 | sudo apt-get update && sudo apt-get install tensorflow-model-serverOptionally, install the api client:
96 |sudo pip install tensorflow-serving-api --no-cache-dirThen serve the TensorFlow model using:
98 |tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/mnt/hgfs/tfserve/trained/tensorflow-mnist2017-10-04 14:53:15.291409: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:284] Loading SavedModel: success. Took 113175 microseconds.
100 | 2017-10-04 14:53:15.293200: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: mnist version: 1}
101 | 2017-10-04 14:53:15.299338: I tensorflow_serving/model_servers/main.cc:288] Running ModelServer at 0.0.0.0:9000 ...Or the tfestimators model using:
tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/mnt/hgfs/tfserve/trained/tfestimators-mtcars2017-10-04 15:33:21.279211: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:284] Loading SavedModel: success. Took 43820 microseconds.
105 | 2017-10-04 15:33:21.281129: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: mnist version: 1507156394}
106 | 2017-10-04 15:33:21.284161: I tensorflow_serving/model_servers/main.cc:288] Running ModelServer at 0.0.0.0:9000 ...
107 | One can use saved_model_cli to inspect model contents, as in:
saved_model_cli show --dir /mnt/hgfs/tfserve/trained/tensorflow-mnist/1Manually download mnist_client.py and mnist_input_data.py. Then run:
115 |python mnist_client.py --num_tests=1000 --server=localhost:9000
116 |
117 | Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
118 | Extracting /tmp/train-images-idx3-ubyte.gz
119 | Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
120 | Extracting /tmp/train-labels-idx1-ubyte.gz
121 | Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
122 | Extracting /tmp/t10k-images-idx3-ubyte.gz
123 | Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
124 | Extracting /tmp/t10k-labels-idx1-ubyte.gz
125 | ........................................
126 | Inference error rate: 9.5%Or while running the Keras model under TF Serving, :
128 |python mnist_client.py --num_tests=1000 --server=localhost:9000
129 |
130 | Extracting /tmp/train-images-idx3-ubyte.gz
131 | Extracting /tmp/train-labels-idx1-ubyte.gz
132 | Extracting /tmp/t10k-images-idx3-ubyte.gz
133 | Extracting /tmp/t10k-labels-idx1-ubyte.gz
134 | ............
135 | Inference error rate: 84.5%TODO: Investigate inference error rate under keras, see: /keras/issues/7848.
137 |The main goal of tfdeploy is to create models in R and then export, test, and deploy those models to environments without R. However, there may be cases when it makes sense to use a saved model from R:
73 |One way to use a deployed model from R would be to execute HTTP requests using a package like httr. For non-deployed models, it is possible to use serve_savedmodel - as we did for local testing - along with a tool like httr. However, there is an easier way to make predictions from a saved model using the predict_savedmodel function.
Using the same MNIST model described previously, we can easily make predictions for new pre-processed images. For example, we can load the MNIST test data set and predict a digit for the first 10 images:
83 |library(keras)
84 | library(tfdeploy)
85 |
86 | test <- dataset_mnist()$test$x
87 | processed_test <- array_reshape(test, dim = c(nrow(test), 784)) / 255
88 | processed_test_list <- lapply(1:10, function(i) {processed_test[i,]})
89 |
90 | predict_savedmodel(processed_test_list, '../savedmodel/')Prediction 1:
92 | $prediction
93 | [1] 3.002971e-37 8.401216e-29 2.932129e-24 4.048731e-22 0.000000e+00 9.172148e-37
94 | [7] 0.000000e+00 1.000000e+00 4.337524e-31 1.772979e-17
95 |
96 | Prediction 2:
97 | $prediction
98 | [1] 0.000000e+00 4.548326e-22 1.000000e+00 2.261879e-31 0.000000e+00 0.000000e+00
99 | [7] 0.000000e+00 0.000000e+00 2.390626e-38 0.000000e+00
100 |
101 | ...A few things to keep in mind:
103 |predict_savedmodel expects the new instances to already be pre-processed.predict_savedmodel requires the new data to be in a list, and it always returns a list. This requirement faciliates models with more complex inputs or ouputs.In this previous example we used predict_savedmodel with the directory, ‘savedmodel’, which was created with the export_savedmodel command. In addition to providing a path to a saved model directory, predict_savedmodel can also be used with a deployed model by supplying a REST URL, a CloudML model by supplying a CloudML name and version, or by supplying a graph object loaded with load_savedmodel.
load_savedmodel can be used alongside of predict_savedmodel if the prediction function will be called multiple times. For example:
predict_savedmodel(instances, 'savedmodel')
112 |
113 | # OR
114 | sess <- tensorflow::tf$Session()
115 | graph <- load_savedmodel(sess, '../savedmodel')
116 | predict_savedmodel(instances, graph)There are many different ways to interact with models from R. Models can be developed directly in-memory. Keras models can be saved and reloaded from HDF5 files, serialized and saved as R data files, or model configurations can be saved to JSON or YAML. TensorFlow, Keras, and tfestimator models can be saved and exported with tfdeploy.
The correct option depends on the task and your goals.
123 || Model | 131 |Can Be Re-trained | 132 |Predictions | 133 |
|---|---|---|
| In memory keras, tfestimators, or tensorflow model | 137 |Yes | 138 |Locally, with predict
139 | |
140 |
| keras model saved in hdf5 | 143 |Yes | 144 |Locally, with predict after load_model_hdf5
145 | |
146 |
| keras model serialized to R object | 149 |Yes | 150 |Locally, with predict after unserialzing |
151 |
| tfdeploy saved model from disk 154 | | 155 |No | 156 |Locally with predict_savedmodel(..., <file path>)
157 | |
158 |
| tfdeploy saved model deployed on CloudMl or RStudio Connect | 161 |No | 162 |Scoring occurs remotely with HTTP requests or predict_savedmodel(..., <url>)
163 | |
164 |
The main goal of tfdeploy is to create models in R and then export, test, and deploy those models to environments without R. However, there may be cases when it makes sense to use a saved model from R:
68 |One way to use a deployed model from R would be to execute HTTP requests using a package like httr. For non-deployed models, it is possible to use serve_savedmodel - as we did for local testing - along with a tool like httr. However, there is an easier way to make predictions from a saved model using the predict_savedmodel function.
Using the same MNIST model described previously, we can easily make predictions for new pre-processed images. For example, we can load the MNIST test data set and create predictions for the first 10 images:
78 |library(keras)
79 | library(tfdeploy)
80 |
81 | test <- dataset_mnist()$test$x
82 | processed_test <- array_reshape(test, dim = c(nrow(test), 784)) / 255
83 | processed_test_list <- lapply(1:10, function(i) {processed_test[i,]})
84 |
85 | predict_savedmodel(processed_test_list, '../savedmodel/')Prediction 1:
87 | $prediction
88 | [1] 3.002971e-37 8.401216e-29 2.932129e-24 4.048731e-22 0.000000e+00 9.172148e-37
89 | [7] 0.000000e+00 1.000000e+00 4.337524e-31 1.772979e-17
90 |
91 | Prediction 2:
92 | $prediction
93 | [1] 0.000000e+00 4.548326e-22 1.000000e+00 2.261879e-31 0.000000e+00 0.000000e+00
94 | [7] 0.000000e+00 0.000000e+00 2.390626e-38 0.000000e+00
95 |
96 | ...A few things to keep in mind:
98 |predict_savedmodel expects the new instance data to be pre-processed.predict_savedmodel requires the new data to be in a list, and it always returns a list. This requirement faciliates models with more complex inputs or ouputs.In the previous example we used predict_savedmodel with the directory, ‘savedmodel’, which was created with the export_savedmodel command. In addition to providing a path to a saved model directory, predict_savedmodel can also be used with a deployed model by supplying a REST URL, a CloudML model by supplying a CloudML name and version, or by supplying a graph object loaded with load_savedmodel.
The last option above references the load_savedmodel function. load_savedmodel should be used alongside of predict_savedmodel if you’ll be calling the prediction function multiple times. load_savedmodel effectively caches the model graph in memory and can speed up repeated calls to predict_savedmodel. This caching is useful, for example, in a Shiny application where user input would drive calls to predict_savedmodel.
# if there will only be one batch of predictions
106 | predict_savedmodel(instances, 'savedmodel')
107 |
108 | # if there will be multiple batches of predictions
109 | sess <- tensorflow::tf$Session()
110 | graph <- load_savedmodel(sess, '../savedmodel')
111 | predict_savedmodel(instances, graph)
112 | # ... more work ...
113 | predict_savedmodel(instances, graph)There are a few distinct ways that a model can be represented in R. The most straightforward representation is the in-memory, R model object. This object is what is created and used while developing and training a model.
119 |A second representation is the on-disk saved model. This representation of the model can be used by the *_savedmodel functions. As a special case, load_savedmodel creates a new R object pointing to the model graph. It is important to keep in mind that these saved models are not the full R model object. For example, you can not update or re-train a graph from a saved model.
Finally, for Keras models there are 2 other representations: HDF5 files and serialized objects. Each of these represenations captures the entire in-memory R object. For example, using save_model_hdf5 and then load_model_hdf5 will result in a model that can be updated or retrained.
If you are developing a model and have access to the in-memory R model object, you should use the model object for predictions using R’s predict function.
If you are developing a Keras model and would like to save the model for use in a different session, you should use the HDF5 file or serialize the model and then save it to an R data format like RDS.
126 |If you are going to deploy a model and want to test, you should export the model using export_savedmodel and then test with either serve_savedmodel and your HTTP client or predict_savedmodel.
If you are using R and want to create predictions from a deployed or saved model, and you don’t have access to the in-memory R model object, you should use predict_savedmodel.