├── README.md ├── conclusion ├── README.md └── media │ └── _placeholder ├── creating-machine-learning-pipelines ├── README.md ├── machine-learning-pipelines.md └── media │ ├── _placeholder │ ├── batch_score.png │ ├── batch_scoring_pipeline_details.png │ ├── data_prep.png │ ├── new_notebook.png │ ├── pipeline_details.png │ └── pipelines.png ├── data-acquisition-understanding ├── README.md ├── accessing-data.md ├── data-wrangling.md ├── loading-and-writing-data.md └── media │ ├── aldermen-data.png │ ├── azure-blobs-crime-data.png │ ├── create-workspace-sdk.png │ ├── crime-aldermen-joined.png │ ├── crime-append-cols.png │ ├── crime-append-rows.png │ ├── crime-date-formats-updated.png │ ├── crime-date-formats.png │ ├── crime-dirty-df.png │ ├── crime-dirty-head-5.png │ ├── crime-location-fill-errors.png │ ├── crime-location-fill-nulls.png │ ├── crime-location-imputed.png │ ├── crime-location-mean-latitude.png │ ├── crime-location-replace.png │ ├── crime-location-sample.png │ ├── data-profiles-output.png │ ├── download-directory-and-output.png │ ├── download-directory.png │ ├── download-file-and-output.png │ ├── dprep-add-column-using-expression.png │ ├── dprep-auto-read-parquet.png │ ├── dprep-auto-read-txt.png │ ├── dprep-auto-read-xlsx.png │ ├── dprep-builder-to-dataflow.png │ ├── dprep-builders-column-types.png │ ├── dprep-crime-txt-transformed.png │ ├── dprep-data-file-format.png │ ├── dprep-detect-file-format-dtypes.png │ ├── dprep-dflow-dtypes.png │ ├── dprep-dflow.png │ ├── dprep-load-crime-dirty-csv.png │ ├── dprep-load-crime-full-csv.png │ ├── dprep-read-csv-infer-column-types.png │ ├── dprep-read-csv-path-only.png │ ├── dprep-read-csv-skip-rows.png │ ├── dprep-read-json-flatten-nested-arrays.png │ ├── dprep-read-json.png │ ├── dprep-read-lines.png │ ├── dprep-read-sql-from-datasource.png │ ├── dprep-source-txt.png │ ├── dprep-transform-builder-new-dataflow.png │ ├── dprep-transform-crime-location-mean-latitude.png │ ├── dprep-transform-location-imputed.png │ ├── dprep-transform-location-sample.png │ ├── dprep-transform-split-columns.png │ ├── dprep-type-conversions-to-dataflow.png │ ├── dprep-write-dflow-with-write-step.png │ ├── dprep-write-error-assignment.png │ ├── dprep-write-initial-dflow.png │ ├── dprep-write-local-file-system.png │ ├── dprep-write-read-back-crime-txt.png │ ├── dprep-write-to-parquet-with-error.png │ ├── read-csv-from-datastore.png │ ├── read-sql-from-datastore.png │ ├── split-column-by-example.png │ ├── upload-command-and-output.png │ └── upload-files-and-output.png ├── data-science-lifecycle ├── README.md ├── media │ └── _placeholder └── tdsp-intro.md ├── devops-for-ai ├── README.md ├── e2e-pipeline-code-sample.md ├── media │ ├── Boards.png │ ├── Commits.png │ ├── CreateProject.png │ ├── Dashboards.png │ ├── DevOpsDashBoard.png │ ├── Files.png │ ├── Jenkins.png │ ├── MakingChanges.png │ ├── Overview.png │ ├── PipRepos.png │ ├── PostGates.png │ ├── Queries.png │ ├── ReleasePipeline.png │ ├── ReleasePipelineFiring.png │ ├── ReleaseTemplates.png │ ├── Repos.png │ ├── Stages.png │ ├── Tasks.png │ ├── TestPlans.png │ ├── TestPlansTab.png │ ├── azureml-data-collection-blob.png │ ├── azureml-data-collection-databricks-1.png │ ├── azureml-data-collection-databricks-2.png │ ├── azureml-data-collection-databricks-3.png │ ├── azureml-data-collection-enable.png │ ├── azureml-data-collection-powerbi-1.png │ ├── azureml-data-collection-powerbi-2.png │ ├── azureml-data-collection-powerbi-3.png │ ├── azureml-model-tagging.png │ ├── azureml-model-versioning.png │ ├── azureml-telemetry-azure-portal.png │ ├── azureml-telemetry-enable.png │ ├── mlops-connect-devops-to-aml.png │ └── pipelines.png ├── mlops-and-devops.md ├── model-version-management-code-sample.md ├── model-webservice-performance-telemetry.md ├── monitoring-data-and-telemetry-code-sample.md ├── monitoring-data-collection.md ├── post-deployment-monitoring-and-management.md └── using-azure-devops.md ├── intro ├── README.md ├── architecture-overview.md ├── environment-setup.md ├── media │ ├── Diagrams.pptx │ ├── _placeholder │ ├── ai-spectrum.png │ ├── aml-experiment-link.png │ ├── aml-experiment-run.png │ ├── aml-model-workflow.png │ ├── aml-workspace-add-compute-form.png │ ├── aml-workspace-add-compute.png │ ├── aml-workspace-script-cloud-shell.png │ ├── azure-cloud-shell.png │ ├── azure-create-resource.png │ ├── azure-machine-learning-taxonomy.png │ ├── azure-ml-service-workspace-create-dialog.png │ ├── azure-ml-service-workspace-deployment.png │ ├── azure-mlservice-workspace-search.png │ ├── azure-notebooks-add-environment-setup-step.png │ ├── azure-notebooks-clone-button.png │ ├── azure-notebooks-profile-link.png │ ├── azure-notebooks-profile.png │ ├── azure-notebooks-project-settings-button.png │ ├── azure-notebooks-terminal.png │ ├── azure-notebooks-upload-github-repo.png │ ├── new-notebook-vm-link.png │ ├── new-notebook-vm.png │ ├── notebook-cells.png │ ├── notebook-vm-jupyter-link.png │ ├── notebook-vm-stop.png │ ├── simple-experiment-run.png │ ├── steps-to-using-azureml.png │ ├── visual-interface-compute-target.png │ ├── visual-interface-create-kubernetes-compute-target.png │ ├── visual-interface-deploy-web-service.png │ ├── visual-interface-experiment.png │ ├── visual-interface-link.png │ ├── visual-interface-predictive-experiment.png │ ├── visual-interface-run.png │ ├── visual-interface-web-service-compute-target.png │ ├── visual-interface-web-service.png │ ├── vs-code-aml-extension-workspace.png │ ├── vs-code-aml-extension.png │ ├── vs-code-intellicode-extension.png │ ├── vs-code-python-interpreter-selected.png │ ├── vs-code-python-select-interpreter.png │ └── vscode-azure-ml-extension-installing.png ├── tools.md └── what-is-azure-machine-learning.md ├── model-deployment ├── README.md ├── batch-inferencing.md ├── deployment-target-options.md ├── deployment-with-onnx.md ├── iot-edge-inferencing.md ├── media │ ├── _placeholder │ ├── aci-deployment-output.png │ ├── aci-test-output-http.png │ ├── aci-test-output.png │ ├── aks-deployment-output.png │ ├── aks-test-output-http.png │ ├── aml-deployment-process.png │ ├── aml-workspace-compute-failed-details.png │ ├── aml-workspace-compute-failed.png │ ├── aml-workspace-deployment-details-aks.png │ ├── aml-workspace-deployment-details.png │ ├── aml-workspace-deployments-aci.png │ ├── aml-workspace-deployments-aks.png │ ├── azure-notebooks-ai-toolkit-iot-edge-run.png │ ├── azure-notebooks-cloned-repo.png │ ├── azure-notebooks-upload-github-repo.png │ ├── azure-portal-pipeline-run.png │ ├── cloud-shell-vm-usage.png │ ├── lambda-architecture.png │ ├── model-deployment-process.png │ ├── onnx_overview.png │ ├── pipeline-run-details.png │ ├── prediction-results.png │ ├── visual-interface-create-predictive-experiment.png │ ├── visual-interface-deploy-web-service-button.png │ ├── visual-interface-deploy-web-service-dialog.png │ ├── visual-interface-run-predictive-experiment.png │ ├── visual-interface-run-training-experiment.png │ ├── visual-interface-save-trained-model-link.png │ ├── visual-interface-save-trained-model.png │ ├── visual-interface-web-service-api-doc.png │ ├── visual-interface-web-service-consume.png │ ├── visual-interface-web-service-test.png │ └── visual-interface-web-services.png └── real-time-inferencing.md ├── modeling ├── README.md ├── aml-compute-options.md ├── aml-estimators.md ├── aml-experiment-runs.md ├── aml-model-registry.md ├── automl-classification-code-sample.md ├── automl-forecasting-code-sample.md ├── automl-regression-code-sample.md ├── automl-understand-models-with-explainability.md ├── capture-query-model-performance-with-aml-experiments.md ├── feature-engineering-training-evaluation-selection.md ├── measure-model-performance.md ├── measure-performance-regression-classification.md ├── media │ ├── _placeholder │ ├── aml-compute-create-in-portal.png │ ├── automl-block-algorithms.png │ ├── automl-classification-execution-metrics.png │ ├── automl-classification-execution-progress.png │ ├── automl-classification-execution-results.png │ ├── automl-explainability-architecture.png │ ├── automl-explainability-numeric.png │ ├── automl-explainability-visual-1.png │ ├── automl-explainability-visual-2.png │ ├── automl-forecasting-actual-evolution.png │ ├── automl-forecasting-execution-metrics.png │ ├── automl-forecasting-execution-progress.png │ ├── automl-forecasting-execution-results.png │ ├── automl-forecasting-testing.png │ ├── automl-how-it-works-simple.png │ ├── automl-how-it-works.png │ ├── automl-model-explainability-iris.png │ ├── automl-regression-execution-metrics.png │ ├── automl-regression-execution-progress.png │ ├── automl-regression-execution-results.png │ ├── automl-regression-testing-rmse.png │ ├── automl-regression-testing.png │ ├── azure-machine-learning-taxonomy-marked.png │ ├── azure-machine-learning-taxonomy.png │ ├── compute-target-in-portal.png │ ├── experiment-and-runs-in-portal.png │ ├── experiment-in-sdk-runs-with-children.png │ ├── experiment-in-sdk-runs.png │ ├── experiment-in-sdk.png │ ├── model-in-portal.png │ ├── model-in-sdk.png │ ├── model-logging-01.png │ ├── model-logging-02.png │ ├── model-logging-03.png │ ├── model-logging-04.png │ ├── model-logging-05.png │ ├── model-logging-sdk-01.png │ ├── model-logging-sdk-02.png │ ├── model-logging-sdk-03.png │ ├── model-logging-sdk-04.png │ ├── model-logging-sdk-05.png │ ├── model-performance-calibration.png │ ├── model-performance-gain.png │ ├── model-performance-lift.png │ ├── model-performance-mlflow.png │ ├── model-performance-precision-recall.png │ ├── model-performance-predicted-vs-true.png │ ├── model-performance-regression-run-metrics.png │ ├── model-performance-residual-histogram.png │ ├── model-performance-roc.png │ ├── model-properties-in-portal.png │ ├── model_monitoring_1.png │ ├── model_monitoring_2.png │ ├── model_perf_1.png │ ├── model_perf_2.png │ ├── model_perf_3.png │ ├── run-in-portal.png │ ├── run-in-sdk-details.png │ ├── runs-and-child-runs.png │ ├── vi_bdtr.png │ ├── vi_cmd.png │ ├── vi_dp_modules.png │ ├── vi_emv.png │ ├── vi_launch.png │ ├── vi_ml_modules.png │ ├── vi_overview.png │ ├── vi_run.png │ ├── vi_sem.png │ ├── vi_smv1.png │ ├── vi_smv2.png │ ├── vi_tc.png │ ├── vi_tm.png │ └── vi_traineval_modules.png ├── model-evaluation.md ├── model-training.md ├── simplify-process-with-automated-ml.md ├── training-evaluating-model-with-visual-interface.md └── training-evaluating-simple-models-with-aml-compute.md └── template.md /README.md: -------------------------------------------------------------------------------- 1 | # Azure Machine Learning developer guide 2 | 3 | For years, Microsoft has been leading the way in AI and in providing the tools and services needed to help organizations of all sizes benefit from the capabilities it provides. Microsoft provides a vast number of technologies that you, as a data scientist, developer, or data engineer, can use to harness the power of AI and machine learning (ML). These technologies cover the [AI and ML spectrum](./intro/what-is-azure-machine-learning.md#the-microsoft-ai-and-ml-spectrum), from fully managed pre-built AI models ([Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/)) to tools and services that enable you to build and deploy your custom models, with [Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/overview-what-is-azure-ml) at its core. 4 | 5 | This guide provides documentation, code samples, and best practices for harnessing the power of Azure Machine Learning. Whether you are a data scientist, data engineer, or developer, this guide is your resource to learn AI fundamentals and how to use Azure Machine Learning service and related tools to grow your AI practice and learn new skills. 6 | 7 | Use the table of contents for diving into any part of the guide. You do not need to read the guide from front to back. It is divided into topics that introduce related concepts and contains detailed instructions and best practices. Each topic is self-contained and links to related content as needed. 8 | 9 | ## Table of contents 10 | 11 | 1. [Intro](./intro/README.md) 12 | - [What is Azure Machine Learning?](./intro/what-is-azure-machine-learning.md) 13 | - [What tools are used to do data engineering, data science, and AI?](./intro/tools.md) 14 | - [Overview of Azure Machine Learning service architecture and concepts](./intro/architecture-overview.md) 15 | - [Getting your environment set up](./intro/environment-setup.md) 16 | 2. [Data acquisition & understanding](./data-acquisition-understanding/README.md) 17 | - [Overview of wrangling, exploring and cleaning data](./data-acquisition-understanding/data-wrangling.md) 18 | - [Accessing data from Azure services and datastores](./data-acquisition-understanding/accessing-data.md) 19 | - [Load, transform and write data with Azure Machine Learning and the AML Data Prep SDK](./data-acquisition-understanding/loading-and-writing-data.md) 20 | 3. [Modeling](./modeling/README.md) 21 | - [Overview of Feature Engineering, Model Training, Model Evaluation and Model Selection](./modeling/feature-engineering-training-evaluation-selection.md) 22 | - [Feature Engineering introduced](./modeling/feature-engineering-training-evaluation-selection.md#feature-engineering-introduced) 23 | - [Model Training introduced](./modeling/model-training.md) 24 | - [Model Evaluation introduced](./modeling/model-evaluation.md) 25 | - [Training and Evaluating a simple model using Azure Machine Learning Visual Interface](./modeling/training-evaluating-model-with-visual-interface.md) 26 | - [Training and Evaluating a few simple models Using Azure Notebooks and Azure Machine Learning compute](./modeling/training-evaluating-simple-models-with-aml-compute.md) 27 | - [Simplifying the process with Automated Machine Learning, a component of Azure Machine Learning service](./modeling/simplify-process-with-automated-ml.md) 28 | - [Understanding automated machine learning generated models, using the model explainability capability of automated machine learning](./modeling/automl-understand-models-with-explainability.md) 29 | - [Using automated machine learning for Classification (Code Sample)](./modeling/automl-classification-code-sample.md) 30 | - [Using automated machine learning for Regression (Code Sample)](./modeling/automl-regression-code-sample.md) 31 | - [Using automated machine learning for Forecasting (Code Sample)](./modeling/automl-forecasting-code-sample.md) 32 | - [Using automated machine learning with model explainability (Code Sample)](./modeling/automl-understand-models-with-explainability.md#Model-explainability-code-sample) 33 | 4. [Model deployment](./model-deployment/README.md) 34 | - [Overview of deployment target options](./model-deployment/deployment-target-options.md) 35 | - [Overview of Real-time inferencing](./model-deployment/real-time-inferencing.md) 36 | - [Overview of Batch inferencing](./model-deployment/batch-inferencing.md) 37 | - [Overview of inferencing at the IoT edge](./model-deployment/iot-edge-inferencing.md) 38 | - [Reducing model deployment dependencies and improving model inferencing performance with ONNX](./model-deployment/deployment-with-onnx.md) 39 | 5. [Creating machine learning pipelines](./creating-machine-learning-pipelines/README.md) 40 | 6. [MLOps - DevOps for AI](./devops-for-ai/README.md) 41 | - [Overview of the relationship between model training pipeline (MLOps) and traditional DevOps application CI/CD pipelines](./devops-for-ai/mlops-and-devops.md) 42 | - [Using Azure DevOps](./devops-for-ai/using-azure-devops.md) 43 | - [Overview of post deployment monitoring and management tasks](./devops-for-ai/post-deployment-monitoring-and-management.md) 44 | - [Production monitoring of model input data and the deployed model predictions using the AML Model data collection](./devops-for-ai/monitoring-data-collection.md) 45 | - [Collecting model web service performance telemetry (e.g., request rates, response times, failure rates and exceptions) with Application Insights](./devops-for-ai/model-webservice-performance-telemetry.md) 46 | - [Monitoring a deployed model's collected data and telemetry (Code Sample)](./devops-for-ai/monitoring-data-and-telemetry-code-sample.md) 47 | - [Model version management (Code Sample)](./devops-for-ai/model-version-management-code-sample.md) 48 | - [Executing an end-to-end DevOps pipeline with Azure Machine Learning and Azure DevOps (Code Sample)](./devops-for-ai/e2e-pipeline-code-sample.md) 49 | 7. [Conclusion](./conclusion/README.md) 50 | -------------------------------------------------------------------------------- /conclusion/README.md: -------------------------------------------------------------------------------- 1 | # Conclusion 2 | 3 | In conclusion, you have seen through this Azure Machine Learning Developer Guide how Microsoft has been leading the way in AI and in providing the tools and services needed to help organizations of all sizes benefit from the capabilities it provides. This Developer Guide helped recognize the vast number of technologies that you, as a data scientist, developer, or data engineer, can use to harness the power of AI and machine learning (ML). From fully managed pre-built AI models ([Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/)) to tools and services that enable you to build and deploy your custom models, with [Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/overview-what-is-azure-ml) at its core. 4 | 5 | This guide provided documentation, code samples and best practices for harnessing the power of Azure Machine Learning. Whether you are a data scientist, data engineer, or developer, keep this guide nearby while learning AI fundamentals and how to use Azure Machine Learning service and related tools to grow your AI practice and learn new skills. 6 | 7 | This guide went through very important topics to understand the different offerings by Microsoft: 8 | - Introduction to Azure Machine Learning 9 | - [Supervised Learning](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/intro/what-is-azure-machine-learning.md) 10 | - [Unsupervised Learning](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/intro/what-is-azure-machine-learning.md) 11 | - [Reinforcement Learning](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/intro/what-is-azure-machine-learning.md) 12 | - [Deep Learning](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/intro/what-is-azure-machine-learning.md) 13 | - [Transfer Learning](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/intro/what-is-azure-machine-learning.md) 14 | - Data Acquisition and Understanding 15 | - [Data Wrangling](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/data-acquisition-understanding/data-wrangling.md) 16 | - [Accessing data from various Azure services](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/data-acquisition-understanding/accessing-data.md) 17 | - [AML Data Prep SDK](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/data-acquisition-understanding/loading-and-writing-data.md) 18 | - Model creation, training, evaluation, selection and deployment 19 | - [Training & evaluating using the Visual Interface](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/modeling/training-evaluating-model-with-visual-interface.md) 20 | - [Training & evaluating in Azure Notebooks](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/modeling/training-evaluating-simple-models-with-aml-compute.md) 21 | - [Working with Automated ML](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/modeling/simplify-process-with-automated-ml.md) 22 | - [Real Time Inferencing](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/model-deployment/real-time-inferencing.md) 23 | - [Batch Inferencing](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/model-deployment/batch-inferencing.md) 24 | - Working with Machine Learning pipelines 25 | - [Learning pipelines using the Azure ML SDK](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/creating-machine-learning-pipelines/machine-learning-pipelines.md) 26 | - Using Azure DevOps to maintain and automate the AI & Machine Learning Cycles during a project 27 | - [MLOps vs traditional DevOps](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/devops-for-ai/mlops-and-devops.md) 28 | - [Post Deployment monitoring and management tasks](https://github.com/solliancenet/Azure-Machine-Learning-Dev-Guide/blob/master/devops-for-ai/post-deployment-monitoring-and-management.md) -------------------------------------------------------------------------------- /conclusion/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/conclusion/media/_placeholder -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/README.md: -------------------------------------------------------------------------------- 1 | # Creating machine learning pipelines 2 | 3 | - [Overview of machine learning pipelines using the Azure Machine Learning SDK](./machine-learning-pipelines.md) 4 | -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/_placeholder -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/batch_score.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/batch_score.png -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/batch_scoring_pipeline_details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/batch_scoring_pipeline_details.png -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/data_prep.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/data_prep.png -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/new_notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/new_notebook.png -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/pipeline_details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/pipeline_details.png -------------------------------------------------------------------------------- /creating-machine-learning-pipelines/media/pipelines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/creating-machine-learning-pipelines/media/pipelines.png -------------------------------------------------------------------------------- /data-acquisition-understanding/README.md: -------------------------------------------------------------------------------- 1 | # Data acquisition & understanding 2 | 3 | - [Overview of wrangling, exploring and cleaning data](./data-wrangling.md) 4 | - [Accessing data from Azure services and datastores](./accessing-data.md) 5 | - [Load, transform and write data with Azure Machine Learning and the AML Data Prep SDK](./loading-and-writing-data.md) 6 | -------------------------------------------------------------------------------- /data-acquisition-understanding/media/aldermen-data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/aldermen-data.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/azure-blobs-crime-data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/azure-blobs-crime-data.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/create-workspace-sdk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/create-workspace-sdk.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-aldermen-joined.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-aldermen-joined.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-append-cols.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-append-cols.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-append-rows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-append-rows.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-date-formats-updated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-date-formats-updated.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-date-formats.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-date-formats.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-dirty-df.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-dirty-df.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-dirty-head-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-dirty-head-5.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-fill-errors.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-fill-errors.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-fill-nulls.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-fill-nulls.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-imputed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-imputed.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-mean-latitude.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-mean-latitude.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-replace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-replace.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/crime-location-sample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/crime-location-sample.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/data-profiles-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/data-profiles-output.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/download-directory-and-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/download-directory-and-output.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/download-directory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/download-directory.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/download-file-and-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/download-file-and-output.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-add-column-using-expression.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-add-column-using-expression.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-auto-read-parquet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-auto-read-parquet.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-auto-read-txt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-auto-read-txt.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-auto-read-xlsx.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-auto-read-xlsx.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-builder-to-dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-builder-to-dataflow.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-builders-column-types.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-builders-column-types.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-crime-txt-transformed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-crime-txt-transformed.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-data-file-format.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-data-file-format.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-detect-file-format-dtypes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-detect-file-format-dtypes.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-dflow-dtypes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-dflow-dtypes.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-dflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-dflow.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-load-crime-dirty-csv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-load-crime-dirty-csv.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-load-crime-full-csv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-load-crime-full-csv.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-csv-infer-column-types.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-csv-infer-column-types.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-csv-path-only.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-csv-path-only.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-csv-skip-rows.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-csv-skip-rows.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-json-flatten-nested-arrays.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-json-flatten-nested-arrays.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-json.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-json.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-lines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-lines.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-read-sql-from-datasource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-read-sql-from-datasource.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-source-txt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-source-txt.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-transform-builder-new-dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-transform-builder-new-dataflow.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-transform-crime-location-mean-latitude.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-transform-crime-location-mean-latitude.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-transform-location-imputed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-transform-location-imputed.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-transform-location-sample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-transform-location-sample.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-transform-split-columns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-transform-split-columns.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-type-conversions-to-dataflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-type-conversions-to-dataflow.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-dflow-with-write-step.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-dflow-with-write-step.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-error-assignment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-error-assignment.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-initial-dflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-initial-dflow.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-local-file-system.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-local-file-system.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-read-back-crime-txt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-read-back-crime-txt.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/dprep-write-to-parquet-with-error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/dprep-write-to-parquet-with-error.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/read-csv-from-datastore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/read-csv-from-datastore.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/read-sql-from-datastore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/read-sql-from-datastore.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/split-column-by-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/split-column-by-example.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/upload-command-and-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/upload-command-and-output.png -------------------------------------------------------------------------------- /data-acquisition-understanding/media/upload-files-and-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-acquisition-understanding/media/upload-files-and-output.png -------------------------------------------------------------------------------- /data-science-lifecycle/README.md: -------------------------------------------------------------------------------- 1 | # The data science lifecycle -------------------------------------------------------------------------------- /data-science-lifecycle/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/data-science-lifecycle/media/_placeholder -------------------------------------------------------------------------------- /data-science-lifecycle/tdsp-intro.md: -------------------------------------------------------------------------------- 1 | # Introduction to the TDSP lifecycle -------------------------------------------------------------------------------- /devops-for-ai/README.md: -------------------------------------------------------------------------------- 1 | # MLOps - DevOps for AI 2 | 3 | - [Overview of the relationship between model training pipeline (MLOps) and traditional DevOps application CI/CD pipelines](./mlops-and-devops.md) 4 | - [Using Azure DevOps](./using-azure-devops.md) 5 | - [Overview of post deployment monitoring and management tasks](./post-deployment-monitoring-and-management.md) 6 | - [Production monitoring of model input data and the deployed model predictions using the AML Model data collection](./monitoring-data-collection.md) 7 | - [Collecting model web service performance telemetry (e.g., request rates, response times, failure rates and exceptions) with Application Insights](./model-webservice-performance-telemetry.md) 8 | - [Monitoring a deployed model's collected data and telemetry (Code Sample)](./monitoring-data-and-telemetry-code-sample.md) 9 | - [Model version management (Code Sample)](./model-version-management-code-sample.md) 10 | - [Executing an end-to-end DevOps pipeline with Azure Machine Learning and Azure DevOps (Code Sample)](./e2e-pipeline-code-sample.md) 11 | -------------------------------------------------------------------------------- /devops-for-ai/media/Boards.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Boards.png -------------------------------------------------------------------------------- /devops-for-ai/media/Commits.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Commits.png -------------------------------------------------------------------------------- /devops-for-ai/media/CreateProject.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/CreateProject.png -------------------------------------------------------------------------------- /devops-for-ai/media/Dashboards.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Dashboards.png -------------------------------------------------------------------------------- /devops-for-ai/media/DevOpsDashBoard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/DevOpsDashBoard.png -------------------------------------------------------------------------------- /devops-for-ai/media/Files.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Files.png -------------------------------------------------------------------------------- /devops-for-ai/media/Jenkins.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Jenkins.png -------------------------------------------------------------------------------- /devops-for-ai/media/MakingChanges.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/MakingChanges.png -------------------------------------------------------------------------------- /devops-for-ai/media/Overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Overview.png -------------------------------------------------------------------------------- /devops-for-ai/media/PipRepos.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/PipRepos.png -------------------------------------------------------------------------------- /devops-for-ai/media/PostGates.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/PostGates.png -------------------------------------------------------------------------------- /devops-for-ai/media/Queries.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Queries.png -------------------------------------------------------------------------------- /devops-for-ai/media/ReleasePipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/ReleasePipeline.png -------------------------------------------------------------------------------- /devops-for-ai/media/ReleasePipelineFiring.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/ReleasePipelineFiring.png -------------------------------------------------------------------------------- /devops-for-ai/media/ReleaseTemplates.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/ReleaseTemplates.png -------------------------------------------------------------------------------- /devops-for-ai/media/Repos.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Repos.png -------------------------------------------------------------------------------- /devops-for-ai/media/Stages.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Stages.png -------------------------------------------------------------------------------- /devops-for-ai/media/Tasks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/Tasks.png -------------------------------------------------------------------------------- /devops-for-ai/media/TestPlans.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/TestPlans.png -------------------------------------------------------------------------------- /devops-for-ai/media/TestPlansTab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/TestPlansTab.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-blob.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-blob.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-databricks-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-databricks-1.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-databricks-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-databricks-2.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-databricks-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-databricks-3.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-enable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-enable.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-powerbi-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-powerbi-1.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-powerbi-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-powerbi-2.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-data-collection-powerbi-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-data-collection-powerbi-3.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-model-tagging.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-model-tagging.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-model-versioning.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-model-versioning.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-telemetry-azure-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-telemetry-azure-portal.png -------------------------------------------------------------------------------- /devops-for-ai/media/azureml-telemetry-enable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/azureml-telemetry-enable.png -------------------------------------------------------------------------------- /devops-for-ai/media/mlops-connect-devops-to-aml.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/mlops-connect-devops-to-aml.png -------------------------------------------------------------------------------- /devops-for-ai/media/pipelines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/devops-for-ai/media/pipelines.png -------------------------------------------------------------------------------- /devops-for-ai/mlops-and-devops.md: -------------------------------------------------------------------------------- 1 | # Overview of the relationship between model training pipeline (MLOps) and traditional DevOps application CI/CD pipelines 2 | 3 | ## MLOps - DevOps for AI overview 4 | 5 | DevOps has become ubiquitous in the world of classical development. Almost all projects that exceed a certain level of complexity become inevitably DevOps projects. Yet there is one category of projects that are stepping out of the line. You’ve guessed it right, it’s the category of Data Science projects. 6 | 7 | When it comes to DevOps, Data Science projects pose a range of special challenges, whether it’s about the technical side of things, the philosophy of the people involved, or the actors involved. Think about one simple example: versioning. While in a “classical” development project versioning refers almost exclusively to source code, in the world of data science it gets another important aspect: data versioning. It’s not enough to know the version of the code for your model, it’s equally important to know the “version” of the data it was trained on. Another interesting question is, for examples, what does a “build” mean in the world of data science? Or a “release”? 8 | 9 | This section is dedicated to the topic of MLOps which is basically the discipline that deals with the application of "classical" DevOps principles to projects that have at least one data science component. As the title implies, MLOps is `DevOps for AI`. 10 | 11 | ## The relationship between MLOps and traditional DevOps application CI/CD pipelines 12 | 13 | When it comes to DevOps principles, any non-trivial project that includes a data science component will need to take advantage of a combination of features from the following two major platforms: 14 | 15 | - [Azure Machine Learning service](https://azure.microsoft.com/services/machine-learning-service/) 16 | - [Azure DevOps](https://azure.microsoft.com/services/devops) 17 | 18 | The official definition of **Azure Machine Learning service** is: 19 | 20 | > *Azure Machine Learning service provides a cloud-based environment you can use to prep data, train, test, deploy, manage, and track machine learning models. Start training on your local machine and then scale out to the cloud. The service fully supports open-source technologies such as PyTorch, TensorFlow, and scikit-learn and can be used for any kind of machine learning, from classical ml to deep learning, supervised and unsupervised learning.* 21 | 22 | The official definition of **Azure DevOps** is: 23 | 24 | > *Azure DevOps provides developer services to support teams to plan work, collaborate on code development, and build and deploy applications. Developers can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server, formerly named Visual Studio Team Foundation Server (TFS).* 25 | 26 | The DevOps mindset applied to data science solutions is commonly referred as Machine Learning Operations (MLOps). The most important aspects of MLOps are: 27 | 28 | - Deploying Machine Learning projects from anywhere 29 | - Monitoring ML solutions for both generic and ML-specific operational issues 30 | - Capturing all data that is necessary for full traceability in the ML lifecycle 31 | - Automating the end-to-end ML lifecycle using a combination of Azure Machine Learning service and Azure DevOps 32 | 33 | ## Two complementary platforms 34 | 35 | If you plan on implementing an end-to-end DevOps approach for your project, you will need to use both platforms (Azure Machine Learning service and Azure DevOps). Although Azure DevOps is a native, all-purpose DevOps platform, it lacks specific features that are required in data science projects. At the same time, Azure Machine Learning service is a native, all-purpose Data Science platform, but it does not have some of the core DevOps features that you will most certainly need. 36 | 37 | Because of the very specific DevOps-type needs that data science projects have, the Azure Machine Learning service has started to provide DevOps-like services centered around the concept of ML Pipelines and model management. ML Pipelines provide the backend for data scientists, data engineers, and IT professionals to collaborate of specific tasks like data preparation (e.g. transformation, normalization), model training and evaluation, and deployment. Because of the term `pipelines`, one might get the sense that ML Pipelines are somewhat of a competitor to Azure DevOps pipelines. In fact, the two of them are designed to work together and their functionalities are complementary. Model management enables you to register and track trained ML models based on their name and version. 38 | 39 | ML Pipelines are described in the [Overview of machine learning pipelines using the Azure Machine Learning SDK](../../creating-machine-learning-pipelines/machine-learning-pipelines.md) section. 40 | 41 | Model management is described in the [Model versioning with the AML model registry](../post-deployment-monitoring-and-management/model-versioning.md) section. 42 | 43 | Next, we will explore the relationships that exist between the various features of Azure Machine Learning service and Azure DevOps. 44 | 45 | ## Feature comparison 46 | 47 | The following table provides details about how the most important MLOps concepts are interpreted by the Azure Machine Learning service and Azure DevOps. 48 | 49 | MLOps concept | Azure Machine Learning service | Azure DevOps | Primary Choice 50 | --- | --- | --- | --- 51 | Boards | Not supported | Provides a set of services that enable you to plan, track, and discuss work across different teams involved in the project. | Azure DevOps 52 | Code Repos | Not Supported | Provides a set of services that enable you to work with Git repos, and collaborate using advanced file management and pull requests. Integrates with boards, enabling you to link source code, branches, and pull requests with work items. | Azure DevOps 53 | Artifacts Management | Provides extensive support for managing machine learning-specific artifacts like experiments, runs, models, compute resources, models, images, deployments, and activities. Also capable of handling various by-products of the ML processes like outputs, logs, images, performance metrics, and many more. | Provides generic support for build artifacts management as well as services for storing Maven, npm, NuGet, and Python packages together. | Azure Machine Learning service (Azure DevOps will be used to handle artifacts required by the CI/CD pipelines). 54 | Pipelines | Provides services geared towards data preparation, model training and evaluation, and deployment. Enables optimization of ML workflows with simplicity, speed, portability, and reuse. Also enables explicit naming and versioning of data sources, inputs, and outputs. | Provides support for generic, all-purpose Continuous Integration / Continuous Delivery pipelines for building, testing, and deploying solutions. Works with any code repo provider, any language, any platform, and any cloud. | Azure DevOps (for high-level orchestration, coordination, build, and release using a CI/CD approach). Specific steps from the Azure DevOps CI/CD pipelines call into Azure Machine Learning service through the Python SDK (includes the launch of ML pipelines driven by the Azure Machine Learning service). 55 | Data Versioning | Provides support through Azure ML Datasets. | Not supported | Azure Machine Learning service 56 | Model Training, Deployment, and Monitoring | Provides extensive support for registering, tracking, packaging, debugging, validating, profiling, converting, and using ML models. Provides support for understanding what data is send to the model, and what predictions it returns. | Not supported | Azure Machine Learning service 57 | 58 | Looking at this table it is easy to observe the complementary nature of Azure Machine Learning service and Azure DevOps in the context of MLOps. Furthermore, the integration between the two is simplified by the [Azure Machine Learning extension](https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.vss-services-azureml) from Azure DevOps which enables the following: 59 | 60 | - Azure Machine Learning workspace selection when defining a service connection in Azure DevOps 61 | - Azure DevOps release pipeline triggering by a trained model created in a training pipeline 62 | 63 | ## A typical scenario 64 | 65 | The typical scenario is defined by the following: 66 | 67 | - Uses a combination of Azure DevOps and Azure Machine Learning service 68 | - Azure Machine Learning Datasets help track and version data 69 | - Azure Machine Learning experiments and runs manage code, data, and compute used to train a model 70 | - Azure Machine Learning model registry captures all the metadata around models 71 | - Azure DevOps Boards are used to manage the agile development process 72 | - Azure DevOps Code Repos are used to manage all source code, including code that controls and interacts with the Azure Machine Learning service (through the Python SDK) 73 | - Azure DevOps release and build pipelines are used to implement the high-level processes. In turn, they rely on the Python SDK to interact with ML pipelines from Azure Machine Learning service 74 | - When a data scientists checks in code into the repo, an Azure DevOps pipeline will start a training run. The results will be then inspected to assess the performance characteristics of the trained model. If the newly trained model passes the quality checks, a different (release) pipeline will be triggered to deploy the model as a service. 75 | 76 | ## Next steps 77 | 78 | You can learn more about Azure Machine by reviewing these links to additional resources: 79 | 80 | - [MLOps: Manage, deploy, and monitor models with Azure Machine Learning Service](https://docs.microsoft.com/azure/machine-learning/service/concept-model-management-and-deployment) 81 | - [Build reusable ML pipelines in Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/concept-ml-pipelines) 82 | - [Create and run a machine learning pipeline by using Azure Machine Learning SDK](https://docs.microsoft.com/azure/machine-learning/service/how-to-create-your-first-pipeline) 83 | - [Train and deploy machine learning models](https://docs.microsoft.com/azure/devops/pipelines/targets/azure-machine-learning) 84 | 85 | Read next: [Using Azure DevOps](./using-azure-devops.md) -------------------------------------------------------------------------------- /devops-for-ai/model-version-management-code-sample.md: -------------------------------------------------------------------------------- 1 | # Model version management (Code Sample) 2 | 3 | Extending the classical DevOps approach to Machine Learning requires, among other things, extending the concept of source code management and versioning to model management and versioning. Through its model registry, Azure Machine Learning service provides the necessary capabilities to register, version, and manage trained machine learning models. 4 | 5 | Here is an exmaple on how to register a model: 6 | 7 | ```python 8 | model_description = 'Deep learning model to classify the descriptions of car components as compliant or non-compliant.' 9 | model = Model.register( 10 | model_path='model.h5', # this points to a local file 11 | model_name=args.model_name, # this is the name the model is registered as 12 | tags={"type": "classification", "run_id": run.id, "build_number": args.build_number}, 13 | description=model_description, 14 | workspace=run.experiment.workspace 15 | ) 16 | ``` 17 | 18 | The code above if from a `train.py` file used to train a model on a remote compute resource provided by Azure Machine Learning service. Notice the `tags` parameter which includes the `build_number` property that gets its value from the `build_number` parameter that is passed to the script when its executed. When the script is executed from a build pipeline in Azure DevOps, this tag property can be used to provide a very useful link between the trained model managed in Azure Machine Learning service and the build that run in Azure DevOps. 19 | 20 | **Note**: Multiple calls to `Model.register()` using the same model name (in the same workspace) will result in new versions being created for the model. Version numbers start from 1 and are incremented by 1 each time the same model (based on name) is registered in a given workspace. 21 | 22 | Here is an example of the result of multiple registrations of the same model: 23 | 24 | ![Trained models versions in Azure Machine Learning service](./media/azureml-model-versioning.png) 25 | 26 | Version 3 of the `compliance-classifier` model has been taged with the build number `20190809.2`: 27 | 28 | ![Trained model tagged with build number in Azure Machine Learning service](./media/azureml-model-tagging.png) 29 | 30 | Once a model is registered it can be retrieved at a later point in time either for deployment or for analysis/reevaluation. 31 | 32 | To retrieve the latest version of a model, use the `Model` class: 33 | 34 | ```python 35 | model = Model(ws, 'compliance-classifier') 36 | ``` 37 | 38 | To retrieve a model with a specific version, use the `Model` class and specify the version number: 39 | 40 | 41 | ```python 42 | model = Model(ws, 'compliance-classifier', version=1) 43 | ``` 44 | 45 | To retrieve a model based on the value of one of its tags (e.g. `build_number`), the the `Model` class and specify the tag value: 46 | 47 | ```python 48 | model = Model(ws, 'compliance-classifier', tags={'build_number': '20190809.2'}) 49 | ``` 50 | 51 | ## Next steps 52 | 53 | You can learn more about model version management by reviewing these links to additional resources: 54 | 55 | - [MLOps: Manage, deploy, and monitor models with Azure Machine Learning Service](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-model-management-and-deployment) 56 | 57 | Read next: [Executing an end-to-end DevOps pipeline with Azure Machine Learning and Azure DevOps (Code Sample)](./e2e-pipeline-code-sample.md) -------------------------------------------------------------------------------- /devops-for-ai/model-webservice-performance-telemetry.md: -------------------------------------------------------------------------------- 1 | # Collecting model web service performance telemetry (e.g., request rates, response times, failure rates and exceptions) with Application Insights 2 | 3 | Azure Machine Learning is integrated with [Azure Application Insights](https://docs.microsoft.com/azure/azure-monitor/app/app-insights-overview). 4 | 5 | The official definition of Azure Application Insights is: 6 | 7 | > _Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms. Use it to monitor your live web application. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It's designed to help you continuously improve performance and usability. It works for apps on a wide variety of platforms including .NET, Node.js and Java EE, hosted on-premises, hybrid, or any public cloud. It integrates with your DevOps process, and has connection points to a variety of development tools. It can monitor and analyze telemetry from mobile apps by integrating with Visual Studio App Center._ 8 | 9 | Enabling Application Insights telemetry on your production model provides the following benefits: 10 | 11 | - Accurate measurement of request rates, response times, and failure rates 12 | - Traceability for all usage 13 | - History of exceptions 14 | 15 | ## Enabling Application Insights integration 16 | 17 | You can use the Azure portal to enable Application Insights integration for a model that has been published in production: 18 | 19 | ![Enable Application Insights integration in Azure Machine Learning](./media/azureml-telemetry-enable.png) 20 | 21 | Deploying a trained model to production is described in the [Overview of deployment target options](../model-deployment/deployment-target-options.md) section. 22 | 23 | Enabling data collection using the SDK is described in the [Monitoring a deployed model's collected data and telemetry (Code Sample)](./monitoring-data-and-telemetry-code-sample.md) section. 24 | 25 | ## Querying and analyzing monitoring data 26 | 27 | You can use the Azure Portal to view and analyze the telemetry data collected by Application Insights: 28 | 29 | ![View Application Insights telemetry data in Azure Portal](./media/azureml-telemetry-azure-portal.png) 30 | 31 | If you prefer to use Power BI to analyze telemetry data, you can do this using one of the following approaches: 32 | 33 | - [Export Analytics queries](https://docs.microsoft.com/azure/azure-monitor/app/export-power-bi#export-analytics-queries) 34 | - [Continuous export and Azure Stream Analytics](https://docs.microsoft.com/azure/azure-monitor/app/export-stream-analytics) 35 | 36 | ## Next steps 37 | 38 | You can learn more about production model telemetry with Azure Application Insights by reviewing these links to additional resources: 39 | 40 | -[Monitor your Azure Machine Learning models with Application Insights](https://docs.microsoft.com/azure/machine-learning/service/how-to-enable-app-insights) 41 | 42 | Read next: [Monitoring a deployed model's collected data and telemetry (Code Sample)](./monitoring-data-and-telemetry-code-sample.md) 43 | -------------------------------------------------------------------------------- /devops-for-ai/monitoring-data-and-telemetry-code-sample.md: -------------------------------------------------------------------------------- 1 | # Monitoring a deployed model's collected data and telemetry (Code Sample) 2 | 3 | ## Enable or disable data collection for a production model 4 | 5 | A typical example for a scoring file used when deploying a trained machine learning model looks like this: 6 | 7 | ```python 8 | #example: scikit-learn and Swagger 9 | import json 10 | import numpy as np 11 | from sklearn.externals import joblib 12 | from sklearn.linear_model import Ridge 13 | from azureml.core.model import Model 14 | 15 | from inference_schema.schema_decorators import input_schema, output_schema 16 | from inference_schema.parameter_types.numpy_parameter_type import NumpyParameterType 17 | 18 | 19 | def init(): 20 | global model 21 | # note here "sklearn_regression_model.pkl" is the name of the model registered under 22 | # this is a different behavior than before when the code is run locally, even though the code is the same. 23 | model_path = Model.get_model_path('sklearn_regression_model.pkl') 24 | # deserialize the model file back into a sklearn model 25 | model = joblib.load(model_path) 26 | 27 | 28 | input_sample = np.array([[10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]) 29 | output_sample = np.array([3726.995]) 30 | 31 | 32 | @input_schema('data', NumpyParameterType(input_sample)) 33 | @output_schema(NumpyParameterType(output_sample)) 34 | def run(data): 35 | try: 36 | result = model.predict(data) 37 | # you can return any datatype as long as it is JSON-serializable 38 | return result.tolist() 39 | except Exception as e: 40 | error = str(e) 41 | return error 42 | ``` 43 | 44 | For more details about deploying trained models with Azure Machine Learning read the [Model deployment](../model-deployment/README.md) section. 45 | 46 | To add data collection support you need to make serveral changes to the scoring file. 47 | 48 | First, add the necessary imports at the begining: 49 | 50 | ```python 51 | from azureml.monitoring import ModelDataCollector 52 | ``` 53 | Next, declare the necessary data collection variables in the `init()` function: 54 | 55 | ```python 56 | global inputs_dc, prediction_dc 57 | inputs_dc = ModelDataCollector("best_model", identifier="inputs", feature_names=["feat1", "feat2", "feat3". "feat4", "feat5", "feat6"]) 58 | prediction_dc = ModelDataCollector("best_model", identifier="predictions", feature_names=["prediction1", "prediction2"]) 59 | ``` 60 | 61 | Finally, you need to add the the actual collection code to the `run(data)` function: 62 | 63 | ```python 64 | data = np.array(data) 65 | result = model.predict(data) 66 | inputs_dc.collect(data) #this call is saving our input data into Azure Blob 67 | prediction_dc.collect(result) #this call is saving our input data into Azure Blob 68 | ``` 69 | Once your scoring file is prepared, you can enable data collection. This can be done either during initial publishing or later, after the model has already been published. 70 | 71 | To enable data collection during deployment, you will set the value of the `collect_model_data` parameter to `True`: 72 | 73 | ```python 74 | aks_config = AksWebservice.deploy_configuration(collect_model_data=True) 75 | ``` 76 | To enable data collection for an already published model, you will use the `update()` method on `AksWebService`: 77 | 78 | ```python 79 | from azureml.core.webservice.aks import AksWebservice 80 | 81 | aks_service = AksWebservice(ws, 'compliance-classifier-service') 82 | aks_service.update(collect_model_data=True) 83 | ``` 84 | 85 | To disable data collection for an already published model, you will use the `update()` method on `AksWebService`: 86 | 87 | ```python 88 | from azureml.core.webservice.aks import AksWebservice 89 | 90 | aks_service = AksWebservice(ws, 'compliance-classifier-service') 91 | aks_service.update(collect_model_data=False) 92 | ``` 93 | 94 | ## Enable or disable Application Insights telemetry for a production model 95 | 96 | To enable telemetry during deployment, you will set the value of the `enable_app_insights` parameter to `True`: 97 | 98 | ```python 99 | aks_config = AksWebservice.deploy_configuration(enable_app_insights=True) 100 | ``` 101 | To enable data collection for an already published model, you will use the `update()` method on `AksWebService`: 102 | 103 | ```python 104 | from azureml.core.webservice.aks import AksWebservice 105 | 106 | aks_service = AksWebservice(ws, 'compliance-classifier-service') 107 | aks_service.update(enable_app_insights=True) 108 | ``` 109 | 110 | To disable data collection for an already published model, you will use the `update()` method on `AksWebService`: 111 | 112 | ```python 113 | from azureml.core.webservice.aks import AksWebservice 114 | 115 | aks_service = AksWebservice(ws, 'compliance-classifier-service') 116 | aks_service.update(enable_app_insights=False) 117 | ``` 118 | 119 | ## Next steps 120 | 121 | You can learn more about monitoring a deployed model's collected data and telemetry by reviewing these links to additional resources: 122 | 123 | - [Collect data for models in production](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-enable-data-collection) 124 | - [Monitor your Azure Machine Learning models with Application Insights](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-enable-app-insights) 125 | 126 | Read next: [Model version management (Code Sample)](./model-version-management-code-sample.md) -------------------------------------------------------------------------------- /devops-for-ai/monitoring-data-collection.md: -------------------------------------------------------------------------------- 1 | # Production monitoring of model input data and the deployed model predictions using the AML Model data collection 2 | 3 | The following types of data can be collected for monitoring purposes: 4 | 5 | - Production model input data 6 | - Model predictions based on production input data 7 | 8 | Azure Machine Learning service will save all data in Azure blob storage (in the associated storage account that gets automatically provisioned when the service itself is provisioned). The path has the following template: 9 | 10 | ``` 11 | /modeldata///////////data.csv 12 | ``` 13 | 14 | Here is an example of a `data.csv` file generated by the data collection feature: 15 | 16 | ![Azure Machine Learning service data collection](./media/azureml-data-collection-blob.png) 17 | 18 | ## Enabling data collection 19 | 20 | To enable data collection, you need to have the proper dependencies added to your scoring file. Once you have these in place, you can use the Azure portal to enable data collection for your production model: 21 | 22 | ![Enable production model data collection in Azure portal](./media/azureml-data-collection-enable.png) 23 | 24 | Deploying a trained model to production is described in the [Overview of deployment target options](../model-deployment/deployment-target-options.md) section. 25 | 26 | Enabling data collection using the SDK is described in the [Monitoring a deployed model's collected data and telemetry (Code Sample)](./monitoring-data-and-telemetry-code-sample.md) section. 27 | 28 | ## Querying and analyzing monitoring data 29 | 30 | Since the collected data is made available in CSV format in blob storage, you can use any tool to query and analyze the data. Some of the most popular options are [Power BI](https://powerbi.microsoft.com/desktop/) and [Azure Databricks](https://azure.microsoft.com/services/databricks/). 31 | 32 | ### Power BI Desktop 33 | 34 | Using Power BI Desktop, you can connect to the storage account used by Azure Machine Learning service to store collected data: 35 | 36 | ![Connect to data collection storage account from Power BI Desktop](./media/azureml-data-collection-powerbi-1.png) 37 | 38 | Once you are connected, you can select the `modeldata` container and access all files: 39 | 40 | ![Select the data collection blob storage container from Power BI Desktop](./media/azureml-data-collection-powerbi-2.png) 41 | 42 | Finaly, using Power BI data language capabilities, you can combine all CSV files into a single virtual data space that will provide data for your Power BI reports and dashboards: 43 | 44 | ![Combine all data collection files in Power BI Desktop](./media/azureml-data-collection-powerbi-3.png) 45 | 46 | Alternatively, you can filter the selection and keep only the days/months/years you are interested in. 47 | 48 | ### Azure Databricks 49 | 50 | In cases where you need some more advanced data processing capabilities and/or superior computing power (e.g. when you collect significant amounts of data), Azure Databricks might be your choice. 51 | 52 | Using an Azure Databricks workspace, you can upload the collected data: 53 | 54 | ![Upload data in Azure Databricks](./media/azureml-data-collection-databricks-1.png) 55 | 56 | From there, you can map the data in the blob storage account container using the Azure Blob Storage connector: 57 | 58 | ![Connect to Azure Blob Storage in Azure Databricks](./media/azureml-data-collection-databricks-2.png) 59 | 60 | All that's left is to use the `spark.conf` utility to set the credentials needed to access the storage account: 61 | 62 | ![Configure blob storage account credentials in Azure Databricks](./media/azureml-data-collection-databricks-3.png) 63 | 64 | You can now use Azure Databricks to perform complex analysis and reporting on data collected from you production model. 65 | 66 | ## Next steps 67 | 68 | You can learn more about production monitoring of model input data and model predictions by reviewing these links to additional resources: 69 | 70 | - [Collect data for models in production](https://docs.microsoft.com/azure/machine-learning/service/how-to-enable-data-collection) 71 | 72 | Read next: [Collecting model web service performance telemetry (e.g., request rates, response times, failure rates and exceptions) with Application Insights](./model-webservice-performance-telemetry.md) 73 | -------------------------------------------------------------------------------- /devops-for-ai/post-deployment-monitoring-and-management.md: -------------------------------------------------------------------------------- 1 | # Overview of post deployment monitoring and management tasks 2 | 3 | When it comes to trained machine learning models, the typical data science project lifecycle includes the following stages: 4 | 5 | - Initial model development 6 | - Model training for production 7 | - Trained model deployment into production 8 | - Production use of the model 9 | - Model retraining for production (a new cycle starts when the updated model is re-deployed into production) 10 | 11 | Identifying the right moment to retrain a model for production is not a trivial task. Retraining too often means too many disruptions for systems that rely on the model and also potentially no significant improvements in performance. Retraining not often enough means potential degradation to the performance of the model. You will need to collect data for your production models in order to be able to perform the following tasks: 12 | 13 | - Identify potential data drifts 14 | - Correctly determine the right moment to retrain and/or optimize your model 15 | - Incorporate monitoring data into the training dataset 16 | 17 | This section addresses the problem of data collection and model versioning in post deployment (production) environments. The following topics will be covered: 18 | 19 | - [Production monitoring of model input data and the deployed model predictions using the AML Model data collection](./monitoring-data-collection.md) 20 | - [Collecting model web service performance telemetry (e.g., request rates, response times, failure rates and exceptions) with Application Insights](./model-webservice-performance-telemetry.md) 21 | 22 | ## Next steps 23 | 24 | You can learn more about post deployment by reviewing these links to additional resources: 25 | 26 | - [Collect data for models in production](https://docs.microsoft.com/azure/machine-learning/service/how-to-enable-data-collection) 27 | - [Detect data drift (preview) on models deployed to Azure Kubernetes Service (AKS)](https://docs.microsoft.com/azure/machine-learning/service/how-to-monitor-data-drift) 28 | - [Monitor your Azure Machine Learning models with Application Insights](https://docs.microsoft.com/azure/machine-learning/service/how-to-enable-app-insights) 29 | 30 | Read next: [Production monitoring of model input data and the deployed model predictions using the AML Model data collection](./monitoring-data-collection.md) 31 | -------------------------------------------------------------------------------- /devops-for-ai/using-azure-devops.md: -------------------------------------------------------------------------------- 1 | # Using Azure DevOps 2 | 3 | Azure DevOps provides developer services to support teams to plan work, collaborate on code development, and build and deploy applications. Developers can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server, formerly named Visual Studio Team Foundation Server (TFS). 4 | 5 | ## Why Use Azure DevOps? 6 | 7 | - It is perfect for turning your idea into a piece of Software. 8 | - Helps you plan your solution with Agile Tools 9 | - Enables source code management using Git and other first class source control systems 10 | - Deploys your solution to Cross Platform CI/CD-enabled systems 11 | - Provides full traceability and visibility across your entire solution development lifecycle. 12 | 13 | ## How to Get started? 14 | 15 | To get started, create an account on https://dev.azure.com 16 | This is where you will have access to all your Projects and also your associated Organization. 17 | 18 | ## Create a Project 19 | 20 | Click on the **Create Project** button to start a new Project in Azure DevOps. 21 | 22 | ![The create new project dialog is displayed.](media/CreateProject.png) 23 | 24 | Once the project is created, you will be able to manage Your project, Boards, Repositories, Pipeline, Test Plans, Artifacts and all your Project Settings from your DevOps Project screen. 25 | 26 | ![The DevOps dashboard is displayed with the previously listed areas highlighted.](media/DevOpsDashBoard.png) 27 | 28 | ## Overview 29 | 30 | In this tab you will be able to see a summary of your Project, project stats and participating members on the team. 31 | 32 | ![The Overview left-hand menu structure is highlighted.](media/Overview.png) 33 | 34 | From the Dashboards sub tab you will be able to generate screens that utilize widgets to drag and drop into your dashboards to make it easy to oversee the health and progress of your project. 35 | 36 | ![An arrow shows moving dashboard widgets onto the custom dashboard.](media/Dashboards.png) 37 | 38 | On the Wiki sub tab, you will be able to create a Wiki repository for your project to share information with the rest of your team regarding the project, where markdown is supported. 39 | 40 | ## Boards 41 | 42 | You can use Azure Boards to display Kanban boards to plan and check your entire release. You will be able to create sprints and move work items from different states until completion and release of your product. 43 | 44 | ![The Boards left-hand menu and its options are highlighted.](media/Boards.png) 45 | 46 | In **Backlogs** you will be able to drag and drop your items to the desired sprint and monitor all your backlog items in the system that are yet to be completed. 47 | 48 | In **Sprints** you will be able to track your progress on the **TaskBoard** and move items from one sprint to another easily. You can even create **Branches** right from the board, if need be. 49 | 50 | In **Queries**, this is where you would create your favorite queries about a group of items in the project's work items to quickly execute them in the future 51 | 52 | ![The Queries blade is displayed with a sample query.](media/Queries.png) 53 | 54 | ## Repos 55 | 56 | ![The Repos left-hand menu item and its sub-components are highlighted.](media/Repos.png) 57 | 58 | Repos in Azure DevOps are extremely important to manage your project files and source control, whether you are using a centralized source control or Distributed source control (like Git) 59 | 60 | The Files sub tab will allow you to view the files and structure of your repository and make changes if desired. 61 | 62 | ![A sample Python file is displayed.](media/Files.png) 63 | 64 | **Commits**, allows you to see how all your branches come together. 65 | 66 | ![The commit branch graph is highlighted.](media/Commits.png) 67 | 68 | **Pushes**, give you a historic view of all the changes made to the remote repository. 69 | 70 | **Branches**, will enable to view all the branches and see if they are ahead or behind master. 71 | 72 | **Tags**, will allow you to view all the Tags created for the project and even create new ones. 73 | 74 | **Pull Requests**, will allow you review and participate in Pull Requests in the project. You can also create policies to fire up a build automatically based on a pull request. 75 | 76 | ## Azure Pipelines 77 | 78 | This is a rich Cross Platform CI/CD system capable of building any language targeting any platform. 79 | 80 | ![The Pipelines left-hand menu and its sub-components are highlighted.](media/pipelines.png) 81 | 82 | From the summary page, you will be able to review commits, test results and perform many actions on your project. The Deployment section in the summary will be able to show you were in the release your project is currently. 83 | 84 | The real power of Azure Pipelines starts manifesting itself when you get to the **Build** edit screen. 85 | 86 | Azure Pipelines can pull your code from the most popular source control systems. 87 | 88 | ![The list of source pipeline repos is displayed.](media/PipRepos.png) 89 | 90 | Azure Pipelines also offer hosted Windows, Mac and Linux operating build agents. 91 | 92 | On the **Releases** sub tab, you will be able to create a new pipelines where you can start with an empty job or drag and drop a featured deployment pipeline from the many templates available in Azure Pipelines. 93 | 94 | ![The releases blade is shown with optional templates.](media/ReleaseTemplates.png) 95 | 96 | You are able to deploy to any platform with Releases, on Mac, Linux and Windows using their respective OS agents. With the integration with App Center, you can also deploy directly to mobile as well. 97 | 98 | You can create a pipeline by adding Stages and Artifacts 99 | 100 | ![The pipeline stages and artifacts are shown.](media/Stages.png) 101 | 102 | You can pull the Artifacts from several sources, for example Jenkins... 103 | 104 | ![The Jenkins artifact source type is selected.](media/Jenkins.png) 105 | 106 | For Stages, you have a wide variety of Tasks to choose from by a simple drag and drop to run on Agents 107 | 108 | ![Stage agent tasks are displayed.](media/Tasks.png) 109 | 110 | ## Artifacts 111 | 112 | Artifacts allow you to apply the same DevOps best practices to the packages you develop and maintain. 113 | 114 | The reality of the great features of Azure DevOps are that you can use them in conjunction with AI & machine Learning and use all the features whether Projects, Boards, repos, Pipelines, Test Plans or Artifacts to effectively be on top of you ML project, your models, your notebooks, your deployments and web services. 115 | 116 | ## Next steps 117 | 118 | You can learn more about Azure DevOps by reviewing these links to additional resources: 119 | 120 | - [What is Azure DevOps](https://docs.microsoft.com/azure/devops/user-guide/what-is-azure-devops) 121 | - [Start using Azure DevOps](https://docs.microsoft.com/azure/devops/user-guide/index) 122 | - [Start using Azure Boards](https://docs.microsoft.com/azure/devops/boards/get-started/index) 123 | - [Start using Azure Repos](https://docs.microsoft.com/azure/devops/repos/get-started/index) 124 | - [Azure Pipelines](https://docs.microsoft.com/azure/devops/pipelines/get-started/index) 125 | 126 | Read next: [Overview of post deployment monitoring and management tasks](./post-deployment-monitoring-and-management.md) 127 | -------------------------------------------------------------------------------- /intro/README.md: -------------------------------------------------------------------------------- 1 | # Intro 2 | 3 | - [What is Azure Machine Learning?](./what-is-azure-machine-learning.md) 4 | - [What tools are used to do data engineering, data science, and AI?](./tools.md) 5 | - [Overview of Azure Machine Learning service architecture and concepts](./architecture-overview.md) 6 | - [Getting your environment set up](./environment-setup.md) 7 | -------------------------------------------------------------------------------- /intro/architecture-overview.md: -------------------------------------------------------------------------------- 1 | # Overview of Azure Machine Learning service architecture and concepts 2 | 3 | Azure Machine Learning service provides a comprehensive environment you can use with a suite of [powerful tools](./tools.md) to perform data science and engineering processes. The components of this environment give you a centralized place to work with all the artifacts involved in the process. 4 | 5 | ## Components and concepts 6 | 7 | The top-level component of the Azure Machine Learning (AML) service is the [workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace). Your first step to using AML is to create a workspace within an Azure region and resource group (a logical grouping of Azure services). When you do this, the script creates all services alongside your workspace. Once created, you reference the workspace any time you need to train, evaluate, monitor, or deploy a model. 8 | 9 | Here is an example of creating a new workspace, using the Machine Learning CLI (command-line interface). See [Getting your environment set up](./environment-setup.md) for more information. 10 | 11 | ```bash 12 | az ml workspace create -w myworkspacename -g myresourcegroupname 13 | ``` 14 | 15 | The following diagram shows the high-level taxonomy of the workspace: 16 | 17 | ![The high-level taxonomy of the Azure Machine Learning workspace](./media/azure-machine-learning-taxonomy.png) 18 | 19 | The diagram includes the following components: 20 | 21 | - A workspace may contain [Notebook VMs](https://docs.microsoft.com/azure/machine-learning/service/quickstart-run-cloud-notebook), which are cloud resources pre-configured with a Python environment necessary to run Azure Machine Learning. 22 | - [User roles](https://docs.microsoft.com/azure/machine-learning/service/how-to-assign-roles) enable you to share your workspace with other users, teams, or projects. 23 | - [Compute targets](https://docs.microsoft.com/azure/machine-learning/service/concept-compute-target) are compute resources where you run your experiments or host your service deployment. The target may be your local machine or a cloud-based resource. 24 | - When you create the workspace, [associated resources](#associated-workspace-resources) are also created for you. 25 | - [Experiments](../modeling/feature-engineering-training-evaluation-selection/model-training/aml-experiment-runs.md) are training runs you use to build your models. You can create and run experiments with 26 | - The [Azure Machine Learning SDK for Python](https://docs.microsoft.com/python/api/overview/azure/ml/intro?view=azure-ml-py). 27 | - The [automated machine learning experiments](../modeling/simplify-process-with-automated-ml/README.md) section in the Azure portal. 28 | - The [visual interface](./tools.md#visual-interface) for drag-and-drop model building and deployment. 29 | - [Pipelines](../creating-machine-learning-pipelines/machine-learning-pipelines.md) are reusable workflows for training and retraining your model. 30 | - [Datasets](https://docs.microsoft.com/azure/machine-learning/service/how-to-explore-prepare-data) aid in the management of the data you use for model training and pipeline creation. 31 | - Once you have a model you want to deploy, you create a registered model. 32 | - Use the registered model and a scoring script to create a [deployment](../model-deployment/deployment-target-options.md). 33 | 34 | ## Associated workspace resources 35 | 36 | Beyond the workspace, there are a few related Azure resources that are used to manage, monitor, and secure the workspace components. These are created for you automatically when you create a new workspace. However, you can choose to use existing Azure services in place of creating new versions. 37 | 38 | - [Azure Container Registry](https://azure.microsoft.com/services/container-registry/): Registers docker containers that you use during training and when you deploy a model. To minimize costs, ACR is lazy-loaded until deployment images are created. This means that the ACR service is provisioned when you first register a model in your workspace. 39 | - [Azure Storage account](https://azure.microsoft.com/services/storage/): Is used as the default data store for the workspace. Jupyter notebooks that are used with your notebook VMs are stored here as well. 40 | - [Azure Application Insights](https://azure.microsoft.com/services/application-insights/): Allows you to monitor your models and view metrics on their usage. 41 | - [Azure Key Vault](https://azure.microsoft.com/services/key-vault/): Securely stores secrets and other sensitive information used by your workspace and any compute targets. 42 | 43 | ## High-level model workflow 44 | 45 | ![The stages shown are Train, Package, Validate, Deploy, and Monitor. An arrow labeled Retrain goes back to Train from Monitor.](media/aml-model-workflow.png 'Azure Machine Learning model workflow') 46 | 47 | ### 1 - Train 48 | 49 | At the core of the modern data science process is [training, evaluating, and selecting machine learning models](../modeling/feature-engineering-training-evaluation-selection/README.md). Leading up to this point, you have collected and prepared the data for training. The next step is to select an algorithm for your model then train it with data that has been evaluated and prepared with the transformations and features required for training. At a very high level, training a model with Azure Machine Learning service involves the following steps: 50 | 51 | - Using your favorite [Python environment](./environment-setup.md), create a machine learning training script along with any associated files. Specify the directory that contains these files, as well as an experiment name. Alternately, use visual interface for a code-free experience. 52 | - Create and configure a compute target for executing the training. During training, the complete directory copies to the compute target before the training script executes. 53 | - Submit the training scripts to the configured compute target. Afterward, the script starts running within the environment and has access to read from and write to [datastores](../data-acquisition-understanding/accessing-data.md#working-with-datastores). Each execution saves a record of the run within the workspace, grouped under experiments. 54 | 55 | Use the automated machine learning feature to select the best model during training automatically. This feature automates experimenting with multiple combinations of parameter values, also referred to as hyperparameter tuning, to accelerate the model training process and keeps a record of the outcomes to help identify potential areas of improvement more quickly. 56 | 57 | ### 2 - Package 58 | 59 | After you have trained your model and have identified the best version, you package it along with all the components you need to use the model, into an image. The image can be either a Docker image or an FPGA image used to deploy your model to a field-programmable gate array. The image saves to the image registry in your workspace. The registry provides a centralized place to store your models so they can be easily copied to new deployment targets, as well as versioned. 60 | 61 | ### 3 - Validate 62 | 63 | Model validation is used to calculate the accuracy of a model. Validation happens during the training process to make sure your chosen algorithm is performing as expected. It is also conducted periodically to ensure your model is still performing well over time with new data. Azure Machine Learning service allows you to query your experiments for logged metrics from current and past runs. Use the metrics to determine whether the run had the desired outcome. If not, begin the retraining process by starting over at step one. 64 | 65 | ### 4 - Deploy 66 | 67 | When you want your model to be available for on-demand access over the web or on an IoT device, you use the Azure Machine Learning SDK to deploy it as a web service in Azure, an FPGA, or to an IoT Edge device. The image that you created when you packaged the model is used to copy instances of the model, scoring script, and any dependencies to your deployment target. Your options for deploying the model as a web service are Azure Container Instance (ACI) and Azure Kubernetes Service (AKS). 68 | 69 | ### 5 - Monitor 70 | 71 | Monitor for changes in the distribution of data between the training dataset and inference data of your deployed model. These changes are sometimes called [data drift](https://docs.microsoft.com/azure/machine-learning/service/concept-data-drift) and indicate degraded prediction performance over time due to how the input data changes during this period. When you detect degraded model performance, the next step is to retrain your model with new data, thus creating a new version of the model. If your model is deployed to a web service or IoT devices, then you would use the new version of the model to redeploy it to those endpoints. The Azure Machine Learning SDK provides tools you can use to redeploy with minimal interruption of these services and endpoints. 72 | 73 | ## Next steps 74 | 75 | - [Quickstart guide for running a cloud notebook on a Notebook VM](https://docs.microsoft.com/azure/machine-learning/service/quickstart-run-cloud-notebook) 76 | - [How to assign user roles](https://docs.microsoft.com/azure/machine-learning/service/how-to-assign-roles) 77 | - [Compute targets](https://docs.microsoft.com/azure/machine-learning/service/concept-compute-target) 78 | - [Running model experiments](../modeling/feature-engineering-training-evaluation-selection/model-training/aml-experiment-runs.md) 79 | - [Azure Machine Learning SDK for Python](https://docs.microsoft.com/python/api/overview/azure/ml/intro?view=azure-ml-py) 80 | - [Automated machine learning experiments](../modeling/simplify-process-with-automated-ml/README.md) 81 | - [Visual interface](./tools.md#visual-interface) 82 | - [Building reusable workflows with Pipelines](../creating-machine-learning-pipelines/machine-learning-pipelines.md) 83 | - [Model deployment target options](../model-deployment/deployment-target-options.md) 84 | 85 | Read next: [What tools are used to do data engineering, data science, and AI?](./tools.md) 86 | -------------------------------------------------------------------------------- /intro/media/Diagrams.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/Diagrams.pptx -------------------------------------------------------------------------------- /intro/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/_placeholder -------------------------------------------------------------------------------- /intro/media/ai-spectrum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/ai-spectrum.png -------------------------------------------------------------------------------- /intro/media/aml-experiment-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-experiment-link.png -------------------------------------------------------------------------------- /intro/media/aml-experiment-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-experiment-run.png -------------------------------------------------------------------------------- /intro/media/aml-model-workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-model-workflow.png -------------------------------------------------------------------------------- /intro/media/aml-workspace-add-compute-form.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-workspace-add-compute-form.png -------------------------------------------------------------------------------- /intro/media/aml-workspace-add-compute.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-workspace-add-compute.png -------------------------------------------------------------------------------- /intro/media/aml-workspace-script-cloud-shell.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/aml-workspace-script-cloud-shell.png -------------------------------------------------------------------------------- /intro/media/azure-cloud-shell.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-cloud-shell.png -------------------------------------------------------------------------------- /intro/media/azure-create-resource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-create-resource.png -------------------------------------------------------------------------------- /intro/media/azure-machine-learning-taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-machine-learning-taxonomy.png -------------------------------------------------------------------------------- /intro/media/azure-ml-service-workspace-create-dialog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-ml-service-workspace-create-dialog.png -------------------------------------------------------------------------------- /intro/media/azure-ml-service-workspace-deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-ml-service-workspace-deployment.png -------------------------------------------------------------------------------- /intro/media/azure-mlservice-workspace-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-mlservice-workspace-search.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-add-environment-setup-step.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-add-environment-setup-step.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-clone-button.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-clone-button.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-profile-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-profile-link.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-profile.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-project-settings-button.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-project-settings-button.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-terminal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-terminal.png -------------------------------------------------------------------------------- /intro/media/azure-notebooks-upload-github-repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/azure-notebooks-upload-github-repo.png -------------------------------------------------------------------------------- /intro/media/new-notebook-vm-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/new-notebook-vm-link.png -------------------------------------------------------------------------------- /intro/media/new-notebook-vm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/new-notebook-vm.png -------------------------------------------------------------------------------- /intro/media/notebook-cells.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/notebook-cells.png -------------------------------------------------------------------------------- /intro/media/notebook-vm-jupyter-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/notebook-vm-jupyter-link.png -------------------------------------------------------------------------------- /intro/media/notebook-vm-stop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/notebook-vm-stop.png -------------------------------------------------------------------------------- /intro/media/simple-experiment-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/simple-experiment-run.png -------------------------------------------------------------------------------- /intro/media/steps-to-using-azureml.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/steps-to-using-azureml.png -------------------------------------------------------------------------------- /intro/media/visual-interface-compute-target.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-compute-target.png -------------------------------------------------------------------------------- /intro/media/visual-interface-create-kubernetes-compute-target.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-create-kubernetes-compute-target.png -------------------------------------------------------------------------------- /intro/media/visual-interface-deploy-web-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-deploy-web-service.png -------------------------------------------------------------------------------- /intro/media/visual-interface-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-experiment.png -------------------------------------------------------------------------------- /intro/media/visual-interface-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-link.png -------------------------------------------------------------------------------- /intro/media/visual-interface-predictive-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-predictive-experiment.png -------------------------------------------------------------------------------- /intro/media/visual-interface-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-run.png -------------------------------------------------------------------------------- /intro/media/visual-interface-web-service-compute-target.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-web-service-compute-target.png -------------------------------------------------------------------------------- /intro/media/visual-interface-web-service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/visual-interface-web-service.png -------------------------------------------------------------------------------- /intro/media/vs-code-aml-extension-workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vs-code-aml-extension-workspace.png -------------------------------------------------------------------------------- /intro/media/vs-code-aml-extension.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vs-code-aml-extension.png -------------------------------------------------------------------------------- /intro/media/vs-code-intellicode-extension.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vs-code-intellicode-extension.png -------------------------------------------------------------------------------- /intro/media/vs-code-python-interpreter-selected.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vs-code-python-interpreter-selected.png -------------------------------------------------------------------------------- /intro/media/vs-code-python-select-interpreter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vs-code-python-select-interpreter.png -------------------------------------------------------------------------------- /intro/media/vscode-azure-ml-extension-installing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/intro/media/vscode-azure-ml-extension-installing.png -------------------------------------------------------------------------------- /model-deployment/README.md: -------------------------------------------------------------------------------- 1 | # Model deployment 2 | 3 | - [Overview of deployment target options](./deployment-target-options.md) 4 | - [Overview of Real-time inferencing](./real-time-inferencing.md) 5 | - [Overview of Batch inferencing](./batch-inferencing.md) 6 | - [Overview of inferencing at the IoT edge](./iot-edge-inferencing.md) 7 | - [Reducing model deployment dependencies and improving model inferencing performance with ONNX](./deployment-with-onnx.md) 8 | -------------------------------------------------------------------------------- /model-deployment/deployment-with-onnx.md: -------------------------------------------------------------------------------- 1 | # Reducing model deployment dependencies and improving model inferencing performance with ONNX 2 | 3 | The challenges of optimizing the performance of machine learning models in production is extremely hard considering the range of hardware capabilities on different platforms such as cloud vs edge, CPU vs GPU etc. and couple that with the variety of frameworks that available for building and training the machine learning models. To support the idea of providing users the flexibility to train their machine learning models on framework of their choice, and run that model anywhere with optimal performance, is at the core of [Open Neural Network Exchange](https://onnx.ai/) (ONNX) format for representing machine learning models. ONNX is an open format, that is supported by a [community of partners](https://onnx.ai/supported-tools), including Microsoft, who create compatible frameworks and tools. 4 | 5 | You can create ONNX models from many frameworks, including PyTorch, Chainer, Microsoft Cognitive Toolkit (CNTK), MXNet, ML.Net, TensorFlow, Keras, SciKit-Learn, and more. ONNX Runtime is optimized for both cloud and edge and works on Linux, Windows, and Mac, and it also integrates with accelerators on different hardware such as TensorRT on NVidia GPUs. [ONNX models can be deployed](https://docs.microsoft.com/azure/machine-learning/service/how-to-build-deploy-onnx#deploy) to the cloud using Azure Machine Learning and ONNX Runtime. They can also be deployed to Windows 10 devices using [Windows ML](https://docs.microsoft.com/windows/ai/). They can even be deployed to other platforms using converters that are available from the ONNX community. 6 | 7 | ![ONNX flow diagram showing training, converters and deployment](media/onnx_overview.png 'ONNX Flow Diagram') 8 | 9 | The interoperability you get with ONNX makes it possible to get great ideas into production faster. With ONNX, data scientists can choose their preferred framework for the job. Similarly, developers can spend less time getting models ready for production, and deploy across the cloud and edge. 10 | 11 | - Data scientists: use the framework of their choice to create and train models 12 | - Developers: deploy models cross-platform with minimal integration work 13 | 14 | In this article, we will look at how to first convert a model train in Keras with Tensorflow backend to ONNX, and then deploy the ONNX model as a web service hosted on Azure Container Instance (ACI) for making predictions. 15 | 16 | ## Converting a model to ONNX and deploying to a web service hosted on ACI from an Azure Notebook (Code Sample) 17 | 18 | In the [intro](../intro/tools.md) section we learned how to get started with Azure notebooks. As described, Azure Notebooks is a pre-configured environment that is ready to work with Azure Machine Learning service. In this section we will look at code samples that were built in Azure Notebooks. 19 | 20 | ### Converting deep learning model to ONNX 21 | 22 | In this example, we will use a pre-trained Keras model (`model`) with Tensorflow backend. The [ONNXMLTools](https://github.com/onnx/onnxmltools) library provides support for converting models from various machine learning libraries, such as Keras, Tensorflow, scikit-learn, etc. to the ONNX format. The following example, show how to convert a Keras model to ONNX format: 23 | 24 | ```python 25 | import onnxmltools 26 | import os 27 | 28 | # Convert the Keras model to ONNX using target operator set version 7 29 | onnx_model_name = '...' 30 | onnx_model = onnxmltools.convert_keras(model, onnx_model_name, target_opset=7) 31 | 32 | # Save the onnx model locally 33 | onnx_model_path = '...' 34 | onnxmltools.utils.save_model(onnx_model, os.path.join(onnx_model_path, onnx_model_name)) 35 | ``` 36 | 37 | Next, let's review how you can use the ONNX format to make predictions. It is important to ensure that the input data shape matches the expected input shape by the ONNX inference session. 38 | 39 | ```python 40 | import onnxruntime 41 | import numpy as np 42 | 43 | # Load the ONNX model and observe the expected input shape 44 | onnx_session = onnxruntime.InferenceSession(onnx_model_path, onnx_model_name) 45 | 46 | # Confirm that the expected input shape and the shape test data: x_test match 47 | print('Expected input shape: ', onnx_session.get_inputs()[0].shape) 48 | print('Input shape: ', x_test.shape) 49 | 50 | # Run ONNX session to make predictions on test data: x_test 51 | x_test = x_test.astype(np.float32) 52 | onnx_session.run(None, {onnx_session.get_inputs()[0].name: x_test}) 53 | ``` 54 | 55 | ### Deploy deep learning ONNX format model as a web service 56 | 57 | #### Create a scoring script 58 | 59 | As described in the article [Overview of deployment target options](deployment-target-options.md) the first step to deploy the model to ACI is to register the model in registry hosted in your Azure Machine Learning Service workspace. Following model registration, you need to create the scoring script. Here an example of the scoring script: 60 | 61 | ```python 62 | %%writefile $score.py 63 | import sys, os, json 64 | import numpy as np 65 | from azureml.core.model import Model 66 | import onnxruntime 67 | 68 | def init(): 69 | global model 70 | 71 | try: 72 | model_name = '...' #model name should be the same as the registered model name 73 | print('Looking for model path for model: ', model_name) 74 | model_path = Model.get_model_path(model_name=model_name) 75 | # Load the ONNX model 76 | print('Looking for model in: ', model_path) 77 | model = onnxruntime.InferenceSession(model_path) 78 | print('Model loaded...') 79 | except Exception as e: 80 | print(e) 81 | 82 | def run(raw_data): 83 | try: 84 | print("Received input: ", raw_data) 85 | input_data = np.array(json.loads(raw_data)).astype(np.float32) 86 | # Run an ONNX session to classify the input 87 | result = model.run(None, {model.get_inputs()[0].name:input_data}) 88 | return result[0] 89 | except Exception as e: 90 | error = str(e) 91 | return error 92 | ``` 93 | 94 | #### Package model 95 | 96 | Next, you package the scoring script in a new Docker image. The image has two main components: (1) the environment file that captures all the required dependencies and (2) scoring script file. 97 | 98 | ```python 99 | # create a Conda dependencies environment file 100 | from azureml.core.conda_dependencies import CondaDependencies 101 | print("Creating conda dependencies file locally...") 102 | conda_packages = ['numpy'] 103 | pip_packages = ['azureml-sdk','onnxruntime'] 104 | mycondaenv = CondaDependencies.create(conda_packages=conda_packages, pip_packages=pip_packages) 105 | 106 | conda_file = '...' 107 | with open(conda_file, 'w') as f: 108 | f.write(mycondaenv.serialize_to_string()) 109 | 110 | runtime = 'python' 111 | 112 | # create container image configuration 113 | from azureml.core.image import ContainerImage 114 | print("Creating container image configuration...") 115 | image_config = ContainerImage.image_configuration(execution_script = 'score.py', 116 | runtime = runtime, conda_file = conda_file) 117 | 118 | # create the image 119 | from azureml.core import Image 120 | image_name = '...' 121 | image = Image.create(name=image_name, models=[model], image_config=image_config, workspace=ws) 122 | 123 | # wait for image creation to finish 124 | image.wait_for_creation(show_output=True) 125 | ``` 126 | 127 | #### Deploy model 128 | 129 | For low-scale and CPU-based, ACI is a good option, and can be easily created on the fly as shown below: 130 | 131 | ```python 132 | from azureml.core.webservice import AciWebservice, Webservice 133 | 134 | aci_name = '...' 135 | 136 | aci_config = AciWebservice.deploy_configuration( 137 | cpu_cores = 1, 138 | memory_gb = 1, 139 | tags = {'name': aci_name}, 140 | description = '...') 141 | 142 | service_name = '...' 143 | 144 | aci_service = Webservice.deploy_from_image(deployment_config=aci_config, 145 | image=image, 146 | name=service_name, 147 | workspace=ws) 148 | 149 | aci_service.wait_for_deployment(show_output=True) 150 | ``` 151 | 152 | #### Test deployment 153 | 154 | Finally, you can test your ACI deployment by consuming your deployed web service endpoint (Scoring URI) over HTTP. You can get the scoring URI by call calling the service object (shown below) or get it from Azure portal as shown in the [Overview of real-time inferencing](real-time-inferencing.md) article. 155 | 156 | ```python 157 | import requests 158 | 159 | url = aci_service.scoring_uri 160 | print('Scoring URI is: {}'.format(url)) 161 | 162 | # call the webservice over HTTP 163 | headers = {'Content-Type':'application/json'} 164 | response = requests.post(url, json.dumps(x_test.tolist()), headers=headers) 165 | 166 | # print the predictions 167 | print(response.text) 168 | ``` 169 | 170 | ## Next steps 171 | 172 | Please see the following additional references on Azure Machine Learning Visual Interface: 173 | 174 | - [ONNX](https://onnx.ai/) 175 | - [ONNX and Azure Machine Learning: Create and accelerate ML models](https://docs.microsoft.com/azure/machine-learning/service/concept-onnx) 176 | - [ONNX on Azure Machine Learning](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/deployment/onnx) 177 | 178 | Read next: [Creating machine learning pipelines](../creating-machine-learning-pipelines/machine-learning-pipelines.md) 179 | -------------------------------------------------------------------------------- /model-deployment/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/_placeholder -------------------------------------------------------------------------------- /model-deployment/media/aci-deployment-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aci-deployment-output.png -------------------------------------------------------------------------------- /model-deployment/media/aci-test-output-http.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aci-test-output-http.png -------------------------------------------------------------------------------- /model-deployment/media/aci-test-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aci-test-output.png -------------------------------------------------------------------------------- /model-deployment/media/aks-deployment-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aks-deployment-output.png -------------------------------------------------------------------------------- /model-deployment/media/aks-test-output-http.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aks-test-output-http.png -------------------------------------------------------------------------------- /model-deployment/media/aml-deployment-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-deployment-process.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-compute-failed-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-compute-failed-details.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-compute-failed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-compute-failed.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-deployment-details-aks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-deployment-details-aks.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-deployment-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-deployment-details.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-deployments-aci.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-deployments-aci.png -------------------------------------------------------------------------------- /model-deployment/media/aml-workspace-deployments-aks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/aml-workspace-deployments-aks.png -------------------------------------------------------------------------------- /model-deployment/media/azure-notebooks-ai-toolkit-iot-edge-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/azure-notebooks-ai-toolkit-iot-edge-run.png -------------------------------------------------------------------------------- /model-deployment/media/azure-notebooks-cloned-repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/azure-notebooks-cloned-repo.png -------------------------------------------------------------------------------- /model-deployment/media/azure-notebooks-upload-github-repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/azure-notebooks-upload-github-repo.png -------------------------------------------------------------------------------- /model-deployment/media/azure-portal-pipeline-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/azure-portal-pipeline-run.png -------------------------------------------------------------------------------- /model-deployment/media/cloud-shell-vm-usage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/cloud-shell-vm-usage.png -------------------------------------------------------------------------------- /model-deployment/media/lambda-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/lambda-architecture.png -------------------------------------------------------------------------------- /model-deployment/media/model-deployment-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/model-deployment-process.png -------------------------------------------------------------------------------- /model-deployment/media/onnx_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/onnx_overview.png -------------------------------------------------------------------------------- /model-deployment/media/pipeline-run-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/pipeline-run-details.png -------------------------------------------------------------------------------- /model-deployment/media/prediction-results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/prediction-results.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-create-predictive-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-create-predictive-experiment.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-deploy-web-service-button.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-deploy-web-service-button.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-deploy-web-service-dialog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-deploy-web-service-dialog.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-run-predictive-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-run-predictive-experiment.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-run-training-experiment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-run-training-experiment.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-save-trained-model-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-save-trained-model-link.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-save-trained-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-save-trained-model.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-web-service-api-doc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-web-service-api-doc.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-web-service-consume.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-web-service-consume.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-web-service-test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-web-service-test.png -------------------------------------------------------------------------------- /model-deployment/media/visual-interface-web-services.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/model-deployment/media/visual-interface-web-services.png -------------------------------------------------------------------------------- /modeling/README.md: -------------------------------------------------------------------------------- 1 | # Modeling 2 | 3 | - [Overview of Feature Engineering, Model Training, Model Evaluation and Model Selection](./feature-engineering-training-evaluation-selection.md) 4 | - [Feature Engineering introduced](./feature-engineering-training-evaluation-selection.md#feature-engineering-introduced) 5 | - [Model Training introduced](./model-training.md) 6 | - [Model Evaluation introduced](./model-evaluation.md) 7 | - [Training and Evaluating a simple model using Azure Machine Learning Visual Interface](./training-evaluating-model-with-visual-interface.md) 8 | - [Training and Evaluating a few simple models Using Azure Notebooks and Azure Machine Learning compute](./training-evaluating-simple-models-with-aml-compute.md) 9 | - [Simplifying the process with Automated Machine Learning, a component of Azure Machine Learning service](./simplify-process-with-automated-ml.md) 10 | - [Understanding automated machine learning generated models, using the model explainability capability of automated machine learning](./automl-understand-models-with-explainability.md) 11 | - [Using automated machine learning for Classification (Code Sample)](./automl-classification-code-sample.md) 12 | - [Using automated machine learning for Regression (Code Sample)](./automl-regression-code-sample.md) 13 | - [Using automated machine learning for Forecasting (Code Sample)](./automl-forecasting-code-sample.md) 14 | - [Using automated machine learning with model explainability (Code Sample)](./automl-understand-models-with-explainability.md#Model-explainability-code-sample) 15 | -------------------------------------------------------------------------------- /modeling/aml-estimators.md: -------------------------------------------------------------------------------- 1 | # Introducing AML Estimators 2 | 3 | An Estimator is an alternative higher-level abstraction used to construct run configurations when training deep learning models. For each training context, the Estimator contains the training code, the parameters, the compute resources specifications, and the runtime environment requirements. While the combination of `RunConfiguration` and `ScriptRunConfig` objects provides the highest level of control and flexibility, the `Estimator` object is designed to make the construction of run configurations easier. 4 | 5 | Here is an example of using an Estimator in its simplest form: 6 | 7 | ```python 8 | from azureml.train.estimator import Estimator 9 | 10 | script_params = { 11 | '--data-folder': ds.as_mount(), 12 | '--regularization': 0.8 13 | } 14 | 15 | # run an experiment from the train.py code in your current directory 16 | estimator = Estimator(source_directory='.', 17 | script_params=script_params, 18 | compute_target=, 19 | entry_script='train.py', 20 | conda_packages=['scikit-learn']) 21 | 22 | # submit the experiment and then wait until complete 23 | run = experiment.submit(estimator) 24 | run.wait_for_completion() 25 | ``` 26 | Notice the parameters sent to the training script, the option to specify any `compute_target` (must be one that has already been created), and the requirement to install the `scikit-learn` Python package via Conda (there is an additional parameter available named `pip_packages` that can be used to specify packages to be installed via pip). 27 | 28 | ## Parallel and distributed (multi-node) training 29 | 30 | One particular limitation of the Estimator created in the code sample above is the fact that is runs in a single-node, non-parallel scenario. Consequently, it does not take advantage of the multi-node clusters that most Azure Machine Learning service compute targets can provide. The constructor of the `Estimator` object provides three additional paramters that control multi-node and parallel scenarios: 31 | 32 | Name | Description 33 | --- | --- 34 | node_count | The number of nodes that will be used to run the training job. A value higher than 1 activates the distributed scenario. Only AML Compute targets are currently supported. 35 | process_count_per_node | The number of worker processes that will run on each node. A value higher than 1 activates the parallel scenario. Note when this value is higher then 1, parallel execution will happen even if the value of `node_count` is 1. Only AML Compute targets are currently supported. 36 | distributed_backend | When either `node_count` or `process_count_per_node` have a value higher than 1, must be set to `mpi` to instruct Azure Machine Learning to use its MPI implementation (based on [Open MPI](https://www.open-mpi.org/)). 37 | 38 | The modified, multi-node and parallelized version of the Estimator will look like this: 39 | 40 | ```python 41 | from azureml.train.estimator import Estimator 42 | 43 | script_params = { 44 | '--data-folder': ds.as_mount(), 45 | '--regularization': 0.8 46 | } 47 | 48 | # run an experiment from the train.py code in your current directory 49 | estimator = Estimator(source_directory='.', 50 | script_params=script_params, 51 | compute_target=, 52 | entry_script='train.py', 53 | node_count = 2, 54 | process_count_per_node = 2, 55 | distributed_backend = 'mpi', 56 | conda_packages=['scikit-learn']) 57 | 58 | # submit the experiment and then wait until complete 59 | run = experiment.submit(estimator) 60 | run.wait_for_completion() 61 | ``` 62 | 63 | ## Using custom Docker images 64 | 65 | One additional option available with Estimators is the possiblity of specifying your own Docker image as a base for building the execution environment. The constructor of the `Estimator` object provides additional paramters to configure the use of custom images. Out of these, the most important two are: 66 | 67 | Name | Description 68 | --- | --- 69 | custom_docker_image | The name of the Docker image to be used. Must be an image from a public repository, like Docker Hub. 70 | environment_definition | Used when an image from a private Docker repository must be used. Note that this parameter controlls also other settings, not related to the image to be used. 71 | 72 | ## Specialized Estimators 73 | 74 | In addition to the general-purpose `Estimator` object, Azure Machine Learning provides the following specialized Estimators: 75 | 76 | Name | Description 77 | --- | --- 78 | SKLearn | Used to launch a [Scikit-learn](https://scikit-learn.org/stable/index.html) training job on a compute target. 79 | TensorFlow | Used to launch a [TensorFlow](https://www.tensorflow.org) training job on a compute target. If the `pip_packages` argument is used to import Keras on top of TensorFlow, this also can act as a [Keras](https://keras.io) Estimator. 80 | PyTorch | Used to launch a [PyTorch](https://pytorch.org) training job on a compute target. 81 | Chainer | Used to launch a [Chainer](https://chainer.org) training job on a compute target. 82 | 83 | ## Next steps 84 | 85 | You can learn more about Estimators by reviewing these links to additional resources: 86 | 87 | - [Train models with Azure Machine Learning using estimator](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-ml-models) 88 | - [Train and register Scikit-learn models at scale with Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-scikit-learn) 89 | - [Train and register Tensorflow models at scale with Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-tensorflow) 90 | - [Train and register Keras models at scale with Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-keras) 91 | - [Train and register PyTorch models at scale with Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch) 92 | - [Train and register Chainer models at scale with Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-chainer) 93 | 94 | Read next: [Introducing AML compute options (local and Azure Machine Learning compute)](./aml-compute-options.md) 95 | -------------------------------------------------------------------------------- /modeling/aml-experiment-runs.md: -------------------------------------------------------------------------------- 1 | # Introducing the AML Experiment and Runs 2 | 3 | An Azure Machine Learning Experiment is a generic context for handling runs. Think about it a logical entity you can use to organize your Model Training processes. 4 | 5 | An Azure Machine Learning Run is used to build a trained model. Contains all artifacts associated with a training process, like output files, metrics, logs, and a snapshot of the directory that contains your scripts. A Run is always part of an Experiment. 6 | 7 | The following picture displays the relationship between an Experiment and its Runs as seen in the Azure portal: 8 | 9 | ![Azure Machine Learning Experiment and associated Runs](./media/experiment-and-runs-in-portal.png) 10 | 11 | You can also interact programatically with Experiments and Runs using either the Azure Machine Learning SDK for Python or the Azure CLI (Command Line Interface). 12 | 13 | Using the SDK, you need to intialize the Workspace class which will then give you access to Experiments and Runs. 14 | 15 | ```python 16 | import azureml.core 17 | from azureml.core import Workspace, Experiment, Run 18 | from azureml.core import ScriptRunConfig 19 | 20 | ws = Workspace.from_config() 21 | exp = Experiment(workspace=ws, name="explore-runs") 22 | 23 | # Start a run and its logging process 24 | run = exp.start_logging() 25 | run.log(name="message", value="Hello from run!") 26 | ``` 27 | 28 | Using the CLI, you need to login, attach a workspace configuration to the folder that contains your training script, and then start the run. `run_config_name` is the name of the run configuration file (run configurations are described later in this article). `train.py` is the name of Python code file that contains your code to be executed. 29 | 30 | ``` 31 | az login 32 | 33 | az ml folder attach -w -g 34 | 35 | az ml run submit-script -c -e train.py 36 | ``` 37 | 38 | ## Experiments 39 | 40 | An Experiment is basically an entity used to group together mutiple runs of a specific script. An Experiment always belongs to a Workspace, and a Run always belongs to an Experiment. A new Experiment is automatically created if you submit a Run the specifies an Experiment name that does not exist. 41 | 42 | An Experiment tracks all its associated Runs. You will be able to see the full list of Runs together with their associated states which can be one of the following: 43 | 44 | - Preparing 45 | - Running 46 | - Completed 47 | - Failed 48 | 49 | As presented in the picture above, using the Azure Portal you can also configure and view charts with various metrics associated with the Runs. 50 | 51 | The same information can be obtained programatically using the SDK. 52 | 53 | ![Get Azure Machine Learning Experiment with the Python SDK](./media/experiment-in-sdk.png) 54 | 55 | You can also get the list of the Runs associated with an Experiment. 56 | 57 | ![Get Azure Machine Learning Runs for Experiment](./media/experiment-in-sdk-runs.png) 58 | 59 | ## Runs 60 | 61 | Submitting a script to train a model produces a Run (in the context of a specified Experiment). The following elements are associated with a Run: 62 | 63 | Name | Description 64 | --- | -- 65 | Attributes (metadata) | Values like Status, Created Time, Duration, Run Id, Run Number, and others. 66 | Tags | String values associated with the Run like `azureml.nodeid`. 67 | Tracked Metrics | Performance metrics like `loss` and `acc`. The list depends on the algorithm used to train the model. 68 | Charts | You can log vectors and/or tables from code using `run.log_list(name, value, description)` and then view charts based on these values in the Azure Portal. 69 | Images | You can log image files or `matplotlib` plots from code using `run.log_image(name, path, plot)` and then view them in the Azure Portal. 70 | Outputs | Output files that are either collected automatically by the experiment or are explicitly uploaded by you. 71 | Logs | Various log files generated during the execution of the training script. 72 | Snapshot | When submitting a run, the directory that contains the script file is sent in a compressed form to the compute target being used and also stored as a snapshot. The actual content of the directory that ends up in the zip file can be controlled by placing in it an `.amlignore` or a `.gitignore` file. 73 | 74 | All these elements can be accessed using the Run class in the SDK. 75 | 76 | ![Get the details of an Azure Machine Learning service Run using the Python SDK](./media/run-in-sdk-details.png) 77 | 78 | There are four different ways to submit a Run in an Experiment using code: 79 | 80 | - Use a Run Configuration (see the section below for details). 81 | - Use an Estimator object (see [Introducing AML Estimators](./aml-estimators.md)) 82 | - Use the Azure CLI extension (via `az ml run submit-script`) 83 | - Use the Visual Studio Code extension (see [Train and tune models from Visual Studio Code](https://docs.microsoft.com/azure/machine-learning/service/how-to-vscode-tools#train-and-tune-models)) 84 | 85 | ### Run configurations 86 | 87 | From the point of view of the development process, the most common scenario is the one in which you start working on your local machine. Once you are satisfied with the code being developed, you will want to run that traning script on a more powerfull/scalable compute resource. The concept of a Run Configuration implemented by the Azure Machine Learning service enables you to run the script of different compute targets without the need to make changes in the script itself. 88 | 89 | A separate Run Configuration will be used for each compute target. The `RunConfiguration` class from the SDK implements the required functionalities. In adition to actually targeting a specific compute target, it also provides support for managing the environment and dependencies on the compute target. 90 | 91 | Once you have access to the `Experiment` object in your code, you can use the `ScriptRunConfig` object to submit a run. `ScriptRunConfig` is a helper class that packages the `RunConfiguration` object mentioned above with the actual execution script that performs the training of the model. 92 | 93 | ```python 94 | import azureml.core 95 | from azureml.core import Workspace, Experiment, Run 96 | from azureml.core import ScriptRunConfig 97 | 98 | ws = Workspace.from_config() 99 | exp = Experiment(workspace=ws, name="explore-runs") 100 | 101 | from azureml.core import ScriptRunConfig 102 | import os 103 | 104 | script_folder = os.getcwd() 105 | # Create a configuration to target local compute. 106 | src = ScriptRunConfig(source_directory = script_folder, script = 'train.py', run_config = run_local) 107 | # Submit the run 108 | run = exp.submit(src) 109 | # Wait for the run to complete 110 | run.wait_for_completion(show_output = True) 111 | ``` 112 | 113 | Switching to a different compute target is just a matter of changing the value of the `run_config` parameter in the `ScriptRunConfig` constructor. This is an example that would target Azure Machine Learning compute: 114 | 115 | ```python 116 | # Create a configuration to target local compute. 117 | src = ScriptRunConfig(source_directory = script_folder, script = 'train.py', run_config = run_local) 118 | # Submit the run 119 | run = exp.submit(src) 120 | # Wait for the run to complete 121 | run.wait_for_completion(show_output = True) 122 | ``` 123 | 124 | ### Child runs 125 | 126 | A Run can have child runs which can be viewed in the Azure Portal by checking the `Include child runs` option. 127 | 128 | ![Azure Machine Learning service Runs and Child Runs](./media/runs-and-child-runs.png) 129 | 130 | The same information can be obtained programatically using the SDK by setting the `include_children` parameter to `True` in the `get_runs()` method of the Experiment object. Notice the difference from the previous example - the list now also includes the child runs. 131 | 132 | ![Get Azure Machine Learning Runs and Child Runs for Experiment](./media/experiment-in-sdk-runs-with-children.png) 133 | 134 | The typicall case involving child runs is the use on [Azure Machine Learning service Pipelines](https://docs.microsoft.com/azure/machine-learning/service/concept-ml-pipelines). 135 | 136 | Conceptually, your code will look like this (parts were omitted for improved clarity): 137 | 138 | ```python 139 | trainStep = PythonScriptStep( 140 | name="train", 141 | script_name="train.py", 142 | arguments=["--model_name", ], 143 | compute_target=aml_compute, 144 | runconfig=run_amlcompute, 145 | source_directory=scripts_folder, 146 | allow_reuse=False 147 | ) 148 | 149 | evaluate_output = PipelineData('evaluate_output', datastore=def_blob_store) 150 | 151 | evaluateStep = PythonScriptStep( 152 | name="evaluate", 153 | script_name="evaluate.py", 154 | arguments=["--model_name", , 155 | "--image_name", , 156 | "--output", evaluate_output], 157 | outputs=[evaluate_output], 158 | compute_target=aml_compute, 159 | runconfig=run_amlcompute, 160 | source_directory=scripts_folder, 161 | allow_reuse=False 162 | ) 163 | 164 | # Link steps 165 | evaluateStep.run_after(trainStep) 166 | steps = [evaluateStep] 167 | 168 | # Create pipeline 169 | pipeline = Pipeline(workspace=ws, steps=steps) 170 | 171 | # Execute pipeline 172 | pipeline_run = Experiment(ws, experiment_name).submit(pipeline) 173 | 174 | # Wait for pipeline to finish 175 | pipeline_run.wait_for_completion(show_output=True) 176 | ``` 177 | For a more detailed discussion about pipelines see the [Overview of machine learning pipelines using the Azure Machine Learning SDK](../../../creating-machine-learning-pipelines/machine-learning-pipelines.md) section of this guide. 178 | 179 | **Note**: You can also directly create child runs using the SDK using the `child_run()` method of the `Run` object. Either way, child runs can only be created using the SDK. 180 | 181 | ## Next steps 182 | 183 | You can learn more about Experiments and Runs by reviewing these links to additional resources: 184 | 185 | - [How Azure Machine Learning service works: Architecture and concepts](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) 186 | - [Start, monitor, and cancel training runs in Python](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-runs) 187 | - [Set up compute targets for model training](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets) 188 | - [Get started with Azure Machine Learning for Visual Studio Code](https://docs.microsoft.com/azure/machine-learning/service/how-to-vscode-tools) 189 | - [Build reusable ML pipelines in Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/concept-ml-pipelines) 190 | 191 | 192 | Read next: [Introducing AML Estimators](./aml-estimators.md) 193 | -------------------------------------------------------------------------------- /modeling/aml-model-registry.md: -------------------------------------------------------------------------------- 1 | # Introducing the AML Model Registry 2 | 3 | The most important result produced by a Run in the context of an Experiment is a Model. Basically, a trained Model is a piece of code that takes some inputs and produces some outputs and is registered with the Azure Machine Learning service. In additon to models trained by Runs, you can also register models that were trained elswhere than on a compute target provided by Azure Machine Learning. In fact, Azure Machine Learning supports any trained model that can be loaded through Python 3, regardless of where it was trained. 4 | 5 | You can use the SDK to register a model programatically. 6 | 7 | ```python 8 | model_description = 'Some model description.' 9 | model = Model.register( 10 | model_path='model.h5', # this points to a local file 11 | model_name=, # this is the name the model is registered as 12 | tags={"type": "classification", "run_id": run.id}, # tags associated to the model 13 | description=model_description, 14 | workspace=run.experiment.workspace 15 | ) 16 | ``` 17 | 18 | All registered models are available through the Azure Portal. 19 | 20 | ![Azure Machine Learning Modes in Azure Portal](./media/model-in-portal.png) 21 | 22 | Every registered model has a set of properties managed by the Azure Machine Learning service. 23 | 24 | ![Azure Machine Learning Model details in Azure Portal](./media/model-properties-in-portal.png) 25 | 26 | Each model (and its associated properties) can also be retrieved programatically using the SDK. 27 | 28 | ![Azure Machine Learning Model details using the Python SDK](./media/model-in-sdk.png) 29 | 30 | ## Model files and versions 31 | 32 | A registered model corresponds to one or more files that make up the model. When the result of a model training contains multiple files, you can register the set of file as a single model in the Azure Machine Learning workspace. Once a model is registered, you can download any of the files that were specified during registration. 33 | 34 | Within the context provided by the same model name, multiple versions cand be managed. Each new model registration referring to the same name results in a new version being created (using increments of 1). It is also recommended to use the tags mechanism provided by the Model Registry to tag each registered model version. This will help a lot later in searching for a specific model or a specific model version. 35 | 36 | ## Next steps 37 | 38 | You can learn more about the Model Registry by reviewing these links to additional resources: 39 | 40 | - [How Azure Machine Learning service works: Architecture and concepts - Models](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#models) 41 | 42 | Read next: [Model Evaluation introduced](./model-evaluation.md) 43 | -------------------------------------------------------------------------------- /modeling/automl-classification-code-sample.md: -------------------------------------------------------------------------------- 1 | # Using automated machine learning for Classification (Code Sample) 2 | 3 | 4 | ## The dataset used in this example 5 | 6 | In this classification example we will solve the problem of [Sonar Classification](https://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+(Sonar,+Mines+vs.+Rocks)) where `y` (the class to be predicted) has a cardinality of 2 (binary classification). In this particular case, `X` consists of 60 numerical features (normalized values of sonar readings at various angles) and `y` has two categories - Rock (R) or Mine (M). The problem at hand is to create a classifier that, given the sonar readings, will predict whether an object being detected is a rock or a mine (or, alternatively, whether an object being detected is a mine or not). The case R (not a mine) is coded as 1 and the case M (a mine) is coded as 2. 7 | 8 | Note: All code snippets in this section are designed to run in [Azure Notebooks](https://notebooks.azure.com/). 9 | 10 | ## Prepare your environment 11 | 12 | The first step you will perform is installing/updating your Azure Machine Learning service SDK: 13 | 14 | ```python 15 | !pip install --upgrade azureml-sdk[notebooks,explain,automl,contrib] 16 | ``` 17 | 18 | Once you have the latest version of the SDK, you will need to do the necessary imports and create a local folder to store various artifacts: 19 | 20 | ```python 21 | import os 22 | import logging 23 | import numpy as np 24 | import pandas as pd 25 | from sklearn.model_selection import train_test_split 26 | 27 | import azureml.core 28 | from azureml.core import Workspace, Experiment 29 | from azureml.core.authentication import InteractiveLoginAuthentication 30 | from azureml.core.compute import ComputeTarget, AmlCompute 31 | from azureml.core.runconfig import RunConfiguration 32 | from azureml.core.conda_dependencies import CondaDependencies 33 | 34 | import azureml.dataprep as dprep 35 | 36 | from azureml.train.automl import AutoMLConfig 37 | from azureml.train.automl.run import AutoMLRun 38 | 39 | from azureml.widgets import RunDetails 40 | 41 | experiment_name = 'sonar-binary-classifier' 42 | project_folder = './sonar-binary-classifier' 43 | 44 | # Create a project_folder if it doesn't exist 45 | if not os.path.exists(project_folder): 46 | os.makedirs(project_folder) 47 | ``` 48 | 49 | Next, get a hold on the compute resource you will use to train your model: 50 | 51 | ```python 52 | cpu_cluster = ComputeTarget(workspace=ws, name='aml01') 53 | ``` 54 | Note: The creation of the `Workspace` variable `ws` is ommited for brewity. Also, we are assuming there already exists an Azure Machine Learning compute resource named `aml01` (which means we are going to submit an AutoML experiment run to a remote compute resource, not the local machine). 55 | 56 | ## Prepare input data 57 | 58 | Once everything is in place, you will start loading, analyzing, and preparing your input data: 59 | 60 | ```python 61 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/sonar/sonar.all-data.csv', 62 | header=dprep.api.dataflow.PromoteHeadersMode.NONE, 63 | infer_column_types=True) 64 | 65 | X = data_flow.keep_columns(['Column{:d}'.format(x) for x in range(1,61)]).to_pandas_dataframe() 66 | y = data_flow.keep_columns(['Column61']).to_pandas_dataframe() 67 | 68 | X_train, X_test, y_train, y_test = train_test_split( 69 | X, y, test_size=0.2, random_state=111) 70 | 71 | data_flow.get_profile() 72 | ``` 73 | 74 | Notice how we are splitting the original data set into input features (`X`) and output(`y`) - the categorical feature we are looking to predict. 75 | 76 | ## Create the AutoML configuration 77 | 78 | The following elements are needed to create an AutoML configuration: 79 | - A data script which will be used on the remote compute resource to get the data to train the model. 80 | - A `RunConfiguration` which provides details about the creation and initialization of the Python environment on the remote compute resource. 81 | - A set of AutoML experiment settings including the number of iterations, the maximum time the experiment is allowed to run, the primary metric used to rank resulting models,and the level of logging. 82 | 83 | The data script must be saved localy (as it will be referenced by the AutoML configuration) and it has basically the same code you used above to prepare input data: 84 | 85 | ```python 86 | %%writefile $project_folder/get_data.py 87 | 88 | import azureml.dataprep as dprep 89 | from sklearn.model_selection import train_test_split 90 | 91 | def get_data(): 92 | 93 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/sonar/sonar.all-data.csv', 94 | header=dprep.api.dataflow.PromoteHeadersMode.NONE, 95 | infer_column_types=True) 96 | 97 | X = data_flow.keep_columns(['Column{:d}'.format(x) for x in range(1,61)]).to_pandas_dataframe() 98 | y = data_flow.keep_columns(['Column61']).to_pandas_dataframe() 99 | 100 | X_train, X_test, y_train, y_test = train_test_split( 101 | X, y, test_size=0.2, random_state=111) 102 | 103 | return { "X" : X_train.values, "y" : y_train.values.flatten() } 104 | ``` 105 | 106 | The run configuration includes details about the remote compute resource, the image used to initialize, and the library dependencies that need to be installed: 107 | 108 | ```python 109 | run_config = RunConfiguration(framework="python") 110 | run_config.target = cpu_cluster 111 | run_config.environment.docker.enabled = True 112 | run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE 113 | 114 | dependencies = CondaDependencies.create( 115 | pip_packages=["scikit-learn", "azureml-sdk[automl]","azureml-dataprep", "azureml-explain-model"]) 116 | run_config.environment.python.conda_dependencies = dependencies 117 | ``` 118 | 119 | Finally, you create the settings object and then initialize the `AutoMLConfig` object: 120 | 121 | ```python 122 | automl_settings = { 123 | "name": experiment_name, 124 | "iteration_timeout_minutes": 10, 125 | "iterations": 10, 126 | "n_cross_validations": 5, 127 | "primary_metric": 'accuracy', 128 | "preprocess": True, 129 | "max_concurrent_iterations": 10, 130 | "verbosity": logging.INFO 131 | } 132 | 133 | automl_config = AutoMLConfig(task='classification', 134 | debug_log='automl_errors.log', 135 | path=project_folder, 136 | compute_target=cpu_cluster, 137 | run_configuration=run_config, 138 | data_script=project_folder + "/get_data.py", 139 | model_explainability=True, 140 | **automl_settings, 141 | ) 142 | ``` 143 | We have now defined a classification task that will use the `accuracy` measure as primary metric for model ranking (and selection). 144 | 145 | ## Running the AutoML experiment 146 | 147 | Once you have the `AutoMLConfig` object properly initialized, you are ready to submit your experiment. Depending on the various settings you used, the experiment will run for several minutes or more. Using the `show_output=True` options enables you to get updates on the execution. 148 | 149 | ![Execution progress for an AutoML classification experiment run](./media/automl-classification-execution-progress.png) 150 | 151 | Note how the best value of the specified metric (accuracy in our case) is tracked throughout the entire execution. In this particular example, the `VotingEnsemble` pipeline yielded the top trained model, with an accuracy of 0.8255. 152 | 153 | Once the execution is finished you can request detailed information about the results. 154 | 155 | ![Execution results for an AutoML classification experiment run](./media/automl-classification-execution-results.png) 156 | 157 | Also, you can get all the metrics recorded during the experiment run. 158 | 159 | ![Execution metrics for an AutoML classification experiment run](./media/automl-classification-execution-metrics.png) 160 | 161 | ## Retrieve the best model and use it on test data 162 | 163 | Now that you have several trained models ranked based on the metric you specified when configuring the AutoML experiment, you can retrieve the best one and either use it to score immediately (on test data for example) or register and deploy is as a service. 164 | 165 | Retrieving the best model and using it to score on test data requires the use of `get_output()`: 166 | 167 | ```python 168 | best_run, fitted_model = run.get_output() 169 | 170 | y_predict = fitted_model.predict(X_test.values) 171 | print(y_predict) 172 | ``` 173 | 174 | You have now successfully configured an AutoML experiment, submitted it to run on a remote compute resource, analyzed the results it produced, retrieved its best trained model, and used this best model to score on test data. 175 | 176 | ## Next steps 177 | 178 | You can learn more about using automated machine learning for Classification by reviewing these links to additional resources: 179 | 180 | - [What is automated machine learning?](https://docs.microsoft.com/azure/machine-learning/service/concept-automated-ml) 181 | 182 | Read next: [Using automated machine learning for Regression (Code Sample)](./automl-regression-code-sample.md) -------------------------------------------------------------------------------- /modeling/automl-forecasting-code-sample.md: -------------------------------------------------------------------------------- 1 | # Using automated machine learning for Forecasting (Code Sample) 2 | 3 | ## The dataset used in this example 4 | 5 | In this regression example we will solve the problem of forecasting the number of bike rentals using the [UCI Bike Sharing Dataset](https://archive.ics.uci.edu/ml/datasets/Bike+Sharing+Dataset) which is based on real data from the Capital Bikeshare company that maintains a bike rental network in Washington DC in the United States. The dataset represents the number of bike rentals within a specific hour of a day in the years 2011 and year 2012 and contains 17389 rows and 17 columns. The raw feature set contains weather conditions (temperature/humidity/wind speed) and the type of the day (holiday/weekday). The field to forecast is the `cnt` count, which represents the bike rentals within a specific hour and which ranges from 1 to 977. 6 | 7 | Note: All code snippets in this section are designed to run in [Azure Notebooks](https://notebooks.azure.com/). 8 | 9 | ## Prepare your environment 10 | 11 | The first step you will perform is installing/updating your Azure Machine Learning service SDK: 12 | 13 | ```python 14 | !pip install --upgrade azureml-sdk[notebooks,explain,automl,contrib] 15 | ``` 16 | 17 | Once you have the latest version of the SDK, you will need to do the necessary imports and create a local folder to store various artifacts: 18 | 19 | ```python 20 | import os 21 | import logging 22 | import numpy as np 23 | import pandas as pd 24 | from sklearn.model_selection import train_test_split 25 | 26 | import azureml.core 27 | from azureml.core import Workspace, Experiment 28 | from azureml.core.authentication import InteractiveLoginAuthentication 29 | from azureml.core.compute import ComputeTarget, AmlCompute 30 | from azureml.core.runconfig import RunConfiguration 31 | from azureml.core.conda_dependencies import CondaDependencies 32 | 33 | import azureml.dataprep as dprep 34 | 35 | from azureml.train.automl import AutoMLConfig 36 | from azureml.train.automl.run import AutoMLRun 37 | 38 | from azureml.widgets import RunDetails 39 | 40 | experiment_name = 'bike-sharing-forecasting' 41 | project_folder = './bike-sharing-forecasting' 42 | 43 | # Create a project_folder if it doesn't exist 44 | if not os.path.exists(project_folder): 45 | os.makedirs(project_folder) 46 | ``` 47 | 48 | Next, get a hold on the compute resource you will use to train your model: 49 | 50 | ```python 51 | cpu_cluster = ComputeTarget(workspace=ws, name='aml01') 52 | ``` 53 | 54 | Note: The creation of the `Workspace` variable `ws` is ommited for brewity. Also, we are assuming there already exists an Azure Machine Learning compute resource named `aml01` (which means we are going to submit an AutoML experiment run to a remote compute resource, not the local machine). 55 | 56 | ## Prepare input data 57 | 58 | Once everything is in place, you will start loading, analyzing, and preparing your input data: 59 | 60 | ```python 61 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/bike/bike-rental-hour.csv', 62 | header=dprep.api.dataflow.PromoteHeadersMode.UNGROUPED, 63 | infer_column_types=True) 64 | 65 | X = data_flow.keep_columns(['instant', 'season','mnth','hr','holiday','weekday','workingday','weathersit','temp','atemp','hum','windspeed']).to_pandas_dataframe() 66 | y = data_flow.keep_columns(['cnt']).to_pandas_dataframe() 67 | 68 | X_train = X[:-20] 69 | X_test = X[-20:] 70 | y_train = y[:-20] 71 | y_test = y[-20:] 72 | 73 | data_flow.get_profile() 74 | ``` 75 | 76 | Notice how we are splitting the original data set into input features (`X`) and output(`y`) - the numerical feature we are looking to forecast. Also notice the list of input features we are considering for our forecasting model. For the purpose of testing our model, we are holding out the last 20 values of the series as test data. 77 | 78 | Let's visualize the evolution of the time series data (and also highlight training and test data): 79 | 80 | ![Actual evolution of time series data with training and test split highlight](./media/automl-forecasting-actual-evolution.png) 81 | 82 | ## Create the AutoML configuration 83 | 84 | The following elements are needed to create an AutoML configuration: 85 | 86 | - A data script which will be used on the remote compute resource to get the data to train the model. 87 | - A `RunConfiguration` which provides details about the creation and initialization of the Python environment on the remote compute resource. 88 | - A set of AutoML experiment settings including the number of iterations, the maximum time the experiment is allowed to run, the primary metric used to rank resulting models,and the level of logging. 89 | 90 | The data script must be saved localy (as it will be referenced by the AutoML configuration) and it has basically the same code you used above to prepare input data: 91 | 92 | ```python 93 | %%writefile $project_folder/get_data.py 94 | 95 | import azureml.dataprep as dprep 96 | from sklearn.model_selection import train_test_split 97 | 98 | def get_data(): 99 | 100 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/bike/bike-rental-hour.csv', 101 | header=dprep.api.dataflow.PromoteHeadersMode.UNGROUPED, 102 | infer_column_types=True) 103 | 104 | X = data_flow.keep_columns(['instant', 'season','mnth','hr','holiday','weekday','workingday','weathersit','temp','atemp','hum','windspeed']).to_pandas_dataframe() 105 | y = data_flow.keep_columns(['cnt']).to_pandas_dataframe() 106 | 107 | X_train = X[:-20] 108 | y_train = y[:-20] 109 | 110 | return { "X" : X_train, "y" : y_train.values.flatten() } 111 | ``` 112 | 113 | The run configuration includes details about the remote compute resource, the image used to initialize, and the library dependencies that need to be installed: 114 | 115 | ```python 116 | run_config = RunConfiguration(framework="python") 117 | run_config.target = cpu_cluster 118 | run_config.environment.docker.enabled = True 119 | run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE 120 | 121 | dependencies = CondaDependencies.create( 122 | pip_packages=["scikit-learn", "azureml-sdk[automl]","azureml-dataprep", "azureml-explain-model"]) 123 | run_config.environment.python.conda_dependencies = dependencies 124 | ``` 125 | 126 | Finally, you create the settings object and then initialize the `AutoMLConfig` object: 127 | 128 | ```python 129 | automl_settings = { 130 | "name": experiment_name, 131 | "iteration_timeout_minutes": 10, 132 | "iterations": 10, 133 | "n_cross_validations": 5, 134 | "primary_metric": 'normalized_root_mean_squared_error', 135 | "max_concurrent_iterations": 10, 136 | "verbosity": logging.INFO, 137 | 138 | "time_column_name": "instant", 139 | "max_horizon": 20 140 | } 141 | 142 | automl_config = AutoMLConfig(task='forecasting', 143 | debug_log='automl_errors.log', 144 | path=project_folder, 145 | compute_target=cpu_cluster, 146 | run_configuration=run_config, 147 | data_script=project_folder + "/get_data.py", 148 | model_explainability=True, 149 | **automl_settings, 150 | ) 151 | ``` 152 | 153 | We have now defined a regression task that will use the `normalized_root_mean_squared_error` measure as primary metric for model ranking (and selection). 154 | 155 | ## Running the AutoML experiment 156 | 157 | Once you have the `AutoMLConfig` object properly initialized, you are ready to submit your experiment. Depending on the various settings you used, the experiment will run for several minutes or more. Using the `show_output=True` options enables you to get updates on the execution. 158 | 159 | ![Execution progress for an AutoML forecasting experiment run](./media/automl-forecasting-execution-progress.png) 160 | 161 | Note how the best value of the specified metric (normalized_root_mean_squared_error in our case) is tracked throughout the entire execution. In this particular example, the `VotingEnsemble` pipeline yielded the top trained model, with an error of 0.0311. 162 | 163 | Once the execution is finished you can request detailed information about the results. 164 | 165 | ![Execution results for an AutoML forecasting experiment run](./media/automl-forecasting-execution-results.png) 166 | 167 | Also, you can get all the metrics recorded during the experiment run. 168 | 169 | ![Execution metrics for an AutoML forecasting experiment run](./media/automl-forecasting-execution-metrics.png) 170 | 171 | ## Retrieve the best model and use it on test data 172 | 173 | Now that you have several trained models ranked based on the metric you specified when configuring the AutoML experiment, you can retrieve the best one and either use it to score immediately (on test data for example) or register and deploy is as a service. 174 | 175 | Retrieving the best model and using it to score on test data requires the use of `get_output()`: 176 | 177 | ```python 178 | best_run, fitted_model = run.get_output() 179 | 180 | y_query = y_test.values.copy().astype(np.float) 181 | y_query.fill(np.nan) 182 | 183 | y_pred, X_trans = fitted_model.forecast(X_test, y_query) 184 | ``` 185 | 186 | You can also create a visual representation of the scoring on test data: 187 | 188 | ![Testing of an AutoML forecast trained model](./media/automl-forecasting-testing.png) 189 | 190 | While not perfect, the model has picked up correctly the trend of the data. Also notice how the first few instants are better predicted than subsequent 191 | 192 | You have now successfully configured an AutoML experiment, submitted it to run on a remote compute resource, analyzed the results it produced, retrieved its best trained model, and used this best model to score on test data. 193 | 194 | ## Next steps 195 | 196 | You can learn more about using automated machine learning for Forecasting by reviewing these links to additional resources: 197 | 198 | - [What is automated machine learning?](https://docs.microsoft.com/azure/machine-learning/service/concept-automated-ml) 199 | 200 | Read next: [Understanding automated machine learning generated models, using the model explainability capability of automated machine learning](./automl-understand-models-with-explainability.md) 201 | -------------------------------------------------------------------------------- /modeling/automl-regression-code-sample.md: -------------------------------------------------------------------------------- 1 | # Using automated machine learning for Regression (Code Sample) 2 | 3 | 4 | 5 | ## The dataset used in this example 6 | 7 | In this regression example we will solve the problem of predicting the number of bike rentals using the [UCI Bike Sharing Dataset](https://archive.ics.uci.edu/ml/datasets/Bike+Sharing+Dataset) which is based on real data from the Capital Bikeshare company that maintains a bike rental network in Washington DC in the United States. The dataset represents the number of bike rentals within a specific hour of a day in the years 2011 and year 2012 and contains 17389 rows and 17 columns. The raw feature set contains weather conditions (temperature/humidity/wind speed) and the type of the day (holiday/weekday). The field to predict is the `cnt` count, which represents the bike rentals within a specific hour and which ranges from 1 to 977. 8 | 9 | Note: All code snippets in this section are designed to run in [Azure Notebooks](https://notebooks.azure.com/). 10 | 11 | ## Prepare your environment 12 | 13 | The first step you will perform is installing/updating your Azure Machine Learning service SDK: 14 | 15 | ```python 16 | !pip install --upgrade azureml-sdk[notebooks,explain,automl,contrib] 17 | ``` 18 | 19 | Once you have the latest version of the SDK, you will need to do the necessary imports and create a local folder to store various artifacts: 20 | 21 | ```python 22 | import os 23 | import logging 24 | import numpy as np 25 | import pandas as pd 26 | from sklearn.model_selection import train_test_split 27 | 28 | import azureml.core 29 | from azureml.core import Workspace, Experiment 30 | from azureml.core.authentication import InteractiveLoginAuthentication 31 | from azureml.core.compute import ComputeTarget, AmlCompute 32 | from azureml.core.runconfig import RunConfiguration 33 | from azureml.core.conda_dependencies import CondaDependencies 34 | 35 | import azureml.dataprep as dprep 36 | 37 | from azureml.train.automl import AutoMLConfig 38 | from azureml.train.automl.run import AutoMLRun 39 | 40 | from azureml.widgets import RunDetails 41 | 42 | experiment_name = 'bike-sharing-regression' 43 | project_folder = './bike-sharing-regression' 44 | 45 | # Create a project_folder if it doesn't exist 46 | if not os.path.exists(project_folder): 47 | os.makedirs(project_folder) 48 | ``` 49 | 50 | Next, get a hold on the compute resource you will use to train your model: 51 | 52 | ```python 53 | cpu_cluster = ComputeTarget(workspace=ws, name='aml01') 54 | ``` 55 | Note: The creation of the `Workspace` variable `ws` is ommited for brewity. Also, we are assuming there already exists an Azure Machine Learning compute resource named `aml01` (which means we are going to submit an AutoML experiment run to a remote compute resource, not the local machine). 56 | 57 | ## Prepare input data 58 | 59 | Once everything is in place, you will start loading, analyzing, and preparing your input data: 60 | 61 | ```python 62 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/bike/bike-rental-hour.csv', 63 | header=dprep.api.dataflow.PromoteHeadersMode.UNGROUPED, 64 | infer_column_types=True) 65 | 66 | X = data_flow.keep_columns(['season','mnth','hr','holiday','weekday','workingday','weathersit','temp','atemp','hum','windspeed']).to_pandas_dataframe() 67 | y = data_flow.keep_columns(['cnt']).to_pandas_dataframe() 68 | 69 | X_train, X_test, y_train, y_test = train_test_split( 70 | X, y, test_size=0.2, random_state=111) 71 | 72 | data_flow.get_profile() 73 | ``` 74 | 75 | Notice how we are splitting the original data set into input features (`X`) and output(`y`) - the numerical feature we are looking to predict. Also notice the list of input features we are considering for our regression model. 76 | 77 | ## Create the AutoML configuration 78 | 79 | The following elements are needed to create an AutoML configuration: 80 | - A data script which will be used on the remote compute resource to get the data to train the model. 81 | - A `RunConfiguration` which provides details about the creation and initialization of the Python environment on the remote compute resource. 82 | - A set of AutoML experiment settings including the number of iterations, the maximum time the experiment is allowed to run, the primary metric used to rank resulting models,and the level of logging. 83 | 84 | The data script must be saved localy (as it will be referenced by the AutoML configuration) and it has basically the same code you used above to prepare input data: 85 | 86 | ```python 87 | %%writefile $project_folder/get_data.py 88 | 89 | import azureml.dataprep as dprep 90 | from sklearn.model_selection import train_test_split 91 | 92 | def get_data(): 93 | 94 | data_flow = dprep.read_csv('https://quickstartsws9073123377.blob.core.windows.net/azureml-blobstore-0d1c4218-a5f9-418b-bf55-902b65277b85/bike/bike-rental-hour.csv', 95 | header=dprep.api.dataflow.PromoteHeadersMode.UNGROUPED, 96 | infer_column_types=True) 97 | 98 | X = data_flow.keep_columns(['season','mnth','hr','holiday','weekday','workingday','weathersit','temp','atemp','hum','windspeed']).to_pandas_dataframe() 99 | y = data_flow.keep_columns(['cnt']).to_pandas_dataframe() 100 | 101 | X_train, X_test, y_train, y_test = train_test_split( 102 | X, y, test_size=0.2, random_state=111) 103 | 104 | return { "X" : X_train.values, "y" : y_train.values.flatten() } 105 | ``` 106 | 107 | The run configuration includes details about the remote compute resource, the image used to initialize, and the library dependencies that need to be installed: 108 | 109 | ```python 110 | run_config = RunConfiguration(framework="python") 111 | run_config.target = cpu_cluster 112 | run_config.environment.docker.enabled = True 113 | run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE 114 | 115 | dependencies = CondaDependencies.create( 116 | pip_packages=["scikit-learn", "azureml-sdk[automl]","azureml-dataprep", "azureml-explain-model"]) 117 | run_config.environment.python.conda_dependencies = dependencies 118 | ``` 119 | 120 | Finally, you create the settings object and then initialize the `AutoMLConfig` object: 121 | 122 | ```python 123 | automl_settings = { 124 | "name": experiment_name, 125 | "iteration_timeout_minutes": 10, 126 | "iterations": 10, 127 | "n_cross_validations": 5, 128 | "primary_metric": 'normalized_root_mean_squared_error', 129 | "preprocess": True, 130 | "max_concurrent_iterations": 10, 131 | "verbosity": logging.INFO 132 | } 133 | 134 | automl_config = AutoMLConfig(task='regression', 135 | debug_log='automl_errors.log', 136 | path=project_folder, 137 | compute_target=cpu_cluster, 138 | run_configuration=run_config, 139 | data_script=project_folder + "/get_data.py", 140 | model_explainability=True, 141 | **automl_settings, 142 | ) 143 | ``` 144 | 145 | We have now defined a regression task that will use the `normalized_root_mean_squared_error` measure as primary metric for model ranking (and selection). 146 | 147 | ## Running the AutoML experiment 148 | 149 | Once you have the `AutoMLConfig` object properly initialized, you are ready to submit your experiment. Depending on the various settings you used, the experiment will run for several minutes or more. Using the `show_output=True` options enables you to get updates on the execution. 150 | 151 | ![Execution progress for an AutoML regression experiment run](./media/automl-regression-execution-progress.png) 152 | 153 | Note how the best value of the specified metric (normalized_root_mean_squared_error in our case) is tracked throughout the entire execution. In this particular example, the `VotingEnsemble` pipeline yielded the top trained model, with an error of 0.0984. 154 | 155 | Once the execution is finished you can request detailed information about the results. 156 | 157 | ![Execution results for an AutoML regression experiment run](./media/automl-regression-execution-results.png) 158 | 159 | Also, you can get all the metrics recorded during the experiment run. 160 | 161 | ![Execution metrics for an AutoML regression experiment run](./media/automl-regression-execution-metrics.png) 162 | 163 | Notice the value of the `root_mean_square_error` metric - 96.04. 164 | 165 | ## Retrieve the best model and use it on test data 166 | 167 | Now that you have several trained models ranked based on the metric you specified when configuring the AutoML experiment, you can retrieve the best one and either use it to score immediately (on test data for example) or register and deploy is as a service. 168 | 169 | Retrieving the best model and using it to score on test data requires the use of `get_output()`: 170 | 171 | ```python 172 | best_run, fitted_model = run.get_output() 173 | 174 | y_predict = fitted_model.predict(X_test.values) 175 | print(y_predict) 176 | ``` 177 | You can also create a visual representation of the scoring on test data: 178 | 179 | ![Testing of an AutoML regression trained model](./media/automl-regression-testing.png) 180 | 181 | Finally, you can calculate the value of the `root_mean_square_error` metric for the test data and compare it with the value resulting from training (96.04 - mentioned above): 182 | 183 | ![Testing of an AutoML regression trained model - Root Mean Squared Error](./media/automl-regression-testing-rmse.png) 184 | 185 | You have now successfully configured an AutoML experiment, submitted it to run on a remote compute resource, analyzed the results it produced, retrieved its best trained model, and used this best model to score on test data. 186 | 187 | ## Next steps 188 | 189 | You can learn more about using automated machine learning for Regression by reviewing these links to additional resources: 190 | 191 | - [What is automated machine learning?](https://docs.microsoft.com/azure/machine-learning/service/concept-automated-ml) 192 | 193 | Read next: [Using automated machine learning for Forecasting (Code Sample)](./automl-forecasting-code-sample.md) -------------------------------------------------------------------------------- /modeling/measure-model-performance.md: -------------------------------------------------------------------------------- 1 | # Tools to measure model performance 2 | 3 | The following categories of tools are available for measuring model performance: 4 | 5 | - Programatic tools - the [Azure Machine Learning SDK for Python](https://docs.microsoft.com/python/api/overview/azure/ml/install?view=azure-ml-py) and the [Azure Command-Line Interface](https://docs.microsoft.com/cli/azure/?view=azure-cli-latest) (CLI) (plus the [CLI extension for Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/reference-azure-machine-learning-cli)) 6 | - Automated tools - [Azure Machine Learning service Automated ML](../../simplify-process-with-automated-ml/README.md) (AutoML) 7 | - Third party tools - [MLFlow](https://mlflow.org/) and [TensorBoard](https://www.tensorflow.org/tensorboard/) 8 | 9 | ## Azure Machine Learning SDK for Pyhton and Azure CLI 10 | 11 | The SDK and the CLI can be used for the following tasks related to monitoring, organizing, and managing experiments and runs: 12 | 13 | - Monitor run performance 14 | - Cancel or fail runs 15 | - Create child runs 16 | - Tag and search for runs 17 | 18 | Training code based on the SDK can log the following types of values: 19 | 20 | - Scalar values (string or numerical) - logging the same metric multiple times will result in a vector 21 | - Vector (list) 22 | - Row 23 | - Table 24 | - Image 25 | - File 26 | - Tag 27 | 28 | These values are typically generated during the model training process and they include telemetry related to model performance. For a detailed discussion about using the SDK to measure model performance, read the [Capturing and querying model performance data with AML Experiments](./capture-query-model-performance-with-aml-experiments.md) section. 29 | 30 | ## Azure Machine Learning service AutoML 31 | 32 | AutoML is an Azure Machine Learning feature that enables you to train and tune a Classification, Regresion, or Time Series Forecasting model using a specific target metric. To achieve this, AutoML iterates through several algorithms and parameterizations producing on each iteration a model and a training score. When it comes to measuring model performance, AutoML provides out-of-the box support for a significant number of metrics and graphical representations. 33 | 34 | The [Measuring performance of classification and regression models](./measure-performance-regression-classification.md) section contains a detailed discussion about the performance metrics supported by AutoML. For an exhaustive list of metrics logged by AutomML, read [Evaluate training accuracy in automated ML with metrics](https://docs.microsoft.com/azure/machine-learning/service/how-to-understand-accuracy-metrics). 35 | 36 | ## MLFlow 37 | 38 | [MLFlow](https://mlflow.org/) is a library designed to manage the life cycle of machine learning experiments, originating from the [Databricks ecosystem](https://databricks.com/blog/2018/06/05/introducing-mlflow-an-open-source-machine-learning-platform.html). MLFlow tracking is a component from MLFlow that provides functions for logging and tracking run metrics and model artifacts. 39 | 40 | The following diagram display the conceptual approach of MLFlow: 41 | 42 | ![](media/model-performance-mlflow.png) 43 | 44 | To configure the integration between MLFlow and Azure Machine Learning service, you will use the `get_ml_flow_tracking_uri` method from the `Workspace` object as show in the following example: 45 | 46 | ```python 47 | import mlflow 48 | from azureml.core import Workspace 49 | 50 | ws = Workspace.from_config() 51 | 52 | mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri()) 53 | 54 | experiment_name = 'experiment_with_mlflow' 55 | mlflow.set_experiment(experiment_name) 56 | 57 | with mlflow.start_run(): 58 | mlflow.log_metric('alpha', 0.03) 59 | ``` 60 | 61 | ## TensorBoard 62 | 63 | The [TensorBoard package](https://docs.microsoft.com/python/api/azureml-tensorboard/?view=azure-ml-py) that is part of the main Azure Machine Learning service SDK for Python enables you to view your runs and metrics in [TensorBoard](https://www.tensorflow.org/tensorboard/). TensorBoard is a suite of web applications for inspecting and understanding your TensorFlow runs and graphs. 64 | 65 | There are two options for integration with the TensorBoard environment: 66 | 67 | - For experiments where TensorBoard-compatible log files are generated (e.g. experiments driven by TensorFlow/Keras, PyTorch, or Chainer estimators), TensorBoard can be launched directly from the run history available in Azure Machine Learning. For more details about these estimators read the [Introducing AML Estimators](../model-training/aml-estimators.md) section. 68 | - For other experiments (e.g. driven by Scikit-learn estimators), the `export_to_tensorboard` method can be used to export the run history of an experiment in a TensorBoard-compatible log format. 69 | 70 | ## Next steps 71 | 72 | You can learn more about the tools available to measure model performance by reviewing these links to additional resources: 73 | 74 | - [Log metrics during training runs in Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/service/how-to-track-experiments) 75 | - [Evaluate training accuracy in automated ML with metrics](https://docs.microsoft.com/azure/machine-learning/service/how-to-understand-accuracy-metrics) 76 | - [Track metrics and deploy models with MLflow and Azure Machine Learning service (Preview)](https://docs.microsoft.com/azure/machine-learning/service/how-to-use-mlflow) 77 | - [Visualize experiment runs and metrics with TensorBoard and Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/service/how-to-monitor-tensorboard) 78 | 79 | Read next: [Capturing and querying model performance data with AML Experiments](./capture-query-model-performance-with-aml-experiments.md) 80 | 81 | -------------------------------------------------------------------------------- /modeling/media/_placeholder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/_placeholder -------------------------------------------------------------------------------- /modeling/media/aml-compute-create-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/aml-compute-create-in-portal.png -------------------------------------------------------------------------------- /modeling/media/automl-block-algorithms.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-block-algorithms.png -------------------------------------------------------------------------------- /modeling/media/automl-classification-execution-metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-classification-execution-metrics.png -------------------------------------------------------------------------------- /modeling/media/automl-classification-execution-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-classification-execution-progress.png -------------------------------------------------------------------------------- /modeling/media/automl-classification-execution-results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-classification-execution-results.png -------------------------------------------------------------------------------- /modeling/media/automl-explainability-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-explainability-architecture.png -------------------------------------------------------------------------------- /modeling/media/automl-explainability-numeric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-explainability-numeric.png -------------------------------------------------------------------------------- /modeling/media/automl-explainability-visual-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-explainability-visual-1.png -------------------------------------------------------------------------------- /modeling/media/automl-explainability-visual-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-explainability-visual-2.png -------------------------------------------------------------------------------- /modeling/media/automl-forecasting-actual-evolution.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-forecasting-actual-evolution.png -------------------------------------------------------------------------------- /modeling/media/automl-forecasting-execution-metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-forecasting-execution-metrics.png -------------------------------------------------------------------------------- /modeling/media/automl-forecasting-execution-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-forecasting-execution-progress.png -------------------------------------------------------------------------------- /modeling/media/automl-forecasting-execution-results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-forecasting-execution-results.png -------------------------------------------------------------------------------- /modeling/media/automl-forecasting-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-forecasting-testing.png -------------------------------------------------------------------------------- /modeling/media/automl-how-it-works-simple.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-how-it-works-simple.png -------------------------------------------------------------------------------- /modeling/media/automl-how-it-works.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-how-it-works.png -------------------------------------------------------------------------------- /modeling/media/automl-model-explainability-iris.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-model-explainability-iris.png -------------------------------------------------------------------------------- /modeling/media/automl-regression-execution-metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-regression-execution-metrics.png -------------------------------------------------------------------------------- /modeling/media/automl-regression-execution-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-regression-execution-progress.png -------------------------------------------------------------------------------- /modeling/media/automl-regression-execution-results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-regression-execution-results.png -------------------------------------------------------------------------------- /modeling/media/automl-regression-testing-rmse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-regression-testing-rmse.png -------------------------------------------------------------------------------- /modeling/media/automl-regression-testing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/automl-regression-testing.png -------------------------------------------------------------------------------- /modeling/media/azure-machine-learning-taxonomy-marked.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/azure-machine-learning-taxonomy-marked.png -------------------------------------------------------------------------------- /modeling/media/azure-machine-learning-taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/azure-machine-learning-taxonomy.png -------------------------------------------------------------------------------- /modeling/media/compute-target-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/compute-target-in-portal.png -------------------------------------------------------------------------------- /modeling/media/experiment-and-runs-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/experiment-and-runs-in-portal.png -------------------------------------------------------------------------------- /modeling/media/experiment-in-sdk-runs-with-children.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/experiment-in-sdk-runs-with-children.png -------------------------------------------------------------------------------- /modeling/media/experiment-in-sdk-runs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/experiment-in-sdk-runs.png -------------------------------------------------------------------------------- /modeling/media/experiment-in-sdk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/experiment-in-sdk.png -------------------------------------------------------------------------------- /modeling/media/model-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-in-portal.png -------------------------------------------------------------------------------- /modeling/media/model-in-sdk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-in-sdk.png -------------------------------------------------------------------------------- /modeling/media/model-logging-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-01.png -------------------------------------------------------------------------------- /modeling/media/model-logging-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-02.png -------------------------------------------------------------------------------- /modeling/media/model-logging-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-03.png -------------------------------------------------------------------------------- /modeling/media/model-logging-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-04.png -------------------------------------------------------------------------------- /modeling/media/model-logging-05.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-05.png -------------------------------------------------------------------------------- /modeling/media/model-logging-sdk-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-sdk-01.png -------------------------------------------------------------------------------- /modeling/media/model-logging-sdk-02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-sdk-02.png -------------------------------------------------------------------------------- /modeling/media/model-logging-sdk-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-sdk-03.png -------------------------------------------------------------------------------- /modeling/media/model-logging-sdk-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-sdk-04.png -------------------------------------------------------------------------------- /modeling/media/model-logging-sdk-05.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-logging-sdk-05.png -------------------------------------------------------------------------------- /modeling/media/model-performance-calibration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-calibration.png -------------------------------------------------------------------------------- /modeling/media/model-performance-gain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-gain.png -------------------------------------------------------------------------------- /modeling/media/model-performance-lift.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-lift.png -------------------------------------------------------------------------------- /modeling/media/model-performance-mlflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-mlflow.png -------------------------------------------------------------------------------- /modeling/media/model-performance-precision-recall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-precision-recall.png -------------------------------------------------------------------------------- /modeling/media/model-performance-predicted-vs-true.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-predicted-vs-true.png -------------------------------------------------------------------------------- /modeling/media/model-performance-regression-run-metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-regression-run-metrics.png -------------------------------------------------------------------------------- /modeling/media/model-performance-residual-histogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-residual-histogram.png -------------------------------------------------------------------------------- /modeling/media/model-performance-roc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-performance-roc.png -------------------------------------------------------------------------------- /modeling/media/model-properties-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model-properties-in-portal.png -------------------------------------------------------------------------------- /modeling/media/model_monitoring_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model_monitoring_1.png -------------------------------------------------------------------------------- /modeling/media/model_monitoring_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model_monitoring_2.png -------------------------------------------------------------------------------- /modeling/media/model_perf_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model_perf_1.png -------------------------------------------------------------------------------- /modeling/media/model_perf_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model_perf_2.png -------------------------------------------------------------------------------- /modeling/media/model_perf_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/model_perf_3.png -------------------------------------------------------------------------------- /modeling/media/run-in-portal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/run-in-portal.png -------------------------------------------------------------------------------- /modeling/media/run-in-sdk-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/run-in-sdk-details.png -------------------------------------------------------------------------------- /modeling/media/runs-and-child-runs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/runs-and-child-runs.png -------------------------------------------------------------------------------- /modeling/media/vi_bdtr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_bdtr.png -------------------------------------------------------------------------------- /modeling/media/vi_cmd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_cmd.png -------------------------------------------------------------------------------- /modeling/media/vi_dp_modules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_dp_modules.png -------------------------------------------------------------------------------- /modeling/media/vi_emv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_emv.png -------------------------------------------------------------------------------- /modeling/media/vi_launch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_launch.png -------------------------------------------------------------------------------- /modeling/media/vi_ml_modules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_ml_modules.png -------------------------------------------------------------------------------- /modeling/media/vi_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_overview.png -------------------------------------------------------------------------------- /modeling/media/vi_run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_run.png -------------------------------------------------------------------------------- /modeling/media/vi_sem.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_sem.png -------------------------------------------------------------------------------- /modeling/media/vi_smv1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_smv1.png -------------------------------------------------------------------------------- /modeling/media/vi_smv2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_smv2.png -------------------------------------------------------------------------------- /modeling/media/vi_tc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_tc.png -------------------------------------------------------------------------------- /modeling/media/vi_tm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_tm.png -------------------------------------------------------------------------------- /modeling/media/vi_traineval_modules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/solliancenet/Azure-Machine-Learning-Dev-Guide/bc3ed90fd47d75cd73a22d5cb2e06196b35a33a9/modeling/media/vi_traineval_modules.png -------------------------------------------------------------------------------- /modeling/model-evaluation.md: -------------------------------------------------------------------------------- 1 | # Model Evaluation introduced 2 | 3 | The evaluation of a Machine Learning model is the process through which a set of performance metrics are calculated for an already trained model in order to assess its performance. While these performance metrics play an important role in understanding the performance on the model they were calculated for, they are also critical in the model selection process - a process through which the best model is selected from a number of candidate trained models. 4 | 5 | Azure Machine Learning service provides a comprehensive environment to implement Model Evaluation, giving you a centralized place to store and access all model performance data. The collection of performance data can be either performed manually (through the training script itself using the [Azure Machine Learning SDK for Python](https://docs.microsoft.com/python/api/overview/azure/ml/install?view=azure-ml-py)) or automatically in the the case of automated training. There are multiple types of data that can be used to store performance data. Examples include numeric and string scalars, lists, tables, images, and custom files. 6 | 7 | Model performance data can be programmatically accessed through the SDK and the Azure Portal can be used to view the raw version of it as well as various graphical representations (like confusion matrices, gain and lift charts, receiver operating characteristics curves, calibration plots, predicted vs. true charts, and histograms of residuals, to name just a few). In addition to the Azure Portal, third party tools like [MLFlow](https://mlflow.org/) and [TensorBoard](https://www.tensorflow.org/tensorboard/) can be integrated. 8 | 9 | In this section we will focus on the following: 10 | 11 | - [Measuring performance of classification and regression models](./measure-performance-regression-classification.md) 12 | - [Tools to measure model performance](./measure-model-performance.md) 13 | - [Capturing and querying model performance data with AML Experiments](./capture-query-model-performance-with-aml-experiments.md) 14 | 15 | ## Next steps 16 | 17 | You can learn more about Model Evaluation by reviewing these links to additional resources: 18 | 19 | - [Start, monitor, and cancel training runs in Python](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-runs) 20 | - [Log metrics during training runs in Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/service/how-to-track-experiments) 21 | - [Track metrics and deploy models with MLflow and Azure Machine Learning service (Preview)](https://docs.microsoft.com/azure/machine-learning/service/how-to-use-mlflow) 22 | - [Visualize experiment runs and metrics with TensorBoard and Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/service/how-to-monitor-tensorboard) 23 | 24 | ### Related topics 25 | 26 | - [Feature Engineering introduced](./feature-engineering-training-evaluation-selection.md#feature-engineering-introduced) 27 | - [Model Training introduced](./model-training.md) 28 | 29 | Read next: [Measuring performance of classification and models](./measure-performance-regression-classification.md) 30 | -------------------------------------------------------------------------------- /modeling/model-training.md: -------------------------------------------------------------------------------- 1 | # Model Training introduced 2 | 3 | The training of a Machine Learning model is the process through which a mathematical model is built from data that contains both inputs and expected outcomes (or only inputs in the case of unsupervised learning). There are several classes of algorithms available to build the model, like classification, regression, clustering, feature learning, and others. 4 | 5 | Azure Machine Learning service provides a comprehensive environment to implement data science processes, giving you a centralized place to work with all the artifacts involved in the process. The top level resource for an instance of the Azure Machine Learning service is the workspace. The following diagram illustrates the high level taxonomy of the workspace: 6 | 7 | ![The high level taxonomy of the Azure Machine Learning workspace](./media/azure-machine-learning-taxonomy.png) 8 | 9 | The following table provides a brief description of all the elements from this taxonomy that are involved in the Model Training process. 10 | 11 | | Name | Description | 12 | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 13 | | Experiment | A generic context for handling runs. Think about it a logical entity you can use to organize your Model Training processes. | 14 | | Run | A Model Training run used to build a trained model. Contains all artifacts associated with a training process, like output files, metrics, logs, and a snapshot of the directory that contains your scripts. | 15 | | Estimator | An Estimator is an alternative higher-level abstraction used to construct run configurations when training deep learning models. | 16 | | Compute target | Defines a compute resource used to either run a Model Training script or host a service deployment (associated with a trained Machine Learning model). | 17 | | Model registry | Keeps track of all models in an Azure Machine Learning workspace. Models are either produced by a Run or originating from outside Azure Machine Learning (and made available via model registration). | 18 | 19 | In this section we will focus on the following: 20 | 21 | - [Introducing the AML Experiment and Runs](./aml-experiment-runs.md) 22 | - [Introducing AML Estimators](./aml-estimators.md) 23 | - [Introducing AML compute options (local and Azure Machine Learning compute)](./aml-compute-options.md) 24 | - [Introducing the AML Model Registry](./aml-model-registry.md) 25 | 26 | The following diagram identifies the positions of these elements in the high level taxonomy of the Azure Machine Learning workspace: 27 | 28 | ![The positions of the elements involved in Model Training in the taxonomy of the Azure Machine Learning workspace](./media/azure-machine-learning-taxonomy-marked.png) 29 | 30 | ## Next steps 31 | 32 | You can learn more about Model Training by reviewing these links to additional resources: 33 | 34 | - [How to choose algorithms for Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/studio/algorithm-choice) 35 | - [How Azure Machine Learning service works: Architecture and concepts](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) 36 | - [What is an Azure Machine Learning service workspace?](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace) 37 | 38 | ### Related topics 39 | 40 | - [Feature Engineering introduced](./feature-engineering-training-evaluation-selection.md#feature-engineering-introduced) 41 | - [Model Evaluation introduced](./model-evaluation.md) 42 | 43 | Read next: [Introducing the AML Experiment and Runs](./aml-experiment-runs.md) 44 | -------------------------------------------------------------------------------- /template.md: -------------------------------------------------------------------------------- 1 | # Title 2 | 3 | Description... 4 | 5 | ## Section 6 | 7 | Text 8 | 9 | ### Sub-section 10 | 11 | Text 12 | 13 | ## Next steps 14 | 15 | - [Reference link]() 16 | - [Reference link]() 17 | 18 | Read next: [Related article]() 19 | --------------------------------------------------------------------------------