├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── NOTICE ├── README.md ├── cloud_formation ├── deploy.sh ├── endpoint-stack.yaml ├── network-stack.yaml ├── nlp-workshop-stack.yaml ├── sagemaker-stack.yaml └── storage-stack.yaml ├── endpoint └── deploy_gpt2_gptj_mme.ipynb ├── main_notebook └── Generate_Text_with_GPT2.ipynb ├── pipeline ├── hf-pipeline.ipynb └── iam_helper.py ├── use_cases └── Pick_the_right_HF_model.ipynb └── without_cfn ├── 1_deploy_gpt2_gptj_mme.ipynb └── 2_generate_text_with_gpt2.ipynb /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Hugging Face on Amazon SageMaker and AWS Workshop 2 | 3 | You’ve just been hired by the Chicago Tribune to start a new poetry column. Congrats! The catch? You need to write a new poem every day. And it can’t just be any old string of syllables, you need it to be fresh, authentic, to resonate with the times and carry a sense of rhyme. You need it to delight your readers, to drive up the Tribune’s daily readership numbers and grow their social media presence. How are you going to accomplish this? With the help of Hugging Face and NLP models on SageMaker of course! 4 | 5 | In this 90-minute hands-on workshop you will use prebuilt notebooks to fine-tune text generation models in the style of your favorite poets, on Amazon SageMaker. You will customize your notebooks to generate new lines of poetry that seem promising, and publish your trained model artifact onto an S3 bucket. You will use *SageMaker training to fine-tune your models* on larger sets of data, speeding up your train time with the SageMaker Training Compiler 6 | 7 | You will build a *multi-model endpoint on SageMaker to deploy models* with different styles of text generation, fine-tuned for different authors, and combine all of them to write different styles of poetry. Lastly, you’ll wrap your work into a *SageMaker Pipelines workflow* to push new changes to your models as new topics become more relevant. 8 | 9 | Additionally, we'll provide links to explain *model performance with SageMaker Clarify*, and *tune your hyper-parameters with Automatic Model Tuning.* You’ll see how to optimize your models for performance with the SageMaker *Inference Recommender*, ultimately compiling your model for enhanced inference times with SageMaker Neo. 10 | 11 | ## Repository Structure 12 | 13 | - `main_notebook` - The data science starting point. This is where most users will begin. You'll use Hugging Face to test, fine-tune, and deploy NLP models. We'll use SageMaker Training Compiler to speed up our model, running on one ml.p3.2xlarge instance equipped with one NVIDIA V100. This training job should take ~15 minutes under current settings, with just a few epochs. 14 | - `pipeline` - This is the MLOps starting point. If you want to learn how to build pipelines using SageMaker, this is a great notebook to step through. 15 | - `cloud_formation` - This is the AWS admin starting point. If you want to learn how to deploy this entire workshop into new AWS accounts using yaml and Cloud Formation, this is where to go. 16 | - `use_cases` - This is where to start if you are looking to find the right Hugging Face model and use case. We also identify 40+ notebooks built by both Hugging Face and AWS that demonstrate various aspects of Hugging Face and SageMaker together. 17 | - `endpoint` - This is where to go if you want to focus on the endpoints. Here we are deploying two endpoints. One is a multi-model endpoint to host a variety of GPT-2 fine-tuned poetry generator models. The other is a GPT-J 6B model hosted on a single endpoint. Both of these can be accomplished without GPU's, however we note that performance tends to increase with hardware upgrades, and as such typically recommend GPU for NLP model hosting as much as possible. 18 | 19 | ## Get started 20 | If you are in an AWS-hosted workshop, please follow the steps provided by your workshop team. If you are stepping through this by yourself, please start with the `without_cfn` directory. First, execute the `1_deploy_gpt2_gptj_mme.ipynb` to create the multi-model endpoint, and optionally a GPT-J endpoint. Second, execute the `2_Generate_Text_with_GPT2.ipynb`. This uses the MME you just created to host multiple Hugging Face Models. Currently it does not build the SM Pipeline. 21 | 22 | If you are using the AWS event engine, then please [click here to download the entire infrastructure via CloudFormation!](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://nlp-workshop-models.s3.us-east-1.amazonaws.com/templates/nlp-workshop-stack.yaml&stackName=PoetryOfNLP) Please note that this will deploy 2 Hugging Face endpoints, 1 SageMaker Pipeline, and 1 SageMaker Studio domain. This uses the pipeline and a parameter store to deploy onto the MME. You'll use the `main_notebook/Generate_Text_with_GPT2.ipynb` in this case. 23 | 24 | 25 | ## Use a Text Classifier for "Adversarial" Approaches 26 | While the focus of this workshop is on text generation, we also wanted to highlight another key aspect of NLP, text classification. Towards that end we've trained and have deployed onto the HF Hub our very own classifier, `anne_bradstreet`. Specifically this model is a binary classifier trained to know the difference between genuine text written by Anne Bradsteet, the first published writer out of North America and the subject of our workshop, vs text generated by a generic GPT-2 model conditioned on prompts written by Bradsteet. You can [see our model hosted on the HF Hub here](https://huggingface.co/edubz/anne_bradstreet). 27 | 28 | ## What about other NLP use cases? 29 | If you're not that interested in text generation, and maybe would instead to prefer focusing on other NLP use cases like named entity recognition, question answering, text classification, translation, or anything else, please go ahead and [look at our notebook on the topic here](https://github.com/aws-samples/hugging-face-workshop/blob/main/use_cases/Pick_the_right_HF_model.ipynb). 30 | 31 | ## Where to go from here? 32 | If you've made it to the end of the workshop and are wondering what's next, consider looking at [the new O'Reilly book *Natural Language Processing with Transformers* by the Hugging Face team.](https://www.amazon.com/Natural-Language-Processing-Transformers-Applications/dp/1098103246/ref=sr_1_3?crid=1U6AVYBVZNY07&keywords=natural+language+processing+with+transformers&qid=1642008897&sprefix=natural+language+processing+%2Caps%2C148&sr=8-3) This does a great job recapping many of the topics you've learned here, exploring them in even more detail, and applying on an even greater variety of scenarios. 33 | 34 | Enjoy! 35 | 36 | ## Security 37 | 38 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 39 | 40 | ## License 41 | 42 | This project is licensed under the Apache-2.0 License. 43 | 44 | -------------------------------------------------------------------------------- /cloud_formation/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Exepcts environment variables to be set as follows 4 | # 5 | # AWS_REGION: target region for deployment. 6 | # INFRA_BUCKET: s3 bucket in target region. deploy script will stage required static assets to thie bucket 7 | 8 | if [ -z $INFRA_BUCKET ] || [ -z $AWS_REGION ]; then 9 | echo "Error: Environment Variables INFRA_BUCKET and AWS_REGION are required" 10 | echo "Please set environment variables INFRA_BUCKET and AWS_REGION and rerun" 11 | exit 1 12 | fi 13 | echo "Using S3 Bucket: $INFRA_BUCKET" 14 | 15 | echo "Copying templates to s3 bucket" 16 | aws s3 cp ./endpoint-stack.yaml s3://$INFRA_BUCKET/templates/ 17 | aws s3 cp ./network-stack.yaml s3://$INFRA_BUCKET/templates/ 18 | aws s3 cp ./nlp-workshop-stack.yaml s3://$INFRA_BUCKET/templates/ 19 | aws s3 cp ./sagemaker-stack.yaml s3://$INFRA_BUCKET/templates/ 20 | aws s3 cp ./storage-stack.yaml s3://$INFRA_BUCKET/templates/ 21 | aws s3 cp ./gpt-j.tar.gz s3://$INFRA_BUCKET/gpt-j/model.tar.gz 22 | aws s3 cp ./gpt-2.tar.gz s3://$INFRA_BUCKET/gpt-2/model.tar.gz 23 | 24 | echo "Staging notebooks to s3 bucket" 25 | ( 26 | cd ../notebooks 27 | zip -r ../notebooks.zip . 28 | aws s3 cp ../notebooks.zip s3://$INFRA_BUCKET 29 | rm ../notebooks.zip 30 | ) 31 | 32 | export STACK_NAME=NLP-Lab 33 | echo "Deploying cloudformation stack" 34 | aws cloudformation create-stack \ 35 | --stack-name $STACK_NAME \ 36 | --template-url https://${INFRA_BUCKET}.s3.${AWS_REGION}.amazonaws.com/templates/nlp-workshop-stack.yaml \ 37 | --parameters ParameterKey=InfraResourcesBucket,ParameterValue=$INFRA_BUCKET \ 38 | --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM 39 | 40 | -------------------------------------------------------------------------------- /cloud_formation/endpoint-stack.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: Poetry of NLP Workshop Endpoint Stack 3 | Parameters: 4 | InfraResourcesBucket: 5 | Type: String 6 | Description: Amazon S3 bucket containing environment creation resources 7 | SageMakerStackName: 8 | Type: String 9 | Description: The name of the Stack with the SageMaker domain 10 | StorageStackName: 11 | Type: String 12 | Description: The name of the Storage Stack 13 | Resources: 14 | GPT2Model: 15 | Type: "AWS::SageMaker::Model" 16 | Properties: 17 | ModelName: GPT2 18 | ExecutionRoleArn: 19 | Fn::ImportValue: !Sub ${SageMakerStackName}-SageMaker-Execution-Role-Arn 20 | PrimaryContainer: 21 | Image: !Sub 763104351884.dkr.ecr.${AWS::Region}.amazonaws.com/huggingface-pytorch-inference:1.7.1-transformers4.6.1-cpu-py36-ubuntu18.04 22 | Mode: MultiModel 23 | ModelDataUrl: !Sub s3://${InfraResourcesBucket}/gpt-2/model.tar.gz 24 | MultiModelConfig: 25 | ModelCacheSetting: Enabled 26 | EndpointConfig: 27 | Type: AWS::SageMaker::EndpointConfig 28 | Properties: 29 | EndpointConfigName: NLPPoetryLabMME 30 | ProductionVariants: 31 | - 32 | InitialInstanceCount: 1 33 | InitialVariantWeight: 1 34 | InstanceType: ml.m5.xlarge 35 | ModelName: !GetAtt GPT2Model.ModelName 36 | VariantName: BaseModel 37 | MultiModelEndpoint: 38 | Type: AWS::SageMaker::Endpoint 39 | Properties: 40 | EndpointConfigName: !GetAtt EndpointConfig.EndpointConfigName 41 | ParameterStoreEndpointName: 42 | Type: AWS::SSM::Parameter 43 | Properties: 44 | Name: hf-endpoint 45 | Type: String 46 | Value: !GetAtt MultiModelEndpoint.EndpointName 47 | GPTJModel: 48 | Type: AWS::SageMaker::Model 49 | Properties: 50 | ModelName: GPTJ 51 | ExecutionRoleArn: 52 | Fn::ImportValue: !Sub ${SageMakerStackName}-SageMaker-Execution-Role-Arn 53 | PrimaryContainer: 54 | Image: !Sub 763104351884.dkr.ecr.${AWS::Region}.amazonaws.com/huggingface-pytorch-inference:1.7.1-transformers4.6.1-cpu-py36-ubuntu18.04 55 | Mode: SingleModel 56 | ModelDataUrl: !Sub s3://${InfraResourcesBucket}/gpt-j/model.tar.gz 57 | GPTJEndpointConfig: 58 | Type: AWS::SageMaker::EndpointConfig 59 | Properties: 60 | EndpointConfigName: GPTJ 61 | ProductionVariants: 62 | - InitialInstanceCount: 1 63 | InitialVariantWeight: 1 64 | InstanceType: ml.m5.2xlarge 65 | ModelName: !GetAtt GPTJModel.ModelName 66 | VariantName: BaseModel 67 | GPTJModelEndpoint: 68 | Type: AWS::SageMaker::Endpoint 69 | Properties: 70 | EndpointConfigName: !GetAtt GPTJEndpointConfig.EndpointConfigName 71 | ParameterStoreGPTJEndpointName: 72 | Type: AWS::SSM::Parameter 73 | Properties: 74 | Name: gptj-endpoint 75 | Type: String 76 | Value: !GetAtt GPTJModelEndpoint.EndpointName 77 | PipelineFunctionsRole: 78 | Type: AWS::IAM::Role 79 | Properties: 80 | AssumeRolePolicyDocument: 81 | Version: 2012-10-17 82 | Statement: 83 | - Effect: Allow 84 | Principal: 85 | Service: 86 | - lambda.amazonaws.com 87 | Action: 88 | - "sts:AssumeRole" 89 | Path: / 90 | ManagedPolicyArns: 91 | - arn:aws:iam::aws:policy/AmazonSageMakerFullAccess 92 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 93 | S3InvokeFunctionPermission: 94 | Type: "AWS::Lambda::Permission" 95 | Properties: 96 | Action: "lambda:InvokeFunction" 97 | FunctionName: !GetAtt StartPipelineFunction.Arn 98 | Principal: "s3.amazonaws.com" 99 | SourceAccount: !Ref AWS::AccountId 100 | SourceArn: 101 | Fn::ImportValue: !Sub ${StorageStackName}-Data-Pipeline-Bucket-Arn 102 | ReadParameterStoreFunction: 103 | Type: AWS::Lambda::Function 104 | Properties: 105 | Description: Reads parameter store and returns values for use in SageMaker Pipeline 106 | Runtime: python3.8 107 | Handler: handler.handler 108 | Role: !GetAtt PipelineFunctionsRole.Arn 109 | Code: 110 | ZipFile: | 111 | import json 112 | import boto3 113 | 114 | PARAM_ENDPOINT = 'hf-endpoint' 115 | PARAM_TEST = 'hf-test-data' 116 | PARAM_ARTIFACT = 'hf-s3-location' 117 | 118 | 119 | def lambda_handler(event, context): 120 | ssm_client = boto3.client("ssm") 121 | 122 | response = ssm_client.get_parameter( 123 | Name=PARAM_ENDPOINT 124 | ) 125 | endpoint = response['Parameter']['Value'] 126 | 127 | response = ssm_client.get_parameter( 128 | Name=PARAM_TEST 129 | ) 130 | test_s3 = response['Parameter']['Value'] 131 | 132 | response = ssm_client.get_parameter( 133 | Name=PARAM_ARTIFACT 134 | ) 135 | artifact_s3 = response['Parameter']['Value'] 136 | 137 | return { 138 | "statusCode": 200, 139 | "endpoint": json.dumps(endpoint), 140 | "test_s3": json.dumps(test_s3), 141 | "artifact_s3": json.dumps(artifact_s3) 142 | } 143 | CopyModelArtifactFunction: 144 | Type: AWS::Lambda::Function 145 | Properties: 146 | Description: Reads parameter store and returns values for use in SageMaker Pipeline 147 | Runtime: python3.8 148 | Handler: handler.handler 149 | Role: !GetAtt PipelineFunctionsRole.Arn 150 | Code: 151 | ZipFile: | 152 | import json 153 | import boto3 154 | from urllib.parse import urlparse 155 | 156 | def lambda_handler(event, context): 157 | 158 | s3_client = boto3.client("s3") 159 | 160 | input_artifact = event["model_path"] 161 | artifact_path = event["artifact_path"].lstrip('"').rstrip('"') 162 | 163 | print("got input artifact {0} and artifact path {1}".format(input_artifact, artifact_path)) 164 | 165 | input_p = urlparse(input_artifact) 166 | artifact_p = urlparse(artifact_path) 167 | 168 | input_key = input_p.path.lstrip('/') 169 | input_bucket = input_p.netloc 170 | input_name = input_p.path.split('/')[-1] 171 | 172 | print("Input key = {0}, input bucket = {1}, input name = {2}".format(input_key, input_bucket, input_name)) 173 | 174 | artifact_key = artifact_p.path.lstrip('/') + input_name 175 | artifact_bucket = artifact_p.netloc 176 | 177 | print("Artifact key = {0}, artifact bucket = {1}".format(artifact_key, artifact_bucket)) 178 | 179 | copy_source = { 180 | 'Bucket': input_bucket, 181 | 'Key': input_key 182 | } 183 | s3_client.copy(copy_source, artifact_bucket, artifact_key) 184 | print("Copying {0} to {1}/{2}".format(json.dumps(copy_source), artifact_bucket, artifact_key)) 185 | 186 | return { 187 | "statusCode": 200, 188 | "model_key": input_name 189 | } 190 | InvokeModelFromMMEFunction: 191 | Type: AWS::Lambda::Function 192 | Properties: 193 | Description: Reads parameter store and returns values for use in SageMaker Pipeline 194 | Runtime: python3.8 195 | Handler: handler.handler 196 | Role: !GetAtt PipelineFunctionsRole.Arn 197 | Code: 198 | ZipFile: | 199 | import json 200 | import boto3 201 | from urllib.parse import urlparse 202 | 203 | TMP_FILE = '/tmp/body.txt' 204 | 205 | def lambda_handler(event, context): 206 | 207 | sm_client = boto3.client("sagemaker-runtime") 208 | s3_client = boto3.client("s3") 209 | 210 | endpoint = event["endpoint"].lstrip('"').rstrip('"') 211 | model = event["model"] 212 | test = event["test"].lstrip('"').rstrip('"') 213 | 214 | print("endpoint = {0}, model = {1}, test = {2}".format(endpoint, model, test)) 215 | 216 | test_p = urlparse(test) 217 | 218 | s3_client.download_file(test_p.netloc, test_p.path.lstrip('/'), TMP_FILE) 219 | 220 | with open(TMP_FILE, 'r') as F: 221 | input_data = {'input': F.read() } 222 | print(f"{input_data}") 223 | response = sm_client.invoke_endpoint( 224 | EndpointName = endpoint, 225 | ContentType = "application/json", 226 | TargetModel = model, 227 | Body = json.dumps(input_data).encode('utf-8')) 228 | output = response['Body'].read().decode("utf-8") 229 | 230 | return { 231 | "statusCode": 200, 232 | "inference": output 233 | } 234 | StartPipelineFunction: 235 | Type: "AWS::Lambda::Function" 236 | Properties: 237 | MemorySize: 1024 238 | Runtime: "python3.8" 239 | Timeout: 300 240 | Role: !GetAtt PipelineFunctionsRole.Arn 241 | Handler: "index.handler" 242 | Code: 243 | ZipFile: | 244 | import json 245 | import boto3 246 | import os 247 | import traceback 248 | 249 | client = boto3.client('sagemaker') 250 | pname = os.environ['PipelineName'] 251 | 252 | def lambda_handler(event, context): 253 | try: 254 | files = [] 255 | for record in event['Records']: 256 | bucket = record['s3']['bucket']['name'] 257 | key = record['s3']['object']['key'] 258 | files.append(key) 259 | print("Detected create event: { 0 }/{1}".format(bucket, key)) 260 | 261 | response = client.start_pipeline_execution( 262 | PipelineName=pname, 263 | PipelineParameters=[ 264 | { 265 | 'Name': 'ModelLocation', 266 | 'Value': f"s3://{bucket}/{key}" 267 | } 268 | ], 269 | PipelineExecutionDescription=f"Execution for {','.join(files)}", 270 | ) 271 | except Exception as e: 272 | trc = traceback.format_exc() 273 | print("Error launching pipeline {0}: { 1 } - { 2 }".format(pname, str(e), trc)) 274 | Environment: 275 | Variables: 276 | PipelineName: “huggingfacedeploypipeline” 277 | PoeticPipeline: 278 | Type: AWS::SageMaker::Pipeline 279 | Properties: 280 | PipelineDefinition: 281 | PipelineDefinitionBody: !Sub | 282 | { 283 | "Version":"2020-12-01", 284 | "Metadata":{}, 285 | "Parameters":[ 286 | { 287 | "Name":"ModelLocation", 288 | "Type":"String", 289 | "DefaultValue":"s3://" 290 | }], 291 | "PipelineExperimentConfig":{ 292 | "ExperimentName":{ 293 | "Get":"Execution.PipelineName" 294 | }, 295 | "TrialName":{ 296 | "Get":"Execution.PipelineExecutionId"} 297 | }, 298 | "Steps":[ 299 | { 300 | "Name":"ReadParamsStep", 301 | "Type":"Lambda", 302 | "Arguments":{}, 303 | "FunctionArn":"${ReadParameterStoreFunction.Arn}", 304 | "OutputParameters":[ 305 | { 306 | "OutputName":"endpoint", 307 | "OutputType":"String" 308 | }, 309 | { 310 | "OutputName":"test_s3", 311 | "OutputType":"String" 312 | }, 313 | { 314 | "OutputName":"artifact_s3", 315 | "OutputType":"String" 316 | } 317 | ] 318 | }, 319 | { 320 | "Name":"CopyArtifactStep", 321 | "Type":"Lambda", 322 | "Arguments":{ 323 | "model_path":{ 324 | "Get":"Parameters.ModelLocation" 325 | }, 326 | "artifact_path":{ 327 | "Get":"Steps.ReadParamsStep.OutputParameters['artifact_s3']" 328 | } 329 | }, 330 | "FunctionArn":"${CopyModelArtifactFunction.Arn}", 331 | "OutputParameters":[ 332 | { 333 | "OutputName":"model_key", 334 | "OutputType":"String" 335 | } 336 | ] 337 | }, 338 | { 339 | "Name":"TestStep", 340 | "Type":"Lambda", 341 | "Arguments":{ 342 | "endpoint":{ 343 | "Get":"Steps.ReadParamsStep.OutputParameters['endpoint']" 344 | }, 345 | "model":{ 346 | "Get":"Steps.CopyArtifactStep.OutputParameters['model_key']" 347 | }, 348 | "test":{ 349 | "Get":"Steps.ReadParamsStep.OutputParameters['test_s3']"} 350 | }, 351 | "FunctionArn":"${InvokeModelFromMMEFunction.Arn}", 352 | "OutputParameters":[ 353 | { 354 | "OutputName":"inference", 355 | "OutputType":"String" 356 | } 357 | ] 358 | } 359 | ] 360 | } 361 | PipelineDescription: Pipeline for poets 362 | PipelineDisplayName: PoetryOfNLP 363 | PipelineName: PoetryOfNLP 364 | RoleArn: !GetAtt PipelineFunctionsRole.Arn 365 | 366 | 367 | -------------------------------------------------------------------------------- /cloud_formation/network-stack.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: Poetry of NLP Workshop SageMaker Stack 3 | Resources: 4 | VPC: 5 | Type: AWS::EC2::VPC 6 | Properties: 7 | CidrBlock: 10.0.0.0/16 8 | PublicSubnet: 9 | Type: AWS::EC2::Subnet 10 | Properties: 11 | CidrBlock: 10.0.0.0/20 12 | VpcId: !Ref VPC 13 | PrivateSubnet: 14 | Type: AWS::EC2::Subnet 15 | Properties: 16 | CidrBlock: 10.0.32.0/20 17 | VpcId: !Ref VPC 18 | Outputs: 19 | NetworkStackName: 20 | Description: AWS CloudFormation Stack Name for Network Stack 21 | Value: !Ref AWS::StackName 22 | Export: 23 | Name: NetworkStackName 24 | VpcId: 25 | Description: VPC for Amazon SageMaker Domain 26 | Value: !Ref VPC 27 | Export: 28 | Name: !Sub ${AWS::StackName}-Vpc-Id 29 | PrivateSubnetId: 30 | Description: Private Subnet ID 31 | Value: !Ref PrivateSubnet 32 | Export: 33 | Name: !Sub ${AWS::StackName}-Private-Subnet-Id 34 | PublicSubnetId: 35 | Description: Public Subnet ID 36 | Value: !Ref PublicSubnet 37 | Export: 38 | Name: !Sub ${AWS::StackName}-Public-Subnet-Id 39 | -------------------------------------------------------------------------------- /cloud_formation/nlp-workshop-stack.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: Poetry of NLP Workshop Root Stack 3 | Parameters: 4 | InfraResourcesBucket: 5 | Type: String 6 | Description: Enter the S3 bucket where workshop cloudformation resources have been uploaded 7 | Resources: 8 | NetworkStack: 9 | Type: AWS::CloudFormation::Stack 10 | Properties: 11 | TemplateURL: !Sub https://${InfraResourcesBucket}.s3.${AWS::Region}.amazonaws.com/templates/network-stack.yaml 12 | StorageStack: 13 | Type: AWS::CloudFormation::Stack 14 | Properties: 15 | TemplateURL: !Sub https://${InfraResourcesBucket}.s3.${AWS::Region}.amazonaws.com/templates/storage-stack.yaml 16 | SageMakerStack: 17 | Type: AWS::CloudFormation::Stack 18 | Properties: 19 | TemplateURL: !Sub https://${InfraResourcesBucket}.s3.${AWS::Region}.amazonaws.com/templates/sagemaker-stack.yaml 20 | Parameters: 21 | NetworkStackName: !GetAtt NetworkStack.Outputs.NetworkStackName 22 | InfraResourcesBucket: !Ref InfraResourcesBucket 23 | EndpointStack: 24 | Type: AWS::CloudFormation::Stack 25 | DependsOn: SageMakerStack 26 | Properties: 27 | TemplateURL: !Sub https://${InfraResourcesBucket}.s3.${AWS::Region}.amazonaws.com/templates/endpoint-stack.yaml 28 | Parameters: 29 | InfraResourcesBucket: !Ref InfraResourcesBucket 30 | SageMakerStackName: !GetAtt SageMakerStack.Outputs.SageMakerStackName 31 | StorageStackName: !GetAtt StorageStack.Outputs.StorageStackName 32 | -------------------------------------------------------------------------------- /cloud_formation/sagemaker-stack.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: Poetry of NLP Workshop SageMaker Stack 3 | Parameters: 4 | NetworkStackName: 5 | Type: String 6 | Description: AWS CloudFormation Stack Name for Network Stack 7 | InfraResourcesBucket: 8 | Type: String 9 | Description: Amazon S3 bucket containing environment creation resources 10 | Resources: 11 | SageMakerExecutionRole: 12 | Type: AWS::IAM::Role 13 | Properties: 14 | AssumeRolePolicyDocument: 15 | Version: 2012-10-17 16 | Statement: 17 | - Effect: Allow 18 | Principal: 19 | Service: 20 | - sagemaker.amazonaws.com 21 | Action: 22 | - "sts:AssumeRole" 23 | Path: / 24 | ManagedPolicyArns: 25 | - arn:aws:iam::aws:policy/AmazonSageMakerFullAccess 26 | - arn:aws:iam::aws:policy/AWSCodeCommitPowerUser 27 | - arn:aws:iam::aws:policy/AmazonS3FullAccess 28 | SageMakerDomain: 29 | Type: AWS::SageMaker::Domain 30 | Properties: 31 | DomainName: NLPWorkshop 32 | AuthMode: IAM 33 | DefaultUserSettings: 34 | ExecutionRole: !GetAtt SageMakerExecutionRole.Arn 35 | SubnetIds: 36 | - Fn::ImportValue: !Sub ${NetworkStackName}-Private-Subnet-Id 37 | VpcId: 38 | Fn::ImportValue: !Sub ${NetworkStackName}-Vpc-Id 39 | SageMakerUser: 40 | Type: AWS::SageMaker::UserProfile 41 | Properties: 42 | DomainId: !Ref SageMakerDomain 43 | UserProfileName: LabUser 44 | JupyterApp: 45 | Type: AWS::SageMaker::App 46 | DependsOn: SageMakerUser 47 | Properties: 48 | AppName: default 49 | AppType: JupyterServer 50 | DomainId: !GetAtt SageMakerDomain.DomainId 51 | UserProfileName: LabUser 52 | SourceRepo: 53 | Type: AWS::CodeCommit::Repository 54 | Properties: 55 | Code: 56 | BranchName: main 57 | S3: 58 | Bucket: !Ref InfraResourcesBucket 59 | Key: notebooks.zip 60 | RepositoryDescription: Poetry of NLP Workshop resources 61 | RepositoryName: poetry-nlp 62 | SageMakerCodeRepository: 63 | Type: AWS::SageMaker::CodeRepository 64 | Properties: 65 | CodeRepositoryName: poetry-nlp 66 | GitConfig: 67 | Branch: main 68 | RepositoryUrl: !GetAtt SourceRepo.CloneUrlHttp 69 | ParameterModelStoreLocation: 70 | Type: AWS::SSM::Parameter 71 | Properties: 72 | Name: hf-s3-location 73 | Type: String 74 | Value: !Sub s3://sagemaker-${AWS::Region}-${AWS::AccountId}/multi-model 75 | Outputs: 76 | SageMakerStackName: 77 | Description: AWS CloudFormation Stack Name for SageMaker Stack 78 | Value: !Ref AWS::StackName 79 | Export: 80 | Name: SageMakerStackName 81 | SageMakerExecutionRoleArn: 82 | Description: ARN for the SageMaker Execution Role 83 | Value: !GetAtt SageMakerExecutionRole.Arn 84 | Export: 85 | Name: !Sub ${AWS::StackName}-SageMaker-Execution-Role-Arn 86 | -------------------------------------------------------------------------------- /cloud_formation/storage-stack.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: Poetry of NLP Workshop Storage Stack 3 | Resources: 4 | ModelStorageBucket: 5 | Type: AWS::S3::Bucket 6 | Properties: 7 | BucketName: !Sub "sagemaker-models-${AWS::AccountId}" 8 | BucketEncryption: 9 | ServerSideEncryptionConfiguration: 10 | - ServerSideEncryptionByDefault: 11 | SSEAlgorithm: AES256 12 | DataPipelineBucket: 13 | Type: AWS::S3::Bucket 14 | Properties: 15 | BucketName: !Sub "data-${AWS::AccountId}" 16 | BucketEncryption: 17 | ServerSideEncryptionConfiguration: 18 | - ServerSideEncryptionByDefault: 19 | SSEAlgorithm: AES256 20 | ParameterStoreTestDataLocation: 21 | Type: AWS::SSM::Parameter 22 | Properties: 23 | Name: hf-test-data 24 | Type: String 25 | Value: !Sub s3://${DataPipelineBucket}/test/poems.txt 26 | Outputs: 27 | StorageStackName: 28 | Description: AWS CloudFormation Stack Name for Storage Stack 29 | Value: !Ref AWS::StackName 30 | Export: 31 | Name: StorageStackName 32 | DataPipelineBucketArn: 33 | Description: AWS CloudFormation Stack Arn for Storage Stack 34 | Value: !GetAtt DataPipelineBucket.Arn 35 | Export: 36 | Name: !Sub ${AWS::StackName}-Data-Pipeline-Bucket-Arn 37 | -------------------------------------------------------------------------------- /endpoint/deploy_gpt2_gptj_mme.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Deploying GPT-2 and GPT-J" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In this notebook, we will be using Hugging Face models and SageMaker Hugging Face-specific API's to deploy both GPT-2 and GPT-J. We will also showcase how to deploy what would could be GPT2 models fine-tuned on different datasets to the same SageMaker instance as a Multi Model Endpoint. This will allow you to get real-time predictions from several models, while only paying for one running endpoint instance." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "*****\n", 22 | "## Deploying GTP-2 to SageMaker Multi-Model Endpoint" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "!pip install -U transformers\n", 32 | "!pip install -U sagemaker" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "### Get sagemaker session, role and default bucket\n", 40 | "If you are going to use Sagemaker in a local environment (not SageMaker Studio or Notebook Instances), you need access to an IAM Role with the required permissions for Sagemaker. You can find more about this [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html)." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "import sagemaker\n", 50 | "import boto3\n", 51 | "\n", 52 | "sess = sagemaker.Session()\n", 53 | "# sagemaker session bucket -> used for uploading data, models and logs\n", 54 | "# sagemaker will automatically create this bucket if it not exists\n", 55 | "sagemaker_session_bucket=None\n", 56 | "if sagemaker_session_bucket is None and sess is not None:\n", 57 | " # set to default bucket if a bucket name is not given\n", 58 | " sagemaker_session_bucket = sess.default_bucket()\n", 59 | "\n", 60 | "try:\n", 61 | " role = sagemaker.get_execution_role()\n", 62 | "except ValueError:\n", 63 | " iam = boto3.client('iam')\n", 64 | " role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n", 65 | "\n", 66 | "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n", 67 | "region = sess.boto_region_name\n", 68 | "sm_client = boto3.client('sagemaker')\n", 69 | "\n", 70 | "print(f\"sagemaker role arn: {role}\")\n", 71 | "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n", 72 | "print(f\"sagemaker session region: {region}\")" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "### Load GPT-2 model and tokenizer, save them to the same folder with Transformers `save_pretrained` utility " 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "import transformers \n", 89 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 90 | "\n", 91 | "tokenizer = AutoTokenizer.from_pretrained('gpt2')\n", 92 | "model = AutoModelForCausalLM.from_pretrained('gpt2')\n", 93 | "\n", 94 | "model.save_pretrained('gpt2-model/')\n", 95 | "tokenizer.save_pretrained('gpt2-model/')" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "# This cell is meant to test that the model can be loaded from the local artifact\n", 105 | "# model = AutoModelForCausalLM.from_pretrained('gpt2-model/')" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": {}, 111 | "source": [ 112 | "Test model generation, by generating 5 different sequences for the same prompt." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "model.eval()\n", 122 | "\n", 123 | "text = \"A rose by any other name would smell as sweet, by William Shakespeare.\"\n", 124 | "input_ids = tokenizer.encode(text, return_tensors = 'pt')\n", 125 | "\n", 126 | "sample_outputs = model.generate(input_ids,\n", 127 | " do_sample = True, \n", 128 | " max_length = 70,\n", 129 | " num_return_sequences = 5) #to test how long we can generate and it be coh\n", 130 | "\n", 131 | "print(\"Output:\\n\" + 100 * '-')\n", 132 | "for i, sample_output in enumerate(sample_outputs):\n", 133 | " print(\"{}: {}...\".format(i, tokenizer.decode(sample_output, skip_special_tokens = True)))\n", 134 | " print('')" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "### Tar model and tokenizer artifacts, upload to S3" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "import tarfile \n", 151 | "\n", 152 | "with tarfile.open('gpt2-model.tar.gz', 'w:gz') as f:\n", 153 | " f.add('gpt2-model/',arcname='.')\n", 154 | "f.close()\n", 155 | "\n", 156 | "prefix = 'gpt2-hf-workshop/gpt2-test'" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "Check out the file contents and structure of the model.tar.gz artifact." 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "! tar -ztvf gpt2-model.tar.gz" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "We will upload the same model package twice with different names, to simulate deploying 2 models to the same endpoint." 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "! aws s3 cp gpt2-model.tar.gz s3://\"$sagemaker_session_bucket\"/\"$prefix\"/gpt2-model1.tar.gz\n", 189 | "! aws s3 cp gpt2-model.tar.gz s3://\"$sagemaker_session_bucket\"/\"$prefix\"/gpt2-model2.tar.gz" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "### Get image URI for Hugging Face inference Deep Learning Container" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "metadata": {}, 203 | "outputs": [], 204 | "source": [ 205 | "from sagemaker import image_uris\n", 206 | "\n", 207 | "hf_inference_dlc = image_uris.retrieve(framework='huggingface', \n", 208 | " region=region, \n", 209 | " version='4.12.3', \n", 210 | " image_scope='inference', \n", 211 | " base_framework_version='pytorch1.9.1', \n", 212 | " py_version='py38', \n", 213 | " container_version='ubuntu20.04', \n", 214 | " instance_type='ml.c5.xlarge')" 215 | ] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "metadata": {}, 220 | "source": [ 221 | "### Use `MultiDataModel`to setup a multi-model endpoint definition\n", 222 | "By setting the `HF_TASK` environment variable, we avoid having to write and test our own inference code. Depending on the task and model you choose, the Hugging Face inference Container will run the appropriate code by default. " 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "from sagemaker.multidatamodel import MultiDataModel\n", 232 | "from sagemaker.predictor import Predictor\n", 233 | "\n", 234 | "hub = {\n", 235 | " 'HF_TASK':'text-generation'\n", 236 | "}\n", 237 | "\n", 238 | "mme = MultiDataModel(\n", 239 | " name='gpt2-models',\n", 240 | " model_data_prefix=f's3://{sagemaker_session_bucket}/{prefix}/',\n", 241 | " image_uri=hf_inference_dlc,\n", 242 | " env=hub,\n", 243 | " predictor_cls=Predictor,\n", 244 | " role=role,\n", 245 | " sagemaker_session=sess,\n", 246 | " )" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "metadata": {}, 252 | "source": [ 253 | "We can see that our model object has already \"registered\" the model artifacts we uploaded to S3 under the `model_data_prefix`." 254 | ] 255 | }, 256 | { 257 | "cell_type": "code", 258 | "execution_count": null, 259 | "metadata": {}, 260 | "outputs": [], 261 | "source": [ 262 | "for model in mme.list_models():\n", 263 | " print(model)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "### Deploy Multi-Model Endpoint and send inference requests to both models" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "metadata": {}, 277 | "outputs": [], 278 | "source": [ 279 | "import datetime\n", 280 | "from sagemaker.serializers import JSONSerializer\n", 281 | "from sagemaker.deserializers import JSONDeserializer\n", 282 | "\n", 283 | "endpoint_name_gpt2 = 'mme-gpt2-'+datetime.datetime.now().strftime(\n", 284 | " \"%Y-%m-%d-%H-%M-%S\"\n", 285 | ")\n", 286 | "\n", 287 | "predictor_gpt2 = mme.deploy(\n", 288 | " initial_instance_count=1,\n", 289 | " instance_type='ml.c5.xlarge',\n", 290 | " serializer=JSONSerializer(),\n", 291 | " deserializer=JSONDeserializer(),\n", 292 | " endpoint_name='mme-gpt2'\n", 293 | " )" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "You can now get predictions from both models; the first request made to each model will take longer than the subsequent, as the model will be loaded from S3." 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "predictor_gpt2.predict({'inputs':'A rose by any other name.'},\n", 310 | " target_model='gpt2-model1.tar.gz')[0]" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [ 319 | "predictor_gpt2.predict({'inputs':'A rose by any other name.'},\n", 320 | " target_model='gpt2-model2.tar.gz')[0]" 321 | ] 322 | }, 323 | { 324 | "cell_type": "markdown", 325 | "metadata": {}, 326 | "source": [ 327 | "### Add new model to endpoint\n", 328 | "To add a new model to our multi-model endpoint, we only have to upload a new model artifact to the same prefix where we uploaded the other models to. You will be able to load and get inferences from this new model as soon as it is uploded to S3. We will again load the same artifact we previously packaged, for demonstration purposes." 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": null, 334 | "metadata": {}, 335 | "outputs": [], 336 | "source": [ 337 | "! aws s3 cp gpt2-model.tar.gz s3://\"$sagemaker_session_bucket\"/\"$prefix\"/gpt2-model3.tar.gz" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "predictor_gpt2.predict({'inputs':'A rose by any other name.'},\n", 347 | " target_model='gpt2-model3.tar.gz')[0]" 348 | ] 349 | }, 350 | { 351 | "cell_type": "markdown", 352 | "metadata": {}, 353 | "source": [ 354 | "\n", 355 | "********************************************************************************************************************************************\n", 356 | "********************************************************************************************************************************************\n", 357 | "\n", 358 | "\n", 359 | "# Deploying GPT-J to SageMaker Endpoint" 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "### Clone sample repo and run model preparation script\n", 367 | "Hugging Face has solidified best practices for deploying GPT-J on Sagemaker in this [repository](https://github.com/philschmid/amazon-sagemaker-gpt-j-sample). Namely, PyTorch utilities are directly used to save the model to disk, instead of `.save_pretrained()`. On deployment, this helps in reducing model loading time by 10x. Check out this [blog post](https://huggingface.co/blog/gptj-sagemaker) to learn more." 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": null, 373 | "metadata": {}, 374 | "outputs": [], 375 | "source": [ 376 | "%%bash\n", 377 | "git clone https://github.com/philschmid/amazon-sagemaker-gpt-j-sample.git\n", 378 | "\n", 379 | "mv amazon-sagemaker-gpt-j-sample/convert_gptj.py \\\n", 380 | " amazon-sagemaker-gpt-j-sample/requirements.txt \\\n", 381 | " amazon-sagemaker-gpt-j-sample/code/ .\n", 382 | "\n", 383 | "rm -r amazon-sagemaker-gpt-j-sample/\n", 384 | "pip install -r requirements.txt" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "The `convert_gpj.py` script will save the model, tokenizer and inference script to disk, create a tar file with those artifacts, and upload them to an S3 bucket of our choice, under the `gpt-j` prefix. We also directly get the S3 URI for our model artifact from the script's execution. " 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "output = !python3 convert_gptj.py --bucket_name \"$sagemaker_session_bucket\"\n", 401 | "model_uri = output[0]" 402 | ] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "metadata": {}, 407 | "source": [ 408 | "We now use the `HuggingFaceModel` API to deploy GPT-J to a SageMaker endpoint, from which we can get real time predictions. Due to the way we saved our model, it is important that the Transformers and PyTorch version you use match the ones installed in the environment where you are running this notebook." 409 | ] 410 | }, 411 | { 412 | "cell_type": "code", 413 | "execution_count": null, 414 | "metadata": {}, 415 | "outputs": [], 416 | "source": [ 417 | "from sagemaker.huggingface import HuggingFaceModel\n", 418 | "\n", 419 | "huggingface_model = HuggingFaceModel(\n", 420 | " model_data=model_uri,\n", 421 | " transformers_version='4.12.3',\n", 422 | " pytorch_version='1.9.1',\n", 423 | " py_version='py38',\n", 424 | " role=role, \n", 425 | " )\n" 426 | ] 427 | }, 428 | { 429 | "cell_type": "code", 430 | "execution_count": null, 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "endpoint_name_gptj = 'gptj-'+datetime.datetime.now().strftime(\n", 435 | " \"%Y-%m-%d-%H-%M-%S\"\n", 436 | ")\n", 437 | "\n", 438 | "predictor_gptj = huggingface_model.deploy(\n", 439 | " initial_instance_count=1,\n", 440 | " instance_type='ml.g4dn.xlarge',\n", 441 | " endpoint_name=endpoint_name_gptj\n", 442 | ")" 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": {}, 448 | "source": [ 449 | "Now we can get real-time predictions from our model!" 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": null, 455 | "metadata": {}, 456 | "outputs": [], 457 | "source": [ 458 | "predictor_gptj.predict({\n", 459 | " \"inputs\": \"Can you please let us know more details about your \",\n", 460 | "})" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "### Cleanup " 468 | ] 469 | }, 470 | { 471 | "cell_type": "code", 472 | "execution_count": null, 473 | "metadata": {}, 474 | "outputs": [], 475 | "source": [ 476 | "predictor_gpt2.delete_model()\n", 477 | "predictor_gpt2.delete_endpoint()\n", 478 | "predictor_gptj.delete_model()\n", 479 | "predictor_gptj.delete_endpoint()" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "Utility: To load endpoint from name" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": {}, 493 | "outputs": [], 494 | "source": [ 495 | "# from sagemaker.predictor import Predictor\n", 496 | "\n", 497 | "# predictor = Predictor(\n", 498 | "# endpoint_name='mme-test-gpt2',\n", 499 | "# serializer=JSONSerializer(),\n", 500 | "# deserializer=JSONDeserializer())" 501 | ] 502 | } 503 | ], 504 | "metadata": { 505 | "instance_type": "ml.t3.medium", 506 | "kernelspec": { 507 | "display_name": "Python 3 (PyTorch 1.8 Python 3.6 CPU Optimized)", 508 | "language": "python", 509 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/1.8.1-cpu-py36" 510 | }, 511 | "language_info": { 512 | "codemirror_mode": { 513 | "name": "ipython", 514 | "version": 3 515 | }, 516 | "file_extension": ".py", 517 | "mimetype": "text/x-python", 518 | "name": "python", 519 | "nbconvert_exporter": "python", 520 | "pygments_lexer": "ipython3", 521 | "version": "3.6.13" 522 | } 523 | }, 524 | "nbformat": 4, 525 | "nbformat_minor": 4 526 | } 527 | -------------------------------------------------------------------------------- /main_notebook/Generate_Text_with_GPT2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Text generation with Pretrained GPT2 models from Hugging Face on Amazon SageMaker\n", 8 | "## The Poetry of NLP\n", 9 | "\n", 10 | "You’ve just been hired by the Chicago Tribune to start a new poetry column. Congrats! The catch? You need to write a new poem every day. And it can’t just be any old string of syllables, you need it to be fresh, authentic, to resonate with the times and carry a sense of rhyme. You need it to delight your readers, to drive up the Tribune’s daily readership numbers and grow their social media presence. How are you going to accomplish this? With the help of Hugging Face and NLP models on SageMaker of course! " 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### In this notebook, we'll execute the following steps.\n", 18 | "\n", 19 | "1. Use the Hugging Face transformfers SDK to download pretrained NLP models and test them locally.\n", 20 | "2. Select a dataset from among our favorite authors.\n", 21 | "3. Finetune the pretrained model using SageMaker training.\n", 22 | "4. Deploy the model into S3.\n", 23 | "5. Trigger a pipeline to test and deploy the model onto a multi-container endpoint.\n", 24 | "6. Test your multi-model endpoint locally to write poetry and text in the style of your favorite authors. " 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "Please note, this notebook was built on SageMaker Studio, using an ml.t3.medium kernel gatway application, and the Python 3.6 PyTorch 1.8 CPU Jupyter Kernel." 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "### Step 0. Install the transformers SDK locally." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 6, 44 | "metadata": {}, 45 | "outputs": [ 46 | { 47 | "name": "stdout", 48 | "output_type": "stream", 49 | "text": [ 50 | "Overwriting requirements.txt\n" 51 | ] 52 | } 53 | ], 54 | "source": [ 55 | "%%writefile requirements.txt \n", 56 | "\n", 57 | "transformers==4.6.1\n", 58 | "datasets" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 4, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "!pip install -r requirements.txt" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "### Step 1. Download a pretrained GPT2 model and test locally.\n", 75 | "We're using the Transformers SDK syntax available on the model card here: https://huggingface.co/gpt2 \n", 76 | "\n", 77 | "To make this model even better, we'll use a version of GPT2 that **has already been finetuned to generate poetry!**" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 35, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 87 | "\n", 88 | "poem_gpt = \"ismaelfaro/gpt2-poems.en\"\n", 89 | "\n", 90 | "tokenizer = AutoTokenizer.from_pretrained(poem_gpt)\n", 91 | "\n", 92 | "base_model = AutoModelForCausalLM.from_pretrained(poem_gpt)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 50, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "from transformers import set_seed\n", 102 | "\n", 103 | "def get_outputs(sample_outputs, tokenizer):\n", 104 | " # takes a tokenizer, and raw output from the model, decodes these and formats nicely\n", 105 | " rt = []\n", 106 | "\n", 107 | " print(\"Output:\\n\" + 100 * '-')\n", 108 | " for i, sample_output in enumerate(sample_outputs):\n", 109 | " txt = tokenizer.decode(sample_output, skip_special_tokens = True)\n", 110 | " print(\"{}: {}...\".format(i, txt))\n", 111 | " print('')\n", 112 | " rt.append(txt)\n", 113 | " \n", 114 | " return rt\n", 115 | "\n", 116 | "# setting the seed helps us ensure reproducibility. when the seed is consistent, we know the model results will be consistent\n", 117 | "set_seed(42)\n", 118 | "\n", 119 | "text = \"A rose by any other name\"\n", 120 | "\n", 121 | "input_ids = tokenizer.encode(text, return_tensors = 'pt')\n", 122 | "\n", 123 | "sample_outputs = base_model.generate(input_ids,\n", 124 | " do_sample = True, \n", 125 | " max_length = 70,\n", 126 | " num_return_sequences = 5) \n", 127 | "\n", 128 | "generic_outputs = get_outputs(sample_outputs, tokenizer)" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "Interesting and entertaining! Clearly this model knows the form of poetry. It is obviously generating short lines, with a newline, and it seems to pick up some interesting concepts. Now, let's see if we can fine-tune this poem writer to fit the style of an author we have in mind." 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "### Step 2. Fine-tune the GPT2 Poem model with Anne Bradstreet.\n", 143 | "Now, we're going to fine-tune this model using another, much smaller, dataset. Then later we'll use a text classifier trained to evaluate this style of writer, and see how well our new text performs!" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "If you're curious, take a look at some of the top authors in the English language available through this open domain site.\n", 151 | "https://www.public-domain-poetry.com/topauthors.php \n", 152 | "\n", 153 | "For the purposes of this workshop we'll stick to the longer poem pasted below. On your time time, outside of the workshop, if you'd like to modify this to work with a different text you are welcome to do so.\n", 154 | "\n", 155 | "Poke around at some of the available poems, and copy and paste what you like into this `train.txt` file below. We'll format that for finetuning GPT2 in the next step. In this notebook we're using a poem from Anne Bradstreet, a North American writer from the 17th Century.\n", 156 | "\n", 157 | "You may not have known this, but Anne Bradstreet was the first writer to be published in the North America!" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 34, 163 | "metadata": { 164 | "jupyter": { 165 | "source_hidden": true 166 | } 167 | }, 168 | "outputs": [ 169 | { 170 | "name": "stdout", 171 | "output_type": "stream", 172 | "text": [ 173 | "Writing train.txt\n" 174 | ] 175 | } 176 | ], 177 | "source": [ 178 | "%%writefile train.txt\n", 179 | "\n", 180 | "A Dialogue Between Old England And New\n", 181 | "\n", 182 | " By Anne Bradstreet\n", 183 | "\n", 184 | " New England.\n", 185 | "\n", 186 | " Alas, dear Mother, fairest Queen and best,\n", 187 | " With honour, wealth, and peace happy and blest,\n", 188 | " What ails thee hang thy head, and cross thine arms,\n", 189 | " And sit i� the dust to sigh these sad alarms?\n", 190 | " What deluge of new woes thus over-whelm\n", 191 | " The glories of thy ever famous Realm?\n", 192 | " What means this wailing tone, this mournful guise?\n", 193 | " Ah, tell thy Daughter; she may sympathize.\n", 194 | "\n", 195 | " Old England.\n", 196 | "\n", 197 | " Art ignorant indeed of these my woes,\n", 198 | " Or must my forced tongue these griefs disclose,\n", 199 | " And must my self dissect my tatter�d state,\n", 200 | " Which Amazed Christendom stands wondering at?\n", 201 | " And thou a child, a Limb, and dost not feel\n", 202 | " My weak�ned fainting body now to reel?\n", 203 | " This physic-purging-potion I have taken\n", 204 | " Will bring Consumption or an Ague quaking,\n", 205 | " Unless some Cordial thou fetch from high,\n", 206 | " Which present help may ease my malady.\n", 207 | " If I decease, dost think thou shalt survive?\n", 208 | " Or by my wasting state dost think to thrive?\n", 209 | " Then weigh our case, if �t be not justly sad.\n", 210 | " Let me lament alone, while thou art glad.\n", 211 | "\n", 212 | " New England.\n", 213 | "\n", 214 | " And thus, alas, your state you much deplore\n", 215 | " In general terms, but will not say wherefore.\n", 216 | " What Medicine shall I seek to cure this woe,\n", 217 | " If th� wound�s so dangerous, I may not know?\n", 218 | " But you, perhaps, would have me guess it out.\n", 219 | " What, hath some Hengist like that Saxon stout\n", 220 | " By fraud and force usurp�d thy flow�ring crown,\n", 221 | " Or by tempestuous Wars thy fields trod down?\n", 222 | " Or hath Canutus, that brave valiant Dane,\n", 223 | " The regal peaceful Sceptre from thee ta�en?\n", 224 | " Or is �t a Norman whose victorious hand\n", 225 | " With English blood bedews thy conquered Land?\n", 226 | " Or is �t intestine Wars that thus offend?\n", 227 | " Do Maud and Stephen for the Crown contend?\n", 228 | " Do Barons rise and side against their King,\n", 229 | " And call in Foreign aid to help the thing?\n", 230 | " Must Edward be depos�d? Or is �t the hour\n", 231 | " That second Richard must be clapp�d i� th� Tower?\n", 232 | " Or is it the fatal jar, again begun,\n", 233 | " That from the red, white pricking Roses sprung?\n", 234 | " Must Richmond�s aid the Nobles now implore\n", 235 | " To come and break the tushes of the Boar?\n", 236 | " If none of these, dear Mother, what�s your woe?\n", 237 | " Pray, do not fear Spain�s bragging Armado.\n", 238 | " Doth your Ally, fair France, conspire your wrack,\n", 239 | " Or doth the Scots play false behind your back?\n", 240 | " Doth Holland quit you ill for all your love?\n", 241 | " Whence is this storm, from Earth or Heaven above?\n", 242 | " Is �t drought, is �t Famine, or is �t Pestilence?\n", 243 | " Dost feel the smart, or fear the consequence?\n", 244 | " Your humble Child entreats you shew your grief.\n", 245 | " Though Arms nor Purse she hath for your relief�\n", 246 | " Such is her poverty,�yet shall be found\n", 247 | " A suppliant for your help, as she is bound.\n", 248 | "\n", 249 | " Old England.\n", 250 | "\n", 251 | " I must confess some of those Sores you name\n", 252 | " My beauteous Body at this present maim,\n", 253 | " But foreign Foe nor feigned friend I fear,\n", 254 | " For they have work enough, thou knowest, elsewhere.\n", 255 | " Nor is it Alcie�s son and Henry�s Daughter\n", 256 | " Whose proud contention cause this slaughter;\n", 257 | " Nor Nobles siding to make John no King,\n", 258 | " French Louis unjustly to the Crown to bring;\n", 259 | " No Edward, Richard, to lose rule and life,\n", 260 | " Nor no Lancastrians to renew old strife;\n", 261 | " No Crook-backt Tyrant now usurps the Seat,\n", 262 | " Whose tearing tusks did wound, and kill, and threat.\n", 263 | " No Duke of York nor Earl of March to soil\n", 264 | " Their hands in Kindred�s blood whom they did foil;\n", 265 | " No need of Tudor Roses to unite:\n", 266 | " None knows which is the Red or which the White.\n", 267 | " Spain�s braving Fleet a second time is sunk.\n", 268 | " France knows how of my fury she hath drunk\n", 269 | " By Edward third and Henry fifth of fame;\n", 270 | " Her Lilies in my Arms avouch the same.\n", 271 | " My Sister Scotland hurts me now no more,\n", 272 | " Though she hath been injurious heretofore.\n", 273 | " What Holland is, I am in some suspense,\n", 274 | " But trust not much unto his Excellence.\n", 275 | " For wants, sure some I feel, but more I fear;\n", 276 | " And for the Pestilence, who knows how near?\n", 277 | " Famine and Plague, two sisters of the Sword,\n", 278 | " Destruction to a Land doth soon afford.\n", 279 | " They�re for my punishments ordain�d on high,\n", 280 | " Unless thy tears prevent it speedily.\n", 281 | " But yet I answer not what you demand\n", 282 | " To shew the grievance of my troubled Land.\n", 283 | " Before I tell the effect I�ll shew the cause,\n", 284 | " Which are my sins�the breach of sacred Laws:\n", 285 | " Idolatry, supplanter of a N ation,\n", 286 | " With foolish superstitious adoration,\n", 287 | " Are lik�d and countenanc�d by men of might,\n", 288 | " The Gospel is trod down and hath no right.\n", 289 | " Church Offices are sold and bought for gain\n", 290 | " That Pope had hope to find Rome here again.\n", 291 | " For Oaths and Blasphemies did ever ear\n", 292 | " From Beelzebub himself such language hear?\n", 293 | " What scorning of the Saints of the most high!\n", 294 | " What injuries did daily on them lie!\n", 295 | " What false reports, what nick-names did they take,\n", 296 | " Not for their own, but for their Master�s sake!\n", 297 | " And thou, poor soul, wast jeer�d among the rest;\n", 298 | " Thy flying for the Truth I made a jest.\n", 299 | " For Sabbath-breaking and for Drunkenness\n", 300 | " Did ever Land profaneness more express?\n", 301 | " From crying bloods yet cleansed am not I,\n", 302 | " Martyrs and others dying causelessly.\n", 303 | " How many Princely heads on blocks laid down\n", 304 | " For nought but title to a fading Crown!\n", 305 | " �Mongst all the cruelties which I have done,\n", 306 | " Oh, Edward�s Babes, and Clarence�s hapless Son,\n", 307 | " O Jane, why didst thou die in flow�ring prime?�\n", 308 | " Because of Royal Stem, that was thy crime.\n", 309 | " For Bribery, Adultery, for Thefts, and Lies\n", 310 | " Where is the Nation I can�t paralyze?\n", 311 | " With Usury, Extortion, and Oppression,\n", 312 | " These be the Hydras of my stout transgression;\n", 313 | " These be the bitter fountains, heads, and roots\n", 314 | " Whence flow�d the source, the sprigs, the boughs, and fruits.\n", 315 | " Of more than thou canst hear or I relate,\n", 316 | " That with high hand I still did perpetrate,\n", 317 | " For these were threat�ned the woeful day\n", 318 | " I mocked the Preachers, put it fair away.\n", 319 | " The Sermons yet upon record do stand\n", 320 | " That cried destruction to my wicked Land.\n", 321 | " These Prophets� mouths (all the while) was stopt,\n", 322 | " Unworthily, some backs whipt, and ears crept;\n", 323 | " Their reverent cheeks bear the glorious marks\n", 324 | " Of stinking, stigmatizing Romish Clerks;\n", 325 | " Some lost their livings, some in prison pent,\n", 326 | " Some grossly fined, from friends to exile went:\n", 327 | " Their silent tongues to heaven did vengeance cry,\n", 328 | " Who heard their cause, and wrongs judg�d righteously,\n", 329 | " And will repay it sevenfold in my lap.\n", 330 | " This is fore-runner of my after-clap.\n", 331 | " Nor took I warning by my neighbors� falls.\n", 332 | " I saw sad Germany�s dismantled walls,\n", 333 | " I saw her people famish�d, Nobles slain,\n", 334 | " Her fruitful land a barren heath remain.\n", 335 | " I saw (unmov�d) her Armies foil�d and fled,\n", 336 | " Wives forc�d, babes toss�d, her houses calcined.\n", 337 | " I saw strong Rochelle yield�d to her foe,\n", 338 | " Thousands of starved Christians there also.\n", 339 | " I saw poor Ireland bleeding out her last,\n", 340 | " Such cruelty as all reports have past.\n", 341 | " Mine heart obdurate stood not yet aghast.\n", 342 | " Now sip I of that cup, and just �t may be\n", 343 | " The bottom dregs reserved are for me.\n", 344 | "\n", 345 | " New England.\n", 346 | "\n", 347 | " To all you�ve said, sad mother, I assent.\n", 348 | " Your fearful sins great cause there �s to lament.\n", 349 | " My guilty hands (in part) hold up with you,\n", 350 | " A sharer in your punishment�s my due.\n", 351 | " But all you say amounts to this effect,\n", 352 | " Not what you feel, but what you do expect.\n", 353 | " Pray, in plain terms, what is your present grief?\n", 354 | " Then let�s join heads and hands for your relief.\n", 355 | "\n", 356 | " Old England.\n", 357 | "\n", 358 | " Well, to the matter, then. There�s grown of late\n", 359 | " �Twixt King and Peers a question of state:\n", 360 | " Which is the chief, the law, or else the King?\n", 361 | " One saith, it�s he; the other, no such thing.\n", 362 | " My better part in Court of Parliament\n", 363 | " To ease my groaning land shew their intent\n", 364 | " To crush the proud, and right to each man deal,\n", 365 | " To help the Church, and stay the Common-Weal.\n", 366 | " So many obstacles comes in their way\n", 367 | " As puts me to a stand what I should say.\n", 368 | " Old customs, new Prerogatives stood on.\n", 369 | " Had they not held law fast, all had been gone,\n", 370 | " Which by their prudence stood them in such stead\n", 371 | " They took high Strafford lower by the head,\n", 372 | " And to their Laud be �t spoke they held �n th� Tower\n", 373 | " All England�s metropolitan that hour.\n", 374 | " This done, an Act they would have passed fain\n", 375 | " No prelate should his Bishopric retain.\n", 376 | " Here tugg�d they hard indeed, for all men saw\n", 377 | " This must be done by Gospel, not by law.\n", 378 | " Next the Militia they urged sore.\n", 379 | " This was denied, I need not say wherefore.\n", 380 | " The King, displeased, at York himself absents.\n", 381 | " They humbly beg return, shew their intents.\n", 382 | " The writing, printing, posting to and fro,\n", 383 | " Shews all was done; I�ll therefore let it go.\n", 384 | " But now I come to speak of my disaster.\n", 385 | " Contention�s grown �twixt Subjects and their Master,\n", 386 | " They worded it so long they fell to blows,\n", 387 | " That thousands lay on heaps. Here bleeds my woes.\n", 388 | " I that no wars so many years have known\n", 389 | " Am now destroy�d and slaughter�d by mine own.\n", 390 | " But could the field alone this strife decide,\n", 391 | " One battle, two, or three I might abide,\n", 392 | " But these may be beginnings of more woe�\n", 393 | " Who knows, the worst, the best may overthrow!\n", 394 | " Religion, Gospel, here lies at the stake,\n", 395 | " Pray now, dear child, for sacred Zion�s sake,\n", 396 | " Oh, pity me in this sad perturbation,\n", 397 | " My plundered Towns, my houses� devastation,\n", 398 | " My ravisht virgins, and my young men slain,\n", 399 | " My wealthy trading fallen, my dearth of grain.\n", 400 | " The seedtime�s come, but Ploughman hath no hope\n", 401 | " Because he knows not who shall inn his crop.\n", 402 | " The poor they want their pay, their children bread,\n", 403 | " Their woful mothers� tears unpitied.\n", 404 | " If any pity in thy heart remain,\n", 405 | " Or any child-like love thou dost retain,\n", 406 | " For my relief now use thy utmost skill,\n", 407 | " And recompense me good for all my ill.\n", 408 | "\n", 409 | " New England.\n", 410 | "\n", 411 | " Dear mother, cease complaints, and wipe your eyes,\n", 412 | " Shake off your dust, cheer up, and now arise.\n", 413 | " You are my mother, nurse, I once your flesh,\n", 414 | " Your sunken bowels gladly would refresh.\n", 415 | " Your griefs I pity much but should do wrong,\n", 416 | " To weep for that we both have pray�d for long,\n", 417 | " To see these latter days of hop�d-for good,\n", 418 | " That Right may have its right, though �t be with blood.\n", 419 | " After dark Popery the day did clear;\n", 420 | " But now the Sun in�s brightness shall appear.\n", 421 | " Blest be the Nobles of thy Noble Land\n", 422 | " With (ventur�d lives) for truth�s defence that stand.\n", 423 | " Blest be thy Commons, who for Common good\n", 424 | " And thy infringed Laws have boldly stood.\n", 425 | " Blest be thy Counties, who do aid thee still\n", 426 | " With hearts and states to testify their will.\n", 427 | " Blest be thy Preachers, who do cheer thee on.\n", 428 | " Oh, cry: the sword of God and Gideon!\n", 429 | " And shall I not on them wish Mero�s curse\n", 430 | " That help thee not with prayers, arms, and purse?\n", 431 | " And for my self, let miseries abound\n", 432 | " If mindless of thy state I e�er be found.\n", 433 | " These are the days the Church�s foes to crush,\n", 434 | " To root out Prelates, head, tail, branch, and rush.\n", 435 | " Let�s bring Baal�s vestments out, to make a fire,\n", 436 | " Their Mitres, Surplices, and all their tire,\n", 437 | " Copes, Rochets, Croziers, and such trash,\n", 438 | " And let their names consume, but let the flash\n", 439 | " Light Christendom, and all the world to see\n", 440 | " We hate Rome�s Whore, with all her trumpery.\n", 441 | " Go on, brave Essex, shew whose son thou art,\n", 442 | " Not false to King, nor Country in thy heart,\n", 443 | " But those that hurt his people and his Crown,\n", 444 | " By force expel, destroy, and tread them down.\n", 445 | " Let Gaols be fill�d with th� remnant of that pack,\n", 446 | " And sturdy Tyburn loaded till it crack.\n", 447 | " And ye brave Nobles, chase away all fear,\n", 448 | " And to this blessed Cause closely adhere.\n", 449 | " O mother, can you weep and have such Peers?\n", 450 | " When they are gone, then drown your self in tears,\n", 451 | " If now you weep so much, that then no more\n", 452 | " The briny Ocean will o�erflow your shore.\n", 453 | " These, these are they (I trust) with Charles our king,\n", 454 | " Out of all mists such glorious days will bring\n", 455 | " That dazzled eyes, beholding, much shall wonder\n", 456 | " At that thy settled Peace, thy wealth, and splendour,\n", 457 | " Thy Church and Weal establish�d in such manner\n", 458 | " That all shall joy that thou display�dst thy banner,\n", 459 | " And discipline erected so, I trust,\n", 460 | " That nursing Kings shall come and lick thy dust.\n", 461 | " Then Justice shall in all thy Courts take place\n", 462 | " Without respect of persons or of case.\n", 463 | " Then bribes shall cease, and suits shall not stick long,\n", 464 | " Patience and purse of Clients for to wrong.\n", 465 | " Then High Commissions shall fall to decay,\n", 466 | " And Pursuivants and Catchpoles want their pay.\n", 467 | " So shall thy happy Nation ever flourish,\n", 468 | " When truth and righteousness they thus shall nourish.\n", 469 | " When thus in Peace, thine Armies brave send out\n", 470 | " To sack proud Rome, and all her vassals rout.\n", 471 | " There let thy name, thy fame, and valour shine,\n", 472 | " As did thine Ancestors� in Palestine,\n", 473 | " And let her spoils full pay with int�rest be\n", 474 | " Of what unjustly once she poll�d from thee.\n", 475 | " Of all the woes thou canst let her be sped,\n", 476 | " Execute to th� full the vengeance threatened.\n", 477 | " Bring forth the beast that rul�d the world with�s beck,\n", 478 | " And tear his flesh, and set your feet on�s neck,\n", 479 | " And make his filthy den so desolate\n", 480 | " To th� �stonishment of all that knew his state.\n", 481 | " This done, with brandish�d swords to Turkey go,�\n", 482 | " (For then what is it but English blades dare do?)\n", 483 | " And lay her waste, for so�s the sacred doom,\n", 484 | " And do to Gog as thou hast done to Rome.\n", 485 | " Oh Abraham�s seed, lift up your heads on high,\n", 486 | " For sure the day of your redemption�s nigh.\n", 487 | " The scales shall fall from your long blinded eyes,\n", 488 | " And him you shall adore who now despise.\n", 489 | " Then fullness of the Nations in shall flow,\n", 490 | " And Jew and Gentile to one worship go.\n", 491 | " Then follows days of happiness and rest.\n", 492 | " Whose lot doth fall to live therein is blest.\n", 493 | " No Canaanite shall then be found �n th� land,\n", 494 | " And holiness on horses� bells shall stand.\n", 495 | " If this make way thereto, then sigh no more,\n", 496 | " But if at all thou didst not see �t before.\n", 497 | " Farewell, dear mother; Parliament, prevail,\n", 498 | " And in a while you�ll tell another tale.\n" 499 | ] 500 | }, 501 | { 502 | "cell_type": "markdown", 503 | "metadata": {}, 504 | "source": [ 505 | "### Step 3. Format your training data for Hugging Face on Amazon SageMaker.\n", 506 | "Now, let's parse your training data to format it for finetuning GPT2 and training on Hugging Face. " 507 | ] 508 | }, 509 | { 510 | "cell_type": "code", 511 | "execution_count": 37, 512 | "metadata": {}, 513 | "outputs": [], 514 | "source": [ 515 | "data = []\n", 516 | "\n", 517 | "with open('train.txt') as f:\n", 518 | " for row in f.readlines():\n", 519 | " d = row.strip()\n", 520 | " if len(d) > 1:\n", 521 | " data.append(d)" 522 | ] 523 | }, 524 | { 525 | "cell_type": "code", 526 | "execution_count": 38, 527 | "metadata": {}, 528 | "outputs": [ 529 | { 530 | "name": "stdout", 531 | "output_type": "stream", 532 | "text": [ 533 | "Found 304 valid objects in the training data.\n" 534 | ] 535 | } 536 | ], 537 | "source": [ 538 | "print ('Found {} valid objects in the training data.'.format(len(data)))" 539 | ] 540 | }, 541 | { 542 | "cell_type": "code", 543 | "execution_count": 39, 544 | "metadata": {}, 545 | "outputs": [ 546 | { 547 | "name": "stdout", 548 | "output_type": "stream", 549 | "text": [ 550 | "['A Dialogue Between Old England And New', 'By Anne Bradstreet', 'New England.', 'Alas, dear Mother, fairest Queen and best,', 'With honour, wealth, and peace happy and blest,', 'What ails thee hang thy head, and cross thine arms,', 'And sit i� the dust to sigh these sad alarms?', 'What deluge of new woes thus over-whelm', 'The glories of thy ever famous Realm?', 'What means this wailing tone, this mournful guise?']\n" 551 | ] 552 | } 553 | ], 554 | "source": [ 555 | "print (data[:10])" 556 | ] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": 49, 561 | "metadata": {}, 562 | "outputs": [], 563 | "source": [ 564 | "import sagemaker\n", 565 | "\n", 566 | "sess = sagemaker.Session()\n", 567 | "bucket = sess.default_bucket() \n", 568 | "\n", 569 | "train_file_name = 'train.txt'\n", 570 | "s3_train_data = 's3://{}/gpt2/{}'.format(bucket, train_file_name)\n", 571 | "\n", 572 | "!aws s3 cp {train_file_name} {s3_train_data}" 573 | ] 574 | }, 575 | { 576 | "cell_type": "code", 577 | "execution_count": 44, 578 | "metadata": {}, 579 | "outputs": [], 580 | "source": [ 581 | "import sagemaker\n", 582 | "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", 583 | "\n", 584 | "# gets role for executing training job\n", 585 | "role = sagemaker.get_execution_role()\n", 586 | "hyperparameters = {\n", 587 | " 'model_name_or_path':\"ismaelfaro/gpt2-poems.en\",\n", 588 | " 'output_dir':'/opt/ml/model',\n", 589 | " 'do_train':True,\n", 590 | " 'train_file': '/opt/ml/input/data/train/{}'.format(train_file_name),\n", 591 | " 'num_train_epochs': 5,\n", 592 | " # set batch size to 22 if using SM training compiler\n", 593 | " \"per_device_train_batch_size\": 64,\n", 594 | " # add your remaining hyperparameters\n", 595 | " # more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/pytorch/language-modeling\n", 596 | "}\n", 597 | "\n", 598 | "# git configuration to download our fine-tuning script\n", 599 | "git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}\n", 600 | "\n", 601 | "# creates Hugging Face estimator\n", 602 | "huggingface_estimator = HuggingFace(\n", 603 | " entry_point='run_clm.py',\n", 604 | " source_dir='./examples/pytorch/language-modeling',\n", 605 | " instance_type='ml.p3.2xlarge',\n", 606 | " instance_count=1,\n", 607 | " role=role,\n", 608 | " git_config=git_config,\n", 609 | " transformers_version='4.11.0',\n", 610 | " pytorch_version='1.9.0',\n", 611 | " py_version='py38',\n", 612 | " hyperparameters = hyperparameters,\n", 613 | " # pass the training compiler config to speed up your job\n", 614 | " compiler_config=TrainingCompilerConfig(),\n", 615 | " environment = {'GPU_NUM_DEVICES':'1'},\n", 616 | " disable_profiler = True,\n", 617 | " debugger_hook_config = False\n", 618 | ")\n", 619 | "\n", 620 | "# starting the train job\n", 621 | "# should take about 13 minutes to run on current settings\n", 622 | "huggingface_estimator.fit({'train':s3_train_data}, wait = True)" 623 | ] 624 | }, 625 | { 626 | "cell_type": "markdown", 627 | "metadata": {}, 628 | "source": [ 629 | "### Step 4. Test your trained model locally" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": 4, 635 | "metadata": {}, 636 | "outputs": [ 637 | { 638 | "name": "stdout", 639 | "output_type": "stream", 640 | "text": [ 641 | "\n", 642 | "2022-03-07 17:46:31 Starting - Preparing the instances for training\n", 643 | "2022-03-07 17:46:31 Downloading - Downloading input data\n", 644 | "2022-03-07 17:46:31 Training - Training image download completed. Training in progress.\n", 645 | "2022-03-07 17:46:31 Uploading - Uploading generated training model\n", 646 | "2022-03-07 17:46:31 Completed - Training job completed\n" 647 | ] 648 | } 649 | ], 650 | "source": [ 651 | "from sagemaker.huggingface import HuggingFace\n", 652 | "\n", 653 | "# redefining if you need to restart your kernel \n", 654 | "# huggingface_estimator = HuggingFace.attach('\", anne_clf, anne_tokenizer)" 879 | ] 880 | }, 881 | { 882 | "cell_type": "markdown", 883 | "metadata": {}, 884 | "source": [ 885 | "Now, run some tests of your own. Try different invocation parameters. What seems to get you the highest Anne scores?" 886 | ] 887 | }, 888 | { 889 | "cell_type": "markdown", 890 | "metadata": {}, 891 | "source": [ 892 | "### Step 4. Deploy your fine-tuned model onto a SageMaker multi-model endpoint\n", 893 | "Now, let's deploy this model onto SageMaker. In particular we will trigger a pipeline to update an existing multi-model endpoint, and then invoke our model from that endpoint. \n", 894 | "\n", 895 | "We'll also list all available models from that endpoint, and test out generating text with each of these. Who knows, maybe we'll stumble on something good enough for the Tribune!" 896 | ] 897 | }, 898 | { 899 | "cell_type": "code", 900 | "execution_count": null, 901 | "metadata": {}, 902 | "outputs": [], 903 | "source": [ 904 | "# mme = " 905 | ] 906 | }, 907 | { 908 | "cell_type": "code", 909 | "execution_count": null, 910 | "metadata": {}, 911 | "outputs": [], 912 | "source": [ 913 | "# your model isn't here yet\n", 914 | "# list(mme.list_models())" 915 | ] 916 | }, 917 | { 918 | "cell_type": "code", 919 | "execution_count": null, 920 | "metadata": {}, 921 | "outputs": [], 922 | "source": [ 923 | "# now invoke the pipeline" 924 | ] 925 | }, 926 | { 927 | "cell_type": "code", 928 | "execution_count": null, 929 | "metadata": {}, 930 | "outputs": [], 931 | "source": [ 932 | "## ADD A GIF OF VIEWING THE PIPELINE INVOCATION HERE" 933 | ] 934 | }, 935 | { 936 | "cell_type": "markdown", 937 | "metadata": {}, 938 | "source": [ 939 | "### Step 5. Test your fine-tuned model on SageMaker multi-model endpoint" 940 | ] 941 | }, 942 | { 943 | "cell_type": "code", 944 | "execution_count": null, 945 | "metadata": {}, 946 | "outputs": [], 947 | "source": [ 948 | "# now, point to the S3 path for the model you just created, and add a name for it\n", 949 | "# mme.add_model(model_data_source=s3_model_data, model_data_path='My-Finetuned-Model')" 950 | ] 951 | }, 952 | { 953 | "cell_type": "code", 954 | "execution_count": null, 955 | "metadata": {}, 956 | "outputs": [], 957 | "source": [ 958 | "# show your model\n", 959 | "# list(mme.list_models())" 960 | ] 961 | }, 962 | { 963 | "cell_type": "code", 964 | "execution_count": 36, 965 | "metadata": {}, 966 | "outputs": [], 967 | "source": [ 968 | "# predictor = sagemaker.predictor.Predictor(endpoint_name = 'hf-multi-gpt2-2022-02-23-23-29-24', sagemaker_session=sess)\n", 969 | "predictor.serializer = sagemaker.serializers.JSONSerializer()\n", 970 | "predictor.deserializer = sagemaker.deserializers.JSONDeserializer()" 971 | ] 972 | }, 973 | { 974 | "cell_type": "code", 975 | "execution_count": 38, 976 | "metadata": {}, 977 | "outputs": [ 978 | { 979 | "data": { 980 | "text/plain": [ 981 | "[{'generated_text': 'A rose by any other name has many meanings. It is the source of a whole variety of food, art, life-affirming practices and ideas, but it is also the keystone in a very great series of cultural expressions.\\n\\nAnd'}]" 982 | ] 983 | }, 984 | "execution_count": 38, 985 | "metadata": {}, 986 | "output_type": "execute_result" 987 | } 988 | ], 989 | "source": [ 990 | "# predictor = point to MME predictor here \n", 991 | "predictor.predict({\"inputs\":'A rose by any other name'}, target_model='anne-bradstreet-gpt2')" 992 | ] 993 | }, 994 | { 995 | "cell_type": "markdown", 996 | "metadata": {}, 997 | "source": [ 998 | "### Step 7. Write poetry for the Chicago Tribune\n", 999 | "Now - select your favorite lines from each output from GPT, and pass it in to the model. Feel free to modify the parameters using kwargs. When you are finished, you can submit your poem to our GitHub workshop page!\n", 1000 | "\n", 1001 | "**Please note** every time you invoke a new model via MME AWS is copying the model artifact from S3 to the SageMaker endpoint. That means **expect a big time delay whenever you invoke a new model.** \n", 1002 | "\n", 1003 | "One way to get around that is with model compilation, ie running SageMaker Neo to decrease the size, and thereby the runtime, of that model.\n", 1004 | "\n", 1005 | "In the poem below, I manually copied my favorite line from each output of the model, and fed it in to the generator. I manually pasted all of my favorites into the markdown file you see below.\n", 1006 | "\n", 1007 | "---" 1008 | ] 1009 | }, 1010 | { 1011 | "cell_type": "markdown", 1012 | "metadata": {}, 1013 | "source": [ 1014 | "### My poem - A rose by any other model\n", 1015 | "\n", 1016 | "A rose by any other name has many meanings.
\n", 1017 | "When all that has been presented to us is a form of exaggeration.
\n", 1018 | "The language will not preserve.
\n", 1019 | "However, the old idea of he who has no business vainly passing by without any other
\n", 1020 | "Some unending mizzen, deceived and deceived, seems ever more absurd and likely to harm our time.
\n", 1021 | "We tuck his back into the sea which is on the plain almost as soon as we lose sight of him.
\n", 1022 | "A mariner shall pass.
\n", 1023 | "And I may leave nothing to thee till thou return, for as I said, My hand am strong when thou shouldst require it.
\n", 1024 | "This comes out of Kant\\'s conviction that we have nothing in our minds
" 1025 | ] 1026 | }, 1027 | { 1028 | "cell_type": "code", 1029 | "execution_count": 48, 1030 | "metadata": {}, 1031 | "outputs": [ 1032 | { 1033 | "data": { 1034 | "text/plain": [ 1035 | "[{'generated_text': 'A rose by any other name has many meanings. The language will not preserve, however, the old idea of \"outly,\" when all that has been presented to us is a form of exaggeration with a foul word, or a dearth of quality: and he who has no business vainly passing by without any other means, not to mention those'}]" 1036 | ] 1037 | }, 1038 | "execution_count": 48, 1039 | "metadata": {}, 1040 | "output_type": "execute_result" 1041 | } 1042 | ], 1043 | "source": [ 1044 | "text = 'A rose by any other name has many meanings.'\n", 1045 | "predictor.predict({\"inputs\":text, \n", 1046 | " 'parameters':{'max_length':70,\n", 1047 | " 'do_sample':True, \n", 1048 | " # only pick tokens at and above this probability level\n", 1049 | " 'top_p':0.99,\n", 1050 | " # only pick from this many tokens\n", 1051 | " 'top_k':600}}, \n", 1052 | " target_model='anne-bradstreet-gpt2')" 1053 | ] 1054 | }, 1055 | { 1056 | "cell_type": "code", 1057 | "execution_count": 49, 1058 | "metadata": {}, 1059 | "outputs": [ 1060 | { 1061 | "data": { 1062 | "text/plain": [ 1063 | "[{'generated_text': 'However, the old idea of he who has no business vainly passing by without any other means than some unending mizzen, deceived and deceived, seems ever more absurd and likely to harm our time. We tuck his back into the sea which is on the plain almost as soon as we lose sight of him. A mariner shall pass'}]" 1064 | ] 1065 | }, 1066 | "execution_count": 49, 1067 | "metadata": {}, 1068 | "output_type": "execute_result" 1069 | } 1070 | ], 1071 | "source": [ 1072 | "text = 'However, the old idea of he who has no business vainly passing by without any other means'\n", 1073 | "\n", 1074 | "predictor.predict({\"inputs\":text, \n", 1075 | " 'parameters':{'max_length':70,\n", 1076 | " 'do_sample':True, \n", 1077 | " # only pick tokens at and above this probability level\n", 1078 | " 'top_p':0.99,\n", 1079 | " # only pick from this many tokens\n", 1080 | " 'top_k':600}}, \n", 1081 | " target_model='anne-bradstreet-gpt2')" 1082 | ] 1083 | }, 1084 | { 1085 | "cell_type": "code", 1086 | "execution_count": 50, 1087 | "metadata": {}, 1088 | "outputs": [ 1089 | { 1090 | "data": { 1091 | "text/plain": [ 1092 | "[{'generated_text': 'A mariner shall pass in front of us with His entrails over His joints, ye are worthy masters. And I may leave nothing to thee till thou return, for as I said, My hand am strong when thou shouldst require it.\\n\\n\"He quivered, trembling and stirred his fist. At this they communicated thine own'}]" 1093 | ] 1094 | }, 1095 | "execution_count": 50, 1096 | "metadata": {}, 1097 | "output_type": "execute_result" 1098 | } 1099 | ], 1100 | "source": [ 1101 | "text = 'A mariner shall pass'\n", 1102 | "\n", 1103 | "predictor.predict({\"inputs\":text, \n", 1104 | " 'parameters':{'max_length':70,\n", 1105 | " 'do_sample':True, \n", 1106 | " # only pick tokens at and above this probability level\n", 1107 | " 'top_p':0.99,\n", 1108 | " # only pick from this many tokens\n", 1109 | " 'top_k':600}}, \n", 1110 | " target_model='anne-bradstreet-gpt2')" 1111 | ] 1112 | }, 1113 | { 1114 | "cell_type": "code", 1115 | "execution_count": 53, 1116 | "metadata": {}, 1117 | "outputs": [ 1118 | { 1119 | "data": { 1120 | "text/plain": [ 1121 | "[{'generated_text': 'A rose by any other model of fashion or sense. No such value can be seen with \"feminine models,\" as I have argued. Although the image is of a sensuous figure, it is not of a woman, for there is a more interesting way of stating it. This comes out of Kant\\'s conviction that we have nothing in our minds'}]" 1122 | ] 1123 | }, 1124 | "execution_count": 53, 1125 | "metadata": {}, 1126 | "output_type": "execute_result" 1127 | } 1128 | ], 1129 | "source": [ 1130 | "text = 'A rose by any other model'\n", 1131 | "\n", 1132 | "predictor.predict({\"inputs\":text, \n", 1133 | " 'parameters':{'max_length':70,\n", 1134 | " 'do_sample':True, \n", 1135 | " # only pick tokens at and above this probability level\n", 1136 | " 'top_p':0.99,\n", 1137 | " # only pick from this many tokens\n", 1138 | " 'top_k':100}}, \n", 1139 | " target_model='anne-bradstreet-gpt2')" 1140 | ] 1141 | }, 1142 | { 1143 | "cell_type": "markdown", 1144 | "metadata": {}, 1145 | "source": [ 1146 | "### Optional - use a pretrained GPTJ" 1147 | ] 1148 | }, 1149 | { 1150 | "cell_type": "markdown", 1151 | "metadata": {}, 1152 | "source": [ 1153 | "Use the Endpoints notebook in this repository to deploy and test a GPT-J 6B endpoint. Compare the generation to that of your fine-tuned GPT-2 model. Add some of the lines to your poem if you like!" 1154 | ] 1155 | }, 1156 | { 1157 | "cell_type": "markdown", 1158 | "metadata": {}, 1159 | "source": [ 1160 | "### Optional- Use Other Pretrained Models and Hugging Face Datasets\n", 1161 | "\n", 1162 | "**Available pretrained models and datasets from Hugging Face**\n", 1163 | "\n", 1164 | "**Datasets**\n", 1165 | "- Shakespeare:\n", 1166 | " - https://huggingface.co/datasets/tiny_shakespeare \n", 1167 | "\n", 1168 | "**Pretrained models**\n", 1169 | "- Chinese Poetry:\n", 1170 | " - https://huggingface.co/caixin1998/chinese-poetry-gpt2 \n", 1171 | "- Hebrew Poetry:\n", 1172 | " - https://huggingface.co/Norod78/hebrew_poetry-gpt_neo-small \n", 1173 | "- Arabic Poetry:\n", 1174 | " - https://huggingface.co/akhooli/gpt2-small-arabic-poetry \n", 1175 | "- Russian Poetry:\n", 1176 | " - https://huggingface.co/TuhinColumbia/russianpoetrymany \n", 1177 | "- Persian Poetry:\n", 1178 | " - https://huggingface.co/mitra-mir/BERT-Persian-Poetry \n", 1179 | "- Italian Poetry:\n", 1180 | " - https://huggingface.co/TuhinColumbia/italianpoetrymany " 1181 | ] 1182 | }, 1183 | { 1184 | "cell_type": "markdown", 1185 | "metadata": {}, 1186 | "source": [ 1187 | "# Conclusion - Use Hugging Face and Text Generation on Amazon SageMaker for Your Organization\n", 1188 | "Now that you've learned how to test, finetune, deploy and utilize a text generation model on SageMaker, let's understand how to apply that within your organzation.\n", 1189 | "\n", 1190 | "First, think to yourself, does my organization already produce a lot of written text? Do we write documentation, scripts, blog posts, documents, answers to questions, customer messaging, etc? Odds are, you do. \n", 1191 | "\n", 1192 | "Then, ask yourself, where do we already have a large volume of written text I can easily access? That may be your existing public documentation, your existing blog posts, etc. First, run through this notebook and use some of your own data to finetune a GPT model. See how well that performs, then consider scaling to large models, including GPT-J. \n", 1193 | "\n", 1194 | "If you really aren't seeing the performance you want, [consider training a model from scratch!](https://github.com/nlp-with-transformers/notebooks/blob/main/10_transformers-from-scratch.ipynb )\n", 1195 | "\n", 1196 | "Look at this and other examples within [Hugging Face's SageMaker example notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker), and similar examples on the [SageMaker repository!](https://github.com/aws/amazon-sagemaker-examples/search?q=hugging+face)\n", 1197 | "\n", 1198 | "Remember that in order to get the best performance we **combined a variety of computer-generated and human-discriminated approaches**. Further work could improve on this by training discriminator NLP models, such as text classifiers in certain styles, to make the generation and improvement process even faster." 1199 | ] 1200 | } 1201 | ], 1202 | "metadata": { 1203 | "instance_type": "ml.t3.medium", 1204 | "kernelspec": { 1205 | "display_name": "Python 3 (PyTorch 1.8 Python 3.6 CPU Optimized)", 1206 | "language": "python", 1207 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/1.8.1-cpu-py36" 1208 | }, 1209 | "language_info": { 1210 | "codemirror_mode": { 1211 | "name": "ipython", 1212 | "version": 3 1213 | }, 1214 | "file_extension": ".py", 1215 | "mimetype": "text/x-python", 1216 | "name": "python", 1217 | "nbconvert_exporter": "python", 1218 | "pygments_lexer": "ipython3", 1219 | "version": "3.6.13" 1220 | } 1221 | }, 1222 | "nbformat": 4, 1223 | "nbformat_minor": 4 1224 | } 1225 | -------------------------------------------------------------------------------- /pipeline/hf-pipeline.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Hugging Face Deployment Pipeline\n", 8 | "\n", 9 | "This notebook creates a SageMaker Pipeline to deploy a trained Hugging Face model to a multi-model SageMaker inference endpoint.\n", 10 | "\n", 11 | "## Inputs\n", 12 | "\n", 13 | "* S3 location for model artifact\n", 14 | "\n", 15 | "## Assumptions\n", 16 | "\n", 17 | "* Endpoint is already created with appropriate inference script\n", 18 | "* Endpoint name registered in Parameter Store under key `hf-endpoint`\n", 19 | "* Test data location in S3 registered in Parameter Store under key `hf-test-data`\n", 20 | "* Multi-model storage location in S3 registered in Parameter Store under key `hf-s3-location`\n", 21 | "* This notebook's execution role needs permission to create new IAM roles and Lambda functions\n", 22 | "* This notebook's execution role needs the policy `AmazonSageMakerPipelinesIntegrations`" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "## Steps\n", 30 | "\n", 31 | "* Look up endpoint name\n", 32 | "* Look up artifact storage path\n", 33 | "* Look up test data location\n", 34 | "* Copy model to artifact storage path\n", 35 | "* Invoke endpoint with new model and return results" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "### Register pipeline input" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 138, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "from sagemaker.workflow.parameters import ParameterInteger, ParameterString\n", 52 | "\n", 53 | "input_s3_loc = ParameterString(name=\"ModelLocation\", default_value=\"s3://\")" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "### Look up endpoint name, artifact path, test data location" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 139, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "Overwriting lambda_param_store.py\n" 73 | ] 74 | } 75 | ], 76 | "source": [ 77 | "%%writefile lambda_param_store.py\n", 78 | "\n", 79 | "\"\"\"\n", 80 | "This Lambda function looks up three input parameters from Parameter Store\n", 81 | "\"\"\"\n", 82 | "\n", 83 | "import json\n", 84 | "import boto3\n", 85 | "\n", 86 | "PARAM_ENDPOINT = 'hf-endpoint'\n", 87 | "PARAM_TEST = 'hf-test-data'\n", 88 | "PARAM_ARTIFACT = 'hf-s3-location'\n", 89 | "\n", 90 | "def lambda_handler(event, context):\n", 91 | "\n", 92 | " ssm_client = boto3.client(\"ssm\")\n", 93 | " \n", 94 | " response = ssm_client.get_parameter(\n", 95 | " Name=PARAM_ENDPOINT\n", 96 | " )\n", 97 | " endpoint = response['Parameter']['Value']\n", 98 | " \n", 99 | " response = ssm_client.get_parameter(\n", 100 | " Name=PARAM_TEST\n", 101 | " )\n", 102 | " test_s3 = response['Parameter']['Value']\n", 103 | " \n", 104 | " response = ssm_client.get_parameter(\n", 105 | " Name=PARAM_ARTIFACT\n", 106 | " )\n", 107 | " artifact_s3 = response['Parameter']['Value']\n", 108 | "\n", 109 | " return {\n", 110 | " \"statusCode\": 200,\n", 111 | " \"endpoint\": json.dumps(endpoint),\n", 112 | " \"test_s3\": json.dumps(test_s3),\n", 113 | " \"artifact_s3\": json.dumps(artifact_s3)\n", 114 | " }" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 140, 120 | "metadata": {}, 121 | "outputs": [ 122 | { 123 | "name": "stdout", 124 | "output_type": "stream", 125 | "text": [ 126 | "Using ARN from existing role: lambda-deployment-role\n" 127 | ] 128 | } 129 | ], 130 | "source": [ 131 | "from iam_helper import create_lambda_role\n", 132 | "\n", 133 | "lambda_role = create_lambda_role(\"lambda-deployment-role\") " 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 141, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "from sagemaker.workflow.lambda_step import (\n", 143 | " LambdaStep,\n", 144 | " LambdaOutput,\n", 145 | " LambdaOutputTypeEnum,\n", 146 | ")\n", 147 | "from sagemaker.lambda_helper import Lambda\n", 148 | "import time\n", 149 | "\n", 150 | "# Use the current time to define unique names for the resources created\n", 151 | "current_time = time.strftime(\"%m-%d-%H-%M-%S\", time.localtime())\n", 152 | "function_name = \"sagemaker-demo-hf-lambda-step\" + current_time\n", 153 | "\n", 154 | "func = Lambda(\n", 155 | " function_name=function_name,\n", 156 | " execution_role_arn=lambda_role,\n", 157 | " script=\"lambda_param_store.py\",\n", 158 | " handler=\"lambda_param_store.lambda_handler\",\n", 159 | " timeout=30,\n", 160 | " memory_size=512,\n", 161 | ")" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 142, 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "output_param_endpoint = LambdaOutput(output_name=\"endpoint\", output_type=LambdaOutputTypeEnum.String)\n", 171 | "output_param_test_loc = LambdaOutput(output_name=\"test_s3\", output_type=LambdaOutputTypeEnum.String)\n", 172 | "output_param_artifact_loc = LambdaOutput(output_name=\"artifact_s3\", output_type=LambdaOutputTypeEnum.String)" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 143, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "step_params_lambda = LambdaStep(\n", 182 | " name=\"ReadParamsStep\",\n", 183 | " lambda_func=func,\n", 184 | " inputs={},\n", 185 | " outputs=[output_param_endpoint, output_param_test_loc, output_param_artifact_loc] )" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "### Step to copy artifact to new location" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 144, 198 | "metadata": {}, 199 | "outputs": [ 200 | { 201 | "name": "stdout", 202 | "output_type": "stream", 203 | "text": [ 204 | "Overwriting lambda_artifact.py\n" 205 | ] 206 | } 207 | ], 208 | "source": [ 209 | "%%writefile lambda_artifact.py\n", 210 | "\n", 211 | "\"\"\"\n", 212 | "This Lambda function copies a model artifact to the multi-model endpoint artifact path\n", 213 | "\"\"\"\n", 214 | "\n", 215 | "import json\n", 216 | "import boto3\n", 217 | "from urllib.parse import urlparse\n", 218 | "\n", 219 | "def lambda_handler(event, context):\n", 220 | "\n", 221 | " s3_client = boto3.client(\"s3\")\n", 222 | " \n", 223 | " input_artifact = event[\"model_path\"]\n", 224 | " artifact_path = event[\"artifact_path\"].lstrip('\"').rstrip('\"')\n", 225 | " \n", 226 | " print(\"got input artifact {0} and artifact path {1}\".format(input_artifact, artifact_path))\n", 227 | " \n", 228 | " input_p = urlparse(input_artifact)\n", 229 | " artifact_p = urlparse(artifact_path)\n", 230 | " \n", 231 | " input_key = input_p.path.lstrip('/')\n", 232 | " input_bucket = input_p.netloc\n", 233 | " input_name = input_p.path.split('/')[-1]\n", 234 | " \n", 235 | " print(\"Input key = {0}, input bucket = {1}, input name = {2}\".format(input_key, input_bucket, input_name))\n", 236 | " \n", 237 | " artifact_key = artifact_p.path.lstrip('/') + input_name\n", 238 | " artifact_bucket = artifact_p.netloc\n", 239 | " \n", 240 | " print(\"Artifact key = {0}, artifact bucket = {1}\".format(artifact_key, artifact_bucket))\n", 241 | " \n", 242 | " copy_source = {\n", 243 | " 'Bucket': input_bucket,\n", 244 | " 'Key': input_key\n", 245 | " }\n", 246 | " s3_client.copy(copy_source, artifact_bucket, artifact_key)\n", 247 | " print(\"Copying {0} to {1}/{2}\".format(json.dumps(copy_source), artifact_bucket, artifact_key))\n", 248 | "\n", 249 | " return {\n", 250 | " \"statusCode\": 200,\n", 251 | " \"model_key\": input_name\n", 252 | " }" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": 145, 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "output_model_loc = LambdaOutput(output_name=\"model_key\", output_type=LambdaOutputTypeEnum.String)" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 146, 267 | "metadata": {}, 268 | "outputs": [], 269 | "source": [ 270 | "current_time = time.strftime(\"%m-%d-%H-%M-%S\", time.localtime())\n", 271 | "function_name_2 = \"sagemaker-demo-hf-lambda-step-2\" + current_time\n", 272 | "func2 = Lambda(\n", 273 | " function_name=function_name_2,\n", 274 | " execution_role_arn=lambda_role,\n", 275 | " script=\"lambda_artifact.py\",\n", 276 | " handler=\"lambda_artifact.lambda_handler\",\n", 277 | " timeout=600,\n", 278 | " memory_size=512,\n", 279 | ")" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": 147, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [ 288 | "step_artifact_lambda = LambdaStep(\n", 289 | " name=\"CopyArtifactStep\",\n", 290 | " lambda_func=func2,\n", 291 | " inputs={\n", 292 | " \"model_path\": input_s3_loc,\n", 293 | " \"artifact_path\": output_param_artifact_loc\n", 294 | " },\n", 295 | " outputs=[output_model_loc] )" 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "### Invoke endpoint with new model" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": 148, 308 | "metadata": {}, 309 | "outputs": [ 310 | { 311 | "name": "stdout", 312 | "output_type": "stream", 313 | "text": [ 314 | "Overwriting lambda_test.py\n" 315 | ] 316 | } 317 | ], 318 | "source": [ 319 | "%%writefile lambda_test.py\n", 320 | "\n", 321 | "\"\"\"\n", 322 | "This Lambda function invokes a specific model on a multi-model endpoint\n", 323 | "\"\"\"\n", 324 | "\n", 325 | "import json\n", 326 | "import boto3\n", 327 | "from urllib.parse import urlparse\n", 328 | "\n", 329 | "TMP_FILE = '/tmp/body.txt'\n", 330 | "\n", 331 | "def lambda_handler(event, context):\n", 332 | "\n", 333 | " sm_client = boto3.client(\"sagemaker-runtime\")\n", 334 | " s3_client = boto3.client(\"s3\")\n", 335 | " \n", 336 | " endpoint = event[\"endpoint\"].lstrip('\"').rstrip('\"')\n", 337 | " model = event[\"model\"]\n", 338 | " test = event[\"test\"].lstrip('\"').rstrip('\"')\n", 339 | " \n", 340 | " print(\"endpoint = {0}, model = {1}, test = {2}\".format(endpoint, model, test))\n", 341 | " \n", 342 | " test_p = urlparse(test)\n", 343 | " \n", 344 | " s3_client.download_file(test_p.netloc, test_p.path.lstrip('/'), TMP_FILE)\n", 345 | " \n", 346 | " with open(TMP_FILE, 'r') as F:\n", 347 | " input_data = {'input': F.read() }\n", 348 | " print(f\"{input_data}\")\n", 349 | " response = sm_client.invoke_endpoint(\n", 350 | " EndpointName = endpoint,\n", 351 | " ContentType = \"application/json\",\n", 352 | " TargetModel = model,\n", 353 | " Body = json.dumps(input_data).encode('utf-8'))\n", 354 | " output = response['Body'].read().decode(\"utf-8\")\n", 355 | "\n", 356 | " return {\n", 357 | " \"statusCode\": 200,\n", 358 | " \"inference\": output\n", 359 | " }" 360 | ] 361 | }, 362 | { 363 | "cell_type": "code", 364 | "execution_count": 149, 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "current_time = time.strftime(\"%m-%d-%H-%M-%S\", time.localtime())\n", 369 | "function_name_3 = \"sagemaker-demo-hf-lambda-step-3\" + current_time\n", 370 | "func3 = Lambda(\n", 371 | " function_name=function_name_3,\n", 372 | " execution_role_arn=lambda_role,\n", 373 | " script=\"lambda_test.py\",\n", 374 | " handler=\"lambda_test.lambda_handler\",\n", 375 | " timeout=600,\n", 376 | " memory_size=1024,\n", 377 | ")" 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 150, 383 | "metadata": {}, 384 | "outputs": [], 385 | "source": [ 386 | "output_test = LambdaOutput(output_name=\"inference\", output_type=LambdaOutputTypeEnum.String)" 387 | ] 388 | }, 389 | { 390 | "cell_type": "code", 391 | "execution_count": 151, 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "step_test_lambda = LambdaStep(\n", 396 | " name=\"TestStep\",\n", 397 | " lambda_func=func3,\n", 398 | " inputs={\n", 399 | " \"endpoint\": output_param_endpoint,\n", 400 | " \"model\": output_model_loc,\n", 401 | " \"test\": output_param_test_loc,\n", 402 | " },\n", 403 | " outputs=[output_test] )" 404 | ] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "metadata": {}, 409 | "source": [ 410 | "### Define Pipeline" 411 | ] 412 | }, 413 | { 414 | "cell_type": "code", 415 | "execution_count": 152, 416 | "metadata": {}, 417 | "outputs": [], 418 | "source": [ 419 | "from sagemaker.workflow.pipeline import Pipeline\n", 420 | "import sagemaker\n", 421 | "import boto3\n", 422 | "\n", 423 | "region = sagemaker.Session().boto_region_name\n", 424 | "sm_client = boto3.client(\"sagemaker\")\n", 425 | "boto_session = boto3.Session(region_name=region)\n", 426 | "sagemaker_session = sagemaker.session.Session(\n", 427 | " boto_session=boto_session, sagemaker_client=sm_client\n", 428 | ")\n", 429 | "\n", 430 | "pipeline = Pipeline(\n", 431 | " name=f\"HuggingFaceDeployPipeline\",\n", 432 | " parameters=[\n", 433 | " input_s3_loc\n", 434 | " ],\n", 435 | " steps=[step_params_lambda, step_artifact_lambda, step_test_lambda],\n", 436 | " sagemaker_session=sagemaker_session,\n", 437 | ")" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": 153, 443 | "metadata": {}, 444 | "outputs": [ 445 | { 446 | "data": { 447 | "text/plain": [ 448 | "{'Version': '2020-12-01',\n", 449 | " 'Metadata': {},\n", 450 | " 'Parameters': [{'Name': 'ModelLocation',\n", 451 | " 'Type': 'String',\n", 452 | " 'DefaultValue': 's3://'}],\n", 453 | " 'PipelineExperimentConfig': {'ExperimentName': {'Get': 'Execution.PipelineName'},\n", 454 | " 'TrialName': {'Get': 'Execution.PipelineExecutionId'}},\n", 455 | " 'Steps': [{'Name': 'ReadParamsStep',\n", 456 | " 'Type': 'Lambda',\n", 457 | " 'Arguments': {},\n", 458 | " 'FunctionArn': 'arn:aws:lambda:us-east-1:752304587005:function:sagemaker-demo-hf-lambda-step02-24-00-34-23',\n", 459 | " 'OutputParameters': [{'OutputName': 'endpoint', 'OutputType': 'String'},\n", 460 | " {'OutputName': 'test_s3', 'OutputType': 'String'},\n", 461 | " {'OutputName': 'artifact_s3', 'OutputType': 'String'}]},\n", 462 | " {'Name': 'CopyArtifactStep',\n", 463 | " 'Type': 'Lambda',\n", 464 | " 'Arguments': {'model_path': {'Get': 'Parameters.ModelLocation'},\n", 465 | " 'artifact_path': {'Get': \"Steps.ReadParamsStep.OutputParameters['artifact_s3']\"}},\n", 466 | " 'FunctionArn': 'arn:aws:lambda:us-east-1:752304587005:function:sagemaker-demo-hf-lambda-step-202-24-00-34-26',\n", 467 | " 'OutputParameters': [{'OutputName': 'model_key', 'OutputType': 'String'}]},\n", 468 | " {'Name': 'TestStep',\n", 469 | " 'Type': 'Lambda',\n", 470 | " 'Arguments': {'endpoint': {'Get': \"Steps.ReadParamsStep.OutputParameters['endpoint']\"},\n", 471 | " 'model': {'Get': \"Steps.CopyArtifactStep.OutputParameters['model_key']\"},\n", 472 | " 'test': {'Get': \"Steps.ReadParamsStep.OutputParameters['test_s3']\"}},\n", 473 | " 'FunctionArn': 'arn:aws:lambda:us-east-1:752304587005:function:sagemaker-demo-hf-lambda-step-302-24-00-34-28',\n", 474 | " 'OutputParameters': [{'OutputName': 'inference', 'OutputType': 'String'}]}]}" 475 | ] 476 | }, 477 | "execution_count": 153, 478 | "metadata": {}, 479 | "output_type": "execute_result" 480 | } 481 | ], 482 | "source": [ 483 | "import json\n", 484 | "\n", 485 | "json.loads(pipeline.definition())" 486 | ] 487 | }, 488 | { 489 | "cell_type": "code", 490 | "execution_count": 154, 491 | "metadata": {}, 492 | "outputs": [ 493 | { 494 | "data": { 495 | "text/plain": [ 496 | "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:752304587005:pipeline/huggingfacedeploypipeline',\n", 497 | " 'ResponseMetadata': {'RequestId': 'c5a8607b-dc5e-4160-b0b3-7936722bf9ab',\n", 498 | " 'HTTPStatusCode': 200,\n", 499 | " 'HTTPHeaders': {'x-amzn-requestid': 'c5a8607b-dc5e-4160-b0b3-7936722bf9ab',\n", 500 | " 'content-type': 'application/x-amz-json-1.1',\n", 501 | " 'content-length': '93',\n", 502 | " 'date': 'Thu, 24 Feb 2022 00:34:32 GMT'},\n", 503 | " 'RetryAttempts': 0}}" 504 | ] 505 | }, 506 | "execution_count": 154, 507 | "metadata": {}, 508 | "output_type": "execute_result" 509 | } 510 | ], 511 | "source": [ 512 | "role = sagemaker.get_execution_role()\n", 513 | "\n", 514 | "pipeline.upsert(role_arn=role)" 515 | ] 516 | }, 517 | { 518 | "cell_type": "markdown", 519 | "metadata": {}, 520 | "source": [ 521 | "### Execute pipeline" 522 | ] 523 | }, 524 | { 525 | "cell_type": "code", 526 | "execution_count": 155, 527 | "metadata": {}, 528 | "outputs": [], 529 | "source": [ 530 | "#execution = pipeline.start(parameters = {\"ModelLocation\": \"s3://sagemaker-us-east-1-<>/mmtest/m2.tar.gz\"})" 531 | ] 532 | }, 533 | { 534 | "cell_type": "code", 535 | "execution_count": 173, 536 | "metadata": {}, 537 | "outputs": [], 538 | "source": [ 539 | "execution.wait()" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": null, 545 | "metadata": {}, 546 | "outputs": [], 547 | "source": [] 548 | } 549 | ], 550 | "metadata": { 551 | "instance_type": "ml.t3.medium", 552 | "kernelspec": { 553 | "display_name": "Python 3 (Data Science)", 554 | "language": "python", 555 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" 556 | }, 557 | "language_info": { 558 | "codemirror_mode": { 559 | "name": "ipython", 560 | "version": 3 561 | }, 562 | "file_extension": ".py", 563 | "mimetype": "text/x-python", 564 | "name": "python", 565 | "nbconvert_exporter": "python", 566 | "pygments_lexer": "ipython3", 567 | "version": "3.7.10" 568 | } 569 | }, 570 | "nbformat": 4, 571 | "nbformat_minor": 4 572 | } 573 | -------------------------------------------------------------------------------- /pipeline/iam_helper.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | 4 | iam = boto3.client('iam') 5 | 6 | def create_lambda_role(role_name): 7 | try: 8 | response = iam.create_role( 9 | RoleName = role_name, 10 | AssumeRolePolicyDocument = json.dumps({ 11 | "Version": "2012-10-17", 12 | "Statement": [ 13 | { 14 | "Effect": "Allow", 15 | "Principal": { 16 | "Service": "lambda.amazonaws.com" 17 | }, 18 | "Action": "sts:AssumeRole" 19 | } 20 | ] 21 | }), 22 | Description='Role for Lambda to call SSM' 23 | ) 24 | 25 | role_arn = response['Role']['Arn'] 26 | 27 | response = iam.attach_role_policy( 28 | RoleName=role_name, 29 | PolicyArn='arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' 30 | ) 31 | 32 | response = iam.attach_role_policy( 33 | PolicyArn='arn:aws:iam::aws:policy/AmazonSageMakerFullAccess', 34 | RoleName=role_name 35 | ) 36 | 37 | response = iam.attach_role_policy( 38 | PolicyArn='arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess', 39 | RoleName=role_name 40 | ) 41 | 42 | response = iam.attach_role_policy( 43 | PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess', 44 | RoleName=role_name 45 | ) 46 | 47 | return role_arn 48 | 49 | except iam.exceptions.EntityAlreadyExistsException: 50 | print(f'Using ARN from existing role: {role_name}') 51 | response = iam.get_role(RoleName=role_name) 52 | return response['Role']['Arn'] 53 | -------------------------------------------------------------------------------- /use_cases/Pick_the_right_HF_model.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Find the right use case, model, and feature for Hugging Face on SageMaker \n", 8 | "If you are looking for helping finding the right use case and model for your data science project with Hugging Face on SageMaker and AWS, you've come to the right place! \n", 9 | "\n", 10 | "### A few questions to ask yourself:\n", 11 | "1. Where can I apply my NLP project to provide the highest value to my business, while doing this in a low risk, secure way?\n", 12 | "2. Do I already have a relevant dataset?\n", 13 | "3. If I do, is this primarily in English, or do I need multilingual?\n", 14 | "4. Does my domain include specialized terminology, words that wouldn't be common on Wikipedia?\n", 15 | "5. For whichever model seems appropriate, what dataset does it primarily use? What objective metric does this model report on this dataset? Am I comfortable with that level of performance in the use case I am consdiering targeting.\n", 16 | "\n", 17 | "Use your own anwers to those questions to determine which use case, and which model(s), will be most impactful to your teams. Data science projects commonly start with some amount of early exploration, anywhere from 1 day to a few weeks, to determine if a given approach is viable. \n", 18 | "\n", 19 | "\n", 20 | "**Pro tip** - timebox your experiments, be frugal with your teams' time, and move quickly to find a viable solution. Only after you have some quantitative evidence that a given model is working well on your dataset sample is it time to scale up to larger datasets, larger models, larger training, and larger hosting clusters.\n", 21 | "\n", 22 | "---\n", 23 | "\n", 24 | "*Please note: When you pick any pretrained model, make sure that model has been tested on a dataset. Take some time to research what objective metric that model is reporting, usually Hugging Face will report this directly in the model card. Ask yourself - how similar is this research dataset to my own dataset? Am I ok with this level of performance in my use case? How can I design my overall ML solution to accomodate this?*\n", 25 | "\n", 26 | "---" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": {}, 32 | "source": [ 33 | "# Use Cases and Models" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "### Text Classification\n", 41 | "\n", 42 | "Top Use Cases\n", 43 | "- Spam filter\n", 44 | "- Support ticket routing\n", 45 | "- Chat bot analysis\n", 46 | "- Tagging customer feedback and product reviews\n", 47 | "- Social media analysis\n", 48 | "\n", 49 | "\n", 50 | " Top Models \n", 51 | "- [Search engine result optimization](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2)\n", 52 | "- [Sentiment detection](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n", 53 | "- [Multi-lingual](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "### Named Entity Recgonition\n", 61 | "Top Models\n", 62 | "\n", 63 | "- [Extract named entities (nouns) with XLM](https://huggingface.co/xlm-roberta-large-finetuned-conll03-english)\n", 64 | "- Some non-English languages:\n", 65 | " - [German](https://huggingface.co/flair/ner-german)\n", 66 | " - [French](https://huggingface.co/flair/ner-french)\n", 67 | " - [Dutch](https://huggingface.co/flair/ner-dutch )\n", 68 | "- [Finetune NER for disease related languages, with PubMed](https://huggingface.co/beatrice-portelli/DiLBERT )" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "### Question Answering\n", 76 | "Top Models\n", 77 | "- [RoBERTa base Squad 2](https://huggingface.co/deepset/roberta-base-squad2 )\n", 78 | "- [BERT large uncased whole word masking](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad) \n", 79 | "- [BioBERT](https://huggingface.co/dmis-lab/biobert-large-cased-v1.1-squad )" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "### Text Summarization\n", 87 | "\n", 88 | "Top Models\n", 89 | "- [DistiliBart](https://huggingface.co/sshleifer/distilbart-cnn-6-6)\n", 90 | "- [MT5 Multilingual](https://huggingface.co/csebuetnlp/mT5_multilingual_XLSum)\n", 91 | "- [Financial Summarization](https://huggingface.co/human-centered-summarization/financial-summarization-pegasus )" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "### Text Generation\n", 99 | "Top Models\n", 100 | "- [Distiligpt2](https://huggingface.co/distilgpt2)\n", 101 | "- [GPT2](https://huggingface.co/gpt2 )\n", 102 | "- [EleutherAI GPT J 6B](https://huggingface.co/EleutherAI/gpt-j-6B)" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "### Translation\n", 110 | "Top Models\n", 111 | "- [T5 Small](https://huggingface.co/t5-small)\n", 112 | "- [Helsinki NLP's Opus MT / ZH / EN](https://huggingface.co/Helsinki-NLP/opus-mt-zh-en)\n", 113 | "- [T5 3b](https://huggingface.co/t5-3b)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "### Computer vision\n", 121 | "Top Models\n", 122 | "- [Image Classification](https://huggingface.co/google/vit-base-patch16-224)\n", 123 | "- [Image Segmentation](https://huggingface.co/facebook/detr-resnet-50-panoptic)\n", 124 | "- [Object detection](https://huggingface.co/facebook/detr-resnet-50)\n", 125 | "- [Text to image](https://huggingface.co/flax-community/dalle-mini)\n", 126 | "- [Image to text](https://huggingface.co/kha-white/manga-ocr-base)" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "---\n", 134 | "# Frequently Asked Questions" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "#### 1. Do I need a big model or a small model? Does size matter in NLP models?\n", 142 | "Generally, yes, size is very significant in NLP models. Recently researchers have developed techniques to deliver accuracy gains for even smaller model sizes, but the general rule stands. A model with more paramters (ie, larger) will on average be more accurate than a model with smaller parameters. So to answer this question, first ask yourself, where am I in my experimentation process? If you are very early, then please start with a small model. If you are later, then consider using a larger one." 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "#### 2. Can I use the pretrained model as it is, or do I need to fine tune it?\n", 150 | "The answer to this question strictly depends on how similar your dataset is to the dataset the model was trained against. For example, if the domain you are evaluating is primarily in English, with non domain-specific terminology, using casual English as found in Wikipedia, then you may be fine with hosting the model as it for your use case. However, if your dataset is very different from the one the model was trained on, then you may want to consider fine tuning. Generally speaking, fine-tuned models are expected to be more accurate than generic models. This means the only reason you'd deploy a pretrained model directly is if (a) your dataset is almost exactly identical to what the model was trained on, and/or (b) the value of deploying it quickly significantly outweights the time it would take to fine tune it. Pro tip - fine tuning really is not very difficult at all, unless you are dealling with larger scale datasets." 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "#### 3. What if I need to train a model from scratch? Can I do that on SageMaker too?\n", 158 | "Typically you'd only train a model from scratch if the pretrained models are so completely different from your dataset that even large-scale fine tuning won't deliver the performance you need. This may be the case for new languages, new domains, or new syntaxes that vary dramatically from Wikipedia or other Common Crawl samples. If you do need to train a model from scratch, yes absolutely you can do this on SageMaker. Typically customers look at our managed distributed training libraries, like model and data parallel, for the absolute best performance, but you can also bring 3rd party distributed frameworks to scale. Anticipate jobs that run for hours to days to weeks, and cluster sizes ranging from 10s to 100s to 1000s of GPUS or custom accelerators." 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "#### 4. Aren't NLP models biased? How do I navigate that?\n", 166 | "Yes! Please assume that any biases inherent in the dataset the model was trained on will be carried into the model itself. This is commonly visibile when comparing across social traits, such as gender and employment, or countless other examples. For this reason some teams layer on additional \"sensitivity\" models that look for red flags in text, and route it to manual review prior to exposing to customers. You can [detect and monitor for bias in text classifiers using SageMaker Clarify.]( https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/sagemaker_processing/fairness_and_explainability/text_explainability/text_explainability.ipynb)" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "#### 5. Should I use PyTorch or TensorFlow? Do I need to write my own model? What if I want to write my own model?\n", 174 | "Actually the Hugging Face SDK does an excellent job making it extremely easy to switch back and forth between PyTorch and TensorFlow. You'll need to have at least one of those backends installed wherever you are running the Transformers SDK, but that choice is entirely on you. Additionally, Hugging Face absolutely supports defining your own models. You are welcome to bring your own networks in both TensorFlow (Keras) and PyTorch, and use the rest of their functionality. On top of that, HF also lets you convert models from one framework to another. So if a particular model you are evaluating is available only in 1 framework, don't fear, you can just use their SDK to convert it into the one of your choice.\n" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": {}, 180 | "source": [ 181 | "---\n", 182 | "# Hugging Face on SageMaker Examples" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "#### Example notebooks x 46 built by Hugging Face and AWS\n", 190 | "\n", 191 | "1. Getting started\n", 192 | " - [Train and host with PyTorch](https://github.com/huggingface/notebooks/blob/master/sagemaker/01_getting_started_pytorch/sagemaker-notebook.ipynb)\n", 193 | " - [Train and host with TensorFlow](https://github.com/huggingface/notebooks/blob/master/sagemaker/02_getting_started_tensorflow/sagemaker-notebook.ipynb)\n", 194 | "\n", 195 | "2. SageMaker Training\n", 196 | " - [Training Compiler](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-training-compiler/huggingface)\n", 197 | " - [Data parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/data_parallel/bert/pytorch_smdataparallel_bert_demo.ipynb)\n", 198 | " - [Model parallel](https://github.com/aws/amazon-sagemaker-examples/blob/main/training/distributed_training/pytorch/model_parallel/gpt-j/train_gptj_smp_notebook.ipynb)\n", 199 | " - [Hyperparameter tuning](https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/hyperparameter_tuning/huggingface_multiclass_text_classification_20_newsgroups/hpo_huggingface_text_classification_20_newsgroups.ipynb)\n", 200 | " - [Spot instances](https://github.com/huggingface/notebooks/blob/master/sagemaker/05_spot_instances/sagemaker-notebook.ipynb)\n", 201 | "\n", 202 | "3. SageMaker Hosting\n", 203 | " - [Deploy a pretrained model from the HF hub](https://github.com/huggingface/notebooks/tree/master/sagemaker/11_deploy_model_from_hf_hub)\n", 204 | " - [Deploy a finetuned model from s3](https://github.com/huggingface/notebooks/blob/master/sagemaker/10_deploy_model_from_s3/deploy_transformer_model_from_s3.ipynb)\n", 205 | " - [Deploy and autoscale](https://github.com/huggingface/notebooks/blob/master/sagemaker/13_deploy_and_autoscaling_transformers/sagemaker-notebook.ipynb) \n", 206 | " - [Serverless Inference](https://github.com/aws/amazon-sagemaker-examples/blob/35723de714e95d6152bfb1ff4c141fc069e37b8a/serverless-inference/huggingface-serverless-inference/huggingface-text-classification-serverless-inference.ipynb)\n", 207 | " - [Asynchronous Inference](https://github.com/huggingface/notebooks/blob/master/sagemaker/16_async_inference_hf_hub/sagemaker-notebook.ipynb)\n", 208 | "\n", 209 | "4. Operationalize and optimize\n", 210 | " - [Pipelines](https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/end_to_end/nlp_mlops_company_sentiment/02_nlp_company_earnings_analysis_pipeline.ipynb)\n", 211 | " - [Inference recommender](https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/sagemaker-inference-recommender/huggingface-inference-recommender/huggingface-inference-recommender.ipynb)\n", 212 | " - [SageMaker Neo and Inferentia](https://github.com/aws/amazon-sagemaker-examples/blob/a26b5c4e6a81c714707244d2407305984c584f51/sagemaker_neo_compilation_jobs/deploy_huggingface_model_on_Inf1_instance/inf1_bert_compile_and_deploy.ipynb)\n", 213 | " - [SageMaker Clarify](https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/sagemaker_processing/fairness_and_explainability/text_explainability/text_explainability.ipynb)\n", 214 | " - [NVIDIA Triton Server on SageMaker](https://github.com/aws/amazon-sagemaker-examples/blob/4b9615508a08b474d7a89827cb21d67c8f12ad1c/sagemaker-triton/nlp_bert/triton_nlp_bert.ipynb)" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 4, 220 | "metadata": {}, 221 | "outputs": [ 222 | { 223 | "name": "stdout", 224 | "output_type": "stream", 225 | "text": [ 226 | " " 227 | ] 228 | } 229 | ], 230 | "source": [ 231 | "" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [] 240 | } 241 | ], 242 | "metadata": { 243 | "instance_type": "ml.t3.medium", 244 | "kernelspec": { 245 | "display_name": "Python 3 (Data Science)", 246 | "language": "python", 247 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" 248 | }, 249 | "language_info": { 250 | "codemirror_mode": { 251 | "name": "ipython", 252 | "version": 3 253 | }, 254 | "file_extension": ".py", 255 | "mimetype": "text/x-python", 256 | "name": "python", 257 | "nbconvert_exporter": "python", 258 | "pygments_lexer": "ipython3", 259 | "version": "3.7.10" 260 | } 261 | }, 262 | "nbformat": 4, 263 | "nbformat_minor": 4 264 | } 265 | -------------------------------------------------------------------------------- /without_cfn/1_deploy_gpt2_gptj_mme.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Deploying GPT-2 and GPT-J" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In this notebook, we will be using Hugging Face models and SageMaker Hugging Face-specific API's to deploy both GPT-2 and GPT-J. We will also showcase how to deploy what would could be GPT2 models fine-tuned on different datasets to the same SageMaker instance as a Multi Model Endpoint. This will allow you to get real-time predictions from several models, while only paying for one running endpoint instance." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "*****\n", 22 | "## Deploying GTP-2 to SageMaker Multi-Model Endpoint" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "!pip install -U transformers\n", 32 | "!pip install -U sagemaker" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "### Get sagemaker session, role and default bucket\n", 40 | "If you are going to use Sagemaker in a local environment (not SageMaker Studio or Notebook Instances), you need access to an IAM Role with the required permissions for Sagemaker. You can find more about this [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html)." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "import sagemaker\n", 50 | "import boto3\n", 51 | "\n", 52 | "sess = sagemaker.Session()\n", 53 | "# sagemaker session bucket -> used for uploading data, models and logs\n", 54 | "# sagemaker will automatically create this bucket if it not exists\n", 55 | "sagemaker_session_bucket=None\n", 56 | "if sagemaker_session_bucket is None and sess is not None:\n", 57 | " # set to default bucket if a bucket name is not given\n", 58 | " sagemaker_session_bucket = sess.default_bucket()\n", 59 | "\n", 60 | "try:\n", 61 | " role = sagemaker.get_execution_role()\n", 62 | "except ValueError:\n", 63 | " iam = boto3.client('iam')\n", 64 | " role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n", 65 | "\n", 66 | "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n", 67 | "region = sess.boto_region_name\n", 68 | "sm_client = boto3.client('sagemaker')\n", 69 | "\n", 70 | "print(f\"sagemaker role arn: {role}\")\n", 71 | "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n", 72 | "print(f\"sagemaker session region: {region}\")" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "### Load GPT-2 model and tokenizer, save them to the same folder with Transformers `save_pretrained` utility " 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "import transformers \n", 89 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 90 | "\n", 91 | "tokenizer = AutoTokenizer.from_pretrained('gpt2')\n", 92 | "model = AutoModelForCausalLM.from_pretrained('gpt2')\n", 93 | "\n", 94 | "model.save_pretrained('gpt2-model/')\n", 95 | "tokenizer.save_pretrained('gpt2-model/')" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "### Tar model and tokenizer artifacts, upload to S3" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "import tarfile \n", 112 | "\n", 113 | "with tarfile.open('gpt2-model.tar.gz', 'w:gz') as f:\n", 114 | " f.add('gpt2-model/',arcname='.')\n", 115 | "f.close()\n", 116 | "\n", 117 | "prefix = 'gpt2-hf-workshop/gpt2-test'" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "Check out the file contents and structure of the model.tar.gz artifact." 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": {}, 131 | "outputs": [], 132 | "source": [ 133 | "! tar -ztvf gpt2-model.tar.gz" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "We will upload the same model package twice with different names, to simulate deploying 2 models to the same endpoint." 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "! aws s3 cp gpt2-model.tar.gz s3://\"$sagemaker_session_bucket\"/\"$prefix\"/gpt2-model1.tar.gz\n", 150 | "! aws s3 cp gpt2-model.tar.gz s3://\"$sagemaker_session_bucket\"/\"$prefix\"/gpt2-model2.tar.gz" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "### Get image URI for Hugging Face inference Deep Learning Container" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "from sagemaker import image_uris\n", 167 | "\n", 168 | "hf_inference_dlc = image_uris.retrieve(framework='huggingface', \n", 169 | " region=region, \n", 170 | " version='4.12.3', \n", 171 | " image_scope='inference', \n", 172 | " base_framework_version='pytorch1.9.1', \n", 173 | " py_version='py38', \n", 174 | " container_version='ubuntu20.04', \n", 175 | " instance_type='ml.m5.xlarge')" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "### Use `MultiDataModel`to setup a multi-model endpoint definition\n", 183 | "By setting the `HF_TASK` environment variable, we avoid having to write and test our own inference code. Depending on the task and model you choose, the Hugging Face inference Container will run the appropriate code by default. " 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "from sagemaker.multidatamodel import MultiDataModel\n", 193 | "from sagemaker.predictor import Predictor\n", 194 | "\n", 195 | "hub = {\n", 196 | " 'HF_TASK':'text-generation'\n", 197 | "}\n", 198 | "\n", 199 | "mme = MultiDataModel(\n", 200 | " name='gpt2-models',\n", 201 | " model_data_prefix=f's3://{sagemaker_session_bucket}/{prefix}/',\n", 202 | " image_uri=hf_inference_dlc,\n", 203 | " env=hub,\n", 204 | " predictor_cls=Predictor,\n", 205 | " role=role,\n", 206 | " sagemaker_session=sess,\n", 207 | " )" 208 | ] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": {}, 213 | "source": [ 214 | "We can see that our model object has already \"registered\" the model artifacts we uploaded to S3 under the `model_data_prefix`." 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "for model in mme.list_models():\n", 224 | " print(model)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "### Deploy Multi-Model Endpoint and send inference requests to both models" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "import datetime\n", 241 | "from sagemaker.serializers import JSONSerializer\n", 242 | "from sagemaker.deserializers import JSONDeserializer\n", 243 | "\n", 244 | "endpoint_name_gpt2 = 'mme-gpt2-'+datetime.datetime.now().strftime(\n", 245 | " \"%Y-%m-%d-%H-%M-%S\"\n", 246 | ")\n", 247 | "\n", 248 | "predictor_gpt2 = mme.deploy(\n", 249 | " initial_instance_count=1,\n", 250 | " instance_type='ml.m5.xlarge',\n", 251 | " serializer=JSONSerializer(),\n", 252 | " deserializer=JSONDeserializer(),\n", 253 | " endpoint_name='mme-gpt2',\n", 254 | " wait = False\n", 255 | " )" 256 | ] 257 | }, 258 | { 259 | "cell_type": "markdown", 260 | "metadata": {}, 261 | "source": [ 262 | "\n", 263 | "********************************************************************************************************************************************\n", 264 | "********************************************************************************************************************************************\n", 265 | "\n", 266 | "\n", 267 | "# Deploying GPT-J to SageMaker Endpoint" 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": null, 273 | "metadata": {}, 274 | "outputs": [], 275 | "source": [ 276 | "from sagemaker.huggingface import HuggingFaceModel\n", 277 | "import sagemaker\n", 278 | "\n", 279 | "role = sagemaker.get_execution_role()\n", 280 | "# Hub Model configuration. https://huggingface.co/models\n", 281 | "hub = {\n", 282 | "\t'HF_MODEL_ID':'EleutherAI/gpt-j-6B',\n", 283 | "\t'HF_TASK':'text-generation'\n", 284 | "}\n", 285 | "\n", 286 | "# create Hugging Face Model Class\n", 287 | "huggingface_model = HuggingFaceModel(\n", 288 | "\ttransformers_version='4.6.1',\n", 289 | "\tpytorch_version='1.7.1',\n", 290 | "\tpy_version='py36',\n", 291 | "\tenv=hub,\n", 292 | "\trole=role, \n", 293 | ")\n", 294 | "\n", 295 | "# deploy model to SageMaker Inference\n", 296 | "predictor = huggingface_model.deploy(\n", 297 | "\tinitial_instance_count=1, # number of instances\n", 298 | "\tinstance_type='ml.m5.xlarge',\n", 299 | " wait = False# ec2 instance type\n", 300 | ")\n", 301 | "\n" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": null, 307 | "metadata": {}, 308 | "outputs": [], 309 | "source": [ 310 | "# predictor.predict({\n", 311 | "# \t'inputs': \"Can you please let us know more details about your \"\n", 312 | "# })" 313 | ] 314 | } 315 | ], 316 | "metadata": { 317 | "instance_type": "ml.t3.medium", 318 | "kernelspec": { 319 | "display_name": "Python 3 (PyTorch 1.8 Python 3.6 CPU Optimized)", 320 | "language": "python", 321 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/1.8.1-cpu-py36" 322 | }, 323 | "language_info": { 324 | "codemirror_mode": { 325 | "name": "ipython", 326 | "version": 3 327 | }, 328 | "file_extension": ".py", 329 | "mimetype": "text/x-python", 330 | "name": "python", 331 | "nbconvert_exporter": "python", 332 | "pygments_lexer": "ipython3", 333 | "version": "3.6.13" 334 | } 335 | }, 336 | "nbformat": 4, 337 | "nbformat_minor": 4 338 | } 339 | -------------------------------------------------------------------------------- /without_cfn/2_generate_text_with_gpt2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Text generation with Pretrained GPT2 models from Hugging Face on Amazon SageMaker\n", 8 | "## The Poetry of NLP\n", 9 | "\n", 10 | "You’ve just been hired by the Chicago Tribune to start a new poetry column. Congrats! The catch? You need to write a new poem every day. And it can’t just be any old string of syllables, you need it to be fresh, authentic, to resonate with the times and carry a sense of rhyme. You need it to delight your readers, to drive up the Tribune’s daily readership numbers and grow their social media presence. How are you going to accomplish this? With the help of Hugging Face and NLP models on SageMaker of course! " 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "#### In this notebook, we'll execute the following steps.\n", 18 | "\n", 19 | "1. Use the Hugging Face transformfers SDK to download pretrained NLP models and test them locally.\n", 20 | "2. Select a dataset from among our favorite authors.\n", 21 | "3. Finetune the pretrained model using SageMaker training.\n", 22 | "4. Deploy the model into S3.\n", 23 | "5. Trigger a pipeline to test and deploy the model onto a multi-container endpoint.\n", 24 | "6. Test your multi-model endpoint locally to write poetry and text in the style of your favorite authors. " 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "Please note, this notebook was built on SageMaker Studio, using an ml.t3.medium kernel gatway application, and the Python 3.6 PyTorch 1.8 CPU Jupyter Kernel." 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "### Step 0. Install the transformers SDK locally." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "%%writefile requirements.txt \n", 48 | "\n", 49 | "transformers==4.6.1" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [ 58 | "!pip install -r requirements.txt" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": {}, 64 | "source": [ 65 | "### Step 1. Download a pretrained GPT2 model and test locally.\n", 66 | "We're using the Transformers SDK syntax available on the model card here: https://huggingface.co/gpt2 \n", 67 | "\n", 68 | "To make this model even better, we'll use a version of GPT2 that **has already been finetuned to generate poetry!**" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 78 | "\n", 79 | "poem_gpt = \"ismaelfaro/gpt2-poems.en\"\n", 80 | "\n", 81 | "tokenizer = AutoTokenizer.from_pretrained(poem_gpt)\n", 82 | "\n", 83 | "base_model = AutoModelForCausalLM.from_pretrained(poem_gpt)" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "from transformers import set_seed\n", 93 | "\n", 94 | "def get_outputs(sample_outputs, tokenizer):\n", 95 | " # takes a tokenizer, and raw output from the model, decodes these and formats nicely\n", 96 | " rt = []\n", 97 | "\n", 98 | " print(\"Output:\\n\" + 100 * '-')\n", 99 | " for i, sample_output in enumerate(sample_outputs):\n", 100 | " txt = tokenizer.decode(sample_output, skip_special_tokens = True)\n", 101 | " print(\"{}: {}...\".format(i, txt))\n", 102 | " print('')\n", 103 | " rt.append(txt)\n", 104 | " \n", 105 | " return rt\n", 106 | "\n", 107 | "# setting the seed helps us ensure reproducibility. when the seed is consistent, we know the model results will be consistent\n", 108 | "set_seed(42)\n", 109 | "\n", 110 | "text = \"A rose by any other name\"\n", 111 | "\n", 112 | "input_ids = tokenizer.encode(text, return_tensors = 'pt')\n", 113 | "\n", 114 | "sample_outputs = base_model.generate(input_ids,\n", 115 | " do_sample = True, \n", 116 | " max_length = 70,\n", 117 | " num_return_sequences = 5) \n", 118 | "\n", 119 | "generic_outputs = get_outputs(sample_outputs, tokenizer)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "Interesting and entertaining! Clearly this model knows the form of poetry. It is obviously generating short lines, with a newline, and it seems to pick up some interesting concepts. Now, let's see if we can fine-tune this poem writer to fit the style of an author we have in mind." 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "### Step 2. Fine-tune the GPT2 Poem model with Anne Bradstreet.\n", 134 | "Now, we're going to fine-tune this model using another, much smaller, dataset. Then later we'll use a text classifier trained to evaluate this style of writer, and see how well our new text performs!" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "If you're curious, take a look at some of the top authors in the English language available through this open domain site.\n", 142 | "https://www.public-domain-poetry.com/topauthors.php \n", 143 | "\n", 144 | "For the purposes of this workshop we'll stick to the longer poem pasted below. On your time time, outside of the workshop, if you'd like to modify this to work with a different text you are welcome to do so.\n", 145 | "\n", 146 | "Poke around at some of the available poems, and copy and paste what you like into this `train.txt` file below. We'll format that for finetuning GPT2 in the next step. In this notebook we're using a poem from Anne Bradstreet, a North American writer from the 17th Century.\n", 147 | "\n", 148 | "You may not have known this, but Anne Bradstreet was the first writer to be published in the North America!" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": { 155 | "jupyter": { 156 | "source_hidden": true 157 | } 158 | }, 159 | "outputs": [], 160 | "source": [ 161 | "%%writefile train.txt\n", 162 | "\n", 163 | "A Dialogue Between Old England And New\n", 164 | "\n", 165 | " By Anne Bradstreet\n", 166 | "\n", 167 | " New England.\n", 168 | "\n", 169 | " Alas, dear Mother, fairest Queen and best,\n", 170 | " With honour, wealth, and peace happy and blest,\n", 171 | " What ails thee hang thy head, and cross thine arms,\n", 172 | " And sit i� the dust to sigh these sad alarms?\n", 173 | " What deluge of new woes thus over-whelm\n", 174 | " The glories of thy ever famous Realm?\n", 175 | " What means this wailing tone, this mournful guise?\n", 176 | " Ah, tell thy Daughter; she may sympathize.\n", 177 | "\n", 178 | " Old England.\n", 179 | "\n", 180 | " Art ignorant indeed of these my woes,\n", 181 | " Or must my forced tongue these griefs disclose,\n", 182 | " And must my self dissect my tatter�d state,\n", 183 | " Which Amazed Christendom stands wondering at?\n", 184 | " And thou a child, a Limb, and dost not feel\n", 185 | " My weak�ned fainting body now to reel?\n", 186 | " This physic-purging-potion I have taken\n", 187 | " Will bring Consumption or an Ague quaking,\n", 188 | " Unless some Cordial thou fetch from high,\n", 189 | " Which present help may ease my malady.\n", 190 | " If I decease, dost think thou shalt survive?\n", 191 | " Or by my wasting state dost think to thrive?\n", 192 | " Then weigh our case, if �t be not justly sad.\n", 193 | " Let me lament alone, while thou art glad.\n", 194 | "\n", 195 | " New England.\n", 196 | "\n", 197 | " And thus, alas, your state you much deplore\n", 198 | " In general terms, but will not say wherefore.\n", 199 | " What Medicine shall I seek to cure this woe,\n", 200 | " If th� wound�s so dangerous, I may not know?\n", 201 | " But you, perhaps, would have me guess it out.\n", 202 | " What, hath some Hengist like that Saxon stout\n", 203 | " By fraud and force usurp�d thy flow�ring crown,\n", 204 | " Or by tempestuous Wars thy fields trod down?\n", 205 | " Or hath Canutus, that brave valiant Dane,\n", 206 | " The regal peaceful Sceptre from thee ta�en?\n", 207 | " Or is �t a Norman whose victorious hand\n", 208 | " With English blood bedews thy conquered Land?\n", 209 | " Or is �t intestine Wars that thus offend?\n", 210 | " Do Maud and Stephen for the Crown contend?\n", 211 | " Do Barons rise and side against their King,\n", 212 | " And call in Foreign aid to help the thing?\n", 213 | " Must Edward be depos�d? Or is �t the hour\n", 214 | " That second Richard must be clapp�d i� th� Tower?\n", 215 | " Or is it the fatal jar, again begun,\n", 216 | " That from the red, white pricking Roses sprung?\n", 217 | " Must Richmond�s aid the Nobles now implore\n", 218 | " To come and break the tushes of the Boar?\n", 219 | " If none of these, dear Mother, what�s your woe?\n", 220 | " Pray, do not fear Spain�s bragging Armado.\n", 221 | " Doth your Ally, fair France, conspire your wrack,\n", 222 | " Or doth the Scots play false behind your back?\n", 223 | " Doth Holland quit you ill for all your love?\n", 224 | " Whence is this storm, from Earth or Heaven above?\n", 225 | " Is �t drought, is �t Famine, or is �t Pestilence?\n", 226 | " Dost feel the smart, or fear the consequence?\n", 227 | " Your humble Child entreats you shew your grief.\n", 228 | " Though Arms nor Purse she hath for your relief�\n", 229 | " Such is her poverty,�yet shall be found\n", 230 | " A suppliant for your help, as she is bound.\n", 231 | "\n", 232 | " Old England.\n", 233 | "\n", 234 | " I must confess some of those Sores you name\n", 235 | " My beauteous Body at this present maim,\n", 236 | " But foreign Foe nor feigned friend I fear,\n", 237 | " For they have work enough, thou knowest, elsewhere.\n", 238 | " Nor is it Alcie�s son and Henry�s Daughter\n", 239 | " Whose proud contention cause this slaughter;\n", 240 | " Nor Nobles siding to make John no King,\n", 241 | " French Louis unjustly to the Crown to bring;\n", 242 | " No Edward, Richard, to lose rule and life,\n", 243 | " Nor no Lancastrians to renew old strife;\n", 244 | " No Crook-backt Tyrant now usurps the Seat,\n", 245 | " Whose tearing tusks did wound, and kill, and threat.\n", 246 | " No Duke of York nor Earl of March to soil\n", 247 | " Their hands in Kindred�s blood whom they did foil;\n", 248 | " No need of Tudor Roses to unite:\n", 249 | " None knows which is the Red or which the White.\n", 250 | " Spain�s braving Fleet a second time is sunk.\n", 251 | " France knows how of my fury she hath drunk\n", 252 | " By Edward third and Henry fifth of fame;\n", 253 | " Her Lilies in my Arms avouch the same.\n", 254 | " My Sister Scotland hurts me now no more,\n", 255 | " Though she hath been injurious heretofore.\n", 256 | " What Holland is, I am in some suspense,\n", 257 | " But trust not much unto his Excellence.\n", 258 | " For wants, sure some I feel, but more I fear;\n", 259 | " And for the Pestilence, who knows how near?\n", 260 | " Famine and Plague, two sisters of the Sword,\n", 261 | " Destruction to a Land doth soon afford.\n", 262 | " They�re for my punishments ordain�d on high,\n", 263 | " Unless thy tears prevent it speedily.\n", 264 | " But yet I answer not what you demand\n", 265 | " To shew the grievance of my troubled Land.\n", 266 | " Before I tell the effect I�ll shew the cause,\n", 267 | " Which are my sins�the breach of sacred Laws:\n", 268 | " Idolatry, supplanter of a N ation,\n", 269 | " With foolish superstitious adoration,\n", 270 | " Are lik�d and countenanc�d by men of might,\n", 271 | " The Gospel is trod down and hath no right.\n", 272 | " Church Offices are sold and bought for gain\n", 273 | " That Pope had hope to find Rome here again.\n", 274 | " For Oaths and Blasphemies did ever ear\n", 275 | " From Beelzebub himself such language hear?\n", 276 | " What scorning of the Saints of the most high!\n", 277 | " What injuries did daily on them lie!\n", 278 | " What false reports, what nick-names did they take,\n", 279 | " Not for their own, but for their Master�s sake!\n", 280 | " And thou, poor soul, wast jeer�d among the rest;\n", 281 | " Thy flying for the Truth I made a jest.\n", 282 | " For Sabbath-breaking and for Drunkenness\n", 283 | " Did ever Land profaneness more express?\n", 284 | " From crying bloods yet cleansed am not I,\n", 285 | " Martyrs and others dying causelessly.\n", 286 | " How many Princely heads on blocks laid down\n", 287 | " For nought but title to a fading Crown!\n", 288 | " �Mongst all the cruelties which I have done,\n", 289 | " Oh, Edward�s Babes, and Clarence�s hapless Son,\n", 290 | " O Jane, why didst thou die in flow�ring prime?�\n", 291 | " Because of Royal Stem, that was thy crime.\n", 292 | " For Bribery, Adultery, for Thefts, and Lies\n", 293 | " Where is the Nation I can�t paralyze?\n", 294 | " With Usury, Extortion, and Oppression,\n", 295 | " These be the Hydras of my stout transgression;\n", 296 | " These be the bitter fountains, heads, and roots\n", 297 | " Whence flow�d the source, the sprigs, the boughs, and fruits.\n", 298 | " Of more than thou canst hear or I relate,\n", 299 | " That with high hand I still did perpetrate,\n", 300 | " For these were threat�ned the woeful day\n", 301 | " I mocked the Preachers, put it fair away.\n", 302 | " The Sermons yet upon record do stand\n", 303 | " That cried destruction to my wicked Land.\n", 304 | " These Prophets� mouths (all the while) was stopt,\n", 305 | " Unworthily, some backs whipt, and ears crept;\n", 306 | " Their reverent cheeks bear the glorious marks\n", 307 | " Of stinking, stigmatizing Romish Clerks;\n", 308 | " Some lost their livings, some in prison pent,\n", 309 | " Some grossly fined, from friends to exile went:\n", 310 | " Their silent tongues to heaven did vengeance cry,\n", 311 | " Who heard their cause, and wrongs judg�d righteously,\n", 312 | " And will repay it sevenfold in my lap.\n", 313 | " This is fore-runner of my after-clap.\n", 314 | " Nor took I warning by my neighbors� falls.\n", 315 | " I saw sad Germany�s dismantled walls,\n", 316 | " I saw her people famish�d, Nobles slain,\n", 317 | " Her fruitful land a barren heath remain.\n", 318 | " I saw (unmov�d) her Armies foil�d and fled,\n", 319 | " Wives forc�d, babes toss�d, her houses calcined.\n", 320 | " I saw strong Rochelle yield�d to her foe,\n", 321 | " Thousands of starved Christians there also.\n", 322 | " I saw poor Ireland bleeding out her last,\n", 323 | " Such cruelty as all reports have past.\n", 324 | " Mine heart obdurate stood not yet aghast.\n", 325 | " Now sip I of that cup, and just �t may be\n", 326 | " The bottom dregs reserved are for me.\n", 327 | "\n", 328 | " New England.\n", 329 | "\n", 330 | " To all you�ve said, sad mother, I assent.\n", 331 | " Your fearful sins great cause there �s to lament.\n", 332 | " My guilty hands (in part) hold up with you,\n", 333 | " A sharer in your punishment�s my due.\n", 334 | " But all you say amounts to this effect,\n", 335 | " Not what you feel, but what you do expect.\n", 336 | " Pray, in plain terms, what is your present grief?\n", 337 | " Then let�s join heads and hands for your relief.\n", 338 | "\n", 339 | " Old England.\n", 340 | "\n", 341 | " Well, to the matter, then. There�s grown of late\n", 342 | " �Twixt King and Peers a question of state:\n", 343 | " Which is the chief, the law, or else the King?\n", 344 | " One saith, it�s he; the other, no such thing.\n", 345 | " My better part in Court of Parliament\n", 346 | " To ease my groaning land shew their intent\n", 347 | " To crush the proud, and right to each man deal,\n", 348 | " To help the Church, and stay the Common-Weal.\n", 349 | " So many obstacles comes in their way\n", 350 | " As puts me to a stand what I should say.\n", 351 | " Old customs, new Prerogatives stood on.\n", 352 | " Had they not held law fast, all had been gone,\n", 353 | " Which by their prudence stood them in such stead\n", 354 | " They took high Strafford lower by the head,\n", 355 | " And to their Laud be �t spoke they held �n th� Tower\n", 356 | " All England�s metropolitan that hour.\n", 357 | " This done, an Act they would have passed fain\n", 358 | " No prelate should his Bishopric retain.\n", 359 | " Here tugg�d they hard indeed, for all men saw\n", 360 | " This must be done by Gospel, not by law.\n", 361 | " Next the Militia they urged sore.\n", 362 | " This was denied, I need not say wherefore.\n", 363 | " The King, displeased, at York himself absents.\n", 364 | " They humbly beg return, shew their intents.\n", 365 | " The writing, printing, posting to and fro,\n", 366 | " Shews all was done; I�ll therefore let it go.\n", 367 | " But now I come to speak of my disaster.\n", 368 | " Contention�s grown �twixt Subjects and their Master,\n", 369 | " They worded it so long they fell to blows,\n", 370 | " That thousands lay on heaps. Here bleeds my woes.\n", 371 | " I that no wars so many years have known\n", 372 | " Am now destroy�d and slaughter�d by mine own.\n", 373 | " But could the field alone this strife decide,\n", 374 | " One battle, two, or three I might abide,\n", 375 | " But these may be beginnings of more woe�\n", 376 | " Who knows, the worst, the best may overthrow!\n", 377 | " Religion, Gospel, here lies at the stake,\n", 378 | " Pray now, dear child, for sacred Zion�s sake,\n", 379 | " Oh, pity me in this sad perturbation,\n", 380 | " My plundered Towns, my houses� devastation,\n", 381 | " My ravisht virgins, and my young men slain,\n", 382 | " My wealthy trading fallen, my dearth of grain.\n", 383 | " The seedtime�s come, but Ploughman hath no hope\n", 384 | " Because he knows not who shall inn his crop.\n", 385 | " The poor they want their pay, their children bread,\n", 386 | " Their woful mothers� tears unpitied.\n", 387 | " If any pity in thy heart remain,\n", 388 | " Or any child-like love thou dost retain,\n", 389 | " For my relief now use thy utmost skill,\n", 390 | " And recompense me good for all my ill.\n", 391 | "\n", 392 | " New England.\n", 393 | "\n", 394 | " Dear mother, cease complaints, and wipe your eyes,\n", 395 | " Shake off your dust, cheer up, and now arise.\n", 396 | " You are my mother, nurse, I once your flesh,\n", 397 | " Your sunken bowels gladly would refresh.\n", 398 | " Your griefs I pity much but should do wrong,\n", 399 | " To weep for that we both have pray�d for long,\n", 400 | " To see these latter days of hop�d-for good,\n", 401 | " That Right may have its right, though �t be with blood.\n", 402 | " After dark Popery the day did clear;\n", 403 | " But now the Sun in�s brightness shall appear.\n", 404 | " Blest be the Nobles of thy Noble Land\n", 405 | " With (ventur�d lives) for truth�s defence that stand.\n", 406 | " Blest be thy Commons, who for Common good\n", 407 | " And thy infringed Laws have boldly stood.\n", 408 | " Blest be thy Counties, who do aid thee still\n", 409 | " With hearts and states to testify their will.\n", 410 | " Blest be thy Preachers, who do cheer thee on.\n", 411 | " Oh, cry: the sword of God and Gideon!\n", 412 | " And shall I not on them wish Mero�s curse\n", 413 | " That help thee not with prayers, arms, and purse?\n", 414 | " And for my self, let miseries abound\n", 415 | " If mindless of thy state I e�er be found.\n", 416 | " These are the days the Church�s foes to crush,\n", 417 | " To root out Prelates, head, tail, branch, and rush.\n", 418 | " Let�s bring Baal�s vestments out, to make a fire,\n", 419 | " Their Mitres, Surplices, and all their tire,\n", 420 | " Copes, Rochets, Croziers, and such trash,\n", 421 | " And let their names consume, but let the flash\n", 422 | " Light Christendom, and all the world to see\n", 423 | " We hate Rome�s Whore, with all her trumpery.\n", 424 | " Go on, brave Essex, shew whose son thou art,\n", 425 | " Not false to King, nor Country in thy heart,\n", 426 | " But those that hurt his people and his Crown,\n", 427 | " By force expel, destroy, and tread them down.\n", 428 | " Let Gaols be fill�d with th� remnant of that pack,\n", 429 | " And sturdy Tyburn loaded till it crack.\n", 430 | " And ye brave Nobles, chase away all fear,\n", 431 | " And to this blessed Cause closely adhere.\n", 432 | " O mother, can you weep and have such Peers?\n", 433 | " When they are gone, then drown your self in tears,\n", 434 | " If now you weep so much, that then no more\n", 435 | " The briny Ocean will o�erflow your shore.\n", 436 | " These, these are they (I trust) with Charles our king,\n", 437 | " Out of all mists such glorious days will bring\n", 438 | " That dazzled eyes, beholding, much shall wonder\n", 439 | " At that thy settled Peace, thy wealth, and splendour,\n", 440 | " Thy Church and Weal establish�d in such manner\n", 441 | " That all shall joy that thou display�dst thy banner,\n", 442 | " And discipline erected so, I trust,\n", 443 | " That nursing Kings shall come and lick thy dust.\n", 444 | " Then Justice shall in all thy Courts take place\n", 445 | " Without respect of persons or of case.\n", 446 | " Then bribes shall cease, and suits shall not stick long,\n", 447 | " Patience and purse of Clients for to wrong.\n", 448 | " Then High Commissions shall fall to decay,\n", 449 | " And Pursuivants and Catchpoles want their pay.\n", 450 | " So shall thy happy Nation ever flourish,\n", 451 | " When truth and righteousness they thus shall nourish.\n", 452 | " When thus in Peace, thine Armies brave send out\n", 453 | " To sack proud Rome, and all her vassals rout.\n", 454 | " There let thy name, thy fame, and valour shine,\n", 455 | " As did thine Ancestors� in Palestine,\n", 456 | " And let her spoils full pay with int�rest be\n", 457 | " Of what unjustly once she poll�d from thee.\n", 458 | " Of all the woes thou canst let her be sped,\n", 459 | " Execute to th� full the vengeance threatened.\n", 460 | " Bring forth the beast that rul�d the world with�s beck,\n", 461 | " And tear his flesh, and set your feet on�s neck,\n", 462 | " And make his filthy den so desolate\n", 463 | " To th� �stonishment of all that knew his state.\n", 464 | " This done, with brandish�d swords to Turkey go,�\n", 465 | " (For then what is it but English blades dare do?)\n", 466 | " And lay her waste, for so�s the sacred doom,\n", 467 | " And do to Gog as thou hast done to Rome.\n", 468 | " Oh Abraham�s seed, lift up your heads on high,\n", 469 | " For sure the day of your redemption�s nigh.\n", 470 | " The scales shall fall from your long blinded eyes,\n", 471 | " And him you shall adore who now despise.\n", 472 | " Then fullness of the Nations in shall flow,\n", 473 | " And Jew and Gentile to one worship go.\n", 474 | " Then follows days of happiness and rest.\n", 475 | " Whose lot doth fall to live therein is blest.\n", 476 | " No Canaanite shall then be found �n th� land,\n", 477 | " And holiness on horses� bells shall stand.\n", 478 | " If this make way thereto, then sigh no more,\n", 479 | " But if at all thou didst not see �t before.\n", 480 | " Farewell, dear mother; Parliament, prevail,\n", 481 | " And in a while you�ll tell another tale.\n" 482 | ] 483 | }, 484 | { 485 | "cell_type": "markdown", 486 | "metadata": {}, 487 | "source": [ 488 | "### Step 3. Format your training data for Hugging Face on Amazon SageMaker.\n", 489 | "Now, let's parse your training data to format it for finetuning GPT2 and training on Hugging Face. " 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": null, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "data = []\n", 499 | "\n", 500 | "with open('train.txt') as f:\n", 501 | " for row in f.readlines():\n", 502 | " d = row.strip()\n", 503 | " if len(d) > 1:\n", 504 | " data.append(d)" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": null, 510 | "metadata": {}, 511 | "outputs": [], 512 | "source": [ 513 | "print ('Found {} valid objects in the training data.'.format(len(data)))" 514 | ] 515 | }, 516 | { 517 | "cell_type": "code", 518 | "execution_count": null, 519 | "metadata": {}, 520 | "outputs": [], 521 | "source": [ 522 | "print (data[:10])" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": null, 528 | "metadata": {}, 529 | "outputs": [], 530 | "source": [ 531 | "import sagemaker\n", 532 | "\n", 533 | "sess = sagemaker.Session()\n", 534 | "bucket = sess.default_bucket() \n", 535 | "\n", 536 | "train_file_name = 'train.txt'\n", 537 | "s3_train_data = 's3://{}/gpt2/{}'.format(bucket, train_file_name)\n", 538 | "\n", 539 | "!aws s3 cp {train_file_name} {s3_train_data}" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": null, 545 | "metadata": {}, 546 | "outputs": [], 547 | "source": [ 548 | "import sagemaker\n", 549 | "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n", 550 | "\n", 551 | "# gets role for executing training job\n", 552 | "role = sagemaker.get_execution_role()\n", 553 | "hyperparameters = {\n", 554 | " 'model_name_or_path':\"ismaelfaro/gpt2-poems.en\",\n", 555 | " 'output_dir':'/opt/ml/model',\n", 556 | " 'do_train':True,\n", 557 | " 'train_file': '/opt/ml/input/data/train/{}'.format(train_file_name),\n", 558 | " 'num_train_epochs': 5,\n", 559 | " # set batch size to 22 if using SM training compiler\n", 560 | " \"per_device_train_batch_size\": 64,\n", 561 | " # add your remaining hyperparameters\n", 562 | " # more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/pytorch/language-modeling\n", 563 | "}\n", 564 | "\n", 565 | "# git configuration to download our fine-tuning script\n", 566 | "git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}\n", 567 | "\n", 568 | "# creates Hugging Face estimator\n", 569 | "huggingface_estimator = HuggingFace(\n", 570 | " entry_point='run_clm.py',\n", 571 | " source_dir='./examples/pytorch/language-modeling',\n", 572 | " instance_type='ml.p3.2xlarge',\n", 573 | " instance_count=1,\n", 574 | " role=role,\n", 575 | " git_config=git_config,\n", 576 | " transformers_version='4.11.0',\n", 577 | " pytorch_version='1.9.0',\n", 578 | " py_version='py38',\n", 579 | " hyperparameters = hyperparameters,\n", 580 | " # pass the training compiler config to speed up your job\n", 581 | " compiler_config=TrainingCompilerConfig(),\n", 582 | " environment = {'GPU_NUM_DEVICES':'1'},\n", 583 | " disable_profiler = True,\n", 584 | " debugger_hook_config = False\n", 585 | ")\n", 586 | "\n", 587 | "# starting the train job\n", 588 | "# should take about 13 minutes to run on current settings\n", 589 | "huggingface_estimator.fit({'train':s3_train_data}, wait = True)" 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "### Step 4. Test your trained model locally" 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "execution_count": null, 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "from sagemaker.huggingface import HuggingFace\n", 606 | "import time\n", 607 | "\n", 608 | "# redefining if you need to restart your kernel \n", 609 | "# huggingface_estimator = HuggingFace.attach('huggingface-pytorch-trcomp-training-2022-03-14-17-11-36-978')\n", 610 | "try:\n", 611 | " s3_model_data = huggingface_estimator.model_data\n", 612 | " local_model_path = 'gpt2_finetuned'\n", 613 | " \n", 614 | "except:\n", 615 | " time.sleep(5)\n", 616 | " s3_model_data = huggingface_estimator.model_data\n", 617 | " local_model_path = 'gpt2_finetuned'\n", 618 | " " 619 | ] 620 | }, 621 | { 622 | "cell_type": "code", 623 | "execution_count": null, 624 | "metadata": {}, 625 | "outputs": [], 626 | "source": [ 627 | "!mkdir {local_model_path}\n", 628 | "!aws s3 cp {s3_model_data} {local_model_path}\n", 629 | "!tar -xvf {local_model_path}/model.tar.gz -C {local_model_path}\n", 630 | "!rm {local_model_path}/model.tar.gz" 631 | ] 632 | }, 633 | { 634 | "cell_type": "code", 635 | "execution_count": null, 636 | "metadata": {}, 637 | "outputs": [], 638 | "source": [ 639 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 640 | "\n", 641 | "# optional - rerun this if you need to restart your kernel. We are actually using the same tokenizer from before\n", 642 | "tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n", 643 | "local_model_path = 'gpt2_finetuned'\n", 644 | "model = AutoModelForCausalLM.from_pretrained(local_model_path)" 645 | ] 646 | }, 647 | { 648 | "cell_type": "code", 649 | "execution_count": null, 650 | "metadata": {}, 651 | "outputs": [], 652 | "source": [ 653 | "# step to make sure we can run inference with this model locally\n", 654 | "model.eval()" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": null, 660 | "metadata": {}, 661 | "outputs": [], 662 | "source": [ 663 | "from transformers import set_seed\n", 664 | "\n", 665 | "set_seed(42)\n", 666 | "\n", 667 | "text = \"A rose by any other name \"\n", 668 | "input_ids = tokenizer.encode(text, return_tensors = 'pt')\n", 669 | "\n", 670 | "sample_outputs = model.generate(input_ids,\n", 671 | " do_sample = True, \n", 672 | " max_length = 70,\n", 673 | " num_return_sequences = 5) \n", 674 | " \n", 675 | "bradsteet_raw = get_outputs(sample_outputs, tokenizer)" 676 | ] 677 | }, 678 | { 679 | "cell_type": "markdown", 680 | "metadata": {}, 681 | "source": [ 682 | "Interesting, it certainly looks different. Let's see if we can modify this output using different paramters to invoke the trained model." 683 | ] 684 | }, 685 | { 686 | "cell_type": "code", 687 | "execution_count": null, 688 | "metadata": {}, 689 | "outputs": [], 690 | "source": [ 691 | "sample_outputs = model.generate(input_ids, \n", 692 | " max_length=70,\n", 693 | " do_sample=True, \n", 694 | " # only pick tokens at and above this probability level\n", 695 | " top_p=0.85,\n", 696 | " # only pick from this many tokens\n", 697 | " top_k=200,\n", 698 | " num_return_sequences = 5) \n", 699 | "\n", 700 | "\n", 701 | "bradstreet_top_85 = get_outputs(sample_outputs, tokenizer)\n" 702 | ] 703 | }, 704 | { 705 | "cell_type": "markdown", 706 | "metadata": {}, 707 | "source": [ 708 | "Wow! Quite a difference - not all of these seem very much like Bradstreet, and just much more generic. Yet the logical coherence on some of them is strong. Let's try it again with a smaller top_k and top_p." 709 | ] 710 | }, 711 | { 712 | "cell_type": "code", 713 | "execution_count": null, 714 | "metadata": {}, 715 | "outputs": [], 716 | "source": [ 717 | "sample_outputs = model.generate(input_ids, \n", 718 | " max_length=70,\n", 719 | " do_sample=True, \n", 720 | " # only pick tokens at and above this probability level\n", 721 | " top_p=0.95,\n", 722 | " # only pick from this many tokens\n", 723 | " top_k=110,\n", 724 | " num_return_sequences = 5) \n", 725 | "\n", 726 | "\n", 727 | "bradstreet_top_95 = get_outputs(sample_outputs, tokenizer)\n" 728 | ] 729 | }, 730 | { 731 | "cell_type": "markdown", 732 | "metadata": {}, 733 | "source": [ 734 | "Interesting - under these terms the model seems even more generic. You can still pick up a hint of that very old English style of writing, and yet the social media base terms come even more to the surface." 735 | ] 736 | }, 737 | { 738 | "cell_type": "markdown", 739 | "metadata": {}, 740 | "source": [ 741 | "### Step 5. Load a Text Classifier to Quantify Our Generated Text\n", 742 | "Now, we're going to use another model from the HF Hub. This time it's a text classifier, built specifically to give a strong signal for whether or not our text seems like it's in the style of Anne Bradstreet." 743 | ] 744 | }, 745 | { 746 | "cell_type": "code", 747 | "execution_count": null, 748 | "metadata": {}, 749 | "outputs": [], 750 | "source": [ 751 | "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n", 752 | "\n", 753 | "anne_model_name = 'edubz/anne_bradstreet'\n", 754 | "\n", 755 | "anne_tokenizer = AutoTokenizer.from_pretrained(anne_model_name)\n", 756 | "\n", 757 | "anne_clf = AutoModelForSequenceClassification.from_pretrained(anne_model_name)" 758 | ] 759 | }, 760 | { 761 | "cell_type": "code", 762 | "execution_count": null, 763 | "metadata": {}, 764 | "outputs": [], 765 | "source": [ 766 | "from scipy.special import softmax\n", 767 | "\n", 768 | "def invoke_locally(text, anne_clf, anne_tokenizer):\n", 769 | " \n", 770 | " input_ids = anne_tokenizer(text, return_tensors = 'pt')\n", 771 | "\n", 772 | " output = anne_clf(**input_ids)\n", 773 | "\n", 774 | " logits = output['logits'].detach().numpy().tolist()[0]\n", 775 | "\n", 776 | " res = softmax(logits).tolist()\n", 777 | "\n", 778 | " conf = max(res)\n", 779 | "\n", 780 | " label = res.index(conf)\n", 781 | " \n", 782 | " if label == 0:\n", 783 | " label_str = 'Not Anne'\n", 784 | " elif label == 1:\n", 785 | " label_str = 'Anne'\n", 786 | " \n", 787 | " return {'confidence': conf, 'label':label_str }" 788 | ] 789 | }, 790 | { 791 | "cell_type": "code", 792 | "execution_count": null, 793 | "metadata": {}, 794 | "outputs": [], 795 | "source": [ 796 | "invoke_locally(\"Alas, dear Mother, fairest Queen and best\", anne_clf, anne_tokenizer)" 797 | ] 798 | }, 799 | { 800 | "cell_type": "code", 801 | "execution_count": null, 802 | "metadata": {}, 803 | "outputs": [], 804 | "source": [ 805 | "invoke_locally(\"A rose by any other name\", anne_clf, anne_tokenizer)" 806 | ] 807 | }, 808 | { 809 | "cell_type": "code", 810 | "execution_count": null, 811 | "metadata": {}, 812 | "outputs": [], 813 | "source": [ 814 | "invoke_locally(\"Wow I am enjoying this workshop\", anne_clf, anne_tokenizer)" 815 | ] 816 | }, 817 | { 818 | "cell_type": "markdown", 819 | "metadata": {}, 820 | "source": [ 821 | "Now, run some tests of your own. Try different invocation parameters. What seems to get you the highest Anne scores?" 822 | ] 823 | }, 824 | { 825 | "cell_type": "markdown", 826 | "metadata": {}, 827 | "source": [ 828 | "### Step 6. Deploy your fine-tuned model onto a SageMaker multi-model endpoint\n", 829 | "*Now, if you haven't already, please execute `1_deploy_gpt2_gptj_mme.ipynb`.* \n", 830 | "\n", 831 | "Now, let's deploy this model onto SageMaker. In particular we will save this model to disk, and then load it onto a multi-model endpoint.\n", 832 | "\n", 833 | "\n", 834 | "We'll also list all available models from that endpoint, and test out generating text with each of these. Who knows, maybe we'll stumble on something good enough for the Tribune!" 835 | ] 836 | }, 837 | { 838 | "cell_type": "code", 839 | "execution_count": null, 840 | "metadata": {}, 841 | "outputs": [], 842 | "source": [ 843 | "# optional - rerun this if you need to restart your kernel. We are actually using the same tokenizer from before\n", 844 | "\n", 845 | "import sagemaker\n", 846 | "from transformers import AutoTokenizer, AutoModelForCausalLM\n", 847 | "\n", 848 | "sess = sagemaker.Session()\n", 849 | "bucket = sess.default_bucket() \n", 850 | "\n", 851 | "local_model_path = 'gpt2_finetuned'\n", 852 | "\n", 853 | "\n", 854 | "tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n", 855 | "\n", 856 | "model = AutoModelForCausalLM.from_pretrained(local_model_path)" 857 | ] 858 | }, 859 | { 860 | "cell_type": "code", 861 | "execution_count": null, 862 | "metadata": {}, 863 | "outputs": [], 864 | "source": [ 865 | "bradstreet_path = 'gpt2-bradstreet-model'\n", 866 | "b_model_name = '{}.tar.gz'.format(bradstreet_path)" 867 | ] 868 | }, 869 | { 870 | "cell_type": "code", 871 | "execution_count": null, 872 | "metadata": {}, 873 | "outputs": [], 874 | "source": [ 875 | "model.save_pretrained('{}/'.format(bradstreet_path))\n", 876 | "tokenizer.save_pretrained('{}/'.format(bradstreet_path))" 877 | ] 878 | }, 879 | { 880 | "cell_type": "code", 881 | "execution_count": null, 882 | "metadata": {}, 883 | "outputs": [], 884 | "source": [ 885 | "!tar -czvf {b_model_name} {bradstreet_path}" 886 | ] 887 | }, 888 | { 889 | "cell_type": "code", 890 | "execution_count": null, 891 | "metadata": {}, 892 | "outputs": [], 893 | "source": [ 894 | "!aws s3 cp {b_model_name} s3://{bucket}/{prefix}/{b_model_name}" 895 | ] 896 | }, 897 | { 898 | "cell_type": "markdown", 899 | "metadata": {}, 900 | "source": [ 901 | "### Step 7. Test your fine-tuned model on SageMaker multi-model endpoint" 902 | ] 903 | }, 904 | { 905 | "cell_type": "code", 906 | "execution_count": null, 907 | "metadata": {}, 908 | "outputs": [], 909 | "source": [ 910 | "import boto3\n", 911 | "\n", 912 | "client = boto3.client('sagemaker')\n", 913 | "\n", 914 | "endpoints = client.list_endpoints()" 915 | ] 916 | }, 917 | { 918 | "cell_type": "code", 919 | "execution_count": null, 920 | "metadata": {}, 921 | "outputs": [], 922 | "source": [ 923 | "for e in endpoints['Endpoints']:\n", 924 | " name = e['EndpointName']\n", 925 | " if 'mme' in name:\n", 926 | " print (name)\n", 927 | " mme_name = name" 928 | ] 929 | }, 930 | { 931 | "cell_type": "code", 932 | "execution_count": null, 933 | "metadata": {}, 934 | "outputs": [], 935 | "source": [ 936 | "import sagemaker\n", 937 | "sess=sagemaker.Session()\n", 938 | "\n", 939 | "predictor = sagemaker.predictor.Predictor(endpoint_name = mme_name, sagemaker_session=sess)\n", 940 | "predictor.serializer = sagemaker.serializers.JSONSerializer()\n", 941 | "predictor.deserializer = sagemaker.deserializers.JSONDeserializer()" 942 | ] 943 | }, 944 | { 945 | "cell_type": "code", 946 | "execution_count": null, 947 | "metadata": {}, 948 | "outputs": [], 949 | "source": [ 950 | "# the first time you invoke a model on an MME it will take longer to respond b/c the model is being copied from S3\n", 951 | "predictor.predict({\"inputs\":'A rose by any other name'}, target_model=b_model_name)" 952 | ] 953 | }, 954 | { 955 | "cell_type": "code", 956 | "execution_count": null, 957 | "metadata": {}, 958 | "outputs": [], 959 | "source": [ 960 | "predictor.predict({\"inputs\":'My country, my land, my home'}, target_model=b_model_name)" 961 | ] 962 | }, 963 | { 964 | "cell_type": "markdown", 965 | "metadata": {}, 966 | "source": [ 967 | "### Step 8. Write poetry for the Chicago Tribune\n", 968 | "Now - select your favorite lines from each output from GPT, and pass it in to the model. Feel free to modify the parameters using kwargs. When you are finished, you can submit your poem to our GitHub workshop page!\n", 969 | "\n", 970 | "**Please note** every time you invoke a new model via MME AWS is copying the model artifact from S3 to the SageMaker endpoint. That means **expect a big time delay whenever you invoke a new model.** \n", 971 | "\n", 972 | "One way to get around that is with model compilation, ie running SageMaker Neo to decrease the size, and thereby the runtime, of that model.\n", 973 | "\n", 974 | "In the poem below, I manually copied my favorite line from each output of the model, and fed it in to the generator. I manually pasted all of my favorites into the markdown file you see below.\n", 975 | "\n", 976 | "---" 977 | ] 978 | }, 979 | { 980 | "cell_type": "markdown", 981 | "metadata": {}, 982 | "source": [ 983 | "### My poem - A rose by any other model\n", 984 | "\n", 985 | "A rose by any other name has many meanings.
\n", 986 | "When all that has been presented to us is a form of exaggeration.
\n", 987 | "The language will not preserve.
\n", 988 | "However, the old idea of he who has no business vainly passing by without any other
\n", 989 | "Some unending mizzen, deceived and deceived, seems ever more absurd and likely to harm our time.
\n", 990 | "We tuck his back into the sea which is on the plain almost as soon as we lose sight of him.
\n", 991 | "A mariner shall pass.
\n", 992 | "And I may leave nothing to thee till thou return, for as I said, My hand am strong when thou shouldst require it.
\n", 993 | "This comes out of Kant\\'s conviction that we have nothing in our minds
" 994 | ] 995 | }, 996 | { 997 | "cell_type": "code", 998 | "execution_count": null, 999 | "metadata": {}, 1000 | "outputs": [], 1001 | "source": [ 1002 | "text = 'A rose by any other name has many meanings.'\n", 1003 | "predictor.predict({\"inputs\":text, \n", 1004 | " 'parameters':{'max_length':70,\n", 1005 | " 'do_sample':True, \n", 1006 | " # only pick tokens at and above this probability level\n", 1007 | " 'top_p':0.99,\n", 1008 | " # only pick from this many tokens\n", 1009 | " 'top_k':600}}, \n", 1010 | " target_model=b_model_name)" 1011 | ] 1012 | }, 1013 | { 1014 | "cell_type": "code", 1015 | "execution_count": null, 1016 | "metadata": {}, 1017 | "outputs": [], 1018 | "source": [ 1019 | "text = 'However, the old idea of he who has no business vainly passing by without any other means'\n", 1020 | "\n", 1021 | "predictor.predict({\"inputs\":text, \n", 1022 | " 'parameters':{'max_length':70,\n", 1023 | " 'do_sample':True, \n", 1024 | " # only pick tokens at and above this probability level\n", 1025 | " 'top_p':0.99,\n", 1026 | " # only pick from this many tokens\n", 1027 | " 'top_k':600}}, \n", 1028 | " target_model=b_model_name)" 1029 | ] 1030 | }, 1031 | { 1032 | "cell_type": "code", 1033 | "execution_count": null, 1034 | "metadata": {}, 1035 | "outputs": [], 1036 | "source": [ 1037 | "text = 'If two people try to communicate true love'\n", 1038 | "\n", 1039 | "predictor.predict({\"inputs\":text, \n", 1040 | " 'parameters':{'max_length':70,\n", 1041 | " 'do_sample':True, \n", 1042 | " # only pick tokens at and above this probability level\n", 1043 | " 'top_p':0.99,\n", 1044 | " # only pick from this many tokens\n", 1045 | " 'top_k':600}}, \n", 1046 | " target_model=b_model_name)" 1047 | ] 1048 | }, 1049 | { 1050 | "cell_type": "code", 1051 | "execution_count": null, 1052 | "metadata": {}, 1053 | "outputs": [], 1054 | "source": [ 1055 | "text = 'A rose by any other model'\n", 1056 | "\n", 1057 | "predictor.predict({\"inputs\":text, \n", 1058 | " 'parameters':{'max_length':70,\n", 1059 | " 'do_sample':True, \n", 1060 | " # only pick tokens at and above this probability level\n", 1061 | " 'top_p':0.99,\n", 1062 | " # only pick from this many tokens\n", 1063 | " 'top_k':100}}, \n", 1064 | " target_model=b_model_name)" 1065 | ] 1066 | }, 1067 | { 1068 | "cell_type": "markdown", 1069 | "metadata": {}, 1070 | "source": [ 1071 | "### Optional - use a pretrained GPTJ" 1072 | ] 1073 | }, 1074 | { 1075 | "cell_type": "markdown", 1076 | "metadata": {}, 1077 | "source": [ 1078 | "Use the Endpoints notebook in this repository to deploy and test a GPT-J 6B endpoint. Compare the generation to that of your fine-tuned GPT-2 model. Add some of the lines to your poem if you like!" 1079 | ] 1080 | }, 1081 | { 1082 | "cell_type": "markdown", 1083 | "metadata": {}, 1084 | "source": [ 1085 | "### Optional- Use Other Pretrained Models and Hugging Face Datasets\n", 1086 | "\n", 1087 | "**Available pretrained models and datasets from Hugging Face**\n", 1088 | "\n", 1089 | "**Datasets**\n", 1090 | "- Shakespeare:\n", 1091 | " - https://huggingface.co/datasets/tiny_shakespeare \n", 1092 | "\n", 1093 | "**Pretrained models**\n", 1094 | "- Chinese Poetry:\n", 1095 | " - https://huggingface.co/caixin1998/chinese-poetry-gpt2 \n", 1096 | "- Hebrew Poetry:\n", 1097 | " - https://huggingface.co/Norod78/hebrew_poetry-gpt_neo-small \n", 1098 | "- Arabic Poetry:\n", 1099 | " - https://huggingface.co/akhooli/gpt2-small-arabic-poetry \n", 1100 | "- Russian Poetry:\n", 1101 | " - https://huggingface.co/TuhinColumbia/russianpoetrymany \n", 1102 | "- Persian Poetry:\n", 1103 | " - https://huggingface.co/mitra-mir/BERT-Persian-Poetry \n", 1104 | "- Italian Poetry:\n", 1105 | " - https://huggingface.co/TuhinColumbia/italianpoetrymany " 1106 | ] 1107 | }, 1108 | { 1109 | "cell_type": "markdown", 1110 | "metadata": {}, 1111 | "source": [ 1112 | "# Conclusion - Use Hugging Face and Text Generation on Amazon SageMaker for Your Organization\n", 1113 | "Now that you've learned how to test, finetune, deploy and utilize a text generation model on SageMaker, let's understand how to apply that within your organzation.\n", 1114 | "\n", 1115 | "First, think to yourself, does my organization already produce a lot of written text? Do we write documentation, scripts, blog posts, documents, answers to questions, customer messaging, etc? Odds are, you do. \n", 1116 | "\n", 1117 | "Then, ask yourself, where do we already have a large volume of written text I can easily access? That may be your existing public documentation, your existing blog posts, etc. First, run through this notebook and use some of your own data to finetune a GPT model. See how well that performs, then consider scaling to large models, including GPT-J. \n", 1118 | "\n", 1119 | "If you really aren't seeing the performance you want, [consider training a model from scratch!](https://github.com/nlp-with-transformers/notebooks/blob/main/10_transformers-from-scratch.ipynb )\n", 1120 | "\n", 1121 | "Look at this and other examples within [Hugging Face's SageMaker example notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker), and similar examples on the [SageMaker repository!](https://github.com/aws/amazon-sagemaker-examples/search?q=hugging+face)\n", 1122 | "\n", 1123 | "Remember that in order to get the best performance we **combined a variety of computer-generated and human-discriminated approaches**. Further work could improve on this by training discriminator NLP models, such as text classifiers in certain styles, to make the generation and improvement process even faster." 1124 | ] 1125 | } 1126 | ], 1127 | "metadata": { 1128 | "instance_type": "ml.t3.medium", 1129 | "kernelspec": { 1130 | "display_name": "Python 3 (PyTorch 1.8 Python 3.6 CPU Optimized)", 1131 | "language": "python", 1132 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/1.8.1-cpu-py36" 1133 | }, 1134 | "language_info": { 1135 | "codemirror_mode": { 1136 | "name": "ipython", 1137 | "version": 3 1138 | }, 1139 | "file_extension": ".py", 1140 | "mimetype": "text/x-python", 1141 | "name": "python", 1142 | "nbconvert_exporter": "python", 1143 | "pygments_lexer": "ipython3", 1144 | "version": "3.6.13" 1145 | } 1146 | }, 1147 | "nbformat": 4, 1148 | "nbformat_minor": 4 1149 | } 1150 | --------------------------------------------------------------------------------