├── .gitignore ├── images ├── FileAttach.png ├── aud-claim.PNG ├── token-issuer.PNG ├── AmazonQLambdaHook.png ├── settings-designer.png ├── settings-outputs.png ├── qnaitem_lambdahook.png └── qnaitem_lambdahook_example.png ├── CODE_OF_CONDUCT.md ├── LICENSE ├── lambdas ├── mistral-7b-instruct-chat-llm │ ├── src │ │ ├── llm.py │ │ ├── cfnresponse.py │ │ └── settings.py │ └── template.yml ├── llama-2-13b-chat-llm │ ├── src │ │ ├── llm.py │ │ ├── cfnresponse.py │ │ └── settings.py │ └── template.yml ├── ai21-llm │ ├── src │ │ ├── cfnresponse.py │ │ ├── settings.py │ │ ├── llm.py │ │ └── lambdahook.py │ └── template.yml ├── anthropic-llm │ ├── src │ │ ├── cfnresponse.py │ │ ├── llm.py │ │ └── settings.py │ └── template.yml ├── bedrock-embeddings-and-llm │ ├── src │ │ ├── cfnresponse.py │ │ ├── embeddings.py │ │ ├── testModel.py │ │ ├── llm.py │ │ ├── settings.py │ │ └── lambdahook.py │ └── template.yml └── qna_bot_qbusiness_lambdahook │ ├── template.yml │ ├── src │ └── lambdahook.py │ └── README.md ├── CONTRIBUTING.md ├── CHANGELOG.md ├── publish.sh └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | */python/ 3 | */*/python/ 4 | out/ 5 | release.sh 6 | -------------------------------------------------------------------------------- /images/FileAttach.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/FileAttach.png -------------------------------------------------------------------------------- /images/aud-claim.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/aud-claim.PNG -------------------------------------------------------------------------------- /images/token-issuer.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/token-issuer.PNG -------------------------------------------------------------------------------- /images/AmazonQLambdaHook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/AmazonQLambdaHook.png -------------------------------------------------------------------------------- /images/settings-designer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/settings-designer.png -------------------------------------------------------------------------------- /images/settings-outputs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/settings-outputs.png -------------------------------------------------------------------------------- /images/qnaitem_lambdahook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/qnaitem_lambdahook.png -------------------------------------------------------------------------------- /images/qnaitem_lambdahook_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/qnabot-on-aws-plugin-samples/HEAD/images/qnaitem_lambdahook_example.png -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT No Attribution 2 | 3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 13 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 15 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 16 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /lambdas/mistral-7b-instruct-chat-llm/src/llm.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | import io 5 | from typing import Dict 6 | 7 | # grab environment variables 8 | SAGEMAKER_ENDPOINT_NAME = os.environ['SAGEMAKER_ENDPOINT_NAME'] 9 | runtime= boto3.client('runtime.sagemaker') 10 | 11 | def transform_input(prompt: Dict, model_kwargs: Dict) -> bytes: 12 | input_str = json.dumps( 13 | { 14 | "inputs": prompt, 15 | "parameters": model_kwargs, 16 | } 17 | ) 18 | return input_str.encode("utf-8") 19 | 20 | 21 | def call_llm(parameters, prompt): 22 | data = transform_input(prompt, parameters) 23 | response = runtime.invoke_endpoint(EndpointName=SAGEMAKER_ENDPOINT_NAME, 24 | ContentType='application/json', 25 | Body=data) 26 | generated_text = json.loads(response['Body'].read().decode("utf-8")) 27 | return generated_text[0]["generated_text"] 28 | 29 | 30 | def lambda_handler(event, context): 31 | print("Event: ", json.dumps(event)) 32 | prompt = event["prompt"] 33 | parameters = event["parameters"] 34 | generated_text = call_llm(parameters, prompt) 35 | print("Result:", json.dumps(generated_text)) 36 | return { 37 | 'generated_text': generated_text 38 | } 39 | -------------------------------------------------------------------------------- /lambdas/llama-2-13b-chat-llm/src/llm.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | import io 5 | from typing import Dict 6 | 7 | # grab environment variables 8 | SAGEMAKER_ENDPOINT_NAME = os.environ['SAGEMAKER_ENDPOINT_NAME'] 9 | runtime= boto3.client('runtime.sagemaker') 10 | 11 | def transform_input(prompt: Dict, model_kwargs: Dict) -> bytes: 12 | input_str = json.dumps( 13 | { 14 | "inputs": [ 15 | [ 16 | {"role": "user", "content": prompt}, 17 | ] 18 | ], 19 | "parameters": model_kwargs, 20 | } 21 | ) 22 | 23 | return input_str.encode("utf-8") 24 | 25 | 26 | def call_llm(parameters, prompt): 27 | 28 | data = transform_input(prompt, parameters) 29 | 30 | response = runtime.invoke_endpoint(EndpointName=SAGEMAKER_ENDPOINT_NAME, 31 | ContentType='application/json', 32 | CustomAttributes="accept_eula=true", 33 | Body=data) 34 | 35 | generated_text = json.loads(response['Body'].read().decode()) 36 | 37 | return generated_text[0]["generation"]["content"] 38 | 39 | 40 | def lambda_handler(event, context): 41 | print("Event: ", json.dumps(event)) 42 | prompt = event["prompt"] 43 | parameters = event["parameters"] 44 | generated_text = call_llm(parameters, prompt) 45 | print("Result:", json.dumps(generated_text)) 46 | return { 47 | 'generated_text': generated_text 48 | } 49 | -------------------------------------------------------------------------------- /lambdas/ai21-llm/src/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 2 | # This file is licensed to you under the AWS Customer Agreement (the "License"). 3 | # You may not use this file except in compliance with the License. 4 | # A copy of the License is located at http://aws.amazon.com/agreement/ . 5 | # This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. 6 | # See the License for the specific language governing permissions and limitations under the License. 7 | 8 | from __future__ import print_function 9 | import urllib3 10 | import json 11 | 12 | SUCCESS = "SUCCESS" 13 | FAILED = "FAILED" 14 | 15 | http = urllib3.PoolManager() 16 | 17 | 18 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None): 19 | responseUrl = event['ResponseURL'] 20 | 21 | print(responseUrl) 22 | 23 | responseBody = { 24 | 'Status' : responseStatus, 25 | 'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name), 26 | 'PhysicalResourceId' : physicalResourceId or context.log_stream_name, 27 | 'StackId' : event['StackId'], 28 | 'RequestId' : event['RequestId'], 29 | 'LogicalResourceId' : event['LogicalResourceId'], 30 | 'NoEcho' : noEcho, 31 | 'Data' : responseData 32 | } 33 | 34 | json_responseBody = json.dumps(responseBody) 35 | 36 | print("Response body:") 37 | print(json_responseBody) 38 | 39 | headers = { 40 | 'content-type' : '', 41 | 'content-length' : str(len(json_responseBody)) 42 | } 43 | 44 | try: 45 | response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody) 46 | print("Status code:", response.status) 47 | 48 | except Exception as e: 49 | 50 | print("send(..) failed executing http.request(..):", e) 51 | -------------------------------------------------------------------------------- /lambdas/anthropic-llm/src/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 2 | # This file is licensed to you under the AWS Customer Agreement (the "License"). 3 | # You may not use this file except in compliance with the License. 4 | # A copy of the License is located at http://aws.amazon.com/agreement/ . 5 | # This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. 6 | # See the License for the specific language governing permissions and limitations under the License. 7 | 8 | from __future__ import print_function 9 | import urllib3 10 | import json 11 | 12 | SUCCESS = "SUCCESS" 13 | FAILED = "FAILED" 14 | 15 | http = urllib3.PoolManager() 16 | 17 | 18 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None): 19 | responseUrl = event['ResponseURL'] 20 | 21 | print(responseUrl) 22 | 23 | responseBody = { 24 | 'Status' : responseStatus, 25 | 'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name), 26 | 'PhysicalResourceId' : physicalResourceId or context.log_stream_name, 27 | 'StackId' : event['StackId'], 28 | 'RequestId' : event['RequestId'], 29 | 'LogicalResourceId' : event['LogicalResourceId'], 30 | 'NoEcho' : noEcho, 31 | 'Data' : responseData 32 | } 33 | 34 | json_responseBody = json.dumps(responseBody) 35 | 36 | print("Response body:") 37 | print(json_responseBody) 38 | 39 | headers = { 40 | 'content-type' : '', 41 | 'content-length' : str(len(json_responseBody)) 42 | } 43 | 44 | try: 45 | response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody) 46 | print("Status code:", response.status) 47 | 48 | except Exception as e: 49 | 50 | print("send(..) failed executing http.request(..):", e) 51 | -------------------------------------------------------------------------------- /lambdas/llama-2-13b-chat-llm/src/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 2 | # This file is licensed to you under the AWS Customer Agreement (the "License"). 3 | # You may not use this file except in compliance with the License. 4 | # A copy of the License is located at http://aws.amazon.com/agreement/ . 5 | # This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. 6 | # See the License for the specific language governing permissions and limitations under the License. 7 | 8 | from __future__ import print_function 9 | import urllib3 10 | import json 11 | 12 | SUCCESS = "SUCCESS" 13 | FAILED = "FAILED" 14 | 15 | http = urllib3.PoolManager() 16 | 17 | 18 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None): 19 | responseUrl = event['ResponseURL'] 20 | 21 | print(responseUrl) 22 | 23 | responseBody = { 24 | 'Status' : responseStatus, 25 | 'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name), 26 | 'PhysicalResourceId' : physicalResourceId or context.log_stream_name, 27 | 'StackId' : event['StackId'], 28 | 'RequestId' : event['RequestId'], 29 | 'LogicalResourceId' : event['LogicalResourceId'], 30 | 'NoEcho' : noEcho, 31 | 'Data' : responseData 32 | } 33 | 34 | json_responseBody = json.dumps(responseBody) 35 | 36 | print("Response body:") 37 | print(json_responseBody) 38 | 39 | headers = { 40 | 'content-type' : '', 41 | 'content-length' : str(len(json_responseBody)) 42 | } 43 | 44 | try: 45 | response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody) 46 | print("Status code:", response.status) 47 | 48 | except Exception as e: 49 | 50 | print("send(..) failed executing http.request(..):", e) 51 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 2 | # This file is licensed to you under the AWS Customer Agreement (the "License"). 3 | # You may not use this file except in compliance with the License. 4 | # A copy of the License is located at http://aws.amazon.com/agreement/ . 5 | # This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. 6 | # See the License for the specific language governing permissions and limitations under the License. 7 | 8 | from __future__ import print_function 9 | import urllib3 10 | import json 11 | 12 | SUCCESS = "SUCCESS" 13 | FAILED = "FAILED" 14 | 15 | http = urllib3.PoolManager() 16 | 17 | 18 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None): 19 | responseUrl = event['ResponseURL'] 20 | 21 | print(responseUrl) 22 | 23 | responseBody = { 24 | 'Status' : responseStatus, 25 | 'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name), 26 | 'PhysicalResourceId' : physicalResourceId or context.log_stream_name, 27 | 'StackId' : event['StackId'], 28 | 'RequestId' : event['RequestId'], 29 | 'LogicalResourceId' : event['LogicalResourceId'], 30 | 'NoEcho' : noEcho, 31 | 'Data' : responseData 32 | } 33 | 34 | json_responseBody = json.dumps(responseBody) 35 | 36 | print("Response body:") 37 | print(json_responseBody) 38 | 39 | headers = { 40 | 'content-type' : '', 41 | 'content-length' : str(len(json_responseBody)) 42 | } 43 | 44 | try: 45 | response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody) 46 | print("Status code:", response.status) 47 | 48 | except Exception as e: 49 | 50 | print("send(..) failed executing http.request(..):", e) 51 | -------------------------------------------------------------------------------- /lambdas/mistral-7b-instruct-chat-llm/src/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Amazon Web Services, Inc. or its affiliates. All Rights Reserved. 2 | # This file is licensed to you under the AWS Customer Agreement (the "License"). 3 | # You may not use this file except in compliance with the License. 4 | # A copy of the License is located at http://aws.amazon.com/agreement/ . 5 | # This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. 6 | # See the License for the specific language governing permissions and limitations under the License. 7 | 8 | from __future__ import print_function 9 | import urllib3 10 | import json 11 | 12 | SUCCESS = "SUCCESS" 13 | FAILED = "FAILED" 14 | 15 | http = urllib3.PoolManager() 16 | 17 | 18 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None): 19 | responseUrl = event['ResponseURL'] 20 | 21 | print(responseUrl) 22 | 23 | responseBody = { 24 | 'Status' : responseStatus, 25 | 'Reason' : reason or "See the details in CloudWatch Log Stream: {}".format(context.log_stream_name), 26 | 'PhysicalResourceId' : physicalResourceId or context.log_stream_name, 27 | 'StackId' : event['StackId'], 28 | 'RequestId' : event['RequestId'], 29 | 'LogicalResourceId' : event['LogicalResourceId'], 30 | 'NoEcho' : noEcho, 31 | 'Data' : responseData 32 | } 33 | 34 | json_responseBody = json.dumps(responseBody) 35 | 36 | print("Response body:") 37 | print(json_responseBody) 38 | 39 | headers = { 40 | 'content-type' : '', 41 | 'content-length' : str(len(json_responseBody)) 42 | } 43 | 44 | try: 45 | response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody) 46 | print("Status code:", response.status) 47 | 48 | except Exception as e: 49 | 50 | print("send(..) failed executing http.request(..):", e) 51 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/embeddings.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | 5 | # Defaults 6 | DEFAULT_MODEL_ID = os.environ.get("DEFAULT_MODEL_ID","amazon.titan-embed-text-v1") 7 | AWS_REGION = os.environ["AWS_REGION_OVERRIDE"] if "AWS_REGION_OVERRIDE" in os.environ else os.environ["AWS_REGION"] 8 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", f'https://bedrock-runtime.{AWS_REGION}.amazonaws.com') 9 | EMBEDDING_MAX_WORDS = os.environ.get("EMBEDDING_MAX_WORDS") or 6000 # limit 8k token ~ 6k words 10 | 11 | # global variables - avoid creating a new client for every request 12 | client = None 13 | 14 | # limit number of words to avoid exceeding model token limit 15 | def truncate_text(text, n=500): 16 | words = text.split() 17 | if (len(words) > n): 18 | print(f"Truncating input text from {len(words)} to {n} words") 19 | truncated_words = words[:n] 20 | truncated_text = " ".join(truncated_words) 21 | return truncated_text 22 | else: 23 | return text 24 | 25 | def get_client(): 26 | print("Connecting to Bedrock Service: ", ENDPOINT_URL) 27 | client = boto3.client(service_name='bedrock-runtime', region_name=AWS_REGION, endpoint_url=ENDPOINT_URL) 28 | return client 29 | 30 | """ 31 | Example Test Event: 32 | { 33 | "inputText": "Why is the sky blue?" 34 | } 35 | """ 36 | def lambda_handler(event, context): 37 | print("Event:", json.dumps(event)) 38 | global client 39 | modelId = DEFAULT_MODEL_ID 40 | max_words = EMBEDDING_MAX_WORDS 41 | text = truncate_text(event["inputText"].strip(), int(max_words)) 42 | body = json.dumps({"inputText": text}) 43 | if (client is None): 44 | client = get_client() 45 | response = client.invoke_model(body=body, modelId=modelId, accept='application/json', contentType='application/json') 46 | response_body = json.loads(response.get('body').read()) 47 | print("Embeddings length:", len(response_body["embedding"])) 48 | return response_body 49 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/testModel.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | import cfnresponse 5 | import llm 6 | 7 | """ 8 | Example Test Event: 9 | { 10 | "RequestType": "Create", 11 | "ResourceProperties": { 12 | "EmbeddingsModelId": "amazon.titan-embed-text-v1", 13 | "TextModelId": "amazon.titan-text-express-v1" 14 | } 15 | } 16 | """ 17 | def lambda_handler(event, context): 18 | print("Event: ", json.dumps(event)) 19 | global client 20 | status = cfnresponse.SUCCESS 21 | responseData = {} 22 | reason = "Success" 23 | modelId = "" 24 | if event['RequestType'] != 'Delete': 25 | prompt = "\n\nHuman: Why is the sky blue?\n\nAssistant:" 26 | try: 27 | embeddingsModelId = event['ResourceProperties'].get('EmbeddingsModelId', '') 28 | llmModelId = event['ResourceProperties'].get('LLMModelId', '') 29 | client = llm.get_client() 30 | # Test EmbeddingsModel 31 | modelId = embeddingsModelId 32 | body = json.dumps({"inputText": prompt}) 33 | print(f"Testing {modelId} - {body}") 34 | client.invoke_model(body=body, modelId=modelId, accept='application/json', contentType='application/json') 35 | # Test LLMModel 36 | modelId = llmModelId 37 | parameters = { 38 | "modelId": modelId, 39 | "temperature": 0 40 | } 41 | print(f"Testing {modelId}") 42 | llm.call_llm(parameters, prompt) 43 | except Exception as e: 44 | status = cfnresponse.FAILED 45 | reason = f"Exception thrown testing ModelId='{modelId}'. Check that Amazon Bedrock is available in your region, and that models ('{embeddingsModelId}' and '{llmModelId}') are activated in your Amazon Bedrock account - {e}" 46 | print(f"Status: {status}, Reason: {reason}") 47 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /lambdas/llama-2-13b-chat-llm/src/settings.py: -------------------------------------------------------------------------------- 1 | import cfnresponse 2 | import json 3 | 4 | # Default prompt temnplates 5 | LLAMA2_GENERATE_QUERY_PROMPT_TEMPLATE = """

Human: Here is a chat history in tags:

{history}

Human: And here is a follow up question or statement from the human in tags:

{input}

Human: Rephrase the follow up question or statement as a standalone question or statement that makes sense without reading the chat history.

Assistant: Here is the rephrased follow up question or statement:""" 6 | LLAMA2_QA_PROMPT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Documents: {context} Instruction: Based on the above documents, provide a detailed answer for {query} Answer "don't know" if not present in the document. Solution:""" 7 | 8 | def getModelSettings(model): 9 | params = { 10 | "temperature":0.1, 11 | "max_new_tokens":256, 12 | "top_p":0.5 13 | } 14 | settings = { 15 | 'LLM_GENERATE_QUERY_MODEL_PARAMS': json.dumps(params), 16 | 'LLM_QA_MODEL_PARAMS': json.dumps(params), 17 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': LLAMA2_GENERATE_QUERY_PROMPT_TEMPLATE, 18 | 'LLM_QA_PROMPT_TEMPLATE': LLAMA2_QA_PROMPT_TEMPLATE 19 | } 20 | 21 | return settings 22 | 23 | def lambda_handler(event, context): 24 | print("Event: ", json.dumps(event)) 25 | status = cfnresponse.SUCCESS 26 | responseData = {} 27 | reason = "" 28 | if event['RequestType'] != 'Delete': 29 | try: 30 | model = event['ResourceProperties'].get('Model', '') 31 | responseData = getModelSettings(model) 32 | except Exception as e: 33 | print(e) 34 | status = cfnresponse.FAILED 35 | reason = f"Exception thrown: {e}" 36 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /lambdas/mistral-7b-instruct-chat-llm/src/settings.py: -------------------------------------------------------------------------------- 1 | import cfnresponse 2 | import json 3 | 4 | # Default prompt temnplates 5 | MISTRAL_GENERATE_QUERY_PROMPT_TEMPLATE = """[INST] You are a helpful assistant.
Here is a chat history in tags:

{history}


And here is a follow up question or statement from the human in tags:

{input}
[/INST]

[INST]Rephrase the follow up question or statement as a standalone question or statement that makes sense without reading the chat history.[/INST]""" 6 | MISTRAL_QA_PROMPT_TEMPLATE = """[INST]You are an AI chatbot. Carefully read the following context and conversation history and then provide a short answer to question at the end. If the answer cannot be determined from the history or the context, reply saying "Sorry, I don\'t know".

Context: {context}

History:
{history}

{input}[/INST]""" 7 | 8 | def getModelSettings(model): 9 | params = { 10 | "model":model, 11 | "temperature":0.1, 12 | "max_new_tokens":512, 13 | "top_p":0.5, 14 | "top_k":50, 15 | "do_sample":True, 16 | } 17 | settings = { 18 | 'LLM_GENERATE_QUERY_MODEL_PARAMS': json.dumps(params), 19 | 'LLM_QA_MODEL_PARAMS': json.dumps(params), 20 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': MISTRAL_GENERATE_QUERY_PROMPT_TEMPLATE, 21 | 'LLM_QA_PROMPT_TEMPLATE': MISTRAL_QA_PROMPT_TEMPLATE 22 | } 23 | 24 | return settings 25 | 26 | def lambda_handler(event, context): 27 | print("Event: ", json.dumps(event)) 28 | status = cfnresponse.SUCCESS 29 | responseData = {} 30 | reason = "" 31 | if event['RequestType'] != 'Delete': 32 | try: 33 | model = event['ResourceProperties'].get('Model', '') 34 | responseData = getModelSettings(model) 35 | except Exception as e: 36 | print(e) 37 | status = cfnresponse.FAILED 38 | reason = f"Exception thrown: {e}" 39 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /lambdas/ai21-llm/src/settings.py: -------------------------------------------------------------------------------- 1 | import cfnresponse 2 | import json 3 | 4 | # Default prompt temnplates 5 | AI21_GENERATE_QUERY_PROMPT_TEMPLATE = """

Human: Here is a chat history in tags:

{history}

Human: And here is a follow up question or statement from the human in tags:

{input}

Human: Rephrase the follow up question or statement as a standalone question or statement that makes sense without reading the chat history.

Assistant: Here is the rephrased follow up question or statement:""" 6 | AI21_QA_PROMPT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Documents: {context} Instruction: Based on the above documents, provide a detailed answer for {query} Answer "don't know" if not present in the document. Solution:""" 7 | 8 | def getModelSettings(modelType): 9 | params = { 10 | "model_type": modelType, 11 | "temperature": 0, 12 | "maxTokens": 256, 13 | "minTokens": 0, 14 | "topP": 1 15 | } 16 | lambdahook_args = {"Prefix":"LLM Answer:", "Model_params": params} 17 | settings = { 18 | 'LLM_GENERATE_QUERY_MODEL_PARAMS': json.dumps(params), 19 | 'LLM_QA_MODEL_PARAMS': json.dumps(params), 20 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': AI21_GENERATE_QUERY_PROMPT_TEMPLATE, 21 | 'LLM_QA_PROMPT_TEMPLATE': AI21_QA_PROMPT_TEMPLATE, 22 | 'QNAITEM_LAMBDAHOOK_ARGS': json.dumps(lambdahook_args) 23 | } 24 | 25 | return settings 26 | 27 | def lambda_handler(event, context): 28 | print("Event: ", json.dumps(event)) 29 | status = cfnresponse.SUCCESS 30 | responseData = {} 31 | reason = "" 32 | if event['RequestType'] != 'Delete': 33 | try: 34 | modelType = event['ResourceProperties'].get('ModelType', '') 35 | responseData = getModelSettings(modelType) 36 | except Exception as e: 37 | print(e) 38 | status = cfnresponse.FAILED 39 | reason = f"Exception thrown: {e}" 40 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /lambdas/anthropic-llm/src/llm.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import urllib3 4 | import boto3 5 | from botocore.exceptions import ClientError 6 | 7 | # Defaults 8 | API_KEY_SECRET_NAME = os.environ['API_KEY_SECRET_NAME'] 9 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", "https://api.anthropic.com/v1/complete") 10 | DEFAULT_MODEL = os.environ.get("DEFAULT_MODEL","claude-instant-1") 11 | MAX_TOKENS_TO_SAMPLE = 256 12 | 13 | def get_secret(secret_name): 14 | print("Getting API key from Secrets Manager") 15 | secrets_client = boto3.client('secretsmanager') 16 | try: 17 | response = secrets_client.get_secret_value( 18 | SecretId=secret_name 19 | ) 20 | except ClientError as e: 21 | raise e 22 | api_key = response['SecretString'] 23 | return api_key 24 | 25 | def call_llm(parameters, prompt): 26 | api_key = get_secret(API_KEY_SECRET_NAME) 27 | # # Default parameters 28 | data = { 29 | "max_tokens_to_sample": MAX_TOKENS_TO_SAMPLE, 30 | "model": DEFAULT_MODEL 31 | } 32 | data.update(parameters) 33 | data["prompt"] = prompt 34 | headers = { 35 | "anthropic-version": "2023-06-01", 36 | "x-api-key": api_key, 37 | "content-type": "application/json", 38 | "accept": "application/json" 39 | } 40 | http = urllib3.PoolManager() 41 | try: 42 | response = http.request( 43 | "POST", 44 | ENDPOINT_URL, 45 | body=json.dumps(data), 46 | headers=headers 47 | ) 48 | if response.status != 200: 49 | raise Exception(f"Error: {response.status} - {response.data}") 50 | generated_text = json.loads(response.data)["completion"].strip() 51 | return generated_text 52 | except Exception as err: 53 | print(err) 54 | raise 55 | 56 | """ 57 | Example Test Event: 58 | { 59 | "prompt": "\n\nHuman:Why is the sky blue?\n\nAssistant:", 60 | "parameters": { 61 | "model": "claude-instant-1", 62 | "temperature": 0 63 | } 64 | } 65 | For supported parameters, see the link to Anthropic docs: https://docs.anthropic.com/claude/reference/complete_post 66 | """ 67 | def lambda_handler(event, context): 68 | print("Event: ", json.dumps(event)) 69 | global secret 70 | prompt = event["prompt"] 71 | parameters = event["parameters"] 72 | generated_text = call_llm(parameters, prompt) 73 | print("Result:", json.dumps(generated_text)) 74 | return { 75 | 'generated_text': generated_text 76 | } 77 | -------------------------------------------------------------------------------- /lambdas/ai21-llm/src/llm.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import urllib3 4 | import boto3 5 | from botocore.exceptions import ClientError 6 | 7 | # Defaults 8 | API_KEY_SECRET_NAME = os.environ['API_KEY_SECRET_NAME'] 9 | DEFAULT_MODEL_TYPE = os.environ.get("DEFAULT_MODEL_TYPE","j2-mid") 10 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", "https://api.ai21.com/studio/v1/{MODEL_TYPE}/complete") 11 | MAX_TOKENS = 256 12 | 13 | def get_secret(secret_name): 14 | print("Getting API key from Secrets Manager") 15 | secrets_client = boto3.client('secretsmanager') 16 | try: 17 | response = secrets_client.get_secret_value( 18 | SecretId=secret_name 19 | ) 20 | except ClientError as e: 21 | raise e 22 | api_key = response['SecretString'] 23 | return api_key 24 | 25 | def call_llm(parameters, prompt): 26 | api_key = get_secret(API_KEY_SECRET_NAME) 27 | # Default parameters 28 | data = { 29 | "maxTokens": MAX_TOKENS 30 | } 31 | data.update(parameters) 32 | data["prompt"] = prompt 33 | headers = { 34 | "Authorization": f"Bearer {api_key}", 35 | "content-type": "application/json", 36 | "accept": "application/json" 37 | } 38 | # Endpoint URL is a template, so we need to replace the model type with the one specified in parameters 39 | endpoint_url = ENDPOINT_URL.format(MODEL_TYPE=parameters.get("model_type", DEFAULT_MODEL_TYPE)) 40 | http = urllib3.PoolManager() 41 | try: 42 | response = http.request( 43 | "POST", 44 | endpoint_url, 45 | body=json.dumps(data), 46 | headers=headers 47 | ) 48 | if response.status != 200: 49 | raise Exception(f"Error: {response.status} - {response.data}") 50 | generated_text = json.loads(response.data)["completions"][0]["data"]["text"].strip() 51 | return generated_text 52 | except Exception as err: 53 | print(err) 54 | raise 55 | 56 | """ 57 | Example Test Event: 58 | { 59 | "prompt": "Why is the sky blue?\nAssistant:", 60 | "parameters": { 61 | "model_type": "j2-mid", 62 | "temperature": 0 63 | } 64 | } 65 | For supported parameters, see the link to AI21 docs: https://docs.ai21.com/reference/j2-complete-ref 66 | """ 67 | def lambda_handler(event, context): 68 | print("Event: ", json.dumps(event)) 69 | global secret 70 | prompt = event["prompt"] 71 | parameters = event["parameters"] 72 | generated_text = call_llm(parameters, prompt) 73 | print("Result:", json.dumps(generated_text)) 74 | return { 75 | 'generated_text': generated_text 76 | } 77 | -------------------------------------------------------------------------------- /lambdas/anthropic-llm/src/settings.py: -------------------------------------------------------------------------------- 1 | import cfnresponse 2 | import json 3 | 4 | # Default prompt temnplates 5 | ANTHROPIC_GENERATE_QUERY_PROMPT_TEMPLATE = """

Human: Here is a chat history in tags:

{history}

Human: And here is a follow up question or statement from the human in tags:

{input}

Human: Rephrase the follow up question or statement as a standalone question or statement that makes sense without reading the chat history.

Assistant: Here is the rephrased follow up question or statement:""" 6 | ANTHROPIC_QA_PROMPT_TEMPLATE = """

Human: You are a friendly AI assistant. You provide answers only based on the provided reference passages. Here are reference passages in tags:

{context}

If the references contain the information needed to respond, then write a confident response in under 50 words, quoting the relevant references.
Otherwise, if you can make an informed guess based on the reference passages, then write a less condident response in under 50 words, stating your assumptions.
Finally, if the references do not have any relevant information, then respond saying \\"Sorry, I don't know\\".

{query}

Assistant: According to the reference passages, in under 50 words:""" 7 | 8 | def getModelSettings(model): 9 | params = { 10 | "model": model, 11 | "temperature": 0, 12 | "maxTokens": 256, 13 | "minTokens": 0, 14 | "topP": 1 15 | } 16 | # claude-3 message API params are slightly different 17 | provider = modelId.split(".")[0] 18 | if provider == "anthropic": 19 | if modelId.startswith("anthropic.claude-3"): 20 | params = { 21 | "model": model, 22 | "temperature": 0, 23 | "max_tokens": 256, 24 | "top_p": 1, 25 | "system": "You are a helpful AI assistant." 26 | } 27 | 28 | settings = { 29 | 'LLM_GENERATE_QUERY_MODEL_PARAMS': json.dumps(params), 30 | 'LLM_QA_MODEL_PARAMS': json.dumps(params), 31 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': ANTHROPIC_GENERATE_QUERY_PROMPT_TEMPLATE, 32 | 'LLM_QA_PROMPT_TEMPLATE': ANTHROPIC_QA_PROMPT_TEMPLATE 33 | } 34 | 35 | return settings 36 | 37 | def lambda_handler(event, context): 38 | print("Event: ", json.dumps(event)) 39 | status = cfnresponse.SUCCESS 40 | responseData = {} 41 | reason = "" 42 | if event['RequestType'] != 'Delete': 43 | try: 44 | model = event['ResourceProperties'].get('Model', '') 45 | responseData = getModelSettings(model) 46 | except Exception as e: 47 | print(e) 48 | status = cfnresponse.FAILED 49 | reason = f"Exception thrown: {e}" 50 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. -------------------------------------------------------------------------------- /lambdas/ai21-llm/src/lambdahook.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import urllib3 4 | import boto3 5 | from botocore.exceptions import ClientError 6 | 7 | # Defaults 8 | API_KEY_SECRET_NAME = os.environ['API_KEY_SECRET_NAME'] 9 | DEFAULT_MODEL_TYPE = os.environ.get("DEFAULT_MODEL_TYPE","j2-mid") 10 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", "https://api.ai21.com/studio/v1/{MODEL_TYPE}/complete") 11 | MAX_TOKENS = 256 12 | 13 | def get_secret(secret_name): 14 | print("Getting API key from Secrets Manager") 15 | secrets_client = boto3.client('secretsmanager') 16 | try: 17 | response = secrets_client.get_secret_value( 18 | SecretId=secret_name 19 | ) 20 | except ClientError as e: 21 | raise e 22 | api_key = response['SecretString'] 23 | return api_key 24 | 25 | def get_llm_response(parameters, prompt): 26 | api_key = get_secret(API_KEY_SECRET_NAME) 27 | # Default parameters 28 | data = { 29 | "maxTokens": MAX_TOKENS 30 | } 31 | data.update(parameters) 32 | data["prompt"] = prompt 33 | headers = { 34 | "Authorization": f"Bearer {api_key}", 35 | "content-type": "application/json", 36 | "accept": "application/json" 37 | } 38 | # Endpoint URL is a template, so we need to replace the model type with the one specified in parameters 39 | endpoint_url = ENDPOINT_URL.format(MODEL_TYPE=parameters.get("model_type", DEFAULT_MODEL_TYPE)) 40 | http = urllib3.PoolManager() 41 | try: 42 | response = http.request( 43 | "POST", 44 | endpoint_url, 45 | body=json.dumps(data), 46 | headers=headers 47 | ) 48 | if response.status != 200: 49 | raise Exception(f"Error: {response.status} - {response.data}") 50 | generated_text = json.loads(response.data)["completions"][0]["data"]["text"].strip() 51 | return generated_text 52 | except Exception as err: 53 | print(err) 54 | raise 55 | 56 | def get_args_from_lambdahook_args(event): 57 | parameters = {} 58 | lambdahook_args_list = event["res"]["result"].get("args",[]) 59 | print("LambdaHook args: ", lambdahook_args_list) 60 | if len(lambdahook_args_list): 61 | try: 62 | parameters = json.loads(lambdahook_args_list[0]) 63 | except Exception as e: 64 | print(f"Failed to parse JSON:", lambdahook_args_list[0], e) 65 | print("..continuing") 66 | return parameters 67 | 68 | def format_response(event, llm_response, prefix): 69 | # set plaintext, markdown, & ssml response 70 | if prefix in ["None", "N/A", "Empty"]: 71 | prefix = None 72 | plainttext = llm_response 73 | markdown = llm_response 74 | ssml = llm_response 75 | if prefix: 76 | plainttext = f"{prefix}\n\n{plainttext}" 77 | markdown = f"**{prefix}**\n\n{markdown}" 78 | # add plaintext, markdown, and ssml fields to event.res 79 | event["res"]["message"] = plainttext 80 | event["res"]["session"]["appContext"] = { 81 | "altMessages": { 82 | "markdown": markdown, 83 | "ssml": ssml 84 | } 85 | } 86 | #TODO - can we determine when LLM has a good answer or not? 87 | #For now, always assume it's a good answer. 88 | #QnAbot sets session attribute qnabot_gotanswer True when got_hits > 0 89 | event["res"]["got_hits"] = 1 90 | return event 91 | 92 | def lambda_handler(event, context): 93 | print("Received event: %s" % json.dumps(event)) 94 | # args = {"Prefix:"", "Model_params":{"max_tokens":256}, "Prompt":""} 95 | args = get_args_from_lambdahook_args(event) 96 | # prompt set from args, or from req.question if not specified in args. 97 | prompt = args.get("Prompt", event["req"]["question"]) 98 | model_params = args.get("Model_params",{}) 99 | llm_response = get_llm_response(model_params, prompt) 100 | prefix = args.get("Prefix","LLM Answer:") 101 | event = format_response(event, llm_response, prefix) 102 | print("Returning response: %s" % json.dumps(event)) 103 | return event 104 | -------------------------------------------------------------------------------- /lambdas/llama-2-13b-chat-llm/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: QnABot on AWS LLM Plugin for Llama-2-13b-chat model deployed using SageMaker JumpStart (https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/) 3 | 4 | Parameters: 5 | SageMakerEndpointName: 6 | Type: String 7 | Description: Get the SageMaker Endpoint Name for Llama 2 13b Chat LLM Model from Amazon SageMaker console > Choose Inference in the left panel > Choose Endpoints. Refer to this link on how to deploy the Llama-2-chat model in SageMaker JumpStart - https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/ 8 | Default: 'jumpstart-dft-meta-textgeneration-llama-2-13b-f' 9 | 10 | Resources: 11 | LambdaFunctionRole: 12 | Type: AWS::IAM::Role 13 | Properties: 14 | AssumeRolePolicyDocument: 15 | Version: '2012-10-17' 16 | Statement: 17 | - Effect: Allow 18 | Principal: 19 | Service: lambda.amazonaws.com 20 | Action: sts:AssumeRole 21 | ManagedPolicyArns: 22 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 23 | Policies: 24 | - PolicyDocument: 25 | Version: 2012-10-17 26 | Statement: 27 | - Effect: Allow 28 | Action: 29 | - "sagemaker:InvokeEndpoint" 30 | Resource: 31 | - !Sub arn:${AWS::Partition}:sagemaker:${AWS::Region}:${AWS::AccountId}:endpoint/${SageMakerEndpointName} 32 | PolicyName: SageMakerPolicy 33 | 34 | LambdaFunction: 35 | Type: AWS::Lambda::Function 36 | Properties: 37 | Handler: "llm.lambda_handler" 38 | Role: !GetAtt 'LambdaFunctionRole.Arn' 39 | MemorySize: 128 40 | Timeout: 60 41 | Runtime: python3.10 42 | Environment: 43 | Variables: 44 | SAGEMAKER_ENDPOINT_NAME: !Ref SageMakerEndpointName 45 | Code: ./src 46 | Metadata: 47 | cfn_nag: 48 | rules_to_suppress: 49 | - id: W89 50 | reason: Lambda function is not communicating with any VPC resources. 51 | - id: W92 52 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 53 | 54 | OutputSettingsFunctionRole: 55 | Type: AWS::IAM::Role 56 | Properties: 57 | AssumeRolePolicyDocument: 58 | Version: '2012-10-17' 59 | Statement: 60 | - Effect: Allow 61 | Principal: 62 | Service: lambda.amazonaws.com 63 | Action: sts:AssumeRole 64 | ManagedPolicyArns: 65 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 66 | 67 | OutputSettingsFunction: 68 | Type: AWS::Lambda::Function 69 | Properties: 70 | Handler: settings.lambda_handler 71 | Role: !GetAtt 'OutputSettingsFunctionRole.Arn' 72 | Runtime: python3.10 73 | Timeout: 10 74 | MemorySize: 128 75 | Code: ./src 76 | Metadata: 77 | cfn_nag: 78 | rules_to_suppress: 79 | - id: W89 80 | reason: Lambda function is not communicating with any VPC resources. 81 | - id: W92 82 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 83 | 84 | OutputSettings: 85 | Type: Custom::OutputSettings 86 | Properties: 87 | ServiceToken: !GetAtt OutputSettingsFunction.Arn 88 | Model: !Ref SageMakerEndpointName 89 | 90 | Outputs: 91 | LLMLambdaArn: 92 | Description: Lambda function ARN (use for QnABot param "LLMLambdaArn") 93 | Value: !GetAtt LambdaFunction.Arn 94 | 95 | QnABotSettingGenerateQueryPromptTemplate: 96 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_PROMPT_TEMPLATE" 97 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_PROMPT_TEMPLATE 98 | 99 | QnABotSettingGenerateQueryModelParams: 100 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_MODEL_PARAMS" 101 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_MODEL_PARAMS 102 | 103 | QnABotSettingQAPromptTemplate: 104 | Description: QnABot Designer Setting "LLM_QA_PROMPT_TEMPLATE" 105 | Value: !GetAtt OutputSettings.LLM_QA_PROMPT_TEMPLATE 106 | 107 | QnABotSettingQAModelParams: 108 | Description: QnABot Designer Setting "LLM_QA_MODEL_PARAMS" 109 | Value: !GetAtt OutputSettings.LLM_QA_MODEL_PARAMS -------------------------------------------------------------------------------- /lambdas/mistral-7b-instruct-chat-llm/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: QnABot on AWS LLM Plugin for Mistral-7b-instruct-chat model deployed using SageMaker JumpStart (https://aws.amazon.com/blogs/machine-learning/mistral-7b-foundation-models-from-mistral-ai-are-now-available-in-amazon-sagemaker-jumpstart/) 3 | 4 | Parameters: 5 | SageMakerEndpointName: 6 | Type: String 7 | Description: Get the SageMaker Endpoint Name for Mistral 7b Instruct Chat LLM Model from Amazon SageMaker console > Choose Inference in the left panel > Choose Endpoints. Refer to this link on how to deploy the Mistral 7B Instruct model in SageMaker JumpStart - httpshttps://aws.amazon.com/blogs/machine-learning/mistral-7b-foundation-models-from-mistral-ai-are-now-available-in-amazon-sagemaker-jumpstart/ 8 | Default: 'jumpstart-dft-hf-llm-mistral-7b-instruct' 9 | 10 | Resources: 11 | LambdaFunctionRole: 12 | Type: AWS::IAM::Role 13 | Properties: 14 | AssumeRolePolicyDocument: 15 | Version: '2012-10-17' 16 | Statement: 17 | - Effect: Allow 18 | Principal: 19 | Service: lambda.amazonaws.com 20 | Action: sts:AssumeRole 21 | ManagedPolicyArns: 22 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 23 | Policies: 24 | - PolicyDocument: 25 | Version: 2012-10-17 26 | Statement: 27 | - Effect: Allow 28 | Action: 29 | - "sagemaker:InvokeEndpoint" 30 | Resource: 31 | - !Sub arn:${AWS::Partition}:sagemaker:${AWS::Region}:${AWS::AccountId}:endpoint/${SageMakerEndpointName} 32 | PolicyName: SageMakerPolicy 33 | 34 | LambdaFunction: 35 | Type: AWS::Lambda::Function 36 | Properties: 37 | Handler: "llm.lambda_handler" 38 | Role: !GetAtt 'LambdaFunctionRole.Arn' 39 | MemorySize: 128 40 | Timeout: 60 41 | Runtime: python3.10 42 | Environment: 43 | Variables: 44 | SAGEMAKER_ENDPOINT_NAME: !Ref SageMakerEndpointName 45 | Code: ./src 46 | Metadata: 47 | cfn_nag: 48 | rules_to_suppress: 49 | - id: W89 50 | reason: Lambda function is not communicating with any VPC resources. 51 | - id: W92 52 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 53 | 54 | OutputSettingsFunctionRole: 55 | Type: AWS::IAM::Role 56 | Properties: 57 | AssumeRolePolicyDocument: 58 | Version: '2012-10-17' 59 | Statement: 60 | - Effect: Allow 61 | Principal: 62 | Service: lambda.amazonaws.com 63 | Action: sts:AssumeRole 64 | ManagedPolicyArns: 65 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 66 | 67 | OutputSettingsFunction: 68 | Type: AWS::Lambda::Function 69 | Properties: 70 | Handler: settings.lambda_handler 71 | Role: !GetAtt 'OutputSettingsFunctionRole.Arn' 72 | Runtime: python3.10 73 | Timeout: 10 74 | MemorySize: 128 75 | Code: ./src 76 | Metadata: 77 | cfn_nag: 78 | rules_to_suppress: 79 | - id: W89 80 | reason: Lambda function is not communicating with any VPC resources. 81 | - id: W92 82 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 83 | 84 | OutputSettings: 85 | Type: Custom::OutputSettings 86 | Properties: 87 | ServiceToken: !GetAtt OutputSettingsFunction.Arn 88 | Model: !Ref SageMakerEndpointName 89 | 90 | Outputs: 91 | LLMLambdaArn: 92 | Description: Lambda function ARN (use for QnABot param "LLMLambdaArn") 93 | Value: !GetAtt LambdaFunction.Arn 94 | 95 | QnABotSettingGenerateQueryPromptTemplate: 96 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_PROMPT_TEMPLATE" 97 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_PROMPT_TEMPLATE 98 | 99 | QnABotSettingGenerateQueryModelParams: 100 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_MODEL_PARAMS" 101 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_MODEL_PARAMS 102 | 103 | QnABotSettingQAPromptTemplate: 104 | Description: QnABot Designer Setting "LLM_QA_PROMPT_TEMPLATE" 105 | Value: !GetAtt OutputSettings.LLM_QA_PROMPT_TEMPLATE 106 | 107 | QnABotSettingQAModelParams: 108 | Description: QnABot Designer Setting "LLM_QA_MODEL_PARAMS" 109 | Value: !GetAtt OutputSettings.LLM_QA_MODEL_PARAMS -------------------------------------------------------------------------------- /lambdas/anthropic-llm/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Description: QnABot on AWS LLM Plugin for Anthropic - v0.1.2 3 | 4 | Parameters: 5 | APIKey: 6 | Type: String 7 | Description: Anthropic API Key (stored in Secrets Manager - see stack Outputs) 8 | Default: '' 9 | NoEcho: true 10 | 11 | LLMModel: 12 | Type: String 13 | Default: claude-instant-1 14 | AllowedValues: 15 | - claude-instant-1 16 | - claude-1 17 | - claude-2 18 | Description: Anthropic LLM Model 19 | 20 | Resources: 21 | ApiKeySecret: 22 | Type: AWS::SecretsManager::Secret 23 | Properties: 24 | Description: API Key 25 | Name: !Ref AWS::StackName 26 | SecretString: !Ref APIKey 27 | Metadata: 28 | cfn_nag: 29 | rules_to_suppress: 30 | - id: W77 31 | reason: No requirement for custom CMK, secret will not be shared cross-account. 32 | 33 | LambdaFunctionRole: 34 | Type: AWS::IAM::Role 35 | Properties: 36 | AssumeRolePolicyDocument: 37 | Version: '2012-10-17' 38 | Statement: 39 | - Effect: Allow 40 | Principal: 41 | Service: lambda.amazonaws.com 42 | Action: sts:AssumeRole 43 | ManagedPolicyArns: 44 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 45 | Policies: 46 | - PolicyDocument: 47 | Version: 2012-10-17 48 | Statement: 49 | - Effect: Allow 50 | Action: 51 | - 'secretsmanager:GetResourcePolicy' 52 | - 'secretsmanager:GetSecretValue' 53 | - 'secretsmanager:DescribeSecret' 54 | - 'secretsmanager:ListSecretVersionIds' 55 | Resource: !Ref ApiKeySecret 56 | PolicyName: SecretsManagerPolicy 57 | 58 | LambdaFunction: 59 | Type: AWS::Lambda::Function 60 | Properties: 61 | Handler: "llm.lambda_handler" 62 | Role: !GetAtt 'LambdaFunctionRole.Arn' 63 | MemorySize: 128 64 | Timeout: 60 65 | Runtime: python3.10 66 | Environment: 67 | Variables: 68 | API_KEY_SECRET_NAME: !Ref AWS::StackName 69 | Code: ./src 70 | Metadata: 71 | cfn_nag: 72 | rules_to_suppress: 73 | - id: W89 74 | reason: Lambda function is not communicating with any VPC resources. 75 | - id: W92 76 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 77 | 78 | OutputSettingsFunctionRole: 79 | Type: AWS::IAM::Role 80 | Properties: 81 | AssumeRolePolicyDocument: 82 | Version: '2012-10-17' 83 | Statement: 84 | - Effect: Allow 85 | Principal: 86 | Service: lambda.amazonaws.com 87 | Action: sts:AssumeRole 88 | ManagedPolicyArns: 89 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 90 | 91 | OutputSettingsFunction: 92 | Type: AWS::Lambda::Function 93 | Properties: 94 | Handler: settings.lambda_handler 95 | Role: !GetAtt 'OutputSettingsFunctionRole.Arn' 96 | Runtime: python3.10 97 | Timeout: 10 98 | MemorySize: 128 99 | Code: ./src 100 | Metadata: 101 | cfn_nag: 102 | rules_to_suppress: 103 | - id: W89 104 | reason: Lambda function is not communicating with any VPC resources. 105 | - id: W92 106 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 107 | 108 | 109 | OutputSettings: 110 | Type: Custom::OutputSettings 111 | Properties: 112 | ServiceToken: !GetAtt OutputSettingsFunction.Arn 113 | Model: !Ref LLMModel 114 | 115 | Outputs: 116 | APIKeySecret: 117 | Description: Link to Secrets Manager console to input API Key 118 | Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/secretsmanager/secret?region=${AWS::Region}&name=${AWS::StackName}" 119 | 120 | LLMLambdaArn: 121 | Description: Lambda function ARN (use for QnABot param "LLMLambdaArn") 122 | Value: !GetAtt LambdaFunction.Arn 123 | 124 | QnABotSettingGenerateQueryPromptTemplate: 125 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_PROMPT_TEMPLATE" 126 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_PROMPT_TEMPLATE 127 | 128 | QnABotSettingGenerateQueryModelParams: 129 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_MODEL_PARAMS" 130 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_MODEL_PARAMS 131 | 132 | QnABotSettingQAPromptTemplate: 133 | Description: QnABot Designer Setting "LLM_QA_PROMPT_TEMPLATE" 134 | Value: !GetAtt OutputSettings.LLM_QA_PROMPT_TEMPLATE 135 | 136 | QnABotSettingQAModelParams: 137 | Description: QnABot Designer Setting "LLM_QA_MODEL_PARAMS" 138 | Value: !GetAtt OutputSettings.LLM_QA_MODEL_PARAMS -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/llm.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | 5 | # Defaults 6 | DEFAULT_MODEL_ID = os.environ.get("DEFAULT_MODEL_ID","anthropic.claude-instant-v1") 7 | AWS_REGION = os.environ["AWS_REGION_OVERRIDE"] if "AWS_REGION_OVERRIDE" in os.environ else os.environ["AWS_REGION"] 8 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", f'https://bedrock-runtime.{AWS_REGION}.amazonaws.com') 9 | DEFAULT_MAX_TOKENS = 256 10 | 11 | # global variables - avoid creating a new client for every request 12 | client = None 13 | 14 | def get_client(): 15 | print("Connecting to Bedrock Service: ", ENDPOINT_URL) 16 | client = boto3.client(service_name='bedrock-runtime', region_name=AWS_REGION, endpoint_url=ENDPOINT_URL) 17 | return client 18 | 19 | def get_request_body(modelId, parameters, prompt): 20 | provider = modelId.split(".")[0] 21 | request_body = None 22 | if provider == "anthropic": 23 | # claude-3 models use new messages format 24 | if modelId.startswith("anthropic.claude-3"): 25 | request_body = { 26 | "anthropic_version": "bedrock-2023-05-31", 27 | "messages": [{"role": "user", "content": [{'type':'text','text': prompt}]}], 28 | "max_tokens": DEFAULT_MAX_TOKENS 29 | } 30 | request_body.update(parameters) 31 | else: 32 | request_body = { 33 | "prompt": prompt, 34 | "max_tokens_to_sample": DEFAULT_MAX_TOKENS 35 | } 36 | request_body.update(parameters) 37 | elif provider == "ai21": 38 | request_body = { 39 | "prompt": prompt, 40 | "maxTokens": DEFAULT_MAX_TOKENS 41 | } 42 | request_body.update(parameters) 43 | elif provider == "amazon": 44 | textGenerationConfig = { 45 | "maxTokenCount": DEFAULT_MAX_TOKENS 46 | } 47 | textGenerationConfig.update(parameters) 48 | request_body = { 49 | "inputText": prompt, 50 | "textGenerationConfig": textGenerationConfig 51 | } 52 | elif provider == "cohere": 53 | request_body = { 54 | "prompt": prompt, 55 | "max_tokens": DEFAULT_MAX_TOKENS 56 | } 57 | request_body.update(parameters) 58 | elif provider == "meta": 59 | request_body = { 60 | "prompt": prompt, 61 | "max_gen_len": DEFAULT_MAX_TOKENS 62 | } 63 | request_body.update(parameters) 64 | else: 65 | raise Exception("Unsupported provider: ", provider) 66 | return request_body 67 | 68 | def get_generate_text(modelId, response): 69 | provider = modelId.split(".")[0] 70 | generated_text = None 71 | response_body = json.loads(response.get("body").read()) 72 | print("Response body: ", json.dumps(response_body)) 73 | if provider == "anthropic": 74 | # claude-3 models use new messages format 75 | if modelId.startswith("anthropic.claude-3"): 76 | generated_text = response_body.get("content")[0].get("text") 77 | else: 78 | generated_text = response_body.get("completion") 79 | elif provider == "ai21": 80 | generated_text = response_body.get("completions")[0].get("data").get("text") 81 | elif provider == "amazon": 82 | generated_text = response_body.get("results")[0].get("outputText") 83 | elif provider == "cohere": 84 | generated_text = response_body.get("generations")[0].get("text") 85 | elif provider == "meta": 86 | generated_text = response_body.get("generation") 87 | else: 88 | raise Exception("Unsupported provider: ", provider) 89 | return generated_text 90 | 91 | def call_llm(parameters, prompt): 92 | global client 93 | modelId = parameters.pop("modelId", DEFAULT_MODEL_ID) 94 | body = get_request_body(modelId, parameters, prompt) 95 | print("ModelId", modelId, "- Body: ", body) 96 | if (client is None): 97 | client = get_client() 98 | response = client.invoke_model(body=json.dumps(body), modelId=modelId, accept='application/json', contentType='application/json') 99 | generated_text = get_generate_text(modelId, response) 100 | return generated_text 101 | 102 | 103 | """ 104 | Example Test Event: 105 | { 106 | "prompt": "\n\nHuman:Why is the sky blue?\n\nAssistant:", 107 | "parameters": { 108 | "modelId": "anthropic.claude-3-sonnet-20240229-v1:0", 109 | "temperature": 0, 110 | "system": "You are an AI assistant that always answers in ryhming couplets" 111 | } 112 | } 113 | For supported parameters for each provider model, see Bedrock docs: https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers 114 | """ 115 | def lambda_handler(event, context): 116 | print("Event: ", json.dumps(event)) 117 | prompt = event["prompt"] 118 | parameters = event["parameters"] 119 | generated_text = call_llm(parameters, prompt) 120 | print("Result:", json.dumps(generated_text)) 121 | return { 122 | 'generated_text': generated_text 123 | } -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 5 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 6 | 7 | ## [Unreleased] 8 | ## [0.1.17] - 2024-07-08 9 | - Amazon Q Business Expert plugin now supports Identity Center authentication - PR #30 10 | 11 | ## [0.1.16] - 2024-07-02 12 | ### Added 13 | - Amazon Bedrock LLM plugin now suports anthropic.claude-3-haiku model - PR #28. 14 | 15 | ## [0.1.15] - 2024-03-07 16 | ### Added 17 | - Amazon Bedrock LLM plugin now suports anthropic.claude-3-sonnet model, and deprecates anthropic.claude-v1 - PR #26 & PR #27. 18 | 19 | ## [0.1.14] - 2023-12-22 20 | ### Added 21 | - Amazon Q Business Expert plugin now suports optional file attachments via Lex Web UI (v0.20.4) attach option and the new userFileUpload session attribute - PR #23. 22 | 23 | 24 | ## [0.1.13] - 2023-12-06 25 | ### Added 26 | - Bedrock plugin updates to support new text models - #22 27 | - amazon.titan-text-lite-v1 28 | - anthropic.claude-v2:1 29 | - cohere.command-text-v14 30 | - cohere.command-light-text-v14 31 | - meta.llama2-13b-chat-v1 32 | - meta.llama2-70b-chat-v1 33 | 34 | ## [0.1.12] - 2023-11-29 35 | ### Added 36 | - Amazon Q, your business expert now integrates with QnABot as a fallback answer source, using QnAbot's using Lambda hooks with CustomNoMatches/no_hits. For more information see: [QnABot LambdaHook for Amazon Q, your business expert (preview)](./lambdas/qna_bot_qbusiness_lambdahook/README.md) 37 | 38 | ## [0.1.11] - 2023-11-07 39 | ### Fixed 40 | - Error in Bedrock QnABotSettingQAPromptTemplate output - prompt does not terminate with '\n\nAssistant:` and generates error from Bedrock - #13 41 | 42 | ## [0.1.10] - 2023-10-27 43 | ### Fixed 44 | - Prompt bug: question not denoted by an XML tag, so LLM gets confused about what it's answering - PR #13 45 | - Upgrade issue in BedrockBoto3Layer due to bedrock boto3 zip - PR #16 46 | 47 | ## [0.1.9] - 2023-10-26 48 | ### Added 49 | - Added Amazon Bedrock support for configuring LLM as a fallback source of answers, using Lambda hooks with CustomNoMatches/no_hits - PR #11 50 | 51 | ## [0.1.8] - 2023-10-10 52 | ### Added 53 | - Added Mistral 7b Instruct LLM - PR #10 54 | 55 | ## [0.1.7] - 2023-10-05 56 | ### Fixed 57 | - Bedrock embeddings function now strips any leading or trailing whitespace from input strings before generating embeddings to avoid whitespace affecting accuracy. 58 | 59 | ## [0.1.6] - 2023-10-03 60 | ### Fixed 61 | - Test that Bedrock service and selected models are available in account during stack create/update to avoid downstream failures. 62 | 63 | ## [0.1.5] - 2023-09-30 64 | ### Fixed 65 | - Increased default EMBEDDINGS_SCORE_THRESHOLD to reduce poor quality QnA matches 66 | - Fix typo in QA_PROMPT_TEMPLATE 67 | 68 | ## [0.1.4] - 2023-09-28 69 | ### Added 70 | - Remove preview Bedrock sdk extensions 71 | - Update to Bedrock GA model identifiers 72 | - Update to `bedrock-runtime` endpoints/service name 73 | - Use latest Titan embeddings model 74 | - Add `EmbeddingsLambdaDimensions` to Bedrock plugin stack outputs 75 | - Add new [Lambda Hook function](./README.md#optional-use-the-llm-as-a-fallback-source-of-answers-using-lambda-hooks-with-customnomatchesno_hits) to AI21 Plugin (others coming later) 76 | 77 | ## [0.1.3] - 2023-09-21 78 | ### Added 79 | - Added LLama-2-13b-Chat plugin - PR #5 80 | 81 | ## [0.1.2] - 2023-08-10 82 | ### Added 83 | - Cfn nag fixes 84 | 85 | ## [0.1.1] - 2023-08-08 86 | ### Added 87 | - Update default value for BedrockPreviewSdkUrl parameter in Amazon Bedrock plugin template 88 | 89 | ## [0.1.0] - 2023-07-27 90 | ### Added 91 | - Initial release 92 | 93 | [Unreleased]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/compare/main...develop 94 | [0.1.17]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.17 95 | [0.1.16]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.16 96 | [0.1.15]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.15 97 | [0.1.14]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.14 98 | [0.1.13]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.13 99 | [0.1.12]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.12 100 | [0.1.11]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.11 101 | [0.1.10]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.10 102 | [0.1.9]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.9 103 | [0.1.8]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.8 104 | [0.1.7]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.7 105 | [0.1.6]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.6 106 | [0.1.5]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.5 107 | [0.1.4]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.4 108 | [0.1.3]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.3 109 | [0.1.2]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.2 110 | [0.1.1]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.1 111 | [0.1.0]: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/releases/tag/v0.1.0 112 | -------------------------------------------------------------------------------- /lambdas/ai21-llm/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Description: QnABot on AWS LLM Plugin for AI21 - v0.1.2 3 | 4 | Parameters: 5 | APIKey: 6 | Type: String 7 | Description: AI21 API Key (stored in Secrets Manager - see stack Outputs) 8 | Default: '' 9 | NoEcho: true 10 | 11 | LLMModelType: 12 | Type: String 13 | Default: j2-mid 14 | AllowedValues: 15 | - j2-light 16 | - j2-mid 17 | - j2-ultra 18 | Description: AI21 LLM Model Type 19 | 20 | Resources: 21 | ApiKeySecret: 22 | Type: AWS::SecretsManager::Secret 23 | Properties: 24 | Description: API Key 25 | Name: !Ref AWS::StackName 26 | SecretString: !Ref APIKey 27 | Metadata: 28 | cfn_nag: 29 | rules_to_suppress: 30 | - id: W77 31 | reason: No requirement for custom CMK, secret will not be shared cross-account. 32 | 33 | LambdaFunctionRole: 34 | Type: AWS::IAM::Role 35 | Properties: 36 | AssumeRolePolicyDocument: 37 | Version: '2012-10-17' 38 | Statement: 39 | - Effect: Allow 40 | Principal: 41 | Service: lambda.amazonaws.com 42 | Action: sts:AssumeRole 43 | ManagedPolicyArns: 44 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 45 | Policies: 46 | - PolicyDocument: 47 | Version: 2012-10-17 48 | Statement: 49 | - Effect: Allow 50 | Action: 51 | - 'secretsmanager:GetResourcePolicy' 52 | - 'secretsmanager:GetSecretValue' 53 | - 'secretsmanager:DescribeSecret' 54 | - 'secretsmanager:ListSecretVersionIds' 55 | Resource: !Ref ApiKeySecret 56 | PolicyName: SecretsManagerPolicy 57 | 58 | LambdaFunction: 59 | Type: AWS::Lambda::Function 60 | Properties: 61 | Handler: "llm.lambda_handler" 62 | Role: !GetAtt 'LambdaFunctionRole.Arn' 63 | MemorySize: 128 64 | Timeout: 60 65 | Runtime: python3.10 66 | Environment: 67 | Variables: 68 | API_KEY_SECRET_NAME: !Ref AWS::StackName 69 | Code: ./src 70 | Metadata: 71 | cfn_nag: 72 | rules_to_suppress: 73 | - id: W89 74 | reason: Lambda function is not communicating with any VPC resources. 75 | - id: W92 76 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 77 | 78 | QnaItemLambdaHookFunction: 79 | Type: AWS::Lambda::Function 80 | Properties: 81 | # LambdaHook name must start with 'QNA-' to match QnAbot invoke policy 82 | FunctionName: !Sub "QNA-LAMBDAHOOK-${AWS::StackName}" 83 | Handler: "lambdahook.lambda_handler" 84 | Role: !GetAtt 'LambdaFunctionRole.Arn' 85 | MemorySize: 128 86 | Timeout: 60 87 | Runtime: python3.10 88 | Environment: 89 | Variables: 90 | API_KEY_SECRET_NAME: !Ref AWS::StackName 91 | Code: ./src 92 | Metadata: 93 | cfn_nag: 94 | rules_to_suppress: 95 | - id: W89 96 | reason: Lambda function is not communicating with any VPC resources. 97 | - id: W92 98 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 99 | 100 | OutputSettingsFunctionRole: 101 | Type: AWS::IAM::Role 102 | Properties: 103 | AssumeRolePolicyDocument: 104 | Version: '2012-10-17' 105 | Statement: 106 | - Effect: Allow 107 | Principal: 108 | Service: lambda.amazonaws.com 109 | Action: sts:AssumeRole 110 | ManagedPolicyArns: 111 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 112 | 113 | OutputSettingsFunction: 114 | Type: AWS::Lambda::Function 115 | Properties: 116 | Handler: settings.lambda_handler 117 | Role: !GetAtt 'OutputSettingsFunctionRole.Arn' 118 | Runtime: python3.10 119 | Timeout: 10 120 | MemorySize: 128 121 | Code: ./src 122 | Metadata: 123 | cfn_nag: 124 | rules_to_suppress: 125 | - id: W89 126 | reason: Lambda function is not communicating with any VPC resources. 127 | - id: W92 128 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 129 | 130 | OutputSettings: 131 | Type: Custom::OutputSettings 132 | Properties: 133 | ServiceToken: !GetAtt OutputSettingsFunction.Arn 134 | ModelType: !Ref LLMModelType 135 | 136 | Outputs: 137 | APIKeySecret: 138 | Description: Link to Secrets Manager console to input API Key 139 | Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/secretsmanager/secret?region=${AWS::Region}&name=${AWS::StackName}" 140 | 141 | LLMLambdaArn: 142 | Description: Lambda function ARN (use for QnABot param "LLMLambdaArn") 143 | Value: !GetAtt LambdaFunction.Arn 144 | 145 | QnABotSettingGenerateQueryPromptTemplate: 146 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_PROMPT_TEMPLATE" 147 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_PROMPT_TEMPLATE 148 | 149 | QnABotSettingGenerateQueryModelParams: 150 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_MODEL_PARAMS" 151 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_MODEL_PARAMS 152 | 153 | QnABotSettingQAPromptTemplate: 154 | Description: QnABot Designer Setting "LLM_QA_PROMPT_TEMPLATE" 155 | Value: !GetAtt OutputSettings.LLM_QA_PROMPT_TEMPLATE 156 | 157 | QnABotSettingQAModelParams: 158 | Description: QnABot Designer Setting "LLM_QA_MODEL_PARAMS" 159 | Value: !GetAtt OutputSettings.LLM_QA_MODEL_PARAMS 160 | 161 | QnAItemLambdaHookFunctionName: 162 | Description: QnA Item Lambda Hook Function Name (use with no_hits item for optional ask-the-LLM fallback) 163 | Value: !Ref QnaItemLambdaHookFunction 164 | 165 | QnAItemLambdaHookArgs: 166 | Description: QnA Item Lambda Hook Args (use with no_hits item for optional ask-the-LLM fallback) 167 | Value: !GetAtt OutputSettings.QNAITEM_LAMBDAHOOK_ARGS -------------------------------------------------------------------------------- /publish.sh: -------------------------------------------------------------------------------- 1 | # This script will perform the following tasks: 2 | # 1. Remove any old build files from previous runs. 3 | # 2. Create a deployment S3 bucket to store build artifacts if not already existing 4 | # 3. Installing required libraries and package them into ZIP files for Lambda layer creation. It will spin up a Docker container to install the packages to ensure architecture compatibility 5 | # 4. Package the CloudFormation template and upload it to the S3 bucket 6 | # 7 | # To deploy to non-default region, set AWS_DEFAULT_REGION to supported region 8 | # See: https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/supported-aws-regions.html - E.g. 9 | # export AWS_DEFAULT_REGION=eu-west-1 10 | 11 | USAGE="$0 [public]" 12 | 13 | BUCKET=$1 14 | [ -z "$BUCKET" ] && echo "Cfn bucket name is required parameter. Usage $USAGE" && exit 1 15 | 16 | PREFIX=$2 17 | [ -z "$PREFIX" ] && echo "Prefix is required parameter. Usage $USAGE" && exit 1 18 | 19 | # Remove trailing slash from prefix if needed 20 | [[ "${PREFIX}" == */ ]] && PREFIX="${PREFIX%?}" 21 | 22 | ACL=$3 23 | if [ "$ACL" == "public" ]; then 24 | echo "Published S3 artifacts will be acessible by public (read-only)" 25 | PUBLIC=true 26 | else 27 | echo "Published S3 artifacts will NOT be acessible by public." 28 | PUBLIC=false 29 | fi 30 | 31 | # Config 32 | LAYERS_DIR=$PWD/layers 33 | LAMBDAS_DIR=$PWD/lambdas 34 | 35 | # Create bucket if it doesn't already exist 36 | echo "------------------------------------------------------------------------------" 37 | aws s3api list-buckets --query 'Buckets[].Name' | grep "\"$BUCKET\"" > /dev/null 2>&1 38 | if [ $? -ne 0 ]; then 39 | echo "Creating S3 bucket: $BUCKET" 40 | aws s3 mb s3://${BUCKET} || exit 1 41 | aws s3api put-bucket-versioning --bucket ${BUCKET} --versioning-configuration Status=Enabled || exit 1 42 | else 43 | echo "Using existing bucket: $BUCKET" 44 | fi 45 | echo "------------------------------------------------------------------------------" 46 | 47 | 48 | # get bucket region for owned accounts 49 | region=$(aws s3api get-bucket-location --bucket $BUCKET --query "LocationConstraint" --output text) || region="us-east-1" 50 | [ -z "$region" -o "$region" == "None" ] && region=us-east-1; 51 | echo "Bucket in region: $region" 52 | 53 | if $PUBLIC; then 54 | echo "Enabling ACLs on bucket" 55 | aws s3api put-public-access-block --bucket ${BUCKET} --public-access-block-configuration "BlockPublicPolicy=false" 56 | aws s3api put-bucket-ownership-controls --bucket ${BUCKET} --ownership-controls="Rules=[{ObjectOwnership=BucketOwnerPreferred}]" 57 | fi 58 | 59 | echo "------------------------------------------------------------------------------" 60 | echo "Installing Python packages for AWS Lambda Layers" 61 | echo "------------------------------------------------------------------------------" 62 | if [ -d "$LAYERS_DIR" ]; then 63 | LAYERS=$(ls $LAYERS_DIR) 64 | pushd $LAYERS_DIR 65 | for layer in $LAYERS; do 66 | echo "Installing packages for: $layer" 67 | # ref docs: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-pycache 68 | pip install \ 69 | --quiet \ 70 | --platform manylinux2014_x86_64 \ 71 | --target=package \ 72 | --implementation cp \ 73 | --python-version 3.10 \ 74 | --only-binary=:all: \ 75 | --no-compile \ 76 | --requirement ${layer}/requirements.txt \ 77 | --target=${layer}/python 2>&1 | \ 78 | grep -v "WARNING: Target directory" 79 | echo "Done installing dependencies for $layer" 80 | done 81 | popd 82 | else 83 | echo "Directory $LAYERS_DIR does not exist. Skipping" 84 | fi 85 | 86 | echo "------------------------------------------------------------------------------" 87 | echo "Packaging CloudFormation artifacts" 88 | echo "------------------------------------------------------------------------------" 89 | LAMBDAS=$(ls $LAMBDAS_DIR) 90 | for lambda in $LAMBDAS; do 91 | dir=$LAMBDAS_DIR/$lambda 92 | pushd $dir 93 | echo "PACKAGING $lambda" 94 | mkdir -p ./out 95 | template=${lambda}.yaml 96 | s3_template=s3://${BUCKET}/${PREFIX}/${template} 97 | https_template="https://s3.${region}.amazonaws.com/${BUCKET}/${PREFIX}/${template}" 98 | # avoid re-packaging source zips if only file timestamps have changed - per https://blog.revolve.team/2022/05/19/lambda-build-consistency/ 99 | [ -d "$LAYERS_DIR" ] && find $LAYERS_DIR -exec touch -a -m -t"202307230000.00" {} \; 100 | find ./src -exec touch -a -m -t"202307230000.00" {} \; 101 | aws cloudformation package \ 102 | --template-file ./template.yml \ 103 | --output-template-file ./out/${template} \ 104 | --s3-bucket $BUCKET --s3-prefix $PREFIX \ 105 | --region ${region} || exit 1 106 | echo "Uploading template file to: ${s3_template}" 107 | aws s3 cp ./out/${template} ${s3_template} 108 | echo "Validating template" 109 | aws cloudformation validate-template --template-url ${https_template} > /dev/null || exit 1 110 | popd 111 | done 112 | 113 | if $PUBLIC; then 114 | echo "------------------------------------------------------------------------------" 115 | echo "Setting public read ACLs on published artifacts" 116 | echo "------------------------------------------------------------------------------" 117 | files=$(aws s3api list-objects --bucket ${BUCKET} --prefix ${PREFIX} --query "(Contents)[].[Key]" --output text) 118 | for file in $files 119 | do 120 | aws s3api put-object-acl --acl public-read --bucket ${BUCKET} --key $file 121 | done 122 | fi 123 | 124 | echo "------------------------------------------------------------------------------" 125 | echo "Outputs" 126 | echo "------------------------------------------------------------------------------" 127 | for lambda in $LAMBDAS; do 128 | stackname=QNABOTPLUGIN-$(echo $lambda | tr '[:lower:]' '[:upper:]' | tr '_' '-') 129 | template="https://s3.${region}.amazonaws.com/${BUCKET}/${PREFIX}/${lambda}.yaml" 130 | echo $stackname 131 | echo "==============" 132 | echo " - Template URL: $template" 133 | echo " - Deploy URL: https://${region}.console.aws.amazon.com/cloudformation/home?region=${region}#/stacks/create/review?templateURL=${template}&stackName=${stackname}" 134 | echo "" 135 | done 136 | echo "All done!" 137 | exit 0 138 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/settings.py: -------------------------------------------------------------------------------- 1 | import cfnresponse 2 | import json 3 | 4 | # Default prompt templates 5 | AMAZON_GENERATE_QUERY_PROMPT_TEMPLATE = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.
Chat History:
{history}
Follow up question: {input}
Standalone question:""" 6 | AMAZON_QA_PROMPT_TEMPLATE = """

Human: You are a friendly AI assistant. Answer the question in tags only based on the provided reference passages. Here are reference passages in tags:

{context}

If the references contain the information needed to respond, then write a confident response in under 50 words, quoting the relevant references.
Otherwise, if you can make an informed guess based on the reference passages, then write a less confident response in under 50 words, stating your assumptions.
Finally, if the references do not have any relevant information, then respond saying \\"Sorry, I don't know\\".

{query}


Assistant: According to the reference passages, in under 50 words:""" 7 | ANTHROPIC_GENERATE_QUERY_PROMPT_TEMPLATE = """

Human: Here is a chat history in tags:

{history}

Human: And here is a follow up question or statement from the human in tags:

{input}

Human: Rephrase the follow up question or statement as a standalone question or statement that makes sense without reading the chat history.

Assistant: Here is the rephrased follow up question or statement:""" 8 | ANTHROPIC_QA_PROMPT_TEMPLATE = AMAZON_QA_PROMPT_TEMPLATE 9 | AI21_GENERATE_QUERY_PROMPT_TEMPATE = ANTHROPIC_GENERATE_QUERY_PROMPT_TEMPLATE 10 | AI21_QA_PROMPT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Documents: {context} Instruction: Based on the above documents, provide a detailed answer for {query} Answer "don't know" if not present in the document. Solution:""" 11 | COHERE_GENERATE_QUERY_PROMPT_TEMPLATE = AMAZON_GENERATE_QUERY_PROMPT_TEMPLATE 12 | COHERE_QA_PROMPT_TEMPLATE = AMAZON_QA_PROMPT_TEMPLATE 13 | META_GENERATE_QUERY_PROMPT_TEMPLATE = AMAZON_GENERATE_QUERY_PROMPT_TEMPLATE 14 | META_QA_PROMPT_TEMPLATE = AMAZON_QA_PROMPT_TEMPLATE 15 | 16 | def getEmbeddingSettings(modelId): 17 | provider = modelId.split(".")[0] 18 | settings = {} 19 | # Currently, only Amazon embeddings are supported 20 | if provider == "amazon": 21 | settings.update({ 22 | "EMBEDDINGS_SCORE_THRESHOLD": 0.8, 23 | "EMBEDDINGS_SCORE_ANSWER_THRESHOLD": 0.6, 24 | "EMBEDDINGS_TEXT_PASSAGE_SCORE_THRESHOLD": 0.7, 25 | "EMBEDDINGS_DIMENSIONS": 1536 26 | }) 27 | else: 28 | raise Exception("Unsupported provider for embeddings: ", provider) 29 | return settings 30 | 31 | def getModelSettings(modelId): 32 | params = { 33 | "modelId": modelId, 34 | "temperature": 0 35 | } 36 | params_qa = params.copy() 37 | # claude-3 message API params are slightly different 38 | provider = modelId.split(".")[0] 39 | if provider == "anthropic": 40 | if modelId.startswith("anthropic.claude-3"): 41 | params = { 42 | "modelId": modelId, 43 | "temperature": 0, 44 | "max_tokens": 256, 45 | "top_p": 1 46 | } 47 | # add optional system prompt to qa params 48 | params_qa = { 49 | "modelId": modelId, 50 | "temperature": 0, 51 | "max_tokens": 256, 52 | "top_p": 1, 53 | "system": "You are a helpful AI assistant." 54 | } 55 | else: 56 | params.update({"max_tokens_to_sample": 256}) 57 | params_qa.update({"max_tokens_to_sample": 256}) 58 | lambdahook_args = {"Prefix":"LLM Answer:", "Model_params": params} 59 | settings = { 60 | 'LLM_GENERATE_QUERY_MODEL_PARAMS': json.dumps(params), 61 | 'LLM_QA_MODEL_PARAMS': json.dumps(params_qa), 62 | 'QNAITEM_LAMBDAHOOK_ARGS': json.dumps(lambdahook_args) 63 | } 64 | provider = modelId.split(".")[0] 65 | if provider == "anthropic": 66 | settings.update({ 67 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': ANTHROPIC_GENERATE_QUERY_PROMPT_TEMPLATE, 68 | 'LLM_QA_PROMPT_TEMPLATE': ANTHROPIC_QA_PROMPT_TEMPLATE 69 | }) 70 | elif provider == "ai21": 71 | settings.update({ 72 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': AI21_GENERATE_QUERY_PROMPT_TEMPATE, 73 | 'LLM_QA_PROMPT_TEMPLATE': AI21_QA_PROMPT_TEMPLATE 74 | }) 75 | elif provider == "amazon": 76 | settings.update({ 77 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': AMAZON_GENERATE_QUERY_PROMPT_TEMPLATE, 78 | 'LLM_QA_PROMPT_TEMPLATE': AMAZON_QA_PROMPT_TEMPLATE 79 | }) 80 | elif provider == "cohere": 81 | settings.update({ 82 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': COHERE_GENERATE_QUERY_PROMPT_TEMPLATE, 83 | 'LLM_QA_PROMPT_TEMPLATE': COHERE_QA_PROMPT_TEMPLATE 84 | }) 85 | elif provider == "meta": 86 | settings.update({ 87 | 'LLM_GENERATE_QUERY_PROMPT_TEMPLATE': META_GENERATE_QUERY_PROMPT_TEMPLATE, 88 | 'LLM_QA_PROMPT_TEMPLATE': META_QA_PROMPT_TEMPLATE 89 | }) 90 | else: 91 | raise Exception("Unsupported provider: ", provider) 92 | return settings 93 | 94 | def lambda_handler(event, context): 95 | print("Event: ", json.dumps(event)) 96 | status = cfnresponse.SUCCESS 97 | responseData = {} 98 | reason = "" 99 | if event['RequestType'] != 'Delete': 100 | try: 101 | llmModelId = event['ResourceProperties'].get('LLMModelId', '') 102 | embeddingsModelId = event['ResourceProperties'].get('EmbeddingsModelId', '') 103 | responseData = getModelSettings(llmModelId) 104 | responseData.update(getEmbeddingSettings(embeddingsModelId)) 105 | except Exception as e: 106 | print(e) 107 | status = cfnresponse.FAILED 108 | reason = f"Exception thrown: {e}" 109 | cfnresponse.send(event, context, status, responseData, reason=reason) -------------------------------------------------------------------------------- /lambdas/qna_bot_qbusiness_lambdahook/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: > 3 | Amazon Q (Business) Lambda Hook function for using with 'QnABot on AWS'. 4 | Use with the 'no_hits' (CustomNoMatches) item to use Amazon Q when no good answers are found by other methods - v0.1.17 5 | 6 | Parameters: 7 | AmazonQAppId: 8 | Type: String 9 | AllowedPattern: "^[a-zA-Z0-9][a-zA-Z0-9-]{35}$" 10 | Description: Amazon Q Application ID 11 | 12 | IDCApplicationARN: 13 | Type: String 14 | Description: ARN of the Identity Center customer managed application created for QBusiness 15 | 16 | DynamoDBTableName: 17 | Type: String 18 | Description: DynamoDB Table Name used for caching QBusiness credentials 19 | 20 | AmazonQRegion: 21 | Type: String 22 | Default: "us-east-1" 23 | AllowedPattern: "^[a-z]{2}-[a-z]+-[0-9]+$" 24 | Description: Amazon Q Region 25 | 26 | AmazonQEndpointUrl: 27 | Type: String 28 | Default: "" 29 | Description: (Optional) Amazon Q Endpoint (leave empty for default endpoint) 30 | 31 | Resources: 32 | QManagedPolicy: 33 | Type: AWS::IAM::ManagedPolicy 34 | Properties: 35 | PolicyDocument: 36 | Version: "2012-10-17" 37 | Statement: 38 | - Sid: AllowQChat 39 | Effect: Allow 40 | Action: 41 | - "qbusiness:ChatSync" 42 | Resource: !Sub "arn:${AWS::Partition}:qbusiness:${AWS::Region}:${AWS::AccountId}:application/${AmazonQAppId}" 43 | 44 | QServiceRole: 45 | Type: AWS::IAM::Role 46 | Properties: 47 | AssumeRolePolicyDocument: 48 | Version: 2012-10-17 49 | Statement: 50 | - Effect: Allow 51 | Principal: 52 | AWS: 53 | - !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:root 54 | Action: 55 | - sts:AssumeRole 56 | - sts:SetContext 57 | Path: / 58 | ManagedPolicyArns: 59 | - !Ref QManagedPolicy 60 | 61 | QBusinessModelLayer: 62 | Type: "AWS::Lambda::LayerVersion" 63 | Properties: 64 | Content: ../../layers/qbusiness_boto3_model 65 | CompatibleRuntimes: 66 | - python3.12 67 | 68 | KMSKey: 69 | Type: "AWS::KMS::Key" 70 | Properties: 71 | KeySpec: "SYMMETRIC_DEFAULT" 72 | KeyUsage: "ENCRYPT_DECRYPT" 73 | KeyPolicy: 74 | Version: "2012-10-17" 75 | Statement: 76 | - Effect: Allow 77 | Principal: 78 | AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" 79 | Action: "kms:*" 80 | Resource: "*" 81 | 82 | CredentialsTable: 83 | Type: AWS::DynamoDB::Table 84 | Properties: 85 | AttributeDefinitions: 86 | - AttributeName: "jti" 87 | AttributeType: "S" 88 | KeySchema: 89 | - AttributeName: "jti" 90 | KeyType: "HASH" 91 | BillingMode: PAY_PER_REQUEST 92 | SSESpecification: 93 | SSEEnabled: True 94 | TableName: !Ref DynamoDBTableName 95 | TimeToLiveSpecification: 96 | AttributeName: ExpiresAt 97 | Enabled: true 98 | 99 | LambdaFunctionRole: 100 | Type: AWS::IAM::Role 101 | Properties: 102 | AssumeRolePolicyDocument: 103 | Version: "2012-10-17" 104 | Statement: 105 | - Effect: Allow 106 | Principal: 107 | Service: lambda.amazonaws.com 108 | Action: sts:AssumeRole 109 | ManagedPolicyArns: 110 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 111 | Policies: 112 | - PolicyDocument: 113 | Version: 2012-10-17 114 | Statement: 115 | - Effect: Allow 116 | Action: 117 | - "qbusiness:ChatSync" 118 | Resource: !Sub "arn:aws:qbusiness:${AWS::Region}:${AWS::AccountId}:application/${AmazonQAppId}" 119 | PolicyName: QBusinessPolicy 120 | - PolicyDocument: 121 | Version: 2012-10-17 122 | Statement: 123 | - Effect: Allow 124 | Action: 125 | - "s3:GetObject" 126 | Resource: "arn:aws:s3:::*/*" 127 | PolicyName: S3ImportBucketPolicy 128 | - PolicyDocument: 129 | Version: 2012-10-17 130 | Statement: 131 | - Effect: Allow 132 | Action: 133 | - "dynamodb:PutItem" 134 | - "dynamodb:GetItem" 135 | Resource: 136 | - !Sub "arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTableName}" 137 | PolicyName: DynamoDbPolicy 138 | - PolicyDocument: 139 | Version: 2012-10-17 140 | Statement: 141 | - Effect: Allow 142 | Action: 143 | - "kms:Decrypt" 144 | - "kms:Encrypt" 145 | Resource: 146 | - !Sub "arn:${AWS::Partition}:kms:${AWS::Region}:${AWS::AccountId}:key/${KMSKey}" 147 | PolicyName: KmsPolicy 148 | - PolicyDocument: 149 | Version: 2012-10-17 150 | Statement: 151 | - Effect: Allow 152 | Action: 153 | - "sso-oauth:CreateTokenWithIAM" 154 | Resource: "*" 155 | PolicyName: OICDPolicy 156 | - PolicyDocument: 157 | Version: 2012-10-17 158 | Statement: 159 | - Effect: Allow 160 | Action: 161 | - "sts:AssumeRole" 162 | - "sts:SetContext" 163 | Resource: 164 | - !GetAtt QServiceRole.Arn 165 | PolicyName: AllowAssumeQRole 166 | 167 | QnaItemLambdaHookFunction: 168 | Type: AWS::Lambda::Function 169 | Properties: 170 | # LambdaHook name must start with 'QNA-' to match QnAbot invoke policy 171 | FunctionName: !Sub "QNA-LAMBDAHOOK-${AWS::StackName}" 172 | Handler: lambdahook.lambda_handler 173 | Role: !GetAtt "LambdaFunctionRole.Arn" 174 | Runtime: python3.12 175 | Layers: 176 | - !Ref QBusinessModelLayer 177 | Timeout: 60 178 | MemorySize: 128 179 | Environment: 180 | Variables: 181 | AWS_DATA_PATH: /opt/model 182 | AMAZONQ_APP_ID: !Ref AmazonQAppId 183 | AMAZONQ_ROLE_ARN: !GetAtt QServiceRole.Arn 184 | DYNAMODB_CACHE_TABLE_NAME: !Ref CredentialsTable 185 | KMS_KEY_ID: !Ref KMSKey 186 | IDC_CLIENT_ID: !Ref IDCApplicationARN 187 | AMAZONQ_REGION: !Ref AmazonQRegion 188 | AMAZONQ_ENDPOINT_URL: !Ref AmazonQEndpointUrl 189 | Code: ./src 190 | Metadata: 191 | cfn_nag: 192 | rules_to_suppress: 193 | - id: W89 194 | reason: No VPC resources. 195 | - id: W92 196 | reason: No requirements to set reserved concurrencies. 197 | 198 | Outputs: 199 | QnAItemLambdaHookFunctionName: 200 | Description: QnA Item Lambda Hook Function Name (use with no_hits item for optional ask-Amazon-Q-Business fallback) 201 | Value: !Ref "QnaItemLambdaHookFunction" 202 | 203 | QnAItemLambdaHookArgs: 204 | Description: QnA Item Lambda Hook Args (use with no_hits item for optional ask-the-LLM fallback) 205 | Value: '{"Prefix":"Amazon Q Answer:", "ShowContextText":true, "ShowSourceLinks":true}' 206 | 207 | QnAItemLambdaFunctionRoleArn: 208 | Description: ARN of the Role created for executing the Lambda function 209 | Value: !GetAtt LambdaFunctionRole.Arn 210 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/src/lambdahook.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | import os 4 | 5 | # Defaults 6 | DEFAULT_MODEL_ID = os.environ.get("DEFAULT_MODEL_ID","anthropic.claude-instant-v1") 7 | AWS_REGION = os.environ["AWS_REGION_OVERRIDE"] if "AWS_REGION_OVERRIDE" in os.environ else os.environ["AWS_REGION"] 8 | ENDPOINT_URL = os.environ.get("ENDPOINT_URL", f'https://bedrock-runtime.{AWS_REGION}.amazonaws.com') 9 | DEFAULT_MAX_TOKENS = 256 10 | 11 | # global variables - avoid creating a new client for every request 12 | client = None 13 | 14 | def get_client(): 15 | print("Connecting to Bedrock Service: ", ENDPOINT_URL) 16 | client = boto3.client(service_name='bedrock-runtime', region_name=AWS_REGION, endpoint_url=ENDPOINT_URL) 17 | return client 18 | 19 | def get_request_body(modelId, parameters, prompt): 20 | provider = modelId.split(".")[0] 21 | request_body = None 22 | if provider == "anthropic": 23 | # claude-3 models use new messages format 24 | if modelId.startswith("anthropic.claude-3"): 25 | request_body = { 26 | "anthropic_version": "bedrock-2023-05-31", 27 | "messages": [{"role": "user", "content": [{'type':'text','text': prompt}]}], 28 | "max_tokens": DEFAULT_MAX_TOKENS 29 | } 30 | request_body.update(parameters) 31 | else: 32 | request_body = { 33 | "prompt": prompt, 34 | "max_tokens_to_sample": DEFAULT_MAX_TOKENS 35 | } 36 | request_body.update(parameters) 37 | elif provider == "ai21": 38 | request_body = { 39 | "prompt": prompt, 40 | "maxTokens": DEFAULT_MAX_TOKENS 41 | } 42 | request_body.update(parameters) 43 | elif provider == "amazon": 44 | textGenerationConfig = { 45 | "maxTokenCount": DEFAULT_MAX_TOKENS 46 | } 47 | textGenerationConfig.update(parameters) 48 | request_body = { 49 | "inputText": prompt, 50 | "textGenerationConfig": textGenerationConfig 51 | } 52 | elif provider == "cohere": 53 | request_body = { 54 | "prompt": prompt, 55 | "max_tokens": DEFAULT_MAX_TOKENS 56 | } 57 | request_body.update(parameters) 58 | elif provider == "meta": 59 | request_body = { 60 | "prompt": prompt, 61 | "max_gen_len": DEFAULT_MAX_TOKENS 62 | } 63 | request_body.update(parameters) 64 | else: 65 | raise Exception("Unsupported provider: ", provider) 66 | return request_body 67 | 68 | def get_generate_text(modelId, response): 69 | provider = modelId.split(".")[0] 70 | generated_text = None 71 | response_body = json.loads(response.get("body").read()) 72 | print("Response body: ", json.dumps(response_body)) 73 | if provider == "anthropic": 74 | # claude-3 models use new messages format 75 | if modelId.startswith("anthropic.claude-3"): 76 | generated_text = response_body.get("content")[0].get("text") 77 | else: 78 | generated_text = response_body.get("completion") 79 | elif provider == "ai21": 80 | generated_text = response_body.get("completions")[0].get("data").get("text") 81 | elif provider == "amazon": 82 | generated_text = response_body.get("results")[0].get("outputText") 83 | elif provider == "cohere": 84 | generated_text = response_body.get("generations")[0].get("text") 85 | elif provider == "meta": 86 | generated_text = response_body.get("generation") 87 | else: 88 | raise Exception("Unsupported provider: ", provider) 89 | return generated_text 90 | 91 | def replace_template_placeholders(prompt, event): 92 | # history 93 | history_array = json.loads(event["req"]["_userInfo"].get("chatMessageHistory","[]")) 94 | history_str = '\n'.join(f"{key}: {value}" for item in history_array for key, value in item.items()) 95 | prompt = prompt.replace("{history}", history_str) 96 | # TODO - replace additional prompt template placeholders - eg query, input, session attributes, user info 97 | return prompt 98 | 99 | def format_prompt(modelId, prompt): 100 | provider = modelId.split(".")[0] 101 | if provider == "anthropic": 102 | # Claude models prior to v3 required 'Human/Assistant' formatting 103 | if not modelId.startswith("anthropic.claude-3"): 104 | print("Model provider is Anthropic v2. Checking prompt format.") 105 | if not prompt.startswith("\n\nHuman:") or not prompt.startswith("\n\nSystem:"): 106 | prompt = "\n\nHuman: " + prompt 107 | print("Prepended '\\n\\nHuman:'") 108 | if not prompt.endswith("\n\nAssistant:"): 109 | prompt = prompt + "\n\nAssistant:" 110 | print("Appended '\\n\\nHuman:'") 111 | print(f"Prompt: {json.dumps(prompt)}") 112 | return prompt 113 | 114 | def get_llm_response(modelId, parameters, prompt): 115 | global client 116 | body = get_request_body(modelId, parameters, prompt) 117 | print("ModelId", modelId, "- Body: ", body) 118 | if (client is None): 119 | client = get_client() 120 | response = client.invoke_model(body=json.dumps(body), modelId=modelId, accept='application/json', contentType='application/json') 121 | generated_text = get_generate_text(modelId, response) 122 | return generated_text 123 | 124 | def get_args_from_lambdahook_args(event): 125 | parameters = {} 126 | lambdahook_args_list = event["res"]["result"].get("args",[]) 127 | print("LambdaHook args: ", lambdahook_args_list) 128 | if len(lambdahook_args_list): 129 | try: 130 | parameters = json.loads(lambdahook_args_list[0]) 131 | except Exception as e: 132 | print(f"Failed to parse JSON:", lambdahook_args_list[0], e) 133 | print("..continuing") 134 | return parameters 135 | 136 | def format_response(event, llm_response, prefix): 137 | # set plaintext, markdown, & ssml response 138 | if prefix in ["None", "N/A", "Empty"]: 139 | prefix = None 140 | plainttext = llm_response 141 | markdown = llm_response 142 | ssml = llm_response 143 | if prefix: 144 | plainttext = f"{prefix}\n\n{plainttext}" 145 | markdown = f"**{prefix}**\n\n{markdown}" 146 | # add plaintext, markdown, and ssml fields to event.res 147 | event["res"]["message"] = plainttext 148 | event["res"]["session"]["appContext"] = { 149 | "altMessages": { 150 | "markdown": markdown, 151 | "ssml": ssml 152 | } 153 | } 154 | #TODO - can we determine when LLM has a good answer or not? 155 | #For now, always assume it's a good answer. 156 | #QnAbot sets session attribute qnabot_gotanswer True when got_hits > 0 157 | event["res"]["got_hits"] = 1 158 | return event 159 | 160 | def lambda_handler(event, context): 161 | print("Received event: %s" % json.dumps(event)) 162 | # args = {"Prefix:"", "Model_params":{"modelId":"anthropic.claude-instant-v1", "max_tokens":256}, "Prompt":""} 163 | args = get_args_from_lambdahook_args(event) 164 | model_params = args.get("Model_params",{}) 165 | modelId = model_params.pop("modelId", DEFAULT_MODEL_ID) 166 | # prompt set from args, or from req.question if not specified in args. 167 | prompt = args.get("Prompt", event["req"]["question"]) 168 | prompt = format_prompt(modelId, prompt) 169 | prompt = replace_template_placeholders(prompt, event) 170 | llm_response = get_llm_response(modelId, model_params, prompt) 171 | prefix = args.get("Prefix","LLM Answer:") 172 | event = format_response(event, llm_response, prefix) 173 | print("Returning response: %s" % json.dumps(event)) 174 | return event 175 | -------------------------------------------------------------------------------- /lambdas/qna_bot_qbusiness_lambdahook/src/lambdahook.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | import base64 4 | import json 5 | import os 6 | import random 7 | import string 8 | import uuid 9 | import boto3 10 | 11 | AMAZONQ_APP_ID = os.environ.get("AMAZONQ_APP_ID") 12 | AMAZONQ_REGION = os.environ.get("AMAZONQ_REGION") or os.environ["AWS_REGION"] 13 | AMAZONQ_ENDPOINT_URL = os.environ.get("AMAZONQ_ENDPOINT_URL") or f'https://qbusiness.{AMAZONQ_REGION}.api.aws' 14 | print("AMAZONQ_ENDPOINT_URL:", AMAZONQ_ENDPOINT_URL) 15 | 16 | 17 | def get_amazonq_response(prompt, context, attachments, qbusiness_client): 18 | print(f"get_amazonq_response: prompt={prompt}, app_id={AMAZONQ_APP_ID}, context={context}") 19 | input = { 20 | "applicationId": AMAZONQ_APP_ID, 21 | "userMessage": prompt 22 | } 23 | if context: 24 | if context["conversationId"]: 25 | input["conversationId"] = context["conversationId"] 26 | if context["parentMessageId"]: 27 | input["parentMessageId"] = context["parentMessageId"] 28 | else: 29 | input["clientToken"] = str(uuid.uuid4()) 30 | 31 | if attachments: 32 | input["attachments"] = attachments 33 | 34 | print("Amazon Q Input: ", input) 35 | try: 36 | resp = qbusiness_client.chat_sync(**input) 37 | except Exception as e: 38 | print("Amazon Q Exception: ", e) 39 | resp = { 40 | "systemMessage": "Amazon Q Error: " + str(e) 41 | } 42 | print("Amazon Q Response: ", json.dumps(resp, default=str)) 43 | return resp 44 | 45 | 46 | def get_settings_from_lambdahook_args(event): 47 | lambdahook_settings = {} 48 | lambdahook_args_list = event["res"]["result"].get("args", []) 49 | print("LambdaHook args: ", lambdahook_args_list) 50 | if len(lambdahook_args_list): 51 | try: 52 | lambdahook_settings = json.loads(lambdahook_args_list[0]) 53 | except Exception as e: 54 | print(f"Failed to parse JSON:", lambdahook_args_list[0], e) 55 | print("..continuing") 56 | return lambdahook_settings 57 | 58 | 59 | def get_args_from_lambdahook_args(event): 60 | parameters = {} 61 | lambdahook_args_list = event["res"]["result"].get("args", []) 62 | print("LambdaHook args: ", lambdahook_args_list) 63 | if len(lambdahook_args_list): 64 | try: 65 | parameters = json.loads(lambdahook_args_list[0]) 66 | except Exception as e: 67 | print(f"Failed to parse JSON:", lambdahook_args_list[0], e) 68 | print("..continuing") 69 | return parameters 70 | 71 | 72 | def getS3File(s3Path): 73 | if s3Path.startswith("s3://"): 74 | s3Path = s3Path[5:] 75 | s3 = boto3.resource('s3') 76 | bucket, key = s3Path.split("/", 1) 77 | obj = s3.Object(bucket, key) 78 | return obj.get()['Body'].read() 79 | 80 | 81 | def getAttachments(event): 82 | userFilesUploaded = event["req"]["session"].get("userFilesUploaded", []) 83 | attachments = [] 84 | for userFile in userFilesUploaded: 85 | print(f"getAttachments: userFile={userFile}") 86 | attachments.append({ 87 | "data": getS3File(userFile["s3Path"]), 88 | "name": userFile["fileName"] 89 | }) 90 | # delete userFilesUploaded from session 91 | event["res"]["session"].pop("userFilesUploaded", None) 92 | return attachments 93 | 94 | 95 | def format_response(event, amazonq_response): 96 | # get settings, if any, from lambda hook args 97 | # e.g: {"Prefix":"", "ShowContext": False} 98 | lambdahook_settings = get_settings_from_lambdahook_args(event) 99 | prefix = lambdahook_settings.get("Prefix", "Amazon Q Answer:") 100 | showContextText = lambdahook_settings.get("ShowContextText", True) 101 | showSourceLinks = lambdahook_settings.get("ShowSourceLinks", True) 102 | # set plaintext, markdown, & ssml response 103 | if prefix in ["None", "N/A", "Empty"]: 104 | prefix = None 105 | plainttext = amazonq_response["systemMessage"] 106 | markdown = amazonq_response["systemMessage"] 107 | ssml = amazonq_response["systemMessage"] 108 | if prefix: 109 | plainttext = f"{prefix}\n\n{plainttext}" 110 | markdown = f"**{prefix}**\n\n{markdown}" 111 | if showContextText: 112 | contextText = "" 113 | for source in amazonq_response.get("sourceAttributions", []): 114 | title = source.get("title", "title missing") 115 | snippet = source.get("snippet", "snippet missing") 116 | url = source.get("url") 117 | if url: 118 | contextText = f'{contextText}
{title}' 119 | else: 120 | contextText = f'{contextText}
{title}' 121 | # Returning too large of a snippet can break QnABot by exceeding the event payload size limit 122 | contextText = f"{contextText}
{snippet}\n"[:5000] 123 | if contextText: 124 | markdown = f'{markdown}\n
Context

{contextText}

' 125 | if showSourceLinks: 126 | sourceLinks = [] 127 | for source in amazonq_response.get("sourceAttribution", []): 128 | title = source.get("title", "link (no title)") 129 | url = source.get("url") 130 | if url: 131 | sourceLinks.append(f'{title}') 132 | if len(sourceLinks): 133 | markdown = f'{markdown}
Sources: ' + ", ".join(sourceLinks) 134 | 135 | # add plaintext, markdown, and ssml fields to event.res 136 | event["res"]["message"] = plainttext 137 | event["res"]["session"]["appContext"] = { 138 | "altMessages": { 139 | "markdown": markdown, 140 | "ssml": ssml 141 | } 142 | } 143 | # preserve conversation context in session 144 | amazonq_context = { 145 | "conversationId": amazonq_response.get("conversationId"), 146 | "parentMessageId": amazonq_response.get("systemMessageId") 147 | } 148 | event["res"]["session"]["qnabotcontext"]["amazonq_context"] = amazonq_context 149 | # TODO - can we determine when Amazon Q has a good answer or not? 150 | # For now, always assume it's a good answer. 151 | # QnAbot sets session attribute qnabot_gotanswer True when got_hits > 0 152 | event["res"]["got_hits"] = 1 153 | return event 154 | 155 | 156 | def get_idc_iam_credentials(jwt): 157 | sso_oidc_client = boto3.client('sso-oidc') 158 | idc_sso_resp = sso_oidc_client.create_token_with_iam( 159 | clientId=os.environ.get("IDC_CLIENT_ID"), 160 | grantType="urn:ietf:params:oauth:grant-type:jwt-bearer", 161 | assertion=jwt, 162 | ) 163 | 164 | print(idc_sso_resp) 165 | idc_sso_id_token_jwt = json.loads(base64.b64decode(idc_sso_resp['idToken'].split('.')[1] + '==').decode()) 166 | 167 | sts_context = idc_sso_id_token_jwt["sts:identity_context"] 168 | sts_client = boto3.client('sts') 169 | session_name = "qbusiness-idc-" + "".join( 170 | random.choices(string.ascii_letters + string.digits, k=32) 171 | ) 172 | assumed_role_object = sts_client.assume_role( 173 | RoleArn=os.environ.get("AMAZONQ_ROLE_ARN"), 174 | RoleSessionName=session_name, 175 | ProvidedContexts=[{ 176 | "ProviderArn": "arn:aws:iam::aws:contextProvider/IdentityCenter", 177 | "ContextAssertion": sts_context 178 | }] 179 | ) 180 | creds_object = assumed_role_object['Credentials'] 181 | 182 | return creds_object 183 | 184 | 185 | def lambda_handler(event, context): 186 | print("Received event: %s" % json.dumps(event)) 187 | args = get_args_from_lambdahook_args(event) 188 | # prompt set from args, or from the original query if not specified in args. 189 | userInput = event["req"]["llm_generated_query"]["orig"] 190 | qnabotcontext = event["req"]["session"].get("qnabotcontext", {}) 191 | amazonq_context = qnabotcontext.get("amazonq_context", {}) 192 | attachments = getAttachments(event) 193 | 194 | # Get the IDC IAM credentials 195 | # Parse session JWT token to get the jti 196 | token = (event['req']['session']['idtokenjwt']) 197 | decoded_token = json.loads(base64.b64decode(token.split('.')[1] + '==').decode()) 198 | jti = decoded_token['jti'] 199 | 200 | dynamo_resource = boto3.resource('dynamodb') 201 | dynamo_table = dynamo_resource.Table(os.environ.get('DYNAMODB_CACHE_TABLE_NAME')) 202 | 203 | kms_client = boto3.client('kms') 204 | kms_key_id = os.environ.get("KMS_KEY_ID") 205 | 206 | # Check if JTI exists in caching DB 207 | response = dynamo_table.get_item(Key={'jti': jti}) 208 | 209 | if 'Item' in response: 210 | creds = json.loads((kms_client.decrypt( 211 | KeyId=kms_key_id, 212 | CiphertextBlob=response['Item']['Credentials'].value))['Plaintext']) 213 | else: 214 | creds = get_idc_iam_credentials(token) 215 | exp = creds['Expiration'].timestamp() 216 | creds.pop('Expiration') 217 | # Encrypt the credentials and store them in the caching DB 218 | encrypted_creds = \ 219 | kms_client.encrypt(KeyId=kms_key_id, 220 | Plaintext=bytes(json.dumps(creds).encode()))['CiphertextBlob'] 221 | dynamo_table.put_item(Item={'jti': jti, 'ExpiresAt': int(exp), 'Credentials': encrypted_creds}) 222 | 223 | # Assume the qbusiness role with the IDC IAM credentials to create the qbusiness client 224 | assumed_session = boto3.Session( 225 | aws_access_key_id=creds['AccessKeyId'], 226 | aws_secret_access_key=creds['SecretAccessKey'], 227 | aws_session_token=creds['SessionToken'] 228 | ) 229 | 230 | qbusiness_client = assumed_session.client("qbusiness") 231 | amazonq_response = get_amazonq_response(userInput, amazonq_context, attachments, qbusiness_client) 232 | event = format_response(event, amazonq_response) 233 | print("Returning response: %s" % json.dumps(event)) 234 | return event 235 | -------------------------------------------------------------------------------- /lambdas/qna_bot_qbusiness_lambdahook/README.md: -------------------------------------------------------------------------------- 1 | # QnABot LambdaHook for Amazon Q Business (preview) 2 | 3 | | :zap: The QnAbot LambdaHook for Amazon Q Business has been updated to accomodate migration to IAM Identicy Center. Please note some manual configuration steps are required and outlined below as part of the deployment process. | 4 | |-----------------------------------------| 5 | 6 | Amazon Q is a new generative AI-powered application that helps users get work done. Amazon Q can become your tailored business expert and let you discover content, brainstorm ideas, or create summaries using your company’s data safely and securely. For more information see: [Introducing Amazon Q, a new generative AI-powered assistant (preview)](https://aws.amazon.com/blogs/aws/introducing-amazon-q-a-new-generative-ai-powered-assistant-preview) 7 | 8 | In this repo we share a project which lets you use Amazon Q's generative AI to enable QnABot users to access your organization's data and knowledge sources via conversational question-answering. You can connect to your organization data via data source connectors and integrate it with the QnABot LambdaHook plugin for Amazon Q to enable access to your QnABot users. It allows your users to converse with Amazon Q using QnABot to ask questions and get answers based on company data, get help creating new content such as emails, and performing tasks. 9 | 10 | NEW! This plugin now supports attachments! Use the newest version of the [Lex Web UI](http://amazon.com/chatbotui) - version 0.20.4 or later - to add local file attachments to your conversation. There's more information on this feature in the Lex Web UI [File Upload README](https://github.com/aws-samples/aws-lex-web-ui/blob/master/README-file-upload.md). 11 | 12 | It's pretty cool. It's easy to deploy in your own AWS Account, and add to your own QnABot. We show you how below. 13 | 14 | ![Amazon Q Demo](../../images/AmazonQLambdaHook.png) 15 | 16 | ## Deploy Amazon Q (your business expert) as a fallback source of answers, using Lambda hooks with CustomNoMatches/no_hits 17 | 18 | ### Prerequisites 19 | 20 | 1. An existing deployment of a Q Business application. Please reference the AWS docs for creating a new [Q Business application](https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/create-application.html) 21 | 2. A deployment of the Lex Web UI with login enabled is required for this stack. To learn more about deploying the Web UI see the [Github repo for the solution](https://github.com/aws-samples/aws-lex-web-ui). This Cognito should be integrated with the same identity provider as your Identity Center (in the below example we will use IAM Identity Center as the IDP). 22 | 3. The Cognito user pool created by the Web UI will need to be added as **Trusted token issuer** to Identity Center by doing the following steps 23 | 1. Go to Identity Center and click on `Settings`, then `Create trusted token issuer` 24 | 2. The issuer URL will be `https://cognito-idp.[region].amazonaws.com/[cognito-pool-id]` and you will need to provide which attributes should map between the two. 25 | ![Issuer](../../images/token-issuer.PNG) 26 | 4. A custom application will need to be created in Identity Center to handle the connection between your Q Business application and your Cognito pool. Follow these steps to create the application. 27 | 1. Go to Identity Center and click on `Applications` then `Add application` 28 | 2. Select `I have an application I want to set up` and `OAuth 2.0` on the next page for Selecting Application type, then hit `Next` 29 | 3. For `Application URL`, provide the **Web experience URL** of your Q Business application (if you have a custom domain for your Q Business application, you would use the URL of that domain). You can either opt to assign specific users/groups to this application or allow any Identity Center users/groups to access the application. Your Q Business subscriptions will still apply however so only users with a subscription can successfully chat with the application. Then hit `Next` 30 | 4. Select the Trusted token issuer that was created in Step 2 of this guide, you will now need an aud claim so that the token issuer can identify the application. This aud claim is created when you deploy the Lex Web UI and can be found within the Coginto User pool. To find this value go to your Cognito user pool and select the `App integration` tab and scroll to the bottom. The aud claim is the **Client ID** value found under the App client list. Take this value and paste it into the aud claim field, then select `Next` 31 | ![Claim](../../images/aud-claim.PNG) 32 | 5. You will need to wait until after you deploy the CloudFormation stack to provide the role on the Specify application credentials page. For now, provide any existing IAM role in your environment and hit `Next`. 33 | 34 | ### Deploy a new Amazon Q (Business) Plugin stack 35 | 36 | Use AWS CloudFormation to deploy one or more of the sample plugin Lambdas in your own AWS account (if you do not have an AWS account, please see [How do I create and activate a new Amazon Web Services account?](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/)): 37 | 38 | 1. Log into the [AWS console](https://console.aws.amazon.com/) if you are not already. 39 | 2. Choose one of the **Launch Stack** buttons below for your desired AWS region to open the AWS CloudFormation console and create a new stack. 40 | 3. On the CloudFormation `Create Stack` page, click `Next` 41 | 4. Enter the following parameters: 42 | 1. `Stack Name`: Name your stack, e.g. QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK. 43 | 2. `AmazonQAppId`: Existing Amazon Q Application ID (copy from AWS console) 44 | 3. `AmazonQRegion`: Amazon Q Region (us-east-1, or us-west-2) 45 | 4. `DynamoDBTableName`: DynamoDB table that will be used to cache encrypted user credentials for question answering with QBusiness. 46 | 5. `IDCApplicationARN`: ARN of the Identity Center customer managed application created for QBusiness (see prerequisites for steps to create) 47 | 5. Launch the stack. 48 | 6. When your QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK Plugin CloudFormation stack status is CREATE_COMPLETE, choose the **Outputs** tab. Look for the output `QnAItemLambdaFunctionRoleArn` and modify your existing Identity Center application with this value by following these steps. 49 | 1. Go to Identity Center and click on `Applications` and find the application created for the QBusiness plugin. Click on the application to view more details. 50 | 2. Select `Actions->Edit configuration` to modify the settings of the application 51 | 3. Expand the Application credentials and paste the ARN obtained from the Outputs section. 52 | 4. Hit `Save changes` 53 | 54 | #### N. Virginia (us-east-1) 55 | Plugin | Launch Stack | Template URL 56 | --- | --- | --- 57 | QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml&stackName=QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml 58 | 59 | #### Oregon (us-west-2) 60 | Plugin | Launch Stack | Template URL 61 | --- | --- | --- 62 | QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml&stackName=QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml 63 | 64 | ## After your Amazon Q Plugin stack is deployed 65 | Configure QnAbot to prompt Amazon Q directly by configuring the AmazonQ LambdaHook function `QnAItemLambdaHookFunctionName` as a Lambda Hook for the QnABot [CustomNoMatches](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/using-keyword-filters-for.html) `no_hits` item. When QnABot cannot answer a question by any other means, it reverts to the `no_hits` item, which, when configured with this Lambda Hook function, will relay the question to Amazon Q. 66 | 67 | When your QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK Plugin CloudFormation stack status is CREATE_COMPLETE, choose the **Outputs** tab. Look for the outputs `QnAItemLambdaHookFunctionName` and `QnAItemLambdaHookArgs`. Use these values in the LambdaHook section of your no_hits item. You can change the value of "Prefix', or use "None" if you don't want to prefix the LLM answer. 68 | 69 | The default behavior is to relay the user's query to Amazon Q Business as the user input. If LLM_QUERY_GENERATION is enabled, the generated (disambiguated) query will be used, otherwise the user's utterance is used. 70 | Alternatively, you can supply an explicit `"Prompt"` key in the `QnAItemLambdaHookArgs` value. For example setting `QnAItemLambdaHookArgs` to `{"Prefix":"Amazon Q Answer:", "ShowContextText":true, "ShowSourceLinks":true, "Prompt":"Why is the sky blue?"}` will ignore the user's input and simply use the configured prompt instead. You may find this useful if you use the function as a Lambda Hook for QnA items that match explicit lists of utterances/questions, and you want to normalise these into a single static question to ask Amazon Q. Prompts supplied in this manner do not (yet) support variable substitution (eg to substitute user attributes, session attributes, etc. into the prompt). If you feel that would be a useful feature, please create a feature request issue in the repo, or, better yet, implement it, and submit a Pull Request! 71 | 72 | ### Say hello 73 | > Time to say Hi! 74 | 75 | 1. Go to QnAbot 76 | 2. Launch the Web client 77 | 4. Say *Hello*. And start asking questions! 78 | 5. Enjoy. 79 | 80 | ### Using file attachments 81 | 82 | This plugin now supports attachments! Use the newest version of the [Lex Web UI](http://amazon.com/chatbotui) - version 0.20.4 or later - to add local file attachments to your conversation. There's more information on this feature in the Lex Web UI [File Upload README](https://github.com/aws-samples/aws-lex-web-ui/blob/master/README-file-upload.md). 83 | When deploying or updating your Lex Web UI, you can reuse QnABot's existing **ImportBucket** name as the **UploadBucket** parameter - it already has a CORS policy that will work, and the Q Business plugin lambda role already grants read access to uploads in this bucket. To find your QnaBot's ImportBucket, use the `Resources` tab in the QnABot stack to search for the bucket reasorce with the logical name **ImportBucket**. 84 | 85 | Here's an example of what you can do with attachments - it is a beautiful thing! 86 | 87 | ![Amazon Q Demo](../../images/FileAttach.png) 88 | 89 | -------------------------------------------------------------------------------- /lambdas/bedrock-embeddings-and-llm/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Description: QnABot on AWS LLM Plugin for Bedrock - v0.1.15 3 | 4 | Parameters: 5 | 6 | EmbeddingsModelId: 7 | Type: String 8 | Default: amazon.titan-embed-text-v1 9 | AllowedValues: 10 | - amazon.titan-embed-text-v1 11 | Description: Bedrock Embeddings ModelId 12 | 13 | LLMModelId: 14 | Type: String 15 | Default: anthropic.claude-3-haiku-20240307-v1:0 16 | AllowedValues: 17 | - amazon.titan-text-express-v1 18 | - amazon.titan-text-lite-v1 19 | - ai21.j2-ultra-v1 20 | - ai21.j2-mid-v1 21 | - anthropic.claude-instant-v1 22 | - anthropic.claude-v2 23 | - anthropic.claude-v2:1 24 | - anthropic.claude-3-haiku-20240307-v1:0 25 | - anthropic.claude-3-sonnet-20240229-v1:0 26 | - anthropic.claude-3-opus-20240229-v1:0 27 | - cohere.command-text-v14 28 | - cohere.command-light-text-v14 29 | - meta.llama2-13b-chat-v1 30 | - meta.llama2-70b-chat-v1 31 | Description: Bedrock LLM ModelId 32 | 33 | Resources: 34 | 35 | BedrockBoto3Bucket: 36 | Type: AWS::S3::Bucket 37 | Properties: 38 | BucketEncryption: 39 | ServerSideEncryptionConfiguration: 40 | - ServerSideEncryptionByDefault: 41 | SSEAlgorithm: AES256 42 | Metadata: 43 | cfn_nag: 44 | rules_to_suppress: 45 | - id: W51 46 | reason: Bucket is to store boto3 package and does not contain any sensitive information. 47 | - id: W35 48 | reason: Bucket is to store boto3 package and does not contain any sensitive information. 49 | 50 | BedrockBoto3ZipFunctionRole: 51 | Type: AWS::IAM::Role 52 | Properties: 53 | AssumeRolePolicyDocument: 54 | Version: '2012-10-17' 55 | Statement: 56 | - Effect: Allow 57 | Principal: 58 | Service: lambda.amazonaws.com 59 | Action: sts:AssumeRole 60 | ManagedPolicyArns: 61 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 62 | Policies: 63 | - PolicyDocument: 64 | Version: 2012-10-17 65 | Statement: 66 | - Effect: Allow 67 | Action: 68 | - 's3:PutObject' 69 | - 's3:DeleteObject' 70 | - 's3:ListBucket' 71 | Resource: 72 | !Sub 'arn:aws:s3:::${BedrockBoto3Bucket}*' 73 | PolicyName: S3Policy 74 | 75 | BedrockBoto3ZipFunction: 76 | Type: AWS::Lambda::Function 77 | Properties: 78 | Handler: index.handler 79 | Runtime: python3.10 80 | Role: !GetAtt 'BedrockBoto3ZipFunctionRole.Arn' 81 | Timeout: 60 82 | MemorySize: 512 83 | Environment: 84 | Variables: 85 | BOTO3_BUCKET: !Ref BedrockBoto3Bucket 86 | Code: 87 | ZipFile: | 88 | import os 89 | import sys 90 | import re 91 | import shutil 92 | import subprocess 93 | import boto3 94 | import zipfile 95 | import urllib3 96 | import json 97 | from datetime import datetime 98 | import cfnresponse 99 | boto3_bucket = os.environ['BOTO3_BUCKET'] 100 | 101 | def upload_file_to_s3(file_path, bucket, key): 102 | s3 = boto3.client('s3') 103 | s3.upload_file(file_path, bucket, key) 104 | print(f"Upload successful. {file_path} uploaded to {bucket}/{key}") 105 | 106 | def make_zip_filename(): 107 | now = datetime.now() 108 | timestamp = now.strftime('%Y%m%d_%H%M%S') 109 | filename = f'BedrockBoto3SDK_{timestamp}.zip' 110 | return filename 111 | 112 | def zipdir(path, zipname): 113 | zipf = zipfile.ZipFile(zipname, 'w', zipfile.ZIP_DEFLATED) 114 | for root, dirs, files in os.walk(path): 115 | for file in files: 116 | zipf.write(os.path.join(root, file), 117 | os.path.relpath(os.path.join(root, file), 118 | os.path.join(path, '..'))) 119 | zipf.close() 120 | 121 | def deleteObject(existingZip): 122 | print(f'Deleting Existing Zip: {existingZip}') 123 | s3_client = boto3.client('s3') 124 | s3_client.delete_object(Bucket=existingZip["Bucket"], Key=existingZip["Key"]) 125 | return 126 | 127 | def handler(event, context): 128 | print("Event: ", json.dumps(event)) 129 | physicalResourceId = event.get("PhysicalResourceId", None) 130 | existingZip = None 131 | if physicalResourceId: 132 | try: 133 | existingZip = json.loads(physicalResourceId) 134 | except: 135 | existingZip = "" 136 | responseData={} 137 | reason="" 138 | status = cfnresponse.SUCCESS 139 | try: 140 | if event['RequestType'] != 'Delete': 141 | os.chdir('/tmp') 142 | print(f"running pip install boto3==1.28.57") 143 | subprocess.check_call([sys.executable, "-m", "pip", "install", "boto3==1.28.57", "-t", "python" ]) 144 | boto3_zip_name = make_zip_filename() 145 | zipdir("python",boto3_zip_name) 146 | print(f"uploading {boto3_zip_name} to s3 bucket {boto3_bucket}") 147 | upload_file_to_s3(boto3_zip_name, boto3_bucket, boto3_zip_name) 148 | responseData = {"Bucket": boto3_bucket, "Key": boto3_zip_name} 149 | physicalResourceId = json.dumps(responseData) 150 | else: 151 | print(f"RequestType is: {event['RequestType']}") 152 | except Exception as e: 153 | print(e) 154 | status = cfnresponse.FAILED 155 | reason = f"Exception thrown: {e}" 156 | # delete any previously created zip file (during update or delete), so that bucket can be deleted 157 | if existingZip: 158 | deleteObject(existingZip) 159 | cfnresponse.send(event, context, status, responseData, reason=reason, physicalResourceId=physicalResourceId) 160 | Metadata: 161 | cfn_nag: 162 | rules_to_suppress: 163 | - id: W89 164 | reason: Lambda function is not communicating with any VPC resources. 165 | - id: W92 166 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 167 | 168 | BedrockBoto3Zip: 169 | Type: Custom::BedrockBoto3Zip 170 | Properties: 171 | ServiceToken: !GetAtt BedrockBoto3ZipFunction.Arn 172 | # Rerun BedrockBoto3ZipFunction if any of the following parameters change 173 | BOTO3_BUCKET: !Ref BedrockBoto3Bucket 174 | VERSION: 1 175 | 176 | BedrockBoto3Layer: 177 | Type: "AWS::Lambda::LayerVersion" 178 | Properties: 179 | Content: 180 | S3Bucket: !GetAtt BedrockBoto3Zip.Bucket 181 | S3Key: !GetAtt BedrockBoto3Zip.Key 182 | CompatibleRuntimes: 183 | - python3.10 184 | 185 | LambdaFunctionRole: 186 | Type: AWS::IAM::Role 187 | Properties: 188 | AssumeRolePolicyDocument: 189 | Version: '2012-10-17' 190 | Statement: 191 | - Effect: Allow 192 | Principal: 193 | Service: lambda.amazonaws.com 194 | Action: sts:AssumeRole 195 | ManagedPolicyArns: 196 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 197 | Policies: 198 | - PolicyDocument: 199 | Version: 2012-10-17 200 | Statement: 201 | - Effect: Allow 202 | Action: 203 | - "bedrock:InvokeModel" 204 | Resource: 205 | - !Sub "arn:${AWS::Partition}:bedrock:*::foundation-model/*" 206 | - !Sub "arn:${AWS::Partition}:bedrock:*:${AWS::AccountId}:custom-model/*" 207 | PolicyName: BedrockPolicy 208 | 209 | EmbeddingsLambdaFunction: 210 | Type: AWS::Lambda::Function 211 | Properties: 212 | Handler: embeddings.lambda_handler 213 | Role: !GetAtt 'LambdaFunctionRole.Arn' 214 | Runtime: python3.11 215 | Layers: 216 | - !Ref BedrockBoto3Layer 217 | Timeout: 60 218 | MemorySize: 128 219 | Environment: 220 | Variables: 221 | DEFAULT_MODEL_ID: !Ref EmbeddingsModelId 222 | EMBEDDING_MAX_WORDS: 6000 223 | Code: ./src 224 | Metadata: 225 | cfn_nag: 226 | rules_to_suppress: 227 | - id: W89 228 | reason: Lambda function is not communicating with any VPC resources. 229 | - id: W92 230 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 231 | 232 | LLMLambdaFunction: 233 | Type: AWS::Lambda::Function 234 | Properties: 235 | Handler: llm.lambda_handler 236 | Role: !GetAtt 'LambdaFunctionRole.Arn' 237 | Runtime: python3.11 238 | Layers: 239 | - !Ref BedrockBoto3Layer 240 | Timeout: 60 241 | MemorySize: 128 242 | Code: ./src 243 | Metadata: 244 | cfn_nag: 245 | rules_to_suppress: 246 | - id: W89 247 | reason: Lambda function is not communicating with any VPC resources. 248 | - id: W92 249 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 250 | 251 | QnaItemLambdaHookFunction: 252 | Type: AWS::Lambda::Function 253 | Properties: 254 | # LambdaHook name must start with 'QNA-' to match QnAbot invoke policy 255 | FunctionName: !Sub "QNA-LAMBDAHOOK-${AWS::StackName}" 256 | Handler: "lambdahook.lambda_handler" 257 | Role: !GetAtt 'LambdaFunctionRole.Arn' 258 | Runtime: python3.11 259 | Timeout: 60 260 | MemorySize: 128 261 | Layers: 262 | - !Ref BedrockBoto3Layer 263 | Code: ./src 264 | Metadata: 265 | cfn_nag: 266 | rules_to_suppress: 267 | - id: W89 268 | reason: Lambda function is not communicating with any VPC resources. 269 | - id: W92 270 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 271 | 272 | OutputSettingsFunctionRole: 273 | Type: AWS::IAM::Role 274 | Properties: 275 | AssumeRolePolicyDocument: 276 | Version: '2012-10-17' 277 | Statement: 278 | - Effect: Allow 279 | Principal: 280 | Service: lambda.amazonaws.com 281 | Action: sts:AssumeRole 282 | ManagedPolicyArns: 283 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 284 | 285 | OutputSettingsFunction: 286 | Type: AWS::Lambda::Function 287 | Properties: 288 | Handler: settings.lambda_handler 289 | Role: !GetAtt 'OutputSettingsFunctionRole.Arn' 290 | Runtime: python3.11 291 | Timeout: 10 292 | MemorySize: 128 293 | Code: ./src 294 | Metadata: 295 | cfn_nag: 296 | rules_to_suppress: 297 | - id: W89 298 | reason: Lambda function is not communicating with any VPC resources. 299 | - id: W92 300 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 301 | 302 | OutputSettings: 303 | Type: Custom::OutputSettings 304 | Properties: 305 | ServiceToken: !GetAtt OutputSettingsFunction.Arn 306 | EmbeddingsModelId: !Ref EmbeddingsModelId 307 | LLMModelId: !Ref LLMModelId 308 | LastUpdate: '03/07/2024 12:20' 309 | 310 | TestBedrockModelFunction: 311 | Type: AWS::Lambda::Function 312 | Properties: 313 | Handler: testModel.lambda_handler 314 | Role: !GetAtt 'LambdaFunctionRole.Arn' 315 | Runtime: python3.11 316 | Layers: 317 | - !Ref BedrockBoto3Layer 318 | Timeout: 60 319 | MemorySize: 128 320 | Code: ./src 321 | Metadata: 322 | cfn_nag: 323 | rules_to_suppress: 324 | - id: W89 325 | reason: Lambda function is not communicating with any VPC resources. 326 | - id: W92 327 | reason: No requirements to set reserved concurrencies, function will not be invoked simultaneously. 328 | 329 | TestBedrockModel: 330 | Type: Custom::TestBedrockModel 331 | Properties: 332 | ServiceToken: !GetAtt TestBedrockModelFunction.Arn 333 | EmbeddingsModelId: !Ref EmbeddingsModelId 334 | LLMModelId: !Ref LLMModelId 335 | 336 | Outputs: 337 | 338 | BedrockBoto3Layer: 339 | Description: Lambda layer for Boto3 Bedrock SDK extensions 340 | Value: !Ref BedrockBoto3Layer 341 | 342 | EmbeddingsLambdaArn: 343 | Description: Lambda function for LLM (use for QnABot param "EmbeddingsLambdaArn") 344 | Value: !GetAtt 'EmbeddingsLambdaFunction.Arn' 345 | 346 | EmbeddingsLambdaDimensions: 347 | Description: Embeddings dimensions (use for QnABot param "EmbeddingsLambdaDimensions") 348 | Value: !GetAtt OutputSettings.EMBEDDINGS_DIMENSIONS 349 | 350 | LLMLambdaArn: 351 | Description: Lambda function for LLM (use for QnABot param "LLMLambdaArn") 352 | Value: !GetAtt LLMLambdaFunction.Arn 353 | 354 | QnABotSettingEmbeddingsScoreThreshold: 355 | Description: QnABot Designer Setting "EMBEDDINGS_SCORE_THRESHOLD" 356 | Value: !GetAtt OutputSettings.EMBEDDINGS_SCORE_THRESHOLD 357 | 358 | QnABotSettingEmbeddingsScoreAnswerThreshold: 359 | Description: QnABot Designer Setting "EMBEDDINGS_SCORE_ANSWER_THRESHOLD" 360 | Value: !GetAtt OutputSettings.EMBEDDINGS_SCORE_ANSWER_THRESHOLD 361 | 362 | QnABotSettingEmbeddingsTextPassageScoreThreshold: 363 | Description: QnABot Designer Setting "EMBEDDINGS_TEXT_PASSAGE_SCORE_THRESHOLD" 364 | Value: !GetAtt OutputSettings.EMBEDDINGS_TEXT_PASSAGE_SCORE_THRESHOLD 365 | 366 | QnABotSettingGenerateQueryPromptTemplate: 367 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_PROMPT_TEMPLATE" 368 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_PROMPT_TEMPLATE 369 | 370 | QnABotSettingGenerateQueryModelParams: 371 | Description: QnABot Designer Setting "LLM_GENERATE_QUERY_MODEL_PARAMS" 372 | Value: !GetAtt OutputSettings.LLM_GENERATE_QUERY_MODEL_PARAMS 373 | 374 | QnABotSettingQAPromptTemplate: 375 | Description: QnABot Designer Setting "LLM_QA_PROMPT_TEMPLATE" 376 | Value: !GetAtt OutputSettings.LLM_QA_PROMPT_TEMPLATE 377 | 378 | QnABotSettingQAModelParams: 379 | Description: QnABot Designer Setting "LLM_QA_MODEL_PARAMS" 380 | Value: !GetAtt OutputSettings.LLM_QA_MODEL_PARAMS 381 | 382 | QnAItemLambdaHookFunctionName: 383 | Description: QnA Item Lambda Hook Function Name (use with no_hits item for optional ask-the-LLM fallback) 384 | Value: !Ref QnaItemLambdaHookFunction 385 | 386 | QnAItemLambdaHookArgs: 387 | Description: QnA Item Lambda Hook Args (use with no_hits item for optional ask-the-LLM fallback) 388 | Value: !GetAtt OutputSettings.QNAITEM_LAMBDAHOOK_ARGS 389 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # QnABot on AWS Sample Plugins 2 | 3 | This repository provides sample plugin Lambda functions for use with the [QnABot on AWS](https://aws.amazon.com/solutions/implementations/qnabot-on-aws/) solution. 4 | 5 | The directions below explain how to build and deploy the plugins. For more information on the QnABot solution itself, see the [QnABot on AWS Solution Implementation Guide](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/solution-overview.html). 6 | 7 | ### Contents: 8 | 9 | *Plugins to extend QnABot LLM integration* 10 | 1. Amazon Bedrock Embeddings and LLM: Uses Amazon Bedrock service API 11 | 2. AI21 LLM: Uses AI21's Jurassic model API - requires an AI21 account with an API Key 12 | 3. Anthropic LLM: Uses Anthropic's Claude model API - requires an Anthropic account with an API Key 13 | 4. Llama 2 13b Chat LLM: Uses Llama 2 13b Chat model - requires Llama-2-chat model to be deployed via SageMaker JumpStart. Refer to [Llama 2 foundation models from Meta are now available in Amazon SageMaker JumpStart](https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/) on how to deploy the Llama-2-chat model in SageMaker JumpStart. 14 | 5. Mistral 7b Instruct LLM: Uses Mistral 7b Instruct model - requires Mistral 7b Instruct model to be deployed via SageMaker JumpStart. Refer to [Mistral 7B foundation models from Mistral AI are now available in Amazon SageMaker JumpStart](https://aws.amazon.com/blogs/machine-learning/mistral-7b-foundation-models-from-mistral-ai-are-now-available-in-amazon-sagemaker-jumpstart/) on how to deploy the Mistral 7B Instruct model in SageMaker JumpStart. 15 | 6. Amazon Q, your business expert: Integrates Amazon Q, your business expert (preview) with QnABot as a fallback answer source, using QnAbot's Lambda hooks with CustomNoMatches/no_hits. For more information see: [QnABot LambdaHook for Amazon Q, your business expert (preview)](./lambdas/qna_bot_qbusiness_lambdahook/README.md) 16 | 17 | ### (optional) Build and Publish QnABot Plugins CloudFormation artifacts 18 | 19 | _Note: Perform this step only if you want to create deployment artifacts in your own account. Otherwise, we have hosted a CloudFormation template for 1-click deployment in the [deploy](#deploy) section_. 20 | 21 | *Pre-requisite*: You must already have the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) installed and configured. You can use an AWS Cloud9 environment. 22 | 23 | Copy the plugins GitHub repo to your local machine. 24 | Either: 25 | - use the `git` command: `git clone https://github.com/aws-samples/qnabot-on-aws-plugin-samples.git` 26 | - OR, download and expand the ZIP file from the GitHub page: https://github.com/aws-samples/qnabot-on-aws-plugin-samples/archive/refs/heads/main.zip 27 | 28 | Then use the [publish.sh](./publish.sh) bash script in the project root directory to build the project and deploy CloudFormation templates to your own deployment bucket. 29 | 30 | Run the script with up to 3 parameters: 31 | ``` 32 | ./publish.sh [public] 33 | 34 | - : name of S3 bucket to deploy CloudFormation templates and code artifacts. If bucket does not exist, it will be created. 35 | - : artifacts will be copied to the path specified by this prefix (path/to/artifacts/) 36 | - public: (optional) Adding the argument "public" will set public-read acl on all published artifacts, for sharing with any account. 37 | ``` 38 | 39 | To deploy to a non-default region, set environment variable `AWS_DEFAULT_REGION` to a region supported by QnABot. See: [Supported AWS Regions](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/supported-aws-regions.html) 40 | E.g. to deploy in Ireland run `export AWS_DEFAULT_REGION=eu-west-1` before running the publish script. 41 | 42 | It downloads package dependencies, builds code zipfiles, and copies templates and zip files to the cfn_bucket. 43 | When completed, it displays the CloudFormation templates S3 URLs and 1-click URLs for launching the stack creation in CloudFormation console, e.g.: 44 | ``` 45 | ------------------------------------------------------------------------------ 46 | Outputs 47 | ------------------------------------------------------------------------------ 48 | QNABOTPLUGIN-AI21-LLM 49 | ============== 50 | - Template URL: https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/ai21-llm.yaml 51 | - Deploy URL: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/ai21-llm.yaml&stackName=QNABOTPLUGIN-AI21-LLM 52 | 53 | QNABOTPLUGIN-ANTHROPIC-LLM 54 | ============== 55 | - Template URL: https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/anthropic-llm.yaml 56 | - Deploy URL: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/anthropic-llm.yaml&stackName=QNABOTPLUGIN-ANTHROPIC-LLM 57 | 58 | QNABOTPLUGIN-BEDROCK-EMBEDDINGS-LLM 59 | ============== 60 | - Template URL: https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/bedrock-embeddings-llm.yaml 61 | - Deploy URL: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/bedrock-embeddings-llm.yaml&stackName=QNABOTPLUGIN-BEDROCK-EMBEDDINGS-LLM 62 | 63 | QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM 64 | ============== 65 | - Template URL: https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/llama-2-13b-chat-llm.yaml 66 | - Deploy URL: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/llama-2-13b-chat-llm.yaml&stackName=QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM 67 | 68 | QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM 69 | ============== 70 | - Template URL: https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/mistral-7b-instruct-chat-llm.yaml 71 | - Deploy URL: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/xxxxx-cfn-bucket/qnabot-plugins/mistral-7b-instruct-chat-llm.yaml&stackName=QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM 72 | ``` 73 | 74 | ### Deploy a new Plugin stack 75 | 76 | Use AWS CloudFormation to deploy one or more of the sample plugin Lambdas in your own AWS account (if you do not have an AWS account, please see [How do I create and activate a new Amazon Web Services account?](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/)): 77 | 78 | 1. Log into the [AWS console](https://console.aws.amazon.com/) if you are not already. 79 | *Note: Ensure that your IAM Role/User have permissions to create and manage the necessary resources and components for this application.* 80 | 2. Choose one of the **Launch Stack** buttons below for your desired LLM and AWS region to open the AWS CloudFormation console and create a new stack. 81 | 3. On the CloudFormation `Create Stack` page, click `Next` 82 | 4. Enter the following parameters: 83 | 1. `Stack Name`: Name your stack, e.g. QNABOTPLUGIN-LLM-AI21. 84 | 2. `APIKey`: Your Third-Party vendor account API Key, if applicable. The API Key is securely stored in AWS Secrets Manager. 85 | 3. `LLMModelId` and `EmbeddingsModelId` (for Bedrock), `LLMModel` (for Anthropic), `LLMModelType` (for AI21): Choose one of the available models to be used depending on the model provider. 86 | 87 | #### N. Virginia (us-east-1) 88 | Plugin | Launch Stack | Template URL 89 | --- | --- | --- 90 | QNABOT-BEDROCK-EMBEDDINGS-AND-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/bedrock-embeddings-and-llm.yaml&stackName=QNABOTPLUGIN-BEDROCK-EMBEDDINGS-AND-LLM) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/bedrock-embeddings-and-llm.yaml 91 | QNABOT-AI21-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/ai21-llm.yaml&stackName=QNABOTPLUGIN-AI21-LLM) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/ai21-llm.yaml 92 | QNABOT-ANTHROPIC-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/anthropic-llm.yaml&stackName=QNABOTPLUGIN-ANTHROPIC-LLM) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/anthropic-llm.yaml 93 | QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/llama-2-13b-chat-llm.yaml&stackName=QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/llama-2-13b-chat-llm.yaml 94 | QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/mistral-7b-instruct-chat-llm.yaml&stackName=QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/mistral-7b-instruct-chat-llm.yaml 95 | QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml&stackName=QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK) | https://s3.us-east-1.amazonaws.com/aws-ml-blog/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml 96 | 97 | #### Oregon (us-west-2) 98 | Plugin | Launch Stack | Template URL 99 | --- | --- | --- 100 | QNABOT-BEDROCK-EMBEDDINGS-AND-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/bedrock-embeddings-and-llm.yaml&stackName=QNABOTPLUGIN-BEDROCK-EMBEDDINGS-AND-LLM) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/bedrock-embeddings-and-llm.yaml 101 | QNABOT-AI21-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/ai21-llm.yaml&stackName=QNABOTPLUGIN-AI21-LLM) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/ai21-llm.yaml 102 | QNABOT-ANTHROPIC-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/anthropic-llm.yaml&stackName=QNABOTPLUGIN-ANTHROPIC-LLM) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/anthropic-llm.yaml 103 | QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/llama-2-13b-chat-llm.yaml&stackName=QNABOTPLUGIN-LLAMA-2-13B-CHAT-LLM) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/llama-2-13b-chat-llm.yaml 104 | QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/mistral-7b-instruct-chat-llm.yaml&stackName=QNABOTPLUGIN-MISTRAL-7B-INSTRUCT-CHAT-LLM) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/mistral-7b-instruct-chat-llm.yaml 105 | QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK | [![Launch Stack](https://cdn.rawgit.com/buildkite/cloudformation-launch-stack-button-svg/master/launch-stack.svg)](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml&stackName=QNABOTPLUGIN-QNA-BOT-QBUSINESS-LAMBDAHOOK) | https://s3.us-west-2.amazonaws.com/aws-ml-blog-us-west-2/artifacts/qnabot-on-aws-plugin-samples/qna_bot_qbusiness_lambdahook.yaml 106 | 107 | ## After your Plugin stack is deployed 108 | 109 | ### Configure a new or existing QnABot stack deployment to use your new LLM Plugin function 110 | 111 | When your CloudFormation stack status is CREATE_COMPLETE, choose the **Outputs** tab 112 | - Copy the value for `LLMLambdaArn` 113 | - Deploy a new QnABot Stack ([instructions](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/step-1-launch-the-stack.html)) or Update an existing QnABot stack ([instructions](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/update-the-solution.html)), selecting **LLMApi** parameter as `LAMBDA`, and for **LLMLambdaArn** parameter enter the Lambda Arn copied above. 114 | 115 | For more information, see [QnABot LLM README - Lambda Function](https://github.com/aws-solutions/qnabot-on-aws/blob/main/docs/LLM_Retrieval_and_generative_question_answering/README.md#2-lambda-function) 116 | 117 | 118 | ### Update QnABot Settings 119 | 120 | When the QnABot Cloudformation stack status is CREATE_COMPLETE or UPDATE_COMPLETE: 121 | - Keep your QnABot plugins CloudFormation stack **Outputs** tab open 122 | - In a new browser window, log into QnABot Content Designer (You can find the URL in the **Outputs** tab of your QnABot CloudFormation stack `ContentDesignerURL`). You will need to set your password for the first login. 123 | - From the Content Designer tools (☰) menu, choose **Settings** 124 | - From your QnABot plugins CloudFormation stack **Outputs** tab, copy setting values from each of the outputs named `QnABotSetting...` 125 | - use this copied value for the corresponding QnABot setting (identified in the output Description column) 126 | - do this for all settings. Note: the Bedrock stack has additional settings for Embeddings score thresholds. 127 | - Choose **Save** when complete. 128 | 129 | *Copy Stack Outputs:* 130 | 131 | Settings 132 | 133 | *To corresponding Designer Settings:* 134 | 135 | Settings 136 | - In a new browser window, access the QnABot Client URL (You can find the URL in the **Outputs** tab of your QnABot CloudFormation stack `ClientURL`), and start interacting with the QnA bot! 137 | 138 | ### (Optional) Configure QnABot to use your new Embeddings function *(currently only available for Bedrock)* 139 | 140 | When your CloudFormation stack status is **CREATE_COMPLETE**, choose the **Outputs** tab 141 | - Copy the value for `EmbeddingsLambdaArn` and `EmbeddingsLambdaDimensions` 142 | - Deploy a new QnABot Stack ([instructions](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/step-1-launch-the-stack.html)) or Update an existing QnABot stack ([instructions](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/update-the-solution.html)), selecting **EmbeddingsApi** as `LAMBDA`, and for **EmbeddingsLambdaArn** and **EmbeddingsLambdaDimensions** enter the Lambda Arn and embedding dimension values copied above. 143 | 144 | For more information, see [QnABot Embeddings README - Lambda Function](https://github.com/aws-solutions/qnabot-on-aws/tree/main/docs/semantic_matching_using_LLM_embeddings#3-lambda-function) 145 | 146 | ### (Optional) Modify Region and Endpoint URL 147 | 148 | The default AWS region and endpoint URL are set based on the CloudFormation deployed region and the default third-party LLM provider/Bedrock endpoint URL. To override the endpoint URL: 149 | 150 | - Once your CloudFormation stack status shows **CREATE_COMPLETE**, go to the Outputs tab and copy the Lambda Function Name [refer to green highlighted field above]. 151 | - In Lambda Functions, search for the Function Name. 152 | - Go to the Configuration tab, edit Environment Variables, add **ENDPOINT_URL** to override the endpoint URL. 153 | 154 | ### (Optional) Modify Third Party API Keys in Secrets Manager 155 | 156 | When your CloudFormation stack status is CREATE_COMPLETE, choose the **Outputs** tab. Use the link for `APIKeySecret` to open AWS Secrets Manager to inspect or edit your API Key in `Secret value`. 157 | 158 | ### Use the LLM as a fallback source of answers, using Lambda hooks with CustomNoMatches/no_hits 159 | 160 | Optionally configure QnAbot to prompt the LLM directly by configuring the LLM Plugin LambdaHook function `QnAItemLambdaHookFunctionName` as a Lambda Hook for the QnABot [CustomNoMatches](https://docs.aws.amazon.com/solutions/latest/qnabot-on-aws/keyword-filters-and-custom-dont-know-answers.html) `no_hits` item. When QnABot cannot answer a question by any other means, it reverts to the `no_hits` item, which, when configured with this Lambda Hook function, will relay the question to the LLM. 161 | 162 | When your Plugin CloudFormation stack status is CREATE_COMPLETE, choose the **Outputs** tab. Look for the outputs `QnAItemLambdaHookFunctionName` and `QnAItemLambdaHookArgs`. Use these values in the LambdaHook section of your no_hits item. You can change the value of "Prefix', or use "None" if you don't want to prefix the LLM answer. 163 | 164 | The default behavior is to relay the user's query to the LLM as the prompt. If LLM_QUERY_GENERATION is enabled, the generated (disambiguated) query will be used, otherwise the user's utterance is used. You can override this behavior by supplying an explicit `"Prompt"` key in the `QnAItemLambdaHookArgs` value. For example setting `QnAItemLambdaHookArgs` to `{"Prefix": "LLM Answer:", "Model_params": {"modelId": "anthropic.claude-instant-v1", "temperature": 0}, "Prompt":"Why is the sky blue?"}` will ignore the user's input and simply use the configured prompt instead. Prompts supplied in this manner do not (yet) support variable substitution (eg to substitute user attributes, session attributes, etc. into the prompt). If you feel that would be a useful feature, please create a feature request issue in the repo, or, better yet, implement it, and submit a Pull Request! 165 | 166 | Currently the Lambda hook option has been implemented only in the Bedrock, AI21, and (new!) AmazonQ (Business) plugins. 167 | For more infomation on the Amazon Q plugin, see [QnABot LambdaHook for Amazon Q, your business expert (preview)](./lambdas/qna_bot_qbusiness_lambdahook/README.md) 168 | 169 | 170 | LambdaHook 171 | 172 | LambdaHook 173 | 174 | 175 | ## Security 176 | 177 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 178 | 179 | ## License 180 | 181 | This library is licensed under the MIT-0 License. See the LICENSE file. 182 | --------------------------------------------------------------------------------