8 |
9 | ## Contents
10 |
11 | Outline the file contents of the repository. It helps users navigate the codebase, build configuration and any related assets.
12 |
13 | | Folder | Description |
14 | |-------------------|--------------------------------------------|
15 | | [azure-custom-speech](azure-custom-speech) | Sample data and instructions to create custom transcription model using Azure Speech service (Step 1 in above diagram). This step produces a sample custom speech model. This step also enables speech logging to capture real-time call audio. |
16 | | [azure-speech-streaming-reactjs](azure-speech-streaming-reactjs) | Java Script applications that simulates real-time call intelligence (Step 2 in above diagram). This application also captures audio conversation that could be used in the next step for batch call analytics. |
17 | | [call-batch-analytics](call-batch-analytics) | ARM template file and deployment guide for performing ingestion & batch analytics of calls using various Azure AI services (Step 3 in above diagram). This part of the solution can be used either with data output from step 2 OR using sample call recordings (if you have that). |
18 | | [powerbi](powerbi) | Template files and deployment guide for visualizing call insights using Power BI (Step 4 in above diagram). |
19 |
20 |
21 | ## Prerequisites
22 |
23 | * An existing [Azure Account](https://azure.microsoft.com/free/)
24 | * Ensure you have [Node.js](https://nodejs.org/en/download/) installed. Required for `Step 2` only.
25 | * Ensure you have [Power BI](https://powerbi.microsoft.com/en-us/downloads/) installed. Required for `Step 4` only.
26 |
27 |
28 | ## Dependencies
29 |
30 | This solution is modular and some part of the solution can be used independently and some components depends on other steps to be completed. In summary, real-time and batch call analytics can be used independently. Below is a list of dependencies:
31 | * Step 2 depends on Step 1 to be completed. Custom Speech model created in Step 1 is used in Step 2. Step 2 can be used without step 1 with minor code modifications (for advanced users only).
32 | * Step 3 and Step 4 can be used independently, if you have sample call recordings. If you don't have sample call recordings, then use Step 2 to simulate business conversations and capture the recording that you could use in this step.
33 |
34 | ## Getting started
35 |
36 | Follow the individual instructions for each step of the solution provided within above `Folders`.
37 |
38 |
--------------------------------------------------------------------------------
/azure-custom-speech/README.md:
--------------------------------------------------------------------------------
1 | ## What is Custom Speech?
2 |
3 | Azure [Custom Speech](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/custom-speech-overview) is a set of online tools that allow you to evaluate and improve the Microsoft speech-to-text accuracy for your applications, tools, and products. This powerful feature will be used in this sample to customize the Azure speech model for Insurance language - this helps model more accurately transcribe insurance words and phrases. Similarly, you could use the Cusom Speech to customize the model for different business scenarios for different industries - medical, finance, manufactoring, retail etc.
4 |
5 | # Steps to create & deploy a Custom Speech model
6 |
7 | Before you can do anything with Custom Speech, you'll need an Azure account and a Speech service subscription. After you have an account, you can prep your data, train and test your models, inspect recognition quality, evaluate accuracy, and ultimately deploy and use the custom speech-to-text model.
8 |
9 | This diagram highlights the pieces that make up the [Custom Speech area of the Speech Studio](https://aka.ms/speechstudio/customspeech). Use the links below to learn more about each step.
10 |
11 | 
12 |
13 | 1. [Bring your own storage (BYOS) for Speech logging](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-encryption-of-data-at-rest#bring-your-own-storage-byos-for-customization-and-logging) feature will be used to enable speech logging. Azure Speech Service supports automatically logging audio content to customer managed Azure storage account. This is very useful when you are using speech for real-time interaction and want the service to capture audio for post analysis, archival, compliance etc. This will allow us capture real-time audio for batch analytics. In this pattern, speech service will return transcription in real-time to the client and also capture audio in the Azure storage account specified here. `You might have to request access to this preview feature. Follow instructions [here](https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-encryption-of-data-at-rest#bring-your-own-storage-byos-for-customization-and-logging)'
14 |
15 | 2. [Create Speech Service resource](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices). Create a Speech service resource using Azure portal. Select “Yes” for bring your own storage box when creating a new Azure Speech service resource. `NOTE: You need to create a Speech Resource with a paid (S0) key. The free key account will not work for the batch analytics step.`
16 |
17 | 3. `Create a new custom speech project` on [Speech Studio](https://speech.microsoft.com/customspeech) using the newly created speech resource from the previous step. Content like data, models, tests, and endpoints are organized into *projects* in the [Speech Studio](https://speech.microsoft.com/customspeech). Each project is specific to a domain and country/language. For example, you might create a project for call centers that use English in the United States.
18 |
19 | To create your first project, select **Speech-to-text/Custom speech**, and then select **New Project**. Follow the instructions provided by the wizard to create your project. After you create a project, you should see four tabs: **Data**, **Testing**, **Training**, and **Deployment**.
20 |
21 | 4. Upload test data to customize the speech model for business lanaguage and terms. You can use sample language adaptation data provided under `azure-custom-speech\sampledata\InsuranceLanguage-Data1.txt`. Select "Plain text" as the data type when uploading txt file as shown below.
22 |
23 | 
24 |
25 | 5. `Train a model`. Improve the accuracy of your speech-to-text model by providing related text (<200 MB). This data helps to train the speech-to-text model.
26 |
27 | 6. `Deploy a model`. After training, if you're satisfied with the result, you can deploy your model to a custom endpoint. Create a new custom endpoint and select “Enable content logging (audio and diagnostic information) for this endpoint.” as shown in screenshot below.
28 |
29 | 
30 |
31 | 7. `Get Custom Speech Model Endpoint ID`- Once you have successully deployed a custom speech model. click on the deployed model and get the "Endpoint ID" as shown below. You will need the "Endpoint ID", "Service region" and "Subscription Key" in the next step.
32 |
33 | 
34 |
--------------------------------------------------------------------------------
/azure-custom-speech/sampledata/InsuranceLanguage-Data1.txt:
--------------------------------------------------------------------------------
1 | The insurance industry safeguards the assets of its policyholders by transferring
2 | risk from an individual or business to an insurance company. Insurance companies act as financial intermediaries in that they invest the premiums they collect
3 | for providing this service. Insurance company size is usually measured by net
4 | premiums written, that is, premium revenues less amounts paid for reinsurance.
5 | There are three main insurance sectors: property/casualty, life/health and health
6 | insurance. Property/casualty (P/C) consists mainly of auto, home and commercial insurance. Life/health (L/H) consists mainly of life insurance and annuity
7 | products. Health insurance is offered by private health insurance companies
8 | and some L/H and P/C insurers, as well as by government programs such as
9 | Medicare.
10 |
11 | All types of insurance are regulated by the states, with each state having its
12 | own set of statutes and rules. State insurance departments oversee insurer solvency, market conduct and, to a greater or lesser degree, review and rule on
13 | requests for rate increases for coverage. The National Association of Insurance
14 | Commissioners develops model rules and regulations for the industry, many
15 | of which must be approved by state legislatures. The McCarran-Ferguson Act,
16 | passed by Congress in 1945, refers to continued state regulation of the insurance
17 | industry as being in the public interest. Under the 1999 Gramm-Leach-Bliley
18 | Financial Services Modernization Act, insurance activities—whether conducted
19 | by banks, broker-dealers or insurers—are regulated by the states. However, there
20 | have been, and continue to be, challenges to state regulation from some segments of the federal government as well as from some financial services firms.
21 |
22 | Insurers are required to use statutory accounting principles (SAP) when filing
23 | annual financial reports with state regulators and the Internal Revenue Service. SAP,
24 | which evolved to enhance the industry’s financial stability, is more conservative
25 | than the generally accepted accounting principles (GAAP), established by the independent Financial Accounting Standards Board (FASB). The Securities and Exchange
26 | Commission (SEC) requires publicly owned companies to report their financial
27 | results using GAAP rules. Insurers outside the United States use standards that differ from SAP and GAAP. As global markets developed, the need for more uniform
28 | accounting standards became clear. In 2001 the International Accounting Standards
29 | Board (IASB), an independent international accounting standards setting organization, began work on a set of standards, called International Financial Reporting
30 | Standards (IFRS) that it hopes will be used around the world. Since 2001 over 100
31 | countries have required or permitted the use of IFRS.
32 | In 2007 the SEC voted to stop requiring non-U.S. companies that use IFRS
33 | to re-issue their financial reports for U.S. investors using GAAP. In 2008 the
34 | National Association of Insurance Commissioners began to explore ways to
35 | move from statutory accounting principles to IFRS. Also in 2008, the FASB and
36 | IASB undertook a joint project to develop a common and improved framework
37 | for financial reporting.
38 |
39 | Property/casualty and life insurance policies were once sold almost exclusively
40 | by agents—either by captive agents, representing one insurance company, or
41 | by independent agents, representing several companies. Insurance companies
42 | selling through captive agents and/or by mail, telephone or via the Internet
43 | are called “direct writers.” However, the distinctions between direct writers and
44 | independent agency companies have been blurring since the 1990s, when insurers began to use multiple channels to reach potential customers. In addition, in
45 | the 1980s banks began to explore the possibility of selling insurance through
46 | independent agents, usually buying agencies for that purpose. Other distribution channels include sales through professional organizations and through
47 | workplaces.
48 |
49 | Auto insurance protects against financial loss in the event of an accident. It is a
50 | contract between the policyholder and the insurance company. The policyholder agrees to pay the premium and the insurance company agrees to pay losses as
51 | defined in the policy.
52 | Auto insurance provides property, liability and medical coverage:
53 | Property coverage pays for damage to, or theft of, the car.
54 | Liability coverage pays for the policyholder’s legal responsibility to
55 | others for bodily injury or property damage.
56 | Medical coverage pays for the cost of treating injuries, rehabilitation
57 | and sometimes lost wages and funeral expenses.
58 | Most states require drivers to have auto liability insurance before they can legally drive a car. (Liability insurance pays the other driver’s medical, car repair and
59 | other costs when the policyholder is at fault in an auto accident.) All states have
60 | laws that set the minimum amounts of insurance or other financial security
61 | drivers have to pay for the harm caused by their negligence behind the wheel if
62 | an accident occurs. Most auto policies are for six months to a year. A basic auto
63 | insurance policy is comprised of six different kinds of coverage, each of which is
64 | priced separately (see below).
65 | Bodily Injury Liability
66 | This coverage applies to injuries that the policyholder and family members listed on the policy cause to someone else. These individuals are also covered when
67 | driving other peoples’ cars with permission. As motorists in serious accidents
68 | may be sued for large amounts, drivers can opt to buy more than the staterequired minimum to protect personal assets such as homes and savings.
69 | Medical Payments or Personal Injury Protection (PIP)
70 | This coverage pays for the treatment of injuries to the driver and passengers
71 | of the policyholder’s car. At its broadest, PIP can cover medical payments,
72 | lost wages and the cost of replacing services normally performed by someone
73 | injured in an auto accident. It may also cover funeral costs.
74 | Property Damage Liability
75 | This coverage pays for damage policyholders (or someone driving the car with
76 | their permission) may cause to someone else’s property. Usually, this means
77 | damage to someone else’s car, but it also includes damage to lamp posts, telephone poles, fences, buildings or other structures hit in an accident.
78 |
79 | This coverage pays for damage to the policyholder’s car resulting from a collision with another car, an object or as a result of flipping over. It also covers
80 | damage caused by potholes. Collision coverage is generally sold with a deductible of $250 to $1,000—the higher the deductible, the lower the premium. Even
81 | if policyholders are at fault for an accident, collision coverage will reimburse
82 | them for the costs of repairing the car, minus the deductible. If the policyholder
83 | is not at fault, the insurance company may try to recover the amount it paid
84 | from the other driver’s insurance company, a process known as subrogation. If
85 | the company is successful, policyholders will also be reimbursed for the deductible.
86 |
87 | This coverage reimburses for loss due to theft or damage caused by something
88 | other than a collision with another car or object, such as fire, falling objects,
89 | missiles, explosions, earthquakes, windstorms, hail, flood, vandalism and riots,
90 | or contact with animals such as birds or deer. Comprehensive insurance is usually sold with a $100 to $300 deductible, though policyholders may opt for a
91 | higher deductible as a way of lowering their premium. Comprehensive insurance may also reimburse the policyholder if a windshield is cracked or shattered.
92 | Some companies offer separate glass coverage with or without a deductible.
93 | States do not require the purchase of collision or comprehensive coverage, but
94 | lenders may insist borrowers carry it until a car loan is paid off. It may also be a
95 | requirement of some dealerships if a car is leased.
96 | 6. Uninsured and Underinsured Motorist Coverage
97 | Uninsured motorist coverage will reimburse the policyholder, a member of the
98 | family or a designated driver if one of them is hit by an uninsured or a hit-andrun driver. Underinsured motorist coverage comes into play when an at-fault
99 | driver has insufficient insurance to pay for the other driver’s total loss. This coverage will also protect a policyholder who is hit while a pedestrian.
100 |
101 | Homeowners insurance provides financial protection against disasters. It is a package policy, which means that it covers both damage to property and liability, or
102 | legal responsibility, for any injuries and property damage policyholders or their
103 | families cause to other people. This includes damage caused by household pets.
104 | Damage caused by most disasters is covered but there are exceptions. Standard
105 | homeowners policies do not cover flooding, earthquakes or poor maintenance.
106 | Flood coverage, however, is available in the form of a separate policy both from
107 | the National Flood Insurance Program (NFIP) and from a few private insurers. Earthquake coverage is available either in the form of an endorsement or
108 | as a separate policy. Most maintenance-related problems are the homeowners’
109 | responsibility.
110 | A standard homeowners insurance policy includes four essential types of
111 | coverage. They include:
112 | Coverage for the Structure of the Home
113 | This part of a policy pays to repair or rebuild a home if it is damaged or
114 | destroyed by fire, hurricane, hail, lightning or other disaster listed in the policy.
115 | It will not pay for damage caused by a flood, earthquake or routine wear and
116 | tear. Most standard policies also cover structures that are not attached to a
117 | house such as a garage, tool shed or gazebo. Generally, these structures are covered for about 10 percent of the total amount of insurance on the structure of
118 | the home.
119 | Coverage for Personal Belongings
120 | Furniture, clothes, sports equipment and other personal items are covered if
121 | they are stolen or destroyed by fire, hurricane or other insured disaster. Most
122 | companies provide coverage for 50 to 70 percent of the amount of insurance on
123 | the structure of a home. This part of the policy includes off-premises coverage.
124 | This means that belongings are covered anywhere in the world, unless the policyholder has decided against off-premises coverage. Expensive items like jewelry,
125 | furs and silverware are covered, but there are usually dollar limits if they are stolen. To insure these items to their full value, individuals can purchase a special
126 | personal property endorsement or floater and insure the item for its appraised
127 | value.
128 | Trees, plants and shrubs are also covered under standard homeowners insurance—generally up to about $500 per item. Perils covered are theft, fire, lightning, explosion, vandalism, riot and even falling aircraft. They are not covered
129 | for damage by wind or disease
130 |
131 | Liability coverage protects against the cost of lawsuits for bodily injury or property damage that policyholders or family members cause to other people. It also
132 | pays for damage caused by pets. The liability portion of the policy pays for both
133 | the cost of defending the policyholder in court and any court awards—up to the
134 | limit of the policy. Coverage is not just in the home but extends to anywhere
135 | in the world. Liability limits generally start at about $100,000. However, experts
136 | recommend that homeowners purchase at least $300,000 worth of protection.
137 | An umbrella or excess liability policy, which provides broader coverage, including claims for libel and slander, as well as higher liability limits, can be added to
138 | the policy. Generally, umbrella policies cost between $200 to $350 for $1 million of additional liability protection.
139 | Homeowners policies also provide no-fault medical coverage. In the event
140 | that someone is injured in a policyholder’s home, the injured person can simply submit medical bills to the policyholder’s insurance company. In this way
141 | expenses are paid without a liability claim being filed. This coverage, however,
142 | does not pay the medical bills for the policyholder’s own family or pets.
143 | 4. Additional Living Expenses
144 | This pays the additional costs of living away from home if a house is inhabitable due to damage from a fire, storm or other insured disaster. It covers hotel
145 | bills, restaurant meals and other extra living expenses incurred while the home
146 | is being rebuilt. Coverage for additional living expenses differs from company to
147 | company. Many policies provide coverage for about 20 percent of the insurance
148 | on a house. The coverage can be increased for an additional premium. Some
149 | companies sell a policy that provides an unlimited amount of loss-of-use coverage, but for a limited amount of time.
150 | Additional living expense coverage also reimburses homeowners who rent
151 | out part of their home for the rent that would have been collected from a tenant if the home had not been destroyed.
152 | Types of Homeowners Insurance Policies
153 | There are several types of homeowners insurance policies that differ in the amount
154 | of insurance coverage they provide. The different types are fairly standard throughout the country. However, individual states and companies may offer policies that
155 | are slightly different or go by other names such as “standard” or “deluxe.” People
156 | who rent the homes they live in have specific renters policies.
157 |
--------------------------------------------------------------------------------
/azure-custom-speech/sampledata/b0013.wav:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-custom-speech/sampledata/b0013.wav
--------------------------------------------------------------------------------
/azure-custom-speech/sampledata/test_audio.wav:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-custom-speech/sampledata/test_audio.wav
--------------------------------------------------------------------------------
/azure-custom-speech/sampledata/test_audio_stereo.wav:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-custom-speech/sampledata/test_audio_stereo.wav
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/.gitignore:
--------------------------------------------------------------------------------
1 | # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
2 |
3 | # dependencies
4 | /speechexpressbackend/node_modules
5 | /speechreactfrontend/node_modules
6 | /front-end-ui/node_modules
7 | /node_modules
8 | /.pnp
9 | .pnp.js
10 |
11 | # testing
12 | /coverage
13 |
14 | # production
15 | /build
16 |
17 | # misc
18 | .DS_Store
19 | .env.local
20 | .env.development.local
21 | .env.test.local
22 | .env.production.local
23 |
24 | npm-debug.log*
25 | yarn-debug.log*
26 | yarn-error.log*
27 |
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/README.md:
--------------------------------------------------------------------------------
1 | # Real-time Transcription using Azure Speech in ReactJS
2 |
3 | This sample simulates call center intelligence in real-time using Azure AI services. The code also records the conversation to Azure storage using the conversation ID provided by the user on web UI.
4 |
5 | This sample shows design pattern examples for authentication token exchange and management, as well as capturing audio from a microphone or file for speech-to-text conversions.
6 |
7 | Below architecture diagram depicts key components and API/communication sequence used in this sample
8 | 
9 |
10 | This sample uses Express.js backend framework which allows you to make http calls from any front end. ReactJS is used for frontend app. *NOTE*: This sample is only using the Azure Speech SDK - it does not use Azure Bot Service and Direct Line Speech channel.
11 |
12 | * **Express.js**: Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications. It facilitates the rapid development of Node based web applications.
13 |
14 | * **React.js** often referred to as React or ReactJS is a JavaScript library responsible for building a hierarchy of UI components or in other words, responsible for the rendering of UI components. It provides support for both frontend and server-side.
15 |
16 | ## Prerequisites
17 |
18 | 1. This article assumes that you have an Azure account and Speech service subscription. If you don't have an account and subscription, [try the Speech service for free](https://docs.microsoft.com/azure/cognitive-services/speech-service/overview#try-the-speech-service-for-free).
19 | 1. Ensure you have [Node.js](https://nodejs.org/en/download/) installed.
20 |
21 | ## How to run the app
22 |
23 | 1. Clone this repo. This repo has two apps as shown in the architecture diagram above:
24 | * speechreactfrontend folder is for the "ReactJS Frontend" component and
25 | * speechexpressbackend folder is for the "ExpressJS Backend" component
26 |
27 |
28 | 2. **Prepare and run the Speech service Express.js backend**
29 | - Go to speechexpressbacked directory and run `npm install -all` to install dependencies.
30 | - Update the “.env” file with your Azure Speech service key and Azure region. Azure Region value examples: “eastus2”, “westus”
31 | - Start Speech service backend app by running `‘npm start’`
32 | - If you are running this locally then try accessing below URLs from browser to verify that the backend component is working as expected
33 | * `http://localhost:8080/api/sayhello`
34 | * `http://localhost:8080/api/get-speech-token`
35 | - If you have deployed speechexpressbacked app to Azure App Service (as per instructions below) then you can verify using URLs from browser as below:
36 | * `https://<
48 |
49 |
50 | + If you have also deployed the frontend ReactJS to Azure App Service then use the deployed app service URL which you can find on Azure portal for your App Service. Example: `https://myspeechreactfrontend.azurewebsites.net`
51 |
52 |
53 |
54 | ## Deploying sample code to Azure App Service
55 | You can deploy your Node.js app using VS Code and the Azure App Service extension. Follow instructions [Deploy NodeJS using Azure App Service]:https://docs.microsoft.com/en-us/azure/app-service/quickstart-nodejs?pivots=platform-linux#deploy-to-azure that explains how to deploy any node app to Azure App Service.
56 |
57 | * To deploy **speechexpressbacked** to Azure App Service, select the “speechexpressbacked” as the root folder when prompted in the VS code.
58 | - Validate that your ExpressJS backend is successfully deployed by trying to access one of the two APIs hosted by your backend
59 | - `https://<‘process.env.WEB_PORT || 8080’ |
73 |
74 |
75 | ## Change recognition language
76 |
77 | To change the source recognition language, change the locale strings in `App.js` lines **32** and **66**, which sets the recognition language property on the `SpeechConfig` object.
78 |
79 | ```javascript
80 | speechConfig.speechRecognitionLanguage = 'en-US'
81 | ```
82 |
83 | For a full list of supported locales, see the [language support article](https://docs.microsoft.com/azure/cognitive-services/speech-service/language-support#speech-to-text).
84 |
85 | ## Speech-to-text from microphone
86 |
87 | To convert speech-to-text using a microphone, run the app and then click **Convert speech to text from your mic.**. This will prompt you for access to your microphone, and then listen for you to speak. The following function `sttFromMic` in `App.js` contains the implementation.
88 |
89 | ```javascript
90 | async sttFromMic() {
91 | const tokenObj = await getTokenOrRefresh();
92 | const speechConfig = speechsdk.SpeechConfig.fromAuthorizationToken(tokenObj.authToken, tokenObj.region);
93 | speechConfig.speechRecognitionLanguage = 'en-US';
94 |
95 | const audioConfig = speechsdk.AudioConfig.fromDefaultMicrophoneInput();
96 | const recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig);
97 |
98 | this.setState({
99 | displayText: 'speak into your microphone...'
100 | });
101 |
102 | recognizer.recognizeOnceAsync(result => {
103 | let displayText;
104 | if (result.reason === ResultReason.RecognizedSpeech) {
105 | displayText = `RECOGNIZED: Text=${result.text}`
106 | } else {
107 | displayText = 'ERROR: Speech was cancelled or could not be recognized. Ensure your microphone is working properly.';
108 | }
109 |
110 | this.setState({
111 | displayText: displayText
112 | });
113 | });
114 | }
115 | ```
116 |
117 | Running speech-to-text from a microphone is done by creating an `AudioConfig` object and using it with the recognizer.
118 |
119 | ```javascript
120 | const audioConfig = speechsdk.AudioConfig.fromDefaultMicrophoneInput();
121 | const recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig);
122 | ```
123 |
124 | ## Speech-to-text from file
125 |
126 | To convert speech-to-text from an audio file, run the app and then click **Convert speech to text from an audio file.**. This will open a file browser and allow you to select an audio file. The following function `fileChange` is bound to an event handler that detects the file change.
127 |
128 | ```javascript
129 | async fileChange(event) {
130 | const audioFile = event.target.files[0];
131 | console.log(audioFile);
132 | const fileInfo = audioFile.name + ` size=${audioFile.size} bytes `;
133 |
134 | this.setState({
135 | displayText: fileInfo
136 | });
137 |
138 | const tokenObj = await getTokenOrRefresh();
139 | const speechConfig = speechsdk.SpeechConfig.fromAuthorizationToken(tokenObj.authToken, tokenObj.region);
140 | speechConfig.speechRecognitionLanguage = 'en-US';
141 |
142 | const audioConfig = speechsdk.AudioConfig.fromWavFileInput(audioFile);
143 | const recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig);
144 |
145 | recognizer.recognizeOnceAsync(result => {
146 | let displayText;
147 | if (result.reason === ResultReason.RecognizedSpeech) {
148 | displayText = `RECOGNIZED: Text=${result.text}`
149 | } else {
150 | displayText = 'ERROR: Speech was cancelled or could not be recognized. Ensure your microphone is working properly.';
151 | }
152 |
153 | this.setState({
154 | displayText: fileInfo + displayText
155 | });
156 | });
157 | }
158 | ```
159 |
160 | You need the audio file as a JavaScript [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File) object, so you can grab it directly off the event target using `const audioFile = event.target.files[0];`. Next, you use the file to create the `AudioConfig` and then pass it to the recognizer.
161 |
162 | ```javascript
163 | const audioConfig = speechsdk.AudioConfig.fromWavFileInput(audioFile);
164 | const recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig);
165 | ```
166 |
167 | ## Token exchange process
168 |
169 | This sample application shows an example design pattern for retrieving and managing tokens, a common task when using the Speech JavaScript SDK in a browser environment. A simple Express back-end is implemented in the same project under `server/index.js`, which abstracts the token retrieval process.
170 |
171 | The reason for this design is to prevent your speech key from being exposed on the front-end, since it can be used to make calls directly to your subscription. By using an ephemeral token, you are able to protect your speech key from being used directly. To get a token, you use the Speech REST API and make a call using your speech key and region. In the Express part of the app, this is implemented in `index.js` behind the endpoint `/api/get-speech-token`, which the front-end uses to get tokens.
172 |
173 | ```javascript
174 | app.get('/api/get-speech-token', async (req, res, next) => {
175 | res.setHeader('Content-Type', 'application/json');
176 | const speechKey = process.env.SPEECH_KEY;
177 | const speechRegion = process.env.SPEECH_REGION;
178 |
179 | if (speechKey === 'paste-your-speech-key-here' || speechRegion === 'paste-your-speech-region-here') {
180 | res.status(400).send('You forgot to add your speech key or region to the .env file.');
181 | } else {
182 | const headers = {
183 | headers: {
184 | 'Ocp-Apim-Subscription-Key': speechKey,
185 | 'Content-Type': 'application/x-www-form-urlencoded'
186 | }
187 | };
188 |
189 | try {
190 | const tokenResponse = await axios.post(`https://${speechRegion}.api.cognitive.microsoft.com/sts/v1.0/issueToken`, null, headers);
191 | res.send({ token: tokenResponse.data, region: speechRegion });
192 | } catch (err) {
193 | res.status(401).send('There was an error authorizing your speech key.');
194 | }
195 | }
196 | });
197 | ```
198 |
199 | In the request, you create a `Ocp-Apim-Subscription-Key` header, and pass your speech key as the value. Then you make a request to the **issueToken** endpoint for your region, and an authorization token is returned. In a production application, this endpoint returning the token should be *restricted by additional user authentication* whenever possible.
200 |
201 | On the front-end, `token_util.js` contains the helper function `getTokenOrRefresh` that is used to manage the refresh and retrieval process.
202 |
203 | ```javascript
204 | export async function getTokenOrRefresh() {
205 | const cookie = new Cookie();
206 | const speechToken = cookie.get('speech-token');
207 |
208 | if (speechToken === undefined) {
209 | try {
210 | const res = await axios.get('/api/get-speech-token');
211 | const token = res.data.token;
212 | const region = res.data.region;
213 | cookie.set('speech-token', region + ':' + token, {maxAge: 540, path: '/'});
214 |
215 | console.log('Token fetched from back-end: ' + token);
216 | return { authToken: token, region: region };
217 | } catch (err) {
218 | console.log(err.response.data);
219 | return { authToken: null, error: err.response.data };
220 | }
221 | } else {
222 | console.log('Token fetched from cookie: ' + speechToken);
223 | const idx = speechToken.indexOf(':');
224 | return { authToken: speechToken.slice(idx + 1), region: speechToken.slice(0, idx) };
225 | }
226 | }
227 | ```
228 |
229 | This function uses the `universal-cookie` library to store and retrieve the token from local storage. It first checks to see if there is an existing cookie, and in that case it returns the token without hitting the Express back-end. If there is no existing cookie for a token, it makes the call to `/api/get-speech-token` to fetch a new one. Since we need both the token and its corresponding region later, the cookie is stored in the format `token:region` and upon retrieval is spliced into each value.
230 |
231 | Tokens for the service expire after 10 minutes, so the sample uses the `maxAge` property of the cookie to act as a trigger for when a new token needs to be generated. It is reccommended to use 9 minutes as the expiry time to act as a buffer, so we set `maxAge` to **540 seconds**.
232 |
233 | In `App.js` you use `getTokenOrRefresh` in the functions for speech-to-text from a microphone, and from a file. Finally, use the `SpeechConfig.fromAuthorizationToken` function to create an auth context using the token.
234 |
235 | ```javascript
236 | const tokenObj = await getTokenOrRefresh();
237 | const speechConfig = speechsdk.SpeechConfig.fromAuthorizationToken(tokenObj.authToken, tokenObj.region);
238 | ```
239 |
240 | In many other Speech service samples, you will see the function `SpeechConfig.fromSubscription` used instead of `SpeechConfig.fromAuthorizationToken`, but by **avoiding the usage** of `fromSubscription` on the front-end, you prevent your speech subscription key from becoming exposed, and instead utilize the token authentication process. `fromSubscription` is safe to use in a Node.js environment, or in other Speech SDK programming languages when the call is made on a back-end, but it is best to avoid using in a browser-based JavaScript environment.
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/common/images/sampleoutputrealtimetranscription.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-speech-streaming-reactjs/common/images/sampleoutputrealtimetranscription.PNG
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/common/images/speechstreamingdiagram.PNG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-speech-streaming-reactjs/common/images/speechstreamingdiagram.PNG
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.env:
--------------------------------------------------------------------------------
1 | REACT_APP_PLATFORM=desktop
2 | REACT_APP_BACKEND_API="http://localhost:8080"
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.env.desktop:
--------------------------------------------------------------------------------
1 | REACT_APP_PLATFORM=desktop
2 | REACT_APP_BACKEND_API="http://localhost:8080"
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.env.hosted.microsoft:
--------------------------------------------------------------------------------
1 | REACT_APP_PLATFORM=hosted
2 | REACT_APP_BACKEND_API="https://speechexpressbackendamc.azurewebsites.net"
3 | REACT_APP_CLIENT_ID=6798e375-a31f-48b0-abcb-87f06d70d0b6
4 | REACT_APP_REDIRECT_URI=https://speechreactfrontendamc.azurewebsites.net/
5 | REACT_APP_POST_LOGOUT_REDIRECT_URI=http://localhost:3000/
6 | REACT_APP_TENANT_ID=72f988bf-86f1-41af-91ab-2d7cd011db47
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.env.hosted.progger:
--------------------------------------------------------------------------------
1 | REACT_APP_PLATFORM=hosted
2 | REACT_APP_BACKEND_API="https://speechexpressbackendamc.azurewebsites.net"
3 | REACT_APP_CLIENT_ID=6798e375-a31f-48b0-abcb-87f06d70d0b6
4 | REACT_APP_REDIRECT_URI=http://localhost:3000/
5 | REACT_APP_POST_LOGOUT_REDIRECT_URI=http://localhost:3000/
6 | REACT_APP_TENANT_ID=6e8d4ac4-6168-4505-ae45-4206d841b472
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.github/workflows/azure-static-web-apps-wonderful-wave-0f526c610.yml:
--------------------------------------------------------------------------------
1 | name: Azure Static Web Apps CI/CD
2 |
3 | on:
4 | push:
5 | branches:
6 | - master
7 | pull_request:
8 | types: [opened, synchronize, reopened, closed]
9 | branches:
10 | - master
11 |
12 | jobs:
13 | build_and_deploy_job:
14 | if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
15 | runs-on: ubuntu-latest
16 | name: Build and Deploy Job
17 | steps:
18 | - uses: actions/checkout@v2
19 | with:
20 | submodules: true
21 | - name: Build And Deploy
22 | id: builddeploy
23 | uses: Azure/static-web-apps-deploy@v1
24 | with:
25 | azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_WONDERFUL_WAVE_0F526C610 }}
26 | repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
27 | action: "upload"
28 | ###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
29 | # For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
30 | app_location: "/" # App source code path
31 | api_location: "" # Api source code path - optional
32 | output_location: "" # Built app content directory - optional
33 | ###### End of Repository/Build Configurations ######
34 |
35 | close_pull_request_job:
36 | if: github.event_name == 'pull_request' && github.event.action == 'closed'
37 | runs-on: ubuntu-latest
38 | name: Close Pull Request Job
39 | steps:
40 | - name: Close Pull Request
41 | id: closepullrequest
42 | uses: Azure/static-web-apps-deploy@v1
43 | with:
44 | azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_WONDERFUL_WAVE_0F526C610 }}
45 | action: "close"
46 |
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/.gitignore:
--------------------------------------------------------------------------------
1 | node_modules
2 |
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/build/asset-manifest.json:
--------------------------------------------------------------------------------
1 | {
2 | "files": {
3 | "main.js": "/static/js/main.a03b50d2.chunk.js",
4 | "main.js.map": "/static/js/main.a03b50d2.chunk.js.map",
5 | "runtime-main.js": "/static/js/runtime-main.4452493b.js",
6 | "runtime-main.js.map": "/static/js/runtime-main.4452493b.js.map",
7 | "static/css/2.8c0e6b67.chunk.css": "/static/css/2.8c0e6b67.chunk.css",
8 | "static/js/2.cd9c6385.chunk.js": "/static/js/2.cd9c6385.chunk.js",
9 | "static/js/2.cd9c6385.chunk.js.map": "/static/js/2.cd9c6385.chunk.js.map",
10 | "index.html": "/index.html",
11 | "static/css/2.8c0e6b67.chunk.css.map": "/static/css/2.8c0e6b67.chunk.css.map",
12 | "static/js/2.cd9c6385.chunk.js.LICENSE.txt": "/static/js/2.cd9c6385.chunk.js.LICENSE.txt"
13 | },
14 | "entrypoints": [
15 | "static/js/runtime-main.4452493b.js",
16 | "static/css/2.8c0e6b67.chunk.css",
17 | "static/js/2.cd9c6385.chunk.js",
18 | "static/js/main.a03b50d2.chunk.js"
19 | ]
20 | }
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/build/favicon.ico:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/amulchapla/CallCenterIntelligenceAzureAI/f5a6952db758d1f325ea1f112624d2135a7bc8f0/azure-speech-streaming-reactjs/front-end-ui/build/favicon.ico
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/build/index.html:
--------------------------------------------------------------------------------
1 | ","\n")})),e.next=13;break;case 12:i.result.reason===G.a.NoMatch&&(r+="\n");case 13:case"end":return e.stop()}}),e)})));return function(t,n){return e.apply(this,arguments)}}(),e.next=6,t.startContinuousRecognitionAsync();case 6:return e.next=8,this.setState({isStreaming:!0});case 8:case"end":return e.stop()}}),e,this)})));return function(t,n){return e.apply(this,arguments)}}()},{key:"render",value:function(){return Object(T.jsx)(T.Fragment,{children:Object(T.jsxs)(P,{children:[Object(T.jsx)(v.a,{children:Object(T.jsx)(F,{profile:Object(T.jsx)(Y,{processAccessToken:this.processAccessToken}),debugData:this.state.debugConsole,nlpOutput:this.state.displayNLPOutput,text:this.state.displayText,dashboard:Object(T.jsx)(X,{isStreaming:this.state.isStreaming,AudioEnabled:this.state.AudioRecordingEnabled,onToggleClick:this.handleAudioRecordingSwitch,onMicRecordClick:this.handleMicRecorderClick})})}),Object(T.jsx)(v.c,{children:Object(T.jsx)(F,{profile:Object(T.jsx)(Y,{}),debugData:this.state.debugConsole,nlpOutput:this.state.displayNLPOutput,text:this.state.displayText,dashboard:Object(T.jsx)(X,{isStreaming:this.state.isStreaming,AudioEnabled:this.state.AudioRecordingEnabled,onToggleClick:this.handleAudioRecordingSwitch,onMicRecordClick:this.handleMicRecorderClick})})})]})})}}]),n}(r.Component),ee=n(212),te=n(292),ne=n(291),re=n(210),ce=n.n(re),ae=Object(ne.a)({palette:{primary:{main:"#556cd6"},secondary:{main:"#19857b"},error:{main:ce.a.A400},background:{default:"#fff"}}}),ie=n(290),se=n(20),oe=new ie.a(y);!oe.getActiveAccount()&&oe.getAllAccounts().length>0&&oe.setActiveAccount(oe.getAllAccounts()[0]),oe.enableAccountStorageEvents(),oe.addEventCallback((function(e){if(e.eventType===se.a.LOGIN_SUCCESS&&e.payload.account){var t=e.payload.account;oe.setActiveAccount(t)}})),i.a.render(Object(T.jsx)(c.a.StrictMode,{children:Object(T.jsx)(ee.a,{children:Object(T.jsx)(te.a,{theme:ae,children:Object(T.jsx)(v.b,{instance:oe,children:Object(T.jsx)($,{})})})})}),document.getElementById("root"))},85:function(e,t){}},[[275,1,2]]]); 2 | //# sourceMappingURL=main.a03b50d2.chunk.js.map -------------------------------------------------------------------------------- /azure-speech-streaming-reactjs/front-end-ui/build/static/js/main.a03b50d2.chunk.js.map: -------------------------------------------------------------------------------- 1 | {"version":3,"sources":["authConfig.js","components/SignInButton.jsx","components/SignOutButton.jsx","components/PageLayout.jsx","components/OutputWindows.jsx","token_util.js","components/Dashboard.jsx","components/Profile.jsx","App.jsx","styles/theme.js","index.js"],"names":["ua","window","navigator","userAgent","msie","indexOf","msie11","isIE","isEdge","isFirefox","msalConfig","auth","clientId","process","authority","redirectUri","postLogoutRedirectUri","cache","cacheLocation","storeAuthStateInCookie","system","loggerOptions","loggerCallback","level","message","containsPii","LogLevel","Error","console","error","Info","info","Verbose","debug","Warning","warn","loginRequest","scopes","SignInButton","instance","useMsal","DropdownButton","variant","className","drop","title","Dropdown","Item","as","onClick","loginType","loginPopup","catch","e","log","loginRedirect","SignOutButton","logoutType","logoutPopup","mainWindowRedirectUri","logoutRedirect","PageLayout","props","isAuthenticated","useIsAuthenticated","isHosted","DesktopLayout","children","HostedLayout","Container","fluid","Navbar","bg","Brand","Layout","OutputWindows","Row","Col","Card","border","style","height","Body","Header","Text","profile","dashboard","text","nlpOutput","debugData","BACKEND_API","getTokenOrRefresh","accessToken","a","cookie","Cookie","undefined","speechToken","get","headers","axios","res","token","data","region","endpoint_id","set","maxAge","path","authToken","response","idx","slice","getKeyPhrases","requestText","transcript","post","keyPhrasesExtracted","entityExtracted","Dashboard","record_status","verb","accounts","useState","setAccessToken","AudioEnabled","button_message","button_variant","isStreaming","useEffect","onMicRecordClick","Button","onToggleClick","request","account","acquireTokenSilent","then","acquireTokenPopup","RequestAccessToken","ProfileContent","name","speechsdk","require","App","handleAudioRecordingSwitch","state","AudioRecordingEnabled","setState","handleMicRecorderClick","bind","color","value","displayText","displayNLPOutput","debugConsole","this","delay","ms","Promise","setTimeout","InitializeStream","recognizer","sttFromMic","stopMicStream","tokenObj","customSpeechEndpoint","speechConfig","SpeechConfig","fromAuthorizationToken","endpointId","setServiceProperty","ServicePropertyChannel","UriQueryParameter","speechRecognitionLanguage","audioConfig","AudioConfig","fromDefaultMicrophoneInput","SpeechRecognizer","stopContinuousRecognitionAsync","resultText","nlpText","sessionStarted","s","sessionId","recognized","result","reason","ResultReason","RecognizedSpeech","nlpObj","keyPhraseText","JSON","stringify","length","entityText","replace","NoMatch","startContinuousRecognitionAsync","processAccessToken","Component","theme","createMuiTheme","palette","primary","main","secondary","red","A400","background","default","msalInstance","PublicClientApplication","getActiveAccount","getAllAccounts","setActiveAccount","enableAccountStorageEvents","addEventCallback","event","eventType","EventType","LOGIN_SUCCESS","payload","ReactDOM","render","StrictMode","ThemeProvider","document","getElementById"],"mappings":"gYAIMA,EAAKC,OAAOC,UAAUC,UACtBC,EAAOJ,EAAGK,QAAQ,SAClBC,EAASN,EAAGK,QAAQ,YAGpBE,EAAOH,EAAO,GAAKE,EAAS,EAC5BE,EAHSR,EAAGK,QAAQ,SAGF,EAClBI,EAHUT,EAAGK,QAAQ,WAGC,EAGfK,EAAa,CACtBC,KAAM,CACFC,SAAUC,uCACVC,UAAU,qCAAD,OAAuCD,wCAChDE,YAAaF,yBACbG,sBAAuBH,0BAE3BI,MAAO,CACHC,cAAe,eACfC,uBAAwBZ,GAAQC,GAAUC,GAE9CW,OAAQ,CACJC,cAAe,CACXC,eAAgB,SAACC,EAAOC,EAASC,GAC7B,IAAIA,EAGJ,OAAQF,GACJ,KAAKG,IAASC,MAEV,YADAC,QAAQC,MAAML,GAElB,KAAKE,IAASI,KAEV,YADAF,QAAQG,KAAKP,GAEjB,KAAKE,IAASM,QAEV,YADAJ,QAAQK,MAAMT,GAElB,KAAKE,IAASQ,QAEV,YADAN,QAAQO,KAAKX,GAEjB,QACI,YAQXY,EAAe,CACxBC,OAAQ,CAAC,c,0CC7CAC,EAAe,WACxB,IAAQC,EAAaC,cAAbD,SAaR,OACI,cAACE,EAAA,EAAD,CAAgBC,QAAQ,YAAYC,UAAU,UAAUC,KAAK,OAAOC,MAAM,UAA1E,SAEI,cAACC,EAAA,EAASC,KAAV,CAAeC,GAAG,SAASC,QAAS,WAdxB,IAACC,EACC,WADDA,EAcyC,YAZtDX,EAASY,WAAWf,GAAcgB,OAAM,SAAAC,GACpCzB,QAAQ0B,IAAID,MAEK,aAAdH,GACPX,EAASgB,cAAcnB,GAAcgB,OAAM,SAAAC,GACvCzB,QAAQ0B,IAAID,OAOhB,uCClBCG,EAAgB,WACzB,IAAQjB,EAAaC,cAAbD,SAcR,OACI,cAACE,EAAA,EAAD,CAAgBC,QAAQ,YAAYC,UAAU,UAAUC,KAAK,OAAOC,MAAM,WAA1E,SAEI,cAACC,EAAA,EAASC,KAAV,CAAeC,GAAG,SAASC,QAAS,WAfvB,IAACQ,EACC,WADDA,EAeyC,YAbvDlB,EAASmB,YAAY,CACjB1C,sBAAuB,IACvB2C,sBAAuB,MAEL,aAAfF,GACPlB,EAASqB,eAAe,CACpB5C,sBAAuB,OAO3B,wC,SCdC6C,EAAa,SAACC,GACvB,IAAMC,EAAkBC,cAIhBC,GAAW,EAKnB,SAASC,IACL,OAAO,mCAAGJ,EAAMK,WAGpB,SAASC,IAEL,OACI,qCACA,eAACC,EAAA,EAAD,CAAWC,OAAK,EAAhB,UACEL,EAAW,eAACM,EAAA,EAAD,CAAQC,GAAG,OAAO9B,QAAQ,OAA1B,UACT,cAAC6B,EAAA,EAAOE,MAAR,qCACEV,EAAkB,cAAC,EAAD,IAAoB,cAAC,EAAD,OAChC,sBACZ,0BAEEA,EAAkB,mCAAGD,EAAMK,WAAe,yBAKpD,SAASO,IAEL,OAAIT,EACO,cAACG,EAAD,IAEA,cAACF,EAAD,IAIf,OACI,cAACQ,EAAD,K,mCC9CKC,EAAgB,SAACb,GAC1B,OACE,eAAC,IAAD,CAAWQ,OAAK,EAAhB,UACA,eAACM,EAAA,EAAD,WACE,cAACC,EAAA,EAAD,UACE,cAACC,EAAA,EAAD,CAAMN,GAAG,YAAYO,OAAO,UAAUC,MAAO,CAAEC,OAAQ,SAAvD,SACE,eAACH,EAAA,EAAKI,KAAN,WACA,cAACJ,EAAA,EAAKK,OAAN,uBACA,cAACL,EAAA,EAAKM,KAAN,UAAYtB,EAAMuB,iBAItB,cAACR,EAAA,EAAD,UACE,cAACC,EAAA,EAAD,CAAMN,GAAG,YAAYO,OAAO,UAAUC,MAAO,CAAEC,OAAQ,SAAvD,SACE,eAACH,EAAA,EAAKI,KAAN,WACA,cAACJ,EAAA,EAAKK,OAAN,wBACA,cAACL,EAAA,EAAKM,KAAN,UAAYtB,EAAMwB,sBAKxB,uBACA,eAACV,EAAA,EAAD,WACE,cAACC,EAAA,EAAD,UACA,cAACC,EAAA,EAAD,CAAMS,KAAK,UAAUf,GAAG,OAAOO,OAAO,UAAUC,MAAO,CAAEC,OAAQ,SAAjE,SACE,eAACH,EAAA,EAAKI,KAAN,WACA,cAACJ,EAAA,EAAKK,OAAN,2CACA,cAACL,EAAA,EAAKM,KAAN,UAAYtB,EAAMyB,cAIpB,cAACV,EAAA,EAAD,UACA,cAACC,EAAA,EAAD,CAAMS,KAAK,UAAUf,GAAG,OAAOO,OAAO,UAAUC,MAAO,CAAEC,OAAQ,SAAjE,SACE,eAACH,EAAA,EAAKI,KAAN,WACA,cAACJ,EAAA,EAAKK,OAAN,iCACA,cAACL,EAAA,EAAKM,KAAN,UAAYtB,EAAM0B,sBAKtB,cAACZ,EAAA,EAAD,UACE,cAACC,EAAA,EAAD,UACA,cAACC,EAAA,EAAD,CAAMS,KAAK,UAAUf,GAAG,OAAOO,OAAO,SAASC,MAAO,CAAEC,OAAQ,SAAhE,SACE,eAACH,EAAA,EAAKI,KAAN,WACA,cAACJ,EAAA,EAAKK,OAAN,oCACA,cAACL,EAAA,EAAKM,KAAN,UAAYtB,EAAM2B,yB,2BCjDtBC,EAAc7E,gDAEb,SAAe8E,EAAtB,kC,4CAAO,WAAiCC,GAAjC,6BAAAC,EAAA,yDACGC,EAAS,IAAIC,SAICC,KAHdC,EAAcH,EAAOI,IAAI,iBAF5B,iCAQKtE,QAAQ0B,IAAI,8CACN6C,EAAU,CAAC,eAAgB,mBAAoB,cAAgB,UAAhB,OAA2BP,GAAe,8BAA+B,KAC9HhE,QAAQ0B,IAAI6C,GAVjB,SAWuBC,IAAMF,IAAIR,EAAc,wBAAyB,CAACS,YAXzE,cAWWE,EAXX,OAYWC,EAAQD,EAAIE,KAAKD,MACjBE,EAASH,EAAIE,KAAKC,OAClBC,EAAcJ,EAAIE,KAAKE,YAC7BX,EAAOY,IAAI,eAAgBF,EAAS,IAAMF,EAAO,CAACK,OAAQ,IAAKC,KAAM,MAErEhF,QAAQ0B,IAAI,gCAAkCgD,GAjBnD,kBAkBY,CAAEO,UAAWP,EAAOE,OAAQA,EAAQC,YAAaA,IAlB7D,yCAoBK7E,QAAQ0B,IAAI,KAAIwD,SAASP,MApB9B,kBAqBY,CAAEM,UAAW,KAAMhF,MAAO,KAAIiF,SAASP,OArBnD,uCAwBC3E,QAAQ0B,IAAI,8BAAgC2C,GACtCc,EAAMd,EAAY5F,QAAQ,KAzBjC,kBA0BQ,CAAEwG,UAAWZ,EAAYe,MAAMD,EAAM,GAAIP,OAAQP,EAAYe,MAAM,EAAGD,KA1B9E,2D,sBA8BA,SAAeE,EAAtB,oC,4CAAO,WAA6BC,EAAatB,GAA1C,mBAAAC,EAAA,sEAGOU,EAAO,CAACY,WAAYD,GACpBf,EAAU,CAAE,eAAgB,mBAAoB,cAAgB,UAAhB,OAA2BP,IAJlF,SAKmBQ,IAAMgB,KAAK1B,EAAc,sBAAuBa,EAAM,CAACJ,YAL1E,cAKOE,EALP,yBAOQA,EAAIE,MAPZ,yDAUQ,CAACc,oBAAqB,OAAQC,gBAAiB,UAVvD,0D,0DC7BMC,EAAY,SAACzD,GACtB,IAAI0D,EAAgB,WAChBC,EAAO,SACX,EAA+BjF,cAAvBD,EAAR,EAAQA,SAAUmF,EAAlB,EAAkBA,SAClB,EAAsCC,mBAAS,MAA/C,mBAAO/B,EAAP,KAAoBgC,EAApB,KAEI9D,EAAM+D,cACNL,EAAgB,WAChBC,EAAO,YAEPD,EAAgB,YAChBC,EAAO,UAGX,IAAIjG,EAAU,iBAAmBiG,EAAO,IACpCK,EAAiB,sBACjBC,EAAiB,UAuCrB,OAtCIjE,EAAMkE,aACNF,EAAiB,qBACjBC,EAAiB,WAEjBD,EAAiB,sBACjBC,EAAiB,WA2BrBE,qBAAW,WACHrC,GACA9B,EAAMoE,iBAAiBtC,KAE5B,CAACA,IAGA,mCACI,kCACI,qBAAIX,OAAO,MAAX,UACI,6BAAI,yDAA4BuC,OAChC,6BAAI,cAACW,EAAA,EAAD,CAAQlF,QAASa,EAAMsE,cAAvB,SAAuC5G,SAE/C,qBAAIyD,OAAO,MAAX,UACI,6BAAI,+DACJ,6BAAI,cAACkD,EAAA,EAAD,CAAQzF,QAASqF,EAAgB9E,QAvBrD,YAhBA,WACI,IAAMoF,EAAO,2BACNjG,GADM,IAETkG,QAASZ,EAAS,GAClBrF,OAAQ,CAAE,mEAGdE,EAASgG,mBAAmBF,GAASG,MAAK,SAAC1B,GACvCc,EAAed,EAASlB,gBACzBxC,OAAM,SAACC,GACNd,EAASkG,kBAAkBJ,GAASG,MAAK,SAAC1B,GACtCc,EAAed,EAASlB,oBAOhC8C,IAqBgB,SAAyDZ,eChEpEa,EAAiB,SAAC7E,GAC3B,MAA+BtB,cAAvBD,EAAR,EAAQA,SAAUmF,EAAlB,EAAkBA,SAClB,EAAsCC,mBAAS,MAA/C,mBAAO/B,EAAP,KAAoBgC,EAApB,KA0BA,OAtBmB,EAwBX,mCACI,kCACE,qBAAIjF,UAAU,aAAd,qBAAoC+E,EAAS,GAAGkB,QAC/ChD,EACC,uDAEA,cAACuC,EAAA,EAAD,CAAQzF,QAAQ,YAAYO,QAzB5C,WACI,IAAMoF,EAAO,2BACNjG,GADM,IAETkG,QAASZ,EAAS,KAGtBnF,EAASgG,mBAAmBF,GAASG,MAAK,SAAC1B,GACvCc,EAAed,EAASlB,gBACzBxC,OAAM,SAACC,GACNd,EAASkG,kBAAkBJ,GAASG,MAAK,SAAC1B,GACtCc,EAAed,EAASlB,oBAepB,uCAOR,mCACI,gCACI,oBAAIjD,UAAU,aAAd,sCClCdkG,EAAYC,EAAQ,KAILC,E,kDACnB,WAAYjF,GAAQ,IAAD,8BACf,cAAMA,IA6CVkF,2BAA6B,WAEvB,EAAKC,MAAMC,sBACb,EAAKC,SAAS,CAACD,uBAAuB,IAEtC,EAAKC,SAAS,CAACD,uBAAuB,KAhDtC,EAAKE,uBAAyB,EAAKA,uBAAuBC,KAA5B,gBAC9B,EAAKL,2BAA6B,EAAKA,2BAA2BK,KAAhC,gBAElC,EAAKJ,MAAQ,CACXrD,YAAa,KACbsD,uBAAuB,EACvBlB,aAAa,EACb/D,UAAU,EACVqF,MAAO,QACPC,MAAO,GACPC,YAAa,kDACbC,iBAAkB,+CAClBC,aAAc,sCAfD,E,iGAmBnB,WAA6B9D,GAA7B,iBAAAC,EAAA,0DAKM8D,KAAKV,MAAMjB,YALjB,uBAMI2B,KAAKR,SAAS,CAACO,aAAe,4BANlC,SAOUC,KAAKR,SAAS,CAACnB,aAAa,IAPtC,gCAQW,MARX,uBAUU2B,KAAKR,SAAS,CAACnB,aAAa,IAVtC,cAaQ4B,EAAQ,SAAAC,GAAE,OAAI,IAAIC,SAAQ,SAAAzD,GAAG,OAAI0D,WAAW1D,EAAKwD,OAbzD,UAc6BF,KAAKK,iBAAiBpE,GAdnD,eAcUqE,EAdV,iBAeUN,KAAKO,WAAWD,EAAYrE,GAftC,yBAgBUgE,EAAM,KAhBhB,eAkBMD,KAAKR,SAAS,CAACO,aAAe,gCAC9BC,KAAKR,SAAS,CAACO,aAAe,8CAnBpC,UAoBYE,EAAM,KApBlB,WAsBWD,KAAKV,MAAMjB,YAtBtB,0CAuBU2B,KAAKQ,cAAcF,GAvB7B,iD,6HAqCA,sBAAApE,EAAA,0F,2HAIF,WAAuBD,GAAvB,uBAAAC,EAAA,sEAC2BF,EAAkBC,GAD7C,cACUwE,EADV,OAEUC,EAAuBD,EAAS3D,YAChC6D,EAAezB,EAAU0B,aAAaC,uBAAuBJ,EAASvD,UAAWuD,EAAS5D,QAC5FmD,KAAKV,MAAMC,wBAGboB,EAAaG,WAAaJ,EAG1BC,EAAaI,mBAAmB,qBAAsBf,KAAKV,MAAMM,MAAOV,EAAU8B,uBAAuBC,oBAG3GN,EAAaO,0BAA4B,QACnCC,EAAcjC,EAAUkC,YAAYC,6BACpCf,EAAa,IAAIpB,EAAUoC,iBAAiBX,EAAcQ,GAfpE,kBAgBWb,GAhBX,iD,yHAoBE,WAAoBA,GAApB,SAAApE,EAAA,sEACQoE,EAAWiB,iCADnB,uBAEQvB,KAAKR,SAAS,CAACnB,aAAc,IAFrC,OAGE2B,KAAKR,SAAS,CAACO,aAAe,0BAHhC,gD,sHAOA,WAAiBO,EAAYrE,GAA7B,wBAAAC,EAAA,6DAEMsF,EAAa,GACbC,EAAU,IAEdnB,EAAWoB,eAAiB,SAACC,EAAGjI,GAC9B8H,EAAa,eAAiB9H,EAAEkI,UAChC,EAAKpC,SAAS,CACZK,YAAa2B,KAIjBlB,EAAWuB,WAAX,uCAAwB,WAAOF,EAAGjI,GAAV,mBAAAwC,EAAA,yDAEnBxC,EAAEoI,OAAOC,SAAWC,IAAaC,iBAFd,wBAIdT,GAAU,YAAS9H,EAAEoI,OAAOlG,MAC5B,EAAK4D,SAAS,CACZK,YAAa2B,IAND,SAUOlE,EAAc5D,EAAEoI,OAAOlG,KAAMK,GAVpC,OAURiG,EAVQ,QAaRC,EAAgBC,KAAKC,UAAUH,EAAOxE,sBAE3B4E,OAAS,KACtBb,GAAW,KAAOU,EAClB,EAAK3C,SAAS,CAAEM,iBAAkB2B,MAIhCc,EAAaH,KAAKC,UAAUH,EAAOvE,kBAE3B2E,OAAS,KACnBb,GAAW,KAAOc,EAClB,EAAK/C,SAAS,CAAEM,iBAAkB2B,EAAQe,QAAQ,QAAS,SAzBjD,wBA6BT9I,EAAEoI,OAAOC,SAAWC,IAAaS,UAEtCjB,GAAU,MA/BI,4CAAxB,wDAZF,SA8CQlB,EAAWoC,kCA9CnB,uBA+CQ1C,KAAKR,SAAS,CAACnB,aAAc,IA/CrC,gD,6EAkDA,WACE,OACE,mCACA,eAAC,EAAD,WACE,cAAC,IAAD,UACE,cAAC,EAAD,CAAe3C,QAAS,cAAC,EAAD,CAAgBiH,mBAAoB3C,KAAK2C,qBAAuB7G,UAAWkE,KAAKV,MAAMS,aAAclE,UAAWmE,KAAKV,MAAMQ,iBAAkBlE,KAAMoE,KAAKV,MAAMO,YAAalE,UAAW,cAAC,EAAD,CAAW0C,YAAa2B,KAAKV,MAAMjB,YAAaH,aAAc8B,KAAKV,MAAMC,sBAAuBd,cAAeuB,KAAKX,2BAA4Bd,iBAAkByB,KAAKP,6BAEtX,cAAC,IAAD,UACE,cAAC,EAAD,CAAe/D,QAAS,cAAC,EAAD,IAAmBI,UAAWkE,KAAKV,MAAMS,aAAclE,UAAWmE,KAAKV,MAAMQ,iBAAkBlE,KAAMoE,KAAKV,MAAMO,YAAalE,UAAW,cAAC,EAAD,CAAW0C,YAAa2B,KAAKV,MAAMjB,YAAaH,aAAc8B,KAAKV,MAAMC,sBAAuBd,cAAeuB,KAAKX,2BAA4Bd,iBAAkByB,KAAKP,sC,GAlJhTmD,a,mDChBpBC,GAAQC,aAAe,CAClCC,QAAS,CACPC,QAAS,CACPC,KAAM,WAERC,UAAW,CACTD,KAAM,WAER/K,MAAO,CACL+K,KAAME,KAAIC,MAEZC,WAAY,CACVC,QAAS,W,mBCDTC,GAAe,IAAIC,KAAwBzM,IAG5CwM,GAAaE,oBAAsBF,GAAaG,iBAAiBpB,OAAS,GAE7EiB,GAAaI,iBAAiBJ,GAAaG,iBAAiB,IAI9DH,GAAaK,6BAEbL,GAAaM,kBAAiB,SAACC,GAC7B,GAAIA,EAAMC,YAAcC,KAAUC,eAAiBH,EAAMI,QAAQvF,QAAS,CACxE,IAAMA,EAAUmF,EAAMI,QAAQvF,QAC9B4E,GAAaI,iBAAiBhF,OAKlCwF,IAASC,OACP,cAAC,IAAMC,WAAP,UACE,cAAC,KAAD,UACE,cAACC,GAAA,EAAD,CAAezB,MAAOA,GAAtB,SACE,cAAC,IAAD,CAAcjK,SAAU2K,GAAxB,SACA,cAAC,EAAD,YAKNgB,SAASC,eAAe,U","file":"static/js/main.a03b50d2.chunk.js","sourcesContent":["import { LogLevel } from \"@azure/msal-browser\";\r\n// Browser check variables\r\n// If you support IE, our recommendation is that you sign-in using Redirect APIs\r\n// If you as a developer are testing using Edge InPrivate mode, please add \"isEdge\" to the if check\r\nconst ua = window.navigator.userAgent;\r\nconst msie = ua.indexOf(\"MSIE \");\r\nconst msie11 = ua.indexOf(\"Trident/\");\r\nconst msedge = ua.indexOf(\"Edge/\");\r\nconst firefox = ua.indexOf(\"Firefox\");\r\nconst isIE = msie > 0 || msie11 > 0;\r\nconst isEdge = msedge > 0;\r\nconst isFirefox = firefox > 0; // Only needed if you need to support the redirect flow in Firefox incognito\r\n\r\n// Config object to be passed to Msal on creation\r\nexport const msalConfig = {\r\n auth: {\r\n clientId: process.env.REACT_APP_CLIENT_ID,\r\n authority: `https://login.microsoftonline.com/${process.env.REACT_APP_TENANT_ID}`,\r\n redirectUri: process.env.REACT_APP_REDIRECT_URI,\r\n postLogoutRedirectUri: process.env.REACT_APP_POST_LOGOUT_REDIRECT_URI\r\n },\r\n cache: {\r\n cacheLocation: \"localStorage\",\r\n storeAuthStateInCookie: isIE || isEdge || isFirefox\r\n },\r\n system: {\r\n loggerOptions: {\r\n loggerCallback: (level, message, containsPii) => {\r\n if (containsPii) {\t\r\n return;\t\r\n }\r\n switch (level) {\t\r\n case LogLevel.Error:\t\r\n console.error(message);\t\r\n return;\t\r\n case LogLevel.Info:\t\r\n console.info(message);\t\r\n return;\t\r\n case LogLevel.Verbose:\t\r\n console.debug(message);\t\r\n return;\t\r\n case LogLevel.Warning:\t\r\n console.warn(message);\t\r\n return;\t\r\n default:\r\n return;\r\n }\r\n }\r\n }\r\n }\r\n};\r\n\r\n// Add here scopes for id token to be used at MS Identity Platform endpoints.\r\nexport const loginRequest = {\r\n scopes: [\"User.Read\"]\r\n};\r\n\r\n// Add here the endpoints for MS Graph API services you would like to use.\r\nexport const graphConfig = {\r\n graphMeEndpoint: \"https://graph.microsoft.com/v1.0/me\"\r\n};","import React from \"react\";\r\nimport { useMsal } from \"@azure/msal-react\";\r\nimport { loginRequest } from \"../authConfig\";\r\nimport DropdownButton from \"react-bootstrap/DropdownButton\";\r\nimport Dropdown from \"react-bootstrap/esm/Dropdown\";\r\n\r\n/**\r\n * Renders a drop down button with child buttons for logging in with a popup or redirect\r\n */\r\nexport const SignInButton = () => {\r\n const { instance } = useMsal();\r\n\r\n const handleLogin = (loginType) => {\r\n if (loginType === \"popup\") {\r\n instance.loginPopup(loginRequest).catch(e => {\r\n console.log(e);\r\n });\r\n } else if (loginType === \"redirect\") {\r\n instance.loginRedirect(loginRequest).catch(e => {\r\n console.log(e);\r\n });\r\n }\r\n }\r\n return (\r\n
\r\n
\r\n
| Audio Recording is {record_status} | \r\n\r\n |
| Start Or Stop Streaming | \r\n\r\n |
', '\\n') });\r\n } \r\n }\r\n\r\n else if (e.result.reason === ResultReason.NoMatch) {\r\n //resultText += `\\nNo Match`\r\n resultText += `\\n`\r\n } \r\n };\r\n await recognizer.startContinuousRecognitionAsync();\r\n await this.setState({isStreaming : true});\r\n}\r\n\r\n render() {\r\n return (\r\n <>\r\n
', '\n') }); 138 | } 139 | } 140 | 141 | else if (e.result.reason === ResultReason.NoMatch) { 142 | //resultText += `\nNo Match` 143 | resultText += `\n` 144 | } 145 | }; 146 | await recognizer.startContinuousRecognitionAsync(); 147 | await this.setState({isStreaming : true}); 148 | } 149 | 150 | render() { 151 | return ( 152 | <> 153 |
| Audio Recording is {record_status} | 65 |66 | |
| Start Or Stop Streaming | 69 |70 | |
29 |
37 |
12 |
17 | );
18 | };
--------------------------------------------------------------------------------
/azure-speech-streaming-reactjs/front-end-ui/src/components/SignInButton.jsx:
--------------------------------------------------------------------------------
1 | import React from "react";
2 | import { useMsal } from "@azure/msal-react";
3 | import { loginRequest } from "../authConfig";
4 | import DropdownButton from "react-bootstrap/DropdownButton";
5 | import Dropdown from "react-bootstrap/esm/Dropdown";
6 |
7 | /**
8 | * Renders a drop down button with child buttons for logging in with a popup or redirect
9 | */
10 | export const SignInButton = () => {
11 | const { instance } = useMsal();
12 |
13 | const handleLogin = (loginType) => {
14 | if (loginType === "popup") {
15 | instance.loginPopup(loginRequest).catch(e => {
16 | console.log(e);
17 | });
18 | } else if (loginType === "redirect") {
19 | instance.loginRedirect(loginRequest).catch(e => {
20 | console.log(e);
21 | });
22 | }
23 | }
24 | return (
25 | First Name: {props.graphData.givenName}
13 | {/*Last Name: {props.graphData.surname}
14 |Email: {props.graphData.userPrincipalName}
15 |Id: {props.graphData.id}
*/} 16 |Hello World
7 | ); 8 | }; -------------------------------------------------------------------------------- /azure-speech-streaming-reactjs/front-end-ui/src/config.json: -------------------------------------------------------------------------------- 1 | { 2 | 3 | } -------------------------------------------------------------------------------- /azure-speech-streaming-reactjs/front-end-ui/src/graph.js: -------------------------------------------------------------------------------- 1 | import { graphConfig } from "./authConfig"; 2 | 3 | /** 4 | * Attaches a given access token to a MS Graph API call. Returns information about the user 5 | * @param accessToken 6 | */ 7 | export async function callMsGraph(accessToken) { 8 | const headers = new Headers(); 9 | const bearer = `Bearer ${accessToken}`; 10 | 11 | headers.append("Authorization", bearer); 12 | 13 | const options = { 14 | method: "GET", 15 | headers: headers 16 | }; 17 | 18 | return fetch(graphConfig.graphMeEndpoint, options) 19 | .then(response => response.json()) 20 | .catch(error => console.log(error)); 21 | } 22 | -------------------------------------------------------------------------------- /azure-speech-streaming-reactjs/front-end-ui/src/index.js: -------------------------------------------------------------------------------- 1 | import 'bootstrap/dist/css/bootstrap.css'; 2 | import React from 'react'; 3 | import ReactDOM from 'react-dom'; 4 | import App from "./App.jsx"; 5 | import { BrowserRouter as Router } from "react-router-dom"; 6 | import { ThemeProvider } from '@material-ui/core/styles'; 7 | import { theme } from "./styles/theme"; 8 | import { MsalProvider } from "@azure/msal-react"; 9 | 10 | 11 | // MSAL Imports 12 | import { PublicClientApplication, EventType } from "@azure/msal-browser"; 13 | import { msalConfig } from "./authConfig"; 14 | 15 | // MSAL configuration 16 | const msalInstance = new PublicClientApplication(msalConfig); 17 | 18 | // Default to using the first account if no account is active on page load 19 | if (!msalInstance.getActiveAccount() && msalInstance.getAllAccounts().length > 0) { 20 | // Account selection logic is app dependent. Adjust as needed for different use cases. 21 | msalInstance.setActiveAccount(msalInstance.getAllAccounts()[0]); 22 | } 23 | 24 | // Optional - This will update account state if a user signs in from another tab or window 25 | msalInstance.enableAccountStorageEvents(); 26 | 27 | msalInstance.addEventCallback((event) => { 28 | if (event.eventType === EventType.LOGIN_SUCCESS && event.payload.account) { 29 | const account = event.payload.account; 30 | msalInstance.setActiveAccount(account); 31 | } 32 | }); 33 | 34 | 35 | ReactDOM.render( 36 |', '\n') }); 98 | } 99 | 100 | //Display PII Detected 101 | const piiText = JSON.stringify(nlpObj.piiExtracted); 102 | if(piiText.length > 21){ 103 | nlpText += "\n" + piiText; 104 | this.setState({ displayNLPOutput: nlpText.replace('
', '\n') }); 105 | } 106 | } 107 | else if (e.result.reason === ResultReason.NoMatch) { 108 | //resultText += `\nNo Match` 109 | resultText += `\n` 110 | } 111 | 112 | }; 113 | 114 | recognizer.startContinuousRecognitionAsync(); 115 | } 116 | 117 | async fileChange(event) { 118 | const audioFile = event.target.files[0]; 119 | console.log(audioFile); 120 | const fileInfo = audioFile.name + ` size=${audioFile.size} bytes `; 121 | 122 | this.setState({ 123 | displayText: fileInfo 124 | }); 125 | 126 | const tokenObj = await getTokenOrRefresh(); 127 | const speechConfig = speechsdk.SpeechConfig.fromAuthorizationToken(tokenObj.authToken, tokenObj.region); 128 | speechConfig.speechRecognitionLanguage = 'en-US'; 129 | 130 | const audioConfig = speechsdk.AudioConfig.fromWavFileInput(audioFile); 131 | const recognizer = new speechsdk.SpeechRecognizer(speechConfig, audioConfig); 132 | 133 | recognizer.recognizeOnceAsync(result => { 134 | let displayText; 135 | if (result.reason === ResultReason.RecognizedSpeech) { 136 | displayText = `RECOGNIZED: Text=${result.text}` 137 | } else { 138 | displayText = 'ERROR: Speech was cancelled or could not be recognized. Ensure your microphone is working properly.'; 139 | } 140 | 141 | this.setState({ 142 | displayText: fileInfo + displayText 143 | }); 144 | }); 145 | } 146 | 147 | render() { 148 | return ( 149 |
Realtime Call Intelligence - powered by Azure AI
152 | NOTE: This covnersation will be recorded for demo purpose.
153 | -----------------------------------------------------------
154 |
155 |
162 |
163 |
164 |
165 | this.sttFromMic()}>
166 | STEP 2 - Click on Microphone and start talking for real-time insights..
167 |
168 |
169 | ----- Speech-to-text Output ---------------------------------------------------- AI-powered Call Insights ------
170 |
171 |
172 |
179 |
180 |
173 |
175 | {this.state.displayText}
174 |
176 |
178 | {this.state.displayNLPOutput}
177 |