├── README.md
├── hello_world
└── hello_world.ipynb
├── mining_usecase
├── discovery_mining.ipynb
└── wikipedia_mining.ipynb
├── pro_con_customization
├── img
│ ├── after_choose_model.png
│ ├── after_create_ds.png
│ ├── after_uploading_file.png
│ ├── before_choose_model.png
│ ├── before_start_train.png
│ ├── before_uploading_file.png
│ ├── choose_file.png
│ ├── docker_out.png
│ ├── new_tab.png
│ ├── run_docker.png
│ ├── train_out.png
│ ├── train_out_2.png
│ └── ui.png
├── pro_con_customization.ipynb
├── pro_con_customization_with_answers.ipynb
├── pro_con_customization_with_answers_and_results.ipynb
├── wealth_test_set.csv
├── wealth_train_set.csv
└── wealth_util.py
├── requirements.txt
├── run_kpa_scripts
├── run_keypoint_analysis_script.py
└── run_keypoint_analysis_script_two_stances.py
├── survey_usecase
├── kpa_quick_start_tutorial-with_results.ipynb
├── legacy_sdk
│ ├── dataset_austin.csv
│ ├── dataset_austin_sentences.csv
│ ├── kpa_parameters.pdf
│ ├── kpa_quick_start_tutorial-with_results.ipynb
│ ├── kpa_quick_start_tutorial.ipynb
│ ├── run_warmup_batches.sh
│ └── warm_up_cache_eap_tutorial.sh
└── new_sdk
│ ├── Changelog.md
│ ├── README.md
│ ├── dataset_austin.csv
│ ├── kpa_quick_start_tutorial-with_results.ipynb
│ ├── kpa_quick_start_tutorial.ipynb
│ ├── kps_parameters.pdf
│ ├── kps_results
│ ├── austin_demo_sentences.csv
│ ├── kps_result_2016.csv
│ ├── kps_result_2016.json
│ ├── kps_result_2016_hierarchical.docx
│ ├── kps_result_2016_kps_summary.csv
│ └── kps_result_2016_unmatched_sents.csv
│ └── run_keypoint_summarization_script.py
└── tabs.png
/README.md:
--------------------------------------------------------------------------------
1 | # Project Debater Early Access Program tutorials
2 | ## Introduction
3 | Project Debater is the first AI to successfully engage with a human in a live debate. In February 2019, Project Debater debated Mr. Harish Natarajan, one of the world's leading professional debaters in an event held in San Francisco and broadcast live worldwide. The underlying technologies that enabled the live event are now available to you as software services, denoted as Project Debater services, that include core natural language understanding capabilities, argument mining, text summarization and narrative generation. More information on the various services, code examples and online demos can be found at https://early-access-program.debater.res.ibm.com.
4 |
5 | When you have a large collection of texts representing people’s opinions (such as product reviews, survey answers or posts from social media), it is difficult to understand the key issues that come up in the data. Going over thousands of comments is prohibitively expensive. Existing automated approaches are often limited to identifying recurring phrases or concepts and the overall sentiment toward them, but do not provide detailed or actionable insights.
6 |
7 | In this tutorial you will gain hands-on experience in using Project Debater services for analyzing and deriving insights from open-ended answers. The data we will use is a community survey conducted in the city of Austin in the years 2016 and 2017. We will analyze their open-ended answers in different ways by using four Debater services, the Argument Quality service, the Key Point Analysis service, the Term Wikifier service and the Term Relater service, and we will see how they can be combined into a powerful text analysis tool.
8 |
9 | ## Prerequisites
10 | To follow this tutorial, you first need to receive credentials for the Debater APIs early-access-program site (https://early-access-program.debater.res.ibm.com) by sending a request email to: `project.debater@il.ibm.com`
11 |
12 | ## Available tutorials
13 | * The tutorial `survey_usecase` demonstrates how to use Project Debater services for analyzing and finding insights in free text survey data.
14 | * The tutorial `pro_con_customization` demonstrates how to customize the Project Debater services to specific domains and tasks by fine-tuning the model with user provided examples. In addition to an APIKEY, this tutorial requires a Docker image of the pro-con service of the Debater Early Access Program. Please contact `eladv@il.ibm.com` for more information.
15 |
16 | ## Estimated time
17 | It should take about 1 hour to complete the `survey_usecase` tutorial.
18 |
19 | ## Environment setup
20 | In order to run the tutorials, you need an Python Anaconda environment with the Early Access Program SDK installed on it and a Jupyter notebook.
21 |
22 | * Create a conda env:
23 | `conda create --name python=3.7`
24 |
25 | * Activate env:
26 | `conda activate `
27 |
28 | * Get credentials for the Early Access Program site by sending an email to:
29 | `project.debater@il.ibm.com`
30 |
31 | * Login to the Project Debater Early Access Program web site:
32 | https://early-access-program.debater.res.ibm.com/
33 |
34 | * Download and install the Python SDK according to the instructions in:
35 | https://early-access-program.debater.res.ibm.com/download_sdk.html
36 |
37 | * To check that the environment is set correctly, try running a code example, e.g.:
38 | https://early-access-program.debater.res.ibm.com/#keypoints
39 |
40 | * Get the API-KEY from the site:
41 | 
42 |
43 |
44 | ## Run the tutorial
45 |
46 | * Clone the tutorial repository:
47 | `git clone https://github.com/IBM/debater-eap-tutorial.git`
48 |
49 | * Install the required packages:
50 | `cd debater-eap-tutorial`
51 | `pip install -r requirements.txt`
52 |
53 | * Run Jupyter Notebook (use the api-key from the site):
54 | `env DEBATER_API_KEY=APIKEY_PLACE_HOLDER jupyter notebook`
55 |
56 | * To check that the environment is set correctly, try running notebook `hello_world/hello_world.ipynb`.
57 |
58 | * Open `survey_usecase/new_sdk/kpa_quick_start_tutorial.ipynb` notebook for a simple quick-start tutorial or `survey_usecase/new_sdk/kpa_quick_start_tutorial-with_results.ipynb` for a version with results.
59 |
60 | * The tutorial is self-explanatory, and demonstrates how to use Key Point Analysis and its key features.
61 |
62 | * Feel free to contact us if you face problems or have questions at:
`project.debater@il.ibm.com`
--------------------------------------------------------------------------------
/hello_world/hello_world.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Hello World"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "The purpose of this notebook is to check that your environment is set up correctly and that you are able to access the Debater Early Access APIs. It also demonstrates how to call the Pro/Con Service."
15 | ]
16 | },
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {},
20 | "source": [
21 | "Set up imports"
22 | ]
23 | },
24 | {
25 | "cell_type": "code",
26 | "execution_count": null,
27 | "metadata": {},
28 | "outputs": [],
29 | "source": [
30 | "from debater_python_api.api.debater_api import DebaterApi\n",
31 | "import os"
32 | ]
33 | },
34 | {
35 | "cell_type": "markdown",
36 | "metadata": {},
37 | "source": [
38 | "Check that DEBATER_API_KEY environment variable is set up correctly"
39 | ]
40 | },
41 | {
42 | "cell_type": "code",
43 | "execution_count": null,
44 | "metadata": {},
45 | "outputs": [],
46 | "source": [
47 | "if not 'DEBATER_API_KEY' in os.environ:\n",
48 | " raise ValueError(\"Please set the DEBATER_API_KEY environment variable to the API KEY found in the API-KEY \"\n",
49 | " \"tab of the Early Access Program Website\")\n",
50 | "api_key = os.environ['DEBATER_API_KEY']\n",
51 | "if api_key == \"\":\n",
52 | " raise ValueError(\"Please set the content of the DEBATER_API_KEY environment variable to the API KEY found in the API-KEY \"\n",
53 | " \"tab of the Early Access Program Website. It currently uses a placeholder template\")"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "metadata": {},
59 | "source": [
60 | "Initialize a DebaterApi object with the api key, and return a pro/con client object."
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": null,
66 | "metadata": {},
67 | "outputs": [],
68 | "source": [
69 | "debater_api = DebaterApi(api_key)\n",
70 | "pro_con_client = debater_api.get_pro_con_client()"
71 | ]
72 | },
73 | {
74 | "cell_type": "markdown",
75 | "metadata": {},
76 | "source": [
77 | "Initialize the data to be sent to the pro/con service. In this case, it is list containing a single pair of a \n",
78 | "sentence and a topic:"
79 | ]
80 | },
81 | {
82 | "cell_type": "code",
83 | "execution_count": null,
84 | "metadata": {},
85 | "outputs": [],
86 | "source": [
87 | "topic = 'This tutorial is helpful.'\n",
88 | "sentence = 'This tutorial works as expected.' \n",
89 | "\n",
90 | "sentence_topic_dicts = [{'sentence' : sentence, 'topic' : topic } ]"
91 | ]
92 | },
93 | {
94 | "cell_type": "markdown",
95 | "metadata": {},
96 | "source": [
97 | "Call the pro/con client. The client performs a REST call to the service and receives the pro/con scores as a list (one score per pair of topic/sentence submitted)"
98 | ]
99 | },
100 | {
101 | "cell_type": "code",
102 | "execution_count": null,
103 | "metadata": {},
104 | "outputs": [],
105 | "source": [
106 | "scores = pro_con_client.run(sentence_topic_dicts)"
107 | ]
108 | },
109 | {
110 | "cell_type": "markdown",
111 | "metadata": {},
112 | "source": [
113 | "Print the score and validate it is in the right range."
114 | ]
115 | },
116 | {
117 | "cell_type": "code",
118 | "execution_count": null,
119 | "metadata": {},
120 | "outputs": [],
121 | "source": [
122 | "print(\"Score: \"+\"{:.4f}\".format(scores[0]))\n",
123 | "\n",
124 | "if (scores[0] > 0.99):\n",
125 | " print(\"Hello world! It works.\")"
126 | ]
127 | }
128 | ],
129 | "metadata": {
130 | "kernelspec": {
131 | "display_name": "eap",
132 | "language": "python",
133 | "name": "eap"
134 | },
135 | "language_info": {
136 | "codemirror_mode": {
137 | "name": "ipython",
138 | "version": 3
139 | },
140 | "file_extension": ".py",
141 | "mimetype": "text/x-python",
142 | "name": "python",
143 | "nbconvert_exporter": "python",
144 | "pygments_lexer": "ipython3",
145 | "version": "3.8.8"
146 | }
147 | },
148 | "nbformat": 4,
149 | "nbformat_minor": 4
150 | }
151 |
--------------------------------------------------------------------------------
/pro_con_customization/img/after_choose_model.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/after_choose_model.png
--------------------------------------------------------------------------------
/pro_con_customization/img/after_create_ds.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/after_create_ds.png
--------------------------------------------------------------------------------
/pro_con_customization/img/after_uploading_file.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/after_uploading_file.png
--------------------------------------------------------------------------------
/pro_con_customization/img/before_choose_model.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/before_choose_model.png
--------------------------------------------------------------------------------
/pro_con_customization/img/before_start_train.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/before_start_train.png
--------------------------------------------------------------------------------
/pro_con_customization/img/before_uploading_file.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/before_uploading_file.png
--------------------------------------------------------------------------------
/pro_con_customization/img/choose_file.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/choose_file.png
--------------------------------------------------------------------------------
/pro_con_customization/img/docker_out.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/docker_out.png
--------------------------------------------------------------------------------
/pro_con_customization/img/new_tab.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/new_tab.png
--------------------------------------------------------------------------------
/pro_con_customization/img/run_docker.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/run_docker.png
--------------------------------------------------------------------------------
/pro_con_customization/img/train_out.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/train_out.png
--------------------------------------------------------------------------------
/pro_con_customization/img/train_out_2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/train_out_2.png
--------------------------------------------------------------------------------
/pro_con_customization/img/ui.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/pro_con_customization/img/ui.png
--------------------------------------------------------------------------------
/pro_con_customization/pro_con_customization.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# About this tutorial\n",
8 | "In this tutorial, we will focus on the pro-con and the narrative generation services. \n",
9 | "\n",
10 | "We will use a set of arguments collected from the crowd for the \"That's Debatable\" show. The crowd was asked to contribute arguments for or against the topic **\"It is time to redistribute the wealth\"**.\n",
11 | "\n",
12 | "We will start by running the pro-con service, in order to predict whether an argument is for or against the topic. Next, we will see how we can improve the classification results. First, by selecting a classification *threshold* , and then by fine-tuning to model to the specific topic. \n",
13 | "\n",
14 | "Finally, we will use these predictions in order to generate a narrative for the topic. "
15 | ]
16 | },
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {},
20 | "source": [
21 | "# 1. Running pro-con service "
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {},
27 | "source": [
28 | "The input to the pro-con service is a list of dictionaries, each containing two keys: 'sentence' and 'topic', as follows - \n",
29 | "\n",
30 | "`arg_topic_dicts = [{'sentence': 'sentence_a', 'topic': 'topic_a'}, {'sentence': 'sentence_b', 'topic': 'topic_b'}]`\n",
31 | "\n",
32 | "We will begin by collecting the data for these dictionaries, as well as the ground truth labels."
33 | ]
34 | },
35 | {
36 | "cell_type": "code",
37 | "execution_count": null,
38 | "metadata": {},
39 | "outputs": [],
40 | "source": [
41 | "import os\n",
42 | "from debater_python_api.api.debater_api import DebaterApi\n",
43 | "from debater_python_api.api.clients.narrative_generation_client import Polarity\n",
44 | "import csv\n",
45 | "\n",
46 | "arguments_file = 'wealth_test_set.csv'\n",
47 | "topic = 'It is time to redistribute the wealth'\n",
48 | "\n",
49 | "with open(arguments_file, encoding='utf8') as csv_file:\n",
50 | " reader = csv.DictReader(csv_file)\n",
51 | " examples = list(reader)\n",
52 | " arguments = [example['sentence'] for example in examples]\n",
53 | " labels = [example['label'] for example in examples]"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "metadata": {},
59 | "source": [
60 | "Next, we will create the debater_api object:"
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": null,
66 | "metadata": {},
67 | "outputs": [],
68 | "source": [
69 | "api_key = os.environ['DEBATER_API_KEY']\n",
70 | "debater_api = DebaterApi(api_key)"
71 | ]
72 | },
73 | {
74 | "cell_type": "markdown",
75 | "metadata": {},
76 | "source": [
77 | "Now we are ready to run the pro con service. \n",
78 | "\n",
79 | "The service returns a list of floats in the range [-1, 1], where each float is the prediction (score) for the corresponding sentence and topic. The sign of the score indicates if the service predicts that the argument supports the topic (positive score), or contests the topic (negative score). The absolute value of the score indicates the confidence of the prediction. Low absolute values indicates low confidence, while high absolute values indicates high confidence.\n",
80 | "\n",
81 | "Documentation for the pro-con service can be found here: https://early-access-program.debater.res.ibm.com/#pro_con\n"
82 | ]
83 | },
84 | {
85 | "cell_type": "markdown",
86 | "metadata": {},
87 | "source": [
88 | "## Your turn!\n",
89 | "\n",
90 | "Please write the code for the following:\n",
91 | "\n",
92 | "1. Create a *pro_con_client* by calling the method `get_pro_con_client()` of the `debater_api` object and storing the resulting object in the `pro_con_client` variable.\n",
93 | "2. Create a list in the format required for this client - list of dictionaries, where each dictionary contains two keys - `topic` and `sentence`. Use `topic` and `arguments` variables previously defined. Name the list 'sentence_topics_dicts'.\n",
94 | "3. Run the pro-con client on this list, using the `run(sentence_topic_dicts)` method of *pro_con_client*, while storing the result in `pro_con_scores`."
95 | ]
96 | },
97 | {
98 | "cell_type": "code",
99 | "execution_count": null,
100 | "metadata": {
101 | "scrolled": false
102 | },
103 | "outputs": [],
104 | "source": [
105 | "#PLEASE WRITE YOUR CODE HERE\n"
106 | ]
107 | },
108 | {
109 | "cell_type": "markdown",
110 | "metadata": {},
111 | "source": [
112 | "# 2. Using pro con results\n",
113 | "Now, we measure the accuracy of the results. We will compare the service prediction to the labels created by human annotators. The label field contains three possible values: '1' for supporting arguments, '-1' for contesting arguments, and '0' for neutral or mixed arguments.\n",
114 | "\n",
115 | "We will measure how many arguments were classified as supporting arguments and how many of these were actually labeled as supporting arguments. We can then print the accuracy of the pro predictions. \n",
116 | "\n",
117 | "We will repeat these steps for the contesting arguments as well. "
118 | ]
119 | },
120 | {
121 | "cell_type": "code",
122 | "execution_count": null,
123 | "metadata": {
124 | "scrolled": true
125 | },
126 | "outputs": [],
127 | "source": [
128 | "argument_score_label_list = list(zip(arguments, pro_con_scores, labels))\n",
129 | "\n",
130 | "predicted_pro = [triple for triple in argument_score_label_list if triple[1] > 0]\n",
131 | "predicted_pro_labeled_pro = [triple for triple in predicted_pro if triple[2] == '1']\n",
132 | "print('predicted pro accuracy {}'.format(len(predicted_pro_labeled_pro) / len(predicted_pro)))\n",
133 | "\n",
134 | "predicted_con = [triple for triple in argument_score_label_list if triple[1] < 0]\n",
135 | "predicted_con_labeled_con = [triple for triple in predicted_con if triple[2] == '-1']\n",
136 | "print('predicted con accuracy {}'.format(len(predicted_con_labeled_con) / len(predicted_con)))\n"
137 | ]
138 | },
139 | {
140 | "cell_type": "markdown",
141 | "metadata": {},
142 | "source": [
143 | "# 3. Looking into polarity errors:\n",
144 | "We saw that the accuracy of our classifier is about 70%. This is very nice result, but let's look at the arguments that we missed and try to understand if we can improve the results."
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "execution_count": null,
150 | "metadata": {
151 | "scrolled": true
152 | },
153 | "outputs": [],
154 | "source": [
155 | "def sign(x):\n",
156 | " if x > 0:\n",
157 | " return '1'\n",
158 | " return '-1'\n",
159 | "\n",
160 | "\n",
161 | "arguments_with_wrong_polarity = [triple for triple in argument_score_label_list if sign(triple[1]) != triple[2]]\n",
162 | "print('Arguments with wrong polarity:')\n",
163 | "for triple in arguments_with_wrong_polarity[:10]:\n",
164 | " print('score: {}, label: {}, argument: {}'.format(triple[1], triple[2], triple[0]))"
165 | ]
166 | },
167 | {
168 | "cell_type": "markdown",
169 | "metadata": {},
170 | "source": [
171 | "We can see that some of the arguments which were predicted incorrectly were labeled as '0'. It means that the human annotators didn't think that these arguments have a clear stance. In addition, we can see that although there are a few arguments with relatively high confidence (high absolute value) in this list, most of the arguments have a relatively low prediction confidence. \n",
172 | "\n",
173 | "A possible improvement will be to take into account not only the sign of the score but also its confidence. We can decide to classify to either pro or con only if the prediction confidence is higher than some threshold and classify remaining arguments as undetermined or neutral.\n",
174 | "\n",
175 | "How should choose such a threshold? Choosing a high threshold improves the precision of the classification, but reduces the coverage - it means that we will take into account only a limited portion of the arguments. "
176 | ]
177 | },
178 | {
179 | "cell_type": "markdown",
180 | "metadata": {},
181 | "source": [
182 | "# 4. Generating coverage-precision curve\n",
183 | "\n",
184 | "In order to choose a threshold that provides the optimal trade off between precision and coverage, we will generate the coverage-precision curve for our pro-con results. For each given threshold, we can calculate the corresponding precision and coverage, and add the point (coverage, precision) to a graph on the coverage-precision plane. We will get a curve, parameterized by the threshold, which enables us to choose the correct threshold for our use case. "
185 | ]
186 | },
187 | {
188 | "cell_type": "code",
189 | "execution_count": null,
190 | "metadata": {
191 | "scrolled": true
192 | },
193 | "outputs": [],
194 | "source": [
195 | "from wealth_util import calculate_statistics, plot_graph\n",
196 | "\n",
197 | "pro_con_statistics = calculate_statistics('./wealth_test_set.csv', pro_con_scores)\n",
198 | "plot_graph(examples_list=[pro_con_statistics], labels=['prod'])"
199 | ]
200 | },
201 | {
202 | "cell_type": "markdown",
203 | "metadata": {},
204 | "source": [
205 | "We can see that at 50% coverage, precision is about 93% for the pro arguments and almost 95% for the con arguments. For a coverage of 80%, precision will drop to about 82% in both for pro and con arguments. "
206 | ]
207 | },
208 | {
209 | "cell_type": "markdown",
210 | "metadata": {},
211 | "source": [
212 | "# 5. Fine-tuning the pro-con service\n",
213 | "In order to further improve the results, we can fine-tune the model we use on our data. The generic model which we used has been trained on over 400,000 examples, collected from both newspapers and magazines corpus and from the crowd. The model is a transformer based model (https://en.wikipedia.org/wiki/Transformer_(machine_learning_model). It can be fine-tuned on sample data from the desired domain in order to improve the service results. Experiments show that fine-tuning on even a relatively small sample from the desired domain can significantly improve results in the said domain.\n",
214 | "\n",
215 | "The IBM cloud instance of Project Debater Early Access Program offers GPU based services. It is also possible to get a docker image of a given service and run it locally. Both CPU/GPU images can be provided.\n",
216 | "\n",
217 | "Since the cloud instances are shared between all of our users, we do not allow to fine-tune them. But as it is very simple to run our services locally, a user can get the image, fine-tune it quickly, and use it for inference.\n",
218 | "\n",
219 | "In this tutorial we will show an example of running the pro-con service from a docker image on the virtual machine. Since the virtual machine doesn't have a GPU, we will use a CPU based image. It is important to mention that a GPU instance will run much faster than a CPU instance (we use Nvidia K80 and run about 3000 examples per minute, while running the same service on a mac with i9 core, 2.3GHz with 64 GB ram yields about 600 examples per minute).\n",
220 | "\n",
221 | "In general, in order to fine-tune the pro-con service, we need a CSV file with three columns: motion (topic), sentence (argument), and label. Each line should be one example, with the topic and argument, and the correct label. The label enumeration should be '1' for pro arguments, '0' for con arguments, and '2' for neutral arguments. Note that these labels are different from the labels the service returns at inference - a real number in the range [-1,1] where its sign represents its class (positive for pro arguments and negative for con arguments).\n",
222 | "\n",
223 | "Sometimes even a few dozens of examples can improve the basic model, and the more the better. A training parameter that can be configured is the number of training epochs. This parameter defines how many rounds we will apply to our training set. On the one hand, training many times will fit the model better to the data, but on the other hand, it can cause overfit, which means that it will match very well the training data set at the cost of lower results on the actual data. A simple rule of thumb is that we should try to increase the number of epochs when the training set is larger. At this tutorial, we use less than 500 examples, and therefore we will set the number of epochs to one. \n",
224 | "\n",
225 | "It is important not to train the model on the same arguments set that you use for evaluating the model since it may overfit to the training set.\n",
226 | "\n",
227 | "In our tutorial, we will use a different set of arguments collected from the same event: [wealth_train_set.csv](./wealth_train_set.csv). \n",
228 | "\n",
229 | "## Your turn! Lets start to run a local pro-con service and recreate it.\n",
230 | "\n",
231 | "1. Run local pro-con service:\n",
232 | "- Open a new terminal tab (File -> New Tab).\n",
233 | "\n",
234 | "- Run a docker command for runn the pro-con docker image and assign port 8000 to it:\n",
235 | "`docker run -p 8000:8000 us.icr.io/ris2-debater/pro-con:4.4.b37`\n",
236 | "\n",
237 | "- wait until this text appears: \"run: server started\"\n",
238 | "\n",
239 | "\n",
240 | "2. Fine-tune it\n",
241 | "- open the following url in chrome: http://localhost:8000/model.html\n",
242 | "\n",
243 | "- under the title \"Training data sets\", click on \"create\" button (you don't have to give it a name).\n",
244 | "\n",
245 | "- click on \"Browse\" button, and select wealth_train_set.csv.\n",
246 | "\n",
247 | "- click Upload\n",
248 | "\n",
249 | "- Give name to the new model near \"Train new model\" set \"epochs\" to 1, and click \"start training\".\n",
250 | "\n",
251 | "\n",
252 | "- please note that there is a UI issue that causes the log output to wrap itself \n",
253 | "\n",
254 | "- wait until the new model name appears under “Available models”.\n",
255 | "- Select the radio button that correspond to the new model.\n",
256 | "\n",
257 | "\n",
258 | "\n",
259 | "Now you have a local service, fine tune with example from the same topic. \n"
260 | ]
261 | },
262 | {
263 | "cell_type": "markdown",
264 | "metadata": {},
265 | "source": [
266 | "# 6. Coverage-precision curve for the fine-tuned model.\n",
267 | "\n",
268 | "In order to compare between the generic model and the fine-tuned model, you will generate again the precision coverage curve for the fine-tuned model, and then plot the two curves on the same graph. \n",
269 | "\n",
270 | "Start by setting the client host to http://localhost:8000 and then generate the pro-con scores the same way you did for the generic model. Store the scores at the variable `ft_pro_con_scores`"
271 | ]
272 | },
273 | {
274 | "cell_type": "code",
275 | "execution_count": null,
276 | "metadata": {},
277 | "outputs": [],
278 | "source": [
279 | "pro_con_client.set_host('http://localhost:8000')\n",
280 | "ft_pro_con_scores = pro_con_client.run(sentence_topic_dicts)"
281 | ]
282 | },
283 | {
284 | "cell_type": "markdown",
285 | "metadata": {},
286 | "source": [
287 | "Now we can calculate the statistics for the fine-tuned model, and plot the old and new statistics onthe same graph."
288 | ]
289 | },
290 | {
291 | "cell_type": "code",
292 | "execution_count": null,
293 | "metadata": {},
294 | "outputs": [],
295 | "source": [
296 | "ft_pro_con_statistics = calculate_statistics('./wealth_test_set.csv', ft_pro_con_scores)\n",
297 | "plot_graph(examples_list=[pro_con_statistics, ft_pro_con_statistics], labels=['prod', 'ft'])"
298 | ]
299 | },
300 | {
301 | "cell_type": "markdown",
302 | "metadata": {},
303 | "source": [
304 | "We can see that the fine-tuning improved significantly our pro precision, and it also improved the con predictions for most of the coverage range. We can see that for the con side, for coverage = 0.8, we get a precision of about 0.93."
305 | ]
306 | },
307 | {
308 | "cell_type": "markdown",
309 | "metadata": {},
310 | "source": [
311 | "# 7. Generating a narrative using the fine-tuned model \n",
312 | "\n",
313 | "Now, we ready to use the model predictions we have, with the threshold we chose, to create a narrative. We will use the narrative generation client and create an opoosing speech, from all the arguments identified as con above the 0.8 threshold."
314 | ]
315 | },
316 | {
317 | "cell_type": "code",
318 | "execution_count": null,
319 | "metadata": {},
320 | "outputs": [],
321 | "source": [
322 | "dominant_concept = 'Redistribution of income and wealth'\n",
323 | "\n",
324 | "customizations = \\\n",
325 | " [\n",
326 | " {\n",
327 | " \"title\": \"Number of paragraphs\",\n",
328 | " \"description\": \"Set an upper bound on number of paragraphs (each yielding a theme)\",\n",
329 | " \"type\": \"numParagraphsToSelect\",\n",
330 | " \"items\":\n",
331 | " [\n",
332 | " {\n",
333 | " \"key\": \"max_num_of_paragraphs\",\n",
334 | " \"description\": \"recommended values: from 4 to 8\",\n",
335 | " \"value\": 3,\n",
336 | " \"itemType\": \"single_integer\"\n",
337 | " }\n",
338 | " ]\n",
339 | " },\n",
340 | " {\n",
341 | " \"title\": \"Speech length in minutes\",\n",
342 | " \"description\": \"Set an upper bound on number of minutes the speech should last.\",\n",
343 | " \"type\": \"speechLengthInMinutes\",\n",
344 | " \"items\":\n",
345 | " [\n",
346 | " {\n",
347 | " \"key\": \"max_length_in_minutes\",\n",
348 | " \"description\": \"A float number expressed as decimal with up to two digits to the right of the decimal point. Recommended values: from 2.0 to 4.0\",\n",
349 | " \"value\": 3,\n",
350 | " \"itemType\": \"single_float\",\n",
351 | " }\n",
352 | " ]\n",
353 | " },\n",
354 | " {\n",
355 | " \"title\": \"System Polarity Threshold\",\n",
356 | " \"description\": \"Sets the threshold for filtering of arguments by their polarity score. Can specify absolute value, and / or fraction of the top scoring arguments. When both thresholds are specified, both are employed.\",\n",
357 | " \"type\": \"systemPolarityThreshold\",\n",
358 | " \"items\":\n",
359 | " [\n",
360 | " {\n",
361 | " \"key\": \"system_polarity_absolute_threshold\",\n",
362 | " \"description\": \"Threshold, in the range [0,1]. Arguments whose procon score falls below that threshold will be filtered out.\",\n",
363 | " \"value\": 0.0,\n",
364 | " \"itemType\": \"single_float\",\n",
365 | " },\n",
366 | " {\n",
367 | " \"key\": \"system_polarity_percentage_threshold\",\n",
368 | " \"description\": \"Threshold, in the range [0,1]. Arguments whose rank, in the arguments sorted by their procon score, is not among the top threshold of the list, will be filtered out.\",\n",
369 | " \"value\": 0.8,\n",
370 | " \"itemType\": \"single_float\",\n",
371 | " }\n",
372 | "\n",
373 | " ]\n",
374 | " }\n",
375 | " ]\n",
376 | "\n",
377 | "\n",
378 | "speech = debater_api.get_narrative_generation_client().run(topic=topic,\n",
379 | " dc=dominant_concept,\n",
380 | " sentences=arguments,\n",
381 | " pro_con_scores=pro_con_scores,\n",
382 | " polarity=Polarity.CON,\n",
383 | " customizations=customizations)\n",
384 | "\n",
385 | "print('\\n\\n' + str(speech))"
386 | ]
387 | }
388 | ],
389 | "metadata": {
390 | "kernelspec": {
391 | "display_name": "Python 3",
392 | "language": "python",
393 | "name": "python3"
394 | },
395 | "language_info": {
396 | "codemirror_mode": {
397 | "name": "ipython",
398 | "version": 3
399 | },
400 | "file_extension": ".py",
401 | "mimetype": "text/x-python",
402 | "name": "python",
403 | "nbconvert_exporter": "python",
404 | "pygments_lexer": "ipython3",
405 | "version": "3.7.10"
406 | }
407 | },
408 | "nbformat": 4,
409 | "nbformat_minor": 4
410 | }
411 |
--------------------------------------------------------------------------------
/pro_con_customization/pro_con_customization_with_answers.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# About this tutorial\n",
8 | "In this tutorial, We will focus on the pro-con and the narrative generation services. \n",
9 | "\n",
10 | "We will use a set of arguments collected from the crowd for the \"That's Debatable\" show. The crowd was asked to contribute arguments for or against the topic **\"It is time to redistribute the wealth\"**.\n",
11 | "\n",
12 | "We will start by running the pro-con service, in order to predict whether an argument is for or against the topic. Next, we will see how we can improve the classification results. First, by selecting a classification *threshold* , and then by fine-tuning to model to the specific topic. \n",
13 | "\n",
14 | "Finally, we will use these predictions in order to generate a narrative for the topic. "
15 | ]
16 | },
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {},
20 | "source": [
21 | "# 1. Running pro-con service "
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {},
27 | "source": [
28 | "The input to the pro-con service is a list of dictionaries, each containing two keys: 'sentence' and 'topic', as follows - \n",
29 | "\n",
30 | "`arg_topic_dicts = [{'sentence': 'sentence_a', 'topic': 'topic_a'}, {'sentence': 'sentence_b', 'topic': 'topic_b'}]`\n",
31 | "\n",
32 | "We will begin by collecting the data for these dictionaries, as well as the ground truth labels."
33 | ]
34 | },
35 | {
36 | "cell_type": "code",
37 | "execution_count": null,
38 | "metadata": {},
39 | "outputs": [],
40 | "source": [
41 | "import os\n",
42 | "from debater_python_api.api.debater_api import DebaterApi\n",
43 | "from debater_python_api.api.clients.narrative_generation_client import Polarity\n",
44 | "import csv\n",
45 | "\n",
46 | "arguments_file = 'wealth_test_set.csv'\n",
47 | "topic = 'It is time to redistribute the wealth'\n",
48 | "\n",
49 | "with open(arguments_file, encoding='utf8') as csv_file:\n",
50 | " reader = csv.DictReader(csv_file)\n",
51 | " examples = list(reader)\n",
52 | " arguments = [example['sentence'] for example in examples]\n",
53 | " labels = [example['label'] for example in examples]"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "metadata": {},
59 | "source": [
60 | "Next, we will create the debater_api object:"
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "execution_count": null,
66 | "metadata": {},
67 | "outputs": [],
68 | "source": [
69 | "api_key = os.environ['DEBATER_API_KEY']\n",
70 | "debater_api = DebaterApi(api_key)"
71 | ]
72 | },
73 | {
74 | "cell_type": "markdown",
75 | "metadata": {},
76 | "source": [
77 | "Now we are ready to run the pro con service. \n",
78 | "\n",
79 | "The service returns a list of floats in the range [-1, 1], where each float is the prediction (score) for the corresponding sentence and topic. The sign of the score indicates if the service predicts that the argument supports the topic (positive score), or contests the topic (negative score). The absolute value of the score indicates the confidence of the prediction. Low absolute values indicates low confidence, while high absolute values indicates high confidence.\n",
80 | "\n",
81 | "Documentation for the pro-con service can be found here: https://early-access-program.debater.res.ibm.com/#pro_con\n"
82 | ]
83 | },
84 | {
85 | "cell_type": "markdown",
86 | "metadata": {},
87 | "source": [
88 | "## Your turn!\n",
89 | "\n",
90 | "Please write the code for the following:\n",
91 | "\n",
92 | "1. Create a *pro_con_client* by calling the method `get_pro_con_client()` of the `debater_api` object and storing the resulting object in the `pro_con_client` variable.\n",
93 | "2. Create a list in the format required for this client - list of dictionaries, where each dictionary contains two keys - `topic` and `sentence`. Use `topic` and `arguments` variables previously defined. Name the list 'sentence_topics_dicts'.\n",
94 | "3. Run the pro-con client on this list, using the `run(sentence_topic_dicts)` method of *pro_con_client*, while storing the result in `pro_con_scores`."
95 | ]
96 | },
97 | {
98 | "cell_type": "code",
99 | "execution_count": null,
100 | "metadata": {
101 | "scrolled": false
102 | },
103 | "outputs": [],
104 | "source": [
105 | "pro_con_client = debater_api.get_pro_con_client()\n",
106 | "sentence_topic_dicts = [{'sentence': argument, 'topic': topic} for argument in arguments]\n",
107 | "pro_con_scores = pro_con_client.run(sentence_topic_dicts)"
108 | ]
109 | },
110 | {
111 | "cell_type": "markdown",
112 | "metadata": {},
113 | "source": [
114 | "# 2. Using pro con results\n",
115 | "Now, we measure the accuracy of the results. We will compare the service prediction to the labels created by human annotators. The label field contains three possible values: '1' for supporting arguments, '-1' for contesting arguments, and '0' for neutral or mixed arguments.\n",
116 | "\n",
117 | "We will measure how many arguments were classified as supporting arguments and how many of these were actually labeled as supporting arguments. We can then print the accuracy of the pro predictions. \n",
118 | "\n",
119 | "We will repeat these steps for the contesting arguments as well. "
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": null,
125 | "metadata": {
126 | "scrolled": true
127 | },
128 | "outputs": [],
129 | "source": [
130 | "argument_score_label_list = list(zip(arguments, pro_con_scores, labels))\n",
131 | "\n",
132 | "predicted_pro = [triple for triple in argument_score_label_list if triple[1] > 0]\n",
133 | "predicted_pro_labeled_pro = [triple for triple in predicted_pro if triple[2] == '1']\n",
134 | "print('predicted pro accuracy {}'.format(len(predicted_pro_labeled_pro) / len(predicted_pro)))\n",
135 | "\n",
136 | "predicted_con = [triple for triple in argument_score_label_list if triple[1] < 0]\n",
137 | "predicted_con_labeled_con = [triple for triple in predicted_con if triple[2] == '-1']\n",
138 | "print('predicted con accuracy {}'.format(len(predicted_con_labeled_con) / len(predicted_con)))\n"
139 | ]
140 | },
141 | {
142 | "cell_type": "markdown",
143 | "metadata": {},
144 | "source": [
145 | "# 3. Looking into polarity errors:\n",
146 | "We saw that the accuracy of our classifier is about 70%. This is very nice result, but let's look at the arguments that we missed and try to understand if we can improve the results."
147 | ]
148 | },
149 | {
150 | "cell_type": "code",
151 | "execution_count": null,
152 | "metadata": {
153 | "scrolled": true
154 | },
155 | "outputs": [],
156 | "source": [
157 | "def sign(x):\n",
158 | " if x > 0:\n",
159 | " return '1'\n",
160 | " return '-1'\n",
161 | "\n",
162 | "\n",
163 | "arguments_with_wrong_polarity = [triple for triple in argument_score_label_list if sign(triple[1]) != triple[2]]\n",
164 | "print('Arguments with wrong polarity:')\n",
165 | "for triple in arguments_with_wrong_polarity[:10]:\n",
166 | " print('score: {}, label: {}, argument: {}'.format(triple[1], triple[2], triple[0]))"
167 | ]
168 | },
169 | {
170 | "cell_type": "markdown",
171 | "metadata": {},
172 | "source": [
173 | "We can see that some of the arguments which were predicted incorrectly were labeled as '0'. It means that the human annotators didn't think that these arguments have a clear stance. In addition, we can see that although there are a few arguments with relatively high confidence (high absolute value) in this list, most of the arguments have a relatively low prediction confidence. \n",
174 | "\n",
175 | "A possible improvement will be to take into account not only the sign of the score but also its confidence. We can decide to classify to either pro or con only if the prediction confidence is higher than some threshold and classify remaining arguments as undetermined or neutral.\n",
176 | "\n",
177 | "How should choose such a threshold? Choosing a high threshold improves the precision of the classification, but reduces the coverage - it means that we will take into account only a limited portion of the arguments. "
178 | ]
179 | },
180 | {
181 | "cell_type": "markdown",
182 | "metadata": {},
183 | "source": [
184 | "# 4. Generating coverage-precision curve\n",
185 | "\n",
186 | "In order to choose a threshold that provides the optimal trade off between precision and coverage, we will generate the coverage-precision curve for our pro-con results. For each given threshold, we can calculate the corresponding precision and coverage, and add the point (coverage, precision) to a graph on the coverage-precision plane. We will get a curve, parameterized by the threshold, which enables us to choose the correct threshold for our use case. "
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "execution_count": null,
192 | "metadata": {
193 | "scrolled": true
194 | },
195 | "outputs": [],
196 | "source": [
197 | "from wealth_util import calculate_statistics, plot_graph\n",
198 | "\n",
199 | "pro_con_statistics = calculate_statistics('./wealth_test_set.csv', pro_con_scores)\n",
200 | "plot_graph(examples_list=[pro_con_statistics], labels=['prod'])"
201 | ]
202 | },
203 | {
204 | "cell_type": "markdown",
205 | "metadata": {},
206 | "source": [
207 | "We can see that at 50% coverage, precision is about 93% for the pro arguments and almost 95% for the con arguments. For a coverage of 80%, precision will drop to about 82% in both for pro and con arguments. "
208 | ]
209 | },
210 | {
211 | "cell_type": "markdown",
212 | "metadata": {},
213 | "source": [
214 | "# 5. Fine-tuning the pro-con service\n",
215 | "In order to further improve the results, we can fine-tune the model we use on our data. The generic model which we used has been trained on over 400,000 examples, collected from both newspapers and magazines corpus and from the crowd. The model is a transformer based model (https://en.wikipedia.org/wiki/Transformer_(machine_learning_model). It can be fine-tuned on sample data from the desired domain in order to improve the service results. Experiments show that fine-tuning on even a relatively small sample from the desired domain can significantly improve results in the said domain.\n",
216 | "\n",
217 | "The IBM cloud instance of Project Debater Early Access Program offers GPU based services. It is also possible to get a docker image of a given service and run it locally. Both CPU/GPU images can be provided.\n",
218 | "\n",
219 | "Since the cloud instances are shared between all of our users, we do not allow to fine-tune them. But as it is very simple to run our services locally, a user can get the image, fine-tune it quickly, and use it for inference.\n",
220 | "\n",
221 | "In this tutorial we will show an example of running the pro-con service from a docker image on the virtual machine. Since the virtual machine doesn't have a GPU, we will use a CPU based image. It is important to mention that a GPU instance will run much faster than a CPU instance (we use Nvidia K80 and run about 3000 examples per minute, while running the same service on a mac with i9 core, 2.3GHz with 64 GB ram yields about 600 examples per minute).\n",
222 | "\n",
223 | "In general, in order to fine-tune the pro-con service, we need a CSV file with three columns: motion (topic), sentence (argument), and label. Each line should be one example, with the topic and argument, and the correct label. The label enumeration should be '1' for pro arguments, '0' for con arguments, and '2' for neutral arguments. Note that these labels are different from the labels the service returns at inference - a real number in the range [-1,1] where its sign represents its class (positive for pro arguments and negative for con arguments).\n",
224 | "\n",
225 | "Sometimes even a few dozens of examples can improve the basic model, and the more the better. A training parameter that can be configured is the number of training epochs. This parameter defines how many rounds we will apply to our training set. On the one hand, training many times will fit the model better to the data, but on the other hand, it can cause overfit, which means that it will match very well the training data set at the cost of lower results on the actual data. A simple rule of thumb is that we should try to increase the number of epochs when the training set is larger. At this tutorial, we use less than 500 examples, and therefore we will set the number of epochs to one. \n",
226 | "\n",
227 | "It is important not to train the model on the same arguments set that you use for evaluating the model since it may overfit to the training set.\n",
228 | "\n",
229 | "In our tutorial, we will use a different set of arguments collected from the same event: [wealth_train_set.csv](./wealth_train_set.csv). \n",
230 | "\n",
231 | "## Your turn! Lets start to run a local pro-con service and recreate it.\n",
232 | "\n",
233 | "1. Run local pro-con service:\n",
234 | "- Open a new terminal tab (File -> New Tab).\n",
235 | "\n",
236 | "- Run a docker command for runn the pro-con docker image and assign port 8000 to it:\n",
237 | "`docker run -p 8000:8000 us.icr.io/ris2-debater/pro-con:4.4.b37`\n",
238 | "\n",
239 | "- wait until this text appears: \"run: server started\"\n",
240 | "\n",
241 | "\n",
242 | "2. Fine-tune it\n",
243 | "- open the following url in chrome: http://localhost:8000/model.html\n",
244 | "\n",
245 | "- under the title \"Training data sets\", click on \"create\" button (you don't have to give it a name).\n",
246 | "\n",
247 | "- click on \"Browse\" button, and select wealth_train_set.csv.\n",
248 | "\n",
249 | "- click Upload\n",
250 | "\n",
251 | "- Give name to the new model near \"Train new model\" set \"epochs\" to 1, and click \"start training\".\n",
252 | "\n",
253 | "\n",
254 | "- please note that there is a UI issue that causes the log output to wrap itself \n",
255 | "\n",
256 | "- wait until the new model name appears under “Available models”.\n",
257 | "- Select the radio button that correspond to the new model.\n",
258 | "\n",
259 | "\n",
260 | "\n",
261 | "Now you have a local service, fine tune with example from the same topic. \n"
262 | ]
263 | },
264 | {
265 | "cell_type": "markdown",
266 | "metadata": {},
267 | "source": [
268 | "# 6. Coverage-precision curve for the fine-tuned model.\n",
269 | "\n",
270 | "In order to compare between the generic model and the fine-tuned model, you will generate again the precision coverage curve for the fine-tuned model, and then plot the two curves on the same graph. \n",
271 | "\n",
272 | "Start by setting the client host to http://localhost:8000 and then generate the pro-con scores the same way you did for the generic model. Store the scores at the variable `ft_pro_con_scores`"
273 | ]
274 | },
275 | {
276 | "cell_type": "code",
277 | "execution_count": null,
278 | "metadata": {},
279 | "outputs": [],
280 | "source": [
281 | "pro_con_client.set_host('http://localhost:8000')\n",
282 | "ft_pro_con_scores = pro_con_client.run(sentence_topic_dicts)"
283 | ]
284 | },
285 | {
286 | "cell_type": "markdown",
287 | "metadata": {},
288 | "source": [
289 | "Now we can calculate the statistics for the fine-tuned model, and plot the old and new statistics onthe same graph."
290 | ]
291 | },
292 | {
293 | "cell_type": "code",
294 | "execution_count": null,
295 | "metadata": {},
296 | "outputs": [],
297 | "source": [
298 | "ft_pro_con_statistics = calculate_statistics('./wealth_test_set.csv', ft_pro_con_scores)\n",
299 | "plot_graph(examples_list=[pro_con_statistics, ft_pro_con_statistics], labels=['prod', 'ft'])"
300 | ]
301 | },
302 | {
303 | "cell_type": "markdown",
304 | "metadata": {},
305 | "source": [
306 | "We can see that the fine-tuning improved significantly our pro precision, and it also improved the con predictions for most of the coverage range. We can see that for the con side, for coverage = 0.8, we get a precision of about 0.93."
307 | ]
308 | },
309 | {
310 | "cell_type": "markdown",
311 | "metadata": {},
312 | "source": [
313 | "# 7. Generating a narrative using the fine-tuned model \n",
314 | "\n",
315 | "Now, we ready to use the model predictions we have, with the threshold we chose, to create a narrative. We will use the narrative generation client and create an opoosing speech, from all the arguments identified as con above the 0.8 threshold."
316 | ]
317 | },
318 | {
319 | "cell_type": "code",
320 | "execution_count": null,
321 | "metadata": {},
322 | "outputs": [],
323 | "source": [
324 | "dominant_concept = 'Redistribution of income and wealth'\n",
325 | "\n",
326 | "customizations = \\\n",
327 | " [\n",
328 | " {\n",
329 | " \"title\": \"Number of paragraphs\",\n",
330 | " \"description\": \"Set an upper bound on number of paragraphs (each yielding a theme)\",\n",
331 | " \"type\": \"numParagraphsToSelect\",\n",
332 | " \"items\":\n",
333 | " [\n",
334 | " {\n",
335 | " \"key\": \"max_num_of_paragraphs\",\n",
336 | " \"description\": \"recommended values: from 4 to 8\",\n",
337 | " \"value\": 3,\n",
338 | " \"itemType\": \"single_integer\"\n",
339 | " }\n",
340 | " ]\n",
341 | " },\n",
342 | " {\n",
343 | " \"title\": \"Speech length in minutes\",\n",
344 | " \"description\": \"Set an upper bound on number of minutes the speech should last.\",\n",
345 | " \"type\": \"speechLengthInMinutes\",\n",
346 | " \"items\":\n",
347 | " [\n",
348 | " {\n",
349 | " \"key\": \"max_length_in_minutes\",\n",
350 | " \"description\": \"A float number expressed as decimal with up to two digits to the right of the decimal point. Recommended values: from 2.0 to 4.0\",\n",
351 | " \"value\": 3,\n",
352 | " \"itemType\": \"single_float\",\n",
353 | " }\n",
354 | " ]\n",
355 | " },\n",
356 | " {\n",
357 | " \"title\": \"System Polarity Threshold\",\n",
358 | " \"description\": \"Sets the threshold for filtering of arguments by their polarity score. Can specify absolute value, and / or fraction of the top scoring arguments. When both thresholds are specified, both are employed.\",\n",
359 | " \"type\": \"systemPolarityThreshold\",\n",
360 | " \"items\":\n",
361 | " [\n",
362 | " {\n",
363 | " \"key\": \"system_polarity_absolute_threshold\",\n",
364 | " \"description\": \"Threshold, in the range [0,1]. Arguments whose procon score falls below that threshold will be filtered out.\",\n",
365 | " \"value\": 0.0,\n",
366 | " \"itemType\": \"single_float\",\n",
367 | " },\n",
368 | " {\n",
369 | " \"key\": \"system_polarity_percentage_threshold\",\n",
370 | " \"description\": \"Threshold, in the range [0,1]. Arguments whose rank, in the arguments sorted by their procon score, is not among the top threshold of the list, will be filtered out.\",\n",
371 | " \"value\": 0.8,\n",
372 | " \"itemType\": \"single_float\",\n",
373 | " }\n",
374 | "\n",
375 | " ]\n",
376 | " }\n",
377 | " ]\n",
378 | "\n",
379 | "\n",
380 | "speech = debater_api.get_narrative_generation_client().run(topic=topic,\n",
381 | " dc=dominant_concept,\n",
382 | " sentences=arguments,\n",
383 | " pro_con_scores=pro_con_scores,\n",
384 | " polarity=Polarity.CON,\n",
385 | " customizations=customizations)\n",
386 | "\n",
387 | "print('\\n\\n' + str(speech))"
388 | ]
389 | }
390 | ],
391 | "metadata": {
392 | "kernelspec": {
393 | "display_name": "Python 3",
394 | "language": "python",
395 | "name": "python3"
396 | },
397 | "language_info": {
398 | "codemirror_mode": {
399 | "name": "ipython",
400 | "version": 3
401 | },
402 | "file_extension": ".py",
403 | "mimetype": "text/x-python",
404 | "name": "python",
405 | "nbconvert_exporter": "python",
406 | "pygments_lexer": "ipython3",
407 | "version": "3.7.10"
408 | }
409 | },
410 | "nbformat": 4,
411 | "nbformat_minor": 4
412 | }
413 |
--------------------------------------------------------------------------------
/pro_con_customization/wealth_util.py:
--------------------------------------------------------------------------------
1 | # (C) Copyright IBM Corp. 2020.
2 |
3 | import csv
4 | import matplotlib.pyplot as plt
5 | from sklearn.metrics import classification_report
6 |
7 |
8 | def sign(n):
9 | if n < 0:
10 | return -1
11 | return 1
12 |
13 |
14 | def calculate_statistics(labels_file, pro_con_scores):
15 | with open(labels_file, encoding='utf8') as csv_file:
16 | reader = csv.DictReader(csv_file)
17 | examples = list(reader)
18 |
19 | for i, example in enumerate(examples):
20 | example['score'] = pro_con_scores[i]
21 |
22 | examples = sorted(examples, key=lambda k: abs(k['score']))
23 | for i, example in enumerate(examples):
24 | example['coverage'] = (len(examples) - i) / len(examples)
25 | predicted_classes = [sign(example['score']) for example in examples[i:]]
26 | labels = [int(example['label']) for example in examples[i:]]
27 | labels_set = list(set(predicted_classes))
28 | if len(labels_set) > 1:
29 | example['classification_report'] = classification_report(labels, predicted_classes, digits=4, zero_division=0,
30 | labels=labels_set, output_dict=True)
31 | else:
32 | example['classification_report'] = classification_report(labels, predicted_classes, digits=4, zero_division=0,
33 | output_dict=True)
34 | # generate unclassified predictions
35 | if i > 0:
36 | labels = [int(example['label']) for example in examples[:i]]
37 | predicted_classes = [0 for example in examples[:i]]
38 | classification_report_ = classification_report(labels, predicted_classes, digits=4, zero_division=0,
39 | output_dict=True)
40 | example['classification_report']['0'] = {}
41 | example['classification_report']['0']['precision'] = classification_report_['0']['precision']
42 |
43 | return examples
44 |
45 |
46 | def add_line(examples, label, stance):
47 | stance_2_class_label = {'pro': '1', 'con': '-1', 'neutral': '0'}
48 | class_label = stance_2_class_label[stance]
49 | plt.plot([example['coverage'] for example in examples],
50 | [example['classification_report'][class_label]['precision']
51 | if class_label in example['classification_report'] else 1.0
52 | for example in examples]
53 | , label='{}_{}'.format(label, stance))
54 |
55 |
56 | def plot_stance(stance, examples_list, labels):
57 | for examples, label in zip(examples_list, labels):
58 | add_line(examples, label, stance)
59 | plt.title('precision / coverage - SBC data {}'.format(stance))
60 | plt.xlabel('coverage')
61 | plt.ylabel('precision')
62 | plt.legend(loc='best')
63 |
64 |
65 | def plot_graph(examples_list, labels):
66 | plt.figure(figsize=(12, 8))
67 | plt.subplot(1, 3, 1)
68 | plot_stance('pro', examples_list, labels)
69 | plt.subplot(1, 3, 2)
70 | plot_stance('con', examples_list, labels)
71 | plt.subplot(1, 3, 3)
72 | plot_stance('neutral', examples_list, labels)
73 | plt.show()
74 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | notebook
2 | prettytable
3 |
--------------------------------------------------------------------------------
/run_kpa_scripts/run_keypoint_analysis_script.py:
--------------------------------------------------------------------------------
1 | # (C) Copyright IBM Corporation 2020-2022
2 | # LICENSE: Apache License 2.0 (Apache-2.0)
3 | # http://www.apache.org/licenses/LICENSE-2.0
4 |
5 | import logging
6 |
7 | from debater_python_api.api.clients.key_point_analysis.KpAnalysisUtils import KpAnalysisUtils
8 | from debater_python_api.api.clients.keypoints_client import KpAnalysisTaskFuture, KpAnalysisClient
9 | from debater_python_api.api.debater_api import DebaterApi
10 | import pandas as pd
11 |
12 |
13 | def write_sentences_to_csv(sentences, out_file):
14 | if len(sentences) == 0:
15 | logging.info('there are no sentences, not saving file')
16 | return
17 |
18 | cols = list(sentences[0].keys())
19 | rows = [[s[col] for col in cols] for s in sentences]
20 | df = pd.DataFrame(rows, columns=cols)
21 | df.to_csv(out_file, index=False)
22 |
23 |
24 | def get_comments_ids_and_texts(file, ids_column, comment_text_column):
25 | df = pd.read_csv(file)
26 | id_text = sorted(list(zip(df[ids_column], df[comment_text_column])))
27 | id_text = [t for t in id_text if len(t[1]) < 1000] # comments must have at most 1000 chars
28 | comments_ids = [str(t[0]) for t in id_text]
29 | comments_texts = [str(t[1]) for t in id_text]
30 | return comments_ids, comments_texts
31 |
32 |
33 | if __name__ == '__main__':
34 | KpAnalysisUtils.init_logger()
35 |
36 | # ======================= update params =======================
37 | api_key = ''
38 | file = ''
39 | comment_ids_column = ''
40 | comment_text_column = ''
41 | domain = ''
42 | host = ''
43 | # ======================= update params =======================
44 |
45 | debater_api = DebaterApi(apikey=api_key)
46 | keypoints_client = KpAnalysisClient(apikey=api_key, host=host)
47 |
48 | # what to run
49 | delete_all_domains = False # deletes-all! cannot be undone
50 | restart = False
51 | delete_domain = restart
52 | create_domain = restart
53 | upload_comments = restart
54 | run_kpa = False
55 | download_sentences = False
56 | get_report = False
57 | connect_to_running_job = False
58 | cancel_all_running_jobs_in_the_domain = False
59 |
60 | # how to run
61 | domain_params = {}
62 | run_params = {'sentence_to_multiple_kps': True}
63 |
64 | if delete_all_domains:
65 | answer = input("Are you sure you want to delete ALL domains (cannot be undone)? yes/no:")
66 | if answer == 'yes':
67 | keypoints_client.delete_all_domains_cannot_be_undone()
68 |
69 | if get_report:
70 | KpAnalysisUtils.print_report(keypoints_client.get_full_report())
71 |
72 | if delete_domain:
73 | answer = input(f'Are you sure you want to delete domain: {domain} (cannot be undone)? yes/no:')
74 | if answer == 'yes':
75 | KpAnalysisUtils.delete_domain_ignore_doesnt_exist(keypoints_client, domain)
76 |
77 | if create_domain:
78 | KpAnalysisUtils.create_domain_ignore_exists(keypoints_client, domain, {})
79 |
80 | if upload_comments:
81 | comments_ids, comments_texts = get_comments_ids_and_texts(file, comment_ids_column, comment_text_column)
82 |
83 | keypoints_client.upload_comments(domain=domain,
84 | comments_ids=comments_ids,
85 | comments_texts=comments_texts)
86 |
87 | keypoints_client.wait_till_all_comments_are_processed(domain=domain)
88 |
89 | if download_sentences:
90 | sentences = keypoints_client.get_sentences_for_domain(domain)
91 | write_sentences_to_csv(sentences, file.replace('.csv', '_sentences.csv'))
92 |
93 | future = None
94 | if run_kpa:
95 | keypoints_client.wait_till_all_comments_are_processed(domain=domain)
96 | future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)
97 |
98 | if connect_to_running_job:
99 | future = KpAnalysisTaskFuture(keypoints_client, '')
100 |
101 | if future is not None:
102 | kpa_result = future.get_result(high_verbosity=True, polling_timout_secs=30)
103 | KpAnalysisUtils.write_result_to_csv(kpa_result, file.replace('.csv', '_kpa_results.csv'))
104 |
105 | if cancel_all_running_jobs_in_the_domain:
106 | keypoints_client.cancel_all_extraction_jobs_for_domain(domain=domain)
107 |
--------------------------------------------------------------------------------
/run_kpa_scripts/run_keypoint_analysis_script_two_stances.py:
--------------------------------------------------------------------------------
1 | # (C) Copyright IBM Corporation 2020-2022
2 | # LICENSE: Apache License 2.0 (Apache-2.0)
3 | # http://www.apache.org/licenses/LICENSE-2.0
4 |
5 | import logging
6 | from debater_python_api.api.clients.key_point_analysis.KpAnalysisUtils import KpAnalysisUtils
7 | from debater_python_api.api.clients.keypoints_client import KpAnalysisTaskFuture, \
8 | KpaIllegalInputException, KpAnalysisClient
9 | import pandas as pd
10 |
11 | def write_sentences_to_csv(sentences, out_file):
12 | if len(sentences) == 0:
13 | logging.info('there are no sentences, not saving file')
14 | return
15 |
16 | cols = list(sentences[0].keys())
17 | rows = [[s[col] for col in cols] for s in sentences]
18 | df = pd.DataFrame(rows, columns=cols)
19 | df.to_csv(out_file, index=False)
20 |
21 |
22 | def run_on_sentences_two_stances(domain, stance_to_future):
23 | keypoints_client.wait_till_all_comments_are_processed(domain=domain)
24 | run_params['stances_to_run'] = ['pos']
25 | run_params['stances_threshold'] = 0.5
26 | stance_to_future['pro'] = keypoints_client.start_kp_analysis_job(domain=domain,
27 | run_params=run_params)
28 |
29 | run_params['stances_to_run'] = ['neg', 'sug']
30 | run_params['stances_threshold'] = 0.5
31 | stance_to_future['con'] = keypoints_client.start_kp_analysis_job(domain=domain,
32 | run_params=run_params)
33 |
34 | def get_merge_and_save_results(stance_to_future, file):
35 | stance_to_result = {}
36 | for stance in stance_to_future:
37 | future = stance_to_future[stance]
38 | try:
39 | kpa_result = future.get_result(high_verbosity=True, polling_timout_secs=60)
40 | stance_to_result[stance] = kpa_result
41 | except Exception as e:
42 | if 'maybe the input size is too small' not in e.args[0]:
43 | raise e
44 | else:
45 | print(f'job: {future.get_job_id()} don\'t have enough data for extracting key points')
46 | kpa_result = {'keypoint_matchings': []}
47 | stance_to_result[stance] = kpa_result
48 |
49 | if 'pro' in stance_to_result and 'con' in stance_to_result:
50 | pro_result = KpAnalysisUtils.set_stance_to_result(stance_to_result['pro'], 'pro')
51 | KpAnalysisUtils.write_result_to_csv(pro_result, file.replace('.csv', '_pro_kpa_results.csv'))
52 | con_result = KpAnalysisUtils.set_stance_to_result(stance_to_result['con'], 'con')
53 | KpAnalysisUtils.write_result_to_csv(con_result, file.replace('.csv', '_con_kpa_results.csv'))
54 | merged_result = KpAnalysisUtils.merge_two_results(pro_result, con_result)
55 | KpAnalysisUtils.write_result_to_csv(merged_result, file.replace('.csv', '_merged_kpa_results.csv'))
56 | else:
57 | print('a stance is missing: %s' % str(list(stance_to_result.keys())))
58 |
59 |
60 | def get_comments_ids_and_texts(file, comment_ids_column, comment_text_column):
61 | df = pd.read_csv(file)
62 | id_text = sorted(list(zip(df[comment_ids_column], df[comment_text_column])))
63 | id_text = [t for t in id_text if len(t[1]) < 1000] # comments must have at most 1000 chars
64 | comments_ids = [str(t[0]) for t in id_text]
65 | comments_texts = [str(t[1]) for t in id_text]
66 | return comments_ids, comments_texts
67 |
68 |
69 | if __name__ == '__main__':
70 | KpAnalysisUtils.init_logger()
71 |
72 | # ======================= update params =======================
73 | api_key = ''
74 | file = ''
75 | comment_ids_column = ''
76 | comment_text_column = ''
77 | domain = ''
78 | host = ''
79 | # ======================= update params =======================
80 |
81 | keypoints_client = KpAnalysisClient(apikey=api_key, host=host)
82 |
83 | # what to run
84 | delete_all_domains = False # deletes-all! cannot be undone
85 | restart = False
86 | delete_domain = restart
87 | create_domain = restart
88 | upload_comments = restart
89 | run_kpa_both_stances = False
90 | download_sentences = False
91 | get_report = False
92 | connect_to_running_job = False
93 | cancel_all_running_jobs_in_the_domain = False
94 |
95 | # how to run
96 | domain_params = {'do_stance_analysis': True}
97 | run_params = {'sentence_to_multiple_kps': True}
98 |
99 | if delete_all_domains:
100 | answer = input("Are you sure you want to delete ALL domains (cannot be undone)? yes/no:")
101 | if answer == 'yes':
102 | keypoints_client.delete_all_domains_cannot_be_undone()
103 |
104 | if get_report:
105 | KpAnalysisUtils.print_report(keypoints_client.get_full_report())
106 |
107 | if delete_domain:
108 | answer = input(f'Are you sure you want to delete domain: {domain} (cannot be undone)? yes/no:')
109 | if answer == 'yes':
110 | KpAnalysisUtils.delete_domain_ignore_doesnt_exist(keypoints_client, domain)
111 |
112 | if create_domain:
113 | KpAnalysisUtils.create_domain_ignore_exists(keypoints_client, domain, domain_params)
114 |
115 | if upload_comments:
116 | comments_ids, comments_texts = get_comments_ids_and_texts(file, comment_ids_column, comment_text_column)
117 |
118 | keypoints_client.upload_comments(domain=domain,
119 | comments_ids=comments_ids,
120 | comments_texts=comments_texts)
121 |
122 | keypoints_client.wait_till_all_comments_are_processed(domain=domain)
123 |
124 | if download_sentences:
125 | sentences = keypoints_client.get_sentences_for_domain(domain)
126 | write_sentences_to_csv(sentences, file.replace('.csv', '_sentences.csv'))
127 |
128 | stance_to_future = {}
129 | if run_kpa_both_stances:
130 | run_on_sentences_two_stances(domain, stance_to_future)
131 |
132 | if connect_to_running_job:
133 | stance_to_future['pro'] = KpAnalysisTaskFuture(keypoints_client, '')
134 | stance_to_future['con'] = KpAnalysisTaskFuture(keypoints_client, '')
135 |
136 | if len(stance_to_future.keys()) > 0:
137 | get_merge_and_save_results(stance_to_future, file)
138 |
139 | if cancel_all_running_jobs_in_the_domain:
140 | keypoints_client.cancel_all_extraction_jobs_for_domain(domain=domain)
--------------------------------------------------------------------------------
/survey_usecase/kpa_quick_start_tutorial-with_results.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "79190f16-c04e-4480-a10d-183da359c5bf",
6 | "metadata": {},
7 | "source": [
8 | "# Using Key Point Summarization for analyzing and finding insights in a survey data\n",
9 | "\n",
10 | "A new version of the SDK has recently been introduced. \n",
11 | "\n",
12 | "The tutorial for the updated version in available [here](./new_sdk/kpa_quick_start_tutorial-with_results.ipynb).\n",
13 | "\n",
14 | "The tutorial for the legacy version (to be supported until the end of July 2023) in available [here](./legacy_sdk/kpa_quick_start_tutorial-with_results.ipynb).\n"
15 | ]
16 | }
17 | ],
18 | "metadata": {
19 | "kernelspec": {
20 | "display_name": "Python 3 (ipykernel)",
21 | "language": "python",
22 | "name": "python3"
23 | },
24 | "language_info": {
25 | "codemirror_mode": {
26 | "name": "ipython",
27 | "version": 3
28 | },
29 | "file_extension": ".py",
30 | "mimetype": "text/x-python",
31 | "name": "python",
32 | "nbconvert_exporter": "python",
33 | "pygments_lexer": "ipython3",
34 | "version": "3.8.16"
35 | }
36 | },
37 | "nbformat": 4,
38 | "nbformat_minor": 5
39 | }
40 |
--------------------------------------------------------------------------------
/survey_usecase/legacy_sdk/kpa_parameters.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/survey_usecase/legacy_sdk/kpa_parameters.pdf
--------------------------------------------------------------------------------
/survey_usecase/legacy_sdk/kpa_quick_start_tutorial.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "a7f1683b",
6 | "metadata": {},
7 | "source": [
8 | "# Using *Key Point Analysis* service for analyzing and finding insights in a survey data \n",
9 | "\n",
10 | "#### **Important Notice**: This tutorial describes the legacy SDK, of debater-python-api version up to 4.3.2. The tutorial of the updated SDK, starting from debater-python-api version 5.0.0, is available [here](../new_sdk/kpa_quick_start_tutorial.ipynb).\n",
11 | "\n",
12 | "When you have a large collection of texts representing people’s opinions (such as product reviews, survey answers or social media), it is difficult to understand the key issues that come up in the data. Going over thousands of comments is prohibitively expensive. Existing automated approaches are often limited to identifying recurring phrases or concepts and the overall sentiment toward them, but do not provide detailed or actionable insights.\n",
13 | "\n",
14 | "In this tutorial you will gain hands-on experience in using *Key Point Analysis* (KPA) for analyzing and deriving insights from open-ended answers. \n",
15 | "\n",
16 | "The data we will use is a [community survey conducted in the city of Austin](https://data.austintexas.gov/dataset/Community-Survey/s2py-ceb7). In this survey, the citizens of Austin were asked \"If there was ONE thing you could share with the Mayor regarding the City of Austin (any comment, suggestion, etc.), what would it be?\". "
17 | ]
18 | },
19 | {
20 | "cell_type": "markdown",
21 | "id": "2ab28f81",
22 | "metadata": {},
23 | "source": [
24 | "## 1. Run *Key Point Analysis* (data from 2016)"
25 | ]
26 | },
27 | {
28 | "cell_type": "markdown",
29 | "id": "d85aed7e",
30 | "metadata": {},
31 | "source": [
32 | "Lets first import all required packages for this tutoarial and initialize the *Key Point Analysis* client. The client prints information using the logger and a suitable verbosity level should be set. The client object is configured with an API key. It should be retrieved from the [Project Debater Early Access Program](https://early-access-program.debater.res.ibm.com/) site. In the code bellow it is passed by the enviroment variable *DEBATER_API_KEY* (you may also modify the code and place the api-key directly)."
33 | ]
34 | },
35 | {
36 | "cell_type": "code",
37 | "execution_count": null,
38 | "id": "510552af",
39 | "metadata": {},
40 | "outputs": [],
41 | "source": [
42 | "from debater_python_api.api.clients.keypoints_client import KpAnalysisClient, KpAnalysisTaskFuture\n",
43 | "from debater_python_api.api.clients.key_point_analysis.KpAnalysisUtils import KpAnalysisUtils\n",
44 | "import os\n",
45 | "import csv\n",
46 | "import random\n",
47 | "\n",
48 | "KpAnalysisUtils.init_logger()\n",
49 | "api_key = os.environ['DEBATER_API_KEY']\n",
50 | "host = 'https://keypoint-matching-backend.debater.res.ibm.com'\n",
51 | "keypoints_client = KpAnalysisClient(api_key, host)"
52 | ]
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "id": "71725a3a",
57 | "metadata": {
58 | "tags": []
59 | },
60 | "source": [
61 | "### 1.1 Read the data and run *key point analysis* over it\n",
62 | "Let's read the data from *dataset_austin.csv* file, which holds the Austin survey dataset, and print the first comment."
63 | ]
64 | },
65 | {
66 | "cell_type": "code",
67 | "execution_count": null,
68 | "id": "68c6f083",
69 | "metadata": {},
70 | "outputs": [],
71 | "source": [
72 | "with open('./dataset_austin.csv') as csv_file:\n",
73 | " reader = csv.DictReader(csv_file)\n",
74 | " comments = [dict(d) for d in reader]\n",
75 | "\n",
76 | "print(f'There are {len(comments)} comments in the dataset')\n",
77 | "print(comments[0])"
78 | ]
79 | },
80 | {
81 | "cell_type": "markdown",
82 | "id": "25b55e5e",
83 | "metadata": {},
84 | "source": [
85 | "Each comment is a dictionary with an unique_id 'id' and 'text' and a 'year'. We will first remove all comments with text longer than 1000 characters since this is a systme's limit. Then we will filter the comments and take the ones from 2016. \n",
86 | "\n",
87 | "The *Key Point Analysis* service is able to run over hundreds of thousands of sentences, however since the computation is heavy in resources (particularly GPUs) the trial version is limited to 1000 comments. You may request to increase this limit if needed. Since we want the tutorial to be relativly fast and lightweight, we will only run on a sample of 400 comments. Note that running over a larger set improves both the quality and coverage of the results."
88 | ]
89 | },
90 | {
91 | "cell_type": "code",
92 | "execution_count": null,
93 | "id": "fdb6777e",
94 | "metadata": {},
95 | "outputs": [],
96 | "source": [
97 | "comments = [c for c in comments if len(c['text'])<=1000]\n",
98 | "comments_2016 = [c for c in comments if c['year'] == '2016']\n",
99 | "sample_size = 400\n",
100 | "random.seed(0)\n",
101 | "comments_2016_sample = random.sample(comments_2016, sample_size)"
102 | ]
103 | },
104 | {
105 | "cell_type": "markdown",
106 | "id": "916ebe73",
107 | "metadata": {},
108 | "source": [
109 | "*Key point analysis* is a novel and promising approach for summarization, with an important quantitative angle. This service summarizes a collection of comments on a given topic as a small set of key points. The salience of each key point is given by the number of its matching sentences in the given comments."
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "id": "8ff4b8dc",
115 | "metadata": {},
116 | "source": [
117 | "In order to run *Key Point Analysis*, do the following steps:\n",
118 | "\n",
119 | "### 1.2 Create a domain\n",
120 | "The *Key Point Analysis* service stores the data (and cached-results) in a *domain*. A user can create several domains, one for each dataset. Domains are only accessible to the user who created them.\n",
121 | "\n",
122 | "Create a domin using the **keypoints_client.create_domain(domain=domain, domain_params={})** method. Several params can be passed when creating a domain in the domain_params dictionary as described in the documentation. Leaving it empty gives us a good default behaviour. You can also use **KpAnalysisUtils.create_domain_ignore_exists(client=keypoints_client, domain=domain, domain_params={})** if you don't want an exception to be thrown if the domain already exists (note that in such case the domain_params are not updated and are remained as they where before). In this tutorial we will first delete the domain if it exists, since we want to start with an empty domain.\n",
123 | "\n",
124 | "Full documentation of the supported *domain_params* and how they affect the domain can be found [here](kpa_parameters.pdf)."
125 | ]
126 | },
127 | {
128 | "cell_type": "code",
129 | "execution_count": null,
130 | "id": "f4956e78",
131 | "metadata": {},
132 | "outputs": [],
133 | "source": [
134 | "domain = 'austin_demo'\n",
135 | "KpAnalysisUtils.delete_domain_ignore_doesnt_exist(client=keypoints_client, domain=domain)\n",
136 | "keypoints_client.create_domain(domain=domain, domain_params={})"
137 | ]
138 | },
139 | {
140 | "cell_type": "markdown",
141 | "id": "dd618c78",
142 | "metadata": {},
143 | "source": [
144 | "Few domain related points:\n",
145 | "* We can always delete a domain we no longer need using: **KpAnalysisUtils.delete_domain_ignore_doesnt_exist(client=keypoints_client, domain=domain)**\n",
146 | "* Keep in mind that a domain has a state. It stores all comments that had beed uploaded into it and a cache with all calculations performed over this data.\n",
147 | "* If we want to restart and run over the domain from scratch (no comments and no cache), we can delete the domain and then re-create it or obviously use a different domain. Keep in mind that the cache is also cleared and consecutive runs will take longer."
148 | ]
149 | },
150 | {
151 | "cell_type": "markdown",
152 | "id": "d82c51cd",
153 | "metadata": {},
154 | "source": [
155 | "### 1.3 Upload comments into the domain\n",
156 | "Upload the comments into the domain using the **keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)** method. This method receives the domain, a list of comment_ids and a list of comment_texts. When uploading comments into a domain, the *Key Point Analysis* service splits the comments into sentences and runs a minor cleansing on the sentences. If you have domain-specific knowladge and want to split the comments into sentences yourself, you can upload comments that are already splitted into sentences and set the *dont_split* parameter to True (in the domain_params when creating the domain) and *Key Point Analysis* will use the provided sentences as is. \n",
157 | "\n",
158 | "Note that:\n",
159 | "* Comments_ids must be unique\n",
160 | "* The number of comments_ids must match the number comments_texts\n",
161 | "* Comments_texts must not be longer than 1000 characters\n",
162 | "* Uploading the same comment several times (same domain + comment_id, comment_text is ignored) is not a problem and the comment is only uploaded once (if the comment_text is different, it is NOT updated)."
163 | ]
164 | },
165 | {
166 | "cell_type": "code",
167 | "execution_count": null,
168 | "id": "4da5d650",
169 | "metadata": {},
170 | "outputs": [],
171 | "source": [
172 | "comments_texts = [comment['text'] for comment in comments_2016_sample]\n",
173 | "comments_ids = [comment['id'] for comment in comments_2016_sample]\n",
174 | "keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)"
175 | ]
176 | },
177 | {
178 | "cell_type": "markdown",
179 | "id": "183b1da7",
180 | "metadata": {},
181 | "source": [
182 | "### 1.4 Wait for the comments to be processed\n",
183 | "Comments that are uploaded to the domain are being processed. This takes some times and runs in an async manner. We can't run an analysis before this phase finishes and we need to wait till all comments in the domain are processed using the **keypoints_client.wait_till_all_comments_are_processed(domain=domain)** method."
184 | ]
185 | },
186 | {
187 | "cell_type": "code",
188 | "execution_count": null,
189 | "id": "220393ed",
190 | "metadata": {},
191 | "outputs": [],
192 | "source": [
193 | "keypoints_client.wait_till_all_comments_are_processed(domain=domain)"
194 | ]
195 | },
196 | {
197 | "cell_type": "markdown",
198 | "id": "da852b04",
199 | "metadata": {},
200 | "source": [
201 | "### 1.5 Start a Key Point Analysis job\n",
202 | "Start a *Key Point Analysis* job using the **future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)** method. This method receives the domain and a *run_params*. The run_params is a dictionary with various parameters for customizing the job. Leaving it empty gives us a good default behaviour. The job runs in an async manner therefore the method returns a future object.\n",
203 | "\n",
204 | "Few additional options when running an analysis job:\n",
205 | "* The analysis is performed over all comments in the domain. If we need to run over a subset of the comments (split the data by different GEOs/users types/timeframes etc') we can pass a list of comments_ids to the comments_ids parameter and it will create an analysis using only the provided comments.\n",
206 | "* By default, key points are extracted automatically. When we want to provide key points and match all sentences to these key points we can do so by passing them to the keypoints parameter: **run_param['keypoints'] = [...]**. This enables a mode of work named human-in-the-loop where we first automatically extract key points, then we manually edit them (refine non-perfect key points, remove duplicated and add missing ones) and then run again, this time providing the edited keypoints as a given set of key points.\n",
207 | "* It is also possible to provide key points and let KPA add additional missing key points. To do so pass the key points to the keypoint_candidates parameter: **run_param['keypoint_candidates'] = [...]** (see section 4 for an elaborated example).\n",
208 | "* Full documentation of the supported *domain_params* and *run_params* and how they affect the analysis can be found [here](kpa_parameters.pdf)."
209 | ]
210 | },
211 | {
212 | "cell_type": "code",
213 | "execution_count": null,
214 | "id": "948bf751",
215 | "metadata": {},
216 | "outputs": [],
217 | "source": [
218 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params={})"
219 | ]
220 | },
221 | {
222 | "cell_type": "markdown",
223 | "id": "1a1e4414",
224 | "metadata": {},
225 | "source": [
226 | "### 1.6 Wait for the Key Point Analysis job to finish\n",
227 | "Use the returned future and wait till results are available using the **kpa_result = future.get_result()** method. The method waits for the job to finish and eventually returns the result. The result is a dictionary containing the key points (sorted descendingly according to number of matched sentences) and for each key point has a list of matched sentences (sorted descendingly according to their match score). An additional 'none' key point is added which holds all the sentences that don't match any key point."
228 | ]
229 | },
230 | {
231 | "cell_type": "code",
232 | "execution_count": null,
233 | "id": "455eee99",
234 | "metadata": {},
235 | "outputs": [],
236 | "source": [
237 | "kpa_result_2016 = future.get_result(high_verbosity=True, polling_timout_secs=30)"
238 | ]
239 | },
240 | {
241 | "cell_type": "markdown",
242 | "id": "222b3a24",
243 | "metadata": {},
244 | "source": [
245 | "Let's print the results:"
246 | ]
247 | },
248 | {
249 | "cell_type": "code",
250 | "execution_count": null,
251 | "id": "b6af7396",
252 | "metadata": {},
253 | "outputs": [],
254 | "source": [
255 | "KpAnalysisUtils.print_result(kpa_result_2016, n_sentences_per_kp=2, title='2016 Random sample')"
256 | ]
257 | },
258 | {
259 | "cell_type": "markdown",
260 | "id": "74421ee6",
261 | "metadata": {
262 | "tags": []
263 | },
264 | "source": [
265 | "We can also save the results to file. This creates two files, one with the key points and all matched sentences and another summary file with only the key points and their saliance."
266 | ]
267 | },
268 | {
269 | "cell_type": "code",
270 | "execution_count": null,
271 | "id": "5f5ed2e7",
272 | "metadata": {
273 | "tags": []
274 | },
275 | "outputs": [],
276 | "source": [
277 | "KpAnalysisUtils.write_result_to_csv(kpa_result_2016, 'austin_survey_2016_kpa_results.csv')"
278 | ]
279 | },
280 | {
281 | "cell_type": "markdown",
282 | "id": "4b5a5ab0",
283 | "metadata": {},
284 | "source": [
285 | "It is always possible to cancel a pending/running job in the following way:\n",
286 | "* **keypoints_client.cancel_kp_extraction_job(\\)**\n",
287 | "\n",
288 | "Job Id can be found: \n",
289 | "1. It's printed when a job is started \n",
290 | "2. From the future object: **future.get_job_id()**\n",
291 | "3. From user report: **keypoints_client.get_full_report()** (see bellow)\n",
292 | "\n",
293 | "It is also possibe to stop all jobs in a domain, or even all jobs in all domains (might be simpler since there is no need of the job_id):\n",
294 | "* **keypoints_client.cancel_all_extraction_jobs_for_domain(domain)**\n",
295 | "* **keypoints_client.cancel_all_extraction_jobs_all_domains()**\n",
296 | "\n",
297 | "Please cancel long jobs if the results are no longer needed."
298 | ]
299 | },
300 | {
301 | "cell_type": "markdown",
302 | "id": "ccc10ed9-3a2b-44b9-9375-2e4a6a5bf003",
303 | "metadata": {
304 | "tags": []
305 | },
306 | "source": [
307 | "### 1.7 Modify the run_params and increase coverage\n",
308 | "Each domain has a cache that stores all intermediate results that are calculated during the analysis. Therefore modifing the run_params and running another analysis runs much faster and all intersecting intermediate results are retreived from cache. \n",
309 | "\n",
310 | "Let's run again, but now change the **mapping_policy**. The **mapping_policy** is used when mapping all sentences to the final key points: the default value is **NORMAL**. Changing to **STRICT** will cause only the sentence and key point pairs with very high matching confidence to be considered matched, increasing precision but potentially decreasing coverage. We will change it to **LOOSE**, which matches also sentences and key points with lower confidence, and is therefore expected to increase coverage at cost of precision. We will also increase the number of required key points to 100 using the **n_top_kps** parameter. "
311 | ]
312 | },
313 | {
314 | "cell_type": "code",
315 | "execution_count": null,
316 | "id": "199d24fa",
317 | "metadata": {},
318 | "outputs": [],
319 | "source": [
320 | "run_params = {'mapping_policy':'LOOSE', 'n_top_kps': 100}\n",
321 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)\n",
322 | "kpa_result_2016 = future.get_result(high_verbosity=True, polling_timout_secs=30)\n",
323 | "KpAnalysisUtils.write_result_to_csv(kpa_result_2016, 'austin_survey_2016_kpa_results.csv')\n",
324 | "KpAnalysisUtils.print_result(kpa_result_2016, n_sentences_per_kp=2, title='Random sample')"
325 | ]
326 | },
327 | {
328 | "cell_type": "markdown",
329 | "id": "d61557e9",
330 | "metadata": {},
331 | "source": [
332 | "By changing the mapping policy to **LOOSE** and increasing the number of key points, the coverage was increased from 44% to 68%."
333 | ]
334 | },
335 | {
336 | "cell_type": "markdown",
337 | "id": "4523a7d0",
338 | "metadata": {},
339 | "source": [
340 | "### 1.8 User Report\n",
341 | "When we want to see what domains we have, maybe delete old ones that are not needed, see past and present analysis jobs, perhaps take their job_id and fetch their result \n",
342 | "(via **KpAnalysisTaskFuture(keypoints_client, \\).get_result()** ), \n",
343 | "we can get a report with all the needed information"
344 | ]
345 | },
346 | {
347 | "cell_type": "code",
348 | "execution_count": null,
349 | "id": "c10280d7-d91f-4d0b-b634-5d2dac0900bc",
350 | "metadata": {},
351 | "outputs": [],
352 | "source": [
353 | "report = keypoints_client.get_full_report()\n",
354 | "KpAnalysisUtils.print_report(report)"
355 | ]
356 | },
357 | {
358 | "cell_type": "markdown",
359 | "id": "66f6d46c",
360 | "metadata": {},
361 | "source": [
362 | "## 2. Mapping sentences to multiple key points, and creating a Key-Points-Graphs\n",
363 | "By default, each sentence is mapped to one key point at most (the key point with the highest match-score, that follows the **mapping_policy**). We can run again and ask KPA to map each sentence to all key points that are matched according to the **mapping_policy**, by adding the **sentence_to_multiple_kps** parameter."
364 | ]
365 | },
366 | {
367 | "cell_type": "code",
368 | "execution_count": null,
369 | "id": "b677b860",
370 | "metadata": {},
371 | "outputs": [],
372 | "source": [
373 | "run_params = {'sentence_to_multiple_kps': True, 'n_top_kps': 100}\n",
374 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)\n",
375 | "kpa_2016_job_id = future.get_job_id() # saving the job_id for a following section\n",
376 | "kpa_result_2016 = future.get_result(high_verbosity=True, polling_timout_secs=30)"
377 | ]
378 | },
379 | {
380 | "cell_type": "code",
381 | "execution_count": null,
382 | "id": "bb6a89b7",
383 | "metadata": {},
384 | "outputs": [],
385 | "source": [
386 | "KpAnalysisUtils.print_result(kpa_result_2016, n_sentences_per_kp=2, title='Random sample')"
387 | ]
388 | },
389 | {
390 | "cell_type": "markdown",
391 | "id": "9b5d94bd",
392 | "metadata": {},
393 | "source": [
394 | "Now that sentences are mapped to multiple key points, it is possible to create a *key points graph* by first saving the results as before, then translating the results file into a graph-data json file, then load this json file in our demo graph visualization, available at: [key points graph demo](https://keypoint-matching-ui.ris2-debater-event.us-east.containers.appdomain.cloud/)"
395 | ]
396 | },
397 | {
398 | "cell_type": "code",
399 | "execution_count": null,
400 | "id": "0b9c2614",
401 | "metadata": {},
402 | "outputs": [],
403 | "source": [
404 | "KpAnalysisUtils.write_result_to_csv(kpa_result_2016, 'austin_survey_2016_multiple_kpa_results.csv')\n",
405 | "KpAnalysisUtils.generate_graphs_and_textual_summary('austin_survey_2016_multiple_kpa_results.csv')"
406 | ]
407 | },
408 | {
409 | "cell_type": "markdown",
410 | "id": "818fd69a",
411 | "metadata": {},
412 | "source": [
413 | "**generate_graphs_and_textual_summary** creates 4 files:\n",
414 | "* **austin_survey_2016_multiple_kpa_results_graph_data.json**: a graph_data file that can be loaded to: [key points graph demo](https://keypoint-matching-ui.ris2-debater-event.us-east.containers.appdomain.cloud/). It presents the relations between the key points as a graph of key points.\n",
415 | "* **austin_survey_2016_multiple_kpa_results_hierarchical_graph_data.json**: another graph_data file that can be loaded to the graph-demo-site. This graph is simplified, it's more convenient to extract insights from it.\n",
416 | "* **austin_survey_2016_multiple_kpa_results_hierarchical.txt**: This textual file shows the simplified graph (from the previous bullet) as a list of hierarchical bullets.\n",
417 | "* **austin_survey_2016_multiple_kpa_results_hierarchical.docx**: This Microsoft Word document shows the textual bullets (from the previous bullet) as a user-friendly report."
418 | ]
419 | },
420 | {
421 | "cell_type": "markdown",
422 | "id": "67ff92aa",
423 | "metadata": {},
424 | "source": [
425 | "## 3. Run *Key Point Analysis* incrementally\n",
426 | "### 3.1 Run *Key Point Analysis* incrementally on new data (data from 2016 + 2017)\n",
427 | "A year passed, and we collect additional data (data from 2017). We can now upload the 2017 data to the same domain (austin_demo) and have both 2016 and 2017 data in one domain. "
428 | ]
429 | },
430 | {
431 | "cell_type": "code",
432 | "execution_count": null,
433 | "id": "3a5f9111",
434 | "metadata": {},
435 | "outputs": [],
436 | "source": [
437 | "comments_2017 = [c for c in comments if c['year'] == '2017']\n",
438 | "random.seed(0)\n",
439 | "comments_2017_sample = random.sample(comments_2017, sample_size)\n",
440 | "\n",
441 | "domain = 'austin_demo'\n",
442 | "comments_texts = [comment['text'] for comment in comments_2017_sample]\n",
443 | "comments_ids = [comment['id'] for comment in comments_2017_sample]\n",
444 | "keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)\n",
445 | "keypoints_client.wait_till_all_comments_are_processed(domain=domain)"
446 | ]
447 | },
448 | {
449 | "cell_type": "markdown",
450 | "id": "69cdd33f",
451 | "metadata": {},
452 | "source": [
453 | "We can now run a new analysis over all the data in the domain, as we did before, and automatically extract new key points. We can assume that some will be identical to the key points extracted on the 2016 data, some will be similar and some key points will be new.\n",
454 | "\n",
455 | "A better option is to run a new analysis but provide the keypoints from the 2016 analysis and let *Key Point Analysis* add new key points of 2017 data if there are such. One benefit of this approach is that the new result will mostly use 2016 key point and we will be able to compare between them, see what changed, what improved and what not. Another major benefit for this approach is run-time. 2016 data was already analyzed with these key points and since we have a cache in place much of the computation can be avoided. The 2016 key points can be provided via the: **run_param['keypoint_candidates'] = [...]** parameter, passing a list of strings, or we can use: **run_param['keypoint_candidates_by_job_id'] = ** and provide the job_id of an analysis job. KPA will take the key points from the job's result automatically. We will use this parameter and provide the *kpa_2016_job_id* we saved in advance."
456 | ]
457 | },
458 | {
459 | "cell_type": "code",
460 | "execution_count": null,
461 | "id": "f1b4a120",
462 | "metadata": {},
463 | "outputs": [],
464 | "source": [
465 | "run_params = {'sentence_to_multiple_kps': True,\n",
466 | " 'keypoint_candidates_by_job_id': kpa_2016_job_id, 'n_top_kps': 100}\n",
467 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)\n",
468 | "kpa_result_2016_2017 = future.get_result(high_verbosity=True, polling_timout_secs=30)"
469 | ]
470 | },
471 | {
472 | "cell_type": "code",
473 | "execution_count": null,
474 | "id": "b5a0af0c",
475 | "metadata": {},
476 | "outputs": [],
477 | "source": [
478 | "KpAnalysisUtils.write_result_to_csv(kpa_result_2016_2017, 'austin_survey_2016_2017_kpa_results.csv')\n",
479 | "KpAnalysisUtils.compare_results(kpa_result_2016, kpa_result_2016_2017, '2016', '2016 + 2017')"
480 | ]
481 | },
482 | {
483 | "cell_type": "markdown",
484 | "id": "4a8412e2",
485 | "metadata": {},
486 | "source": [
487 | "### 3.2 Run *Key Point Analysis* incrementaly on new data (2017 independantly)\n",
488 | "Using the **comments_ids** parameter in **start_kp_analysis_job** method, we can run over a subset of the comments in the domain. Let's do that and run an analysis over 2017 comments independantly. We will provide the key points from 2016 since we want to able to compare between them:"
489 | ]
490 | },
491 | {
492 | "cell_type": "code",
493 | "execution_count": null,
494 | "id": "d574a490",
495 | "metadata": {
496 | "tags": []
497 | },
498 | "outputs": [],
499 | "source": [
500 | "comments_ids = [comment['id'] for comment in comments_2017_sample]\n",
501 | "run_params = {'sentence_to_multiple_kps': True,\n",
502 | " 'keypoint_candidates_by_job_id': kpa_2016_job_id, 'n_top_kps': 100}\n",
503 | "future = keypoints_client.start_kp_analysis_job(comments_ids=comments_ids, domain=domain, run_params=run_params)\n",
504 | "kpa_result_2017 = future.get_result(high_verbosity=True, polling_timout_secs=30)\n",
505 | "\n",
506 | "KpAnalysisUtils.write_result_to_csv(kpa_result_2017, 'austin_survey_2017_kpa_results.csv')"
507 | ]
508 | },
509 | {
510 | "cell_type": "code",
511 | "execution_count": null,
512 | "id": "0f8daaef-3aac-43a0-bae7-7b3ef323e3ed",
513 | "metadata": {
514 | "tags": []
515 | },
516 | "outputs": [],
517 | "source": [
518 | "KpAnalysisUtils.compare_results(kpa_result_2016, kpa_result_2017, '2016', '2017')"
519 | ]
520 | },
521 | {
522 | "cell_type": "markdown",
523 | "id": "4aa5b2f6",
524 | "metadata": {},
525 | "source": [
526 | "Running over subsets of the data in the domain enable us to compare results between them (subsets can be data from different GEOs, different organizations, different users (e.g. promoters/detractors) etc')."
527 | ]
528 | },
529 | {
530 | "cell_type": "markdown",
531 | "id": "de996c7b",
532 | "metadata": {
533 | "tags": []
534 | },
535 | "source": [
536 | "## 4. Run *Key Point Analysis* on each stance separately\n",
537 | "In many use-cases (surveys, customer feedback, etc') the comments have positive and/or negative stance, and it is usful to create a KPA analysis on each stance seperatly. Most stance detection models don't perfome too well on survey data (also costumer feedbacks etc') since the comments tend to have many \"suggestions\" in them, and the suggestions tend to apear positive to the model while the user suggests to improve something that needs improvement.\n",
538 | "For that end we trained a stance-model that handles suggestions well and labels each sentence as 'Positive', 'Negative', 'Neutral' and 'Suggestion'. We usually treat Suggestions as negatives and run two separate analysis, first over 'Positive' sentences and second over 'Negative' and 'Suggestions' sentences.\n",
539 | "\n",
540 | "This has the following advantages:\n",
541 | "* Creates a separate positive/negative summary that shows clearly what works well and what needs to be improved.\n",
542 | "* Filters-out neutral sentences that usually don't contain valuable information.\n",
543 | "* Helps the matching model avoid stance mistakes (matching a positive sentence to a negative key point and vice-versa).\n",
544 | "\n",
545 | "Lets run again, over the Austin survey dataset, but this time create two seperate KPA analyses (positive and negative). We will first need to create a new domain and add the domain_param **do_stance_analysis**."
546 | ]
547 | },
548 | {
549 | "cell_type": "code",
550 | "execution_count": null,
551 | "id": "cd950d77",
552 | "metadata": {
553 | "tags": []
554 | },
555 | "outputs": [],
556 | "source": [
557 | "domain = 'austin_demo_two_stances'\n",
558 | "domain_params = {'do_stance_analysis': True}\n",
559 | "KpAnalysisUtils.delete_domain_ignore_doesnt_exist(client=keypoints_client, domain=domain)\n",
560 | "keypoints_client.create_domain(domain=domain, domain_params=domain_params)"
561 | ]
562 | },
563 | {
564 | "cell_type": "markdown",
565 | "id": "3f04dd7e",
566 | "metadata": {},
567 | "source": [
568 | "Let's upload all 2016 comments to the new domain and wait for them to be processed. This time the sentences' stance is also calculated."
569 | ]
570 | },
571 | {
572 | "cell_type": "code",
573 | "execution_count": null,
574 | "id": "5bbbd3ed",
575 | "metadata": {},
576 | "outputs": [],
577 | "source": [
578 | "comments_texts = [comment['text'] for comment in comments_2016]\n",
579 | "comments_ids = [comment['id'] for comment in comments_2016]\n",
580 | "keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)\n",
581 | "keypoints_client.wait_till_all_comments_are_processed(domain=domain)"
582 | ]
583 | },
584 | {
585 | "cell_type": "markdown",
586 | "id": "46bd699f",
587 | "metadata": {},
588 | "source": [
589 | "We can download the processed sentences and save them into a csv if we want to examine the processed data."
590 | ]
591 | },
592 | {
593 | "cell_type": "code",
594 | "execution_count": null,
595 | "id": "61f97380",
596 | "metadata": {},
597 | "outputs": [],
598 | "source": [
599 | "sentences = keypoints_client.get_sentences_for_domain(domain=domain)\n",
600 | "KpAnalysisUtils.write_sentences_to_csv(sentences, f'{domain}_sentences.csv')"
601 | ]
602 | },
603 | {
604 | "cell_type": "markdown",
605 | "id": "1eae7368",
606 | "metadata": {},
607 | "source": [
608 | "And now, run two analyses, one over the positive sentences and one over the negative + suggestions."
609 | ]
610 | },
611 | {
612 | "cell_type": "code",
613 | "execution_count": null,
614 | "id": "11b6e4a3",
615 | "metadata": {},
616 | "outputs": [],
617 | "source": [
618 | "run_params = {'sentence_to_multiple_kps': True, \"n_top_kps\":100}\n",
619 | "run_params['stances_to_run'] = ['pos']\n",
620 | "run_params['stances_threshold'] = 0.5\n",
621 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params)\n",
622 | "kpa_pos_result = future.get_result(high_verbosity=True, polling_timout_secs=30)\n",
623 | "KpAnalysisUtils.print_result(kpa_pos_result, n_sentences_per_kp=2, title='Random sample positives')"
624 | ]
625 | },
626 | {
627 | "cell_type": "markdown",
628 | "id": "52e9d628",
629 | "metadata": {},
630 | "source": [
631 | "As in many surveys, most comments are negative/suggestions therefore the positive analysis is relativly limited. Let's see how the negative analysis goes."
632 | ]
633 | },
634 | {
635 | "cell_type": "code",
636 | "execution_count": null,
637 | "id": "160cbb3e",
638 | "metadata": {},
639 | "outputs": [],
640 | "source": [
641 | "run_params['stances_to_run'] = ['neg', 'sug']\n",
642 | "run_params['stances_threshold'] = 0.5\n",
643 | "future = keypoints_client.start_kp_analysis_job(domain=domain, run_params=run_params, comments_ids=comments_ids)\n",
644 | "kpa_neg_result = future.get_result(high_verbosity=True, polling_timout_secs=30)"
645 | ]
646 | },
647 | {
648 | "cell_type": "markdown",
649 | "id": "8121d10b",
650 | "metadata": {},
651 | "source": [
652 | "Lets print the results:"
653 | ]
654 | },
655 | {
656 | "cell_type": "code",
657 | "execution_count": null,
658 | "id": "d8015b3b",
659 | "metadata": {},
660 | "outputs": [],
661 | "source": [
662 | "KpAnalysisUtils.print_result(kpa_neg_result, n_sentences_per_kp=2, title='Random sample negatives')"
663 | ]
664 | },
665 | {
666 | "cell_type": "markdown",
667 | "id": "6c1e4a64",
668 | "metadata": {},
669 | "source": [
670 | "Reaching a nice 67% coverage, most of the sentences are matched to the 100 automatically extracted key points.\n",
671 | "\n",
672 | "We can increase the stances_threshold when we want to run over less sentences with a stronger stance. This is useful when we have a large dataset with many less-relevant sentences and we want to filter them out.\n",
673 | "\n",
674 | "We can mark the stance in the results:"
675 | ]
676 | },
677 | {
678 | "cell_type": "code",
679 | "execution_count": null,
680 | "id": "5b27b433",
681 | "metadata": {},
682 | "outputs": [],
683 | "source": [
684 | "kpa_pos_result = KpAnalysisUtils.set_stance_to_result(kpa_pos_result, 'pos')\n",
685 | "kpa_neg_result = KpAnalysisUtils.set_stance_to_result(kpa_neg_result, 'neg')"
686 | ]
687 | },
688 | {
689 | "cell_type": "markdown",
690 | "id": "28d9976f",
691 | "metadata": {},
692 | "source": [
693 | "And save the results (both pos/neg seperatly and merged) and create key points graphs' data files as we did before"
694 | ]
695 | },
696 | {
697 | "cell_type": "code",
698 | "execution_count": null,
699 | "id": "9c03646d",
700 | "metadata": {},
701 | "outputs": [],
702 | "source": [
703 | "pos_result_file = 'austin_survey_2016_pro_kpa_results.csv'\n",
704 | "KpAnalysisUtils.write_result_to_csv(kpa_pos_result, pos_result_file)\n",
705 | "KpAnalysisUtils.generate_graphs_and_textual_summary(pos_result_file)\n",
706 | "\n",
707 | "neg_result_file = 'austin_survey_2016_neg_kpa_results.csv'\n",
708 | "KpAnalysisUtils.write_result_to_csv(kpa_neg_result, neg_result_file)\n",
709 | "KpAnalysisUtils.generate_graphs_and_textual_summary(neg_result_file)\n",
710 | "\n",
711 | "kpa_merged_result = KpAnalysisUtils.merge_two_results(kpa_pos_result, kpa_neg_result)\n",
712 | "merged_result_file = 'austin_survey_2016_merged_kpa_results.csv'\n",
713 | "KpAnalysisUtils.write_result_to_csv(kpa_merged_result, merged_result_file)\n",
714 | "KpAnalysisUtils.generate_graphs_and_textual_summary(merged_result_file)"
715 | ]
716 | },
717 | {
718 | "cell_type": "markdown",
719 | "id": "b2922bee",
720 | "metadata": {},
721 | "source": [
722 | "We can also use the incremental approach when running on each stance seperatly. We will need to provide the job_id of the positive analysis of 2016 when running on the positive sentences of 2016 + 2017 and the job_id of negative analysis of 2016 when running on the negative sentences of 2016 + 2017, but for simplicity reasons, we didn't combine the features in this tutorial."
723 | ]
724 | },
725 | {
726 | "cell_type": "markdown",
727 | "id": "9fd476d4",
728 | "metadata": {},
729 | "source": [
730 | "## 5. Cleanup\n",
731 | "If you finished the tutorial and no longer need the domains and the results, cleaning up is always advised:"
732 | ]
733 | },
734 | {
735 | "cell_type": "code",
736 | "execution_count": null,
737 | "id": "27faf5a8",
738 | "metadata": {},
739 | "outputs": [],
740 | "source": [
741 | "KpAnalysisUtils.delete_domain_ignore_doesnt_exist(client=keypoints_client, domain='austin_demo')\n",
742 | "KpAnalysisUtils.delete_domain_ignore_doesnt_exist(client=keypoints_client, domain='austin_demo_two_stances')"
743 | ]
744 | },
745 | {
746 | "cell_type": "markdown",
747 | "id": "a8dbc94e",
748 | "metadata": {},
749 | "source": [
750 | "## 6. Conclusion\n",
751 | "In this tutorial, we showed how to use the *Key Point Analysis* service, and how it provides detailed insights over survey data right out of the box - significantly reducing the effort required by a data scientist to analyze the data. We also demonstrated key *key point analysis* features such as how to modify the analysis parameters and increase coverage, how to use the stance-model and create per-stance results, how to create *key points graph* and further improve the quality and the clarity of the results, and how to incrementally add new data.\n",
752 | "\n",
753 | "Feel free to contact us for questions or assistance: *yoavka@il.ibm.com*"
754 | ]
755 | }
756 | ],
757 | "metadata": {
758 | "kernelspec": {
759 | "display_name": "Python 3 (ipykernel)",
760 | "language": "python",
761 | "name": "python3"
762 | },
763 | "language_info": {
764 | "codemirror_mode": {
765 | "name": "ipython",
766 | "version": 3
767 | },
768 | "file_extension": ".py",
769 | "mimetype": "text/x-python",
770 | "name": "python",
771 | "nbconvert_exporter": "python",
772 | "pygments_lexer": "ipython3",
773 | "version": "3.8.16"
774 | }
775 | },
776 | "nbformat": 4,
777 | "nbformat_minor": 5
778 | }
--------------------------------------------------------------------------------
/survey_usecase/legacy_sdk/run_warmup_batches.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | for (( COUNTER=0; COUNTER<150; COUNTER+=10 )); do
4 | bash ./warm_up_cache_eap_tutorial.sh $COUNTER 10 &
5 | done
6 |
--------------------------------------------------------------------------------
/survey_usecase/legacy_sdk/warm_up_cache_eap_tutorial.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | ApiKeysArray=("" "")
3 |
4 | # $1 index to start
5 | # $2 how many to take
6 | ApiKeysArray=("${ApiKeysArray[@]:$1:$2}")
7 |
8 | for val1 in ${ApiKeysArray[*]}; do
9 | echo 'running on api_key:' $val1
10 | env DEBATER_API_KEY=$val1 jupyter nbconvert --to notebook --ExecutePreprocessor.kernel_name=kpa_env --execute austin_demo_with_answers.ipynb
11 | done
12 |
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/Changelog.md:
--------------------------------------------------------------------------------
1 | # Changelog
2 |
3 | All notable changes to the KPS service starting from 2023-June-26 will be documented in this file.
4 |
5 | ## 2023-June-26
6 | SDK version upgraded.
7 | The current backend version supports both new and old versions of the SDK, so for now you can keep using the old one.
8 | In a month (26.7) the legacy version will no longer be supported.
9 |
10 | Install the new version by:
11 | `pip install debater-python-api`
12 |
13 | Install the old version by:
14 | `pip install debater-python-api==4.3.2`
15 |
16 |
17 | A link to the full tutorial of the new SDK is available here: https://github.com/IBM/debater-eap-tutorial/blob/main/survey_usecase/new_sdk/kpa_quick_start_tutorial-with_results.ipynb
18 |
19 | A link to a lecture covering the tutorial is available here: https://ibm.box.com/s/61kwk9s8d7sm510kd2yz16g8hloufini
20 |
21 | The new SDK has all the capabilities of the legacy SDK and more, with many (but mostly straight forward) API changes.
22 | Our main goals were to simplify the API as much as possible while still allowing flexibility for advanced users, and to allow more analysis options over the results.
23 |
24 | The main changes are:
25 | * KPA is now called KPS.
26 | * The results are now kept as a KpsResult object, that can be saved and loaded to a json file, and allows for more analysis options. The same reports as before can be generated from this object, and they are organized better and contain more information.
27 | * The run_params and domain_params lists have been modified and shortened. We wanted to let you decide on the final required results, without having to deal with guessing numerical thresholds and other internal parameters which are now tuned internally. The updated lists are available in: https://github.com/IBM/debater-eap-tutorial/blob/main/survey_usecase/new_sdk/kps_parameters.pdf .
28 | * Stance handling is different and simpler: we’re now supporting two stances: Pro (positive sentences) and Con (negative sentences and suggestion). It’s also possible to run without stance or to let the system to run on each stance separately and provide the merged results. You can still see the category of each sentence (pos/neg/sug/neut) in the full results.
29 | * Supporting different running modes: by default, the system now runs in a synchronous manner. It’s also possible to run in an asynchronous manner as before, or to run end to end by only providing the comments (without caching and customization).
30 |
31 |
32 |
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/README.md:
--------------------------------------------------------------------------------
1 | # Key Point Summarization (KPS) tutorials
2 | ## Introduction
3 | When you have a large collection of texts representing people’s opinions (such as product reviews, survey answers or posts from social media), it is difficult to understand the key issues that come up in the data. Going over thousands of comments is prohibitively expensive. Existing automated approaches are often limited to identifying recurring phrases or concepts and the overall sentiment toward them, but do not provide detailed or actionable insights.
4 |
5 | In this tutorial you will gain hands-on experience in using Key Point Summarization for analyzing and deriving insights from open-ended answers. The data we will use is a community survey conducted in the city of Austin in the years 2016 and 2017.
6 |
7 | ## Prerequisites
8 | To follow this tutorial, you first need to receive credentials for KPS APIs by sending a request email to: `project.debater@il.ibm.com`
9 | We also assume you have conda package managment system (https://docs.conda.io/en/latest/) available and basic knowledge in Python.
10 |
11 |
12 | ## Estimated time
13 | It should take about 1 hour to complete the `survey_usecase` tutorial.
14 |
15 | ## Environment setup
16 | In order to run the tutorials, you need an Python Anaconda environment with the Early Access Program SDK installed on it and a Jupyter notebook.
17 |
18 | * Create a conda env:
19 | `conda create --name python=3.11`
20 |
21 | * Activate env:
22 | `conda activate `
23 |
24 | * Install KPS's Python SDK:
25 | `pip install debater-python-api`
26 |
27 | * Install Jupyter Notebook:
28 | `pip install notebook`
29 |
30 | * Get an API key for KPS by sending an email to:
31 | `project.debater@il.ibm.com`
32 |
33 |
34 | ## Run the tutorial
35 |
36 | * Clone the tutorial repository:
37 | `git clone https://github.com/IBM/debater-eap-tutorial.git`
38 |
39 | * Run Jupyter Notebook (use the api-key from the site):
40 | `env KPS_API_KEY= jupyter notebook`
41 |
42 |
43 | * Open [kpa_quick_start_tutorial.ipynb](kpa_quick_start_tutorial.ipynb) notebook for a simple quick-start tutorial or [kpa_quick_start_tutorial-with_results.ipynb](kpa_quick_start_tutorial-with_results.ipynb) for a version with results.
44 |
45 | * The tutorial is self-explanatory, and demonstrates how to use Key Point Summarization and its key features.
46 |
47 | * Feel free to contact us if you face problems or have questions at:
`project.debater@il.ibm.com`
48 |
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/kpa_quick_start_tutorial.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "9a14582c-65dc-4d0b-a7b9-4d030844fd23",
6 | "metadata": {
7 | "tags": []
8 | },
9 | "source": [
10 | "# Using *Key Point Summarization* for analyzing and finding insights in a survey data \n",
11 | "\n",
12 | "#### **Important Notice**: This tutorial describes the new SDK that was introduced on June 2023, starting from debater-python-api version 5.0.0. The tutorial of the legacy SDK is available [here](../legacy_sdk/kpa_quick_start_tutorial.ipynb).\n",
13 | "\n",
14 | "When you have a large collection of texts representing people’s opinions (such as product reviews, survey answers or social media), it is difficult to understand the key issues that come up in the data. Going over thousands of comments is prohibitively expensive. Existing automated approaches are often limited to identifying recurring phrases or concepts and the overall sentiment toward them, but do not provide detailed or actionable insights. \n",
15 | "\n",
16 | "In this tutorial you will gain hands-on experience in using *Key Point Summarization* (KPS) for analyzing and deriving insights from open-ended comments. \n",
17 | "\n",
18 | "The data we will use is [a community survey conducted in the city of Austin](https://data.austintexas.gov/dataset/Community-Survey/s2py-ceb7). In this survey, the citizens of Austin were asked \"If there was ONE thing you could share with the Mayor regarding the City of Austin (any comment, suggestion, etc.), what would it be?\". "
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "id": "2ab28f81",
24 | "metadata": {
25 | "tags": []
26 | },
27 | "source": [
28 | "## 1. Initialization"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "id": "d85aed7e",
34 | "metadata": {},
35 | "source": [
36 | "### 1.1 Setup\n",
37 | "\n",
38 | "Let's first import all the required packages for this tutorial and initialize the *Key Point Summarization* client. The client prints information using a logger and a suitable verbosity level should be set. The client object is configured with an API key. To receive an API key please send an email to *Project.Debater@il.ibm.com* and we'll be happy to provide it. In the code below it is stored in the enviroment variable *KPS_API_KEY* (you may also modify the code and place the api-key directly)."
39 | ]
40 | },
41 | {
42 | "cell_type": "code",
43 | "execution_count": null,
44 | "id": "510552af",
45 | "metadata": {},
46 | "outputs": [],
47 | "source": [
48 | "from debater_python_api.api.clients.keypoints_client import KpsClient, KpsJobFuture\n",
49 | "from debater_python_api.api.clients.key_point_summarization.KpsResult import KpsResult\n",
50 | "import os\n",
51 | "import pandas as pd\n",
52 | "import json \n",
53 | "\n",
54 | "pd.set_option('display.max_rows', None)\n",
55 | "KpsClient.init_logger()\n",
56 | "api_key = os.environ['KPS_API_KEY']\n",
57 | "host = 'https://keypoint-matching-backend.debater.res.ibm.com'\n",
58 | "keypoints_client = KpsClient(api_key, host)"
59 | ]
60 | },
61 | {
62 | "cell_type": "markdown",
63 | "id": "71725a3a",
64 | "metadata": {
65 | "tags": []
66 | },
67 | "source": [
68 | "### 1.2 Read the data\n",
69 | "Let's read the data from *dataset_austin.csv* file, which holds the Austin survey dataset, and print the first comment."
70 | ]
71 | },
72 | {
73 | "cell_type": "code",
74 | "execution_count": null,
75 | "id": "68c6f083",
76 | "metadata": {},
77 | "outputs": [],
78 | "source": [
79 | "comments_df = pd.read_csv('./dataset_austin.csv')\n",
80 | "\n",
81 | "print(f'There are {len(comments_df)} comments in the dataset')\n",
82 | "print(dict(comments_df.iloc[0,:]))"
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "id": "25b55e5e",
88 | "metadata": {},
89 | "source": [
90 | "Each comment has a unique_id 'id', a 'text', a 'year' and a 'Council_District'. The *Key Point Summarization* service is able to run over hundreds of thousands of comments. However, to provide good user experience, the KPS evaluation service is limited to 1000 comments per run, each comment with up to 3000 chars. You may request to increase this limit if needed. \n",
91 | "\n",
92 | "We will choose a sample of 1000 comments from 2016 for our analysis."
93 | ]
94 | },
95 | {
96 | "cell_type": "code",
97 | "execution_count": null,
98 | "id": "fdb6777e",
99 | "metadata": {},
100 | "outputs": [],
101 | "source": [
102 | "comments_df = comments_df.dropna()\n",
103 | "comments_df = comments_df[comments_df.text.apply(lambda x: 0)**\n",
619 | "\n",
620 | "It is also possibe to stop all jobs in a domain, or even all jobs in all domains (this might be simpler since no job_id is required):\n",
621 | "\n",
622 | "* **keypoints_client.cancel_all_extraction_jobs_for_domain(\\)**\n",
623 | "* **keypoints_client.cancel_all_extraction_jobs_all_domains()**\n",
624 | "\n",
625 | "\n",
626 | "### 5.4 Fetching the results of a previous job\n",
627 | "If the program terminated unexpectedly after the job was sent, you can still fetch the results using:\n",
628 | "* **kps_result = keypoints_client.get_results_from_job_id(\\)**"
629 | ]
630 | },
631 | {
632 | "cell_type": "markdown",
633 | "id": "67ff92aa",
634 | "metadata": {
635 | "tags": []
636 | },
637 | "source": [
638 | "## 6. Comparative analysis with KPS\n",
639 | "\n",
640 | "The KPS service allows us to easily perform comparisons between subsets of our data and to perform trend analysis over data collected in different times. Let's explore two of these options: \n",
641 | "\n",
642 | "### 6.1 Compare comment subsets\n",
643 | "So far, we ran over all the sampled comments for 2016. Now, suppose that we want to perform the analysis over the same data by district, and compare the feedback of the residents of district 7 with the feedback of the residents of district 10. All we need is our previous KpsResult and the comment_ids of each subset:"
644 | ]
645 | },
646 | {
647 | "cell_type": "code",
648 | "execution_count": null,
649 | "id": "6d41202c-b5d6-452b-aa9c-86aa829b8a72",
650 | "metadata": {},
651 | "outputs": [],
652 | "source": [
653 | "comment_ids_district_7 = list(comments_2016_sample_df[comments_2016_sample_df[\"Council_District\"]==7][\"id\"].astype(str))\n",
654 | "comment_ids_district_10 = list(comments_2016_sample_df[comments_2016_sample_df[\"Council_District\"]==10][\"id\"].astype(str))"
655 | ]
656 | },
657 | {
658 | "cell_type": "markdown",
659 | "id": "7b60284d-a99f-47dc-a7c1-5c5101d83d67",
660 | "metadata": {},
661 | "source": [
662 | "Now, we can compare the full result and the result from each district, using the method **compare_with_comment_subsets()**.\n",
663 | "This method receives a dictionary with mappings from subset names to sets of comment ids, and performs the comparison between the full result and the subset results. For the full result and for each of the subsets, you can see the number and percentage of the comments that match each key point. Let's create the comparison df and print the top 20 key points and the summary row."
664 | ]
665 | },
666 | {
667 | "cell_type": "code",
668 | "execution_count": null,
669 | "id": "bb22305d-b84b-40b9-8982-6a6441a4d72f",
670 | "metadata": {},
671 | "outputs": [],
672 | "source": [
673 | "subsets_dict = {\"district_7\":comment_ids_district_7, \"district_10\":comment_ids_district_10}\n",
674 | "comparison_df = kps_result_2016_no_stance.compare_with_comment_subsets(subsets_dict)\n",
675 | "pd.concat([comparison_df.head(20),comparison_df.tail(1)])"
676 | ]
677 | },
678 | {
679 | "cell_type": "markdown",
680 | "id": "82659631-b225-4162-878f-e43eee3f46a3",
681 | "metadata": {},
682 | "source": [
683 | "From the percentage displayed in the comparison, it is apparent that the residents of district 10 are more concerned about the traffic issues in Austin, while the residents of district 7 care more about affordable housing.\n",
684 | "\n",
685 | "Using this method, we can compare results over subsets of the data in the same domain with a single KPS job. The subsets can be data from different GEOs, different organizations, different times, different users (e.g. promoters/detractors) etc.\n",
686 | "\n",
687 | "This has several advantages over running a separate job for each subset:\n",
688 | "- The service only needs to be called once.\n",
689 | "- We can easily compare the distribution of different subsets over the same list of key points.\n",
690 | "- KPS generates better results for larger datasets, so running over the full data allows to get reacher key points with better coverage. "
691 | ]
692 | },
693 | {
694 | "cell_type": "markdown",
695 | "id": "41727d63-8428-4014-bf1d-eac250627463",
696 | "metadata": {},
697 | "source": [
698 | "### 6.2 Run KPS incrementally \n",
699 | "\n",
700 | "A year passed, and we collect additional data (from 2017). We would like to analyze the new data and compare it to the data from 2016. \n",
701 | "We can upload all the comments from 2017 to the same domain (\"austin_demo\")."
702 | ]
703 | },
704 | {
705 | "cell_type": "code",
706 | "execution_count": null,
707 | "id": "3a5f9111",
708 | "metadata": {},
709 | "outputs": [],
710 | "source": [
711 | "comments_2017_df = comments_df[comments_df['year'] == 2017]\n",
712 | "\n",
713 | "sample_size = 1000\n",
714 | "comments_2017_sample_df = comments_2017_df.sample(n = sample_size, random_state = 1)\n",
715 | "\n",
716 | "domain = 'austin_demo'\n",
717 | "comments_texts_2017 = list(comments_2017_sample_df['text'])\n",
718 | "comments_ids_2017 = list(comments_2017_sample_df['id'].astype(str))\n",
719 | "keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids_2017, comments_texts=comments_texts_2017)"
720 | ]
721 | },
722 | {
723 | "cell_type": "markdown",
724 | "id": "69cdd33f",
725 | "metadata": {},
726 | "source": [
727 | "We can now run a new summarization job over all the data in the domain, as we did before, and automatically extract new key points. Then we can use the **compare_with_comment_subsets** method to compare between the data subsets, as we learned in the previous section. We can assume that some key points will be identical to the key points extracted from the 2016 data, some will be similar and some key points will be new.\n",
728 | "\n",
729 | "A better option is to run a new summarization on the comments from 2017, but provide the keypoints from the 2016 summarization and let *Key Point Summarization* add new key points from the 2017 data if there are such. One benefit of this approach is that the new result will mostly use the 2016 key points and will be consistent with our previous results. Another major benefit for this approach is run time. 2016 data was already analyzed with these key points and now we only need to process the new data. The 2016 key points can be provided via the: **run_param['key_point_candidates'] = [...]** parameter, passing a list of strings, or we can use: **run_param['key_point_candidates_by_job_ids'] = [\\,...]** and provide a list of previous job_ids. KPS will take the key points from the jobs' result automatically.\n",
730 | "\n",
731 | "For simplicity, we'll run over the result without stance analysis. We can also use the incremental approach when running on both stances: we will need to provide the job_id of the positive summarization of 2016 in the run_params_pro and the job_id of negative summarization of 2016 in the run_params_con when running on the data from 2017.\n",
732 | "\n",
733 | "First, let's extract the job id from the result:"
734 | ]
735 | },
736 | {
737 | "cell_type": "code",
738 | "execution_count": null,
739 | "id": "8630e28e-438a-4af0-809a-90d11a16932d",
740 | "metadata": {},
741 | "outputs": [],
742 | "source": [
743 | "stance_to_job_id = kps_result_2016_no_stance.get_stance_to_job_id()\n",
744 | "print(stance_to_job_id)\n",
745 | "job_id_2016_no_stance = stance_to_job_id[\"no-stance\"]"
746 | ]
747 | },
748 | {
749 | "cell_type": "markdown",
750 | "id": "4a8412e2",
751 | "metadata": {},
752 | "source": [
753 | "We can use the **comments_ids** parameter in the **run_kps_job** method, to run on a subset of the comments in the domain. Let's do that and run summarization over the comments from 2017 independantly. We will provide the key points from 2016 as candidates, since we want to able to compare between the the two results:"
754 | ]
755 | },
756 | {
757 | "cell_type": "code",
758 | "execution_count": null,
759 | "id": "d574a490",
760 | "metadata": {
761 | "tags": []
762 | },
763 | "outputs": [],
764 | "source": [
765 | "run_params = {'key_point_candidates_by_job_ids': [job_id_2016_no_stance]}\n",
766 | "kps_result_2017_no_stance = keypoints_client.run_kps_job(domain, run_params=run_params, comments_ids = comments_ids_2017)"
767 | ]
768 | },
769 | {
770 | "cell_type": "markdown",
771 | "id": "c6c11729-a19c-41ca-9000-ec5c51a41bc7",
772 | "metadata": {},
773 | "source": [
774 | "Now, we can compare the kps_result_2016_no_stance with the kps_result_2017_no_stance using the **compare_with_other_results** method. This method receives the title for the current result and a dictionary mapping from results name to KpsResult objects, and returns the comparison table. If the comparison is with a single other result, the change percent is also displayed. "
775 | ]
776 | },
777 | {
778 | "cell_type": "code",
779 | "execution_count": null,
780 | "id": "0f8daaef-3aac-43a0-bae7-7b3ef323e3ed",
781 | "metadata": {
782 | "tags": []
783 | },
784 | "outputs": [],
785 | "source": [
786 | "other_results_dict = {\"2017\": kps_result_2017_no_stance}\n",
787 | "comparison_df = kps_result_2016_no_stance.compare_with_other_results(this_title=\"2016\", other_results_dict=other_results_dict)\n",
788 | "comparison_df"
789 | ]
790 | },
791 | {
792 | "cell_type": "markdown",
793 | "id": "eb17d963-a2d0-461d-a8fa-82a0d62a82b9",
794 | "metadata": {},
795 | "source": [
796 | "In the first 54 rows we can see the key points from 2016 applied to the data from 2017. Then, key points from 2017 that are not covered by the 2016 data are added, e.g. *\"Focus on basic services\"* and *\"Austin is not managing growth well.\"*."
797 | ]
798 | },
799 | {
800 | "cell_type": "markdown",
801 | "id": "5d82f0f0-e408-4967-be7b-cb323e56a3fd",
802 | "metadata": {},
803 | "source": [
804 | "## 7. Running KPS asynchronously\n",
805 | "\n",
806 | "It is also possible to upload comments and run KPS jobs asynchronously. This can be useful when you want to start several jobs simultaneously, and then later collect the results.\n",
807 | "\n",
808 | "Let's create a new domain for sake of the demonstaration. Note that this is not required, as asynchronous and synchronous calls can be used on the same doamin."
809 | ]
810 | },
811 | {
812 | "cell_type": "code",
813 | "execution_count": null,
814 | "id": "70882ed7-8360-470a-9aa7-a2328b72e199",
815 | "metadata": {},
816 | "outputs": [],
817 | "source": [
818 | "domain = 'austin_demo_async'\n",
819 | "keypoints_client.delete_domain_cannot_be_undone(domain=domain)\n",
820 | "keypoints_client.create_domain(domain=domain, domain_params={})"
821 | ]
822 | },
823 | {
824 | "cell_type": "markdown",
825 | "id": "e2053676-6931-4977-aa26-9adae6a17dfc",
826 | "metadata": {},
827 | "source": [
828 | "### 7.1 Uploading comments asynchronously\n",
829 | "\n",
830 | "In order to start loading comments, use the *upload_comments_async* method: "
831 | ]
832 | },
833 | {
834 | "cell_type": "code",
835 | "execution_count": null,
836 | "id": "756ae827-73b5-43bd-87ad-185019342cc1",
837 | "metadata": {},
838 | "outputs": [],
839 | "source": [
840 | "comments_texts = list(comments_2016_sample_df['text'])\n",
841 | "comments_ids = list(comments_2016_sample_df['id'].astype(str))\n",
842 | "keypoints_client.upload_comments_async(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)"
843 | ]
844 | },
845 | {
846 | "cell_type": "markdown",
847 | "id": "9fd476d4",
848 | "metadata": {},
849 | "source": [
850 | "The method uploads the comments and returns immediately. We must wait until all comments finish processing before starting a KPS job. This can be checked via the **are_all_comments_processed(domain=domain)** method, which prints the upload status and returns True when the domain is ready for running jobs:"
851 | ]
852 | },
853 | {
854 | "cell_type": "code",
855 | "execution_count": null,
856 | "id": "ea4e2b15-9631-496f-acfa-43677e50cc86",
857 | "metadata": {},
858 | "outputs": [],
859 | "source": [
860 | "print(keypoints_client.are_all_comments_processed(domain))"
861 | ]
862 | },
863 | {
864 | "cell_type": "markdown",
865 | "id": "862fe93f-ba26-447d-ab67-b2c8c07700d3",
866 | "metadata": {},
867 | "source": [
868 | "You can also use the **wait_till_all_comments_are_processed(domain=domain)** method, that returns only after the comments are processed:"
869 | ]
870 | },
871 | {
872 | "cell_type": "code",
873 | "execution_count": null,
874 | "id": "53ea81e5-1380-44df-adce-fd112e527e33",
875 | "metadata": {},
876 | "outputs": [],
877 | "source": [
878 | "keypoints_client.wait_till_all_comments_are_processed(domain=domain) "
879 | ]
880 | },
881 | {
882 | "cell_type": "markdown",
883 | "id": "caef34d0-0fa9-4858-93ba-63c00a92e543",
884 | "metadata": {},
885 | "source": [
886 | "### 7.2 Running KPS jobs asynchronously\n",
887 | "\n",
888 | "In order to start a job asynchronously, use the **run_kps_job_async** method. This method receives the same arguments as **run_kps_job**, but returns immediately after the job is sent to the server, returning a future object:"
889 | ]
890 | },
891 | {
892 | "cell_type": "code",
893 | "execution_count": null,
894 | "id": "9a784aab-d51b-48e5-8f24-69656b637420",
895 | "metadata": {},
896 | "outputs": [],
897 | "source": [
898 | "future = keypoints_client.run_kps_job_async(domain=domain)"
899 | ]
900 | },
901 | {
902 | "cell_type": "markdown",
903 | "id": "fed9c0ac-1040-451a-b375-4f0c3a49bb81",
904 | "metadata": {},
905 | "source": [
906 | "Use the returned future and wait till results are available using the **kps_result = future.get_result()** method. The method waits for the job to finish and eventually returns the result."
907 | ]
908 | },
909 | {
910 | "cell_type": "code",
911 | "execution_count": null,
912 | "id": "9260613b-1204-446b-9d37-2555c333e89b",
913 | "metadata": {},
914 | "outputs": [],
915 | "source": [
916 | "kps_result_async = future.get_result(high_verbosity=True)"
917 | ]
918 | },
919 | {
920 | "cell_type": "markdown",
921 | "id": "44d2b13b-9b8e-46e0-adf0-13265c113628",
922 | "metadata": {},
923 | "source": [
924 | "The future object can also be used to obtain the job_id, via the **future.get_job_id()** method.\n",
925 | "\n",
926 | "To generate a merged pro and con KpsResult, generate a separate pro_result and con_result using the above flow, and then use the method **KpsResult.get_merged_pro_con_results(pro_result, con_result)**. It's important to only merge pro and con results that were obtained over the same domain and using the same set of comments, otherwise an error will be raised."
927 | ]
928 | },
929 | {
930 | "cell_type": "code",
931 | "execution_count": null,
932 | "id": "f5aaafc9-57cf-4153-8c07-8e9adaa66732",
933 | "metadata": {},
934 | "outputs": [],
935 | "source": [
936 | "future_pro = keypoints_client.run_kps_job_async(domain, stance=\"pro\")\n",
937 | "future_con = keypoints_client.run_kps_job_async(domain, stance=\"con\")\n",
938 | "result_pro = future_pro.get_result()\n",
939 | "result_con = future_con.get_result()\n",
940 | "result_async_merged = KpsResult.get_merged_pro_con_results(pro_result=result_pro, con_result=result_con)"
941 | ]
942 | },
943 | {
944 | "cell_type": "markdown",
945 | "id": "00698571-ebbe-4b82-9472-a680de9f1ae4",
946 | "metadata": {},
947 | "source": [
948 | "## 8. Cleanup\n",
949 | "Once you finished the tutorial and no longer need the domains and the results, cleaning up is always advised:"
950 | ]
951 | },
952 | {
953 | "cell_type": "code",
954 | "execution_count": null,
955 | "id": "27faf5a8",
956 | "metadata": {},
957 | "outputs": [],
958 | "source": [
959 | "keypoints_client.delete_domain_cannot_be_undone(domain='austin_demo')\n",
960 | "keypoints_client.delete_domain_cannot_be_undone(domain='austin_demo_async')"
961 | ]
962 | },
963 | {
964 | "cell_type": "markdown",
965 | "id": "a8dbc94e",
966 | "metadata": {},
967 | "source": [
968 | "## 9. Conclusion\n",
969 | "In this tutorial, we showed how to use the *Key Point Summarization* service, and how it provides detailed insights over survey data right out of the box - significantly reducing the effort required by a data scientist to analyze the data. We also demonstrated key features such as modifying the summarization parameters and increasing coverage, using the stance model and creating per-stance results, incrementally adding new data and comparing different subsets of the data, and running KPS asynchronously.\n",
970 | "\n",
971 | "Feel free to contact us for questions or assistance: *Project.Debater@il.ibm.com*"
972 | ]
973 | },
974 | {
975 | "cell_type": "code",
976 | "execution_count": null,
977 | "id": "6bbb220b-1fc6-453a-9c00-3c6a37c334b7",
978 | "metadata": {},
979 | "outputs": [],
980 | "source": []
981 | }
982 | ],
983 | "metadata": {
984 | "kernelspec": {
985 | "display_name": "Python 3 (ipykernel)",
986 | "language": "python",
987 | "name": "python3"
988 | },
989 | "language_info": {
990 | "codemirror_mode": {
991 | "name": "ipython",
992 | "version": 3
993 | },
994 | "file_extension": ".py",
995 | "mimetype": "text/x-python",
996 | "name": "python",
997 | "nbconvert_exporter": "python",
998 | "pygments_lexer": "ipython3",
999 | "version": "3.8.16"
1000 | },
1001 | "toc-autonumbering": true,
1002 | "toc-showcode": false,
1003 | "toc-showmarkdowntxt": false,
1004 | "toc-showtags": true
1005 | },
1006 | "nbformat": 4,
1007 | "nbformat_minor": 5
1008 | }
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/kps_parameters.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/survey_usecase/new_sdk/kps_parameters.pdf
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/kps_results/kps_result_2016_hierarchical.docx:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/survey_usecase/new_sdk/kps_results/kps_result_2016_hierarchical.docx
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/kps_results/kps_result_2016_kps_summary.csv:
--------------------------------------------------------------------------------
1 | key_point,#comments,comments_coverage,#sentences,sentences_coverage,stance,kp_id,parent_id,n_comments_subtree
2 | Improve traffic flow.,169,0.169,185,0.10204081632653061,con,0,,220
3 | We really need to improve our public transportation.,153,0.153,167,0.09211252068394926,con,1,,153
4 | Improve affordable housing/living.,127,0.127,150,0.0827357970215113,con,2,,154
5 | PROPERTY TAXES ARE TOO HIGH.,83,0.083,96,0.052950910093767234,con,3,,110
6 | Please plan better for growth on our roadways!,78,0.078,85,0.04688361831218974,con,4,0,78
7 | Buying a home in S Austin is prohibitive.,56,0.056,62,0.03419746276889134,con,5,2,56
8 | driving in Austin is terrible,47,0.047,51,0.028130170987313845,con,6,0,47
9 | Water costs too much.,44,0.044,50,0.027578599007170437,con,7,3,52
10 | Don't let Austin become Houston with overdevelopment.,35,0.035,36,0.019856591285162713,con,11,,35
11 | Better pedestrian and biking lifestyle options.,34,0.034,36,0.019856591285162713,con,9,,34
12 | STOP WASTING MONEY.,33,0.033,41,0.022614451185879757,con,8,,35
13 | Stop the gentrification.,32,0.032,36,0.019856591285162713,con,10,2,32
14 | electric rates are too high,28,0.028,30,0.01654715940430226,con,12,7,28
15 | FOCUS ON SUSTAINABILITY INSTEAD OF GROWTH.,26,0.026,26,0.014340871483728626,con,13,,26
16 | Make utility changes more affordable for seniors.,24,0.024,25,0.013789299503585218,con,14,2,24
17 | THE HOMELESS POPULATION NEEDS MORE HELP/ATTENTION,22,0.022,23,0.0126861555432984,con,15,,32
18 | Clean up the homeless population downtown!,20,0.02,22,0.012134583563154992,con,16,15,20
19 | Fewer bicycles for safer roads.,18,0.018,18,0.009928295642581356,con,17,,18
20 | City Council members should represent all citizens.,16,0.016,17,0.009376723662437948,con,19,,16
21 | INCREASE POLICE FORCE.,16,0.016,17,0.009376723662437948,con,20,,16
22 | The permitting process has got to improve.,15,0.015,18,0.009928295642581356,con,18,,15
23 | PLEASE MORE PARKING SPACES.,14,0.014,16,0.00882515168229454,con,22,,14
24 | We need more street lights.,14,0.014,16,0.00882515168229454,con,21,,14
25 | CITY IS REGULATING FAR TOO MUCH.,14,0.014,15,0.00827357970215113,con,23,,14
26 | Quit adding bike lanes on roads,14,0.014,14,0.007722007722007722,con,24,,14
27 | Code issues are hurting the city.,13,0.013,13,0.007170435741864313,con,25,,13
28 | WE NEED BETTER FLOOD PLANNING.,13,0.013,13,0.007170435741864313,con,26,,13
29 | WE ARE WAY TOO CAR DEPENDENT.,11,0.011,12,0.006618863761720904,con,28,1,11
30 | Please keep grass mowed along sidewalks.,10,0.01,13,0.007170435741864313,con,27,,10
31 | Focus on efficient solutions.,9,0.009,9,0.004964147821290678,con,31,,9
32 | "Help our public schools, please.",9,0.009,9,0.004964147821290678,con,30,,9
33 | I think you're doing a great job!,8,0.008,8,0.00441257584114727,pro,34,,8
34 | Austin needs fields for sports!,8,0.008,8,0.00441257584114727,con,35,,8
35 | Growth needs to pay for itself.,8,0.008,8,0.00441257584114727,con,36,8,8
36 | More transparency about usage of tax money.,7,0.007,11,0.006067291781577496,con,29,,7
37 | Please increase curbside recycling services pickup frequency.,7,0.007,8,0.00441257584114727,con,32,,7
38 | Respect efforts for historic districts.,7,0.007,8,0.00441257584114727,con,33,,7
39 | Make our buses a desirable form of transportation.,7,0.007,7,0.003861003861003861,con,37,1,7
40 | The police are too reactive and authoritarian.,6,0.006,7,0.003861003861003861,con,38,,6
41 | Hotel room prices are outrageous on weekends.,6,0.006,6,0.003309431880860452,con,39,,6
42 | I don't feel safe going to work!,6,0.006,6,0.003309431880860452,con,41,,6
43 | More programs for senior citizens.,6,0.006,6,0.003309431880860452,con,40,,6
44 | "City services (water, streets, electric) are outstanding!!",5,0.005,5,0.0027578599007170436,pro,43,,5
45 | Embrace growth and innovation!,5,0.005,5,0.0027578599007170436,con,47,,5
46 | Glad Marc Ott is leaving - incompetent.,5,0.005,5,0.0027578599007170436,con,46,,5
47 | Police response time in unacceptably slow.,5,0.005,5,0.0027578599007170436,con,44,,5
48 | Stop sanctuary city program!,5,0.005,5,0.0027578599007170436,con,45,,5
49 | We need more schools in northwest Austin.,4,0.004,5,0.0027578599007170436,con,42,,4
50 | *total_con,881,0.881,1450,0.7997793712079426,,,,
51 | *matched_con,644,0.644,882,0.4864864864864865,,,,
52 | *total_pro,78,0.078,101,0.05570876999448428,,,,
53 | *matched_pro,12,0.012,12,0.006618863761720904,,,,
54 | *total,1000,1.0,1813,1.0,,,,
55 | *matched,651,0.651,894,0.4931053502482074,,,,
56 |
--------------------------------------------------------------------------------
/survey_usecase/new_sdk/run_keypoint_summarization_script.py:
--------------------------------------------------------------------------------
1 | # (C) Copyright IBM Corporation 2020-2022
2 | # LICENSE: Apache License 2.0 (Apache-2.0)
3 | # http://www.apache.org/licenses/LICENSE-2.0
4 |
5 | import logging
6 | import os
7 | from debater_python_api.api.clients.keypoints_client import KpsClient
8 | import pandas as pd
9 |
10 |
11 | def get_comments_ids_and_texts(file, comment_ids_column, comment_text_column):
12 | df = pd.read_csv(file)
13 | id_text = sorted(list(zip(df[comment_ids_column], df[comment_text_column])))
14 | id_text = [t for t in id_text if len(t[1]) < 3000] # comments must have at most 1000 chars
15 | comments_ids = [str(t[0]) for t in id_text]
16 | comments_texts = [str(t[1]) for t in id_text]
17 | return comments_ids, comments_texts
18 |
19 |
20 | if __name__ == '__main__':
21 | KpsClient.init_logger()
22 |
23 | # ======================= update params =======================
24 | api_key = ''
25 | host = 'https://keypoint-matching-backend.debater.res.ibm.com'
26 | input_file = ''
27 | comment_ids_column = ''
28 | comment_text_column = ''
29 | output_dir = '' # existing output directory
30 | job_name = '' # job mane to appear in output files names
31 | domain = ''
32 | # ======================= update params =======================
33 |
34 | # what to run
35 | restart = False # set to True for recreating the domain and uploading comments
36 | delete_domain = restart
37 | create_domain = restart
38 | upload_comments = restart # set to True for uploading more comments to an existing domain
39 | download_sentences = True
40 | get_report = False
41 | run_kps_job = True
42 | both_stances = True
43 |
44 | # how to run
45 | domain_params = {}
46 | run_params = {}
47 | if run_kps_job and both_stances:
48 | run_params_pro = run_params # change for customized pro run
49 | run_params_con = run_params # change for customized con run
50 |
51 | #########################
52 |
53 | keypoints_client = KpsClient(apikey=api_key, host=host)
54 |
55 | if get_report:
56 | KpsClient.print_report(keypoints_client.get_full_report())
57 |
58 | if delete_domain:
59 | answer = input(f'Are you sure you want to delete domain: {domain} (cannot be undone)? yes/no:')
60 | if answer == 'yes':
61 | keypoints_client.delete_domain_cannot_be_undone(domain)
62 |
63 | if create_domain:
64 | keypoints_client.create_domain(domain, domain_params)
65 |
66 | if upload_comments:
67 | comments_ids, comments_texts = get_comments_ids_and_texts(input_file, comment_ids_column, comment_text_column)
68 | keypoints_client.upload_comments(domain=domain, comments_ids=comments_ids, comments_texts=comments_texts)
69 |
70 | if download_sentences:
71 | sentences_df = keypoints_client.get_sentences_for_domain(domain)
72 | sentences_df.to_csv(os.path.join(output_dir, f'{job_name}_sentences.csv'))
73 |
74 | kps_result = None
75 | if run_kps_job:
76 | if both_stances:
77 | kps_result = keypoints_client.run_kps_job_both_stances(
78 | domain, run_params_con=run_params_con, run_params_pro=run_params_pro)
79 | else:
80 | kps_result = keypoints_client.run_kps_job(domain=domain, run_params=run_params)
81 |
82 | if kps_result:
83 | kps_result.save(os.path.join(output_dir, f'{job_name}.json'))
84 | kps_result.export_to_all_outputs(output_dir=output_dir, result_name=job_name)
85 |
--------------------------------------------------------------------------------
/tabs.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IBM/debater-eap-tutorial/f41b39399197cbe4f5b1579d81f96aaadaba1d35/tabs.png
--------------------------------------------------------------------------------