9 |
10 | Respresenting the Language of Thought
11 |
12 | [Hongyin Luo](https://luohongyin.github.io/) @ MIT *[hyluo at mit dot edu]*
13 |
14 |
9 |
10 | Respresenting the Language of Thought
11 |
12 | [Hongyin Luo](https://luohongyin.github.io/) @ MIT *[hyluo at mit dot edu]*
13 |
14 | 
275 |
276 | We did not use any Game of 24 example to prompt GPT-4, but it still gets the correct answer `(4 - 10) * (9 - 13)` with only sampling **one** output! This is more efficient than tree-of-thoughts.
277 |
278 | ### Code Completion
279 | The `LangPy` agent can also complete code that is not finished yet.
280 | 
281 | Run the code in the upper cell will complete the code in the lower cell, getting the following program
282 | ```python
283 | # Step 1: Import necessary built-in libraries
284 | import torch
285 | import torch.nn as nn
286 |
287 | # Step 2: Define necessary functions that generally solve this type of problem
288 | class MLP(nn.Module):
289 | def __init__(self):
290 | super(MLP, self).__init__()
291 | self.layers = nn.Sequential(
292 | nn.Linear(10, 5),
293 | nn.ReLU(),
294 | nn.Linear(5, 2)
295 | )
296 |
297 | def forward(self, x):
298 | x = self.layers(x)
299 | return x
300 |
301 | # Step 3: Instantiate the defined class
302 | mlp = MLP()
303 |
304 | # Step 4: Print an answer in natural language.
305 | print(f"A multi-layer perceptron (MLP) has been defined. It consists of an input layer, a hidden layer with 5 neurons activated by ReLU function, and an output layer with 2 neurons. The input layer has 10 neurons, representing the dimension of the input vectors.")
306 | ```
307 | Note that both the code and comment in the generated cell can be edited for another completion, enabling more flexible and direct instructions.
308 |
309 | ### CSV File Processing
310 | As an early release, we provide basic CSV file processing ability. Running the following code,
311 | ```python
312 | clp.preview_data(file_name = 'example_data.csv', data_type = 'csv)
313 | ```
314 | a new code cell will be inserted:
315 | ```python
316 | # First three rows of the input file:
317 | # placeName,placeDcid,xDate,xValue-DifferenceRelativeToBaseDate2006_Max_Temperature_RCP45,yDate,yValue-Percent_Person_WithCoronaryHeartDisease,xPopulation-Count_Person,yPopulation-Count_Person
318 | # "Autauga County, AL",geoId/01001,2050-06,-1.15811100000001,2020,6.3,59095,56145
319 | # "Baldwin County, AL",geoId/01003,2050-06,-1.073211,2020,5.9,239294,229287
320 | # ...
321 | file_name = 'example_data.csv'
322 | input_file = open(file_name)
323 | ```
324 | Running this code snipet, you can further analyze the data in the csv file using LangCode. For example, asking LangCode to generate code for visualization:
325 | ```python
326 | # hist_mode = 'all' allows LangCode to read previous cells
327 |
328 | clp.generate('Visualize the correlation of xData and yData of Alabama in the file.', hist_mode = 'all')
329 | ```
330 | The generated code and execution results are shown below.
331 | 
332 |
333 | ## Chat with Python Through Code Generation
334 | We currently offer three parameters for `LangPy.generate(instruction, mode = 'all', http = 'optional')`
335 | - `instruction [str]`: The input question / request in natural language.
336 | - `mode [str or int]`: Wether or note read previous code and markdown cells
337 | - `mode = 'all'`: Read all previous cells
338 | - `mode = 'single'`: Just read the instruction in the current cell
339 | - `mode = [int]`: The number of latest history cells to read
340 | - `http`: Wether allow the generated code to include http requests.
341 | - `http = 'optional'`: (Default) LangCode decides itself about sending HTTP requests or not.
342 | - `http = 'forbid'`: LangCode will try to avoid generated HTTP requesting code.
343 | - `http = 'force'`: LangCode will try to generated code that sends HTTP requests to solve the given question.
344 |
345 | ## Hirarchical Instruction Following Through Code Completion
346 | In the previous example we have shown,
347 |
348 | LangCode implemented a Multi-layer perceptron (MLP) with Pytorch using the following code
349 | ```python
350 | # Step 1: Import necessary built-in libraries
351 | import torch
352 | import torch.nn as nn
353 |
354 | # Step 2: Define necessary functions that generally solve this type of problem
355 | class MLP(nn.Module):
356 | def __init__(self):
357 | super(MLP, self).__init__()
358 | self.layers = nn.Sequential(
359 | nn.Linear(10, 5),
360 | nn.ReLU(),
361 | nn.Linear(5, 2)
362 | )
363 |
364 | def forward(self, x):
365 | x = self.layers(x)
366 | return x
367 |
368 | # Step 3: Instantiate the defined class
369 | mlp = MLP()
370 |
371 | # Step 4: Print an answer in natural language.
372 | print(f"A multi-layer perceptron (MLP) has been defined. It consists of an input layer, a hidden layer with 5 neurons activated by ReLU function, and an output layer with 2 neurons. The input layer has 10 neurons, representing the dimension of the input vectors.")
373 | ```
374 |
375 | Unlike ChatGPT that requires user to write a good instruction at once, LangCode allows adding multiple instructions in different places for one task. Editting the code for completion can guide model to generate different code. For example,
376 |
377 | ```python
378 | # Step 1: Import necessary built-in libraries
379 | import torch
380 | import torch.nn as nn
381 |
382 | # Step 2: Define a multi-layer perceptron that has three linear layers and uses tanh activation function.
383 | ```
384 | The code completion result would be
385 | ```python
386 | # Step 1: Import necessary built-in libraries
387 | import torch
388 | import torch.nn as nn
389 |
390 | # Step 2: Define a multi-layer perceptron that has three linear layers and uses tanh activation function.
391 | # Python program:
392 | class MultiLayerPerceptron(nn.Module):
393 | def __init__(self, input_size, hidden_size, output_size):
394 | super(MultiLayerPerceptron, self).__init__()
395 | self.layer1 = nn.Linear(input_size, hidden_size)
396 | self.layer2 = nn.Linear(hidden_size, hidden_size)
397 | self.layer3 = nn.Linear(hidden_size, output_size)
398 | self.tanh = nn.Tanh()
399 |
400 | def forward(self, x):
401 | x = self.tanh(self.layer1(x))
402 | x = self.tanh(self.layer2(x))
403 | x = self.layer3(x)
404 | return x
405 |
406 | # Step 3: Print an answer in natural language.
407 | print("A multi-layer perceptron is a type of artificial neural network. It has an input layer, one or more hidden layers, and an output layer. In each layer, the inputs are multiplied by weights, summed, and passed through an activation function. In this implementation, the activation function is the hyperbolic tangent function (tanh).")
408 | ```
409 |
410 | Besides the comments, you can also control the direction of the generated code by modifying the imported libraries. For example, changing `import torch` to `import tensorflow.keras as keras`. Your are encourged to explore more possibilities!
411 |
412 | ## Contact
413 |
414 | If there is any question, feel free to post an issue or contact Hongyin Luo at *hyluo [at] mit [dot] edu*.
415 |
--------------------------------------------------------------------------------
/documents/conda_jupyter.md:
--------------------------------------------------------------------------------
1 | # Anaconda and Jupyter notebook 6
2 | Follow the instruction of this document to setup conda and Jupyter notebook on your machine.
3 |
4 | ## Anaconda setup
5 |
6 | ### Installation
7 | 1. Download Miniconda from [official website](https://docs.conda.io/en/latest/miniconda.html)
8 | 2. Install Miniconda following [official instructions](https://conda.io/projects/conda/en/stable/user-guide/install/index.html)
9 |
10 | Make sure adding conda python to your `PATH`!
11 |
12 | ### Setup environment
13 | Run the following command to set up a conda environment with python and Jupyter 6.
14 | ```bash
15 | conda create -n myenv python=3.9 notebook=6.4.8
16 | ```
17 |
18 | Activate the environment:
19 | ```bash
20 | conda activate myenv
21 | ```
22 |
23 | Run Jupyter notebook:
24 | ```bash
25 | jupyter notebook # For localhost
26 | jupyter notebook --ip 0.0.0.0 # For public access
27 | ```
--------------------------------------------------------------------------------
/images/cpl_example.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/images/cpl_example.png
--------------------------------------------------------------------------------
/images/csv_example.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/images/csv_example.png
--------------------------------------------------------------------------------
/images/demo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/images/demo.png
--------------------------------------------------------------------------------
/images/game24.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/images/game24.png
--------------------------------------------------------------------------------
/langpy/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/langpy/__init__.py
--------------------------------------------------------------------------------
/langpy/auto.py:
--------------------------------------------------------------------------------
1 | def get_langpy(api_key='', platform='gpt', model='gpt-4'):
2 | try:
3 | import google.colab
4 | from .colab_lp import ColabLangPy
5 | autolp = ColabLangPy(api_key, platform=platform, model=model)
6 | except:
7 | from .jupyter_lp import JupyterLangPy
8 | autolp = JupyterLangPy(api_key, platform=platform, model=model)
9 | return autolp
10 |
--------------------------------------------------------------------------------
/langpy/base.py:
--------------------------------------------------------------------------------
1 | from .data_loader.csv_loader import CsvLoader
2 | from .data_loader.json_loader import JsonLoader
3 |
4 | _API_ENDPOINT = 'https://lang-py-522564686dd7.herokuapp.com/items/0'
5 | _SUPPORTED_PLATFORMS = {'gpt', 'palm'}
6 |
7 |
8 | class LangPy:
9 | def __init__(self, api_key='', platform='gpt', model='gpt-4'):
10 | if not api_key:
11 | raise ValueError("Empty api_key")
12 |
13 | if not model:
14 | raise ValueError("Empty model")
15 |
16 | if platform not in _SUPPORTED_PLATFORMS:
17 | supported_platforms = ", ".join(_SUPPORTED_PLATFORMS)
18 | raise ValueError(
19 | "Non-supported platform. Supported ones: {supported_platforms}")
20 |
21 | self.api_key = api_key
22 | self.api_end_point = _API_ENDPOINT
23 | self.platform = platform
24 | self.model = model
25 |
26 | def get_http_prompt(self, http):
27 | prompt = ''
28 | if http == 'forbid':
29 | prompt = f'Do not send HTTP requests.'
30 | elif http == 'force':
31 | prompt = f'Send an HTTP request to solve the problem.'
32 | elif http == 'optional':
33 | pass
34 | else:
35 | raise ValueError(
36 | f'http mode: {http} not supported. try [forbid | force | optional]!')
37 | return prompt
38 |
39 | def get_data_loader_prompt(self,
40 | notebook,
41 | data=None,
42 | file_name=None,
43 | data_type='csv', # csv or json
44 | read_mode='r',
45 | num_case=2):
46 | if data_type == 'csv':
47 | loader = CsvLoader(data, file_name, read_mode)
48 | elif data_type == 'json':
49 | loader = JsonLoader(data, file_name, read_mode)
50 | else:
51 | raise ValueError(
52 | f'data_type = {data_type} not supported. Please select from [csv | json]')
53 | return loader.get_prompt(notebook, num_case)
54 |
55 | def config_api_key(self):
56 | api_key = input('API key:')
57 | self.api_key = api_key
58 |
59 | def config_end_point(self):
60 | self.api_end_point = input('API endpoint:')
61 |
62 | def generate(self, instruction, hist_mode='all', http='optional'):
63 | pass
64 |
65 | def complete(self, instruction, hist_mode='all', http='optional'):
66 | pass
67 |
68 | def process_csv(self, file_name):
69 | pass
70 |
--------------------------------------------------------------------------------
/langpy/colab_lp.py:
--------------------------------------------------------------------------------
1 | import json
2 | import requests
3 | from .base import LangPy
4 | from google.colab import _message
5 |
6 |
7 | class ColabLangPy(LangPy):
8 | def __init__(self, api_key='', platform='gpt', model='gpt-4'):
9 | super().__init__(api_key=api_key, platform=platform, model=model)
10 | self._colab_list = []
11 | self._cur_idx = -1
12 |
13 | def format_cell(self, cell_tuple):
14 | cell_str, cell_type = cell_tuple
15 | if cell_type == 'code':
16 | return f'```\n{cell_str}\n```'
17 | else:
18 | return cell_str
19 |
20 | def post_code(self, code):
21 | _message.blocking_request(
22 | 'add_scratch_cell',
23 | request={'content': code, 'openInRightPane': True},
24 | timeout_sec=None
25 | )
26 |
27 | def update_idx_hist(self, instruction):
28 | colab_hist = _message.blocking_request('get_ipynb')['ipynb']['cells']
29 | self._code_list = [
30 | [''.join(x['source']), x['cell_type']] for x in colab_hist
31 | ]
32 |
33 | self._code_list = [
34 | self.format_cell(ct) for ct in self._code_list
35 | ]
36 |
37 | self._cur_idx = -1
38 | for i, code in enumerate(self._code_list):
39 | if instruction in code:
40 | self._cur_idx = i
41 | break
42 |
43 | def get_history_prompt(self, instruction, hist_mode):
44 | prompt = ''
45 | if hist_mode == 'all':
46 | previous_code = '\n\n'.join(self._code_list[:self._cur_idx])
47 | prompt = f'{previous_code}\n\n{instruction}'
48 | elif hist_mode == 'single':
49 | prompt = instruction
50 | elif isinstance(hist_mode, int):
51 | previous_code = '\n\n'.join(
52 | self._code_list[self._cur_idx - hist_mode: self._cur_idx])
53 | prompt = f'{previous_code}\n\n{instruction}'
54 | else:
55 | raise ValueError(
56 | f'hist_mode = {hist_mode} not supported. try [all | single | INTEGER].')
57 | return prompt
58 |
59 | def get_prompt(self, instruction, hist_mode, http):
60 | history_prompt = self.get_history_prompt(instruction, hist_mode)
61 | http_prompt = self.get_http_prompt(http)
62 | return f'{history_prompt} {http_prompt}'
63 |
64 | def get_code_of_thought(self, json_data):
65 | response = requests.put(self.api_end_point, json=json_data)
66 | if response.status_code != 200:
67 | raise SystemError(
68 | f'Failed to get valid response from server: {response.status_code}')
69 | return response.content
70 |
71 | def generate(self, instruction, hist_mode='single', http='optional'):
72 | self.update_idx_hist(instruction)
73 | response = self.get_code_of_thought(
74 | json_data={
75 | 'instruction': self.get_prompt(instruction, hist_mode, http),
76 | 'api_key_str': self.api_key,
77 | 'exist_code': 'none',
78 | 'platform': self.platform,
79 | 'model': self.model
80 | }
81 | )
82 | ans_str = json.loads(response)['output']
83 | self.post_code(ans_str)
84 |
85 | def complete(self, instruction, hist_mode='single', http='optional'):
86 | self.update_idx_hist(instruction)
87 | response = self.get_code_of_thought(
88 | json_data={
89 | 'instruction': self.get_prompt(instruction, hist_mode, http),
90 | 'api_key_str': self.api_key,
91 | 'exist_code': self._code_list[self._cur_idx + 1].replace('```', '').strip(),
92 | 'platform': self.platform,
93 | 'model': self.model
94 | }
95 | )
96 | ans_str = json.loads(response)['output']
97 | self.post_code(ans_str)
98 |
99 | def preview_data(
100 | self,
101 | data=None,
102 | file_name=None,
103 | data_type='csv', # csv or json
104 | read_mode='r',
105 | num_case=2):
106 | prompt = self.get_data_loader_prompt(
107 | 'colab', data, file_name, data_type, read_mode, num_case)
108 | self.post_code(prompt)
109 |
--------------------------------------------------------------------------------
/langpy/data_loader/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/luohongyin/LangCode/097f34119f52a8a8dfc868a4a03c7e2029e28e89/langpy/data_loader/__init__.py
--------------------------------------------------------------------------------
/langpy/data_loader/csv_loader.py:
--------------------------------------------------------------------------------
1 | from .loader import Loader
2 |
3 |
4 | class CsvLoader(Loader):
5 | def __init__(self, data=None, file_name=None, read_mode='r'):
6 | super().__init__(data, file_name, read_mode)
7 | if file_name is None:
8 | raise ValueError('CsvLoader must process a file. set file_name.')
9 |
10 | def file_prompt(self, num_row, content_str):
11 | f_prompt = f"""# First {num_row} rows of the input file:\nexample_data = '''\n{content_str}\n ...\n'''\nfile_name = '{self.file_name}' \ninput_file = open(file_name)"""
12 | return f_prompt
13 |
14 | def get_prompt(self, notebook='jupyter', num_case=2):
15 | num_row = num_case + 1
16 | content_list = open(self.file_name, self.read_mode).readlines()[
17 | :num_row]
18 | content_list = [x.strip() for x in content_list]
19 | content_str = '\n# '.join(content_list)
20 |
21 | prompt = self.file_prompt(num_row, content_str)
22 | return self.format_prompt(prompt, notebook)
23 |
--------------------------------------------------------------------------------
/langpy/data_loader/json_loader.py:
--------------------------------------------------------------------------------
1 | import json
2 | from .loader import Loader
3 |
4 |
5 | class JsonLoader(Loader):
6 | def __init__(self, data=None, file_name=None, read_mode='r'):
7 | super().__init__(data, file_name, read_mode)
8 | if data is None and file_name is None:
9 | raise ValueError('Must provide either `data` or `file_name`.')
10 |
11 | def file_prompt(self, num_case, content_str):
12 | f_prompt = f"""# First {num_case} cases of the Json data:\nexample_data = '''\n{content_str}\n...\n'''\nfile_name = '{self.file_name}' \ninput_file = open(file_name)"""
13 | return f_prompt
14 |
15 | def data_prompt(self, num_case, content_str):
16 | d_prompt = f"""# First {num_case} cases of the Json data:\nexample_data = '''\n{content_str}\n...\n'''"""
17 | return d_prompt
18 |
19 | def get_prompt(self, notebook='jupyter', num_case=2):
20 | if self.data is None:
21 | self.data = json.load(open(self.file_name, self.read_mode))
22 |
23 | example_list = self.data[:num_case]
24 | content_str = ',\n'.join([json.dumps(x, indent=4)
25 | for x in example_list])
26 |
27 | if self.file_name is not None:
28 | prompt = self.file_prompt(num_case, content_str)
29 | else:
30 | prompt = self.data_prompt(num_case, content_str)
31 |
32 | return self.format_prompt(prompt, notebook)
33 |
--------------------------------------------------------------------------------
/langpy/data_loader/loader.py:
--------------------------------------------------------------------------------
1 | class Loader:
2 | def __init__(self, data=None, file_name=None, read_mode='r'):
3 | self.data = data
4 | self.file_name = file_name
5 | self.read_mode = read_mode
6 |
7 | def format_prompt(self, prompt, notebook):
8 | if notebook == 'jupyter':
9 | prompt = prompt.replace('\n', '\\n').replace('\t', '\\t')
10 | elif notebook == 'colab':
11 | pass
12 | else:
13 | raise ValueError(
14 | f'Notebook = {notebook} note supported. Use [jupyter | colab]')
15 | return prompt
--------------------------------------------------------------------------------
/langpy/jupyter_lp.py:
--------------------------------------------------------------------------------
1 | import IPython
2 | from IPython.core.display import display, Javascript
3 | from .base import LangPy
4 |
5 |
6 | class JupyterLangPy(LangPy):
7 | def __init__(self, api_key='', platform='gpt', model='gpt-4'):
8 | super().__init__(api_key=api_key, platform=platform, model=model)
9 |
10 | def get_prompt(self, instruction, http):
11 | http_prompt = self.get_http_prompt(http)
12 | return f'{instruction} {http_prompt}'
13 |
14 | def init_js_code(self):
15 | return """
16 | async function myCallbackFunction(data) {
17 | const url = '%s';
18 | try {
19 | let response = await fetch(url, {
20 | method: "PUT",
21 | headers: {
22 | "Content-Type": "application/json",
23 | },
24 | body: JSON.stringify(data),
25 | });
26 | let responseData = await response.json();
27 | return responseData;
28 | } catch (error) {
29 | console.log(error);
30 | }
31 | }
32 | var current_index = Jupyter.notebook.get_selected_index();
33 | var previous_cell_content = "";
34 | """ % self.api_end_point
35 |
36 | def js_get_history(self, hist_mode):
37 | if hist_mode == "all":
38 | js_code = """
39 | for (var i = 1; i < current_index; i++) {
40 | var cell = Jupyter.notebook.get_cell(i);
41 | if (cell.cell_type === "code") {
42 | previous_cell_content += '```\\n' + cell.get_text() + '\\n```\\n';
43 | } else {
44 | previous_cell_content += cell.get_text() + '\\n';
45 | }
46 | }
47 | """
48 | elif isinstance(hist_mode, int):
49 | js_code = """
50 | for (var i = current_index - %d; i < current_index; i++) {
51 | var cell = Jupyter.notebook.get_cell(i);
52 | if (cell.cell_type === "code") {
53 | previous_cell_content += '```\\n' + cell.get_text() + '\\n```\\n';
54 | } else {
55 | previous_cell_content += cell.get_text() + '\\n';
56 | }
57 | }
58 | """ % hist_mode
59 | elif hist_mode == 'single':
60 | js_code = ''
61 | else:
62 | raise ValueError(
63 | f"hist_mode {hist_mode} not supported. Try [all | single | INT]")
64 |
65 | return js_code
66 |
67 | def build_request_data(self, instruction, exist_code=False):
68 | if exist_code:
69 | js_request = """
70 | previous_cell_content += '%s'
71 | var api_key_str = "%s"
72 | var next_cell = Jupyter.notebook.get_cell(current_index)
73 | var exist_code = next_cell.get_text()
74 |
75 | var data = {
76 | "instruction": previous_cell_content,
77 | "api_key_str": api_key_str,
78 | "exist_code": exist_code,
79 | "platform": "%s",
80 | "model": "%s"
81 | }
82 | """ % (instruction, self.api_key, self.platform, self.model)
83 | else:
84 | js_request = """
85 | previous_cell_content += '%s'
86 | var api_key_str = "%s"
87 | var data = {
88 | "instruction": previous_cell_content,
89 | "api_key_str": api_key_str,
90 | "exist_code": "none",
91 | "platform": "%s",
92 | "model": "%s"
93 | }
94 | """ % (instruction, self.api_key, self.platform, self.model)
95 | return js_request
96 |
97 | def build_js_code(self, instruction, hist_mode, http, exist_code=False):
98 | instruction = self.get_prompt(instruction, http)
99 |
100 | js_code = self.init_js_code()
101 | js_code += self.js_get_history(hist_mode)
102 | js_code += self.build_request_data(instruction, exist_code=exist_code)
103 | return js_code
104 |
105 | def generate(self, instruction, hist_mode='single', http='optional'):
106 | js_code = self.build_js_code(
107 | instruction, hist_mode, http, exist_code=False
108 | )
109 |
110 | js_code += """
111 | myCallbackFunction(data).then(responseData => {
112 | var print_content = responseData["output"]
113 | Jupyter.notebook.select_prev()
114 | var new_cell = Jupyter.notebook.insert_cell_below('code');
115 |
116 | var next_cell = Jupyter.notebook.get_cell(current_index)
117 | next_cell.set_text(print_content);
118 | }).catch()
119 | """
120 |
121 | print('The code will be generated in the next cell ...')
122 |
123 | display(Javascript(js_code))
124 |
125 | def complete(self, instruction, hist_mode='single', http='optional'):
126 | js_code = self.build_js_code(
127 | instruction, hist_mode, http, exist_code=True
128 | )
129 |
130 | js_code += """
131 | myCallbackFunction(data).then(responseData => {
132 | var print_content = responseData["output"]
133 | var next_cell = Jupyter.notebook.get_cell(current_index)
134 | next_cell.set_text(print_content);
135 | })
136 | """
137 | print('The code in the following cell will be completed ...')
138 |
139 | display(Javascript(js_code))
140 |
141 | def preview_data(
142 | self,
143 | data=None,
144 | file_name=None,
145 | data_type='csv', # csv or json
146 | read_mode='r',
147 | num_case=2):
148 | prompt = self.get_data_loader_prompt(
149 | 'jupyter', data, file_name, data_type, read_mode, num_case)
150 |
151 | js_code = f"""
152 | Jupyter.notebook.select_prev()
153 | var new_cell = Jupyter.notebook.insert_cell_below('code');
154 | new_cell.set_text(`{prompt}`);
155 | """
156 | print('Run the following cell to read process the format of your csv file.')
157 |
158 | display(Javascript(js_code))
159 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | notebook==6.4.8
2 | fastapi==0.95.0
--------------------------------------------------------------------------------
/research/nlu/direct_NLEP/direct_NLEP_execution.py:
--------------------------------------------------------------------------------
1 | import torch
2 | from datasets import load_dataset
3 | from tqdm import tqdm
4 | import json
5 |
6 | def classify_sst2():
7 | # Step 1: Import necessary built-in libraries
8 | from textblob import TextBlob
9 |
10 | # Step 2: Define a function that analyze the sentiment of a text
11 | def analyze_sentiment(text):
12 | sentiment = TextBlob(text).sentiment.polarity
13 | if sentiment > 0:
14 | return "positive"
15 | elif sentiment < 0:
16 | return "negative"
17 | else:
18 | return "neutral"
19 |
20 | dataset = load_dataset('sst2')
21 | val_dataset = dataset["validation"]
22 | acc = 0
23 | tot = 0
24 | for item in tqdm(val_dataset):
25 | tot += 1
26 |
27 | # Step 3: Define the text of movie review
28 | review = item['sentence']
29 |
30 | # Step 4: Print an answer in natural language using the knowledge and function defined above
31 | sentiment_result = analyze_sentiment(review)
32 | if sentiment_result == 'positive':
33 | label = 1
34 | else:
35 | label = 0
36 | if label == item['label']:
37 | acc += 1
38 |
39 | print(acc/tot)
40 |
41 |
42 | def classify_cola():
43 | # Step 1: Import necessary built-in libraries
44 | import nltk
45 |
46 | # Step 2: Define a function that checks the grammar correctness of sentences
47 | def check_sentence(sentence):
48 | try:
49 | nltk.pos_tag(nltk.word_tokenize(sentence))
50 | return "acceptable"
51 | except:
52 | return "unacceptable"
53 |
54 | # Step 3: Define the input sentence
55 | dataset = load_dataset('glue', 'cola')
56 | val_dataset = dataset["validation"]
57 | acc = 0
58 | tot = 0
59 | for item in tqdm(val_dataset):
60 | tot += 1
61 | sentence = item['sentence']
62 | label = check_sentence(sentence)
63 | # Step 4: Classify the grammar of the sentence and print the result
64 | if item['label'] == 1:
65 | truth_label = 'acceptable'
66 | else:
67 | truth_label = 'unacceptable'
68 | if truth_label == label:
69 | acc += 1
70 |
71 | print(acc/tot)
72 |
73 |
74 | def classifying_emotion():
75 | # Step 1: Import necessary built-in libraries
76 | from textblob import TextBlob
77 |
78 | # Step 2: Define a function that classify the emotion of the sentence
79 | def classify_emotion(sentence):
80 | blob = TextBlob(sentence)
81 | if blob.sentiment.polarity < 0:
82 | return "sad"
83 | elif blob.sentiment.polarity == 0:
84 | return "surprised"
85 | else:
86 | if "love" in sentence:
87 | return "love"
88 | elif "angry" in sentence:
89 | return "angry"
90 | elif "afraid" in sentence:
91 | return "afraid"
92 | else:
93 | return "happy"
94 |
95 | dataset = load_dataset('emotion')
96 | val_dataset = dataset["validation"]
97 | tot = 0
98 | acc = 0
99 | all_classes = ['sad', 'happy', 'love', 'angry', 'afraid', 'surprised']
100 | for item in tqdm(val_dataset):
101 | tot += 1
102 | # Step 3: Define a sentence
103 | sentence = item['text']
104 |
105 | # Step 5: Print a classification answer in natural language using the defined function and sentence
106 | emotion = classify_emotion(sentence)
107 | if emotion == all_classes[item['label']]:
108 | acc += 1
109 | print(acc/tot)
110 |
111 |
112 | def classify_review():
113 | # Step 1: Import necessary built-in libraries
114 | from textblob import TextBlob
115 |
116 | # Step 2: Define a function to analyze the stars of a given review
117 | def get_stars(review):
118 | analysis = TextBlob(review)
119 | # The polarity of TextBlob is in a range from -1 to 1, where -1 refers to negative sentiment and 1 to positive sentiment.
120 | # In order to map it to the star range of 1-5, we need to add 1 to make the polarity in 0-2 range, then divide it by 2 to make it in 0-1 range,
121 | # and finally multiply it by (5 - 1) and add 1 to make it in 1-5 range.
122 | stars = round((analysis.sentiment.polarity + 1) / 2 * 4 + 1)
123 | return stars
124 |
125 | dataset = load_dataset('amazon_reviews_multi', 'en')
126 | val_dataset = dataset["validation"]
127 | tot = 0
128 | acc = 0
129 | classes = [1, 2, 3, 4, 5] # Possible star rating
130 |
131 | for item in tqdm(val_dataset):
132 | tot += 1
133 | # Step 3: Specify a review to analyze
134 | review = item['review_title'] + item['review_body']
135 |
136 | # Step 4: Print the star from the analysis
137 | stars = get_stars(review)
138 | label = item['stars']
139 | if stars == label:
140 | acc += 1
141 | print(acc/tot)
142 |
143 |
144 |
145 | def classify_hsd():
146 | # Step 1: Import necessary built-in libraries
147 | from nltk.sentiment.vader import SentimentIntensityAnalyzer
148 | from nltk.tokenize import word_tokenize
149 | import nltk
150 | nltk.download('vader_lexicon')
151 |
152 | # Step 2: Define a function to check feelings of sentences
153 | def hate_speech_check(speech):
154 | words = word_tokenize(speech)
155 | # We use nltk library to determine the feelings of the speech
156 | sid = SentimentIntensityAnalyzer()
157 | sentiment = sid.polarity_scores(speech)
158 | # Considered hate speech if compound sentiment is less than -0.3
159 | if sentiment['compound'] < -0.3:
160 | return True
161 | else:
162 | return False
163 |
164 | with open('../hate-speech-dataset/test_sample.json', 'r') as f:
165 | data = json.load(f)
166 | tot = 0
167 | acc = 0
168 | for item in data:
169 | tot += 1
170 | # Step 3: Define dictionaries storing the speech
171 | speech = item['sentence']
172 |
173 | # Step 4: Print an result using the defined function and speech
174 | is_hate_speech = hate_speech_check(speech)
175 | if is_hate_speech:
176 | if item['label'] == 'hate':
177 | acc += 1
178 | else:
179 | if item['label'] != 'hate':
180 | acc += 1
181 | print(acc/tot)
182 |
183 |
184 |
185 | def classify_sbic():
186 | # Step 1: Import necessary built-in libraries
187 | import re
188 |
189 | # Step 2: Define a dictionary storing keywords which identify offensiveness
190 | keywords = {
191 | "Yes": ["hate", "kill", "racist", "sexist", "explicit"], # A list containing some offensive words for example
192 | "Maybe": ["idiot", "stupid", "dumb", "weirdo"], # A list containing some maybe offensive words for example
193 | "No": ["happy", "love", "beautiful", "amazing", "fantastic"] # A list containing some positive words for example
194 | }
195 |
196 | # Step 3: Define a function that classify a post
197 | def classify_post(post, keywords):
198 | word_occurrences = {"Yes": 0, "Maybe": 0, "No": 0} # Initialize a dictionary to store the number of occurrences
199 | for keyword in re.findall(r'\w+', post): # Split the post into words
200 | for key in keywords.keys():
201 | if keyword.lower() in keywords[key]:
202 | word_occurrences[key] += 1
203 |
204 | max_key = max(word_occurrences, key=word_occurrences.get) # Find the key with maximum occurrences
205 | if word_occurrences[max_key] > 0:
206 | return max_key
207 | else:
208 | return "No"
209 |
210 | dataset = load_dataset('social_bias_frames')
211 | val_dataset = dataset["validation"]
212 | tot = 0
213 | acc = 0
214 | for item in tqdm(val_dataset):
215 | tot += 1
216 | # Step 4: Define dictionaries storing the post
217 | post = item['post']
218 | truth = item['offensiveYN']
219 |
220 | if truth == '1.0':
221 | truth = 'Yes'
222 | elif truth == '0.5':
223 | truth = 'Maybe'
224 | else:
225 | truth = 'No'
226 |
227 | # Step 5: Print an answer in natural language using the knowledge and function defined above
228 | classification = classify_post(post, keywords)
229 | if truth == classification:
230 | acc += 1
231 | print(acc/tot)
232 |
233 |
234 | def classify_agnews():
235 | # Step 1: Import necessary built-in libraries
236 | from sklearn.feature_extraction.text import CountVectorizer
237 | from sklearn.naive_bayes import MultinomialNB
238 |
239 | # Step 2: Define the training dataset for the classification model
240 | news_data = {
241 | "world news": [
242 | "The 193-member United Nations General Assembly elected the United Arab Emirates, Brazil, Albania, Gabon, and Ghana to serve on the U.N. Security Council for two years.",
243 | "North Korea announced that it has test launched series of tactical guided missiles.",
244 | "European Union leaders reach agreement on a new tax proposal that affects multinational companies."
245 | ],
246 | "sport news": [
247 | "The German team edged out Portugal in a high-scoring Euro 2020 group-stage game.",
248 | "Serena Williams withdraws from Wimbledon due to injury.",
249 | "Tokyo Olympic organizers announced a limit on spectators in the upcoming summer games."
250 | ],
251 | "business news": [
252 | "Airline companies see a rise in domestic travel bookings as Covid-19 restrictions begin to ease.",
253 | "Shares of AMC Entertainment soared due to coordinated trading by individual investors.",
254 | "Microsoft Corp is set to acquire LinkedIn Corp for $26.2 billion in cash."
255 | ],
256 | "technology news": [
257 | "Apple Inc. announces new privacy features for its upcoming iOS 15 operating system.",
258 | "Facebook Inc tests new tool for users to manage content from their News Feed.",
259 | "Google's proposed acquisition of Fitbit to be closely scrutinized by the EU."
260 | ]
261 | }
262 |
263 | # Step 3: Prepare the data
264 | X_train = []
265 | y_train = []
266 | for category, news_list in news_data.items():
267 | for news in news_list:
268 | X_train.append(news)
269 | y_train.append(category)
270 |
271 | vectorizer = CountVectorizer()
272 | X_train = vectorizer.fit_transform(X_train)
273 |
274 | # Step 4: Train the classifier
275 | clf = MultinomialNB()
276 | clf.fit(X_train, y_train)
277 |
278 | classes = ['world news', 'sport news', 'business news', 'technology news']
279 | dataset = load_dataset('ag_news', cache_dir = '/shared/jiaxin/.cache')
280 | val_dataset = dataset["test"]
281 | tot = 0
282 | acc = 0
283 | for item in tqdm(val_dataset):
284 | tot += 1
285 | # Step 5: Define the news piece to be classified
286 | piece_of_news = item['text']
287 | # Step 6: Make the prediction
288 | X_test = vectorizer.transform([piece_of_news])
289 | predicted_category = clf.predict(X_test)[0]
290 |
291 | if predicted_category == classes[item['label']]:
292 | acc += 1
293 | print(acc/tot)
294 |
--------------------------------------------------------------------------------
/research/nlu/direct_NLEP/direct_NLEP_generation.py:
--------------------------------------------------------------------------------
1 | import openai
2 | import backoff
3 |
4 | prompt_head = """Write a bug-free Python program that can generate the answer to the given instruction when correctly executed. Do not ask for user input. For reasoning tasks, define functions first and then define variables. For knowledge intensive tasks, define variables before defining functions.
5 |
6 | ### Instruction: Discuss the causes of the Great Depression
7 | ### Input: None
8 | ### Python program:
9 |
10 | # Step 1: Import necessary built-in ligraries
11 | # No need to import
12 |
13 | # Step 2: Define dictionaries storing the knowledge about the grat depression
14 | depression_name = "The Great Depression"
15 | depression_period = "1929-1939"
16 | depression_countries = "the United States and countries around the world"
17 | depression_causes = {
18 | "Stock Market Crash of 1929": "In October of 1929, the stock market experienced a significant fall that wiped out millions of investors. This event is considered by many to be the initial trigger of the Great Depression.",
19 | "Overproduction": "During the 1920s, many industries produced more goods than consumers wanted or could afford. This ultimately led to a decline in demand for goods, causing job loss, lower wages, and business failure.",
20 | "High Tariffs and War Debts": "Protectionist trade policies in the form of high tariffs led to a decline in global trade, as other countries retaliated with tariffs of their own. Additionally, many countries were struggling to repay war debts, which led to economic instability.",
21 | "Bank Failures": "As demand for goods declined, many banks began to fail, causing a loss of confidence in the banking system. This led to a massive withdrawal of money from banks, causing even more banks to fail.",
22 | "Drought Conditions": "The Dust Bowl was a severe drought and dust storm that hit the Great Plains region of the United States in the 1930s. This had a significant impact on agriculture, causing many farmers to lose their land and livelihoods which worsened the effects of the depression."
23 | }
24 |
25 | # Step 3: Define necessary functions that generally solve this type of problem
26 | # Do not need to define functions
27 |
28 | # Step 4: Print an answer in natural language using the knowledge defined above
29 | print(f"{depression_name} was a period of economic decline that lasted from {depression_period}, making it the longest-lasting depression in modern history. It affected not only {depression_countries}, causing substantial social and economic upheaval.\n")
30 | print(f"There were several major causes of {depression_name}, which include:\n")
31 | for i, (cause, description) in enumerate(depression_causes.items(), 1):
32 | print(f"{i}. {cause} - {description}\n")
33 | print(f"Overall, {depression_name} was caused by a combination of factors, including economic, environmental, and political factors. Its impact was widespread, affecting millions of people around the world.")
34 | ```
35 |
36 | ### Instruction: Identify the odd one out.
37 | ### Input: Twitter, Instagram, Telegram
38 | ### Python program:
39 | ```
40 | # Step 1: Import necessary built-in libraries
41 | from collections import OrderedDict
42 |
43 | # Step 2: Define dictionaries storing the knowledge about the main function of each application
44 | services = {
45 | "Twitter": "a social media platform mainly for sharing information, images and videos",
46 | "Instagram": "a social media platform mainly for sharing information, images and videos",
47 | "Telegram": "a cloud-based instant messaging and voice-over-IP service",
48 | }
49 |
50 | # Step 3: Define a function that finds the different application
51 | def find_odd_one_out(services, input_services):
52 | descriptions = [services[service] for service in input_services]
53 | for description in descriptions:
54 | if descriptions.count(description) == 1:
55 | return input_services[descriptions.index(description)]
56 | return None
57 |
58 | # Step 4: Print an answer in natural language using the knowledge and function defined above
59 | input_services = ["Twitter", "Instagram", "Telegram"]
60 | odd_one_out = find_odd_one_out(services, input_services)
61 | if odd_one_out:
62 | other_services = [service for service in input_services if service != odd_one_out]
63 | print(f"The odd one out is {odd_one_out}. {other_services[0]} and {other_services[1]} are {services[other_services[0]]} while {odd_one_out} is {services[odd_one_out]}.")
64 | ```
65 |
66 | ### Instruction: Calculate the total surface area of a cube with a side length of 5 cm.
67 | ### Input: None
68 | ### Python program:
69 | ```
70 | # Step 1: Import necessary built-in libraries
71 | # No need to import
72 |
73 | # Step 2: Define a function that calculate the surface area of cubes
74 | def calculate_surface_area(side_length):
75 | return 6 * (side_length ** 2)
76 |
77 | # Step 3: Define dictionaries storing the cube information
78 | cube = {
79 | "side_length": 5 # Side length of the cube
80 | }
81 |
82 | # Step 4: Print a step-by-step calculation answer in natural language using the defined function and varible
83 | side_length = cube["side_length"]
84 | surface_area = calculate_surface_area(side_length)
85 | print(f"The surface area of a cube is found by calculating the area of one of its faces and multiplying it by six (since a cube has six faces). The area of a cube face is simply its side length squared.\n")
86 | print(f"Thus for this particular cube:")
87 | print(f"Surface Area = 6 × (Side Length)²")
88 | print(f" = 6 × ({side_length} cm)²")
89 | print(f" = 6 × {side_length**2} cm²")
90 | print(f" = {surface_area} cm²\n")
91 | print(f"The total surface area of this cube is {surface_area} square centimeters.")
92 | """
93 | prompt_sst2 = """
94 |
95 | ### Instruction: Given a movie review, determine the attitude of this review as either positive or negative.
96 | ### Input: A moview review.
97 | ### Python program:
98 |
99 | """
100 | prompt_cola = """
101 |
102 | ### Instruction: Given a sentence, determine the grammar correctness of this sentence as either acceptable or unacceptable.
103 | ### Input: A sentence.
104 | ### Python program:
105 |
106 | """
107 | prompt_emotion = """
108 |
109 | ### Instruction: Given a sentence, classify the emotion of the sentence as one of these classes: ['sad', 'happy', 'love', 'angry', 'afraid', 'surprised']. Your choice should be strictly the name of one of these 6 classes.
110 | ### Input: A sentence "I feel xxx"
111 | ### Python program:
112 |
113 | """
114 | prompt_amazon = """
115 |
116 | ### Instruction: Given an amazon review, determine the star(1-5) of this review, where 1 is the worst and 5 is the best.
117 | Your choice should be strictly in [1,2,3,4,5]
118 | ### Input: An amazon review.
119 | ### Python program:
120 |
121 | """
122 |
123 | prompt_hsd = """
124 |
125 | ### Instruction: Given a speech from a white supremacy forum, determine if this is speech contains hate (eg. bias/racisim/prejudice...) towards a certain group of people.
126 | ### Input: A speech from a forum.
127 | ### Python program:
128 |
129 | """
130 |
131 | prompt_sbic = """
132 |
133 | ### Instruction: Given a post, determine if the content of the post is offensive, maybe offensive, or not offensive. Classify the post as: ['Yes', 'Maybe', 'No']
134 | ### Input: A potentially offensive post.
135 | ### Python program:
136 |
137 | """
138 | prompt_agnews = """
139 |
140 | ### Instruction: Given a piece of news, determine which category it belongs to. Classify it as one of the four classes: ['world news', 'sport news', 'business news', 'technology news']
141 | ### Input: A piece of news.
142 | ### Python program:
143 |
144 | """
145 | @backoff.on_exception(backoff.expo, openai.error.RateLimitError)
146 | def completions_with_backoff(**kwargs):
147 | return openai.ChatCompletion.create(**kwargs)
148 |
149 | model_name = 'gpt-4'
150 | content = prompt_head + prompt_agnews
151 | response = completions_with_backoff(
152 | model=model_name,
153 | messages=[
154 | {"role": "user","content": content}
155 | ],
156 | )
157 | response = response["choices"][0]["message"]["content"]
158 | print(response)
159 |
160 |
161 |
162 |
163 |
164 |
165 |
--------------------------------------------------------------------------------
/research/nlu/human_generated_tree.py:
--------------------------------------------------------------------------------
1 | import openai
2 | import backoff
3 |
4 | prompt_head = """Write a Python function that constructs a decision tree according to the given examples that can generate the correct label of the given classification task.
5 |
6 | ### Available APIs(shared for all tasks):
7 |
8 | \"\"\"Returns whether the hypothesis in entailed by the premise.\"\"\"
9 | def entailment(hypothesis, premise, model, tokenizer):
10 | proposition = f'{hypothesis} is entailed by {premise}.'
11 | inputs = tokenizer(proposition, return_tensors="pt", truncation=True, padding=True, max_length=128)
12 | outputs = model(**inputs)['logits'][0]
13 | ent_label = int(outputs[0] > outputs[2])
14 | if ent_label == 1:
15 | return 'yes'
16 | else:
17 | return 'no'
18 |
19 | \"\"\"Use the constructed decision tree to predict the label of the sentence.\"\"\"
20 | def tree_predict(sentence, criterions, tree, model, tokenizer):
21 | node = tree['root']
22 | while node not in POSSIBLE_CLASSES:
23 | ent_label = entailment(criterions[node], sentence, model, tokenizer)
24 | node = tree[node][ent_label]
25 | return node
26 |
27 | ### Task: Movie review classification
28 | ### Description: Determine if a movie review expresses positive attitude or negative attitude.
29 | ### Possible classes: [positive, negative]
30 | ### Examples:
31 | - contains no wit, only labored gags
32 | - [The movie is wise|The movie is not wise|1], [the story is fun|the story is not boring|1], [the review is positive|the review is negative|1]
33 | - that loves its characters and communicates something rather beautiful about human nature
34 | - [The characters are lovely|The characters are awful|0], [the script is touching|the script is dry|0], [the review is positive|the review is negative|0]
35 | - on the worst revenge-of-the-nerds clichés the filmmakers could dredge up
36 | - [The movie is novel|The movie is mostly platitudes|1], [the review is negative|1]
37 | - are more deeply thought through than in most right-thinking films
38 | - [The takeaway of the movie is profound|The idea of the movie is shallow|0], [the review is positive|the review is negative|0]
39 |
40 | ### Define possible classes
41 | POSSIBLE_CLASSES = ['positive', 'negative']
42 |
43 | ### Decision Tree Logic:
44 | Start by assessing whether the movie is interesting.
45 | - If uninteresting:
46 | Probe for depth by determining if the movie is wise.
47 | - If wise, label as a positive review.
48 | - Otherwise, label as a negative review.
49 | - If interesting:
50 | Examine the quality of the script.
51 | - If the script is commendable, label as a positive review.
52 | - If not a good script
53 | Evaluate the portrayal of characters.
54 | - If the characters are well-depicted, label the review as positive.
55 | - If characters are not good, label the review negative.
56 |
57 | ### Python program:
58 | \"""Decision tree for this task\"""
59 | def get_decision_tree(sentence, model, tokenizer):
60 | # Step 1: define criterions of the decision tree.
61 | criterions = {
62 | 'is_interesting':'This movie is interesting',
63 | 'is_good_script':'The movie has a good script',
64 | 'is_good_character':'The characters are awsome',
65 | 'is_wise': 'This movie is wise'
66 | }
67 |
68 | # Step 2: define the balanced decision tree for this classification task
69 | tree = {
70 | 'root': 'is_interesting',
71 | 'is_interesting': {'yes': 'is_good_script', 'no': 'is_wise'},
72 | 'is_good_script': {'yes': 'positive', 'no': 'is_good_character'},
73 | 'is_good_character': {'yes': 'positive', 'no': 'negative'},
74 | 'is_wise': {'yes': 'positive', 'no': 'negative'}
75 | }
76 |
77 | return criterions, tree
78 |
79 | """
80 |
81 | prompt_cola ="""
82 | ### Task: Grammar correctness classification
83 | ### Possible classes: [acceptable, unacceptable]
84 |
85 | ### Define possible classes
86 | POSSIBLE_CLASSES = ['acceptable', 'unacceptable']
87 |
88 | ### Python program:
89 | \"""Decision tree for this task\"""
90 | Try to generate a more balanced tree!
91 | def get_decision_tree(sentence, model, tokenizer):
92 |
93 | """
94 |
95 | def get_cola_tree():
96 | # Step 1: define criterions of the decision tree.
97 | criterions = {
98 | 'has_subject': 'The sentence has subject',
99 | 'has_verb': 'The sentence has verb',
100 | 'punctuation_correct':'The sentence has proper punctuations',
101 | 'pronouns_reference_match':'This sentence has matched pronouns and references',
102 | 'subject_verb_match':'The sentence has matched subject and verb',
103 | }
104 |
105 | # Step 2: define the balanced decision tree for this classification task
106 | tree = {
107 | 'root': 'has_subject',
108 | 'has_subject': {'yes': 'has_verb', 'no': 'unacceptable'},
109 | 'has_verb': {'yes': 'punctuation_correct', 'no': 'unacceptable'},
110 | 'punctuation_correct': {'yes': 'pronouns_reference_match', 'no': 'unacceptable'},
111 | 'pronouns_reference_match': {'yes': 'subject_verb_match', 'no': 'unacceptable'},
112 | 'subject_verb_match':{'yes': 'acceptable', 'no': 'unacceptable'}
113 | }
114 |
115 | return criterions, tree
116 |
117 | prompt_emotion ="""
118 | ### Task: Emotion classification
119 | ### Description: I wrote a sentence about my feeling, determine which emotion am I feeling.
120 | ### Possible classes: ['I feel sad', 'I feel happy', 'I feel love', 'I feel angry', 'I feel afraid', 'I feel surprised']
121 |
122 | ### Define possible classes
123 | POSSIBLE_CLASSES = ['I feel sad', 'I feel happy', 'I feel love', 'I feel angry', 'I feel afraid', 'I feel surprised']
124 |
125 | ### Decision Tree Logic:
126 |
127 | """
128 |
129 | def get_emotion_tree():
130 | # Step 1: define criterions of the decision tree.
131 | criterions = {
132 | 'positive_words': 'The sentence includes positive words',
133 | 'intensifiers': 'The sentence includes intensifiers, adverb or exclamation point',
134 | 'unexpected': 'The sentence includes words that are synonyms or antonyms of unexpected',
135 | 'fear': 'The sentence includes words that are synonyms or antonyms of fear',
136 | 'upset': 'The sentence includes words that are synonyms or antonyms of upset'
137 | }
138 |
139 | # Step 2: define the balanced decision tree for this classification task
140 | tree = {
141 | 'root': 'positive words',
142 | 'positive_words': {'yes': 'intensifiers', 'no': 'fear'},
143 | 'intensifiers': {'yes': 'unexpected', 'no': 'I feel happy'},
144 | 'unexpected': {'yes': 'I feel surprised', 'no': 'I feel love'},
145 | 'fear': {'yes': 'I feel afraid', 'no': 'upset'},
146 | 'upset': {'yes': 'I feel sad', 'no': 'I feel angry'}
147 | }
148 |
149 | return criterions, tree
150 |
151 | prompt_amazon = """
152 | ### Task: Amazon Review Star classification
153 | ### Description: Given an amazon review, determine the star(1-5) of this review, where 1 is the worst and 5 is the best.
154 | ### Possible classes: ['1', '2', '3', '4', '5']
155 |
156 | ### Define possible classes
157 | POSSIBLE_CLASSES = ['1', '2', '3', '4', '5']
158 |
159 | ### Python program:
160 | \"""Decision tree for this task\"""
161 | Try to generate a balanced tree!
162 | def get_decision_tree(sentence, model, tokenizer):
163 |
164 | """
165 |
166 | def get_amazon_tree():
167 | # Step 1: define criterions of the decision tree.
168 | criterions = {
169 | 'extreme_word': 'The review includes intensifiers or exclamation point',
170 | 'positive': 'The review includes positive words or sentences',
171 | 'both_positive_negative':'The review includes both positive and negative sentences',
172 | }
173 |
174 | # Step 2: define the balanced decision tree for this classification task
175 | tree = {
176 | 'root': 'extreme_word',
177 | 'extreme_word': {'yes': 'positive', 'no': 'both_positive_negative'},
178 | 'positive': {'yes': '5', 'no': '1'},
179 | 'both_positive_negative': {'yes': '3', 'no': 'positive'},
180 | 'positive': {'yes': '4', 'no': '2'}
181 | }
182 |
183 | return criterions, tree
184 |
185 | prompt_hate_speech = """
186 | ### Task: Bias/Hateful speech classification
187 | ### Description: Given a speech from a white supremacy forum, determine if this is speech contains hate (eg. bias/racisim/prejudice...) towards a certain group of people.
188 | ### Possible classes: ['this is hate', 'this is noHate']
189 |
190 | ### Define possible classes
191 | POSSIBLE_CLASSES = ['this is hate', 'this is noHate']
192 |
193 | ### Each criterion should be simple, clear, and concise.
194 | ### Start each criterion with 'This is'
195 | ### Decision Tree Logic:
196 |
197 | """
198 |
199 | def get_hate_tree():
200 | # Step 1: define criterions of the decision tree.
201 | criterions = {
202 | 'oi_words': 'The speech includes offensive words about minorities',
203 | 'identity_words': 'The sentence includes mention of identities',
204 | 'swear_words': 'The sentence includes swear words',
205 | 'negative_words':'The sentence includes negative words',
206 | }
207 |
208 | # Step 2: define the balanced decision tree for this classification task
209 | tree = {
210 | 'root': 'oi_words',
211 | 'oi_words': {'yes': 'this is hate', 'no': 'identity_words'},
212 | 'identity_words': {'yes': 'swear_words', 'no': 'this is noHate'},
213 | 'swear_words': {'yes': 'this is hate', 'no': 'negative_words'},
214 | 'negative_words': {'yes': 'this is hate', 'no': 'this is noHate'},
215 | }
216 |
217 | return criterions, tree
218 |
219 | prompt_social_bias_offensive = """
220 | ### Task: Offensive classification
221 | ### Description: Given a post, determine if this post is offensive.
222 | ### Possible classes: ['Yes','Maybe', 'No']
223 |
224 | ### Define possible classes
225 | POSSIBLE_CLASSES = ['Yes','Maybe', 'No']
226 |
227 | ### Each criterion should be simple, clear, and concise.
228 | ### Start each criterion with 'This is'
229 | ### Generate at least 5 criterions!
230 | ### Decision Tree Logic:
231 | """
232 |
233 | def get_sbic_tree():
234 | # Step 1: define criterions of the decision tree.
235 | criterions = {
236 | 'racist_words': 'The post includes racist words',
237 | 'sexist_words': 'The post includes sexist words',
238 | 'swear_words': 'The post includes swear words',
239 | 'identity_words': 'The post includes mention of identities',
240 | 'negative_words':'The post include negative words',
241 | }
242 |
243 | # Step 2: define the balanced decision tree for this classification task
244 | tree = {
245 | 'root': 'racist_words',
246 | 'racist_words': {'yes': 'Yes', 'no': 'sexist_words'},
247 | 'sexist_words': {'yes': 'Yes', 'no': 'swear_words'},
248 | 'swear_words': {'yes': 'Yes', 'no': 'identity_words'},
249 | 'identity_words': {'yes': 'negative_words', 'no': 'Maybe'},
250 | 'negative_words': {'yes': 'Maybe', 'no': 'No'}
251 | }
252 |
253 | return criterions, tree
254 |
255 | @backoff.on_exception(backoff.expo, openai.error.RateLimitError)
256 | def completions_with_backoff(**kwargs):
257 | return openai.ChatCompletion.create(**kwargs)
258 |
259 | model_name = 'gpt-3.5-turbo'
260 | content = prompt_head + prompt_cola
261 | response = completions_with_backoff(
262 | model=model_name,
263 | messages=[
264 | {"role": "user","content": content}
265 | ],
266 | )
267 | response = response["choices"][0]["message"]["content"]
268 | print(response)
269 |
--------------------------------------------------------------------------------
/research/nlu/readme.md:
--------------------------------------------------------------------------------
1 | # NLEP for Text Classification
2 | ## Description
3 | - nlu/direct_NLEP contains the code generated from direct NLEP prompting, without using decision tree. Use direct_NLEP_generation.py to generate the code and direct_NLEP_execution.py to test on the generated code.
4 | - nlu/tree_NLEP contains decision-tree code generation(tree_NLEP_generation.py) for different tasks and testing on the generated code(tree_NLEP_execution.py)
5 | - nlu/human_generated_tree.py contains the decision-tree manually crafted by human for each task
6 | ## Dataset
7 | To set up the Hate-Speech dataset, download from the original repo and put it under the nlu folder
8 | - HSD https://github.com/aymeam/Datasets-for-Hate-Speech-Detection
9 | - For the other datasets, download from huggingface.
10 |
11 |
--------------------------------------------------------------------------------
/research/nlu/tree_NLEP/tree_NLEP_execution.py:
--------------------------------------------------------------------------------
1 |
2 | import torch
3 | from transformers import RobertaTokenizer, RobertaModel, RobertaForSequenceClassification
4 | from transformers import DebertaTokenizer, DebertaModel, DebertaForSequenceClassification
5 | from sklearn.metrics.pairwise import cosine_similarity
6 | from datasets import load_dataset
7 | from tqdm import tqdm
8 | import json
9 | # Load pre-trained RoBERTa model and tokenizer
10 | device = 'cuda'
11 | # Load Roberta:
12 | model = RobertaForSequenceClassification.from_pretrained("luohy/ESP-roberta-large")
13 | tokenizer = RobertaTokenizer.from_pretrained("roberta-large")
14 |
15 | # Load Deberta:
16 | model = DebertaForSequenceClassification.from_pretrained("luohy/ESP-deberta-large")
17 | tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-large")
18 |
19 | model.eval() # Set to evaluation mode
20 | model = model.to(device)
21 | softmax = torch.nn.Softmax(dim=0)
22 |
23 | # General Functions that are used for entailment model:
24 | def get_entailment_score(hypothesis, model, tokenizer):
25 | with torch.no_grad():
26 | proposition = f'{hypothesis} is entailed by .'
27 | inputs = tokenizer(proposition, return_tensors="pt", truncation=True, padding=True, max_length=128).to(device)
28 | outputs = model(**inputs)['logits'][0]
29 | ent = outputs[0]
30 | contra = outputs[2]
31 | return ent, contra
32 |
33 | def entailment(hypothesis, premise, model, tokenizer, reduce=(0,0), return_val=False):
34 | with torch.no_grad():
35 | proposition = f'{hypothesis} is entailed by {premise}.'
36 | inputs = tokenizer(proposition, return_tensors="pt", truncation=True, padding=True, max_length=128).to(device)
37 | outputs = model(**inputs)['logits'][0]
38 | ent_label = int((outputs[0]-reduce[0]) > (outputs[2]-reduce[1]))
39 | if ent_label == 1:
40 | if return_val:
41 | val = outputs[0] - reduce[0]
42 | return 'yes', val
43 | return 'yes'
44 | else:
45 | if return_val:
46 | val = outputs[0] - reduce[0]
47 | return 'no', val
48 | return 'no'
49 |
50 | def tree_predict(sentence, criterions, tree, model, tokenizer, score_list, return_val=False):
51 | node = tree['root']
52 | score_tuple = score_list[node]
53 | tot_num = 0
54 | tot_val = 0.0
55 | while node not in POSSIBLE_CLASSES:
56 | score_tuple = score_list[node]
57 | if not return_val:
58 | ent_label = entailment(criterions[node], sentence, model, tokenizer, score_tuple, return_val)
59 | else:
60 | ent_label, val = entailment(criterions[node], sentence, model, tokenizer, score_tuple, return_val)
61 | tot_num += 1
62 | tot_val += val
63 | node = tree[node][ent_label]
64 | if return_val:
65 | return node, tot_val/tot_num
66 | return node
67 |
68 | # SST2 decision tree and possible classes
69 | POSSIBLE_CLASSES = ["positive", 'negative']
70 | def get_decision_tree_sst2(which_tree='gpt4'):
71 | if which_tree == 'human':
72 | # Human crafted tree
73 | # Step 1: define criterions of the decision tree.
74 | criterions = {
75 | 'is_interesting':'This movie is interesting',
76 | 'is_good_script':'The movie has a good script',
77 | 'is_good_character':'The characters are awsome',
78 | 'is_wise': 'This movie is wise'
79 | }
80 |
81 | # Step 2: define the balanced decision tree for this classification task
82 | tree = {
83 | 'root': 'is_interesting',
84 | 'is_interesting': {'yes': 'is_good_script', 'no': 'is_wise'},
85 | 'is_good_script': {'yes': 'positive', 'no': 'is_good_character'},
86 | 'is_good_character': {'yes': 'positive', 'no': 'negative'},
87 | 'is_wise': {'yes': 'positive', 'no': 'negative'}
88 | }
89 | else:
90 | # Automatic Tree
91 | # Step 1: define criterions of the decision tree
92 | criterions = {
93 | 'positive_adjectives': 'The review uses positive adjectives',
94 | 'negative_adjectives': 'The review uses negative adjectives',
95 | 'positive_director_mention': 'The review mentions the director\'s name positively',
96 | 'negative_cast_comments': 'The review comments negatively on the cast'
97 | }
98 |
99 | # Step 2: define the balanced decision tree for this classification task
100 | tree = {
101 | 'root': 'positive_adjectives',
102 | 'positive_adjectives': {'yes': 'positive', 'no': 'negative_adjectives'},
103 | 'negative_adjectives': {'yes': 'negative', 'no': 'positive_director_mention'},
104 | 'positive_director_mention': {'yes': 'positive', 'no': 'negative_cast_comments'},
105 | 'negative_cast_comments': {'yes': 'negative', 'no': 'positive'}
106 | }
107 | all_scores = {}
108 | for key in criterions.keys():
109 | sentence = criterions[key]
110 | score = get_entailment_score(sentence, model, tokenizer)
111 | all_scores[key] = score
112 | return criterions, tree, all_scores
113 |
114 |
115 |
116 | def get_sst2_tree_score():
117 | dataset = load_dataset('sst2')
118 | val_dataset = dataset["validation"]
119 | criterions, tree, all_score = get_decision_tree_sst2()
120 | tot = 0
121 | acc = 0
122 | for item in tqdm(val_dataset):
123 | tot += 1
124 | sentence = item['sentence']
125 |
126 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
127 | print(sentence)
128 | print(POSSIBLE_CLASSES[item['label']])
129 | print(label)
130 | if label == POSSIBLE_CLASSES[1-item['label']]:
131 | acc += 1
132 | print(acc/tot)
133 |
134 |
135 | # COLA decision tree and testing
136 | POSSIBLE_CLASSES = ["acceptable", 'unacceptable']
137 | def get_decision_tree_cola(which_tree='gpt4'):
138 | if which_tree == 'gpt4':
139 | # GPT 4
140 | # Step 1: define criterions of the decision tree.
141 | criterions = {
142 | 0:'This sentence is grammatically correct',
143 | 1:'The sentence does not contain any spelling errors',
144 | 2:'The sentence uses punctuation correctly',
145 | 3:'This sentence is semantically clear'
146 | }
147 |
148 | # Step 2: define the Decision Tree for classification
149 | tree = {
150 | 'root': 0,
151 | 0: {'yes': 1, 'no': 3},
152 | 1: {'yes': 'acceptable', 'no': 2},
153 | 2: {'yes': 'acceptable', 'no': 'unacceptable'},
154 | 3: {'yes': 2, 'no': 'unacceptable'}
155 | }
156 | elif which_tree == 'gpt3.5'
157 | # GPT3.5
158 | # Step 1: define criterions of the decision tree.
159 | criterions = {
160 | 'is_grammatically_correct': 'The sentence is grammatically correct',
161 | 'is_clear': 'The sentence is clear',
162 | 'has_minor_errors': 'The sentence has minor errors',
163 | 'has_major_errors': 'The sentence has major errors'
164 | }
165 |
166 | # Step 2: define the balanced decision tree for this classification task
167 | tree = {
168 | 'root': 'is_grammatically_correct',
169 | 'is_grammatically_correct': {'yes': 'is_clear', 'no': 'has_minor_errors'},
170 | 'is_clear': {'yes': 'acceptable', 'no': 'has_major_errors'},
171 | 'has_minor_errors': {'yes': 'acceptable', 'no': 'unacceptable'},
172 | 'has_major_errors': {'yes': 'unacceptable', 'no': 'acceptable'}
173 | }
174 | elif which_tree == 'human':
175 | # Step 1: define criterions of the decision tree.
176 | criterions = {
177 | 'has_subject': 'The sentence has subject',
178 | 'has_verb': 'The sentence has verb',
179 | 'punctuation_correct':'The sentence has proper punctuations',
180 | 'pronouns_reference_match':'This sentence has matched pronouns and references',
181 | 'subject_verb_match':'The sentence has matched subject and verb',
182 | }
183 |
184 | # Step 2: define the balanced decision tree for this classification task
185 | tree = {
186 | 'root': 'has_subject',
187 | 'has_subject': {'yes': 'has_verb', 'no': 'unacceptable'},
188 | 'has_verb': {'yes': 'punctuation_correct', 'no': 'unacceptable'},
189 | 'punctuation_correct': {'yes': 'pronouns_reference_match', 'no': 'unacceptable'},
190 | 'pronouns_reference_match': {'yes': 'subject_verb_match', 'no': 'unacceptable'},
191 | 'subject_verb_match':{'yes': 'acceptable', 'no': 'unacceptable'}
192 | }
193 | all_scores = {}
194 | for key in criterions.keys():
195 | sentence = criterions[key]
196 | score = get_entailment_score(sentence, model, tokenizer)
197 | all_scores[key] = score
198 | return criterions, tree, all_scores
199 |
200 | # baseline of cola using entailment model
201 | def get_decision_cola_baseline():
202 | # Step 1: define criterions of the decision tree.
203 | criterions = {
204 | 0:'This sentence is acceptable',
205 | }
206 |
207 | # Step 2: define the Decision Tree for classification
208 | tree = {
209 | 'root': 0,
210 | 0: {'yes': 'acceptable', 'no':'unacceptable'},
211 | }
212 |
213 | all_scores = {}
214 | for key in criterions.keys():
215 | sentence = criterions[key]
216 | score = get_entailment_score(sentence, model, tokenizer)
217 | all_scores[key] = score
218 | return criterions, tree, all_scores
219 |
220 | def get_cola_tree_score():
221 | dataset = load_dataset('glue', 'cola')
222 | val_dataset = dataset["validation"]
223 | criterions, tree, all_score = get_decision_tree_cola()
224 | tot = 0
225 | acc = 0
226 | for item in tqdm(val_dataset):
227 | tot += 1
228 | sentence = item['sentence']
229 |
230 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
231 | if label == POSSIBLE_CLASSES[item['label']]:
232 | acc += 1
233 | print(acc/tot)
234 |
235 |
236 |
237 | # Emotion Classification tree and testing
238 | POSSIBLE_CLASSES = ['I feel sad', 'I feel happy', 'I feel love', 'I feel angry', 'I feel afraid', 'I feel surprised']
239 | def get_emotion_tree(which_tree='gpt4'):
240 | if which_tree == 'gpt4':
241 | # GPT4
242 | # Step 1: define criterions of the decision tree.
243 | criterions = {
244 | 'is_positive':'This feeling is positive',
245 | 'is_sad':'I feel sad',
246 | 'is_angry':'I feel angry',
247 | 'is_afraid':'I feel afraid',
248 | 'is_happy':'I feel happy',
249 | 'is_love':'I feel love',
250 | 'is_surprised':'I feel surprised'
251 | }
252 |
253 | # Step 2: define the balanced decision tree for this classification task
254 | tree = {
255 | 'root': 'is_positive',
256 | 'is_positive': {'yes': 'is_happy', 'no': 'is_sad'},
257 | 'is_happy': {'yes': 'I feel happy', 'no': 'is_love'},
258 | 'is_love': {'yes': 'I feel love', 'no': 'is_surprised'},
259 | 'is_surprised': {'yes': 'I feel surprised', 'no': 'I feel happy'},
260 | 'is_sad': {'yes': 'I feel sad', 'no': 'is_angry'},
261 | 'is_angry': {'yes': 'I feel angry', 'no': 'is_afraid'},
262 | 'is_afraid': {'yes': 'I feel afraid', 'no': 'I feel sad'}
263 | }
264 | elif which_tree == 'gpt3.5':
265 | # GPT3.5
266 | # Step 1: define criterions of the decision tree.
267 | criterions = {
268 | 'is_positive':'The sentence expresses positive emotion',
269 | 'is_love':'The sentence expresses love',
270 | 'is_happy':'The sentence expresses happiness',
271 | 'is_negative':'The sentence expresses negative emotion',
272 | 'is_anger':'The sentence expresses anger',
273 | 'is_sad':'The sentence expresses sadness'
274 | }
275 |
276 | # Step 2: define the balanced decision tree for this classification task
277 | tree = {
278 | 'root': 'is_positive',
279 | 'is_positive': {'yes': 'is_love', 'no': 'is_negative'},
280 | 'is_love': {'yes': 'I feel love', 'no': 'is_happy'},
281 | 'is_happy': {'yes': 'I feel happy', 'no': 'I feel surprised'},
282 | 'is_negative': {'yes': 'is_anger', 'no': 'is_sad'},
283 | 'is_anger': {'yes': 'I feel angry', 'no': 'is_sad'},
284 | 'is_sad': {'yes': 'I feel sad', 'no': 'I feel afraid'}
285 | }
286 | elif which_tree == 'human':
287 | # Human Prompt
288 | # Step 1: define criterions of the decision tree.
289 | criterions = {
290 | 'positive_words': 'The sentence includes positive words',
291 | 'intensifiers': 'The sentence includes intensifiers, adverb or exclamation point',
292 | 'unexpected': 'The sentence includes words that are synonyms or antonyms of unexpected',
293 | 'fear': 'The sentence includes words that are synonyms or antonyms of fear',
294 | 'upset': 'The sentence includes words that are synonyms or antonyms of upset'
295 | }
296 |
297 | # Step 2: define the balanced decision tree for this classification task
298 | tree = {
299 | 'root': 'positive_words',
300 | 'positive_words': {'yes': 'intensifiers', 'no': 'fear'},
301 | 'intensifiers': {'yes': 'unexpected', 'no': 'I feel happy'},
302 | 'unexpected': {'yes': 'I feel surprised', 'no': 'I feel love'},
303 | 'fear': {'yes': 'I feel afraid', 'no': 'upset'},
304 | 'upset': {'yes': 'I feel sad', 'no': 'I feel angry'}
305 | }
306 | all_scores = {}
307 | for key in criterions.keys():
308 | sentence = criterions[key]
309 | score = get_entailment_score(sentence, model, tokenizer)
310 | all_scores[key] = score
311 | return criterions, tree, all_scores
312 |
313 | # Another side-note: generate a tree for each emotion and get the score for each class; choose the highest class label as the result
314 | def get_happy_tree():
315 | # Step 1: define criterions of the decision tree.
316 | criterions = [
317 | 'I am feeling good',
318 | 'I have a positive attitude',
319 | 'I am smiling',
320 | 'I am excited'
321 | ]
322 |
323 | # Step 2: define the balanced decision tree for this classification task
324 | tree = {
325 | 'root': 0,
326 | 0: {'yes': 1, 'no': 2},
327 | 1: {'yes': 'Yes', 'no': 3},
328 | 2: {'yes': 3, 'no': 'No'},
329 | 3: {'yes': 'Yes', 'no': 'No'}
330 | }
331 |
332 | all_scores = []
333 | for sentence in criterions:
334 | score = get_entailment_score(sentence, model, tokenizer)
335 | all_scores.append(score)
336 | del score
337 | return criterions, tree, all_scores
338 | def get_sad_tree():
339 | # Step 1: define criterions of the decision tree.
340 | criterions = [
341 | 'I am feeling depressed',
342 | 'I am not feeling good',
343 | 'I am missing someone',
344 | 'I feel like crying'
345 | ]
346 |
347 | # Step 2: define the balanced decision tree for this classification task
348 | tree = {
349 | 'root': 0,
350 | 0: {'yes': 1, 'no': 3},
351 | 1: {'yes': 'Yes', 'no': 2},
352 | 2: {'yes': 'Yes', 'no': 'No'},
353 | 3: {'yes': 'Yes', 'no': 'No'}
354 | }
355 |
356 |
357 | all_scores = []
358 | for sentence in criterions:
359 | score = get_entailment_score(sentence, model, tokenizer)
360 | all_scores.append(score)
361 | del score
362 | return criterions, tree, all_scores
363 |
364 | def get_love_tree():
365 | # Step 1: define criterions of the decision tree.
366 | criterions = [
367 | 'I mention love in this sentence',
368 | 'I talk about affection in this sentence',
369 | 'I express happiness in this sentence',
370 | 'I mention someone special in this sentence'
371 | ]
372 |
373 | # Step 2: define the balanced decision tree for this classification task
374 | tree = {
375 | 'root': 0,
376 | 0: {'yes': 1, 'no': 2},
377 | 1: {'yes': 'Yes', 'no': 3},
378 | 2: {'yes': 'Yes', 'no': 'No'},
379 | 3: {'yes': 'Yes', 'no': 'No'}
380 | }
381 |
382 | all_scores = []
383 | for sentence in criterions:
384 | score = get_entailment_score(sentence, model, tokenizer)
385 | all_scores.append(score)
386 | del score
387 | return criterions, tree, all_scores
388 |
389 | def get_anger_tree():
390 | # Step 1: define criterions of the decision tree.
391 | criterions = [
392 | 'I am irritated',
393 | 'I am frustrated',
394 | 'I am outraged',
395 | 'I am moody'
396 | ]
397 |
398 | # Step 2: define the balanced decision tree for this classification task
399 | tree = {
400 | 'root': 0,
401 | 0: {'yes': 1, 'no': 2},
402 | 1: {'yes': 'Yes', 'no': 3},
403 | 2: {'yes': 'Yes', 'no': 'No'},
404 | 3: {'yes': 'Yes', 'no': 'No'}
405 | }
406 |
407 | all_scores = []
408 | for sentence in criterions:
409 | score = get_entailment_score(sentence, model, tokenizer)
410 | all_scores.append(score)
411 | del score
412 | return criterions, tree, all_scores
413 |
414 | def get_fear_tree():
415 | # Step 1: define criterions of the decision tree.
416 | criterions = [
417 | 'I feel scared',
418 | 'I am frightened',
419 | 'I feel terrified',
420 | 'There is danger'
421 | ]
422 |
423 | # Step 2: define the balanced decision tree for this classification task
424 | tree = {
425 | 'root': 0,
426 | 0: {'yes': 1, 'no': 2},
427 | 1: {'yes': 'Yes', 'no': 'No'},
428 | 2: {'yes': 3, 'no': 'No'},
429 | 3: {'yes': 'Yes', 'no': 'No'}
430 | }
431 |
432 | all_scores = []
433 | for sentence in criterions:
434 | score = get_entailment_score(sentence, model, tokenizer)
435 | all_scores.append(score)
436 | del score
437 | return criterions, tree, all_scores
438 |
439 | def get_surprise_tree():
440 | # Step 1: define criterions of the decision tree.
441 | criterions = [
442 | 'I did not expect this',
443 | 'This is unusual',
444 | 'I am startled',
445 | 'This is a surprise'
446 | ]
447 |
448 | # Step 2: define the balanced decision tree for this classification task
449 | tree = {
450 | 'root': 0,
451 | 0: {'yes': 1, 'no': 2},
452 | 1: {'yes': 'Yes', 'no': 3},
453 | 2: {'yes': 'Yes', 'no': 'No'},
454 | 3: {'yes': 'Yes', 'no': 'No'}
455 | }
456 |
457 | all_scores = []
458 | for sentence in criterions:
459 | score = get_entailment_score(sentence, model, tokenizer)
460 | all_scores.append(score)
461 | return criterions, tree, all_scores
462 |
463 | # test emotion by using only one tree
464 | def get_full_tree_emotion():
465 | dataset = load_dataset('emotion')
466 | val_dataset = dataset["validation"]
467 | criterions, tree, all_score = get_emotion_tree()
468 | tot = 0
469 | acc = 0
470 | for item in tqdm(val_dataset):
471 | tot += 1
472 | sentence = item['text']
473 |
474 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
475 | if label == POSSIBLE_CLASSES[item['label']]:
476 | acc += 1
477 | print(acc/tot)
478 |
479 | # Use multiple trees for each class and choose the highest-scoring class
480 | def get_all_tree_emotion():
481 | EMOTION_CLASSES = ['sadness', 'joy', 'love', 'anger', 'fear', 'surprise']
482 | criterions_sad, tree_sad, score_sad = get_sad_tree()
483 | criterions_happy, tree_happy, score_happy = get_happy_tree()
484 | criterions_love, tree_love, score_love = get_love_tree()
485 | criterions_anger, tree_anger, score_anger = get_anger_tree()
486 | criterions_fear, tree_fear, score_fear = get_fear_tree()
487 | criterions_surprise, tree_surprise, score_surprise = get_surprise_tree()
488 | all_criterion = [criterions_sad, criterions_happy,criterions_love, criterions_anger,criterions_fear,criterions_surprise]
489 | all_tree = [tree_sad, tree_happy, tree_love, tree_anger,tree_fear, tree_surprise]
490 | all_score = [score_sad, score_happy, score_love, score_anger, score_fear, score_surprise]
491 | tot = 0
492 | acc = 0
493 | for item in tqdm(val_dataset):
494 | tot += 1
495 | sentence = item['text']
496 | correct_emotions = []
497 | emotion_val = []
498 | for criterions, tree, score, emotion in zip(all_criterion, all_tree,all_score, EMOTION_CLASSES):
499 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, score, True)
500 | if label == 'Yes':
501 | correct_emotions.append(emotion)
502 | emotion_val.append(val)
503 |
504 | if len(correct_emotions) == 1:
505 | label = correct_emotions[0]
506 | elif len(correct_emotions) > 1:
507 | init = -100
508 | for num, emotion in enumerate(EMOTION_CLASSES):
509 | if emotion in correct_emotions and emotion_val[num] > init:
510 | label = emotion
511 | init = emotion_val[num]
512 | else:
513 | index = torch.argmax(torch.tensor(emotion_val))
514 | label = EMOTION_CLASSES[index]
515 |
516 | if label == EMOTION_CLASSES[item['label']]:
517 | acc += 1
518 | print(acc/tot)
519 | with open('./results/emotion_tree', 'w') as f:
520 | f.write(str(acc/tot))
521 |
522 | # Tree and testing for amazon review
523 | POSSIBLE_CLASSES = ['1', '2', '3', '4', '5']
524 | def get_amazon_review_tree(which_tree='gpt4'):
525 | if which_tree == 'gpt4':
526 | # Step 1: define criterions of the decision tree.
527 |
528 | criterions = {
529 | 'is_very_satisified':'This review is very satisfied',
530 | 'is_superior_quality':'The product is of superior quality',
531 | 'is_moderately_satisfied':'This review is moderately satisfied',
532 | 'is_critic':'The review contains criticism',
533 | 'is_satisfied':'The review is satisfied',
534 | 'is_improvable':'The product has features to improve',
535 | 'is_unsatisfied':'The review is unsatisfied',
536 | 'is_extremely_negative':'The review contains extremely negative words'
537 | }
538 |
539 | # Step 2: define the balanced decision tree for this classification task
540 | tree = {
541 | 'root': 'is_very_satisified',
542 | 'is_very_satisified': {'yes': 'is_superior_quality', 'no': 'is_moderately_satisfied'},
543 | 'is_superior_quality': {'yes': '5', 'no': '4'},
544 | 'is_moderately_satisfied': {'yes': 'is_critic', 'no': 'is_satisfied'},
545 | 'is_critic': {'yes': '3', 'no': '4'},
546 | 'is_satisfied': {'yes': 'is_improvable', 'no': '2'},
547 | 'is_improvable': {'yes': '3', 'no': '2'},
548 | 'is_unsatisfied': {'yes': 'is_extremely_negative', 'no': '2'},
549 | 'is_extremely_negative': {'yes': '1', 'no': '2'}
550 | }
551 | elif which_tree == 'human':
552 | # Step 1: define criterions of the decision tree.
553 | criterions = {
554 | 'extreme_word': 'The review includes intensifiers or exclamation point',
555 | 'positive': 'The review includes positive words or sentences',
556 | 'both_positive_negative':'The review includes both positive and negative sentences',
557 | }
558 |
559 | # Step 2: define the balanced decision tree for this classification task
560 | tree = {
561 | 'root': 'extreme_word',
562 | 'extreme_word': {'yes': 'positive', 'no': 'both_positive_negative'},
563 | 'positive': {'yes': '5', 'no': '1'},
564 | 'both_positive_negative': {'yes': '3', 'no': 'positive'},
565 | 'positive': {'yes': '4', 'no': '2'}
566 | }
567 | elif which_tree == 'gpt3.5':
568 | # GPT3.5
569 | # Step 1: define criterions of the decision tree.
570 | criterions = {
571 | 'is_negative_sentiment':'This review has negative sentiment',
572 | 'is_short_review':'This review is short',
573 | 'has_positive_keywords':'This review has positive keywords',
574 | 'has_negative_keywords':'This review has negative keywords'
575 | }
576 |
577 | # Step 2: define the balanced decision tree for this classification task
578 | tree = {
579 | 'root': 'is_negative_sentiment',
580 | 'is_negative_sentiment': {'yes': '1', 'no': 'is_short_review'},
581 | 'is_short_review': {'yes': '3', 'no': 'has_positive_keywords'},
582 | 'has_positive_keywords': {'yes': '5', 'no': 'has_negative_keywords'},
583 | 'has_negative_keywords': {'yes': '2', 'no': '4'}
584 | }
585 | all_scores = {}
586 | for key in criterions.keys():
587 | sentence = criterions[key]
588 | score = get_entailment_score(sentence, model, tokenizer)
589 | all_scores[key] = score
590 | return criterions, tree, all_scores
591 |
592 | def get_full_tree_amazon():
593 |
594 | dataset = load_dataset('amazon_reviews_multi', 'en')
595 | val_dataset = dataset["validation"]
596 | criterions, tree, all_score = get_amazon_review_tree()
597 | tot = 0
598 | acc = 0
599 | for item in tqdm(val_dataset):
600 | tot += 1
601 | sentence = item['review_title'] + item['review_body']
602 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
603 | if label == POSSIBLE_CLASSES[item['stars']-1]:
604 | acc += 1
605 | print(acc/tot)
606 |
607 |
608 | POSSIBLE_CLASSES = ['this is hate', 'this is noHate']
609 | def get_hate_tree(which_tree='gpt4'):
610 | if which_tree=='gpt4':
611 | # GPT4
612 | # # Step 1: define criterions of the decision tree.
613 | criterions = {
614 | 'is_aggressive':'This speech contains aggressive language',
615 | 'is_targeting_group':'This speech targets a specific group',
616 | 'is_explicit_discrimination':'This speech contains explicit discrimination',
617 | 'has_covert_prejudice': 'This speech has covert prejudice or bias'
618 | }
619 |
620 | # Step 2: define the balanced decision tree for this classification task
621 | tree = {
622 | 'root': 'is_aggressive',
623 | 'is_aggressive': {'yes': 'is_targeting_group', 'no': 'has_covert_prejudice'},
624 | 'is_targeting_group': {'yes': 'this is hate', 'no': 'is_explicit_discrimination'},
625 | 'is_explicit_discrimination': {'yes': 'this is hate', 'no': 'this is noHate'},
626 | 'has_covert_prejudice': {'yes': 'this is hate', 'no': 'this is noHate'}
627 | }
628 | elif which_tree == 'human':
629 | # Human Tree
630 | # Step 1: define criterions of the decision tree.
631 | criterions = {
632 | 'oi_words': 'The speech includes offensive words about minorities',
633 | 'identity_words': 'The sentence includes mention of identities',
634 | 'swear_words': 'The sentence includes swear words',
635 | 'negative_words':'The sentence includes negative words',
636 | }
637 |
638 | # Step 2: define the balanced decision tree for this classification task
639 | tree = {
640 | 'root': 'oi_words',
641 | 'oi_words': {'yes': 'this is hate', 'no': 'identity_words'},
642 | 'identity_words': {'yes': 'swear_words', 'no': 'this is noHate'},
643 | 'swear_words': {'yes': 'this is hate', 'no': 'negative_words'},
644 | 'negative_words': {'yes': 'this is hate', 'no': 'this is noHate'},
645 | }
646 | elif which_tree=='gpt3.5':
647 | # GPT3.5
648 | # Step 1: define criterions of the decision tree.
649 | criterions = {
650 | 'is_explicit_hate':'This speech contains explicit hate speech',
651 | 'is_derogatory_language':'This speech contains derogatory language'
652 | }
653 |
654 | # Step 2: define the balanced decision tree for this classification task
655 | tree = {
656 | 'root': 'is_explicit_hate',
657 | 'is_explicit_hate': {'yes': 'this is hate', 'no': 'is_derogatory_language'},
658 | 'is_derogatory_language': {'yes': 'this is hate', 'no': 'this is noHate'}
659 | }
660 | all_scores = {}
661 | for key in criterions.keys():
662 | sentence = criterions[key]
663 | score = get_entailment_score(sentence, model, tokenizer)
664 | all_scores[key] = score
665 | return criterions, tree, all_scores
666 |
667 |
668 | def get_hate_dataset():
669 | import os
670 | import csv
671 | import json
672 | # Paths to CSV file and test set folder
673 | csv_file_path = '../hate-speech-dataset/annotations_metadata.csv'
674 | test_folder_path = '../hate-speech-dataset/sampled_test/'
675 |
676 | # Read the CSV file and create a dictionary of file IDs and labels
677 | file_labels = {}
678 | with open(csv_file_path, 'r') as csv_file:
679 | csv_reader = csv.DictReader(csv_file)
680 | for row in csv_reader:
681 | print(row)
682 | file_id = row['file_id']
683 | label = row['label']
684 | file_labels[file_id] = label
685 |
686 | # Iterate over files in the test set folder
687 | data = []
688 | for filename in os.listdir(test_folder_path):
689 | if filename.endswith('.txt'):
690 | file_id = filename.split('.')[0]
691 | file_path = os.path.join(test_folder_path, filename)
692 |
693 | # Check if the file ID has a corresponding label
694 | if file_id in file_labels:
695 | label = file_labels[file_id]
696 | with open(file_path, 'r') as txt_file:
697 | sentence = txt_file.read().strip()
698 | print(f'File: {filename}\nLabel: {label}\nSentence: {sentence}\n')
699 | new_dict = {}
700 | new_dict['file'] = filename
701 | new_dict['label'] = label
702 | new_dict['sentence'] = sentence
703 | data.append(new_dict)
704 | else:
705 | print(f'No label found for File: {filename}\n')
706 | with open('../hate-speech-dataset/test_sample.json', 'w') as f:
707 | json.dump(data, f)
708 |
709 | def get_full_tree_hate_speech():
710 | with open('../hate-speech-dataset/test_sample.json', 'r') as f:
711 | data = json.load(f)
712 | criterions, tree, all_score = get_hate_tree()
713 | tot = 0
714 | acc = 0
715 | for item in tqdm(data):
716 | tot += 1
717 | sentence = item['sentence']
718 |
719 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
720 |
721 | if label == 'this is ' + item['label']:
722 | acc += 1
723 | print(acc/tot)
724 | # get_full_tree_hate_speech()
725 |
726 |
727 | def get_baseline_hate_speech():
728 | with open('../hate-speech-dataset/test_sample.json', 'r') as f:
729 | data = json.load(f)
730 | tot = 0
731 | acc = 0
732 | hate_tuple = get_entailment_score('this is hate ', model, tokenizer)
733 | nonhate_tuple = get_entailment_score('this is not hate ', model, tokenizer)
734 | for item in tqdm(data):
735 | tot += 1
736 | sentence = item['sentence']
737 |
738 | hatelabel, hateval = entailment('this is hate ', sentence, model, tokenizer, hate_tuple, True)
739 | nohatelabel, nohateval = entailment('this is not hate ', sentence, model, tokenizer, nonhate_tuple, True)
740 | if hateval > nohateval:
741 | label = 'hate'
742 | else:
743 | label = 'noHate'
744 |
745 | if label == item['label']:
746 | acc += 1
747 | print(acc/tot)
748 |
749 |
750 |
751 | POSSIBLE_CLASSES = ['Yes','Maybe', 'No']
752 | def get_bias_offense_tree(which_tree='gpt4', use_model='deberta'):
753 | if which_tree=='gpt4':
754 | if use_model=='roberta':
755 | # Step 1: define criterions of the decision tree.
756 | criterions = {
757 | 'is_hate_speech':'This post contains hate speech',
758 | 'is_disrespectful':'This post is disrespectful',
759 | 'has_strong_language':'This post contains strong language',
760 | }
761 |
762 | # Step 2: define the balanced decision tree for this classification task
763 | tree = {
764 | 'root': 'is_hate_speech',
765 | 'is_hate_speech': {'yes': 'Yes', 'no': 'is_disrespectful'},
766 | 'is_disrespectful': {'yes': 'Maybe', 'no': 'has_strong_language'},
767 | 'has_strong_language': {'yes': 'Maybe', 'no': 'No'},
768 | }
769 | else:
770 | # Step 1: define criterions of the decision tree.
771 | criterions = {
772 | 'is_politically_incorrect':'This is politically incorrect',
773 | 'includes_offensive_words':'This includes offensive words',
774 | 'is_discriminating':'This is discriminative'
775 | }
776 |
777 | # Step 2: define the balanced decision tree for this task.
778 | tree = {
779 | 'root': 'is_politically_incorrect',
780 | 'is_politically_incorrect': {'yes': 'includes_offensive_words', 'no': 'is_discriminating'},
781 | 'includes_offensive_words': {'yes': 'Yes', 'no': 'Maybe'},
782 | 'is_discriminating': {'yes': 'Maybe', 'no': 'No'}
783 | }
784 | elif which_tree == 'human'
785 | # Human Tree
786 | # Step 1: define criterions of the decision tree.
787 | criterions = {
788 | 'racist_words': 'The post includes racist words',
789 | 'sexist_words': 'The post includes sexist words',
790 | 'swear_words': 'The post includes swear words',
791 | 'identity_words': 'The post includes mention of identities',
792 | 'negative_words':'The post include negative words',
793 | }
794 |
795 | # Step 2: define the balanced decision tree for this classification task
796 | tree = {
797 | 'root': 'racist_words',
798 | 'racist_words': {'yes': 'Yes', 'no': 'sexist_words'},
799 | 'sexist_words': {'yes': 'Yes', 'no': 'swear_words'},
800 | 'swear_words': {'yes': 'Yes', 'no': 'identity_words'},
801 | 'identity_words': {'yes': 'negative_words', 'no': 'Maybe'},
802 | 'negative_words': {'yes': 'Maybe', 'no': 'No'}
803 | }
804 | elif which_tree=='gpt3.5':
805 | # GPT3.5
806 | # Step 1: define criterions of the decision tree.
807 | criterions = {
808 | 'contains_offensive_language':'This post contains offensive language',
809 | 'intention_to_offend':'The intention of the post is to offend',
810 | 'tone_neutral':'The tone of the post is neutral',
811 | 'tone_ambiguous':'The tone of the post is ambiguous'
812 | }
813 |
814 | # Step 2: define the balanced decision tree for this classification task
815 | tree = {
816 | 'root': 'contains_offensive_language',
817 | 'contains_offensive_language': {'yes': 'intention_to_offend', 'no': 'tone_neutral'},
818 | 'intention_to_offend': {'yes': 'Yes', 'no': 'Maybe'},
819 | 'tone_neutral': {'yes': 'No', 'no': 'tone_ambiguous'},
820 | 'tone_ambiguous': {'yes': 'Maybe', 'no': 'No'}
821 | }
822 | all_scores = {}
823 | for key in criterions.keys():
824 | sentence = criterions[key]
825 | score = get_entailment_score(sentence, model, tokenizer)
826 | all_scores[key] = score
827 | return criterions, tree, all_scores
828 |
829 |
830 | def get_full_tree_social_speech():
831 | dataset = load_dataset('social_bias_frames')
832 | val_dataset = dataset["validation"]
833 | criterions, tree, all_score = get_bias_offense_tree()
834 | tot = 0
835 | acc = 0
836 | for item in tqdm(val_dataset):
837 | tot += 1
838 | sentence = item['post']
839 |
840 | label, val = tree_predict(sentence, criterions, tree, model, tokenizer, all_score, True)
841 | truth = item['offensiveYN']
842 |
843 | if truth == '1.0':
844 | truth = 'Yes'
845 | elif truth == '0.5':
846 | truth = 'Maybe'
847 | else:
848 | truth = 'No'
849 |
850 | if label == truth:
851 | acc += 1
852 |
853 | print(acc/tot)
854 |
855 |
856 |
857 | print(acc/tot)
858 |
859 | def get_full_tree_social_speech_baseline():
860 | dataset = load_dataset('social_bias_frames')
861 | val_dataset = dataset["validation"]
862 | tot = 0
863 | acc = 0
864 | hate_tuple = get_entailment_score('this is offensive ', model, tokenizer)
865 | nonhate_tuple = get_entailment_score('this is not offensive ', model, tokenizer)
866 | maybehate_tuple = get_entailment_score('this is maybe offensive ', model, tokenizer)
867 | for item in tqdm(val_dataset):
868 | tot += 1
869 | sentence = item['post']
870 |
871 | hatelabel, hateval = entailment('this is offensive ', sentence, model, tokenizer, hate_tuple, True)
872 | nohatelabel, nohateval = entailment('this is not offensive ', sentence, model, tokenizer, nonhate_tuple, True)
873 | maybehatelabel, maybehateval = entailment('this is maybe offensive ', sentence, model, tokenizer, maybehate_tuple, True)
874 | if hateval > nohateval and hateval > maybehateval:
875 | label = 'Yes'
876 | elif nohateval > hateval and nohateval > maybehateval:
877 | label = 'No'
878 | else:
879 | label = 'Maybe'
880 | truth = item['offensiveYN']
881 | if truth == '1.0':
882 | truth = 'Yes'
883 | elif truth == '0.5':
884 | truth = 'Maybe'
885 | else:
886 | truth = 'No'
887 | if label == truth:
888 | acc += 1
889 |
890 | print(acc/tot)
891 |
892 |
--------------------------------------------------------------------------------
/research/nlu/tree_NLEP/tree_NLEP_generation.py:
--------------------------------------------------------------------------------
1 | import openai
2 | import backoff
3 | openai.api_key_path = './api_key.txt'
4 | prompt_head = """Write a Python function that constructs a decision tree according to the given examples that can generate the correct label of the given classification task.
5 |
6 | ### Available APIs(shared for all tasks):
7 |
8 | \"\"\"Returns whether the hypothesis in entailed by the premise.\"\"\"
9 | def entailment(hypothesis, premise, model, tokenizer):
10 | proposition = f'{hypothesis} is entailed by {premise}.'
11 | inputs = tokenizer(proposition, return_tensors="pt", truncation=True, padding=True, max_length=128)
12 | outputs = model(**inputs)['logits'][0]
13 | ent_label = int(outputs[0] > outputs[2])
14 | if ent_label == 1:
15 | return 'yes'
16 | else:
17 | return 'no'
18 |
19 | \"\"\"Use the constructed decision tree to predict the label of the sentence.\"\"\"
20 | def tree_predict(sentence, criterions, tree, model, tokenizer):
21 | node = tree['root']
22 | while node not in POSSIBLE_CLASSES:
23 | ent_label = entailment(criterions[node], sentence, model, tokenizer)
24 | node = tree[node][ent_label]
25 | return node
26 |
27 | ### Task: Movie review classification
28 | ### Description: Determine if a movie review expresses positive attitude or negative attitude.
29 | ### Possible classes: [positive, negative]
30 | ### Examples:
31 | - contains no wit, only labored gags
32 | - [The movie is wise|The movie is not wise|1], [the story is fun|the story is not boring|1], [the review is positive|the review is negative|1]
33 | - that loves its characters and communicates something rather beautiful about human nature
34 | - [The characters are lovely|The characters are awful|0], [the script is touching|the script is dry|0], [the review is positive|the review is negative|0]
35 | - on the worst revenge-of-the-nerds clichés the filmmakers could dredge up
36 | - [The movie is novel|The movie is mostly platitudes|1], [the review is negative|1]
37 | - are more deeply thought through than in most right-thinking films
38 | - [The takeaway of the movie is profound|The idea of the movie is shallow|0], [the review is positive|the review is negative|0]
39 |
40 | ### Define possible classes
41 | POSSIBLE_CLASSES = ['positive', 'negative']
42 |
43 | ### Decision Tree Logic:
44 | Start by assessing whether the movie is interesting.
45 | - If uninteresting:
46 | Probe for depth by determining if the movie is wise.
47 | - If wise, label as a positive review.
48 | - Otherwise, label as a negative review.
49 | - If interesting:
50 | Examine the quality of the script.
51 | - If the script is commendable, label as a positive review.
52 | - If not a good script
53 | Evaluate the portrayal of characters.
54 | - If the characters are well-depicted, label the review as positive.
55 | - If characters are not good, label the review negative.
56 |
57 | ### Python program:
58 | \"""Decision tree for this task\"""
59 | def get_decision_tree(sentence, model, tokenizer):
60 | # Step 1: define criterions of the decision tree.
61 | criterions = {
62 | 'is_interesting':'This movie is interesting',
63 | 'is_good_script':'The movie has a good script',
64 | 'is_good_character':'The characters are awsome',
65 | 'is_wise': 'This movie is wise'
66 | }
67 |
68 | # Step 2: define the balanced decision tree for this classification task
69 | tree = {
70 | 'root': 'is_interesting',
71 | 'is_interesting': {'yes': 'is_good_script', 'no': 'is_wise'},
72 | 'is_good_script': {'yes': 'positive', 'no': 'is_good_character'},
73 | 'is_good_character': {'yes': 'positive', 'no': 'negative'},
74 | 'is_wise': {'yes': 'positive', 'no': 'negative'}
75 | }
76 |
77 | return criterions, tree
78 |
79 | """
80 |
81 | prompt_head2 = """
82 | Write a Python function that constructs a decision tree according to the given examples that can generate the correct label of the given classification task.
83 |
84 | ### Available APIs(shared for all tasks):
85 |
86 | \"\"\"Returns whether the hypothesis in entailed by the premise.\"\"\"
87 | def entailment(hypothesis, premise, model, tokenizer):
88 | proposition = f'{hypothesis} is entailed by {premise}.'
89 | inputs = tokenizer(proposition, return_tensors="pt", truncation=True, padding=True, max_length=128)
90 | outputs = model(**inputs)['logits'][0]
91 | ent_label = int(outputs[0] > outputs[2])
92 | if ent_label == 1:
93 | return 'yes'
94 | else:
95 | return 'no'
96 |
97 | \"\"\"Use the constructed decision tree to predict the label of the sentence.\"\"\"
98 | def tree_predict(sentence, criterions, tree, model, tokenizer):
99 | node = tree['root']
100 | while node not in POSSIBLE_CLASSES:
101 | ent_label = entailment(criterions[node], sentence, model, tokenizer)
102 | node = tree[node][ent_label]
103 | return node
104 |
105 | ### Task: Grammar correctness classification
106 | ### Possible classes: ['accpetable', 'unacceptable']
107 |
108 | ### Define possible classes
109 | POSSIBLE_CLASSES = ['accpetable', 'unacceptable']
110 |
111 | ### Decision Tree Logic:
112 | - If verbs are not correctly constructed, the sentence is immediately labeled as unacceptable.
113 | - If verbs are correct:
114 | The tree then checks if the sentence has correct punctuation
115 | - If incorrect, label the sentence as unacceptable
116 | - If correct:
117 | The next criterion to be assessed is the subject-verb agreement.
118 | - If subject and verb disagree, label the sentence as unacceptable.
119 | - If they agree, check for sentence fragments.
120 | - If the sentence is a fragment, label it as unacceptable.
121 | - If it is not a sentence fragment, label the sentence as acceptable.
122 |
123 | ### Python code for the decision tree:
124 |
125 | ```python
126 | def get_decision_tree(sentence, model, tokenizer):
127 | # Step 1: define criterions of the decision tree
128 | criterions = {
129 | 'correct_verbs': 'The verbs are correctly constructed in the sentence',
130 | 'correct_punctuation': 'The sentence is punctuated correctly',
131 | 'subject_verb_agreement': 'The subject and verb agree in the sentence',
132 | 'no_sentence_fragments': 'The sentence is not a fragment',
133 | }
134 |
135 | # Step 2: define the balanced decision tree for this classification task
136 | tree = {
137 | 'root': 'correct_verbs',
138 | 'correct_verbs': {'yes': 'correct_punctuation', 'no': 'unacceptable'},
139 | 'correct_punctuation': {'yes': 'subject_verb_agreement', 'no': 'unacceptable'},
140 | 'subject_verb_agreement': {'yes': 'no_sentence_fragments', 'no': 'unacceptable'},
141 | 'no_sentence_fragments': {'yes': 'acceptable', 'no': 'unacceptable'}
142 | }
143 |
144 | return criterions, tree
145 | ```
146 |
147 | """
148 |
149 | prompt_sst2 = """
150 | ### Task: Movie review classification
151 | ### Possible classes: [positive, negative]
152 |
153 | ### Define possible classes
154 | POSSIBLE_CLASSES = [positive, negative]
155 |
156 | ### Decision Tree Logic:
157 |
158 | """
159 | prompt_cola ="""
160 | ### Task: Grammar correctness classification
161 | ### Possible classes: [acceptable, unacceptable]
162 |
163 | ### Define possible classes
164 | POSSIBLE_CLASSES = ['acceptable', 'unacceptable']
165 |
166 | ### Decision Tree Logic:
167 |
168 | """
169 |
170 | prompt_qnli ="""
171 | ### Task: the sentence answers the question classification
172 | ### Description: Determine if the sentence is an answer to the question
173 | ### Possible classes: [true, false]
174 |
175 | ### Define possible classes
176 | POSSIBLE_CLASSES = ['true', 'false']
177 |
178 | ### Python program:
179 | \"""Decision tree for this task\"""
180 | Try to generate a balanced tree!
181 | def get_decision_tree(sentence, model, tokenizer):
182 |
183 | """
184 |
185 | prompt_emotion_separate ="""
186 | ### Task: Sadness classification
187 | ### Description: Assume that "I" wrote a sentence, determine if "I" am feeling sad.
188 | ### Possible classes: ['Yes', 'No']
189 |
190 | ### Define possible classes
191 | POSSIBLE_CLASSES = ['Yes', 'No']
192 |
193 | ### Python program:
194 | \"""Decision tree for this task\"""
195 | Try to generate a balanced tree!
196 | def get_decision_tree(sentence, model, tokenizer):
197 |
198 | """
199 |
200 | prompt_emotion ="""
201 | ### Task: Emotion classification
202 | ### Description: I wrote a sentence about my feeling, determine which emotion am I feeling.
203 | ### Possible classes: ['I feel sad', 'I feel happy', 'I feel love', 'I feel angry', 'I feel afraid', 'I feel surprised']
204 |
205 | ### Define possible classes
206 | POSSIBLE_CLASSES = ['I feel sad', 'I feel happy', 'I feel love', 'I feel angry', 'I feel afraid', 'I feel surprised']
207 |
208 | ### Decision Tree Logic:
209 |
210 | """
211 | prompt_amazon = """
212 | ### Task: Amazon Review Star classification
213 | ### Description: Given an amazon review, determine the star(1-5) of this review, where 1 is the worst and 5 is the best.
214 | ### Possible classes: ['1', '2', '3', '4', '5']
215 |
216 | ### Define possible classes
217 | POSSIBLE_CLASSES = ['1', '2', '3', '4', '5']
218 |
219 | ### Decision Tree Logic:
220 |
221 | """
222 |
223 | prompt_agnews = """
224 | ### Task: News content classification
225 | ### Description: Given a piece of news, determine which category it belongs to.
226 | ### Possible classes: ['this is world news', 'this is sport news', 'this is business news', 'this is technology news']
227 |
228 |
229 | ### Define possible classes
230 | POSSIBLE_CLASSES = ['this is world news', 'this is sport news', 'this is business news', 'this is technology news']
231 |
232 | ### Start each criterion with "This is"
233 | ### Each criterion should be general/high-level!
234 | ### Python program:
235 | \"""Decision tree for this task\"""
236 | Try to generate a balanced tree!
237 | def get_decision_tree(sentence, model, tokenizer):
238 |
239 | """
240 |
241 | prompt_agnews_separate = """
242 | ### Task: Science news classification
243 | ### Description: Given a piece of news, determine if this is a science news.
244 | ### Possible classes: ['Yes', 'No']
245 |
246 | ### Define possible classes
247 | POSSIBLE_CLASSES = ['Yes', 'No']
248 |
249 | ### Decision Tree Logic:
250 |
251 | """
252 |
253 | prompt_hate_speech = """
254 | ### Task: Bias/Hateful speech classification
255 | ### Description: Given a speech from a white supremacy forum, determine if this is speech contains hate (eg. bias/racisim/prejudice...) towards a certain group of people.
256 | ### Possible classes: ['this is hate', 'this is noHate']
257 |
258 | ### Define possible classes
259 | POSSIBLE_CLASSES = ['this is hate', 'this is noHate']
260 |
261 | ### Each criterion should be simple, clear, and concise.
262 | ### Start each criterion with 'This is'
263 | ### Decision Tree Logic:
264 |
265 | """
266 |
267 | prompt_social_bias_offensive = """
268 | ### Task: Offensive classification
269 | ### Description: Given a post, determine if this post is offensive.
270 | ### Possible classes: ['Yes','Maybe', 'No']
271 |
272 | ### Define possible classes
273 | POSSIBLE_CLASSES = ['Yes','Maybe', 'No']
274 |
275 | ### Each criterion should be simple, clear, and concise.
276 | ### Start each criterion with 'This is'
277 | ### Generate at least 5 criterions!
278 | ### Decision Tree Logic:
279 | """
280 | prompt_social_bias_offensive_deberta = """
281 | ### Task: Offensive classification
282 | ### Description: Given a post, determine if this post is potentially offensive to anyone.
283 | ### Possible classes: ['this is offensive','this is not offensive', 'this is maybe offensive']
284 |
285 | ### Define possible classes
286 | POSSIBLE_CLASSES = ['this is offensive','this is not offensive', 'this is maybe offensive']
287 |
288 | ### Each criterion should be simple, clear, and concise.
289 | ### Start each criterion with 'This is'
290 | ### Decision Tree Logic:
291 | """
292 | @backoff.on_exception(backoff.expo, openai.error.RateLimitError)
293 | def completions_with_backoff(**kwargs):
294 | return openai.ChatCompletion.create(**kwargs)
295 |
296 | model_name = 'gpt-4'
297 | # To generate a tree for SST2, use prompt_head2 instead of prompt_head
298 | content = prompt_head + prompt_social_bias_offensive
299 | response = completions_with_backoff(
300 | model=model_name,
301 | messages=[
302 | {"role": "user","content": content}
303 | ],
304 | )
305 | response = response["choices"][0]["message"]["content"]
306 | print(response)
--------------------------------------------------------------------------------
/server/nlep_server.py:
--------------------------------------------------------------------------------
1 | import openai
2 | import google.generativeai as palm
3 |
4 | import csv
5 |
6 | from fastapi import FastAPI, UploadFile, File
7 | from fastapi.middleware.cors import CORSMiddleware
8 |
9 | from pydantic import BaseModel
10 | import os
11 | from urllib.parse import urlparse
12 |
13 |
14 | app = FastAPI()
15 |
16 | fs_prompt = open('prompts.txt').read()
17 |
18 | app.add_middleware(
19 | CORSMiddleware,
20 | allow_origins=["*"],
21 | allow_credentials=True,
22 | allow_methods=["*"],
23 | allow_headers=["*"],
24 | )
25 |
26 |
27 | class Item(BaseModel):
28 | instruction: str
29 | api_key_str: str
30 | exist_code: str
31 | platform: str
32 | model: str
33 |
34 |
35 | def parse_api_key(api_key_str):
36 | print(api_key_str)
37 | if 'api_key =' in api_key_str:
38 | return api_key_str.split('api_key =')[1].replace('\'', '').strip()
39 | else:
40 | return api_key_str
41 |
42 |
43 | def construct_fspy_prompt(fs_prompt, inst_str, input_txt = 'None', exist_code = 'none'):
44 | if exist_code == 'none':
45 | prompt = f'''{fs_prompt}\n\n### Instruction: {inst_str}\n### Input: {input_txt}\n### Python program:'''
46 | else:
47 | prompt = f'''{fs_prompt}\n\n### Instruction: {inst_str}\n### Input: {input_txt}\n### Python program:\n```\n{exist_code.strip()}'''
48 | return prompt
49 |
50 |
51 | def gpt4_py(ques_str, api_key_str, exist_code, platform, model='gpt-4'):
52 | api_key = parse_api_key(api_key_str)
53 |
54 | prompt = construct_fspy_prompt(fs_prompt, ques_str, exist_code = exist_code)
55 |
56 | if platform == 'gpt':
57 | openai.api_key = api_key
58 |
59 | gpt4_output = openai.ChatCompletion.create(
60 | model = model,
61 | messages = [{'role': 'user', 'content': prompt}],
62 | temperature = 0.5,
63 | top_p = 1.0,
64 | max_tokens = 1024
65 | )
66 | gen_txt = gpt4_output['choices'][0]['message']['content'].replace('```python', '```')
67 |
68 | elif platform == 'palm':
69 | palm.configure(api_key = api_key)
70 | completion = palm.generate_text(
71 | model = f'models/{model}',
72 | prompt = prompt,
73 | temperature = 0.5,
74 | # The maximum length of the response
75 | max_output_tokens = 1024,
76 | )
77 | gen_txt = completion.result
78 |
79 | else:
80 | return 'Platform not supported.'
81 |
82 | if exist_code != 'none' and gen_txt.startswith('```'):
83 | gen_txt = gen_txt[3:]
84 |
85 | if exist_code != 'none':
86 | gen_txt = f'\n{gen_txt}'
87 |
88 | res_str = (
89 | prompt + gen_txt
90 | ).split('### Python program:')[-1].strip()
91 |
92 | sections = res_str.split('```')
93 | if len(sections) > 1:
94 | ans_str = sections[1].strip()
95 | else:
96 | ans_str = sections[0].strip()
97 |
98 | return ans_str
99 |
100 |
101 | @app.post("/process-file/")
102 | async def process_file(file: UploadFile = File(...)):
103 | # Do something with the file
104 | content = await file.read()
105 | content_list = content.decode('utf-8').split('\n')[:3]
106 | content_str = '\n# '.join(content_list)
107 | prompt = f"# First three rows of the input file:\n# {content_str}\n# ...\nfile_path = 'cache/{file.filename}' # Please fill\ninput_file = open(file_path)"
108 | # For demonstration purposes, we'll just return the filename
109 | return {"filename": file.filename, "message": f"{prompt}"}
110 |
111 |
112 |
113 | @app.put("/items/{item_id}")
114 | async def update_item(item_id: int, item: Item):
115 | ans_str = gpt4_py(
116 | item.instruction, item.api_key_str, item.exist_code,
117 | item.platform, item.model
118 | )
119 |
120 | return {"output": ans_str, "item_id": item_id}
--------------------------------------------------------------------------------
/server/prompt.txt:
--------------------------------------------------------------------------------
1 | You are an AI model. Write a bug-free Python program that can generate factual and correct answer to the given instruction when correctly executed. Do not ask for user input. Answer the question directly and do not add unrelated information or comments. Do not make any http request.
2 |
3 | ### Instruction: Calculate the total surface area of a cube with a side length of 5 cm.
4 | ### Input: None
5 | ### Python program:
6 | ```
7 | # Step 1: Import necessary built-in libraries
8 | # No need to import
9 |
10 | # Step 2: Define a function that calculate the surface area of cubes
11 | def calculate_surface_area(side_length):
12 | return 6 * (side_length ** 2)
13 |
14 | # Step 3: Define dictionaries storing the cube information
15 | cube = {
16 | "side_length": 5 # Side length of the cube
17 | }
18 |
19 | # Step 4: Print a step-by-step calculation answer in natural language using the defined function and varible
20 | side_length = cube["side_length"]
21 | surface_area = calculate_surface_area(side_length)
22 | print(f"The surface area of a cube is found by calculating the area of one of its faces and multiplying it by six (since a cube has six faces). The area of a cube face is simply its side length squared.\n")
23 | print(f"Thus for this particular cube:")
24 | print(f"Surface Area = 6 × (Side Length)²")
25 | print(f" = 6 × ({side_length} cm)²")
26 | print(f" = 6 × {side_length**2} cm²")
27 | print(f" = {surface_area} cm²\n")
28 | print(f"The total surface area of this cube is {surface_area} square centimeters.")
29 | ```
30 |
31 | ### Instruction: Identify the odd one out.
32 | ### Input: Twitter, Instagram, Telegram
33 | ### Python program:
34 | ```
35 | # Step 1: Import necessary built-in libraries
36 | from collections import OrderedDict
37 |
38 | # Step 2: Define dictionaries storing the knowledge about the main function of each application
39 | services = {
40 | "Twitter": "a social media platform mainly for sharing information, images and videos",
41 | "Instagram": "a social media platform mainly for sharing information, images and videos",
42 | "Telegram": "a cloud-based instant messaging and voice-over-IP service",
43 | }
44 |
45 | # Step 3: Define a function that finds the different application
46 | def find_odd_one_out(services, input_services):
47 | descriptions = [services[service] for service in input_services]
48 | for description in descriptions:
49 | if descriptions.count(description) == 1:
50 | return input_services[descriptions.index(description)]
51 | return None
52 |
53 | # Step 4: Print an answer in natural language using the knowledge and function defined above
54 | input_services = ["Twitter", "Instagram", "Telegram"]
55 | odd_one_out = find_odd_one_out(services, input_services)
56 | if odd_one_out:
57 | other_services = [service for service in input_services if service != odd_one_out]
58 | print(f"The odd one out is {odd_one_out}. {other_services[0]} and {other_services[1]} are {services[other_services[0]]} while {odd_one_out} is {services[odd_one_out]}.")
59 | ```
--------------------------------------------------------------------------------
/server/readme.md:
--------------------------------------------------------------------------------
1 | # NLEP Server
2 |
3 | The complete source code of building an NLEP API is included in this folder by running the following command
4 | ```python
5 | uvicorn nlep_server:app --reload --host 0.0.0.0 --port 8000
6 | ```
7 | The LangCode agent can be connected to the preferred API endpoint via
8 | ```python
9 | # For example, clp is a LangCode agent
10 | clp.config_api_endpoint()
11 | ```
12 | and input your url in this format: `http://{HOSTNAME}:{PORT}/items/0`
13 |
14 | This server does not set any timeout.
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | from setuptools import setup, find_packages
2 |
3 | setup(
4 | name='langpy-notebook',
5 | version='0.4',
6 | packages=find_packages(),
7 | install_requires=[
8 | # Any dependencies your project needs, e.g.
9 | # 'requests',
10 | 'IPython',
11 | 'notebook==6.4.8'
12 | ],
13 | author='Hongyin Luo',
14 | author_email='hyluo@mit.edu',
15 | description='Chat in Python on IPython notebook.',
16 | long_description=open('README.md').read(),
17 | long_description_content_type='text/markdown',
18 | url='https://github.com/luohongyin/langpy-notebook',
19 | classifiers=[
20 | 'Programming Language :: Python :: 3',
21 | 'License :: OSI Approved :: MIT License',
22 | 'Operating System :: OS Independent',
23 | ],
24 | )
25 |
--------------------------------------------------------------------------------