├── .gitignore ├── README.md ├── assets └── FIND_overview.png ├── requirements.txt └── src ├── __init__.py ├── evaluate_interpretations ├── README.md ├── mse_indicator.py ├── string_indicator.py ├── unit_testing.py └── utils │ ├── eval_data_neurons_entities.json │ ├── eval_data_neurons_relations.json │ ├── eval_data_strings.json │ ├── prompt_eval_entities.txt │ ├── prompt_eval_relations.txt │ └── prompt_eval_strings.txt ├── find-dataset ├── .gitignore ├── neurons_entities │ ├── data.json │ ├── f00000 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00001 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00002 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00003 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00004 │ │ ├── function_code.py │ │ └── initial.json │ └── unit_test_data.json ├── neurons_relations │ ├── data.json │ ├── f00000 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00001 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00002 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00003 │ │ ├── function_code.py │ │ └── initial.json │ ├── f00004 │ │ ├── function_code.py │ │ └── initial.json │ └── unit_test_data.json ├── numeric │ ├── data.json │ ├── f00000 │ │ ├── __pycache__ │ │ │ └── function_code.cpython-38.pyc │ │ ├── function_code.py │ │ └── mlp_approx_model.pt │ ├── f00001 │ │ ├── __pycache__ │ │ │ └── function_code.cpython-38.pyc │ │ ├── function_code.py │ │ └── mlp_approx_model.pt │ ├── f00002 │ │ ├── __pycache__ │ │ │ └── function_code.cpython-38.pyc │ │ ├── function_code.py │ │ └── mlp_approx_model.pt │ ├── f00003 │ │ ├── __pycache__ │ │ │ └── function_code.cpython-38.pyc │ │ ├── function_code.py │ │ └── mlp_approx_model.pt │ ├── f00004 │ │ ├── __pycache__ │ │ │ └── function_code.cpython-38.pyc │ │ ├── function_code.py │ │ └── mlp_approx_model.pt │ └── func_name_dict.json └── strings │ ├── f00000 │ └── function_code.py │ ├── f00001 │ └── function_code.py │ ├── f00002 │ └── function_code.py │ ├── f00003 │ └── function_code.py │ ├── f00004 │ └── function_code.py │ ├── test_data.json │ └── unit_test_data.json ├── make_functions ├── make_numeric │ ├── README.md │ ├── generate_functions.py │ ├── load_mlp.py │ ├── math_functions.py │ ├── train_functions.py │ ├── utils.py │ └── write_source_code.py └── make_strings │ ├── README.md │ ├── __pycache__ │ ├── string_functions.cpython-38.pyc │ └── write_source_code.cpython-38.pyc │ ├── generate_string_functions.py │ ├── string_functions.py │ ├── utils.py │ └── write_source_code.py ├── notebooks ├── .ipynb_checkpoints │ ├── example_interpretations-checkpoint.ipynb │ ├── example_interpretations_entities-checkpoint.ipynb │ ├── example_interpretations_numeric-checkpoint.ipynb │ ├── example_interpretations_relations-checkpoint.ipynb │ └── example_interpretations_strings-checkpoint.ipynb ├── example_interpretations.ipynb ├── example_interpretations_entities.ipynb ├── example_interpretations_numeric.ipynb ├── example_interpretations_relations.ipynb └── example_interpretations_strings.ipynb └── run_interpretations ├── README.md ├── call_interpreter.py ├── collect_interpretations.py └── prompts ├── neurons_entities.txt ├── neurons_relations.txt ├── numeric.txt ├── numeric_mlp.txt ├── prompts ├── neurons_entities.txt ├── neurons_relations.txt ├── numeric.txt ├── numeric_mlp.txt └── strings.txt └── strings.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # Don't track content of these folders: 2 | src/find_dataset/ 3 | src/run_interpretations/results/ 4 | src/make_functions/make_strings/string_functions/ 5 | src/make_functions/make_numeric/numeric_functions/ 6 | /src/run_interpretations/temp/ 7 | *.zip 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # FIND: Function Interpretation and Description Benchmark 2 | **A Function Interpretation Benchmark for Evaluating Interpretability Methods (official implementation)**
3 | ### [Project Page](https://multimodal-interpretability.csail.mit.edu/FIND-benchmark/) | [Arxiv](https://arxiv.org/abs/2309.03886) 4 | 5 | [Sarah Schwettmannn](https://cogconfluence.com/)\*, [Tamar Rott Shaham](https://tamarott.github.io/)\*, [Joanna Materzynska](https://joaanna.github.io/), [Neil Chowdhury](https://nchowdhury.com/), [Shuang Li](https://people.csail.mit.edu/lishuang/), [Jacob Andreas](https://www.mit.edu/~jda/), [David Bau](https://baulab.info/), [Antonio Torralba](https://groups.csail.mit.edu/vision/torralbalab/).
6 | \* equal contribution

7 | ![FIND overview](/assets/FIND_overview.png) 8 | 9 | **This repository is under active development, expect updates!**
10 | 11 | ## Setup 12 | 13 | Clone this repository: 14 | ```bash 15 | git clone https://github.com/multimodal-interpretability/FIND 16 | cd FIND 17 | ``` 18 | Install dependencies: 19 | ```bash 20 | pip install -r requirements.txt 21 | ``` 22 | Then download and unzip the FIND dataset into `./src/find-dataset`: 23 | 24 | 25 | ```bash 26 | wget -P ./src/find_dataset/ https://zenodo.org/record/8039658/files/FIND-dataset.zip 27 | unzip ./src/find_dataset/FIND-dataset.zip -d ./src/ 28 | ``` 29 | We include the dataset structure and examples of 5 functions per category under `./src/find_dataset/` 30 | 31 | ## Run interpretations 32 | To run the interpretation, run `cd ./src/run_interpretations/` and follow the instructions on the README file. 33 | The code will also allow you to add your own interpreter model. 34 | 35 | You can also download the full FIND interpretations benchmark and unzip it to `./src/run_interpretations/`: 36 | ```bash 37 | wget -P ./src/run_interpretations https://data.csail.mit.edu/FIND/FIND-interpretations.zip 38 | unzip ./src/run_interpretations/FIND-interpretations.zip -d ./src/run_interpretations/ 39 | ``` 40 | 41 | See interpretation examples at `./src/notebooks/example_interpretations.ipynb` 42 | 43 | ## Evaluate interpretations 44 | To evaluate the interpretations, run `cd ./src/evaluate_interpretations/` and follow the instructions on the README file. 45 | 46 | ## Generate new functions 47 | To generate a new set of numeric and/or strings functions, run `cd ./src/make_functions/` and follow the instructions on the README file. 48 | -------------------------------------------------------------------------------- /assets/FIND_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/assets/FIND_overview.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | ipython==8.11.0 2 | ipython==8.8.0 3 | matplotlib==3.7.1 4 | numpy==1.23.5 5 | openai==0.27.2 6 | pandas==1.5.2 7 | regex==2022.10.31 8 | scipy==1.10.1 9 | torch==1.13.0 10 | tqdm==4.64.1 11 | transformers==4.28.1 12 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- 1 | # empty 2 | -------------------------------------------------------------------------------- /src/evaluate_interpretations/README.md: -------------------------------------------------------------------------------- 1 | # FIND interpretation benchmark 2 | # Run interpretations 3 | 4 | ### dataset 5 | If you wish to use FIND functions for evaluation, download and unzip the FIND dataset into `./src/find-dataset`: 6 | 7 | ```bash 8 | wget -P ./src/find_dataset/ https://zenodo.org/record/8039658/files/FIND-dataset.zip 9 | unzip ./src/find_dataset/FIND-dataset.zip -d ./src/ 10 | ``` 11 | We include the dataset structure and examples of 5 functions per category under `./src/find_dataset/` 12 | 13 | If you whish to test FIND interpretations, download the full FIND interpretations benchmark and unzip it to `./src/run_interpretations/`: 14 | ```bash 15 | wget -P ./src/run_interpretations https://data.csail.mit.edu/FIND/FIND-interpretations.zip 16 | unzip ./src/run_interpretations/FIND-interpretations.zip -d ./src/run_interpretations/ 17 | ``` 18 | 19 | ## Test interpretations 20 | 21 | ### Code based - MSE indicator (for numeric functions) 22 | ```python mse_indicator.py``` 23 | 24 | ### Code based - string indicator (for string functions approximation) 25 | ```python string_indicator.py``` 26 | 27 | ### Language based - unit testing (for strings and synthetic neurons) 28 | 29 | Download the vicuna-evaluator model: 30 | 31 | ```wget https://data.csail.mit.edu/FIND/vicuna_evaluator.zip``` 32 | 33 | (the training data is available at: 34 | ```wget https://data.csail.mit.edu/FIND/vicuna_evaluator_train_data.json```) 35 | 36 | Install FastChat and the OpenAI API 37 | 38 | ```bash 39 | pip install fschat 40 | pip install openai 41 | ``` 42 | 43 | Then start serving vicuna-evaluator (run these three commands separately): 44 | 45 | ```bash 46 | python -m fastchat.serve.controller 47 | python -m fastchat.serve.model_worker --model-name 'vicuna-13b-v1.1' --model-path /path/to/vicuna-evaluator --num-gpus 2 48 | python -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 49 | ``` 50 | 51 | For string functions: 52 | ```python unit_testing.py --func_category strings --prompt_path ./utils/prompt_eval_strings.txt --eval_data_path ../find_dataset/strings/unit_test_data.json``` 53 | 54 | For synthetic neurons - entities: 55 | ```python unit_testing.py --func_category neurons_entities --prompt_path ./utils/prompt_eval_entities.txt --eval_data_path ../find_dataset/neurons_entities/unit_test_data.json``` 56 | include the flag `--hints` to evaluate interpretation with search initialization. 57 | include the flag `--single_round` for the MILAN settings. 58 | 59 | For synthetic neurons - relations: 60 | ```python unit_testing.py --func_category neurons_relations --prompt_path ./utils/prompt_eval_relations.txt --eval_data_path ../find_dataset/neurons_relations/unit_test_data.json``` 61 | include the flag `--hints` to evaluate interpretation with search initialization. 62 | include the flag `--single_round` for the MILAN settings. -------------------------------------------------------------------------------- /src/evaluate_interpretations/mse_indicator.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import openai 3 | import regex 4 | import re 5 | import pandas as pd 6 | import subprocess 7 | from getpass import getpass 8 | import pandas as pd 9 | import os 10 | from tqdm import tqdm 11 | from IPython import embed 12 | import time 13 | from random import random 14 | import json 15 | import matplotlib.pyplot as plt 16 | import sys 17 | import imp 18 | import numpy as np 19 | import math 20 | 21 | parser = argparse.ArgumentParser(description='Process Arguments') 22 | parser.add_argument('--model', type=str, default='gpt-4', help='model name') 23 | parser.add_argument('--func_category', type=str, default='numeric', help='function category') 24 | parser.add_argument('--test_path', type=str, default='../run_interpretations/results/', help='path to the interretations') 25 | parser.add_argument('--gt_path', type=str, default='../find_dataset/', help='path to the gt functions') 26 | parser.add_argument('--test_range', type=float, default=128, help='range to test the implementation:, the reange for input x is range(-x,x)') 27 | parser.add_argument('--nmse_threshold', type=float, default=0.1, help='threshold for the NMSE indicator') 28 | parser.add_argument('--nfunc', type=int, default=1000, help='number of functions') 29 | args = parser.parse_args() 30 | 31 | def list_mse(x,y): 32 | mse = [] 33 | normalization = [] 34 | for i in range(len(x)): 35 | mse.append(abs(x[i]-y[i])**2) 36 | normalization.append(abs(x[i])**2) 37 | return sum(mse)/len(mse), sum(mse)/sum(normalization) 38 | 39 | 40 | def get_test_func_name(path): 41 | f = open(path,'r') 42 | test_code = f.read() 43 | if test_code == '': return '' 44 | test_func_name = test_code.rsplit('def ')[-1] 45 | test_func_name = test_func_name.rsplit('(')[0] 46 | return test_func_name 47 | 48 | 49 | def main(args): 50 | t_start = time.time() 51 | counter = 0 52 | test_path_all = os.path.join(args.test_path,args.model,args.func_category) 53 | gt_path_all = os.path.join(args.gt_path,args.func_category) 54 | if os.path.exists(os.path.join(test_path_all,'test_res_data.json')): 55 | with open(os.path.join(test_path_all,'test_res_data.json'), 'r') as file: 56 | test_data = json.load(file) 57 | with open(os.path.join(test_path_all,'scores.json'), 'r') as file: 58 | scores_all = json.load(file) 59 | else: 60 | test_data = [] 61 | scores_all = [] 62 | 63 | # for function in tqdm(os.listdir(test_path_all)): 64 | for function_ind in tqdm(range(args.nfunc)): 65 | function = f"f{function_ind:05d}" 66 | if os.path.isfile(os.path.join(test_path_all,function)): continue 67 | path2save = os.path.join(test_path_all,function,'test/') 68 | os.makedirs(path2save,exist_ok=True) 69 | path2gt_func = os.path.join(gt_path_all,function,'function_code.py') 70 | path2test_func = os.path.join(test_path_all,function,'code.py') 71 | x = range(-args.test_range, args.test_range) 72 | func_gt = imp.load_source('function', path2gt_func) 73 | test_func_name = get_test_func_name(path2test_func) 74 | test_out = [] 75 | gt_out = [] 76 | x2plot = [] 77 | try: 78 | func_test = imp.load_source(test_func_name, path2test_func) 79 | for i in x: 80 | gt_val = func_gt.function(float(i+0.5)) 81 | if (not math.isnan(abs(gt_val))) and (not math.isinf(abs(gt_val))): 82 | gt_out.append(gt_val) 83 | exec(f"test_out.append(func_test.{test_func_name}({i+0.5}))") 84 | x2plot.append(i+0.5) 85 | mse,normlized_mse = list_mse(gt_out,test_out) 86 | if normlized_mse", where represents the numerical value of the correct example. 9 | Only include the numerical index of the example. 10 | -------------------------------------------------------------------------------- /src/evaluate_interpretations/utils/prompt_eval_relations.txt: -------------------------------------------------------------------------------- 1 | Below is a textual description of a function f(x): 2 | DESC 3 | Which one of the following pairs of input-output strings are correct executions of f(x)? 4 | Example 1: input = IN0, output = OUT0 5 | Example 2: input = IN1, output = OUT1 6 | Example 3: input = IN2, output = OUT2 7 | Reply in the following format: 8 | "[ANSWER]: ", where represents the numerical value of the correct example. 9 | Only include the numerical index of the example. 10 | -------------------------------------------------------------------------------- /src/evaluate_interpretations/utils/prompt_eval_strings.txt: -------------------------------------------------------------------------------- 1 | Below is a textual description of a function f(x): 2 | DESC 3 | Which one of the following pairs of input-output strings are correct executions of f(x)? 4 | Example 1: input = IN0, output = OUT0 5 | Example 2: input = IN1, output = OUT1 6 | Example 3: input = IN2, output = OUT2 7 | Reply in the following format: 8 | "[ANSWER]: ", where represents the numerical value of the correct example. 9 | Only include the numerical index of the example. 10 | -------------------------------------------------------------------------------- /src/find-dataset/.gitignore: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/data.json: -------------------------------------------------------------------------------- 1 | [{"dir": "f00000", "concept1": "sport organizations", "concept2": null, "initial": ["Metropolitan Museum of Art", "one third", "NFL", "hip-hop", "Japanese", "World Cup", "bus", "Mickey Mouse", "olympics", "webs"]}, {"dir": "f00001", "concept1": "corporations", "concept2": null, "initial": ["stock", "Game of Thrones", "CEO", "French Revolution", "Cold War", "chemistry", "grief", "tennis ", "airplane", "democracy"]}, {"dir": "f00002", "concept1": "elections", "concept2": null, "initial": ["weather", "Tour de France", "voting", "planets", "NFL", "ballot", "map", "Lake Huron", "geometry", "synagogues"]}, {"dir": "f00003", "concept1": "musical works", "concept2": null, "initial": ["digs", "minerals", "classical", "one third", "minutes", "artistic", "map", "piano", "Louvre", "Hamlet"]}, {"dir": "f00004", "concept1": "films", "concept2": null, "initial": ["faith", "Pulp Fiction", "minerals", "pitcher", "puck", "boats", "balance beam", "The Godfather", "train", "seconds"]}, {"dir": "f00005", "concept1": "climate", "concept2": null, "initial": ["stage", "forecast", "humidity", "burial", "Bitcoin", "travel", "Reddit", "Pulitzer Prize", "Mickey Mouse", "temperature"]}, {"dir": "f00006", "concept1": "computer hardware", "concept2": null, "initial": ["genome", "mountains", "map", "dribble", "red", "arch", "CPU", "WWE", "excavation", "RAM"]}, {"dir": "f00007", "concept1": "physics", "concept2": null, "initial": ["fish", "Tour de France", "innocent", "Buddha", "particles", "championships", "temperature", "space", "puck", "55%"]}, {"dir": "f00008", "concept1": "religions and beliefs", "concept2": null, "initial": ["plants", "Christianity", "arch", "olympics", "democracy", "rocks", "knowledge", "one third", "faith", "airplane"]}, {"dir": "f00009", "concept1": "law and justice", "concept2": null, "initial": ["swimming", "Nobel Prize", "television", "guilty", "rocket", "Berlin Wall", "ethics", "innocent", "NBA", "Fortnite"]}, {"dir": "f00010", "concept1": "transport", "concept2": null, "initial": ["tsunami", "classical", "grief", "folding", "Beverly Hills", "Louvre", "Nobel Prize", "bus", "Jesus", "car"]}, {"dir": "f00011", "concept1": "education", "concept2": null, "initial": ["school", "algorithms", "camera", "pilot", "Quran", "Bible", "college", "grief", "Nobel Prize", "World Cup"]}, {"dir": "f00012", "concept1": "politics", "concept2": null, "initial": ["synagogues", "balance beam", "democracy", "school", "government", "part of", "Mickey Mouse", "geometry", "female", "faith"]}, {"dir": "f00013", "concept1": "works of fiction", "concept2": null, "initial": ["highway", "France", "grief", "government", "lens", "Crime and Punishment", "Star Wars", "Allah", "classical", "rink"]}, {"dir": "f00014", "concept1": "games and leisure activities", "concept2": null, "initial": ["half", "helicoptor", "beacon", "boats", "driver", "55%", "puck", "card games", "television", "water"]}, {"dir": "f00015", "concept1": "biology", "concept2": null, "initial": ["WWE", "balance beam", "genome", "reservoir", "earthquake", "car", "factory", "diving", "cells", "The Godfather"]}, {"dir": "f00016", "concept1": "mathematics", "concept2": null, "initial": ["trends", "planes", "creativity", "artifacts", "algebra", "seconds", "Marvel", "geometry", "map", "voting"]}, {"dir": "f00017", "concept1": "geology", "concept2": null, "initial": ["Zeus ", "Michael Phelps", "lens", "expression", "championships", "young", "minerals", "feudalism", "rocks", "seconds"]}, {"dir": "f00018", "concept1": "government and state", "concept2": null, "initial": ["minerals", "rifle", "larva", "government", "Nazis", "Allah", "airplane", "ethics", "boats", "democracy"]}, {"dir": "f00019", "concept1": "food and eating", "concept2": null, "initial": ["light", "Amtrak", "taste", "earthquake", "planes", "contained in", "highway", "suspension", "geometry", "flavor"]}, {"dir": "f00020", "concept1": "air transport", "concept2": null, "initial": ["balance beam", "lanes", "travel", "Brooklyn Bridge", "suspension", "part of", "airplane", "pilot", "ocean", "farming"]}, {"dir": "f00021", "concept1": "road transport", "concept2": null, "initial": ["organs", "DC", "lanes", "badminton", "Zeus ", "55%", "vacation", "car", "deadly", "highway"]}, {"dir": "f00022", "concept1": "rail transport", "concept2": null, "initial": ["train", "tennis ball", "deposit", "farming", "forecast", "Bitcoin", "tennis ", "Amtrak", "classical", "sickness"]}, {"dir": "f00023", "concept1": "cycling", "concept2": null, "initial": ["geometry", "hypothesis", "Cold War", "old", "organs", "Eiffel Tower", "highway", "earthquake", "helmet", "Tour de France"]}, {"dir": "f00024", "concept1": "the New York Times", "concept2": null, "initial": ["green", "Hollywood", "borrowing", "Minecraft", "experiment", "expression", "red", "New Orleans", "editorial", "news"]}, {"dir": "f00025", "concept1": "aircraft", "concept2": null, "initial": ["World Cup", "driver", "helicoptor", "one third", "Jews", "Jesus", "airplane", "humidity", "borrow", "borrowing"]}, {"dir": "f00026", "concept1": "bodies of water", "concept2": null, "initial": ["map", "racing", "Japanese", "television", "Minecraft", "cells", "Pacific Ocean", "series", "software", "Atlantic Ocean"]}, {"dir": "f00027", "concept1": "mineralogy", "concept2": null, "initial": ["voice actors", "rhyme", "democracy", "crystals", "France", "guilty", "helicoptor", "minerals", "Breaking Bad", "Beverly Hills"]}, {"dir": "f00028", "concept1": "occupations", "concept2": null, "initial": ["rhyme", "lawyer", "doctor", "Atlantic Ocean", "equality", "guitar", "genome", "lawyer", "Japanese", "Eiffel Tower"]}, {"dir": "f00029", "concept1": "professional wrestling", "concept2": null, "initial": ["old", "championships", "blue", "Empire State Building", "WWE", "nature", "innocent", "puck", "Nirvana", "Romeo and Juliet"]}, {"dir": "f00030", "concept1": "a quantity indicating a percentage", "concept2": null, "initial": ["travel", "organs", "The New York Times", "helmet", "CPU", "83%", "55%", "Bitcoin", "green", "syntax"]}, {"dir": "f00031", "concept1": "plays", "concept2": null, "initial": ["female", "Brooklyn Bridge", "algebra", "space", "sickness", "Romeo and Juliet", "Allah", "Hamlet", "airplane", "earthquake"]}, {"dir": "f00032", "concept1": "horses", "concept2": null, "initial": ["roots", "rhyme", "saddle", "NFL", "democracy", "owner", "voting", "queen", "racing", "reference"]}, {"dir": "f00033", "concept1": "photography", "concept2": null, "initial": ["lawyer", "stanza", "Buckingham Palace", "borrow", "pitcher", "Michael Phelps", "lens", "camera", "CPU", "pawn"]}, {"dir": "f00034", "concept1": "music", "concept2": null, "initial": ["American Revolution", "script", "melody", "Atlantic Ocean", "rhythym", "airplane", "rocket", "pitcher", "security", "Lake Superior"]}, {"dir": "f00035", "concept1": "fashion", "concept2": null, "initial": ["style", "democracy", "taste", "camera", "trends", "Mickey Mouse", "Nirvana", "school", "temperature", "red"]}, {"dir": "f00036", "concept1": "chess", "concept2": null, "initial": ["security", "Walter White", "television", "tent", "lanes", "queen", "accessibility", "ocean", "New Orleans", "pawn"]}, {"dir": "f00037", "concept1": "racket sports", "concept2": null, "initial": ["tennis ", "doctor", "Disneyland", "badminton", "acting", "Passover", "female", "minerals", "pitcher", "tsunami"]}, {"dir": "f00038", "concept1": "art", "concept2": null, "initial": ["expression", "jazz", "stock", "female", "Amtrak", "old", "World Cup", "creativity", "swimming", "Jews"]}, {"dir": "f00039", "concept1": "disasters", "concept2": null, "initial": ["hip-hop", "water", "tsunami", "Nazis", "83%", "deadly", "Tour de France", "temperature", "earthquake", "borrow"]}, {"dir": "f00040", "concept1": "spacecraft", "concept2": null, "initial": ["biology", "Breaking Bad", "highway", "innocent", "rocket", "Louvre", "oxygen", "beacon", "democracy", "migration"]}, {"dir": "f00041", "concept1": "video games", "concept2": null, "initial": ["melody", "American Revolution", "Minecraft", "Fortnite", "beats", "migration", "seconds", "Bitcoin", "DC", "Pacific Ocean"]}, {"dir": "f00042", "concept1": "the relationship of an element to its class", "concept2": null, "initial": ["boats", "contained in", "part of", "55%", "synagogues", "France", "Game of Thrones", "body armor", "hypothesis", "trends"]}, {"dir": "f00043", "concept1": "basketball", "concept2": null, "initial": ["rifle", "dribble", "tent", "weather", "Game of Thrones", "pawn", "RAM", "NBA", "Crime and Punishment", "vacation"]}, {"dir": "f00044", "concept1": "winter sports", "concept2": null, "initial": ["minutes", "World Cup", "snowboarding", "syntax", "WWE", "artifacts", "skiing", "feudalism", "bus", "CEO"]}, {"dir": "f00045", "concept1": "American football", "concept2": null, "initial": ["female", "NFL", "touchdown", "flowers", "tennis ", "lens", "factory", "The New York Times", "physics", "Mickey Mouse"]}, {"dir": "f00046", "concept1": "golf", "concept2": null, "initial": ["Christianity", "outgoing", "highway", "swimming", "driver", "Bible", "owner", "PGA", "government", "beacon"]}, {"dir": "f00047", "concept1": "baseball", "concept2": null, "initial": ["pitcher", "Breaking Bad", "migration", "batter", "Eiffel Tower", "Amtrak", "reporting", "Beverly Hills", "beats", "Crime and Punishment"]}, {"dir": "f00048", "concept1": "ice hockey", "concept2": null, "initial": ["World Cup", "puck", "stock", "school", "Star Wars", "minutes", "Minecraft", "knowledge", "rink", "bus"]}, {"dir": "f00049", "concept1": "women and feminism", "concept2": null, "initial": ["Big Ben", "deposit", "equality", "empowerment", "highway", "drill", "equality", "antenna", "Atlantic Ocean", "excavation"]}, {"dir": "f00050", "concept1": "lighthouses", "concept2": null, "initial": ["puck", "mourning", "folding", "beacon", "light", "Marvel", "television", "rhyme", "tennis ball", "one third"]}, {"dir": "f00051", "concept1": "tennis", "concept2": null, "initial": ["voting", "racket", "weather", "outgoing", "83%", "beats", "democracy", "highway", "Hollywood", "tennis ball"]}, {"dir": "f00052", "concept1": "water sports", "concept2": null, "initial": ["diving", "guitar", "challenges", "outdoors", "saddle", "swimming", "Buddha", "suspension", "PGA", "bus"]}, {"dir": "f00053", "concept1": "comics", "concept2": null, "initial": ["old", "reporting", "owner", "drill", "tent", "nightlife", "balance beam", "artifacts", "DC", "Marvel"]}, {"dir": "f00054", "concept1": "algorithms", "concept2": null, "initial": ["artifacts", "Hamlet", "computing", "CEO", "reporting", "democracy", "voice actors", "excavation", "outdoors", "efficiency"]}, {"dir": "f00055", "concept1": "tourism", "concept2": null, "initial": ["Empire State Building", "Lake Superior", "vacation", "blue", "serif", "travel", "owner", "Michael Phelps", "algebra", "software"]}, {"dir": "f00056", "concept1": "bridges", "concept2": null, "initial": ["camera", "arch", "earthquake", "diving", "Metropolitan Museum of Art", "suspension", "NFL", "rhythym", "Zeus ", "taste"]}, {"dir": "f00057", "concept1": "books", "concept2": null, "initial": ["literature", "mourning", "CPU", "guitar", "Pulitzer Prize", "Buckingham Palace", "television", "deadly", "minerals", "reading"]}, {"dir": "f00058", "concept1": "linguistics", "concept2": null, "initial": ["genome", "voice actors", "synagogues", "olympics", "syntax", "excavation", "minutes", "puck", "language", "style"]}, {"dir": "f00059", "concept1": "utilization and ownership", "concept2": null, "initial": ["owner", "television", "Pulp Fiction", "Minecraft", "Pulitzer Prize", "neutral", "rhyme", "borrow", "serif", "olympics"]}, {"dir": "f00060", "concept1": "online communities", "concept2": null, "initial": ["Reddit", "rhythym", "Lake Superior", "crops", "rocks", "artistic", "Twitter", "genome", "stanza", "reading"]}, {"dir": "f00061", "concept1": "television", "concept2": null, "initial": ["jazz", "hypothesis", "car", "blueprint", "series", "voting", "shows", "Harry Potter", "Hamlet", "American Revolution"]}, {"dir": "f00062", "concept1": "astronomy", "concept2": null, "initial": ["humidity", "NFL", "planets", "mountains", "stars", "dribble", "accessibility", "Reddit", "pilot", "Pacific Ocean"]}, {"dir": "f00063", "concept1": "disability", "concept2": null, "initial": ["accessibility", "deadly", "Pulitzer Prize", "CPU", "Beverly Hills", "minutes", "challenges", "Amtrak", "creativity", "guilty"]}, {"dir": "f00064", "concept1": "proteins", "concept2": null, "initial": ["folding", "software", "Buddha", "Cold War", "artistic", "biology", "Cajun", "grief", "Christianity", "American Revolution"]}, {"dir": "f00065", "concept1": "human anatomy", "concept2": null, "initial": ["organs", "artifacts", "skeleton", "classical", "Star Wars", "Christianity", "temperature", "rhyme", "weather", "skeleton"]}, {"dir": "f00066", "concept1": "gymnastics", "concept2": null, "initial": ["Game of Thrones", "WWE", "tsunami", "American Revolution", "Marvel", "balance beam", "physics", "Fortnite", "olympics", "stars"]}, {"dir": "f00067", "concept1": "architecture", "concept2": null, "initial": ["Louvre", "dribble", "classical", "queen", "weather", "college", "blueprint", "Empire State Building", "money", "World Cup"]}, {"dir": "f00068", "concept1": "sculpture", "concept2": null, "initial": ["skiing", "statue", "geometry", "reservoir", "rocks", "farming", "dribble", "New Orleans", "clay", "rocket"]}, {"dir": "f00069", "concept1": "archaeology", "concept2": null, "initial": ["excavation", "ethics", "excavation", "reservoir", "artifacts", "olympics", "rink", "excavation", "outdoors", "The New York Times"]}, {"dir": "f00070", "concept1": "theater", "concept2": null, "initial": ["guilty", "rocket", "rhyme", "language", "stage", "knights", "empowerment", "acting", "New Orleans", "The Godfather"]}, {"dir": "f00071", "concept1": "dams", "concept2": null, "initial": ["reservoir", "male", "helicoptor", "rink", "burial", "Reddit", "flavor", "water", "Metropolitan Museum of Art", "Mickey Mouse"]}, {"dir": "f00072", "concept1": "birds and ornithology", "concept2": null, "initial": ["Tour de France", "nest", "Nobel Prize", "migration", "forecast", "serif", "folding", "ballot", "space", "Lake Superior"]}, {"dir": "f00073", "concept1": "computing", "concept2": null, "initial": ["Poseidon", "RAM", "racing", "boats", "Lake Superior", "Ethereum", "reservoir", "software", "algorithms", "Breaking Bad"]}, {"dir": "f00074", "concept1": "time and duration", "concept2": null, "initial": ["school", "seconds", "Nirvana", "RAM", "Alps", "minutes", "saddle", "doctor", "factory", "Berlin Wall"]}, {"dir": "f00075", "concept1": "age", "concept2": null, "initial": ["old", "college", "racing", "driver", "Minecraft", "Ethereum", "melody", "Bible", "young", "Jews"]}, {"dir": "f00076", "concept1": "weapons and military equipment", "concept2": null, "initial": ["Empire State Building", "female", "biology", "olympics", "body armor", "touchdown", "tsunami", "Big Ben", "rifle", "innocent"]}, {"dir": "f00077", "concept1": "ratios and proportions", "concept2": null, "initial": ["oxygen", "death", "Hollywood", "Bitcoin", "boats", "rhythym", "half", "north pole", "television", "one third"]}, {"dir": "f00078", "concept1": "typefaces and typography", "concept2": null, "initial": ["reporting", "north pole", "knights", "script", "planes", "83%", "excavation", "rhythym", "tombstone", "serif"]}, {"dir": "f00079", "concept1": "burials, graves, and memorials", "concept2": null, "initial": ["Walter White", "mourning", "grief", "taste", "cells", "dribble", "Jesus", "Star Wars", "Pulitzer Prize", "grief"]}, {"dir": "f00080", "concept1": "processes and manufacturing", "concept2": null, "initial": ["geometry", "particles", "PGA", "rink", "Louvre", "rhyme", "assembly", "Bible", "factory", "Big Ben"]}, {"dir": "f00081", "concept1": "paleontology", "concept2": null, "initial": ["Big Ben", "Crime and Punishment", "water", "Passover", "money", "drill", "pilot", "excavation", "digs", "weather"]}, {"dir": "f00082", "concept1": "banking", "concept2": null, "initial": ["money", "earthquake", "World Cup", "Empire State Building", "algorithms", "innocent", "deposit", "larva", "security", "faith"]}, {"dir": "f00083", "concept1": "lakes", "concept2": null, "initial": ["chemistry", "Lake Superior", "series", "Zeus ", "nest", "literature", "Lake Huron", "school", "Jews", "CPU"]}, {"dir": "f00084", "concept1": "natural science", "concept2": null, "initial": ["Japanese", "artistic", "Minecraft", "chemistry", "creativity", "Bible", "snowboarding", "Pulp Fiction", "CEO", "physics"]}, {"dir": "f00085", "concept1": "geography", "concept2": null, "initial": ["flowers", "north pole", "trends", "CPU", "Metropolitan Museum of Art", "map", "north pole", "challenges", "roots", "Mickey Mouse"]}, {"dir": "f00086", "concept1": "insects and entomology", "concept2": null, "initial": ["stars", "series", "hypothesis", "antenna", "Nazis", "Poseidon", "flowers", "Berlin Wall", "larva", "literature"]}, {"dir": "f00087", "concept1": "philosophy", "concept2": null, "initial": ["democracy", "ethics", "crystals", "bus", "money", "acting", "American Revolution", "thought", "Nirvana", "Reddit"]}, {"dir": "f00088", "concept1": "poetry", "concept2": null, "initial": ["beats", "rhyme", "NBA", "Minecraft", "boats", "nest", "nightlife", "thought", "stanza", "rocket"]}, {"dir": "f00089", "concept1": "encyclopedias", "concept2": null, "initial": ["knowledge", "experiment", "NFL", "minutes", "beacon", "organs", "Lake Superior", "reference", "young", "car"]}, {"dir": "f00090", "concept1": "television shows", "concept2": null, "initial": ["Game of Thrones", "camera", "rhyme", "melody", "space", "doctor", "Breaking Bad", "part of", "Pulitzer Prize", "Brooklyn Bridge"]}, {"dir": "f00091", "concept1": "awards, prizes and honours", "concept2": null, "initial": ["syntax", "rhythym", "Pulitzer Prize", "earthquake", "Nobel Prize", "NBA", "The New York Times", "Marvel", "contained in", "space"]}, {"dir": "f00092", "concept1": "the Middle Ages", "concept2": null, "initial": ["script", "feudalism", "knights", "security", "thought", "physics", "Dublin", "Twitter", "French Revolution", "badminton"]}, {"dir": "f00093", "concept1": "plants and botany", "concept2": null, "initial": ["tennis ", "writing", "roots", "lawyer", "Empire State Building", "diving", "stems", "crystals", "earthquake", "CEO"]}, {"dir": "f00094", "concept1": "marine biology", "concept2": null, "initial": ["ocean", "minerals", "digs", "ocean", "webs", "tombstone", "folding", "fish", "rocket", "syntax"]}, {"dir": "f00095", "concept1": "color", "concept2": null, "initial": ["rhythym", "blue", "temperature", "outgoing", "Harry Potter", "red", "Marvel", "Louvre", "Empire State Building", "hip-hop"]}, {"dir": "f00096", "concept1": "airports", "concept2": null, "initial": ["planes", "clay", "Big Ben", "books ", "security", "north pole", "borrow", "car", "Michael Phelps", "Pacific Ocean"]}, {"dir": "f00097", "concept1": "science", "concept2": null, "initial": ["minerals", "Big Ben", "Michael Phelps", "oxygen", "hypothesis", "Walter White", "half", "pawn", "experiment", "Nirvana"]}, {"dir": "f00098", "concept1": "anime and manga", "concept2": null, "initial": ["guitar", "hip-hop", "style", "card games", "Japanese", "news", "stock", "beacon", "Christianity", "voice actors"]}, {"dir": "f00099", "concept1": "Christianity", "concept2": null, "initial": ["Jesus", "Metropolitan Museum of Art", "ballot", "forecast", "grief", "acting", "deadly", "Game of Thrones", "Bible", "reading"]}, {"dir": "f00100", "concept1": "Buddhism", "concept2": null, "initial": ["algorithms", "Nazis", "Buddha", "nest", "borrow", "Pulitzer Prize", "tennis ", "Nirvana", "hypothesis", "tennis ball"]}, {"dir": "f00101", "concept1": "Greek mythology", "concept2": null, "initial": ["Empire State Building", "Zeus ", "reservoir", "nest", "synagogues", "minutes", "Michael Phelps", "Nobel Prize", "Pacific Ocean", "Poseidon"]}, {"dir": "f00102", "concept1": "Judaism and the Jewish people", "concept2": null, "initial": ["Passover", "writing", "Passover", "nightlife", "synagogues", "green", "doctor", "Japanese", "suspension", "Jesus"]}, {"dir": "f00103", "concept1": "music genres", "concept2": null, "initial": ["helmet", "Empire State Building", "Louvre", "car", "Walter White", "hip-hop", "classical", "Buddha", "government", "language"]}, {"dir": "f00104", "concept1": "fictional characters", "concept2": null, "initial": ["Walter White", "CEO", "voting", "pilot", "tennis ball", "lens", "French Revolution", "Harry Potter", "hypothesis", "championships"]}, {"dir": "f00105", "concept1": "rap and hip hop", "concept2": null, "initial": ["DC", "language", "beats", "hypothesis", "borrow", "World Cup", "plants", "Allah", "rhyme", "boats"]}, {"dir": "f00106", "concept1": "Islam", "concept2": null, "initial": ["biology", "Allah", "Empire State Building", "artifacts", "RAM", "Tour de France", "Quran", "Pacific Ocean", "melody", "queen"]}, {"dir": "f00107", "concept1": "The Walt Disney Company", "concept2": null, "initial": ["Romeo and Juliet", "Disneyland", "Buckingham Palace", "Mickey Mouse", "female", "Cold War", "water", "minerals", "Louvre", "touchdown"]}, {"dir": "f00108", "concept1": "agriculture", "concept2": null, "initial": ["borrow", "accessibility", "female", "reference", "crops", "Big Ben", "beats", "Nazis", "RAM", "farming"]}, {"dir": "f00109", "concept1": "hiking", "concept2": null, "initial": ["shy", "language", "water", "outdoors", "pawn", "knights", "nest", "mountains", "water", "webs"]}, {"dir": "f00110", "concept1": "New York City", "concept2": null, "initial": ["Minecraft", "Eiffel Tower", "north pole", "Brooklyn Bridge", "beats", "Empire State Building", "rink", "thought", "guilty", "Bible"]}, {"dir": "f00111", "concept1": "gardens", "concept2": null, "initial": ["efficiency", "flowers", "shy", "plants", "creativity", "PGA", "Alps", "forecast", "Empire State Building", "flowers"]}, {"dir": "f00112", "concept1": "personality traits", "concept2": null, "initial": ["outgoing", "crops", "melody", "cells", "literature", "shy", "Eiffel Tower", "crystals", "tent", "stars"]}, {"dir": "f00113", "concept1": "camping", "concept2": null, "initial": ["suspension", "tent", "Buddha", "championships", "nature", "digs", "France", "badminton", "minerals", "outgoing"]}, {"dir": "f00114", "concept1": "gender", "concept2": null, "initial": ["humidity", "female", "beats", "male", "Amtrak", "rhythym", "classical", "Marvel", "neutral", "stock"]}, {"dir": "f00115", "concept1": "cemeteries and graves", "concept2": null, "initial": ["highway", "mourning", "Mickey Mouse", "temperature", "artistic", "swimming", "burial", "feudalism", "tombstone", "series"]}, {"dir": "f00116", "concept1": "London", "concept2": null, "initial": ["borrowing", "Louvre", "syntax", "arch", "hypothesis", "Japanese", "larva", "reservoir", "Big Ben", "Buckingham Palace"]}, {"dir": "f00117", "concept1": "Los Angeles", "concept2": null, "initial": ["rhythym", "queen", "space", "Japanese", "flowers", "CPU", "Hollywood", "digs", "championships", "Beverly Hills"]}, {"dir": "f00118", "concept1": "Chicago", "concept2": null, "initial": ["neutral", "jazz", "puck", "red", "genome", "writing", "seconds", "rhyme", "nightlife", "larva"]}, {"dir": "f00119", "concept1": "Paris", "concept2": null, "initial": ["saddle", "Pacific Ocean", "biology", "Eiffel Tower", "rocks", "taste", "airplane", "Allah", "France", "Ethereum"]}, {"dir": "f00120", "concept1": "Berlin", "concept2": null, "initial": ["racket", "Cold War", "vacation", "rocket", "Berlin Wall", "Eiffel Tower", "clay", "outgoing", "synagogues", "reference"]}, {"dir": "f00121", "concept1": "sailing", "concept2": null, "initial": ["part of", "boats", "serif", "minutes", "water", "championships", "classical", "lanes", "money", "faith"]}, {"dir": "f00122", "concept1": "swimming", "concept2": null, "initial": ["airplane", "Michael Phelps", "plants", "money", "Romeo and Juliet", "young", "snowboarding", "series", "water", "nightlife"]}, {"dir": "f00123", "concept1": "the Holocaust", "concept2": null, "initial": ["Jews", "accessibility", "algorithms", "balance beam", "digs", "Nazis", "trends", "government", "television", "Eiffel Tower"]}, {"dir": "f00124", "concept1": "arachnids and arachnology", "concept2": null, "initial": ["Crime and Punishment", "farming", "Romeo and Juliet", "red", "male", "owner", "webs", "spiders", "dribble", "Ethereum"]}, {"dir": "f00125", "concept1": "musical instruments", "concept2": null, "initial": ["challenges", "television", "outgoing", "guitar", "piano", "Hamlet", "artistic", "beats", "sickness", "feudalism"]}, {"dir": "f00126", "concept1": "meteorology", "concept2": null, "initial": ["expression", "badminton", "folding", "one third", "accessibility", "arch", "forecast", "racket", "weather", "batter"]}, {"dir": "f00127", "concept1": "disease", "concept2": null, "initial": ["deadly", "Beverly Hills", "tennis ball", "rifle", "sickness", "accessibility", "television", "genome", "NBA", "temperature"]}, {"dir": "f00128", "concept1": "Ireland", "concept2": null, "initial": ["green", "Dublin", "planets", "Poseidon", "racing", "French Revolution", "Atlantic Ocean", "ballot", "Amtrak", "pilot"]}, {"dir": "f00129", "concept1": "libraries", "concept2": null, "initial": ["borrowing", "female", "books ", "minerals", "Crime and Punishment", "school", "Marvel", "oxygen", "Metropolitan Museum of Art", "stars"]}, {"dir": "f00130", "concept1": "museums", "concept2": null, "initial": ["stems", "Metropolitan Museum of Art", "Louvre", "rifle", "temperature", "rhythym", "reference", "green", "Nobel Prize", "folding"]}, {"dir": "f00131", "concept1": "journalism", "concept2": null, "initial": ["Poseidon", "reporting", "reference", "hypothesis", "old", "writing", "Passover", "green", "physics", "body armor"]}, {"dir": "f00132", "concept1": "buildings", "concept2": null, "initial": ["French Revolution", "one third", "books ", "Empire State Building", "Eiffel Tower", "water", "ethics", "Empire State Building", "Brooklyn Bridge", "balance beam"]}, {"dir": "f00133", "concept1": "cryptocurrencies", "concept2": null, "initial": ["Ethereum", "racket", "The Godfather", "batter", "vacation", "World Cup", "snowboarding", "Bitcoin", "Atlantic Ocean", "organs"]}, {"dir": "f00134", "concept1": "events and news", "concept2": null, "initial": ["Reddit", "knights", "Brooklyn Bridge", "Game of Thrones", "deadly", "Buddha", "train", "grief", "television", "The New York Times"]}, {"dir": "f00135", "concept1": "Louisiana", "concept2": null, "initial": ["New Orleans", "pitcher", "Japanese", "borrowing", "digs", "creativity", "sickness", "Cajun", "beacon", "touchdown"]}, {"dir": "f00136", "concept1": "revolutions", "concept2": null, "initial": ["genome", "French Revolution", "American Revolution", "larva", "rink", "chemistry", "lanes", "Mickey Mouse", "grief", "Dublin"]}, {"dir": "f00137", "concept1": "mines and mining", "concept2": null, "initial": ["hip-hop", "drill", "oxygen", "minerals", "excavation", "female", "reporting", "Disneyland", "balance beam", "Hollywood"]}, {"dir": "f00138", "concept1": "Switzerland", "concept2": null, "initial": ["old", "airplane", "blueprint", "Metropolitan Museum of Art", "stage", "farming", "lawyer", "Alps", "neutral", "Pulitzer Prize"]}, {"dir": "f00139", "concept1": "World War II", "concept2": null, "initial": ["Japanese", "death", "assembly", "Nirvana", "CPU", "highway", "literature", "trends", "Jews", "Nazis"]}, {"dir": "f00140", "concept1": "fictional characters", "concept2": "anime and manga", "initial": ["roots", "driver", "CPU", "voice actors", "Harry Potter", "car", "Empire State Building", "Pulitzer Prize", "books ", "doctor"]}, {"dir": "f00141", "concept1": "music genres", "concept2": "archaeology", "initial": ["The New York Times", "school", "rhyme", "artifacts", "voice actors", "water", "classical", "Minecraft", "part of", "software"]}, {"dir": "f00142", "concept1": "games and leisure activities", "concept2": "swimming", "initial": ["card games", "Lake Superior", "mountains", "Twitter", "assembly", "Nirvana", "Michael Phelps", "Empire State Building", "writing", "pawn"]}, {"dir": "f00143", "concept1": "gender", "concept2": "hiking", "initial": ["male", "tennis ", "skeleton", "antenna", "planets", "deposit", "Reddit", "mountains", "Big Ben", "queen"]}, {"dir": "f00144", "concept1": "racket sports", "concept2": "buildings", "initial": ["Eiffel Tower", "Alps", "young", "tennis ", "accessibility", "artifacts", "DC", "deadly", "Amtrak", "Bible"]}, {"dir": "f00145", "concept1": "horses", "concept2": "government and state", "initial": ["Minecraft", "democracy", "Hollywood", "beacon", "software", "green", "racing", "webs", "airplane", "folding"]}, {"dir": "f00146", "concept1": "comics", "concept2": "bridges", "initial": ["pilot", "suspension", "grief", "outgoing", "Allah", "Jews", "blue", "Marvel", "train", "suspension"]}, {"dir": "f00147", "concept1": "basketball", "concept2": "chess", "initial": ["helmet", "NBA", "news", "bus", "queen", "mourning", "Reddit", "humidity", "Nirvana", "saddle"]}, {"dir": "f00148", "concept1": "rail transport", "concept2": "Judaism and the Jewish people", "initial": ["language", "one third", "Amtrak", "badminton", "artifacts", "olympics", "forecast", "style", "Passover", "camera"]}, {"dir": "f00149", "concept1": "Berlin", "concept2": "museums", "initial": ["Metropolitan Museum of Art", "water", "artifacts", "pilot", "suspension", "water", "racing", "excavation", "reference", "Berlin Wall"]}, {"dir": "f00150", "concept1": "computing", "concept2": "processes and manufacturing", "initial": ["bus", "snowboarding", "algorithms", "factory", "spiders", "excavation", "Buckingham Palace", "excavation", "suspension", "Dublin"]}, {"dir": "f00151", "concept1": "sculpture", "concept2": "racket sports", "initial": ["excavation", "excavation", "software", "Nirvana", "vacation", "Breaking Bad", "statue", "nest", "piano", "badminton"]}, {"dir": "f00152", "concept1": "proteins", "concept2": "events and news", "initial": ["minerals", "The New York Times", "Louvre", "farming", "pawn", "biology", "olympics", "empowerment", "excavation", "touchdown"]}, {"dir": "f00153", "concept1": "mineralogy", "concept2": "disease", "initial": ["racing", "sickness", "expression", "script", "faith", "minerals", "Eiffel Tower", "Quran", "nature", "Alps"]}, {"dir": "f00154", "concept1": "tennis", "concept2": "food and eating", "initial": ["series", "contained in", "racket", "editorial", "borrow", "Walter White", "racket", "faith", "security", "taste"]}, {"dir": "f00155", "concept1": "philosophy", "concept2": "films", "initial": ["Ethereum", "camera", "reporting", "Louvre", "Poseidon", "Pulp Fiction", "airplane", "equality", "thought", "Brooklyn Bridge"]}, {"dir": "f00156", "concept1": "corporations", "concept2": "women and feminism", "initial": ["empowerment", "Twitter", "Buckingham Palace", "CEO", "pawn", "Empire State Building", "computing", "Cold War", "CEO", "helmet"]}, {"dir": "f00157", "concept1": "New York City", "concept2": "banking", "initial": ["rhythym", "ocean", "Empire State Building", "deposit", "Pulp Fiction", "language", "series", "deadly", "Amtrak", "Walter White"]}, {"dir": "f00158", "concept1": "paleontology", "concept2": "birds and ornithology", "initial": ["Ethereum", "feudalism", "equality", "digs", "young", "flavor", "nest", "digs", "crops", "tennis ball"]}, {"dir": "f00159", "concept1": "swimming", "concept2": "games and leisure activities", "initial": ["tsunami", "creativity", "water", "reference", "piano", "Louvre", "mountains", "television", "television", "puck"]}, {"dir": "f00160", "concept1": "computer hardware", "concept2": "geography", "initial": ["planets", "forecast", "helicoptor", "stems", "lanes", "rifle", "Berlin Wall", "map", "Ethereum", "CPU"]}, {"dir": "f00161", "concept1": "architecture", "concept2": "religions and beliefs", "initial": ["classical", "sickness", "blueprint", "Romeo and Juliet", "Christianity", "Empire State Building", "rocket", "planets", "Minecraft", "Reddit"]}, {"dir": "f00162", "concept1": "Christianity", "concept2": "architecture", "initial": ["excavation", "car", "Louvre", "news", "France", "balance beam", "taste", "Jesus", "Disneyland", "reservoir"]}, {"dir": "f00163", "concept1": "Los Angeles", "concept2": "music", "initial": ["Ethereum", "queen", "old", "rhythym", "burial", "challenges", "shows", "computing", "Hollywood", "Amtrak"]}, {"dir": "f00164", "concept1": "religions and beliefs", "concept2": "sailing", "initial": ["artifacts", "editorial", "rifle", "faith", "Jesus", "nature", "French Revolution", "boats", "Twitter", "racket"]}, {"dir": "f00165", "concept1": "personality traits", "concept2": "art", "initial": ["Fortnite", "empowerment", "farming", "Eiffel Tower", "Star Wars", "algorithms", "Eiffel Tower", "creativity", "Louvre", "outgoing"]}, {"dir": "f00166", "concept1": "the Holocaust", "concept2": "American football", "initial": ["classical", "blueprint", "touchdown", "clay", "Jews", "sickness", "snowboarding", "classical", "camera", "reporting"]}, {"dir": "f00167", "concept1": "sailing", "concept2": "Paris", "initial": ["water", "saddle", "Eiffel Tower", "physics", "equality", "challenges", "rhyme", "money", "Pulitzer Prize", "Hamlet"]}, {"dir": "f00168", "concept1": "chess", "concept2": "gardens", "initial": ["minerals", "plants", "tombstone", "The New York Times", "pawn", "voice actors", "camera", "editorial", "Amtrak", "classical"]}, {"dir": "f00169", "concept1": "food and eating", "concept2": "plants and botany", "initial": ["larva", "reservoir", "queen", "World Cup", "half", "NBA", "Hamlet", "television", "roots", "flavor"]}, {"dir": "f00170", "concept1": "anime and manga", "concept2": "television", "initial": ["olympics", "burial", "Nazis", "Pacific Ocean", "nightlife", "classical", "Passover", "NBA", "series", "Japanese"]}, {"dir": "f00171", "concept1": "government and state", "concept2": "tennis", "initial": ["Nazis", "government", "algorithms", "genome", "tennis ball", "part of", "Atlantic Ocean", "lawyer", "Tour de France", "planets"]}, {"dir": "f00172", "concept1": "Islam", "concept2": "age", "initial": ["experiment", "young", "Allah", "planes", "DC", "airplane", "highway", "The Godfather", "mourning", "Romeo and Juliet"]}, {"dir": "f00173", "concept1": "science", "concept2": "fictional characters", "initial": ["hypothesis", "stanza", "Mickey Mouse", "factory", "boats", "Walter White", "voting", "migration", "borrow", "writing"]}, {"dir": "f00174", "concept1": "disease", "concept2": "spacecraft", "initial": ["Pacific Ocean", "Beverly Hills", "deadly", "rocket", "Eiffel Tower", "water", "algorithms", "challenges", "red", "humidity"]}, {"dir": "f00175", "concept1": "geology", "concept2": "color", "initial": ["rocks", "minerals", "diving", "France", "classical", "outgoing", "ocean", "blue", "rhyme", "light"]}, {"dir": "f00176", "concept1": "mines and mining", "concept2": "personality traits", "initial": ["shy", "Michael Phelps", "excavation", "Marvel", "ethics", "humidity", "World Cup", "death", "outdoors", "airplane"]}, {"dir": "f00177", "concept1": "the relationship of an element to its class", "concept2": "burials, graves, and memorials", "initial": ["Passover", "nest", "hypothesis", "mourning", "rhyme", "Amtrak", "rhyme", "owner", "contained in", "webs"]}, {"dir": "f00178", "concept1": "sport organizations", "concept2": "utilization and ownership", "initial": ["half", "Berlin Wall", "ocean", "innocent", "owner", "NFL", "Zeus ", "stanza", "dribble", "Disneyland"]}, {"dir": "f00179", "concept1": "marine biology", "concept2": "human anatomy", "initial": ["earthquake", "sickness", "CPU", "minerals", "helicoptor", "skeleton", "PGA", "antenna", "ocean", "touchdown"]}, {"dir": "f00180", "concept1": "video games", "concept2": "winter sports", "initial": ["mourning", "rocket", "Zeus ", "innocent", "seconds", "Minecraft", "outgoing", "snowboarding", "NFL", "American Revolution"]}, {"dir": "f00181", "concept1": "agriculture", "concept2": "professional wrestling", "initial": ["chemistry", "farming", "minerals", "championships", "genome", "half", "statue", "jazz", "Buddha", "faith"]}, {"dir": "f00182", "concept1": "Judaism and the Jewish people", "concept2": "proteins", "initial": ["ocean", "young", "farming", "bus", "stock", "Tour de France", "folding", "mourning", "television", "synagogues"]}, {"dir": "f00183", "concept1": "time and duration", "concept2": "arachnids and arachnology", "initial": ["tennis ball", "Atlantic Ocean", "style", "melody", "humidity", "championships", "seconds", "taste", "geometry", "webs"]}, {"dir": "f00184", "concept1": "cemeteries and graves", "concept2": "the Middle Ages", "initial": ["crops", "knights", "style", "Alps", "Metropolitan Museum of Art", "red", "pilot", "Atlantic Ocean", "tombstone", "lens"]}, {"dir": "f00185", "concept1": "education", "concept2": "the Holocaust", "initial": ["Nazis", "roots", "Buddha", "Nazis", "tsunami", "helmet", "school", "hip-hop", "shows", "serif"]}, {"dir": "f00186", "concept1": "baseball", "concept2": "golf", "initial": ["pitcher", "stage", "WWE", "driver", "Nazis", "saddle", "piano", "balance beam", "pitcher", "style"]}, {"dir": "f00187", "concept1": "bodies of water", "concept2": "basketball", "initial": ["planets", "rink", "Atlantic Ocean", "championships", "tennis ", "weather", "Dublin", "dribble", "deposit", "biology"]}, {"dir": "f00188", "concept1": "events and news", "concept2": "Buddhism", "initial": ["rocket", "artifacts", "drill", "Louvre", "Buddha", "Mickey Mouse", "borrow", "Nirvana", "faith", "television"]}, {"dir": "f00189", "concept1": "books", "concept2": "sport organizations", "initial": ["dribble", "sickness", "biology", "championships", "reading", "ocean", "language", "The New York Times", "World Cup", "NFL"]}, {"dir": "f00190", "concept1": "The Walt Disney Company", "concept2": "plays", "initial": ["death", "Mickey Mouse", "Hamlet", "classical", "oxygen", "owner", "skiing", "Christianity", "light", "physics"]}, {"dir": "f00191", "concept1": "arachnids and arachnology", "concept2": "computing", "initial": ["spiders", "pitcher", "The Godfather", "stems", "software", "classical", "voice actors", "Michael Phelps", "syntax", "Mickey Mouse"]}, {"dir": "f00192", "concept1": "astronomy", "concept2": "law and justice", "initial": ["stars", "biology", "computing", "geometry", "tent", "World Cup", "rocket", "one third", "innocent", "hypothesis"]}, {"dir": "f00193", "concept1": "online communities", "concept2": "music genres", "initial": ["money", "jazz", "hip-hop", "Reddit", "piano", "83%", "excavation", "folding", "factory", "farming"]}, {"dir": "f00194", "concept1": "Ireland", "concept2": "Berlin", "initial": ["highway", "Cold War", "temperature", "farming", "Harry Potter", "excavation", "guilty", "Quran", "water", "Dublin"]}, {"dir": "f00195", "concept1": "Chicago", "concept2": "baseball", "initial": ["pitcher", "script", "television", "Pulitzer Prize", "water", "DC", "nightlife", "tent", "batter", "RAM"]}, {"dir": "f00196", "concept1": "transport", "concept2": "tourism", "initial": ["folding", "guitar", "vacation", "ethics", "The New York Times", "deadly", "outgoing", "earthquake", "bus", "assembly"]}, {"dir": "f00197", "concept1": "lakes", "concept2": "journalism", "initial": ["algebra", "style", "books ", "college", "Lake Superior", "ethics", "tennis ", "writing", "RAM", "empowerment"]}, {"dir": "f00198", "concept1": "spacecraft", "concept2": "encyclopedias", "initial": ["Eiffel Tower", "lens", "olympics", "melody", "Pulitzer Prize", "airplane", "fish", "knowledge", "half", "oxygen"]}, {"dir": "f00199", "concept1": "bridges", "concept2": "online communities", "initial": ["stems", "arch", "Quran", "Twitter", "helmet", "PGA", "water", "hip-hop", "efficiency", "CEO"]}] -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00000/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | import numpy as np 4 | import torch 5 | 6 | openai.api_key = "EMPTY" # Not support yet 7 | openai.api_base = {API_BASE} 8 | model = "vicuna-13b-v1.1" 9 | 10 | from transformers import LlamaTokenizer 11 | tokenizer = LlamaTokenizer.from_pretrained({LLAMA_PATH}) 12 | zero_token_id = tokenizer._convert_token_to_id('0') # 29900 13 | one_token_id = tokenizer._convert_token_to_id('1') # 29896 14 | 15 | concept = 'sport organizations' 16 | 17 | def function(input, sys_prompt): 18 | messages=[ 19 | {"role": "system", "content": sys_prompt}, #system prompt 20 | {"role": "user", "content": 21 | f"I am going to give you an input. Return only \"1\" if that input is related to {concept}. "+ 22 | "Return only \"0\" otherwise.\nDo not return any additional information other than the 0 or 1 answer.\n"+ 23 | f"Input: {input}"} 24 | ] 25 | 26 | completion = openai.ChatCompletion.create( 27 | model=model, 28 | messages=messages, 29 | temperature=0.0 30 | ) 31 | 32 | logprobs = np.array(completion.choices[0].logprobs) 33 | output_tokens = tokenizer.encode(completion.choices[0].message.content, return_tensors='pt') 34 | indices_of_zero_or_one = torch.logical_or(output_tokens == zero_token_id, output_tokens == one_token_id) 35 | indices_of_zero_or_one = indices_of_zero_or_one.int() 36 | index_of_choice = indices_of_zero_or_one.nonzero() 37 | index = index_of_choice[0][1].item() 38 | index = index-1 39 | prob_1 = np.exp(logprobs[index, one_token_id]) / (np.exp(logprobs[index, zero_token_id]) + np.exp(logprobs[index, one_token_id])) 40 | 41 | return prob_1 42 | 43 | if __name__ == '__main__': 44 | outputs = '' 45 | prompt = "You are an AI trained to identify words related to input concepts." 46 | for arg in sys.argv[1:]: 47 | outputs += f'({arg}, {function(arg, prompt):.4f}) ' 48 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00000/initial.json: -------------------------------------------------------------------------------- 1 | ["Metropolitan Museum of Art", "one third", "NFL", "hip-hop", "Japanese", "World Cup", "bus", "Mickey Mouse", "olympics", "webs"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00001/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | import numpy as np 4 | import torch 5 | 6 | openai.api_key = "EMPTY" # Not support yet 7 | openai.api_base = {API_BASE} 8 | model = "vicuna-13b-v1.1" 9 | 10 | from transformers import LlamaTokenizer 11 | tokenizer = LlamaTokenizer.from_pretrained({LLAMA_PATH}) 12 | zero_token_id = tokenizer._convert_token_to_id('0') # 29900 13 | one_token_id = tokenizer._convert_token_to_id('1') # 29896 14 | 15 | concept = 'corporations' 16 | 17 | def function(input, sys_prompt): 18 | messages=[ 19 | {"role": "system", "content": sys_prompt}, #system prompt 20 | {"role": "user", "content": 21 | f"I am going to give you an input. Return only \"1\" if that input is related to {concept}. "+ 22 | "Return only \"0\" otherwise.\nDo not return any additional information other than the 0 or 1 answer.\n"+ 23 | f"Input: {input}"} 24 | ] 25 | 26 | completion = openai.ChatCompletion.create( 27 | model=model, 28 | messages=messages, 29 | temperature=0.0 30 | ) 31 | 32 | logprobs = np.array(completion.choices[0].logprobs) 33 | output_tokens = tokenizer.encode(completion.choices[0].message.content, return_tensors='pt') 34 | indices_of_zero_or_one = torch.logical_or(output_tokens == zero_token_id, output_tokens == one_token_id) 35 | indices_of_zero_or_one = indices_of_zero_or_one.int() 36 | index_of_choice = indices_of_zero_or_one.nonzero() 37 | index = index_of_choice[0][1].item() 38 | index = index-1 39 | prob_1 = np.exp(logprobs[index, one_token_id]) / (np.exp(logprobs[index, zero_token_id]) + np.exp(logprobs[index, one_token_id])) 40 | 41 | return prob_1 42 | 43 | if __name__ == '__main__': 44 | outputs = '' 45 | prompt = "You are an AI trained to identify words related to input concepts." 46 | for arg in sys.argv[1:]: 47 | outputs += f'({arg}, {function(arg, prompt):.4f}) ' 48 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00001/initial.json: -------------------------------------------------------------------------------- 1 | ["stock", "Game of Thrones", "CEO", "French Revolution", "Cold War", "chemistry", "grief", "tennis ", "airplane", "democracy"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00002/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | import numpy as np 4 | import torch 5 | 6 | openai.api_key = "EMPTY" # Not support yet 7 | openai.api_base = {API_BASE} 8 | model = "vicuna-13b-v1.1" 9 | 10 | from transformers import LlamaTokenizer 11 | tokenizer = LlamaTokenizer.from_pretrained({LLAMA_PATH}) 12 | zero_token_id = tokenizer._convert_token_to_id('0') # 29900 13 | one_token_id = tokenizer._convert_token_to_id('1') # 29896 14 | 15 | concept = 'elections' 16 | 17 | def function(input, sys_prompt): 18 | messages=[ 19 | {"role": "system", "content": sys_prompt}, #system prompt 20 | {"role": "user", "content": 21 | f"I am going to give you an input. Return only \"1\" if that input is related to {concept}. "+ 22 | "Return only \"0\" otherwise.\nDo not return any additional information other than the 0 or 1 answer.\n"+ 23 | f"Input: {input}"} 24 | ] 25 | 26 | completion = openai.ChatCompletion.create( 27 | model=model, 28 | messages=messages, 29 | temperature=0.0 30 | ) 31 | 32 | logprobs = np.array(completion.choices[0].logprobs) 33 | output_tokens = tokenizer.encode(completion.choices[0].message.content, return_tensors='pt') 34 | indices_of_zero_or_one = torch.logical_or(output_tokens == zero_token_id, output_tokens == one_token_id) 35 | indices_of_zero_or_one = indices_of_zero_or_one.int() 36 | index_of_choice = indices_of_zero_or_one.nonzero() 37 | index = index_of_choice[0][1].item() 38 | index = index-1 39 | prob_1 = np.exp(logprobs[index, one_token_id]) / (np.exp(logprobs[index, zero_token_id]) + np.exp(logprobs[index, one_token_id])) 40 | 41 | return prob_1 42 | 43 | if __name__ == '__main__': 44 | outputs = '' 45 | prompt = "You are an AI trained to identify words related to input concepts." 46 | for arg in sys.argv[1:]: 47 | outputs += f'({arg}, {function(arg, prompt):.4f}) ' 48 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00002/initial.json: -------------------------------------------------------------------------------- 1 | ["weather", "Tour de France", "voting", "planets", "NFL", "ballot", "map", "Lake Huron", "geometry", "synagogues"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00003/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | import numpy as np 4 | import torch 5 | 6 | openai.api_key = "EMPTY" # Not support yet 7 | openai.api_base = {API_BASE} 8 | model = "vicuna-13b-v1.1" 9 | 10 | from transformers import LlamaTokenizer 11 | tokenizer = LlamaTokenizer.from_pretrained({LLAMA_PATH}) 12 | zero_token_id = tokenizer._convert_token_to_id('0') # 29900 13 | one_token_id = tokenizer._convert_token_to_id('1') # 29896 14 | 15 | concept = 'musical works' 16 | 17 | def function(input, sys_prompt): 18 | messages=[ 19 | {"role": "system", "content": sys_prompt}, #system prompt 20 | {"role": "user", "content": 21 | f"I am going to give you an input. Return only \"1\" if that input is related to {concept}. "+ 22 | "Return only \"0\" otherwise.\nDo not return any additional information other than the 0 or 1 answer.\n"+ 23 | f"Input: {input}"} 24 | ] 25 | 26 | completion = openai.ChatCompletion.create( 27 | model=model, 28 | messages=messages, 29 | temperature=0.0 30 | ) 31 | 32 | logprobs = np.array(completion.choices[0].logprobs) 33 | output_tokens = tokenizer.encode(completion.choices[0].message.content, return_tensors='pt') 34 | indices_of_zero_or_one = torch.logical_or(output_tokens == zero_token_id, output_tokens == one_token_id) 35 | indices_of_zero_or_one = indices_of_zero_or_one.int() 36 | index_of_choice = indices_of_zero_or_one.nonzero() 37 | index = index_of_choice[0][1].item() 38 | index = index-1 39 | prob_1 = np.exp(logprobs[index, one_token_id]) / (np.exp(logprobs[index, zero_token_id]) + np.exp(logprobs[index, one_token_id])) 40 | 41 | return prob_1 42 | 43 | if __name__ == '__main__': 44 | outputs = '' 45 | prompt = "You are an AI trained to identify words related to input concepts." 46 | for arg in sys.argv[1:]: 47 | outputs += f'({arg}, {function(arg, prompt):.4f}) ' 48 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00003/initial.json: -------------------------------------------------------------------------------- 1 | ["digs", "minerals", "classical", "one third", "minutes", "artistic", "map", "piano", "Louvre", "Hamlet"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00004/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | import numpy as np 4 | import torch 5 | 6 | openai.api_key = "EMPTY" # Not support yet 7 | openai.api_base = {API_BASE} 8 | model = "vicuna-13b-v1.1" 9 | 10 | from transformers import LlamaTokenizer 11 | tokenizer = LlamaTokenizer.from_pretrained({LLAMA_PATH}) 12 | zero_token_id = tokenizer._convert_token_to_id('0') # 29900 13 | one_token_id = tokenizer._convert_token_to_id('1') # 29896 14 | 15 | concept = 'films' 16 | 17 | def function(input, sys_prompt): 18 | messages=[ 19 | {"role": "system", "content": sys_prompt}, #system prompt 20 | {"role": "user", "content": 21 | f"I am going to give you an input. Return only \"1\" if that input is related to {concept}. "+ 22 | "Return only \"0\" otherwise.\nDo not return any additional information other than the 0 or 1 answer.\n"+ 23 | f"Input: {input}"} 24 | ] 25 | 26 | completion = openai.ChatCompletion.create( 27 | model=model, 28 | messages=messages, 29 | temperature=0.0 30 | ) 31 | 32 | logprobs = np.array(completion.choices[0].logprobs) 33 | output_tokens = tokenizer.encode(completion.choices[0].message.content, return_tensors='pt') 34 | indices_of_zero_or_one = torch.logical_or(output_tokens == zero_token_id, output_tokens == one_token_id) 35 | indices_of_zero_or_one = indices_of_zero_or_one.int() 36 | index_of_choice = indices_of_zero_or_one.nonzero() 37 | index = index_of_choice[0][1].item() 38 | index = index-1 39 | prob_1 = np.exp(logprobs[index, one_token_id]) / (np.exp(logprobs[index, zero_token_id]) + np.exp(logprobs[index, one_token_id])) 40 | 41 | return prob_1 42 | 43 | if __name__ == '__main__': 44 | outputs = '' 45 | prompt = "You are an AI trained to identify words related to input concepts." 46 | for arg in sys.argv[1:]: 47 | outputs += f'({arg}, {function(arg, prompt):.4f}) ' 48 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_entities/f00004/initial.json: -------------------------------------------------------------------------------- 1 | ["faith", "Pulp Fiction", "minerals", "pitcher", "puck", "boats", "balance beam", "The Godfather", "train", "seconds"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/data.json: -------------------------------------------------------------------------------- 1 | [{"dir": "/data/vision/torralba/Functions/neurons_relations//f00000", "concept1": "a country", "concept2": "the capital of that country", "exeption": null, "initial": ["Bulgaria", "Greece", "Adam", "Tokyo", "Marco Rubio", "Fidelio", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00001", "concept1": "a city", "concept2": "the country where that city is located", "exeption": null, "initial": ["Mexico", "Stonehenge", "Bear", "Adam", "Zircon", "Bulgaria", "Spain", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00002", "concept1": "a head of state", "concept2": "the country where that person was head of state", "exeption": null, "initial": ["Marco Rubio", "Zircon", "South Korea", "The Grapes of Wrath", "The Barber of Seville", "Portugal", "Bear", "Tokyo"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00003", "concept1": "a monument", "concept2": "the country where that monument is located", "exeption": null, "initial": ["The Barber of Seville", "UAE", "Australia", "Lena", "L\u00f3pez Obrador", "book", "Tokyo", "Portugal", "Greece"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00004", "concept1": "a monument", "concept2": "the continent where that monument is located", "exeption": null, "initial": ["Rhine", "Tokyo", "Spain", "Fidelio", "Bear", "L\u00f3pez Obrador", "Chichen Itza", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00005", "concept1": "a city in the United States", "concept2": "the state in which that city is located", "exeption": null, "initial": ["Greece", "L\u00f3pez Obrador", "Zircon", "sunshine", "Mexico", "Rhine", "moon", "Wolf", "Bear", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00006", "concept1": "a country", "concept2": "the official language of that country", "exeption": null, "initial": ["L\u00f3pez Obrador", "Tokyo", "Lena", "UAE", "Stonehenge", "Portugal", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00007", "concept1": "a well-known person", "concept2": "the occupation of that person", "exeption": null, "initial": ["Marco Rubio", "Tokyo", "Lena", "Rhine", "Spain", "Wolf", "South Korea", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00008", "concept1": "the name of a person", "concept2": "the most likely gender of that person", "exeption": null, "initial": ["Australia", "Adam", "Albert Einstein", "South Korea", "Dallas", "book", "The Grapes of Wrath", "UAE", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00009", "concept1": "a country", "concept2": "the continent where that country is located", "exeption": null, "initial": ["Australia", "Bear", "L\u00f3pez Obrador", "Zircon", "Wolf", "tree", "Chichen Itza", "Ankara", "Portugal"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00010", "concept1": "an opera", "concept2": "the composer of that opera", "exeption": null, "initial": ["Dallas", "Australia", "Chichen Itza", "Bulgaria", "Spain", "South Korea", "Tokyo", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00011", "concept1": "the title of a book", "concept2": "the author of that book", "exeption": null, "initial": ["Rhine", "Ankara", "Marco Rubio", "Mexico", "Fidelio", "The Grapes of Wrath", "Dallas", "Portugal", "elephant"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00012", "concept1": "a country", "concept2": "another country on its border", "exeption": null, "initial": ["Dallas", "UAE", "Bulgaria", "Lena", "Wolf", "Spain", "South Korea", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00013", "concept1": "a politician", "concept2": "the political party of which this politician is a member", "exeption": null, "initial": ["Marco Rubio", "Australia", "Wolf", "Tokyo", "Lena", "Fidelio", "The Grapes of Wrath", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00014", "concept1": "a river", "concept2": "the continent where that river is located", "exeption": null, "initial": ["Rhine", "UAE", "Bear", "Ankara", "L\u00f3pez Obrador", "Stonehenge", "Adam", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00015", "concept1": "a city", "concept2": "the time zone of that city", "exeption": null, "initial": ["Stonehenge", "Marco Rubio", "L\u00f3pez Obrador", "Greece", "Zircon", "Portugal", "Mexico", "Wolf"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00016", "concept1": "an opera", "concept2": "the language of that opera", "exeption": null, "initial": ["Greece", "Wolf", "Lena", "Rhine", "Portugal", "Marco Rubio", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00017", "concept1": "an animal", "concept2": "the main food source of that animal", "exeption": null, "initial": ["Fidelio", "Spain", "Australia", "Bear", "Tokyo", "Dallas", "Rhine", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00018", "concept1": "an animal", "concept2": "the typical habitat of that animal", "exeption": null, "initial": ["Marco Rubio", "L\u00f3pez Obrador", "Portugal", "Stonehenge", "Greece", "Bear", "South Korea", "Australia"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00019", "concept1": "a gemstone", "concept2": "the color of that gemstone", "exeption": null, "initial": ["Bulgaria", "Chichen Itza", "Dallas", "Portugal", "tree", "Zircon", "Mexico", "Adam", "South Korea"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00020", "concept1": "a country", "concept2": "the color of that country's flag", "exeption": null, "initial": ["Stonehenge", "Adam", "Portugal", "Spain", "The Grapes of Wrath", "Chichen Itza", "Bulgaria", "Fidelio"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00021", "concept1": "a river", "concept2": "the length of that river, in kilometers", "exeption": null, "initial": ["Rhine", "Tokyo", "apple", "Zircon", "The Grapes of Wrath", "The Barber of Seville", "phone", "Bear", "Wolf", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00022", "concept1": "a country", "concept2": "the height of the maximum elevation in that country, in kilometers", "exeption": null, "initial": ["Ankara", "The Barber of Seville", "The Grapes of Wrath", "Adam", "Lena", "L\u00f3pez Obrador", "Stonehenge", "Spain", "star"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00023", "concept1": "a country", "concept2": "the height of the tallest skyscraper in that country, in meters", "exeption": null, "initial": ["Albert Einstein", "Chichen Itza", "Stonehenge", "Spain", "Bear", "Tokyo", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00024", "concept1": "a country", "concept2": "the height of the average male in that country, in meters", "exeption": null, "initial": ["Marco Rubio", "Zircon", "The Barber of Seville", "Spain", "Tokyo", "pencil", "Chichen Itza", "Mexico", "Bear"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00025", "concept1": "a country", "concept2": "the capital of that country", "exeption": "the input country is in Asia", "initial": ["Bulgaria", "Greece", "Adam", "Tokyo", "Marco Rubio", "Fidelio", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00026", "concept1": "a city", "concept2": "the country where that city is located", "exeption": "the input city is in Mexico", "initial": ["Mexico", "Stonehenge", "Bear", "Adam", "Zircon", "Bulgaria", "Spain", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00027", "concept1": "a head of state", "concept2": "the country where that person was head of state", "exeption": "the input is a man", "initial": ["Marco Rubio", "Zircon", "South Korea", "The Grapes of Wrath", "The Barber of Seville", "Portugal", "Bear", "Tokyo"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00028", "concept1": "a monument", "concept2": "the country where that monument is located", "exeption": "the input monument is in New York City", "initial": ["The Barber of Seville", "UAE", "Australia", "Lena", "L\u00f3pez Obrador", "book", "Tokyo", "Portugal", "Greece"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00029", "concept1": "a monument", "concept2": "the continent where that monument is located", "exeption": "the input monument is in South America", "initial": ["Rhine", "Tokyo", "Spain", "Fidelio", "Bear", "L\u00f3pez Obrador", "Chichen Itza", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00030", "concept1": "a city in the United States", "concept2": "the state in which that city is located", "exeption": "the input city is in California", "initial": ["Greece", "L\u00f3pez Obrador", "Zircon", "sunshine", "Mexico", "Rhine", "moon", "Wolf", "Bear", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00031", "concept1": "a country", "concept2": "the official language of that country", "exeption": "the input country is in North America", "initial": ["L\u00f3pez Obrador", "Tokyo", "Lena", "UAE", "Stonehenge", "Portugal", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00032", "concept1": "a well-known person", "concept2": "the occupation of that person", "exeption": "the input person is a woman", "initial": ["Marco Rubio", "Tokyo", "Lena", "Rhine", "Spain", "Wolf", "South Korea", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00033", "concept1": "the name of a person", "concept2": "the most likely gender of that person", "exeption": "the input begins with T", "initial": ["Australia", "Adam", "Albert Einstein", "South Korea", "Dallas", "book", "The Grapes of Wrath", "UAE", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00034", "concept1": "a country", "concept2": "the continent where that country is located", "exeption": "the input country is in Asia", "initial": ["Australia", "Bear", "L\u00f3pez Obrador", "Zircon", "Wolf", "tree", "Chichen Itza", "Ankara", "Portugal"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00035", "concept1": "an opera", "concept2": "the composer of that opera", "exeption": "the input opera is in Italian", "initial": ["Dallas", "Australia", "Chichen Itza", "Bulgaria", "Spain", "South Korea", "Tokyo", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00036", "concept1": "the title of a book", "concept2": "the author of that book", "exeption": "the input book was written after 1900", "initial": ["Rhine", "Ankara", "Marco Rubio", "Mexico", "Fidelio", "The Grapes of Wrath", "Dallas", "Portugal", "elephant"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00037", "concept1": "a country", "concept2": "another country on its border", "exeption": "the input country is in Africa", "initial": ["Dallas", "UAE", "Bulgaria", "Lena", "Wolf", "Spain", "South Korea", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00038", "concept1": "a politician", "concept2": "the political party of which this politician is a member", "exeption": "the input politician is a man", "initial": ["Marco Rubio", "Australia", "Wolf", "Tokyo", "Lena", "Fidelio", "The Grapes of Wrath", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00039", "concept1": "a river", "concept2": "the continent where that river is located", "exeption": "the input is in Africa", "initial": ["Rhine", "UAE", "Bear", "Ankara", "L\u00f3pez Obrador", "Stonehenge", "Adam", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00040", "concept1": "a city", "concept2": "the time zone of that city", "exeption": "the input city is in North America", "initial": ["Stonehenge", "Marco Rubio", "L\u00f3pez Obrador", "Greece", "Zircon", "Portugal", "Mexico", "Wolf"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00041", "concept1": "an opera", "concept2": "the language of that opera", "exeption": "the input opera is in English", "initial": ["Greece", "Wolf", "Lena", "Rhine", "Portugal", "Marco Rubio", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00042", "concept1": "an animal", "concept2": "the main food source of that animal", "exeption": "the input animal is a carnivore", "initial": ["Fidelio", "Spain", "Australia", "Bear", "Tokyo", "Dallas", "Rhine", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00043", "concept1": "an animal", "concept2": "the typical habitat of that animal", "exeption": "the input animal lives on land", "initial": ["Marco Rubio", "L\u00f3pez Obrador", "Portugal", "Stonehenge", "Greece", "Bear", "South Korea", "Australia"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00044", "concept1": "a gemstone", "concept2": "the color of that gemstone", "exeption": "the input gemstone is red", "initial": ["Bulgaria", "Chichen Itza", "Dallas", "Portugal", "tree", "Zircon", "Mexico", "Adam", "South Korea"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00045", "concept1": "a country", "concept2": "the color of that country's flag", "exeption": "the input country is in North America", "initial": ["Stonehenge", "Adam", "Portugal", "Spain", "The Grapes of Wrath", "Chichen Itza", "Bulgaria", "Fidelio"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00046", "concept1": "a river", "concept2": "the length of that river, in kilometers", "exeption": "the input river is in North America", "initial": ["Rhine", "Tokyo", "apple", "Zircon", "The Grapes of Wrath", "The Barber of Seville", "phone", "Bear", "Wolf", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00047", "concept1": "a country", "concept2": "the height of the maximum elevation in that country, in kilometers", "exeption": "the input country is in South America", "initial": ["Ankara", "The Barber of Seville", "The Grapes of Wrath", "Adam", "Lena", "L\u00f3pez Obrador", "Stonehenge", "Spain", "star"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00048", "concept1": "a country", "concept2": "the height of the tallest skyscraper in that country, in meters", "exeption": "the input country is in North America", "initial": ["Albert Einstein", "Chichen Itza", "Stonehenge", "Spain", "Bear", "Tokyo", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00049", "concept1": "a country", "concept2": "the height of the average male in that country, in meters", "exeption": "the input country is in Europe", "initial": ["Marco Rubio", "Zircon", "The Barber of Seville", "Spain", "Tokyo", "pencil", "Chichen Itza", "Mexico", "Bear"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00050", "concept1": "a country", "concept2": "the capital of that country", "exeption": "the input country is in South America", "initial": ["Bulgaria", "Greece", "Adam", "Tokyo", "Marco Rubio", "Fidelio", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00051", "concept1": "a city", "concept2": "the country where that city is located", "exeption": "the input city is in the United States", "initial": ["Mexico", "Stonehenge", "Bear", "Adam", "Zircon", "Bulgaria", "Spain", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00052", "concept1": "a head of state", "concept2": "the country where that person was head of state", "exeption": "the input is a woman", "initial": ["Marco Rubio", "Zircon", "South Korea", "The Grapes of Wrath", "The Barber of Seville", "Portugal", "Bear", "Tokyo"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00053", "concept1": "a monument", "concept2": "the country where that monument is located", "exeption": "the input monument is in France", "initial": ["The Barber of Seville", "UAE", "Australia", "Lena", "L\u00f3pez Obrador", "book", "Tokyo", "Portugal", "Greece"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00054", "concept1": "a monument", "concept2": "the continent where that monument is located", "exeption": "the input monument is in Australia", "initial": ["Rhine", "Tokyo", "Spain", "Fidelio", "Bear", "L\u00f3pez Obrador", "Chichen Itza", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00055", "concept1": "a city in the United States", "concept2": "the state in which that city is located", "exeption": "the input city is in Texas", "initial": ["Greece", "L\u00f3pez Obrador", "Zircon", "sunshine", "Mexico", "Rhine", "moon", "Wolf", "Bear", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00056", "concept1": "a country", "concept2": "the official language of that country", "exeption": "the input country is in South America", "initial": ["L\u00f3pez Obrador", "Tokyo", "Lena", "UAE", "Stonehenge", "Portugal", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00057", "concept1": "a well-known person", "concept2": "the occupation of that person", "exeption": "the input person is a man", "initial": ["Marco Rubio", "Tokyo", "Lena", "Rhine", "Spain", "Wolf", "South Korea", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00058", "concept1": "the name of a person", "concept2": "the most likely gender of that person", "exeption": "the input begins with S", "initial": ["Australia", "Adam", "Albert Einstein", "South Korea", "Dallas", "book", "The Grapes of Wrath", "UAE", "Chichen Itza"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00059", "concept1": "a country", "concept2": "the continent where that country is located", "exeption": "the input country is in Europe", "initial": ["Australia", "Bear", "L\u00f3pez Obrador", "Zircon", "Wolf", "tree", "Chichen Itza", "Ankara", "Portugal"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00060", "concept1": "an opera", "concept2": "the composer of that opera", "exeption": "the input opera is in German", "initial": ["Dallas", "Australia", "Chichen Itza", "Bulgaria", "Spain", "South Korea", "Tokyo", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00061", "concept1": "the title of a book", "concept2": "the author of that book", "exeption": " the input book was written before 1900", "initial": ["Rhine", "Ankara", "Marco Rubio", "Mexico", "Fidelio", "The Grapes of Wrath", "Dallas", "Portugal", "elephant"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00062", "concept1": "a country", "concept2": "another country on its border", "exeption": "the input country is in North America", "initial": ["Dallas", "UAE", "Bulgaria", "Lena", "Wolf", "Spain", "South Korea", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00063", "concept1": "a politician", "concept2": "the political party of which this politician is a member", "exeption": "the input politician is a woman", "initial": ["Marco Rubio", "Australia", "Wolf", "Tokyo", "Lena", "Fidelio", "The Grapes of Wrath", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00064", "concept1": "a river", "concept2": "the continent where that river is located", "exeption": "the input is in Europe", "initial": ["Rhine", "UAE", "Bear", "Ankara", "L\u00f3pez Obrador", "Stonehenge", "Adam", "The Barber of Seville"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00065", "concept1": "a city", "concept2": "the time zone of that city", "exeption": "the input city is in Asia", "initial": ["Stonehenge", "Marco Rubio", "L\u00f3pez Obrador", "Greece", "Zircon", "Portugal", "Mexico", "Wolf"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00066", "concept1": "an opera", "concept2": "the language of that opera", "exeption": "the input opera is in Italian", "initial": ["Greece", "Wolf", "Lena", "Rhine", "Portugal", "Marco Rubio", "Spain", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00067", "concept1": "an animal", "concept2": "the main food source of that animal", "exeption": "the input animal is a herbivore", "initial": ["Fidelio", "Spain", "Australia", "Bear", "Tokyo", "Dallas", "Rhine", "Adam"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00068", "concept1": "an animal", "concept2": "the typical habitat of that animal", "exeption": "the input animal lives in the ocean", "initial": ["Marco Rubio", "L\u00f3pez Obrador", "Portugal", "Stonehenge", "Greece", "Bear", "South Korea", "Australia"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00069", "concept1": "a gemstone", "concept2": "the color of that gemstone", "exeption": "the input gemstone is blue", "initial": ["Bulgaria", "Chichen Itza", "Dallas", "Portugal", "tree", "Zircon", "Mexico", "Adam", "South Korea"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00070", "concept1": "a country", "concept2": "the color of that country's flag", "exeption": "the input country is in Asia", "initial": ["Stonehenge", "Adam", "Portugal", "Spain", "The Grapes of Wrath", "Chichen Itza", "Bulgaria", "Fidelio"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00071", "concept1": "a river", "concept2": "the length of that river, in kilometers", "exeption": "the input river is in Europe", "initial": ["Rhine", "Tokyo", "apple", "Zircon", "The Grapes of Wrath", "The Barber of Seville", "phone", "Bear", "Wolf", "Ankara"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00072", "concept1": "a country", "concept2": "the height of the maximum elevation in that country, in kilometers", "exeption": "the input country is in Africa", "initial": ["Ankara", "The Barber of Seville", "The Grapes of Wrath", "Adam", "Lena", "L\u00f3pez Obrador", "Stonehenge", "Spain", "star"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00073", "concept1": "a country", "concept2": "the height of the tallest skyscraper in that country, in meters", "exeption": "the input country is in Asia", "initial": ["Albert Einstein", "Chichen Itza", "Stonehenge", "Spain", "Bear", "Tokyo", "Ankara", "Mexico"]}, {"dir": "/data/vision/torralba/Functions/neurons_relations//f00074", "concept1": "a country", "concept2": "the height of the average male in that country, in meters", "exeption": "the input country is in South America", "initial": ["Marco Rubio", "Zircon", "The Barber of Seville", "Spain", "Tokyo", "pencil", "Chichen Itza", "Mexico", "Bear"]}] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00000/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | 4 | openai.api_key = "EMPTY" # Not support yet 5 | openai.api_base = {API_BASE} 6 | model = "vicuna-13b-v1.1" 7 | 8 | con1 = 'a country' 9 | con2 = 'the capital of that country' 10 | 11 | def function(input, sys_prompt): 12 | messages=[ 13 | {"role": "system", "content": sys_prompt}, #system prompt 14 | {"role": "user", "content": 15 | f"I am going to give you an input." + 16 | f"If the input is {con1}, reply with only {con2}. " + 17 | f"If the input is not {con1}, return 'undefined'.\n"+ 18 | f"Do not include any additional information other than the output.\n"+ 19 | f"Input:{input}" 20 | } 21 | ] 22 | 23 | completion = openai.ChatCompletion.create( 24 | model=model, 25 | messages=messages, 26 | temperature=0.0 27 | ) 28 | 29 | return completion.choices[0].message.content 30 | 31 | if __name__ == '__main__': 32 | outputs = '' 33 | prompt = "You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible. Do not include additional explanation." 34 | for arg in sys.argv[1:]: 35 | x = arg 36 | outputs += f'({arg}, {function(arg, prompt)}) ' 37 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00000/initial.json: -------------------------------------------------------------------------------- 1 | ["Bulgaria", "Greece", "Adam", "Tokyo", "Marco Rubio", "Fidelio", "Ankara", "Mexico"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00001/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | 4 | openai.api_key = "EMPTY" # Not support yet 5 | openai.api_base = {API_BASE} 6 | model = "vicuna-13b-v1.1" 7 | 8 | con1 = 'a city' 9 | con2 = 'the country where that city is located' 10 | 11 | def function(input, sys_prompt): 12 | messages=[ 13 | {"role": "system", "content": sys_prompt}, #system prompt 14 | {"role": "user", "content": 15 | f"I am going to give you an input." + 16 | f"If the input is {con1}, reply with only {con2}. " + 17 | f"If the input is not {con1}, return 'undefined'.\n"+ 18 | f"Do not include any additional information other than the output.\n"+ 19 | f"Input:{input}" 20 | } 21 | ] 22 | 23 | completion = openai.ChatCompletion.create( 24 | model=model, 25 | messages=messages, 26 | temperature=0.0 27 | ) 28 | 29 | return completion.choices[0].message.content 30 | 31 | if __name__ == '__main__': 32 | outputs = '' 33 | prompt = "You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible. Do not include additional explanation." 34 | for arg in sys.argv[1:]: 35 | x = arg 36 | outputs += f'({arg}, {function(arg, prompt)}) ' 37 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00001/initial.json: -------------------------------------------------------------------------------- 1 | ["Mexico", "Stonehenge", "Bear", "Adam", "Zircon", "Bulgaria", "Spain", "Chichen Itza"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00002/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | 4 | openai.api_key = "EMPTY" # Not support yet 5 | openai.api_base = {API_BASE} 6 | model = "vicuna-13b-v1.1" 7 | 8 | con1 = 'a head of state' 9 | con2 = 'the country where that person was head of state' 10 | 11 | def function(input, sys_prompt): 12 | messages=[ 13 | {"role": "system", "content": sys_prompt}, #system prompt 14 | {"role": "user", "content": 15 | f"I am going to give you an input." + 16 | f"If the input is {con1}, reply with only {con2}. " + 17 | f"If the input is not {con1}, return 'undefined'.\n"+ 18 | f"Do not include any additional information other than the output.\n"+ 19 | f"Input:{input}" 20 | } 21 | ] 22 | 23 | completion = openai.ChatCompletion.create( 24 | model=model, 25 | messages=messages, 26 | temperature=0.0 27 | ) 28 | 29 | return completion.choices[0].message.content 30 | 31 | if __name__ == '__main__': 32 | outputs = '' 33 | prompt = "You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible. Do not include additional explanation." 34 | for arg in sys.argv[1:]: 35 | x = arg 36 | outputs += f'({arg}, {function(arg, prompt)}) ' 37 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00002/initial.json: -------------------------------------------------------------------------------- 1 | ["Marco Rubio", "Zircon", "South Korea", "The Grapes of Wrath", "The Barber of Seville", "Portugal", "Bear", "Tokyo"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00003/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | 4 | openai.api_key = "EMPTY" # Not support yet 5 | openai.api_base = {API_BASE} 6 | model = "vicuna-13b-v1.1" 7 | 8 | con1 = 'a monument' 9 | con2 = 'the country where that monument is located' 10 | 11 | def function(input, sys_prompt): 12 | messages=[ 13 | {"role": "system", "content": sys_prompt}, #system prompt 14 | {"role": "user", "content": 15 | f"I am going to give you an input." + 16 | f"If the input is {con1}, reply with only {con2}. " + 17 | f"If the input is not {con1}, return 'undefined'.\n"+ 18 | f"Do not include any additional information other than the output.\n"+ 19 | f"Input:{input}" 20 | } 21 | ] 22 | 23 | completion = openai.ChatCompletion.create( 24 | model=model, 25 | messages=messages, 26 | temperature=0.0 27 | ) 28 | 29 | return completion.choices[0].message.content 30 | 31 | if __name__ == '__main__': 32 | outputs = '' 33 | prompt = "You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible. Do not include additional explanation." 34 | for arg in sys.argv[1:]: 35 | x = arg 36 | outputs += f'({arg}, {function(arg, prompt)}) ' 37 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00003/initial.json: -------------------------------------------------------------------------------- 1 | ["The Barber of Seville", "UAE", "Australia", "Lena", "L\u00f3pez Obrador", "book", "Tokyo", "Portugal", "Greece"] -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00004/function_code.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import sys 3 | 4 | openai.api_key = "EMPTY" # Not support yet 5 | openai.api_base = {API_BASE} 6 | model = "vicuna-13b-v1.1" 7 | 8 | con1 = 'a monument' 9 | con2 = 'the continent where that monument is located' 10 | 11 | def function(input, sys_prompt): 12 | messages=[ 13 | {"role": "system", "content": sys_prompt}, #system prompt 14 | {"role": "user", "content": 15 | f"I am going to give you an input." + 16 | f"If the input is {con1}, reply with only {con2}. " + 17 | f"If the input is not {con1}, return 'undefined'.\n"+ 18 | f"Do not include any additional information other than the output.\n"+ 19 | f"Input:{input}" 20 | } 21 | ] 22 | 23 | completion = openai.ChatCompletion.create( 24 | model=model, 25 | messages=messages, 26 | temperature=0.0 27 | ) 28 | 29 | return completion.choices[0].message.content 30 | 31 | if __name__ == '__main__': 32 | outputs = '' 33 | prompt = "You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible. Do not include additional explanation." 34 | for arg in sys.argv[1:]: 35 | x = arg 36 | outputs += f'({arg}, {function(arg, prompt)}) ' 37 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/neurons_relations/f00004/initial.json: -------------------------------------------------------------------------------- 1 | ["Rhine", "Tokyo", "Spain", "Fidelio", "Bear", "L\u00f3pez Obrador", "Chichen Itza", "The Barber of Seville"] -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00000/__pycache__/function_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00000/__pycache__/function_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00000/function_code.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00000/mlp_approx_model.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00000/mlp_approx_model.pt -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00001/__pycache__/function_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00001/__pycache__/function_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00001/function_code.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00001/mlp_approx_model.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00001/mlp_approx_model.pt -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00002/__pycache__/function_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00002/__pycache__/function_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00002/function_code.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00002/mlp_approx_model.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00002/mlp_approx_model.pt -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00003/__pycache__/function_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00003/__pycache__/function_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00003/function_code.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00003/mlp_approx_model.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00003/mlp_approx_model.pt -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00004/__pycache__/function_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00004/__pycache__/function_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00004/function_code.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/find-dataset/numeric/f00004/mlp_approx_model.pt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/find-dataset/numeric/f00004/mlp_approx_model.pt -------------------------------------------------------------------------------- /src/find-dataset/numeric/func_name_dict.json: -------------------------------------------------------------------------------- 1 | {"f00991": "function_code.py", "f00805": "corrupted_function_code.py", "f00872": "corrupted_function_code.py", "f00428": "function_code.py", "f00908": "function_code.py", "f00552": "noised_function_code.py", "f00525": "function_code.py", "f00018": "load_mlp.py", "f00162": "function_code.py", "f00081": "load_mlp.py", "f00115": "load_mlp.py", "f00012": "load_mlp.py", "f00186": "function_code.py", "f00065": "load_mlp.py", "f00168": "function_code.py", "f00878": "corrupted_function_code.py", "f00422": "function_code.py", "f00455": "function_code.py", "f00975": "function_code.py", "f00902": "corrupted_function_code.py", "f00896": "function_code.py", "f00558": "function_code.py", "f00366": "function_code.py", "f00285": "function_code.py", "f00311": "corrupted_function_code.py", "f00388": "corrupted_function_code.py", "f00756": "function_code.py", "f00721": "function_code.py", "f00626": "function_code.py", "f00651": "function_code.py", "f00216": "corrupted_function_code.py", "f00382": "function_code.py", "f00261": "function_code.py", "f00127": "load_mlp.py", "f00150": "function_code.py", "f00840": "function_code.py", "f00837": "corrupted_function_code.py", "f00483": "function_code.py", "f00517": "function_code.py", "f00560": "function_code.py", "f00467": "noised_function_code.py", "f00410": "function_code.py", "f00584": "function_code.py", "f00930": "function_code.py", "f00947": "function_code.py", "f00489": "function_code.py", "f00057": "load_mlp.py", "f00020": "load_mlp.py", "f00687": "function_code.py", "f00713": "corrupted_function_code.py", "f00764": "function_code.py", "f00669": "function_code.py", "f00323": "function_code.py", "f00354": "function_code.py", "f00259": "function_code.py", "f00329": "corrupted_function_code.py", "f00253": "function_code.py", "f00224": "function_code.py", "f00719": "function_code.py", "f00663": "function_code.py", "f00614": "noised_function_code.py", "f00780": "function_code.py", "f00966": "noised_function_code.py", "f00911": "function_code.py", "f00885": "corrupted_function_code.py", "f00431": "function_code.py", "f00988": "function_code.py", "f00446": "noised_function_code.py", "f00098": "load_mlp.py", "f00001": "load_mlp.py", "f00195": "corrupted_function_code.py", "f00076": "load_mlp.py", "f00171": "function_code.py", "f00092": "load_mlp.py", "f00106": "load_mlp.py", "f00541": "function_code.py", "f00536": "corrupted_function_code.py", "f00982": "function_code.py", "f00816": "function_code.py", "f00861": "corrupted_function_code.py", "f00205": "noised_function_code.py", "f00391": "function_code.py", "f00272": "function_code.py", "f00308": "function_code.py", "f00635": "function_code.py", "f00642": "function_code.py", "f00738": "function_code.py", "f00648": "corrupted_function_code.py", "f00745": "function_code.py", "f00732": "function_code.py", "f00278": "function_code.py", "f00375": "function_code.py", "f00296": "noised_function_code.py", "f00302": "corrupted_function_code.py", "f00149": "load_mlp.py", "f00044": "load_mlp.py", "f00033": "load_mlp.py", "f00579": "corrupted_function_code.py", "f00923": "noised_function_code.py", "f00954": "corrupted_function_code.py", "f00474": "noised_function_code.py", "f00403": "function_code.py", "f00859": "function_code.py", "f00597": "noised_function_code.py", "f00490": "function_code.py", "f00504": "function_code.py", "f00573": "noised_function_code.py", "f00929": "noised_function_code.py", "f00409": "noised_function_code.py", "f00853": "function_code.py", "f00824": "function_code.py", "f00134": "load_mlp.py", "f00143": "load_mlp.py", "f00039": "load_mlp.py", "f00670": "noised_function_code.py", "f00607": "corrupted_function_code.py", "f00793": "corrupted_function_code.py", "f00240": "function_code.py", "f00237": "corrupted_function_code.py", "f00330": "function_code.py", "f00347": "noised_function_code.py", "f00799": "function_code.py", "f00694": "corrupted_function_code.py", "f00700": "function_code.py", "f00777": "corrupted_function_code.py", "f00699": "function_code.py", "f00794": "noised_function_code.py", "f00600": "function_code.py", "f00677": "function_code.py", "f00230": "function_code.py", "f00247": "corrupted_function_code.py", "f00340": "corrupted_function_code.py", "f00337": "corrupted_function_code.py", "f00770": "corrupted_function_code.py", "f00707": "function_code.py", "f00693": "function_code.py", "f00034": "load_mlp.py", "f00043": "load_mlp.py", "f00139": "load_mlp.py", "f00590": "function_code.py", "f00404": "function_code.py", "f00473": "corrupted_function_code.py", "f00829": "corrupted_function_code.py", "f00509": "function_code.py", "f00953": "corrupted_function_code.py", "f00924": "noised_function_code.py", "f00479": "function_code.py", "f00823": "function_code.py", "f00854": "function_code.py", "f00574": "function_code.py", "f00503": "function_code.py", "f00497": "function_code.py", "f00959": "function_code.py", "f00049": "load_mlp.py", "f00144": "load_mlp.py", "f00133": "load_mlp.py", "f00378": "noised_function_code.py", "f00275": "corrupted_function_code.py", "f00396": "noised_function_code.py", "f00202": "function_code.py", "f00748": "noised_function_code.py", "f00645": "function_code.py", "f00632": "function_code.py", "f00735": "function_code.py", "f00742": "function_code.py", "f00638": "function_code.py", "f00305": "function_code.py", "f00291": "function_code.py", "f00372": "corrupted_function_code.py", "f00208": "function_code.py", "f00441": "function_code.py", "f00436": "function_code.py", "f00882": "noised_function_code.py", "f00916": "function_code.py", "f00961": "corrupted_function_code.py", "f00071": "load_mlp.py", "f00192": "function_code.py", "f00006": "load_mlp.py", "f00198": "corrupted_function_code.py", "f00101": "load_mlp.py", "f00095": "load_mlp.py", "f00176": "function_code.py", "f00866": "function_code.py", "f00811": "function_code.py", "f00985": "function_code.py", "f00531": "noised_function_code.py", "f00546": "function_code.py", "f00888": "function_code.py", "f00619": "function_code.py", "f00763": "noised_function_code.py", "f00714": "function_code.py", "f00680": "function_code.py", "f00229": "noised_function_code.py", "f00353": "function_code.py", "f00324": "function_code.py", "f00223": "noised_function_code.py", "f00254": "function_code.py", "f00359": "function_code.py", "f00787": "noised_function_code.py", "f00613": "function_code.py", "f00664": "corrupted_function_code.py", "f00769": "function_code.py", "f00157": "function_code.py", "f00120": "load_mlp.py", "f00567": "function_code.py", "f00510": "function_code.py", "f00484": "corrupted_function_code.py", "f00830": "corrupted_function_code.py", "f00589": "corrupted_function_code.py", "f00847": "function_code.py", "f00940": "corrupted_function_code.py", "f00937": "function_code.py", "f00583": "noised_function_code.py", "f00417": "function_code.py", "f00460": "corrupted_function_code.py", "f00027": "load_mlp.py", "f00050": "load_mlp.py", "f00316": "noised_function_code.py", "f00282": "corrupted_function_code.py", "f00361": "function_code.py", "f00726": "noised_function_code.py", "f00751": "function_code.py", "f00656": "function_code.py", "f00621": "noised_function_code.py", "f00266": "corrupted_function_code.py", "f00385": "corrupted_function_code.py", "f00211": "function_code.py", "f00288": "function_code.py", "f00978": "noised_function_code.py", "f00522": "function_code.py", "f00555": "function_code.py", "f00875": "function_code.py", "f00802": "function_code.py", "f00458": "corrupted_function_code.py", "f00996": "function_code.py", "f00112": "load_mlp.py", "f00086": "load_mlp.py", "f00165": "corrupted_function_code.py", "f00068": "load_mlp.py", "f00118": "load_mlp.py", "f00062": "load_mlp.py", "f00181": "function_code.py", "f00015": "load_mlp.py", "f00891": "corrupted_function_code.py", "f00905": "function_code.py", "f00972": "function_code.py", "f00528": "noised_function_code.py", "f00808": "function_code.py", "f00452": "function_code.py", "f00425": "corrupted_function_code.py", "f00318": "corrupted_function_code.py", "f00262": "function_code.py", "f00215": "function_code.py", "f00381": "function_code.py", "f00728": "function_code.py", "f00652": "noised_function_code.py", "f00625": "corrupted_function_code.py", "f00722": "corrupted_function_code.py", "f00755": "noised_function_code.py", "f00658": "corrupted_function_code.py", "f00286": "noised_function_code.py", "f00312": "corrupted_function_code.py", "f00365": "corrupted_function_code.py", "f00268": "corrupted_function_code.py", "f00456": "function_code.py", "f00998": "corrupted_function_code.py", "f00421": "noised_function_code.py", "f00901": "corrupted_function_code.py", "f00895": "function_code.py", "f00976": "corrupted_function_code.py", "f00066": "load_mlp.py", "f00011": "load_mlp.py", "f00185": "noised_function_code.py", "f00088": "load_mlp.py", "f00082": "load_mlp.py", "f00116": "load_mlp.py", "f00161": "corrupted_function_code.py", "f00871": "corrupted_function_code.py", "f00992": "corrupted_function_code.py", "f00806": "function_code.py", "f00526": "function_code.py", "f00551": "function_code.py", "f00617": "noised_function_code.py", "f00783": "function_code.py", "f00660": "function_code.py", "f00227": "function_code.py", "f00250": "noised_function_code.py", "f00357": "function_code.py", "f00320": "noised_function_code.py", "f00767": "function_code.py", "f00684": "function_code.py", "f00710": "corrupted_function_code.py", "f00789": "function_code.py", "f00023": "load_mlp.py", "f00054": "load_mlp.py", "f00159": "function_code.py", "f00413": "function_code.py", "f00587": "corrupted_function_code.py", "f00849": "function_code.py", "f00464": "corrupted_function_code.py", "f00944": "corrupted_function_code.py", "f00569": "noised_function_code.py", "f00933": "function_code.py", "f00834": "function_code.py", "f00419": "function_code.py", "f00843": "noised_function_code.py", "f00563": "corrupted_function_code.py", "f00939": "corrupted_function_code.py", "f00480": "function_code.py", "f00514": "function_code.py", "f00029": "load_mlp.py", "f00153": "function_code.py", "f00124": "load_mlp.py", "f00398": "function_code.py", "f00295": "noised_function_code.py", "f00301": "function_code.py", "f00376": "function_code.py", "f00731": "function_code.py", "f00746": "function_code.py", "f00641": "corrupted_function_code.py", "f00636": "function_code.py", "f00271": "function_code.py", "f00206": "function_code.py", "f00392": "function_code.py", "f00535": "function_code.py", "f00918": "noised_function_code.py", "f00542": "corrupted_function_code.py", "f00862": "corrupted_function_code.py", "f00438": "corrupted_function_code.py", "f00981": "function_code.py", "f00815": "function_code.py", "f00091": "load_mlp.py", "f00105": "load_mlp.py", "f00172": "function_code.py", "f00008": "load_mlp.py", "f00178": "function_code.py", "f00075": "load_mlp.py", "f00002": "load_mlp.py", "f00196": "function_code.py", "f00912": "function_code.py", "f00548": "function_code.py", "f00886": "noised_function_code.py", "f00965": "noised_function_code.py", "f00445": "noised_function_code.py", "f00868": "function_code.py", "f00432": "function_code.py", "f00679": "corrupted_function_code.py", "f00774": "function_code.py", "f00697": "corrupted_function_code.py", "f00703": "corrupted_function_code.py", "f00249": "corrupted_function_code.py", "f00344": "corrupted_function_code.py", "f00333": "noised_function_code.py", "f00234": "corrupted_function_code.py", "f00243": "noised_function_code.py", "f00339": "noised_function_code.py", "f00604": "function_code.py", "f00790": "function_code.py", "f00673": "noised_function_code.py", "f00709": "corrupted_function_code.py", "f00140": "load_mlp.py", "f00137": "load_mlp.py", "f00570": "corrupted_function_code.py", "f00493": "function_code.py", "f00507": "corrupted_function_code.py", "f00827": "noised_function_code.py", "f00850": "noised_function_code.py", "f00499": "function_code.py", "f00957": "function_code.py", "f00920": "function_code.py", "f00400": "noised_function_code.py", "f00594": "function_code.py", "f00477": "function_code.py", "f00030": "load_mlp.py", "f00047": "load_mlp.py", "f00130": "load_mlp.py", "f00147": "load_mlp.py", "f00857": "corrupted_function_code.py", "f00599": "corrupted_function_code.py", "f00820": "noised_function_code.py", "f00500": "function_code.py", "f00494": "corrupted_function_code.py", "f00577": "function_code.py", "f00470": "noised_function_code.py", "f00593": "function_code.py", "f00407": "function_code.py", "f00927": "noised_function_code.py", "f00950": "function_code.py", "f00040": "load_mlp.py", "f00037": "load_mlp.py", "f00704": "function_code.py", "f00690": "function_code.py", "f00773": "function_code.py", "f00609": "function_code.py", "f00334": "function_code.py", "f00343": "noised_function_code.py", "f00239": "noised_function_code.py", "f00349": "corrupted_function_code.py", "f00244": "corrupted_function_code.py", "f00233": "corrupted_function_code.py", "f00779": "corrupted_function_code.py", "f00674": "function_code.py", "f00797": "corrupted_function_code.py", "f00603": "function_code.py", "f00812": "noised_function_code.py", "f00986": "function_code.py", "f00448": "noised_function_code.py", "f00865": "function_code.py", "f00545": "function_code.py", "f00968": "function_code.py", "f00532": "function_code.py", "f00078": "load_mlp.py", "f00175": "corrupted_function_code.py", "f00102": "load_mlp.py", "f00096": "load_mlp.py", "f00191": "noised_function_code.py", "f00005": "load_mlp.py", "f00072": "load_mlp.py", "f00108": "load_mlp.py", "f00435": "function_code.py", "f00818": "function_code.py", "f00442": "corrupted_function_code.py", "f00962": "function_code.py", "f00538": "function_code.py", "f00881": "noised_function_code.py", "f00915": "function_code.py", "f00371": "noised_function_code.py", "f00306": "function_code.py", "f00292": "noised_function_code.py", "f00741": "noised_function_code.py", "f00736": "corrupted_function_code.py", "f00631": "function_code.py", "f00646": "function_code.py", "f00298": "function_code.py", "f00395": "noised_function_code.py", "f00201": "noised_function_code.py", "f00276": "function_code.py", "f00129": "load_mlp.py", "f00053": "load_mlp.py", "f00024": "load_mlp.py", "f00934": "function_code.py", "f00519": "corrupted_function_code.py", "f00943": "noised_function_code.py", "f00463": "function_code.py", "f00839": "function_code.py", "f00580": "corrupted_function_code.py", "f00414": "function_code.py", "f00513": "corrupted_function_code.py", "f00949": "function_code.py", "f00487": "function_code.py", "f00564": "corrupted_function_code.py", "f00844": "corrupted_function_code.py", "f00469": "function_code.py", "f00833": "noised_function_code.py", "f00123": "load_mlp.py", "f00154": "function_code.py", "f00059": "load_mlp.py", "f00667": "function_code.py", "f00784": "function_code.py", "f00610": "function_code.py", "f00689": "function_code.py", "f00257": "function_code.py", "f00220": "function_code.py", "f00327": "corrupted_function_code.py", "f00350": "corrupted_function_code.py", "f00717": "noised_function_code.py", "f00683": "noised_function_code.py", "f00760": "function_code.py", "f00971": "function_code.py", "f00892": "function_code.py", "f00906": "function_code.py", "f00426": "function_code.py", "f00451": "noised_function_code.py", "f00182": "function_code.py", "f00016": "load_mlp.py", "f00061": "load_mlp.py", "f00166": "corrupted_function_code.py", "f00111": "load_mlp.py", "f00085": "load_mlp.py", "f00188": "function_code.py", "f00898": "function_code.py", "f00556": "noised_function_code.py", "f00521": "corrupted_function_code.py", "f00801": "corrupted_function_code.py", "f00995": "function_code.py", "f00876": "function_code.py", "f00386": "noised_function_code.py", "f00212": "noised_function_code.py", "f00265": "function_code.py", "f00368": "noised_function_code.py", "f00622": "corrupted_function_code.py", "f00655": "corrupted_function_code.py", "f00758": "corrupted_function_code.py", "f00628": "function_code.py", "f00752": "function_code.py", "f00725": "function_code.py", "f00218": "function_code.py", "f00362": "noised_function_code.py", "f00315": "function_code.py", "f00281": "noised_function_code.py", "f00665": "function_code.py", "f00786": "function_code.py", "f00612": "noised_function_code.py", "f00768": "noised_function_code.py", "f00255": "corrupted_function_code.py", "f00222": "function_code.py", "f00358": "function_code.py", "f00228": "noised_function_code.py", "f00325": "corrupted_function_code.py", "f00352": "function_code.py", "f00618": "corrupted_function_code.py", "f00715": "corrupted_function_code.py", "f00681": "function_code.py", "f00762": "noised_function_code.py", "f00051": "load_mlp.py", "f00026": "load_mlp.py", "f00936": "corrupted_function_code.py", "f00941": "function_code.py", "f00461": "function_code.py", "f00582": "corrupted_function_code.py", "f00416": "noised_function_code.py", "f00511": "function_code.py", "f00485": "corrupted_function_code.py", "f00566": "corrupted_function_code.py", "f00846": "function_code.py", "f00588": "corrupted_function_code.py", "f00831": "noised_function_code.py", "f00121": "load_mlp.py", "f00156": "function_code.py", "f00384": "function_code.py", "f00210": "noised_function_code.py", "f00267": "noised_function_code.py", "f00289": "noised_function_code.py", "f00620": "function_code.py", "f00657": "function_code.py", "f00750": "function_code.py", "f00727": "noised_function_code.py", "f00360": "function_code.py", "f00317": "corrupted_function_code.py", "f00283": "function_code.py", "f00973": "noised_function_code.py", "f00529": "function_code.py", "f00890": "function_code.py", "f00904": "function_code.py", "f00424": "function_code.py", "f00809": "corrupted_function_code.py", "f00453": "function_code.py", "f00119": "load_mlp.py", "f00180": "corrupted_function_code.py", "f00014": "load_mlp.py", "f00063": "load_mlp.py", "f00164": "noised_function_code.py", "f00113": "load_mlp.py", "f00087": "load_mlp.py", "f00069": "load_mlp.py", "f00554": "noised_function_code.py", "f00979": "noised_function_code.py", "f00523": "function_code.py", "f00803": "corrupted_function_code.py", "f00997": "noised_function_code.py", "f00459": "noised_function_code.py", "f00874": "function_code.py", "f00706": "noised_function_code.py", "f00692": "function_code.py", "f00771": "noised_function_code.py", "f00336": "function_code.py", "f00341": "corrupted_function_code.py", "f00246": "function_code.py", "f00231": "corrupted_function_code.py", "f00698": "function_code.py", "f00676": "function_code.py", "f00795": "function_code.py", "f00601": "corrupted_function_code.py", "f00048": "load_mlp.py", "f00132": "load_mlp.py", "f00145": "load_mlp.py", "f00855": "function_code.py", "f00478": "corrupted_function_code.py", "f00822": "corrupted_function_code.py", "f00502": "function_code.py", "f00958": "function_code.py", "f00496": "function_code.py", "f00575": "function_code.py", "f00472": "noised_function_code.py", "f00828": "function_code.py", "f00591": "function_code.py", "f00405": "function_code.py", "f00925": "corrupted_function_code.py", "f00508": "noised_function_code.py", "f00952": "function_code.py", "f00042": "load_mlp.py", "f00035": "load_mlp.py", "f00138": "load_mlp.py", "f00373": "noised_function_code.py", "f00304": "function_code.py", "f00290": "function_code.py", "f00209": "function_code.py", "f00743": "function_code.py", "f00734": "function_code.py", "f00639": "noised_function_code.py", "f00749": "function_code.py", "f00633": "function_code.py", "f00644": "noised_function_code.py", "f00379": "corrupted_function_code.py", "f00397": "function_code.py", "f00203": "function_code.py", "f00274": "function_code.py", "f00810": "corrupted_function_code.py", "f00984": "function_code.py", "f00867": "function_code.py", "f00889": "noised_function_code.py", "f00547": "function_code.py", "f00530": "corrupted_function_code.py", "f00199": "noised_function_code.py", "f00177": "function_code.py", "f00100": "load_mlp.py", "f00094": "load_mlp.py", "f00193": "function_code.py", "f00007": "load_mlp.py", "f00070": "load_mlp.py", "f00437": "function_code.py", "f00440": "function_code.py", "f00960": "function_code.py", "f00883": "corrupted_function_code.py", "f00917": "noised_function_code.py", "f00537": "corrupted_function_code.py", "f00540": "noised_function_code.py", "f00860": "noised_function_code.py", "f00983": "function_code.py", "f00817": "corrupted_function_code.py", "f00093": "load_mlp.py", "f00107": "load_mlp.py", "f00170": "noised_function_code.py", "f00099": "load_mlp.py", "f00077": "load_mlp.py", "f00000": "load_mlp.py", "f00194": "function_code.py", "f00910": "function_code.py", "f00884": "function_code.py", "f00967": "function_code.py", "f00447": "function_code.py", "f00989": "function_code.py", "f00430": "corrupted_function_code.py", "f00279": "function_code.py", "f00297": "noised_function_code.py", "f00303": "function_code.py", "f00374": "corrupted_function_code.py", "f00649": "function_code.py", "f00733": "noised_function_code.py", "f00744": "function_code.py", "f00643": "function_code.py", "f00634": "corrupted_function_code.py", "f00739": "corrupted_function_code.py", "f00273": "function_code.py", "f00204": "function_code.py", "f00390": "function_code.py", "f00309": "corrupted_function_code.py", "f00142": "load_mlp.py", "f00135": "load_mlp.py", "f00038": "load_mlp.py", "f00572": "function_code.py", "f00928": "corrupted_function_code.py", "f00491": "function_code.py", "f00505": "function_code.py", "f00825": "function_code.py", "f00408": "corrupted_function_code.py", "f00852": "function_code.py", "f00955": "noised_function_code.py", "f00578": "corrupted_function_code.py", "f00922": "function_code.py", "f00402": "noised_function_code.py", "f00596": "corrupted_function_code.py", "f00858": "function_code.py", "f00475": "noised_function_code.py", "f00148": "load_mlp.py", "f00032": "load_mlp.py", "f00045": "load_mlp.py", "f00798": "noised_function_code.py", "f00776": "noised_function_code.py", "f00695": "function_code.py", "f00701": "function_code.py", "f00346": "function_code.py", "f00331": "function_code.py", "f00236": "function_code.py", "f00241": "function_code.py", "f00606": "function_code.py", "f00792": "noised_function_code.py", "f00671": "function_code.py", "f00454": "function_code.py", "f00879": "noised_function_code.py", "f00423": "function_code.py", "f00903": "function_code.py", "f00559": "function_code.py", "f00897": "noised_function_code.py", "f00974": "function_code.py", "f00064": "load_mlp.py", "f00013": "load_mlp.py", "f00187": "corrupted_function_code.py", "f00169": "function_code.py", "f00019": "load_mlp.py", "f00080": "load_mlp.py", "f00114": "load_mlp.py", "f00163": "corrupted_function_code.py", "f00873": "corrupted_function_code.py", "f00429": "corrupted_function_code.py", "f00990": "noised_function_code.py", "f00804": "function_code.py", "f00524": "function_code.py", "f00909": "corrupted_function_code.py", "f00553": "function_code.py", "f00260": "corrupted_function_code.py", "f00217": "corrupted_function_code.py", "f00383": "function_code.py", "f00650": "noised_function_code.py", "f00627": "noised_function_code.py", "f00720": "noised_function_code.py", "f00757": "function_code.py", "f00284": "function_code.py", "f00310": "function_code.py", "f00367": "function_code.py", "f00389": "noised_function_code.py", "f00021": "load_mlp.py", "f00056": "load_mlp.py", "f00411": "function_code.py", "f00585": "function_code.py", "f00466": "function_code.py", "f00488": "corrupted_function_code.py", "f00946": "function_code.py", "f00931": "corrupted_function_code.py", "f00836": "function_code.py", "f00841": "corrupted_function_code.py", "f00561": "noised_function_code.py", "f00482": "function_code.py", "f00516": "noised_function_code.py", "f00151": "noised_function_code.py", "f00126": "load_mlp.py", "f00718": "function_code.py", "f00615": "noised_function_code.py", "f00781": "noised_function_code.py", "f00662": "noised_function_code.py", "f00328": "function_code.py", "f00225": "function_code.py", "f00252": "function_code.py", "f00355": "corrupted_function_code.py", "f00322": "function_code.py", "f00258": "function_code.py", "f00765": "function_code.py", "f00686": "function_code.py", "f00712": "corrupted_function_code.py", "f00668": "function_code.py", "f00155": "corrupted_function_code.py", "f00122": "load_mlp.py", "f00058": "load_mlp.py", "f00565": "function_code.py", "f00512": "function_code.py", "f00486": "function_code.py", "f00948": "function_code.py", "f00468": "function_code.py", "f00832": "noised_function_code.py", "f00845": "function_code.py", "f00518": "noised_function_code.py", "f00942": "function_code.py", "f00935": "function_code.py", "f00581": "corrupted_function_code.py", "f00415": "noised_function_code.py", "f00462": "function_code.py", "f00838": "function_code.py", "f00128": "load_mlp.py", "f00025": "load_mlp.py", "f00052": "load_mlp.py", "f00761": "function_code.py", "f00716": "corrupted_function_code.py", "f00682": "corrupted_function_code.py", "f00351": "function_code.py", "f00326": "noised_function_code.py", "f00221": "corrupted_function_code.py", "f00256": "function_code.py", "f00785": "noised_function_code.py", "f00611": "function_code.py", "f00666": "function_code.py", "f00688": "function_code.py", "f00520": "function_code.py", "f00557": "noised_function_code.py", "f00899": "function_code.py", "f00877": "function_code.py", "f00800": "corrupted_function_code.py", "f00994": "function_code.py", "f00110": "load_mlp.py", "f00084": "load_mlp.py", "f00167": "corrupted_function_code.py", "f00189": "noised_function_code.py", "f00060": "load_mlp.py", "f00183": "function_code.py", "f00017": "load_mlp.py", "f00893": "function_code.py", "f00907": "noised_function_code.py", "f00970": "noised_function_code.py", "f00450": "noised_function_code.py", "f00427": "function_code.py", "f00219": "function_code.py", "f00314": "corrupted_function_code.py", "f00280": "function_code.py", "f00363": "noised_function_code.py", "f00629": "function_code.py", "f00724": "noised_function_code.py", "f00753": "corrupted_function_code.py", "f00654": "corrupted_function_code.py", "f00623": "function_code.py", "f00759": "corrupted_function_code.py", "f00264": "noised_function_code.py", "f00387": "function_code.py", "f00213": "corrupted_function_code.py", "f00369": "function_code.py", "f00036": "load_mlp.py", "f00041": "load_mlp.py", "f00592": "function_code.py", "f00406": "noised_function_code.py", "f00471": "noised_function_code.py", "f00951": "function_code.py", "f00926": "function_code.py", "f00821": "function_code.py", "f00598": "function_code.py", "f00856": "function_code.py", "f00576": "function_code.py", "f00501": "function_code.py", "f00495": "function_code.py", "f00146": "load_mlp.py", "f00131": "load_mlp.py", "f00778": "noised_function_code.py", "f00796": "noised_function_code.py", "f00602": "noised_function_code.py", "f00675": "noised_function_code.py", "f00348": "function_code.py", "f00232": "function_code.py", "f00245": "noised_function_code.py", "f00342": "function_code.py", "f00335": "function_code.py", "f00238": "noised_function_code.py", "f00772": "function_code.py", "f00705": "corrupted_function_code.py", "f00691": "noised_function_code.py", "f00608": "function_code.py", "f00819": "noised_function_code.py", "f00443": "corrupted_function_code.py", "f00434": "function_code.py", "f00880": "function_code.py", "f00914": "function_code.py", "f00963": "noised_function_code.py", "f00539": "function_code.py", "f00073": "load_mlp.py", "f00190": "function_code.py", "f00004": "load_mlp.py", "f00109": "load_mlp.py", "f00079": "load_mlp.py", "f00103": "load_mlp.py", "f00097": "load_mlp.py", "f00174": "function_code.py", "f00864": "function_code.py", "f00813": "function_code.py", "f00449": "corrupted_function_code.py", "f00987": "function_code.py", "f00969": "noised_function_code.py", "f00533": "corrupted_function_code.py", "f00544": "noised_function_code.py", "f00299": "function_code.py", "f00277": "noised_function_code.py", "f00394": "function_code.py", "f00200": "corrupted_function_code.py", "f00647": "function_code.py", "f00630": "function_code.py", "f00737": "function_code.py", "f00740": "function_code.py", "f00307": "function_code.py", "f00293": "function_code.py", "f00370": "function_code.py", "f00207": "function_code.py", "f00393": "noised_function_code.py", "f00270": "corrupted_function_code.py", "f00637": "function_code.py", "f00640": "function_code.py", "f00747": "function_code.py", "f00730": "function_code.py", "f00399": "function_code.py", "f00377": "function_code.py", "f00294": "function_code.py", "f00300": "noised_function_code.py", "f00964": "noised_function_code.py", "f00913": "function_code.py", "f00887": "function_code.py", "f00549": "function_code.py", "f00869": "function_code.py", "f00433": "function_code.py", "f00444": "noised_function_code.py", "f00179": "noised_function_code.py", "f00003": "load_mlp.py", "f00197": "function_code.py", "f00074": "load_mlp.py", "f00173": "function_code.py", "f00090": "load_mlp.py", "f00104": "load_mlp.py", "f00009": "load_mlp.py", "f00919": "function_code.py", "f00543": "function_code.py", "f00534": "function_code.py", "f00980": "function_code.py", "f00814": "corrupted_function_code.py", "f00863": "corrupted_function_code.py", "f00439": "function_code.py", "f00672": "function_code.py", "f00605": "function_code.py", "f00791": "noised_function_code.py", "f00708": "corrupted_function_code.py", "f00242": "corrupted_function_code.py", "f00235": "corrupted_function_code.py", "f00338": "corrupted_function_code.py", "f00248": "function_code.py", "f00332": "corrupted_function_code.py", "f00345": "function_code.py", "f00678": "function_code.py", "f00696": "noised_function_code.py", "f00702": "noised_function_code.py", "f00775": "corrupted_function_code.py", "f00046": "load_mlp.py", "f00031": "load_mlp.py", "f00921": "function_code.py", "f00956": "function_code.py", "f00498": "function_code.py", "f00476": "noised_function_code.py", "f00401": "function_code.py", "f00595": "corrupted_function_code.py", "f00492": "corrupted_function_code.py", "f00506": "function_code.py", "f00571": "function_code.py", "f00851": "noised_function_code.py", "f00826": "noised_function_code.py", "f00136": "load_mlp.py", "f00141": "load_mlp.py", "f00364": "function_code.py", "f00287": "corrupted_function_code.py", "f00313": "function_code.py", "f00269": "corrupted_function_code.py", "f00754": "corrupted_function_code.py", "f00723": "function_code.py", "f00659": "noised_function_code.py", "f00729": "noised_function_code.py", "f00624": "function_code.py", "f00653": "function_code.py", "f00319": "function_code.py", "f00214": "corrupted_function_code.py", "f00380": "function_code.py", "f00263": "corrupted_function_code.py", "f00993": "function_code.py", "f00807": "function_code.py", "f00870": "function_code.py", "f00550": "function_code.py", "f00527": "corrupted_function_code.py", "f00160": "function_code.py", "f00083": "load_mlp.py", "f00117": "load_mlp.py", "f00010": "load_mlp.py", "f00184": "corrupted_function_code.py", "f00067": "load_mlp.py", "f00089": "load_mlp.py", "f00420": "corrupted_function_code.py", "f00999": "function_code.py", "f00457": "function_code.py", "f00977": "function_code.py", "f00900": "function_code.py", "f00894": "function_code.py", "f00685": "function_code.py", "f00711": "corrupted_function_code.py", "f00766": "function_code.py", "f00788": "noised_function_code.py", "f00321": "function_code.py", "f00356": "function_code.py", "f00251": "function_code.py", "f00226": "corrupted_function_code.py", "f00661": "noised_function_code.py", "f00616": "corrupted_function_code.py", "f00782": "function_code.py", "f00028": "load_mlp.py", "f00125": "load_mlp.py", "f00152": "noised_function_code.py", "f00418": "function_code.py", "f00842": "noised_function_code.py", "f00835": "noised_function_code.py", "f00481": "corrupted_function_code.py", "f00515": "function_code.py", "f00562": "noised_function_code.py", "f00938": "function_code.py", "f00465": "function_code.py", "f00412": "function_code.py", "f00848": "function_code.py", "f00586": "function_code.py", "f00568": "function_code.py", "f00932": "noised_function_code.py", "f00945": "corrupted_function_code.py", "f00055": "load_mlp.py", "f00022": "load_mlp.py", "f00158": "corrupted_function_code.py"} -------------------------------------------------------------------------------- /src/find-dataset/strings/f00000/function_code.py: -------------------------------------------------------------------------------- 1 | 2 | import numpy as np 3 | import scipy.signal 4 | import sys 5 | 6 | 7 | def function1(string): 8 | 9 | inx = 5 10 | if len(string) 0: 45 | loss = tf.train_function(x, f, func_name, x_test, dirname, verbose=args.verbose) 46 | else: 47 | loss = 'not trained' 48 | 49 | # write source code 50 | noise_type, noise_params = write_function(params, func_name, dirname, corrupted, cparams, cname) 51 | f_module = dirname.replace('/', '.')+'.function_code' 52 | function = import_function(f_module, 'function') 53 | 54 | # copy load_mlp.py to function subdirectory 55 | shutil.copy('load_mlp.py', f'{dirname}/load_mlp.py') 56 | # save meta data 57 | func_dict = {'type': get_type(func_name), 'params': params, 'name': name, 58 | 'dir': dirname, 'approx_mse_error': loss, 'corruption_type': cname, 59 | 'corruption_params': {'std':cparams[0], 'boundary':cparams[1:]}, 'domain': domain, 60 | 'noise_type': noise_type, 'noise_params': noise_params, 'mse_loss': loss} 61 | 62 | if debug: 63 | func_dict = print_functions(dirname, func_name, f, function, corrupted, func_dict) 64 | 65 | data.append(func_dict) 66 | 67 | # Write the data to a JSON file 68 | with open(f"{args.dir}/meta/data.json", "w") as file: 69 | json.dump(data, file) 70 | if args.verbose: 71 | import pprint 72 | pprint.pprint(data) 73 | 74 | 75 | -------------------------------------------------------------------------------- /src/make_functions/make_numeric/load_mlp.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import sys 4 | import os 5 | dir_path = os.path.dirname(os.path.abspath(__file__)) 6 | 7 | class MLP(nn.Module): 8 | def __init__(self): 9 | super(MLP, self).__init__() 10 | self.layers = nn.Sequential( 11 | nn.Linear(1, 64), 12 | nn.ReLU(), 13 | nn.Linear(64, 1) 14 | ) 15 | 16 | def forward(self, x): 17 | return self.layers(x) 18 | 19 | def function(x): 20 | model = MLP() 21 | model.load_state_dict(torch.load(os.path.join(dir_path, 'mlp_approx_model.pt'))) 22 | model.eval() 23 | return model(torch.tensor(x).reshape(-1, 1).float()).item() 24 | 25 | if __name__ == '__main__': 26 | outputs = '' 27 | for arg in sys.argv[1:]: 28 | x = float(arg) 29 | outputs += f'({arg}, {str(function(x))}) ' 30 | print(f'Function input - output pairs: {outputs}') -------------------------------------------------------------------------------- /src/make_functions/make_numeric/math_functions.py: -------------------------------------------------------------------------------- 1 | # math_functions.py 2 | import numpy as np 3 | import scipy.signal 4 | import random 5 | from scipy.special import erf, factorial, gamma 6 | 7 | 8 | compound_func_names = ["sum_two", "sum_three", "multiply_two", "convolution", "compose"] 9 | 10 | 11 | # Simple functions 12 | 13 | # Linear function 14 | def linear(m, b, x): 15 | return m * x + b 16 | 17 | # Polynomial 18 | def polynomial(coeffs, x): 19 | degree = len(coeffs) - 1 20 | result = 0 21 | for i in range(degree+1): 22 | result += coeffs[i] * x**(degree-i) 23 | return result 24 | 25 | #Rational 26 | def rational(num_coeff, denom_coeff, x): 27 | num = sum([a*x**i for i, a in enumerate(num_coeff)]) 28 | denom = sum([b*x**i for i, b in enumerate(denom_coeff)]) 29 | return (num) / (denom) 30 | 31 | # Absolute 32 | def absolute(a, b, x): 33 | return a * np.abs(x) + b 34 | 35 | # Root function 36 | def root(a, b, n, x): 37 | return a*(x ** (1.0 / n)) + b 38 | 39 | # Reciprocal function 40 | def reciprocal(a, x): 41 | return 1/(a*x) 42 | 43 | # Erf 44 | def error_func(x): 45 | return erf(x) 46 | 47 | # Logarithm 48 | def logarithm(a, b, base, x): 49 | return a* (np.log(x) / np.log(base)) + b 50 | 51 | 52 | # Step function 53 | def step(a, b, s, x): 54 | return a * np.heaviside(x, s) + b 55 | 56 | # Rectangle function 57 | def rectangle(w, x): 58 | return np.where(np.abs(x) <= w/2, 1 ,0) 59 | 60 | # ReLU 61 | def relu(a, b, x, l): 62 | return a * np.where(x > 0, x, x * l) + b 63 | 64 | # Sigmoid 65 | def sigmoid(a, b,x, t): 66 | return a*(1 / (1 + np.exp(-x/t))) + b 67 | 68 | # Tanh 69 | def tanh(a, b, c, d, x): 70 | return a * np.tanh((2 * np.pi / b)*(x-c)) + d 71 | 72 | # Sin 73 | def sin(a, b, c, d, x): 74 | return a * np.sin((2 * np.pi / b)*(x-c)) + d 75 | 76 | #Cosine 77 | def cos(a, b, c, d, x): 78 | return a * np.cos((2 * np.pi / b)*(x-c)) + d 79 | 80 | #Tangent 81 | def tan(a, b, c, d, x): 82 | return a * np.tan((2 * np.pi / b)*(x-c)) + d 83 | 84 | #Ceiling 85 | def ceil(x): 86 | return np.ceil(x) 87 | 88 | #Floor 89 | def floor(x): 90 | return np.floor(x) 91 | 92 | #Gaussian 93 | def gaussian(mean, std, x): 94 | return (1 / (std * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - mean) / std) ** 2) 95 | 96 | #Square wave 97 | def square_wave(period, shift, x): 98 | value = np.sin((x-shift) * (2 * np.pi / period)) 99 | return np.where(value>0, 1, -1) 100 | 101 | # Unique functions 102 | # Signum 103 | def signum(x): 104 | return np.sign(x) 105 | 106 | # Sum of two functions 107 | def sum_two(func1, func2, x): 108 | return func1(x) + func2(x) 109 | 110 | # Sum of three functions 111 | def sum_three(func1, func2, func3, x): 112 | return func1(x) + func2(x) + func3(x) 113 | 114 | # Multiply two functions 115 | def multiply_two(func1, func2, x): 116 | return func1(x) * func2(x) 117 | 118 | # Convolution 119 | def convolution(func1, func2, x): 120 | y1 = func1(x) 121 | y2 = func2(x) 122 | return scipy.signal.convolve(y1, y2, mode='same') 123 | 124 | # Composition of functions 125 | def compose(func1, func2, x): 126 | return func1(func2(x)) 127 | 128 | # Interventions 129 | # Interval function 130 | def interval_smaller_function(func1, x, boundary, mean): 131 | if type(x) == int or type(x) == float: 132 | x = np.array(x) 133 | return np.where(x > boundary, func1(x), np.random.normal(loc=mean, scale=0.1, size=x.shape)) 134 | 135 | def interval_larger_function(func1, x, boundary, mean): 136 | if type(x) == int or type(x) == float: 137 | x = np.array(x) 138 | return np.where(x < boundary, func1(x), np.random.normal(loc=mean, scale=0.1, size=x.shape)) 139 | 140 | # Interval Exclude function 141 | def interval_exclude_function(func1, x, a, b, mean): 142 | if type(x) == int or type(x) == float: 143 | x = np.array(x) 144 | return np.where((x > a) & (x < b), np.random.normal(loc=mean, scale=0.1, size=x.shape), func1(x)) 145 | 146 | def constant_fct(a, x): 147 | return np.ones_like(x)*a 148 | 149 | # Sampling function 150 | def sample_single_f(corruption=False, base_func_names=None): 151 | func_dict = {"linear": linear, "polynomial": polynomial, "absolute": absolute, "root": root, "logarithm": logarithm, 152 | "step": step, "signum": signum, "relu": relu, "tanh": tanh, 'constant': constant_fct, 153 | 'sin': sin, 'cos': cos, 'tan': tan, 'ceil': ceil, 'floor': floor, 'reciprocal': reciprocal, 154 | 'gaussian': gaussian, 'rational': rational, 'error_func': error_func, 155 | 'rectangle': rectangle, 'square_wave':square_wave} 156 | if base_func_names is None: 157 | base_func_names = ["linear", "polynomial", "absolute", "root", "logarithm", "step", "signum", "relu", 158 | "tanh", 'constant', 'sin', 'cos', 'tan', 'reciprocal', 'gaussian', 'rational', 159 | 'rectangle', 'square_wave'] 160 | if corruption: 161 | base_func_names = ["linear", 'constant', 'polynomial'] 162 | func_name = np.random.choice(base_func_names) 163 | func = func_dict[func_name] 164 | scale, bias = [round(i, 1) for i in [random.uniform(-30, 30) for _ in range(2)]] 165 | amplitude = round(np.random.choice(np.linspace(-10, 10, 100)), 1) 166 | phase_shift = round(np.random.choice(np.linspace(0.1, 2 * np.pi, 100)), 1) 167 | period = round(np.random.choice(np.linspace(0.1, 2 * np.pi, 100)), 1) 168 | if func_name == "linear": 169 | m = round(random.uniform(-10, 10),1) 170 | b = round(random.uniform(-10, 10),1) 171 | return lambda x: func(m, b, x), (m, b), func_name 172 | elif func_name == "polynomial": 173 | degree = random.randint(2, 5) # Degree of the polynomial 174 | coeffs = [round(i, 1) for i in [random.uniform(-5, 5) for _ in range(degree + 1)]] 175 | return (lambda x: func(coeffs, x)), coeffs, func_name 176 | elif func_name == "rational": 177 | num_order = random.randint(0, 4) # numerator order 178 | denom_order = random.randint(num_order + 1, 5) # denominator order 179 | num_coeff = [round(i, 1) for i in [random.uniform(-10, 10) for _ in range(num_order+1)]] # numerator coefficients 180 | denom_coeff = [round(i, 1) for i in [random.uniform(-10, 10) for _ in range(denom_order+1)]] # denominator coefficients 181 | return lambda x: rational(num_coeff, denom_coeff, x), (num_coeff, denom_coeff), func_name 182 | elif func_name == "reciprocal": 183 | a = round(random.uniform(-10, 10), 1) 184 | return lambda x: reciprocal(a, x), (a,), func_name 185 | elif func_name == "root": 186 | n = random.randint(1, 2) 187 | return lambda x: func(scale, bias, n, x), (scale, bias, n), func_name 188 | elif func_name == "gaussian": 189 | mean = round(random.uniform(-50, 50),1) 190 | std = round(random.uniform(1, 10),1) 191 | return lambda x: gaussian(mean, std, x), (mean, std), func_name 192 | elif func_name == "absolute": 193 | return lambda x: func(scale, bias, x), (scale, bias), func_name 194 | elif func_name == "topower": 195 | a = round(np.random.choice(np.linspace(-2, 2, 100)),1) 196 | return lambda x: func(scale, bias, a, x), (scale, bias, a), func_name 197 | elif func_name == "logarithm": 198 | base = round(random.uniform(1, 10),1) 199 | return lambda x: func(scale, bias, base, x), (scale, bias, base), func_name 200 | elif func_name == "step": 201 | a = round(random.uniform(-100, 100),1) 202 | return lambda x: func(scale, bias, a, x), (scale, bias, a), func_name 203 | elif func_name == "square_wave": 204 | period = round(random.uniform(1, 10),1) # period of the wave, adjust these bounds as you like 205 | shift = round(random.uniform(-5, 5),1) # phase shift, adjust these bounds as you like 206 | return lambda x: square_wave(period, shift, x), (period, shift), func_name 207 | elif func_name == "rectangle": 208 | w = round(random.uniform(1, 10),1) # width of the rectangle, adjust these bounds as you like 209 | return lambda x: rectangle(w, x), (w,), func_name 210 | elif func_name == "signum": 211 | return func, None, func_name 212 | elif func_name == "relu": 213 | leak = round(random.uniform(0, 1),1) 214 | return lambda x: func(scale, bias, x, leak), (scale, bias, leak), func_name 215 | elif func_name == "sigmoid": 216 | temperature = 1 217 | return lambda x: func(scale, bias, x, temperature), (scale, bias, temperature), func_name 218 | elif func_name == "tanh": 219 | return lambda x: func(amplitude, phase_shift, period, bias, x), (amplitude, phase_shift, period, bias), func_name 220 | elif func_name == "sin": 221 | return lambda x: func(amplitude, phase_shift, period, bias, x), (amplitude, phase_shift, period, bias), func_name 222 | elif func_name == "cos": 223 | return lambda x: func(amplitude, phase_shift, period, bias, x), (amplitude, phase_shift, period, bias), func_name 224 | elif func_name == "tan": 225 | return lambda x: func(amplitude, phase_shift, period, bias, x), (amplitude, phase_shift, period, bias), func_name 226 | elif func_name == "constant": 227 | a = np.random.randint(-100, 100) 228 | return lambda x: func(a, x), (a), func_name 229 | elif func_name == "ceil": 230 | return func, None, func_name 231 | elif func_name == "floor": 232 | return func, None, func_name 233 | elif func_name == "pow": 234 | p = np.random.randint(2, 10) 235 | return lambda x: func(x, scale, bias, p), (scale, bias, p), func_name 236 | elif func_name == "error_func": 237 | return func, None, func_name 238 | else: 239 | print(func_name) 240 | raise NotImplementedError 241 | 242 | def generate_noise(y, noise_type, noise_scale): 243 | if noise_type is None: 244 | return 0 245 | elif noise_type == 'gaussian': 246 | return np.random.normal(scale=noise_scale, size=len(y)) 247 | elif noise_type == 'uniform': 248 | return np.random.uniform(low=-noise_scale, high=noise_scale, size=len(y)) 249 | elif noise_type == 'poisson': 250 | return np.random.poisson(lam=noise_scale, size=len(y)) 251 | 252 | def sample_corruption(f_to_corrupt, domain): 253 | """ 254 | Samples a corruption function and returns the corrupted function 255 | Only corrupts a function by changing an interval of the function, 256 | on which the corruption function is applied. The interval length is between 5 and 25. 257 | :param f_to_corrupt: 258 | :return: corrupted function, parameters of corruption function, name of corruption function 259 | """ 260 | cparams = [] 261 | mean_val = np.mean(f_to_corrupt(np.linspace(domain[0], domain[1], 100))) 262 | std = mean_val 263 | cparams.append(std) 264 | a = round(random.uniform(max(domain[0], -100), 100),1) 265 | cparams.append(a) 266 | 267 | 268 | if np.random.choice([True, False]): 269 | if np.random.choice([True, False]): 270 | corrupted = lambda x: interval_smaller_function(f_to_corrupt, x, a, std) 271 | cname = f'interval_smaller' 272 | else: 273 | corrupted = lambda x: interval_larger_function(f_to_corrupt, x, a, std) 274 | cname = f'interval_larger' 275 | else: 276 | b = random.uniform(a + 5, a + 25) 277 | cparams.append(b) 278 | corrupted = lambda x: interval_exclude_function(f_to_corrupt, x, a, b, std) 279 | cname = f'interval_inside' 280 | return corrupted, cparams, cname 281 | 282 | 283 | 284 | 285 | def sample_f(compound_func_names=None, single_func_names1=None, single_func_names2=None): 286 | func_dict = {"sum_two": sum_two, "sum_three": sum_three, "multiply_two": multiply_two, "convolution": convolution, 287 | "compose": compose} 288 | if compound_func_names is None: 289 | compound_functions = [sum_two, sum_three, multiply_two, convolution, compose] 290 | 291 | func_name = np.random.choice(compound_func_names) 292 | func = func_dict[func_name] 293 | func1, params1, name1 = sample_single_f(base_func_names=single_func_names1) 294 | func2, params2, name2 = sample_single_f(base_func_names=single_func_names2) 295 | 296 | if func_name == "sum_three": 297 | func3, params3, name3 = sample_single_f() 298 | func_name = f'{name1};{name2};{name3};{func_name}' 299 | return lambda x: func(func1, func2, func3, x), [params1, params2, params3], func_name 300 | else: 301 | if func_name == "compose" and name2 == "factorial": 302 | func_name = "sum_two" 303 | func = sum_two 304 | func_name = f'{name1};{name2};{func_name}' 305 | return lambda x: func(func1, func2, x), [params1, params2], func_name 306 | 307 | 308 | 309 | def sample(p_simple=0.85): 310 | basic = ["linear", "polynomial", "absolute", "root", "logarithm", "step", "relu", 311 | "tanh", 'constant', 'sin', 'cos', 'reciprocal', 'gaussian', 'rational', 312 | 'rectangle', 'square_wave'] 313 | if random.random() < p_simple: 314 | f, params, f_name = sample_single_f(base_func_names=basic) 315 | return f, params, f_name 316 | else: 317 | compose_only = ["linear", "polynomial", "step", "relu", 'constant', 318 | "ceil", "floor",'rectangle', 'square_wave'] 319 | f, params, f_name = sample_f(compound_func_names=["multiply_two", "sum_two"], 320 | single_func_names1=compose_only, 321 | single_func_names2=compose_only) 322 | return f, params, f_name 323 | 324 | 325 | if __name__ == '__main__': 326 | for i in range(1000): 327 | f, paramm, f_name = sample() 328 | print(f_name) 329 | -------------------------------------------------------------------------------- /src/make_functions/make_numeric/train_functions.py: -------------------------------------------------------------------------------- 1 | import pdb 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.optim as optim 6 | import os 7 | import math_functions as mf 8 | import matplotlib.pyplot as plt 9 | from utils import timer_decorator 10 | 11 | 12 | # Define our MLP model 13 | class MLP(nn.Module): 14 | def __init__(self): 15 | super(MLP, self).__init__() 16 | self.layers = nn.Sequential( 17 | nn.Linear(1, 64), 18 | nn.ReLU(), 19 | nn.Linear(64, 1) 20 | ) 21 | 22 | def forward(self, x): 23 | return self.layers(x) 24 | 25 | @timer_decorator 26 | def train_function(x, f, func_name, x_test, dirname, verbose=False): 27 | device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 28 | y = f(x) 29 | 30 | x = torch.tensor(x, dtype=torch.float32).to(device) 31 | y = torch.tensor(y, dtype=torch.float32).to(device) 32 | 33 | # Initialize the model, loss function and optimizer 34 | model = MLP().to(device) 35 | criterion = nn.MSELoss() 36 | optimizer = optim.Adam(model.parameters(), lr=0.01) 37 | 38 | # Train the model 39 | model.train() 40 | epochs = 10000 41 | for epoch in range(epochs): 42 | optimizer.zero_grad() 43 | outputs = model(x) 44 | loss = criterion(outputs, y) 45 | loss.backward() 46 | optimizer.step() 47 | 48 | if verbose and epoch % 1000 == 0: 49 | print(f'Epoch: {epoch}, Loss: {loss.item()}') 50 | 51 | if verbose: 52 | print("Training complete.") 53 | y_test = torch.tensor(f(x_test), dtype=torch.float32).to(device) 54 | x_test = torch.tensor(x_test, dtype=torch.float32).to(device) 55 | y_pred = model(x_test) 56 | # Test the model with some input 57 | loss = criterion(y_pred, y_test) 58 | torch.save(model.state_dict(), f'{dirname}/mlp_approx_model.pt') 59 | 60 | # plot the result in a figure with two subplots 61 | plt.subplot(3, 1, 1) 62 | # plt.tight_layout() 63 | plt.plot(x.cpu(), y.cpu(), label='real') 64 | plt.title('Real function') 65 | plt.subplot(3, 1, 2) 66 | plt.plot(x.cpu(), model(x).detach().cpu().numpy(), label='predicted') 67 | plt.title('Predicted function') 68 | plt.subplot(3, 1, 3) 69 | plt.plot(x.cpu(), y.cpu(), label='real') 70 | plt.plot(x.cpu(), model(x).detach().cpu().numpy(), label='predicted') 71 | plt.savefig(f'{dirname}/mlp_approx_plot.png') 72 | plt.close() 73 | return loss.item() 74 | 75 | 76 | 77 | 78 | 79 | -------------------------------------------------------------------------------- /src/make_functions/make_numeric/utils.py: -------------------------------------------------------------------------------- 1 | import time 2 | import matplotlib.pyplot as plt 3 | import importlib 4 | import os 5 | import numpy as np 6 | 7 | def get_test_domain(str): 8 | if 'log' in str: 9 | return [1, 100] 10 | elif 'root' in str: 11 | return [0, 100] 12 | else: 13 | return [-100, 100] 14 | 15 | def get_type(str): 16 | if ';' not in str: 17 | return "simple" 18 | else: 19 | if 'compose' in str: 20 | return "compose" 21 | else: 22 | return str.split(';')[0] 23 | 24 | def get_domain(str): 25 | if 'log' in str: 26 | return [1, 1000] 27 | elif 'root' in str: 28 | return [0, 1000] 29 | else: 30 | return [-1000, 1000] 31 | 32 | 33 | def timer_decorator(func): 34 | def wrapper(*args, **kwargs): 35 | start_time = time.time() # get the current time 36 | result = func(*args, **kwargs) # call the original function 37 | end_time = time.time() # get the current time again 38 | print(f"Time elapsed for {func.__name__}: {end_time - start_time} seconds.") 39 | return result # return the result of the original function 40 | return wrapper 41 | 42 | def import_function(module_name, func_name): 43 | module = importlib.import_module(module_name) 44 | function = getattr(module, func_name) 45 | return function 46 | 47 | def create_string(func_name, params): 48 | if type(params) != list: 49 | params = [params] 50 | param_str = ', '.join(str(p) for p in params) if params else '' 51 | return f"{func_name}({param_str})" 52 | 53 | def print_functions(dirname, func_name, f, function, corrupted, func_dict=None): 54 | # check if the code is correct 55 | corrupted_function = import_function(dirname.replace('/', '.') + '.corrupted_function_code', 56 | 'function') 57 | noised_function = import_function(dirname.replace('/', '.') + '.noised_function_code', 'function') 58 | os.makedirs(f'{dirname}/plots/', exist_ok=True) 59 | domain = get_test_domain(func_name) 60 | x = np.linspace(domain[0], domain[1], 1000).reshape(-1, 1) 61 | y = f(x) 62 | plt.plot(x, y) 63 | plt.savefig(f'{dirname}/plots/function.png') 64 | plt.close() 65 | y1 = function(x) 66 | plt.plot(x, y1) 67 | plt.savefig(f'{dirname}/plots/function_from_code.png') 68 | plt.close() 69 | if func_dict is not None: 70 | func_dict['error_code_generated'] = np.mean(y1 - y) 71 | yc = corrupted(x) 72 | plt.plot(x, yc) 73 | plt.savefig(f'{dirname}/plots/corrupted.png') 74 | plt.close() 75 | yc1 = corrupted_function(x) 76 | plt.plot(x, yc1) 77 | plt.savefig(f'{dirname}/plots/corrupted_from_code.png') 78 | plt.close() 79 | 80 | yc = noised_function(x) 81 | plt.plot(x, yc) 82 | plt.plot(x, yc) 83 | plt.savefig(f'{dirname}/plots/noised.png') 84 | plt.close() 85 | if func_dict is not None: 86 | func_dict['error_corrupted_code_generated'] = np.mean(y1 - y) 87 | return func_dict -------------------------------------------------------------------------------- /src/make_functions/make_numeric/write_source_code.py: -------------------------------------------------------------------------------- 1 | # main script 2 | import pdb 3 | import random 4 | import math_functions as mf 5 | import textwrap 6 | import re 7 | 8 | 9 | def write_simple_function(params, func_name, f_name='function'): 10 | code = f""" 11 | def {f_name}(x): 12 | """ 13 | 14 | # For each type of function, we need to generate code accordingly 15 | if func_name == "linear": 16 | code += f""" 17 | return {params[0]}*x + {params[1]} 18 | """ 19 | elif func_name == "polynomial": 20 | degree = len(params) - 1 21 | terms = ["{}*x**{}".format(params[i], degree-i) for i in range(degree+1)] 22 | string = " + ".join(terms) 23 | code += f""" 24 | return {string} 25 | """ 26 | elif func_name == "rational": 27 | code += f""" 28 | return ({' + '.join([f'{params[0][i]}*x**{i}' for i in range(len(params[0]))])}) / ({' + '.join([f'{params[1][i]}*x**{i}' for i in range(len(params[1]))])}) 29 | """ 30 | elif func_name == "root": 31 | code += f""" 32 | return {params[0]}*(x**(1.0 / {params[2]})) + {params[1]} 33 | """ 34 | elif func_name == "reciprocal": 35 | code += f""" 36 | return 1 / ({params[0]} * x) 37 | """ 38 | elif func_name == "error_func": 39 | code += f""" 40 | from math import erf 41 | return erf(x) 42 | """ 43 | elif func_name == "square_wave": 44 | code += f""" 45 | x = np.sin((x - {params[1]}) * (2 * np.pi / {params[0]})) 46 | return np.where(x > 0, 1, -1) 47 | """ 48 | elif func_name == "absolute": 49 | code += f""" 50 | return {params[0]}*np.abs(x)""" 51 | elif func_name == "topower": 52 | code += f""" 53 | return {params[0]}*({params[2]}**x) + {params[1]} 54 | """ 55 | elif func_name == "logarithm": 56 | code += f""" 57 | return {params[0]}*(np.log(x) / np.log({params[2]})) + {params[1]} 58 | """ 59 | elif func_name == "step": 60 | code += f""" 61 | return {params[0]}*np.heaviside(x, {params[2]})+ {params[1]} 62 | """ 63 | elif func_name == "rectangle": 64 | code += f""" 65 | return np.where(abs(x) <= {params[0]}/2,1,0) 66 | """ 67 | elif func_name == "signum": 68 | code += f""" 69 | return np.sign(x) 70 | """ 71 | elif func_name == "relu": 72 | code += f""" 73 | return {params[0]} * np.where(x > 0, x, x * {params[2]}) + {params[1]} 74 | """ 75 | elif func_name == "sigmoid": 76 | code += f""" 77 | return {params[0]} *(1 / (1 + np.exp(-x/{params[2]}))) + {params[1]} 78 | """ 79 | elif func_name == "tanh": 80 | code += f""" 81 | return {params[0]} * np.tanh((2 * np.pi / {params[1]})*(x-{params[2]})) + {params[3]} 82 | """ 83 | elif func_name == "gaussian": 84 | code += f""" 85 | 86 | return (1 / ({params[1]} * np.sqrt(2 * np.pi))) * np.exp(-0.5 * ((x - {params[0]}) / {params[1]}) ** 2) 87 | """ 88 | elif func_name == "student_t": 89 | code += f""" 90 | from scipy.special import gamma 91 | return gamma(({params[0]}+1)/2) / (np.sqrt({params[0]}*np.pi) * gamma({params[0]}/2)) * (1 + x**2/{params[0]}) ** (-({params[0]}+1)/2) 92 | """ 93 | elif func_name == "poisson": 94 | code += f""" 95 | from scipy.special import factorial 96 | return ({params[0]}**x * np.exp(-{params[0]})) / factorial(x) 97 | """ 98 | elif func_name == "sin": 99 | code += f""" 100 | return {params[0]} * np.sin((2 * np.pi / {params[1]})*(x-{params[2]})) + {params[3]} 101 | """ 102 | elif func_name == "cos": 103 | code += f""" 104 | return {params[0]} * np.cos((2 * np.pi / {params[1]})*(x-{params[2]})) + {params[3]} 105 | """ 106 | elif func_name == "tan": 107 | code += f""" 108 | return {params[0]} * np.tan((2 * np.pi / {params[1]})*(x-{params[2]})) + {params[3]} 109 | """ 110 | elif func_name == "constant": 111 | code += f""" 112 | return {params}* np.ones_like(x) 113 | """ 114 | elif func_name == "ceil": 115 | code += f""" 116 | return np.ceil(x) 117 | """ 118 | elif func_name == "floor": 119 | code += f""" 120 | return np.floor(x) 121 | """ 122 | elif func_name == "pow": 123 | code += f""" 124 | x = np.where(x < 0, 0, x) if {params[2]} % 1 != 0 else x 125 | return {params[0]}*np.power(x, {params[2]}) + {params[1]} 126 | """ 127 | 128 | else: 129 | raise ValueError(f"Unknown function name: {func_name}") 130 | 131 | 132 | return code 133 | 134 | 135 | def extract_function_call(string): 136 | # Use a regular expression to find the text between "return" and "\n" 137 | match = re.search(r'return (.*)\n', string) 138 | match1 = re.search(r'return (.*)', string) 139 | if match: 140 | result = match.group(1) 141 | elif match1: 142 | result = match1.group(1) 143 | return result 144 | 145 | def wrtie_noised_function(code, dirname): 146 | start_code = """ 147 | import numpy as np 148 | import scipy.signal 149 | import sys 150 | from scipy.special import gamma, factorial, erf 151 | 152 | """ 153 | clean_function_code = extract_function_call(code) 154 | if random.random() < 0.33: 155 | noise_type = 'normal' 156 | add_code = f""" 157 | def function(x): 158 | return {clean_function_code} + np.random.normal(scale=1)\n""" 159 | elif random.random() < 0.66: 160 | noise_type = 'uniform' 161 | add_code = f""" 162 | def function(x): 163 | return {clean_function_code} + np.random.uniform(low=-1, high=1)\n""" 164 | else: 165 | noise_type = 'poisson' 166 | add_code = f""" 167 | def function(x): 168 | return {clean_function_code} + np.random.poisson(lam=1)\n""" 169 | code = textwrap.dedent(start_code) 170 | add_code = textwrap.dedent(add_code) 171 | code += add_code 172 | run_code = """ 173 | if __name__ == '__main__': 174 | outputs = '' 175 | for arg in sys.argv[1:]: 176 | x = float(arg) 177 | try: 178 | out = function(x) 179 | except: 180 | out = 'None' 181 | outputs += f'({arg}, {out}) ' 182 | print(f'Function input - output pairs: {outputs}') 183 | 184 | """ 185 | code += textwrap.dedent(run_code) 186 | with open(f"{dirname}/noised_function_code.py", "w") as file: 187 | if code[0] == '\n': 188 | code = code[1:] 189 | file.write(code) 190 | return noise_type, 1 191 | 192 | 193 | def write_corrupted_function(code, corrupted, cparams, cname, dirname): 194 | start_code = """ 195 | import numpy as np 196 | import scipy.signal 197 | import sys 198 | from scipy.special import gamma, factorial, erf 199 | 200 | """ 201 | clean_function_code = extract_function_call(code) 202 | 203 | std = cparams[0] 204 | if cname == 'interval_smaller': 205 | corrupted_code = f'return np.where(x > {cparams[1]}, {clean_function_code}, np.random.normal(loc={std},scale=0.1, size=x.shape))\n' 206 | elif cname == 'interval_larger': 207 | corrupted_code = f'return np.where(x < {cparams[1]}, {clean_function_code}, np.random.normal(loc={std}, scale=0.1,size=x.shape))\n' 208 | elif cname == 'interval_inside': 209 | corrupted_code = f'return np.where((x > {cparams[1]}) & (x < {cparams[2]}), np.random.normal(loc={std},scale=0.1, size=x.shape), {clean_function_code})\n' 210 | else: 211 | raise ValueError(f"Unknown corrupted function name: {cname}") 212 | # Add the interval function to the code 213 | new_code1 = f""" 214 | def function(x): 215 | if type(x) == int or type(x) == float: 216 | x = np.array(x) 217 | {corrupted_code} 218 | """ 219 | 220 | code = textwrap.dedent(start_code) 221 | new_code1 = textwrap.dedent(new_code1) 222 | 223 | run_code = """ 224 | if __name__ == '__main__': 225 | outputs = '' 226 | for arg in sys.argv[1:]: 227 | x = float(arg) 228 | try: 229 | out = function(x) 230 | except: 231 | out = 'None' 232 | outputs += f'({arg}, {out}) ' 233 | print(f'Function input - output pairs: {outputs}') 234 | 235 | 236 | """ 237 | code += new_code1 238 | code += textwrap.dedent(run_code) 239 | with open(f"{dirname}/corrupted_function_code.py", "w") as file: 240 | if code[0] == '\n': 241 | code = code[1:] 242 | file.write(code) 243 | 244 | 245 | 246 | def write_function(params, func_name, dirname, corrupted=None, cparams=None, cname=None): 247 | start_code = """ 248 | import numpy as np 249 | import scipy.signal 250 | import sys 251 | from scipy.special import gamma, factorial, erf 252 | 253 | """ 254 | if ';' not in func_name: 255 | code = write_simple_function(params, func_name) 256 | else: 257 | if 'sum_three' in func_name: 258 | params1, params2, params3 = params 259 | func_name1, func_name2, func_name3 = func_name.split(';')[:3] 260 | code1 = write_simple_function(params1, func_name1, f_name='function1') 261 | code2 = write_simple_function(params2, func_name2, f_name='function2') 262 | code3 = write_simple_function(params3, func_name3, f_name='function3') 263 | clean_function_code1 = extract_function_call(code1) 264 | clean_function_code2 = extract_function_call(code2) 265 | clean_function_code3 = extract_function_call(code3) 266 | code = f""" 267 | def function(x): 268 | return {clean_function_code1} + {clean_function_code2} + {clean_function_code3} 269 | """ 270 | else: 271 | params1, params2 = params 272 | func_name1, func_name2, func_name = func_name.split(';') 273 | code1 = write_simple_function(params1, func_name1, f_name='function1') 274 | code2 = write_simple_function(params2, func_name2, f_name='function2') 275 | clean_function_code1 = extract_function_call(code1) 276 | clean_function_code2 = extract_function_call(code2) 277 | if func_name == 'sum_two': 278 | code = f""" 279 | def function(x): 280 | return {clean_function_code1} + {clean_function_code2} 281 | """ 282 | elif func_name == 'multiply_two': 283 | code = f""" 284 | def function(x): 285 | return ({clean_function_code1}) * ({clean_function_code2}) 286 | """ 287 | elif func_name == 'convolution': 288 | raise NameError('Not implemented yet') 289 | elif func_name == 'compose': 290 | clean_function_code1 = clean_function_code1.replace('exp', 'PLACEHOLDER') 291 | clean_function_code1 = clean_function_code1.replace('x', clean_function_code2) 292 | clean_function_code1 = clean_function_code1.replace('PLACEHOLDER', 'exp') 293 | code = f""" 294 | def function(x): 295 | return {clean_function_code1} 296 | """ 297 | else: 298 | raise ValueError('Unknown function name') 299 | 300 | start_code = textwrap.dedent(start_code) 301 | code = textwrap.dedent(code) 302 | run_code = """ 303 | if __name__ == '__main__': 304 | outputs = '' 305 | for arg in sys.argv[1:]: 306 | x = float(arg) 307 | try: 308 | out = function(x) 309 | except: 310 | out = 'None' 311 | outputs += f'({arg}, {out}) ' 312 | print(f'Function input - output pairs: {outputs}') 313 | 314 | 315 | """ 316 | start_code += code 317 | 318 | noise_type, noise_params = wrtie_noised_function(code, dirname) 319 | 320 | if corrupted and cparams and cname: 321 | write_corrupted_function(code, corrupted, cparams, cname, dirname) 322 | 323 | start_code += textwrap.dedent(run_code) 324 | with open(f"{dirname}/function_code.py", "w") as file: 325 | if start_code[0] == '\n': 326 | start_code = start_code[1:] 327 | file.write(start_code) 328 | return noise_type, noise_params 329 | 330 | 331 | 332 | -------------------------------------------------------------------------------- /src/make_functions/make_strings/README.md: -------------------------------------------------------------------------------- 1 | # FIND interpretation benchmark 2 | # Strings Function Generator API 3 | 4 | Welcome to the Strings Function Generator API documentation. Our API allows developers to create custom functions that operate on strings for various applications such as data analysis, machine learning, algorithmic operations and more. 5 | 6 | ## Generating a function library 7 | ```python generate_string_functions.py --num_string_functions 100``` 8 | generates 100 randomly sampled strings functions and saves them into the `string_functions` sub-directory. The `--debug` flag will plot each function to a file. 9 | Each function will have a subdirectory in the `dir` folder: 10 | ``` 11 | string_functions/ 12 | string_functions/f00000/function_code.py 13 | string_functions/f00001/function_code.py 14 | ... 15 | ``` 16 | -------------------------------------------------------------------------------- /src/make_functions/make_strings/__pycache__/string_functions.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/make_functions/make_strings/__pycache__/string_functions.cpython-38.pyc -------------------------------------------------------------------------------- /src/make_functions/make_strings/__pycache__/write_source_code.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/multimodal-interpretability/FIND/670d837a27cfd39e82d8eefd82976657f3b46a58/src/make_functions/make_strings/__pycache__/write_source_code.cpython-38.pyc -------------------------------------------------------------------------------- /src/make_functions/make_strings/generate_string_functions.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pdb 3 | 4 | import string_functions as sf 5 | from write_source_code import write_function 6 | import numpy as np 7 | import importlib 8 | import json 9 | import matplotlib.pyplot as plt 10 | from tqdm import tqdm 11 | 12 | 13 | 14 | 15 | def create_string(func_name, params): 16 | if type(params) != list: 17 | params = [params] 18 | param_str = ', '.join(str(p) for p in params) if params else '' 19 | return f"{func_name}({param_str})" 20 | 21 | def get_type(str): 22 | if ';' not in str: 23 | return "simple" 24 | else: 25 | if 'compose' in str: 26 | return "compose" 27 | else: 28 | return str.split(';')[0] 29 | 30 | 31 | def import_function(module_name, func_name): 32 | module = importlib.import_module(module_name) 33 | function = getattr(module, func_name) 34 | return function 35 | 36 | if __name__ == '__main__': 37 | import argparse 38 | parser = argparse.ArgumentParser(description='Description of your script') 39 | parser.add_argument('--num_string_functions', type=int, help='Number of functions to generate', default=1000) 40 | parser.add_argument('--dir', type=str, default='string_functions') 41 | parser.add_argument('--ratio', type=float, default=0.3) 42 | args = parser.parse_args() 43 | 44 | # args.dir = '%s/composed_fun_%.2f' %(args.dir, args.ratio) 45 | 46 | # Save all meta data in a json file 47 | data = [] 48 | for i in tqdm(range(args.num_string_functions)): 49 | dirname = f'{args.dir}/f{i:05d}' 50 | os.makedirs(dirname, exist_ok=True) 51 | 52 | f, params, func_name = sf.sample(args) 53 | name = create_string(func_name, params) 54 | 55 | 56 | # write source code 57 | write_function(params, func_name, dirname) 58 | 59 | # save meta data 60 | func_dict = {'type': get_type(func_name), 'params': params, 'name': name, 'dir': dirname} 61 | 62 | 63 | print('----------------------------------------------------------') 64 | print(func_dict) 65 | 66 | data.append(func_dict) 67 | 68 | # Write the data to a JSON file 69 | with open(f"{args.dir}/data.json", "w") as file: 70 | json.dump(data, file) 71 | 72 | -------------------------------------------------------------------------------- /src/make_functions/make_strings/utils.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | def timer_decorator(func): 4 | def wrapper(*args, **kwargs): 5 | start_time = time.time() # get the current time 6 | result = func(*args, **kwargs) # call the original function 7 | end_time = time.time() # get the current time again 8 | print(f"Time elapsed for {func.__name__}: {end_time - start_time} seconds.") 9 | return result # return the result of the original function 10 | return wrapper -------------------------------------------------------------------------------- /src/run_interpretations/README.md: -------------------------------------------------------------------------------- 1 | # FIND interpretation benchmark 2 | # Run interpretations 3 | 4 | ## set OPENAI_API_KEY the environment variable 5 | ```export OPENAI_API_KEY= ``` 6 | 7 | ## Collect interpretations from gpt-4 and gpt-3.5-turbo 8 | 9 | ### Numeric functions 10 | ```python collect_interpretations.py --func_category numeric --prompt_path ./prompts/numeric.txt``` 11 | 12 | ### Strings functions 13 | ```python collect_interpretations.py --func_category strings --prompt_path ./prompts/strings.txt``` 14 | 15 | ## Synthetic neurons - entities 16 | Install FastChat and the OpenAI API 17 | ```bash 18 | pip install fschat 19 | pip install openai 20 | ``` 21 | Follow [these instructions](https://huggingface.co/docs/transformers/main/model_doc/llama) to download the llama-13b weights and install dependencies. Then download vicuna-13b delta weights from [here](https://github.com/lm-sys/FastChat#vicuna-weights). 22 | Then start serving Vicuna-13b (run these three commands separately): 23 | ```bash 24 | python -m fastchat.serve.controller 25 | python -m fastchat.serve.model_worker --model-name 'vicuna-13b-v1.1' --model-path /path/to/vicuna-13b --num-gpus 2 26 | python -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 27 | ``` 28 | (this is only needed for synthetic neurons functions) 29 | 30 | Run interpretations: 31 | ```python collect_interpretations.py --func_category neurons_entities --prompt_path ./prompts/neurons_entities.txt --vicuna_server --llama_hf_path ``` 32 | 33 | Run interpretations with search initialization: 34 | ```python collect_interpretations.py --func_category neurons_entities --prompt_path ./prompts/neurons_entities.txt --vicuna_server --llama_hf_path --hints``` 35 | 36 | 37 | ## Synthetic neurons - relations 38 | Lanch vicuna (see instructions in the entities function above). 39 | Run interpretations: 40 | ```python collect_interpretations.py --func_category neurons_relations --prompt_path ./prompts/relations.txt --vicuna_server ``` 41 | Run interpretations with search initialization: 42 | ```python collect_interpretations.py --func_category neurons_relations --prompt_path ./prompts/relations.txt --vicuna_server --hints``` 43 | 44 | ## Define your own interpreter 45 | Open the `collect_interpretations.py` function and define your model according to the instructions. 46 | 47 | ## FIND interpretations benchmark 48 | You can also download the full FIND interpretations benchmark and unzip it to `./src/run_interpretations/`: 49 | ```bash 50 | wget -P ./src/run_interpretations https://data.csail.mit.edu/FIND/FIND-interpretations.zip 51 | unzip ./src/run_interpretations/FIND-interpretations.zip -d ./src/run_interpretations/ 52 | ``` -------------------------------------------------------------------------------- /src/run_interpretations/call_interpreter.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import openai 3 | import regex 4 | 5 | import pandas as pd 6 | from getpass import getpass 7 | import pandas as pd 8 | from tqdm import tqdm 9 | from IPython import embed 10 | import time 11 | from random import random, uniform 12 | import warnings 13 | warnings.filterwarnings("ignore") 14 | 15 | # User inputs: 16 | # Load your API key from an environment variable or secret management service 17 | # openai.api_key = os.getenv("OPENAI_API_KEY") 18 | # OR 19 | # Load your API key manually: 20 | # openai.api_key = API_KEY 21 | 22 | def ask_model(input,model,state,vicuna_server=None): 23 | try: 24 | if model in ['gpt-3.5-turbo','gpt-4','gpt-4-0314']: 25 | state.history.append({'role': 'user', 'content': input}) 26 | r = openai.ChatCompletion.create(model=model, messages=state.history) 27 | resp = r['choices'][0]['message']['content'] 28 | costFactor = [0.03, 0.06] if model == 'gpt-4' else [0.002, 0.002] 29 | state.totalCost += r['usage']['prompt_tokens']/1000*costFactor[0]+r['usage']['completion_tokens']/1000*costFactor[1] 30 | state.history.append({'role': 'assistant', 'content': resp}) 31 | elif ('vicuna' in model): 32 | state.history.append({'role': 'user', 'content': input}) 33 | openai.api_key = "EMPTY" # Not support yet 34 | openai.api_base = vicuna_server 35 | model = "vicuna-13b-v1.1" 36 | r = openai.ChatCompletion.create(model=model, messages=state.history,temperature=0) 37 | resp = r['choices'][0]['message']['content'] 38 | state.history.append({'role': 'assistant', 'content': resp}) 39 | elif ('llama-2-13b' in model): 40 | state.history.append({'role': 'user', 'content': input}) 41 | openai.api_key = "EMPTY" # Not support yet 42 | openai.api_base = vicuna_server 43 | model = "llama-2-13b-chat" 44 | r = openai.ChatCompletion.create(model=model, messages=state.history,temperature=0.7) 45 | resp = r['choices'][0]['message']['content'] 46 | state.history.append({'role': 'assistant', 'content': resp}) 47 | elif ('llama-2-70b' in model): 48 | state.history.append({'role': 'user', 'content': input}) 49 | openai.api_key = "EMPTY" # Not support yet 50 | openai.api_base = vicuna_server 51 | model = "llama-2-70b-chat" 52 | r = openai.ChatCompletion.create(model=model, messages=state.history,temperature=0.7) 53 | resp = r['choices'][0]['message']['content'] 54 | state.history.append({'role': 'assistant', 'content': resp}) 55 | # TODO: add your model her: 56 | # if model == MY_MODEL: 57 | # state.history.append({'role': 'user', 'content': input}) 58 | # resp = GET_YOUR_MODEL_RESPONSE 59 | # state.history.append({'role': 'assistant', 'content': resp}) 60 | else: 61 | print(f"Unrecognize model name: {model}") 62 | return 0 63 | except Exception as e: 64 | print(e) 65 | if model == 'gpt-3.5-turbo': return e 66 | if 'llama' in model: return e 67 | time.sleep(30*random()) 68 | resp = ask_model(input,model,state,vicuna_server) 69 | return resp,state -------------------------------------------------------------------------------- /src/run_interpretations/collect_interpretations.py: -------------------------------------------------------------------------------- 1 | # some code parts are based on the functions provided by the repo: 2 | # https://github.com/d3n7/GPT-4-Unlimited-Tools 3 | 4 | import argparse 5 | # import openai 6 | import regex 7 | import re 8 | import pandas as pd 9 | import subprocess 10 | from getpass import getpass 11 | import pandas as pd 12 | import os 13 | from tqdm import tqdm 14 | import time 15 | from random import random, uniform 16 | import random as rd 17 | import torch.nn as nn 18 | import json 19 | import warnings 20 | import numpy as np 21 | from call_interpreter import ask_model 22 | rd.seed(0000) 23 | 24 | parser = argparse.ArgumentParser(description='Process Arguments') 25 | parser.add_argument('--model', type=str, default='gpt-4', help='model name') 26 | parser.add_argument('--func_category', type=str, default='numeric', help='function category') 27 | parser.add_argument('--debug', action='store_true', help='debug mode, print dialogues to screan', default=False) 28 | parser.add_argument('--dataset_path', type=str, default='./results/', help='a path to save the model outputs') 29 | parser.add_argument('--function_path', type=str, default='../find_dataset/', help='path to the gt functions') 30 | parser.add_argument('--temp_function_path', type=str, default='./temp/', help='path to copy the gt functions') 31 | parser.add_argument('--prompt_path', type=str, default='./prompts/numeric.txt', help='path to prompt to use') 32 | parser.add_argument('--n_func', type=int, default=5, help='specify the amount of functions to run') 33 | parser.add_argument('--vicuna_server', type=str, help='specify vicuna server address') 34 | parser.add_argument('--vicuna_server_module', type=str, help='specify vicuna server address for the backbone model') 35 | parser.add_argument('--llama_hf_path', type=str, help='specify llama path') 36 | parser.add_argument('--hints', action='store_true', help='add_search_initializations', default=False) 37 | parser.add_argument('--single_round', action='store_true', help='add_search_initializations', default=False) 38 | args = parser.parse_args() 39 | 40 | regx = [r"([A-Z]+\(((?:[^()\"']|(?:\"[^\"]*\")|(?:'[^']*')|\((?1)*\))*)\))", 41 | r'''(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|\b[^,]+)'''] 42 | 43 | vicuna_server = args.vicuna_server 44 | vicuna_server_module = args.vicuna_server_module 45 | 46 | class SessionState: 47 | def __init__(self): 48 | self.running = False 49 | self.followup = False 50 | self.prompt = '' 51 | self.command = '' 52 | self.acceptreject = False 53 | self.history = [] 54 | self.totalCost = 0 55 | self.displayChat = False 56 | self.displayCost = False 57 | 58 | newSession = True 59 | manualApproval = False 60 | 61 | def followup(state): 62 | state.followup, state.running = True, True 63 | 64 | def formatTable(table): 65 | lines = '' 66 | for x, i in enumerate(table['GPT Commands']): 67 | lines += '{} - {}\n'.format(table['GPT Commands'][x],table['GPT Explanations'][x]) 68 | return(lines) 69 | 70 | def formatTable_CommandOnly(table): 71 | lines = '' 72 | for x, i in enumerate(table['GPT Commands']): 73 | lines += '{} - {}\n'.format(table['GPT Commands'][x],table['GPT Explanations'][x]) 74 | return(lines) 75 | 76 | commandTable = pd.DataFrame({ 77 | 'GPT Commands': ['PYTHON(function.py)'], 78 | 'GPT Explanations': ['Run a python script with the given file name. Use quotes for the filename argument. Do not use quotes in the function command itself.'], 79 | 'Raw Translation': ['python {}'] 80 | }) 81 | 82 | # return the prompt according to the task 83 | def return_sysPrompt(model): 84 | if model in ['llama-2']: 85 | sysPrompt = 'You now have the ability to execute commands to run external functions. ' \ 86 | 'You are able to access the user\'s machine with these commands. In every message you send, ' \ 87 | 'include "COMMAND: " with your command at the end. Here is a list of commands with ' \ 88 | 'explanations of how they are used:\n{}\nWhen you use a command, the user will respond ' \ 89 | 'with "Response: " followed by the output of the commmand. Use this output to help the ' \ 90 | 'user complete their request. After you receive a task from the user, you must execute {} ' \ 91 | 'to run the external function. You will then receive outputs from the external function to ' \ 92 | 'analyze. You must only analyze outputs produced by the function when you run {}. Do not run ' \ 93 | 'the function any other way. Do not analyze any other outputs besides the ones produced by running {}.' 94 | else: 95 | sysPrompt = 'You now have access to some commands to help complete the user\'s request. ' \ 96 | 'You are able to access the user\'s machine with these commands. In every message you send, ' \ 97 | 'include "COMMAND: " with your command at the end. Here is a list of commands with ' \ 98 | 'explanations of how they are used:\n{}\nWhen you use a command, the user will respond ' \ 99 | 'with "Response: " followed by the output of the commmand. Use this output to help the ' \ 100 | 'user complete their request.' 101 | return sysPrompt 102 | 103 | def runCmd(flag,state): 104 | if flag: 105 | try: 106 | p = subprocess.Popen(state.command, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE) 107 | output, errors = p.communicate() 108 | state.prompt = 'Response: ' + output.decode("utf-8") 109 | # if errors: 110 | # state.prompt += 'Error occurred, please try again (some of the input values might be undefined)' 111 | # state.prompt += 'Errors: ' + errors.decode("utf-8") 112 | except subprocess.CalledProcessError as e: 113 | print('!!!ERROR!!!') 114 | print(e) 115 | # state.prompt = 'Response: ' + e.output.decode("utf-8") 116 | else: 117 | state.prompt = "Response: User rejected this command" 118 | followup(state) 119 | 120 | def define_prompt(prompt_template,dir2func): 121 | prompt = prompt_template.format(DIR2FUNC=os.path.join(dir2func)) 122 | return prompt 123 | 124 | def save_description(response, filepath): 125 | description = response.rsplit('[DESCRIPTION]: ')[-1] 126 | description = description.rsplit('[DOMAIN]: ')[0] 127 | description = description.rsplit('[CODE]: ')[0] 128 | with open(filepath,'w') as f: 129 | f.write(description) 130 | f.close() 131 | 132 | def save_domain(response,filepath): 133 | domain = response.rsplit('[DOMAIN]: ')[-1] 134 | domain = domain.rsplit('[CODE]: ')[0] 135 | with open(filepath,'w') as f: 136 | f.write(domain) 137 | f.close() 138 | 139 | def save_code(response,filepath): 140 | code = response.rsplit('[CODE]:')[-1] 141 | if code.find('```python') != -1: 142 | code = code.rsplit('```python')[-1] 143 | code = code.rsplit('```')[0] 144 | elif code.find('``` python') != -1: 145 | code = code.rsplit('``` python')[-1] 146 | code = code.rsplit('```')[0] 147 | elif code.find('```') != -1: 148 | code = code.rsplit('```',maxsplit=2)[1] 149 | else: 150 | code = '' 151 | 152 | with open(filepath,'w') as f: 153 | f.write(code) 154 | f.close() 155 | 156 | def save_fullhistory(history,filepath): 157 | with open(filepath+'.txt', 'a') as f: 158 | for i in history: 159 | if i['role'] == 'user': 160 | f.write('\nuser:\n'+i['content']) 161 | elif i['role'] == 'assistant': 162 | f.write('\nassistant:\n'+i['content']) 163 | with open(filepath+'.json', 'w') as file: 164 | json.dump(history, file) 165 | f.close() 166 | 167 | 168 | def interp_func(prompt,model,debug=False, single_round=False): 169 | state = SessionState() 170 | sysPrompt = return_sysPrompt(model) 171 | round_count = 0 172 | while True: 173 | state.running = True 174 | if state.running: 175 | state.running = False 176 | if not state.followup: 177 | state.prompt = prompt 178 | if (newSession or state.history == []) and (not state.followup): 179 | if 'llama' in model: 180 | state.history = [{'role': 'system', 'content': sysPrompt.format(formatTable(commandTable),formatTable_CommandOnly(commandTable),formatTable_CommandOnly(commandTable),formatTable_CommandOnly(commandTable))}] 181 | else: 182 | state.history = [{'role': 'system', 'content': sysPrompt.format(formatTable(commandTable))}] 183 | else: 184 | if 'llama' in model: 185 | state.history[0] = {'role': 'system', 'content': sysPrompt.format(formatTable(commandTable),formatTable_CommandOnly(commandTable),formatTable_CommandOnly(commandTable),formatTable_CommandOnly(commandTable))} 186 | else: 187 | state.history[0] = {'role': 'system', 'content': sysPrompt.format(formatTable(commandTable))} 188 | state.followup = False 189 | response,state = ask_model(state.prompt, model, state, vicuna_server) 190 | if len(regex.findall(regx[0], response)) >= 1: 191 | cmd = regex.findall(regx[0], response)[0][0] 192 | pIndex = cmd.index('(') 193 | stem = cmd[:pIndex] 194 | rawArgs = cmd[pIndex+1:][:-1] 195 | cmdId = -1 196 | 197 | for x, i in enumerate(commandTable['GPT Commands']): 198 | if stem in i: 199 | cmdId = x 200 | break 201 | 202 | rawArgs.replace('\n', '\\n') 203 | rawArgs.replace('\\\n', '\\n') 204 | if cmdId == -1: 205 | state.prompt = 'Response: Unrecognized command' 206 | followup(state) 207 | elif "'''" in rawArgs: 208 | state.prompt = 'Response: Error parsing multi-line string (\'\'\') Use a single line with escaped newlines instead (")' 209 | followup(state) 210 | elif '"""' in rawArgs: 211 | state.prompt = 'Response: Error parsing multi-line string (\"\"\") Use a single line with escaped newlines instead (")' 212 | followup(state) 213 | else: 214 | state.command = commandTable['Raw Translation'][cmdId] 215 | args = [] 216 | if rawArgs != '': 217 | args = re.findall(regx[1], rawArgs) #[rawArgs] 218 | state.command = state.command.format(*args) 219 | 220 | singleQuotes = False 221 | for i in args: 222 | if i.startswith("'"): 223 | singleQuotes = True 224 | state.prompt = "Response: Error parsing argument in single quotes. Use double quotes around the argument instead" 225 | followup(state) 226 | break 227 | 228 | if not singleQuotes: 229 | if manualApproval: 230 | state.acceptreject = False 231 | else: 232 | runCmd(1,state) 233 | 234 | round_count+=1 235 | 236 | if state.acceptreject: 237 | print('GPT is trying to run the following command: ' + state.command + '\nPlease accept or reject this request.') 238 | decision = input('Accept or Reject [Accept, Reject]:') 239 | if decision.lower() == 'accept': 240 | state.acceptreject = False 241 | runCmd(1,state) 242 | elif decision.lower() == 'reject': 243 | state.acceptreject = False 244 | runCmd(0,state) 245 | 246 | 247 | if "[DESCRIPTION]" in response or (single_round and round_count==2): 248 | if debug: 249 | print(response) 250 | return(response,state.history) 251 | 252 | if round_count>20: 253 | # raise Warning("Interpretation process exceeded 100 rounds") 254 | return('','') 255 | 256 | if debug: 257 | print(response) 258 | print(state.prompt) 259 | 260 | def copy_func(source,target,category,vicuna_server_module=None,llama_hf_path=None): 261 | func_name = 'function_code.py' 262 | if 'numeric' in args.func_category: 263 | if os.path.exists(f'{source}/mlp_approx_model.pt'): 264 | os.system(f'scp {source}/mlp_approx_model.pt {target}/mlp_approx_model.pt') 265 | if ('neurons_entities' in category) or ('neurons_relations' in category): 266 | temp = open(f'{source}/{func_name}','r').read() 267 | temp = temp.replace('{API_BASE}',f"'{vicuna_server_module}'") 268 | temp = temp.replace('{LLAMA_PATH}',f"'{llama_hf_path}'") 269 | open(target+'/function.py','w').write(temp) 270 | else: 271 | os.system(f'scp {source}/{func_name} {target}/function.py') 272 | return 273 | 274 | 275 | def remove_func(target): 276 | os.system(f'rm {target}') 277 | 278 | def remove_temp_folder(target): 279 | os.system(f'rm -r {target}') 280 | 281 | 282 | def main(args): 283 | with open(args.prompt_path,'r') as f: 284 | prompt_template = f.read() 285 | # os.mkdir(os.path.join(args.dataset_path,args.model,args.func_category)) 286 | t_start = time.time() 287 | count = 0 288 | remove_temp_folder(args.temp_function_path) 289 | os.makedirs(args.temp_function_path, exist_ok=True) 290 | for function in tqdm(os.listdir(args.function_path+args.func_category)): 291 | print(function) 292 | if count >= args.n_func: break 293 | if not os.path.os.path.exists(os.path.join(args.function_path,args.func_category,function,'function_code.py')): continue 294 | path2save = os.path.join(args.dataset_path,args.model,args.func_category,function) 295 | if args.hints: 296 | path2save = os.path.join(args.dataset_path,args.model,args.func_category+'_hints',function) 297 | if args.single_round: 298 | path2save = os.path.join(args.dataset_path,args.model,args.func_category+'_single_round',function) 299 | if os.path.exists(path2save+'/description.txt'): continue 300 | if os.path.isfile(path2save): continue 301 | 302 | os.makedirs(path2save, exist_ok=True) 303 | mlp_flag = False 304 | if 'numeric' in args.func_category: 305 | if int(function[1:]) < 150: 306 | mlp_flag = True 307 | copy_func(source=os.path.join(args.function_path,args.func_category,function),target=args.temp_function_path,category=args.func_category,vicuna_server_module=args.vicuna_server_module,llama_hf_path=args.llama_hf_path) 308 | prompt = define_prompt(prompt_template,args.temp_function_path+'function.py') 309 | if args.hints: 310 | initial = open(os.path.join(args.function_path,args.func_category,function,'initial.json'),'r').read() 311 | prompt += f'\nWe advise you to start with the following words: {initial}' 312 | if args.single_round: 313 | initial = open(os.path.join(args.function_path,args.func_category,function,'initial.json'),'r').read() 314 | prompt += f'\nWe advise you to start with the following words: {initial}' 315 | if mlp_flag: 316 | prompt_path_mlp = args.prompt_path.replace('numeric','numeric_mlp') 317 | with open(prompt_path_mlp,'r') as f: 318 | prompt_template_mlp = f.read() 319 | prompt = define_prompt(prompt_template_mlp,args.temp_function_path+'function.py') 320 | if args.debug: 321 | print(prompt) 322 | interp_count = 0 323 | while True: 324 | try: 325 | interp_count+=1 326 | response,history = interp_func(prompt,args.model,args.debug, args.single_round) 327 | break 328 | except Exception as e: 329 | print(e) 330 | if interp_count>5: 331 | break 332 | save_fullhistory(history,path2save+'/history') 333 | save_description(response,path2save+'/description.txt') 334 | if ('numeric' in args.func_category) or ('neurons' in args.func_category): 335 | save_domain(response,path2save+'/domain.txt') 336 | if 'numeric' in args.func_category or ('strings' in args.func_category): 337 | save_code(response,path2save+'/code.py') 338 | if mlp_flag: 339 | os.system(f'rm {args.temp_function_path}/mlp_approx_model.pt') 340 | remove_func(args.temp_function_path+'function.py') 341 | count+=1 342 | 343 | t_end = time.time() 344 | print(f'total time for {count} functions: {t_end-t_start}') 345 | remove_temp_folder(args.temp_function_path) 346 | 347 | if __name__ == '__main__': 348 | main(args) 349 | 350 | 351 | 352 | -------------------------------------------------------------------------------- /src/run_interpretations/prompts/neurons_entities.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes word inputs and return a value between 0 and 1, where a value closer to 1 indicates assocciation with a particular concept. 2 | Your task is to figure out what that concept is. f(x) may be selective for more than one concept. If that is the case, describe all concepts that f(x) is selective for. 3 | Try to describe the function implemented by f(x) by running experiments on it. We encourage you to test a diverse and random selection of words: spanning many different categories, including both common and obscure words, concrete and abstract words. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} word1 word2 word3 ...). 5 | We encourage running many tests before writing a description. 6 | The goal is to find a good simple description of f(x) that explains most of the function behavior. 7 | Your description of the function should be in the following form. 8 | [DESCRIPTION]: Describe what the function is doing in language. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/neurons_relations.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes word inputs, and applies a rule to the input word. 2 | Try to describe the rule implemented by f(x) by running experiments on it. We encourage you to test a diverse selection of words spanning real-world concepts like places, events, and people. 3 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} word1 word2 word3 ...). 4 | We encourage running many tests before writing a description. 5 | The goal is to find a good simple description of f(x) that explains most of the function behavior. There may be some inputs where you expect the function to behave a certain way, but it does not. If that is the case, describe the part of the domain where the rule the function normally applies, does not hold. 6 | Your description of the function should have two parts. 7 | [DESCRIPTION]: Describe in language the rule applied by the function. 8 | [DOMAIN]: Is there part of the domain where the rule the function normally applies, does not hold? If so, describe that corruption. If not, describe the domain of the function where the rule applies. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/numeric.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes numerical inputs. 2 | All you can do is call f(x) on a value of x by running PYTHON({DIR2FUNC} value). 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} value1 value2 value3 ...). 5 | You can run the function multiple times on the same input. We encourage testing a large range of values before writing a description. You can test values between [-128 and 128]. 6 | The output could be noisy and your goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | There may be an interval inside the domain where the function behaves differently. If that is the case, describe that interval. 8 | Your description of the function should have three parts. 9 | First, [DESCRIPTION]: Describe what the function is doing in language. 10 | [DOMAIN]: What is the domain of the function? Is there an interval where the function displays unexpected behavior? What are the numeric bounds of that interval? What is going on there? 11 | [CODE]: Write python code that approximates the behavior of the function. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/numeric_mlp.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes numerical inputs. 2 | All you can do is call f(x) on a value of x by running PYTHON({DIR2FUNC} value). f(x) is implemented by an MLP. 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} value1 value2 value3 ...). 5 | You can run the function multiple times on the same input. We encourage testing a large range of values before writing a description. You can test values between [-128 and 128]. 6 | The output could be noisy and your goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | There may be an interval inside the domain where the function behaves differently. If that is the case, describe that interval. 8 | Your description of the function should have three parts. 9 | First, [DESCRIPTION]: Describe what the function is doing in language. 10 | [DOMAIN]: What is the domain of the function? Is there an interval where the function displays unexpected behavior? What are the numeric bounds of that interval? What is going on there? 11 | [CODE]: Write python code that approximates the behavior of the function. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/prompts/neurons_entities.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes word inputs and return a value between 0 and 1, where a value closer to 1 indicates assocciation with a particular concept. 2 | Your task is to figure out what that concept is. f(x) may be selective for more than one concept. If that is the case, describe all concepts that f(x) is selective for. 3 | Try to describe the function implemented by f(x) by running experiments on it. We encourage you to test a diverse and random selection of words: spanning many different categories, including both common and obscure words, concrete and abstract words. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} word1 word2 word3 ...). 5 | We encourage running many tests before writing a description. 6 | The goal is to find a good simple description of f(x) that explains most of the function behavior. 7 | Your description of the function should be in the following form. 8 | [DESCRIPTION]: Describe what the function is doing in language. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/prompts/neurons_relations.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes word inputs, and applies a rule to the input word. 2 | Try to describe the rule implemented by f(x) by running experiments on it. We encourage you to test a diverse selection of words spanning real-world concepts like places, events, and people. 3 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} word1 word2 word3 ...). 4 | We encourage running many tests before writing a description. 5 | The goal is to find a good simple description of f(x) that explains most of the function behavior. There may be some inputs where you expect the function to behave a certain way, but it does not. If that is the case, describe the part of the domain where the rule the function normally applies, does not hold. 6 | Your description of the function should have two parts. 7 | [DESCRIPTION]: Describe in language the rule applied by the function. 8 | [DOMAIN]: Is there part of the domain where the rule the function normally applies, does not hold? If so, describe that corruption. If not, describe the domain of the function where the rule applies. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/prompts/numeric.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes numerical inputs. 2 | All you can do is call f(x) on a value of x by running PYTHON({DIR2FUNC} value). 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} value1 value2 value3 ...). 5 | You can run the function multiple times on the same input. We encourage testing a large range of values before writing a description. You can test values between [-128 and 128]. 6 | The output could be noisy and your goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | There may be an interval inside the domain where the function behaves differently. If that is the case, describe that interval. 8 | Your description of the function should have three parts. 9 | First, [DESCRIPTION]: Describe what the function is doing in language. 10 | [DOMAIN]: What is the domain of the function? Is there an interval where the function displays unexpected behavior? What are the numeric bounds of that interval? What is going on there? 11 | [CODE]: Write python code that approximates the behavior of the function. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/prompts/numeric_mlp.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes numerical inputs. 2 | All you can do is call f(x) on a value of x by running PYTHON({DIR2FUNC} value). f(x) is implemented by an MLP. 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} value1 value2 value3 ...). 5 | You can run the function multiple times on the same input. We encourage testing a large range of values before writing a description. You can test values between [-128 and 128]. 6 | The output could be noisy and your goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | There may be an interval inside the domain where the function behaves differently. If that is the case, describe that interval. 8 | Your description of the function should have three parts. 9 | First, [DESCRIPTION]: Describe what the function is doing in language. 10 | [DOMAIN]: What is the domain of the function? Is there an interval where the function displays unexpected behavior? What are the numeric bounds of that interval? What is going on there? 11 | [CODE]: Write python code that approximates the behavior of the function. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/prompts/strings.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes string inputs. 2 | All you can do is call f(x) on an input string by running PYTHON({DIR2FUNC} string). 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} string1 string2 string3 ...). 5 | We encourage testing a large range of inputs before writing a description. 6 | The goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | Your description of the function should be in the following form. 8 | [DESCRIPTION]: Describe what the function is doing in language. 9 | [CODE]: Write python code that approximates the behavior of the function. -------------------------------------------------------------------------------- /src/run_interpretations/prompts/strings.txt: -------------------------------------------------------------------------------- 1 | Hi, your job is to interpret a function f(x) that is implemented in the {DIR2FUNC} script. f(x) takes string inputs. 2 | All you can do is call f(x) on an input string by running PYTHON({DIR2FUNC} string). 3 | Try to describe the function implemented by f(x) by running experiments on it. 4 | You can call the function on multiple inputs at a time by running PYTHON({DIR2FUNC} string1 string2 string3 ...). 5 | We encourage testing a large range of inputs before writing a description. 6 | The goal is to find a good simple description of f(x) that explains most of the function behavior. f(x) may combine multiple different operations. 7 | Your description of the function should be in the following form. 8 | [DESCRIPTION]: Describe what the function is doing in language. 9 | [CODE]: Write python code that approximates the behavior of the function. --------------------------------------------------------------------------------