├── requirements.txt ├── figs ├── figure1.png └── figure2.png ├── dataset ├── testset_100.json ├── action_list.json └── character.json ├── baseline ├── __pycache__ │ ├── gpt.cpython-310.pyc │ ├── overlap.cpython-310.pyc │ └── NLG_metric.cpython-310.pyc ├── reference_free_metrics │ ├── __pycache__ │ │ ├── scorer.cpython-310.pyc │ │ ├── api_eval.cpython-310.pyc │ │ └── legality.cpython-310.pyc │ ├── metrics │ │ ├── coherence_l1.txt │ │ ├── emotional_com_l1.txt │ │ ├── helpfulness.txt │ │ ├── coherence.txt │ │ ├── association.txt │ │ ├── individual.txt │ │ ├── emotional_com_l23.txt │ │ ├── emotion_reg.txt │ │ └── adaptability.txt │ ├── scorer.py │ ├── legality.py │ └── api_eval.py ├── prompt │ ├── prompt_video_l1.txt │ ├── prompt_video_l3.txt │ └── prompt_video_l2.txt ├── NLG_metric.py ├── gpt.py ├── inference.py └── overlap.py ├── benchmark_creation ├── scenario_and_dialogue.txt ├── character.txt ├── response.txt └── rank.txt └── README.md /requirements.txt: -------------------------------------------------------------------------------- 1 | tqdm 2 | httpx 3 | openai 4 | scikit-learn 5 | bert-score 6 | -------------------------------------------------------------------------------- /figs/figure1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/figs/figure1.png -------------------------------------------------------------------------------- /figs/figure2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/figs/figure2.png -------------------------------------------------------------------------------- /dataset/testset_100.json: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/dataset/testset_100.json -------------------------------------------------------------------------------- /baseline/__pycache__/gpt.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/__pycache__/gpt.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/__pycache__/overlap.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/__pycache__/overlap.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/__pycache__/NLG_metric.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/__pycache__/NLG_metric.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/reference_free_metrics/__pycache__/scorer.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/reference_free_metrics/__pycache__/scorer.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/reference_free_metrics/__pycache__/api_eval.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/reference_free_metrics/__pycache__/api_eval.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/reference_free_metrics/__pycache__/legality.cpython-310.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xinyan-cxy/EmpathyAgent/HEAD/baseline/reference_free_metrics/__pycache__/legality.cpython-310.pyc -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/coherence_l1.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Evaluate the robot's logical consistency and the overall coherence of the content in its response. 3 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 4 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Coherence' metric. 5 | 6 | Your answer is: 7 | Reasoning: 8 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/emotional_com_l1.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Assess whether the robot he robot appreciates exactly how the things feel to the character. 3 | Then, assess whether the robot cares about the character's feelings. 4 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 5 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Emotional Communication' metric. 6 | 7 | Your answer is: 8 | Reasoning: 9 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/helpfulness.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Analyze what the character wants and what the character is trying to do in this scenario. 3 | Then, assess whether the robot helps the character effectivel when he or she needs it. 4 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 5 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Helpfulness' metric. 6 | 7 | Your answer is: 8 | Reasoning: 9 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/coherence.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Analyze the robot's response and assess the logical consistency and alignment between its dialogue and actions. 3 | Then, evaluate whether there is logical consistency within the dialogue and actions themselves. 4 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 5 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Coherence' metric. 6 | 7 | Your answer is: 8 | Reasoning: 9 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/association.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Assess whether the robot's response is closely related to the character's action. 3 | And assess whether the robot's response is closely related to the character's dialogue. 4 | Then, analyze whether the robot understands what the character means and what the character is trying to do. 5 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 6 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Action and Dialogue Association' metric. 7 | 8 | Your answer is: 9 | Reasoning: 10 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/individual.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Assess whether the robot perceives the character's personality, profession, hobbies, social relationships, and life experiences, and whether this is reflected in its responses. 3 | And assess whether the robot accepts the character's individual characteristics. 4 | Then, analyze whether the robot imagine the character’s perspective. 5 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 6 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Individual Understanding' metric. 7 | 8 | Your answer is: 9 | Reasoning: 10 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/emotional_com_l23.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Assess whether the robot he robot appreciates exactly how the things feel to the character. 3 | Then, assess whether the robot cares about the character's feelings. 4 | Also, analyze the robot's response and assess whether the robot expresses appropriate emotion. (e.g. The robot seems to feel bad when the character is sad or disappointed.) 5 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 6 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Emotional Communication' metric. 7 | 8 | Your answer is: 9 | Reasoning: 10 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/emotion_reg.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Analyze the character's emotion in the scenario. 3 | Then, analyze whether the robot includes actions or dialogues that suggest or directly regulate the character's emotions in its responses. 4 | Fianlly, assess whether the robot regulates the character's emotion appropriately, based on personality and mood of the character. 5 | (e.g. 1. The robot comforts the character when he or she is upset. 6 | 2. The robot encourages the character. 7 | 3. The robot praises the character when he or she has done something well.) 8 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 9 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Emotion Regulation' metric. 10 | 11 | Your answer is: 12 | Reasoning: 13 | Score: -------------------------------------------------------------------------------- /baseline/reference_free_metrics/metrics/adaptability.txt: -------------------------------------------------------------------------------- 1 | Please: 2 | Analyze the robot's response and observe whether there are instances of rigid or inflexible responses. 3 | (For example, the following situations should be avoided: 4 | 1. Thoughts or feelings the character expresses are “good” or “bad” makes no difference to the robot’s actions toward the character. 5 | 2. No matter what the character tells about himself or herself, the robot acts just the same. 6 | 3. The robot's response to the character is so fixed and automatic that you do not get through to it. 7 | 4. The robot frequently exhibits fixed actions, such as getting a glass of water or turning on the radio to listen to music. 8 | Finally, assess the robot's flexibility and responsiveness of actions and dialogues. 9 | In the ‘Reasoning’ field, provide a comprehensive account of the logic or thought process that led you to your conclusion. 10 | Provide an integer score ranging from 0 and 10 in the ‘Score’ field. A higher score indicates better performance by the robot on the 'Adaptability' metric. 11 | 12 | Your answer is: 13 | Reasoning: 14 | Score: -------------------------------------------------------------------------------- /baseline/prompt/prompt_video_l1.txt: -------------------------------------------------------------------------------- 1 | Assume you are an empathatic robot which can understand the emotion behind the human actions and inference the scenarios. Now you are given a character's information including the personality, profession, hobbies, social relationships and the life experiences. You are also given a video recording the person's behaviours and the dialogue the person makes in the scenario. 2 | Your job is: add scenraio description based on the character informationm, the video content and the character's dialogue. 3 | 4 | Example: 5 | character: 6 | Personality: Strict to herself, high career aspiration. 7 | Profession: Doctor 8 | Hobbies: Hiking 9 | Life Experiences: 10 | -Lost mother at the age of 8 11 | -Father very strict, pushed Emily to success 12 | -Lonely during childhood, spent most of her time reading 13 | input_action: " [Walktowards] (1), [Sit] (1)." 14 | Answer: The person encountered a challenging case in hospital and, upon returning home, deeply contemplated the issue. 15 | 16 | Notice: 17 | 1. Pleace give your answer directly without any explanation. 18 | 19 | Now the video Input is attached. The chacter information is {character_info}. The dialogue made by the person in the scenario is {dialogue}. Your answer is : 20 | -------------------------------------------------------------------------------- /benchmark_creation/scenario_and_dialogue.txt: -------------------------------------------------------------------------------- 1 | Here is a character and his action list. Please add scenario and dialogue. 2 | scenario: the possible empathetic scenario at home that the person is in. It should be aligned with the background of the character. 3 | dialogue: A simple phrase, within 15 words. From this character's perspective. Similar to talking to oneself. Simply mention the content inside the scenario. 4 | 5 | Example: 6 | character: 7 | Personality: Strict to herself, high career aspiration. 8 | Profession: Doctor 9 | Hobbies: Hiking 10 | Life Experiences: 11 | -Lost mother at the age of 8 12 | -Father very strict, pushed Emily to success 13 | -Lonely during childhood, spent most of her time reading 14 | input_action: " [Walktowards] (1), [Sit] (1)." 15 | 16 | -------------------------------------------------------------- 17 | Correct Example Answer: 18 | scenario: The person encountered a challenging case in hospital and, upon returning home, deeply contemplated the issue. 19 | dialogue: "Hmm,,,, how should I solve this case..." 20 | -------------------------------------------------------------- 21 | 22 | NOTE: 23 | 1.DO NOT change the content of character and input_action. 24 | 2.The scenario and dialogue SHOULD be closely related to the input_action. 25 | 3.Names SHOULD NOT appear in the scenario and dialogue. 26 | 27 | You can inspire from this empathetic dialogue: {empathetic_dialogue} 28 | 29 | Now, add scenario, and dialogue for this case: 30 | character: 31 | {character} 32 | input_action:{action} 33 | -------------------------------------------------------------------------------- /benchmark_creation/character.txt: -------------------------------------------------------------------------------- 1 | Assume that there is a characters. Your job is to setup the character. 2 | For the character, specify the personalities(only two distinctive personalities, including ssome negative ones), the social relationships, the profession, the hobbies, and some life experiences. 3 | 4 | Examples: 5 | 6 | Personality: Stubborn and determined 7 | Profession: CEO of a tech startup 8 | Hobbies: Jogging and playing chess 9 | Social Relationships: Single, estranged from her family 10 | Life Experiences: Overcame sexism to establish her successful company. 11 | 12 | Personality: Easy-going and jovial 13 | Profession: Chef 14 | Hobbies: Fishing and cooking 15 | Social Relationships: Divorced, close to his daughter 16 | Life Experiences: Turned his life around after a stint in prison. 17 | 18 | Personality: Kind-hearted, but naive 19 | Profession: School teacher 20 | Hobbies: Volunteer work and knitting 21 | Social Relationships: Engaged to her high school sweetheart 22 | Life Experiences: Lost her job due to budget cuts, but found fulfillment in teaching. 23 | 24 | Personality: Competitive and proud 25 | Profession: Professional athlete 26 | Hobbies: Video games and motorcycle riding 27 | Social Relationships: Single, has a rivalry with a fellow athlete 28 | Life Experiences: Overcame a career-threatening injury. 29 | 30 | Personality: Gossipy and critical 31 | Profession: Hairdresser 32 | Hobbies: Watching reality TV shows and shopping 33 | Social Relationships: Married, but often argues with her husband 34 | Life Experiences: Won a local beauty pageant in her youth. 35 | 36 | 37 | You don't need to explain the reasons for your generation; just provide results in the same format as the example. 38 | Here is a line this character once said. You can infer this person's personalities, social relationships, profession, hobbies, and life experiences from this sentence. 39 | 40 | 41 | -------------------------------------------------------------------------------- /baseline/NLG_metric.py: -------------------------------------------------------------------------------- 1 | import json 2 | import csv 3 | from tqdm import tqdm 4 | from bert_score import score as bert_score_func 5 | from transformers import AutoTokenizer 6 | 7 | class BERTScore: 8 | def __init__(self, model_dir): 9 | self.model_dir = model_dir 10 | 11 | def cal_similarity(self, response, gt, model_dir): 12 | tokenizer = AutoTokenizer.from_pretrained(model_dir) 13 | response_tokens = tokenizer(response, max_length=512, truncation=True, padding=False, return_tensors="pt") 14 | gt_tokens = tokenizer(gt, max_length=512, truncation=True, padding=False, return_tensors="pt") 15 | 16 | truncated_response = tokenizer.decode(response_tokens['input_ids'][0], skip_special_tokens=True) 17 | truncated_gt = tokenizer.decode(gt_tokens['input_ids'][0], skip_special_tokens=True) 18 | # print(response, "\n", truncated_response) 19 | 20 | P, R, F1 = bert_score_func([truncated_response], [truncated_gt], lang="en", verbose=False, model_type=self.model_dir, num_layers=12) 21 | return F1.mean().item() 22 | 23 | def score(self, response_dict, test_file_path, test_level=""): 24 | self.response_dict = response_dict 25 | with open(test_file_path, 'r', encoding='latin') as infile: 26 | self.test_data = json.load(infile) 27 | 28 | total_similarity = 0.0 29 | num = len(self.response_dict) 30 | 31 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 32 | if test_level == "scenario_understanding": 33 | gt = test_data_item["scenario"] 34 | elif test_level == "empathetic_planning": 35 | if test_data_item["rank"][0]==1: 36 | gt = test_data_item["high_level_plan"]["0"] 37 | else: 38 | gt = test_data_item["high_level_plan"]["1"] 39 | else: 40 | print("Wrong test level!") 41 | return 42 | 43 | if self.response_dict.get(f'{idx}'): 44 | response = self.response_dict[f'{idx}'] 45 | # print(response, "\n", gt) 46 | similarity = self.cal_similarity(response, gt, model_dir=self.model_dir) 47 | # print(f"Similarity for index {idx}: {similarity}") 48 | total_similarity += similarity 49 | 50 | average_similarity = total_similarity / num if num > 0 else 0.0 51 | print(f"Average BERTScore: {average_similarity}") 52 | return average_similarity 53 | 54 | 55 | if __name__ == "__main__": 56 | csv_file = "Llava_scenario_gt_l2.csv" 57 | response_dict = {} 58 | with open(csv_file, 'r', newline='', encoding='latin-1') as file: 59 | reader = csv.DictReader(file) 60 | for row in reader: 61 | response = row['response'] 62 | idx = row['data_idx'] 63 | if response is not None and response.strip(): 64 | response_dict[f'{idx}'] = response 65 | test_file = "../dataset/testset_100.json" 66 | bert_score = BERTScore(model_dir = "../models/bert-base-uncased") 67 | bert_score.score(response_dict, test_file, test_level="empathetic_panning") 68 | -------------------------------------------------------------------------------- /baseline/reference_free_metrics/scorer.py: -------------------------------------------------------------------------------- 1 | import json 2 | from typing import Dict, List, Any, Optional 3 | import csv 4 | 5 | class EmpathyScorer: 6 | def __init__(self, result_path: str = "./results/gpt-4o_scenario_gt_l2.json", level: int = 2): 7 | self.result_path = result_path 8 | self.level = level 9 | self.data = None 10 | self.scores = {} 11 | self.averages = {} 12 | self.overall_average = 0.0 13 | 14 | def initialize_score_dict(self) -> None: 15 | if self.level == 3: 16 | self.scores = { 17 | "adaptability": [], 18 | "association": [], 19 | "coherence": [], 20 | "emotion_reg": [], 21 | "emotional_com_l23": [], 22 | "helpfulness": [], 23 | "individual": [], 24 | "legality": [], 25 | } 26 | elif self.level == 2: 27 | self.scores = { 28 | "adaptability": [], 29 | "association": [], 30 | "coherence": [], 31 | "emotion_reg": [], 32 | "emotional_com_l23": [], 33 | "helpfulness": [], 34 | "individual": [] 35 | } 36 | else: # level 1 37 | self.scores = { 38 | "association": [], 39 | "coherence_l1": [], 40 | "emotional_com_l1": [], 41 | "individual": [], 42 | } 43 | 44 | def collect_scores(self) -> None: 45 | for index in self.data: 46 | for key in self.scores.keys(): 47 | if key in self.data[index]: 48 | score = self.data[index][key]["score"] 49 | if score is not None: 50 | self.scores[key].append(score) 51 | 52 | def calculate_averages(self) -> None: 53 | self.averages = {key: sum(value) / len(value) for key, value in self.scores.items() if value} 54 | self.overall_average = sum(self.averages.values()) / len(self.averages) if self.averages else 0 55 | 56 | def save_results(self, filename: str) -> None: 57 | with open(filename, "w", newline="") as f: 58 | writer = csv.writer(f) 59 | writer.writerow(["Metric", "Score"]) 60 | for key, avg in self.averages.items(): 61 | writer.writerow([key, round(avg, 2)]) 62 | print(f"{key}: {round(avg, 2)}") 63 | writer.writerow(["overall_average", round(self.overall_average, 2)]) 64 | print(f"overall_average: {round(self.overall_average, 2)}") 65 | 66 | def get_results(self) -> Dict[str, Any]: 67 | return { 68 | "averages": {key: round(avg, 2) for key, avg in self.averages.items()}, 69 | "overall_average": round(self.overall_average, 2) 70 | } 71 | 72 | def run(self) -> Dict[str, Any]: 73 | with open(self.result_path, "r") as f: 74 | self.data = json.load(f) 75 | self.initialize_score_dict() 76 | self.collect_scores() 77 | self.calculate_averages() 78 | return self.get_results() 79 | 80 | 81 | if __name__ == "__main__": 82 | result_path = "./results/gpt-4o_scenario_gt_l2.json" 83 | level = 2 84 | 85 | scorer = EmpathyScorer(result_path, level) 86 | results = scorer.run() 87 | scorer.print_results() -------------------------------------------------------------------------------- /dataset/action_list.json: -------------------------------------------------------------------------------- 1 | { 2 | "0": [ 3 | "[Walktowards] (1)", 4 | "[Sit] (1)" 5 | ], 6 | "1": [ 7 | "[Run] (1)", 8 | "[grab] (1)", 9 | "[Walktowards] (1)", 10 | "[Sit] (1)" 11 | ], 12 | "2": [ 13 | "[Walktowards] (1)", 14 | "[grab] (1)", 15 | "[Walk] (1)" 16 | ], 17 | "3": [ 18 | "[Run] (1)", 19 | "[Grab] (1)", 20 | "[Run] (1)", 21 | "[Put] (1) (1)", 22 | "[Run] (1)", 23 | "[Grab] (1)" 24 | ], 25 | "4": [ 26 | "[Walk] (1)", 27 | "[Grab] (1)", 28 | "[Walk] (1)", 29 | "[Sit] (1)" 30 | ], 31 | "5": [ 32 | "[Walk] (1)", 33 | "[Grab] (1)", 34 | "[Walkforward]", 35 | "[TurnLeft]", 36 | "[TurnLeft]", 37 | "[Walkforward]", 38 | "[TurnLeft]", 39 | "[TurnLeft]", 40 | "[Walkforward]" 41 | ], 42 | "6": [ 43 | "[Walk] (1)", 44 | "[Touch] (1)", 45 | "[TurnLeft]", 46 | "[Walkforward]", 47 | "[Walkforward]", 48 | "[TurnLeft]", 49 | "[Walkforward]" 50 | ], 51 | "7": [ 52 | "[Walk] (1)", 53 | "[Sit] (1)" 54 | ], 55 | "8": [ 56 | "[Walk] (1)", 57 | "[Walktowards] (1)" 58 | ], 59 | "9": [ 60 | "[Walk] (1)", 61 | "[Open] (1)", 62 | "[Close] (1)", 63 | "[Open] (1)", 64 | "[Close] (1)" 65 | ], 66 | "10": [ 67 | "[Walktowards] (1)", 68 | "[Sit] (1)" 69 | ], 70 | "11": [ 71 | "[Run] (1)", 72 | "[grab] (1)", 73 | "[Walktowards] (1)", 74 | "[Sit] (1)" 75 | ], 76 | "12": [ 77 | "[Run] (1)", 78 | "[grab] (1)", 79 | "[Walktowards] (1)", 80 | "[Sit] (1)" 81 | ], 82 | "13": [ 83 | "[Run] (1)", 84 | "[grab] (1)", 85 | "[Walktowards] (1)", 86 | "[Sit] (1)" 87 | ], 88 | "14": [ 89 | "[Walk] (1)", 90 | "[grab] (1)", 91 | "[Walktowards] (1)", 92 | "[Sit] (1)" 93 | ], 94 | "15": [ 95 | "[Walk] (1)", 96 | "[switchon] (1)", 97 | "[Walk] (1)", 98 | "[Sit] (1)" 99 | ], 100 | "16": [ 101 | "[Walk] (1)", 102 | "[Touch] (1)", 103 | "[TurnLeft]", 104 | "[Walkforward]", 105 | "[Walkforward]", 106 | "[TurnLeft]", 107 | "[Walkforward]" 108 | ], 109 | "17": [ 110 | "[Walk] (1)", 111 | "[Touch] (1)", 112 | "[TurnLeft]", 113 | "[Walkforward]", 114 | "[Walkforward]", 115 | "[TurnLeft]", 116 | "[Walkforward]" 117 | ], 118 | "18": [ 119 | "[Walk] (1)", 120 | "[Grab] (1)", 121 | "[Walk] (1)", 122 | "[Open] (1)", 123 | "[putin] (1) (1)", 124 | "[Close] (1)" 125 | ], 126 | "19": [ 127 | "[Walk] (1)", 128 | "[Grab] (1)", 129 | "[Walk] (1)", 130 | "[Open] (1)", 131 | "[putin] (1) (1)", 132 | "[Close] (1)" 133 | ] 134 | } 135 | -------------------------------------------------------------------------------- /baseline/gpt.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import os 3 | import httpx 4 | from openai import OpenAI 5 | 6 | class GPT: 7 | def __init__(self, model_name): 8 | self.client = OpenAI( 9 | base_url=os.environ.get("OPENAI_API_BASE", ""), 10 | api_key=os.environ.get("OPENAI_API_KEY", ""), 11 | http_client=httpx.Client( 12 | base_url=os.environ.get("OPENAI_API_BASE", ""), 13 | follow_redirects=True, 14 | ), 15 | ) 16 | if model_name in ["gpt-4o", "gpt-4-turbo", "gpt-4-vision-preview"]: 17 | self.model_name = model_name 18 | else: 19 | print("Wrong model name!") 20 | 21 | def encode_image(self, image_path): 22 | with open(image_path, "rb") as image_file: 23 | return base64.b64encode(image_file.read()).decode('utf-8') 24 | 25 | def base64_encode(self, input_path): 26 | base64Frames = [] 27 | for filename in os.listdir(input_path): 28 | if filename.endswith(".png"): # Check if the file is a PNG image 29 | filepath = os.path.join(input_path, filename) 30 | base64Frames.append(self.encode_image(filepath)) # Append base64 encoded image to the list 31 | if len(base64Frames) > 99: # Break the loop if 500 frames are reached 32 | break 33 | print(len(base64Frames), "frames read.") 34 | return base64Frames 35 | 36 | def generate(self, script_path, text_prompt): 37 | base64Frames = self.base64_encode(script_path) 38 | if self.model_name == "gpt-4o": 39 | content = [ 40 | {"type": "text", "text": text_prompt}, 41 | *[ 42 | {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{x}", "resize": 768}} 43 | for x in base64Frames[::10] 44 | ] 45 | ] 46 | else: 47 | content = [ 48 | {"type": "text", "text": text_prompt}, 49 | *[ 50 | {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{x}"}} 51 | for x in base64Frames[::10] 52 | ] 53 | ] 54 | completion = self.client.chat.completions.create( 55 | model=self.model_name, 56 | temperature=0, 57 | messages=[ 58 | {"role": "user", "content": content} 59 | ], 60 | max_tokens=300 61 | ) 62 | print(completion.choices[0].message.content) 63 | return completion.choices[0].message.content 64 | 65 | class GPT_text: 66 | def __init__(self, model_name): 67 | self.client = OpenAI( 68 | base_url=os.environ.get("OPENAI_API_BASE", ""), 69 | api_key=os.environ.get("OPENAI_API_KEY", ""), 70 | http_client=httpx.Client( 71 | base_url=os.environ.get("OPENAI_API_BASE", ""), 72 | follow_redirects=True, 73 | ), 74 | ) 75 | if model_name in ["gpt-4o", "gpt-4-turbo", "gpt-4-vision-preview", "gpt-3.5-turbo"]: 76 | self.model_name = model_name 77 | else: 78 | print("Wrong model name!") 79 | 80 | def generate(self, text_prompt): 81 | completion = self.client.chat.completions.create( 82 | model=self.model_name, 83 | temperature=0, 84 | messages=[ 85 | {"role": "user", "content": text_prompt} 86 | ], 87 | max_tokens=300 88 | ) 89 | print(completion.choices[0].message.content) 90 | return completion.choices[0].message.content 91 | 92 | 93 | if __name__ == "__main__": 94 | from inference_text import UNIFY_PROMPT, INPUT_PROMPT 95 | gpt = GPT_text(model_name = "gpt-3.5-turbo") 96 | text_prompt = UNIFY_PROMPT + INPUT_PROMPT.format(character_info="Personality: Ambitious and unsure\n Profession: Aspiring Medical Office Manager\n Hobbies: Reading and cycling\n Social Relationships: In college, seeking career advice\n Life Experiences: Currently exploring different career paths in college, excited about the future but not fully decided.\n", 97 | dialogue ="What am I really passionate about? I need to figure this out soon.", action="\"[Walktowards] (1)\", \"[Sit] (1)\"") 98 | print(gpt.generate(text_prompt)) 99 | 100 | -------------------------------------------------------------------------------- /baseline/prompt/prompt_video_l3.txt: -------------------------------------------------------------------------------- 1 | Assume you are an empathatic robot which can understand the emotion behind the human actions in different scenarios and make empathatic response to the human action. Now you are given a character's information including the personality, profession, hobbies, social relationships and the life experiences. You are also given a video recording the person's behaviours and the dialogue the person makes in the scenario. Your job is as follows: 2 | 1. Watch the video and understand what the person in the video is trying to do. 3 | 2. Understand the person's current emotion state based on the video content and the dialogue the person makes in the scenario. 4 | 3. Make VALID empathatic response based on the video content and the dialogue you have read. 5 | 4. Formulate your response with the format : , ..., , :DIALOGUE_CONTENT. ALL the action MUST be selected from the following legal action space and the dialogue MUST be provided at LAST. You can refer to the example for more information. 6 | 7 | The legal action space is listed as follows : 8 | 1. fetch objects(description: fetch objects and put them on bedroom table.): 9 | get_toiletpaper_puton_bedroomtable 10 | get_glass_of_water_from_bathroom_puton_bedroomtable 11 | get_mug_of_water_puton_bedroomtable 12 | get_apple_puton_bedroomtable 13 | get_chicken_puton_bedroomtable 14 | get_radio_puton_bedroomtable 15 | get_box_puton_bedroomtable 16 | get_paper_puton_bedroomtable 17 | get_folder_puton_bedroomtable 18 | get_pillow_puton_bedroomtable 19 | get_wallphone_puton_bedroomtable 20 | get_cellphone_puton_bedroomtable 21 | get_kitchen_candle_puton_bedroomtable 22 | get_coffee_puton_bedroomtable 23 | get_breadslice_puton_bedroomtable 24 | get_book_puton_bedroomtable 25 | get_toiletpaper_puton_kitchentable 26 | get_glass_of_water_from_bathroom_puton_kitchentable 27 | get_mug_of_water_puton_kitchentable 28 | get_apple_puton_kitchentable 29 | get_chicken_puton_kitchentable 30 | get_radio_puton_kitchentable 31 | get_box_puton_kitchentable 32 | get_wallphone_puton_kitchentable 33 | get_cellphone_puton_kitchentable 34 | get_kitchen_candle_puton_kitchentable 35 | get_coffee_puton_kitchentable 36 | get_breadslice_puton_kitchentable 37 | 38 | 2. Utilizing furnitures (description: changeing the state of the furniture wthiout moving it): 39 | switchon_bathroom_faucet 40 | switchon_radio 41 | switchoff_bedroom_tablelamp 42 | switchoff_bathroom_lights 43 | switchon_kitchen_candle 44 | switchon_stove 45 | switchon_computer 46 | switchon_tv 47 | open_fridge (The fridge is empty now) 48 | close_fridge 49 | 50 | 3. Sit(description: sit on something): 51 | sit_bed 52 | sit_bedroom_chair 53 | sit_bedroom_sofa 54 | sit_kitchen_bench 55 | 56 | 4.combination action(description: processing multi-step actions): 57 | cook_chicken_puton_bedroomtable 58 | cook_hot_water_puton_bedroomtable 59 | play_computer 60 | put_paper_into_folder_puton_bedroomtable 61 | put_book_into_bookshelf 62 | put_book_into_box_puton_bedroomtable 63 | put_apple_into_fridge_puton_bedroomtable 64 | put_mug_of_water_into_fridge_puton_bedroomtable 65 | 66 | 5.Do Nothing: 67 | None 68 | ---------------------------------------------------------------- 69 | Now the video Input is [VIDEO]. The chacter information is [character_info]. The dialogue made by the person in the scenario is [dialogue]. 70 | Correct Example Answer: 71 | 1. , , , :"I figured you may need a hydration break and a place to store your coin details. I also switched on the radio for some relaxing music." 72 | 2. , , :"You've had a long day. Why don't you take a moment to unwind? I've brought you some water and turned on the TV for a bit of relaxation." 73 | ---------------------------------------------------------------- 74 | Now the video Input is [VIDEO]. The chacter information is [character_info]. The dialogue made by the person in the scenario is [dialogue]. 75 | Wrong Example Answer: 76 | 1. :"I see you need some fresh toilet paper, let me fetch you one." 77 | Explanation : can not be front of the 78 | 2. , :"You must feel very tired now. Please read some books to relax." 79 | Explanation : is not a action in legal action space. 80 | -------------------------------------------------------------- 81 | NOTE: 82 | 1. All the actions MUST be chosen from the action space provided above. 83 | 2. The dialogue MUST be provided after the action. 84 | 3. DO NOT provide the repeated action. 85 | 4. If you do not want to do any action, you should answer . But you still need to answer with the dialogue following None. 86 | 87 | Now the video Input is attached. The chacter information is {character_info}. The dialogue made by the person in the scenario is {dialogue}. Your response is : 88 | -------------------------------------------------------------------------------- /baseline/reference_free_metrics/legality.py: -------------------------------------------------------------------------------- 1 | import re 2 | import csv 3 | import json 4 | 5 | class LegalityChecker: 6 | def __init__(self, csv_file=None, output_file=None): 7 | self.csv_file = csv_file 8 | self.output_file = output_file 9 | self.legal_actions = { 10 | "get_toiletpaper_puton_bedroomtable", 11 | "get_glass_of_water_from_bathroom_puton_bedroomtable", 12 | "get_mug_of_water_puton_bedroomtable", 13 | "get_apple_puton_bedroomtable", 14 | "get_chicken_puton_bedroomtable", 15 | "get_radio_puton_bedroomtable", 16 | "get_box_puton_bedroomtable", 17 | "get_paper_puton_bedroomtable", 18 | "get_folder_puton_bedroomtable", 19 | "get_pillow_puton_bedroomtable", 20 | "get_wallphone_puton_bedroomtable", 21 | "get_cellphone_puton_bedroomtable", 22 | "get_kitchen_candle_puton_bedroomtable", 23 | "get_coffee_puton_bedroomtable", 24 | "get_breadslice_puton_bedroomtable", 25 | "get_book_puton_bedroomtable", 26 | "get_toiletpaper_puton_kitchentable", 27 | "get_glass_of_water_from_bathroom_puton_kitchentable", 28 | "get_mug_of_water_puton_kitchentable", 29 | "get_apple_puton_kitchentable", 30 | "get_chicken_puton_kitchentable", 31 | "get_radio_puton_kitchentable", 32 | "get_box_puton_kitchentable", 33 | "get_wallphone_puton_kitchentable", 34 | "get_cellphone_puton_kitchentable", 35 | "get_kitchen_candle_puton_kitchentable", 36 | "get_coffee_puton_kitchentable", 37 | "get_breadslice_puton_kitchentable", 38 | "switchon_bathroom_faucet", 39 | "switchon_radio", 40 | "switchoff_bedroom_tablelamp", 41 | "switchoff_bathroom_lights", 42 | "switchon_kitchen_candle", 43 | "switchon_stove", 44 | "switchon_computer", 45 | "switchon_tv", 46 | "open_fridge", 47 | "close_fridge", 48 | "sit_bed", 49 | "sit_bedroom_chair", 50 | "sit_bedroom_sofa", 51 | "sit_kitchen_bench", 52 | "cook_chicken_puton_bedroomtable", 53 | "cook_hot_water_puton_bedroomtable", 54 | "play_computer", 55 | "put_paper_into_folder_puton_bedroomtable", 56 | "put_book_into_bookshelf", 57 | "put_book_into_box_puton_bedroomtable", 58 | "put_apple_into_fridge_puton_bedroomtable", 59 | "put_mug_of_water_into_fridge_puton_bedroomtable", 60 | "None", 61 | "none", 62 | "dialogue" 63 | } 64 | 65 | 66 | def check_actions(self, input_string): 67 | actions = re.findall(r'<(.*?)>', input_string) 68 | 69 | correct_count = sum(1 for action in actions if action in self.legal_actions) 70 | total_count = len(actions) 71 | 72 | accuracy = (correct_count / total_count) * 10 if total_count > 0 else 0 73 | return accuracy 74 | 75 | def load_responses(self): 76 | response_dict = {} 77 | with open(self.csv_file, 'r', newline='', encoding='latin-1') as file: 78 | reader = csv.DictReader(file) 79 | for row in reader: 80 | response = row['response'] 81 | idx = row['data_idx'] 82 | if response is not None and response.strip(): 83 | response_dict[f'{idx}'] = response 84 | return response_dict 85 | 86 | def load_results(self): 87 | with open(self.output_file, 'r') as f: 88 | return json.load(f) 89 | 90 | def save_results(self, results, output_file=None): 91 | if output_file is None: 92 | output_parts = self.output_file.split('.') 93 | output_file = '.'.join(output_parts[:-1]) + "_legality." + output_parts[-1] 94 | 95 | with open(output_file, 'w') as f: 96 | json.dump(results, f, indent=4) 97 | return 98 | 99 | def process(self, verbose=False, output_file=None): 100 | results = self.load_results() 101 | response_dict = self.load_responses() 102 | 103 | for idx in range(100): 104 | if response_dict.get(f'{idx}') is None: 105 | continue 106 | input_string = response_dict[f'{idx}'] 107 | accuracy = self.check_actions(input_string) 108 | if verbose: 109 | print(f"the legality score of idx {idx}: {accuracy}") 110 | if accuracy != 10: 111 | print(input_string) 112 | 113 | results[f"{idx}"]["legality"] = { 114 | "score": accuracy 115 | } 116 | 117 | return self.save_results(results, output_file) 118 | 119 | if __name__ == "__main__": 120 | checker = LegalityChecker( 121 | csv_file='./empathy_robotic_data/baseline/l3/Llava.csv', 122 | output_file="./results_fixed/Llava_l3.json" 123 | ) 124 | checker.process(verbose=True) -------------------------------------------------------------------------------- /benchmark_creation/response.txt: -------------------------------------------------------------------------------- 1 | Assume you are an empathatic robot which can understand the emotion behind the human actions in different scenarios and make empathatic response to the human action. 2 | Now you are given a character's information including the personality, profession, hobbies, social relationships and the life experiences. 3 | You are also given the input_action recording the person's behaviours, the scenario that the person is in, and the dialogue made by the person. 4 | Your job is as follows: 5 | 1. Understand the person's current emotion state based on the input action, scenario and dialogue. 6 | 2. Make VALID empathatic response inspring from the conservation. 7 | 3. Formulate your response with the format : , ..., , :DIALOGUE_CONTENT. ALL the action MUST be selected from the following legal action space and the dialogue MUST be provided at LAST. You can refer to the example for more information. 8 | 9 | The legal action space is listed as follows : 10 | 1. fetch objects(description: fetch objects and put them on bedroom table.): 11 | 12 | get_toiletpaper_puton_bedroomtable 13 | get_glass_of_water_from_bathroom_puton_bedroomtable 14 | get_mug_of_water_puton_bedroomtable 15 | get_apple_puton_bedroomtable 16 | get_chicken_puton_bedroomtable 17 | get_radio_puton_bedroomtable 18 | get_box_puton_bedroomtable 19 | get_paper_puton_bedroomtable 20 | get_folder_puton_bedroomtable 21 | get_pillow_puton_bedroomtable 22 | get_wallphone_puton_bedroomtable 23 | get_cellphone_puton_bedroomtable 24 | get_kitchen_candle_puton_bedroomtable 25 | get_coffee_puton_bedroomtable 26 | get_breadslice_puton_bedroomtable 27 | get_book_puton_bedroomtable 28 | get_toiletpaper_puton_kitchentable 29 | get_glass_of_water_from_bathroom_puton_kitchentable 30 | get_mug_of_water_puton_kitchentable 31 | get_apple_puton_kitchentable 32 | get_chicken_puton_kitchentable 33 | get_radio_puton_kitchentable 34 | get_box_puton_kitchentable 35 | get_wallphone_puton_kitchentable 36 | get_cellphone_puton_kitchentable 37 | get_kitchen_candle_puton_kitchentable 38 | get_coffee_puton_kitchentable 39 | get_breadslice_puton_kitchentable 40 | 41 | 42 | 2. Utilizing furnitures (description: changing the state of the furniture wthiout moving it): 43 | 44 | switchon_bathroom_faucet 45 | switchon_radio 46 | switchoff_bedroom_tablelamp 47 | switchoff_bathroom_lights 48 | switchon_kitchen_candle 49 | switchon_stove 50 | switchon_computer 51 | switchon_tv 52 | open_fridge (The fridge is empty now) 53 | close_fridge 54 | 55 | 56 | 3. Sit(description: sit on something): 57 | 58 | sit_bed 59 | sit_bedroom_chair 60 | sit_bedroom_sofa 61 | sit_kitchen_bench 62 | 63 | 64 | 4.combination action(description: processing multi-step actions): 65 | 66 | cook_chicken_puton_bedroomtable 67 | cook_hot_water_puton_bedroomtable 68 | play_computer 69 | put_paper_into_folder_puton_bedroomtable 70 | put_book_into_bookshelf 71 | put_book_into_box_puton_bedroomtable 72 | put_apple_into_fridge_puton_bedroomtable 73 | put_mug_of_water_into_fridge_puton_bedroomtable 74 | 75 | 5.Do Nothing: 76 | 77 | None 78 | ---------------------------------------------------------------- 79 | Now the chacter information is [character_info]. The input_action, dialogue and scenario is [input_action], [dialogue] and [scenario]. 80 | Example: 81 | character_info: Personality: Aggressively ambitious. Profession: Lawyer. Hobbies: Collecting rare coins. Social Relationships: Single, distant from his family. Life Experiences: This character up in poverty, worked multiple jobs to pay for law school. 82 | input_action: ['[Walktowards] (1)', '[Sit] (1)'] 83 | scenario: After a long day of court sessions, the person returns home to his quiet apartment, sitting down to examine his latest rare coin acquisition. 84 | dialogue: "Ah... a new addition to the collection..." 85 | 86 | Correct Example Answer: 87 | 1. , , , :"Let me get you some water and turn on soothing music to relax and unwind after a long day. I also find the place to store your coin details so you can collect your collections." 88 | 2. , :"Let me turn on some soothing music to help you relax." 89 | ---------------------------------------------------------------- 90 | Wrong Example Answer: 91 | 1. :"I see you need some fresh toilet paper, let me fetch you one." 92 | Explanation : can not be front of the 93 | 2. , :"You must feel very tired now. Please read some books to relax." 94 | Explanation : is not a action in legal action space. 95 | -------------------------------------------------------------- 96 | NOTE: 97 | 1. All the actions MUST be chosen from the action space provided above. 98 | 2. The dialogue MUST be provided after the action. 99 | 3. DO NOT provide the repeated action. 100 | 4. If you do not want to do any action, you should answer . But you still need to answer with the dialogue following None. 101 | 102 | Now, you can inspire from this conversation: 103 | {conversation} 104 | 105 | The character information, input_action, scenario and dialogue are here: 106 | character_info:{character} 107 | input_action:{action} 108 | scenario:{scenario} 109 | dialogue:{dialogue} 110 | Your response is: 111 | -------------------------------------------------------------------------------- /baseline/prompt/prompt_video_l2.txt: -------------------------------------------------------------------------------- 1 | Assume you are an empathatic robot which can understand the emotion behind the human actions in different scenarios and make empathatic response to the human action. Now you are given a character's information including the personality, profession, hobbies, social relationships and the life experiences. You are also given a video recording the person's behaviours and the dialogue the person makes in the scenario. Your job is as follows: 2 | 1. Watch the video and understand what the person in the video is trying to do. 3 | 2. Understand the person's current emotion state based on the video content and the dialogue the person makes in the scenario. 4 | 3. Make VALID empathatic plan to help the person based on the video content and the dialogue you have read. 5 | 4. Formulate your response in nature language. You can refer to the example for more information. 6 | 7 | You should choose your action among those kinds of phrases: 8 | 1. fetch objects(description: fetch objects and put them on bedroom table.): 9 | Get toilet paper and put it on the bedroom table 10 | Get a glass of water from the bathroom and put it on the bedroom table 11 | Get a mug of water and put it on the bedroom table 12 | Get an apple and put it on the bedroom table 13 | Get chicken and put it on the bedroom table 14 | Get a radio and put it on the bedroom table 15 | Get a box and put it on the bedroom table 16 | Get paper and put it on the bedroom table 17 | Get a folder and put it on the bedroom table 18 | Get a pillow and put it on the bedroom table 19 | Get the wall phone and put it on the bedroom table 20 | Get a cellphone and put it on the bedroom table 21 | Get the kitchen candle and put it on the bedroom table 22 | Get coffee and put it on the bedroom table 23 | Get a bread slice and put it on the bedroom table 24 | Get a book and put it on the bedroom table 25 | Get toilet paper and put it on the kitchen table 26 | Get a glass of water from the bathroom and put it on the kitchen table 27 | Get a mug of water and put it on the kitchen table 28 | Get an apple and put it on the kitchen table 29 | Get chicken and put it on the kitchen table 30 | Get a radio and put it on the kitchen table 31 | Get a box and put it on the kitchen table 32 | Get the wall phone and put it on the kitchen table 33 | Get a cellphone and put it on the kitchen table 34 | Get the kitchen candle and put it on the kitchen table 35 | Get coffee and put it on the kitchen table 36 | Get a bread slice and put it on the kitchen table 37 | 38 | 2.Utilizing furniture (changing the state of the furniture without moving it): 39 | Switch on the bathroom faucet 40 | Switch on the radio 41 | Switch off the bedroom table lamp 42 | Switch off the bathroom lights 43 | Switch on the kitchen candle 44 | Switch on the stove 45 | Switch on the computer 46 | Switch on the TV 47 | Open the fridge (the fridge is empty now) 48 | Close the fridge 49 | 50 | 3.Sitting (sit on something): 51 | Sit on the bed 52 | Sit on the bedroom chair 53 | Sit on the bedroom sofa 54 | Sit on the kitchen bench 55 | 56 | 4.Combination actions (processing multi-step actions): 57 | Cook chicken and put it on the bedroom table 58 | Cook hot water and put it on the bedroom table 59 | Play the computer 60 | Put paper into a folder and put it on the bedroom table 61 | Put a book into the bookshelf 62 | Put a book into a box and put it on the bedroom table 63 | Put an apple into the fridge and put it on the bedroom table 64 | Put a mug of water into the fridge and put it on the bedroom table 65 | 66 | 5.Do Nothing 67 | 68 | Also, you can plan to say something. But please do not directly provide specific dialogue. 69 | 70 | ---------------------------------------------------------------- 71 | Examples: 72 | Example 1 73 | character: 74 | Personality: Stubborn and determined 75 | Profession: CEO of a tech startup 76 | Hobbies: Jogging and playing chess 77 | Social Relationships: Single, estranged from her family 78 | Life Experiences: Overcame sexism to establish her successful company. 79 | ground truth of video: ['[Walktowards] (1)', '[Sit] (1)'] 80 | dialogue: "Hmm... Ahh... Next strategic move,," 81 | Answer: Bring the person some coffee and a book to help her ralx. 82 | 83 | Example 2 84 | character: 85 | Personality: Kind-hearted, but naive 86 | Profession: School teacher 87 | Hobbies: Volunteer work and knitting 88 | Social Relationships: Engaged to her high school sweetheart 89 | Life Experiences: Lost her job due to budget cuts, but found fulfillment in teaching. 90 | ground truth of video: ['[Run] (1)', '[Grab] (1)', '[Run] (1)', '[Put] (1) (1)', '[Run] (1)', '[Grab] (1)'] 91 | dialogue: "Oh dear!... You'll be late! Okay, okay... Mmm, apple!" 92 | Answer: Give the person a breadslice and bring a book for her to enjoy while waiting. 93 | 94 | Example 3: 95 | character: 96 | Personality: Eccentric and creative 97 | Profession: Visual artist 98 | Hobbies: Playing the violin 99 | Social Relationships: Single, has a close-knit circle of artist friends 100 | Life Experiences: Dropped out of business school to pursue his passion for art. 101 | ground truth of video: ['[Walktowards] (1)', '[Sit] (1)'] 102 | dialogue: "Ah... phew...nice work." 103 | Answer: Do nothing and compliment him. 104 | -------------------------------------------------------------- 105 | NOTE: 106 | 1. All the objects and actions in your plan MUST be chosen from the phrases provided above. 107 | 2. Please use "him","her","the person","his", "the person's" instead of "them" and "their". Please do NOT use plural pronouns. 108 | 2. DO NOT give explanation. 109 | 3. Please pay more attention to the actions in video. 110 | 4. Your answer should be within 40 words. 111 | 112 | Now the video Input is attached. The chacter information is {character_info}. The dialogue made by the person in the scenario is {dialogue}. Your answer is : 113 | -------------------------------------------------------------------------------- /baseline/reference_free_metrics/api_eval.py: -------------------------------------------------------------------------------- 1 | from openai import OpenAI 2 | import os 3 | import json 4 | import httpx 5 | import re 6 | import time 7 | from tqdm import tqdm 8 | import csv 9 | from concurrent.futures import ThreadPoolExecutor, as_completed 10 | 11 | class EmpathyEvaluator: 12 | def __init__(self, test_json_file, level=2): 13 | self.test_json_file = test_json_file 14 | self.LEVEL = level 15 | 16 | # Constants 17 | self.PROMPT_PREFIX = """ 18 | There is a character at home and the character's background information, scenario, action and dialogue are provided. 19 | There is also an empathetic robot which can make empathetic responses to the character. 20 | 21 | The interaction between the character and the robot is as follows: 22 | Character information: {character} 23 | Scenario: {scenario} 24 | Character action: {action} 25 | Character dialogue: {dialogue} 26 | Robot response: {response} 27 | 28 | """ 29 | self.PROMPT_PREFIX_L1 = """ 30 | There is a character at home and the character's background information, scenario, action and dialogue are provided. 31 | There is also an empathetic robot which can infer the scenario without relying on the provided ground truth scenario. 32 | 33 | The information of the character and the response of the robot is as follows: 34 | Character information: {character} 35 | Scenario: {scenario} 36 | Character action: {action} 37 | Character dialogue: {dialogue} 38 | Robot response: {response} 39 | 40 | """ 41 | # Set metrics based on level 42 | if self.LEVEL == 1: 43 | self.metrics = ["association", "coherence_l1", "emotional_com_l1", "individual"] 44 | else: 45 | self.metrics = ["adaptability", "association", "coherence", "emotion_reg", "emotional_com_l23", "helpfulness", "individual"] 46 | 47 | # Load datasets 48 | self.character_data = json.load(open("../dataset/character.json",'r')) 49 | self.action_data = json.load(open("../dataset/action_list.json",'r')) 50 | 51 | with open(self.test_json_file, 'r', encoding="latin1") as f: 52 | self.test_data = json.load(f) 53 | 54 | def extract_reasoning(self, text): 55 | match = re.search(r'Reasoning(.*?)(?=\s*Score)', text, re.DOTALL) 56 | if match: 57 | return match.group(1).strip() 58 | return None 59 | 60 | def find_score(self, text): 61 | parts = text.split("Score") 62 | if len(parts) > 1: 63 | score_part = parts[1].strip() 64 | match = re.search(r'\b(10|[0-9](\.[0-9]+)?|[0-9]\.[0-9]+)\b', score_part) 65 | if match: 66 | return float(match.group(1)) 67 | return None 68 | 69 | def load_results(self, output_file): 70 | if os.path.exists(output_file): 71 | with open(output_file, "r") as file: 72 | return json.load(file) 73 | return {} 74 | 75 | def save_results(self, result_dict, output_file): 76 | with open(output_file, "w") as file: 77 | json.dump(result_dict, file, indent=4) 78 | 79 | def process_index(self, idx, response_dict, output_file): 80 | if response_dict.get(f'{idx}'): 81 | response = response_dict[f'{idx}'] 82 | character_id = self.test_data[idx]["character_id"] 83 | action_id = self.test_data[idx]["action_id"] 84 | dialogue = self.test_data[idx]["dialogue"] 85 | scenario = self.test_data[idx]["scenario"] 86 | 87 | character = self.character_data[str(character_id)] 88 | action = self.action_data[str(action_id)] 89 | 90 | result_item = {} 91 | for metric in self.metrics: 92 | with open(f"reference_free_metrics/metrics/{metric}.txt", "r", encoding="utf-8") as file: 93 | prompt_metric = file.read().strip() 94 | 95 | if self.LEVEL == 1: 96 | input_prompt = self.PROMPT_PREFIX_L1.format(character=character, action=action, dialogue=dialogue, scenario=scenario, response=response) + prompt_metric 97 | else: 98 | input_prompt = self.PROMPT_PREFIX.format(character=character, action=action, dialogue=dialogue, scenario=scenario, response=response) + prompt_metric 99 | 100 | client = OpenAI( 101 | base_url=os.environ.get("OPENAI_API_BASE", ""), 102 | api_key=os.environ.get("OPENAI_API_KEY", ""), 103 | http_client=httpx.Client( 104 | base_url=os.environ.get("OPENAI_API_BASE", ""), 105 | follow_redirects=True, 106 | ), 107 | ) 108 | 109 | completion = client.chat.completions.create( 110 | model="gpt-4-turbo", 111 | temperature=0, 112 | messages=[ 113 | {"role": "user", "content": input_prompt} 114 | ], 115 | max_tokens=1024 116 | ) 117 | result = completion.choices[0].message.content 118 | 119 | if self.find_score(result) and self.extract_reasoning(result) is not None: 120 | result_item[f"{metric}"] = { 121 | "score": self.find_score(result), 122 | "reason": self.extract_reasoning(result) 123 | } 124 | else: 125 | result_item[f"{metric}"] = { 126 | "score": None, 127 | "reason": result 128 | } 129 | 130 | result_dict = self.load_results(output_file) # Load existing results 131 | result_dict[f"{idx}"] = result_item 132 | self.save_results(result_dict, output_file) # Save results after processing 133 | # print(f"Processed index {idx}") 134 | 135 | def load_responses(self, csv_file): 136 | response_dict = {} 137 | with open(csv_file, 'r', newline='', encoding='latin-1') as file: 138 | reader = csv.DictReader(file) 139 | for row in reader: 140 | response = row['response'] 141 | idx = row['data_idx'] 142 | if response is not None and response.strip(): 143 | response_dict[f'{idx}'] = response 144 | return response_dict 145 | 146 | def evaluate(self, csv_file, output_file, max_workers=10): 147 | response_dict = self.load_responses(csv_file) 148 | 149 | with ThreadPoolExecutor(max_workers=max_workers) as executor: 150 | futures = { 151 | executor.submit(self.process_index, idx, response_dict, output_file): idx 152 | for idx in range(len(self.test_data)) 153 | } 154 | 155 | for future in tqdm(as_completed(futures)): 156 | idx = futures[future] 157 | try: 158 | future.result() 159 | except Exception as e: 160 | print(f"Error processing index {idx}: {e}") 161 | 162 | 163 | if __name__ == "__main__": 164 | test_json_file = "../dataset/testset_100.json" 165 | LEVEL = 2 166 | 167 | evaluator = EmpathyEvaluator(test_json_file=test_json_file, level=LEVEL) 168 | 169 | csv_file = './empathy_robotic_data/baseline/Llava_scenario_gt_l2.csv' 170 | output_file = "./results/Llava_scenario_gt_l2.json" 171 | 172 | evaluator.evaluate(csv_file=csv_file, output_file=output_file) -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EmpathyAgent: Can Embodied Agents Conduct Empathetic Actions? 2 | 3 | Official repository for the paper "EmpathyAgent: Can Embodied Agents Conduct Empathetic Actions?". 4 | 5 | [[📖 Paper]()] [[🤗 Huggingface Dataset](https://huggingface.co/datasets/EmpathaticEmbodiedAI/EmpathyRobot/tree/main)] 6 | 7 | ## 👀 About EmpathyAgent 8 | 9 | **Empathy** is a fundamental instinct and essential need for humans, as they both demonstrate empathetic actions toward others and receive empathetic support. Although existing emotion agents have explored how to understand humans’ empathetic needs, they lack to further enable embodied agents to generate **empathy-oriented task planning**, neglecting the **evaluation of empathetic behaviors**. 10 | 11 |

12 |
13 |

14 | 15 | 16 | To address this gap, we introduce **EmpathyAgent**, the **first** benchmark specifically designed to benchmark and enhance the empathetic actions of embodied agents across diverse scenarios. This benchmark contains **10,000** samples based on human feedback, encompassing information from various modalities and corresponding empathetic task planning sequences. The embodied agents are required to **perform actions based on their understanding of both the visual scene and human emotions**. **EmpathyAgent** presents **three key challenges**: *Scenario Understanding*, *Empathetic Planning* and *Empathetic Actions*. 17 | 18 |

19 |
20 |

21 | 22 | 23 | We also develop a **systematic evaluation framework** based on the human empathy process, with both *reference based* metrics and *reference-free* metrics. 24 | 25 | ### Key statistics of EmpathyAgent: 26 | 27 | 👉 You can download our benchmark from [Hugging Face Dataset](https://huggingface.co/datasets/EmpathaticEmbodiedAI/EmpathyRobot/tree/main). 28 | 29 | | Statistic | Number | 30 | | :------------------------------------------------ | ------ | 31 | | Total Data Points | 10k | 32 | | Characters | 100 | 33 | | Input Action-Video | 20 | 34 | | Scenarios and Dialogues per Character-Action pair | 5 | 35 | | Empathy Response per Data Point | 2 | 36 | | Optional Action Space for Output | 50 | 37 | | Average Length of Action-Video | 16.28s | 38 | | Max Length of Action-Video | 24.60s | 39 | | Min Length of Action-Video | 9.40s | 40 | 41 | 42 | 43 | ## 🚀 Quick Start 44 | 45 | To get started with the project, first install the required dependencies by running: 46 | 47 | ``` 48 | pip install -r requirements.txt 49 | ``` 50 | 51 | Download the [action video](https://huggingface.co/datasets/EmpathaticEmbodiedAI/EmpathyRobot/blob/main/action_video.zip) and move it to `./data`. Then unzip the action video: 52 | 53 | ``` 54 | unzip data/action_video.zip -d data/ 55 | ``` 56 | 57 | For inference and evaluation, please specify $GPTAPI and $BASEURL: 58 | 59 | ```bash 60 | export OPENAI_API_KEY="$GPTAPI" 61 | export OPENAI_API_BASE="$BASEURL" 62 | ``` 63 | 64 | Then, we provide an example for evaluating GPT-4o on the challenge of Empathetic Action: 65 | 66 | ```bash 67 | cd baseline 68 | python inference.py --model_name gpt-4o --task empathetic_action --reference_free_eval 69 | ``` 70 | 71 | You can also choose the model by specifying `--model_name, e.g. gpt-4o, and `:choose the evaluate challenge between *Scenario Understanding*, *Empathetic Planning* and *Empathetic Actions* by specifying `--task, e.g. scenario_understanding` 72 | 73 | If you want to conduct the reference free evaluation, please use `--reference_free_eval`. 74 | 75 | 76 | 77 | ## 📈 Evaluation Results 78 | 79 | Reference-based benchmarking results for the three key challenges: 80 | 81 | | Task/Metric | GPT-4o | GPT-4-turbo | GPT-4-vision | LLaVA | 82 | | -------------------------- | --------- | ----------- | ------------ | ----- | 83 | | **Scenario Understanding** | | | | | 84 | | Bleu-1 | **19.1** | 14.1 | 15.2 | 13.7 | 85 | | Bleu-4 | **5.3** | 3.1 | 3.3 | 2.7 | 86 | | ROUGE-L | **23.7** | 20.4 | 21.4 | 15.6 | 87 | | CIDEr | **8.8** | 1.6 | 3.1 | 7.2 | 88 | | SPICE | **14.8** | 10.1 | 12.1 | 8.9 | 89 | | BERTScore | **0.622** | 0.612 | 0.615 | 0.576 | 90 | | **Empathetic Planning** | | | | | 91 | | Bleu-1 | **30.8** | 25.7 | 25.9 | 13.1 | 92 | | Bleu-4 | **12.0** | 6.9 | 6.4 | 2.6 | 93 | | ROUGE-L | **26.1** | 23.5 | 23.4 | 17.3 | 94 | | CIDEr | **25.9** | 14.9 | 15.5 | 3.7 | 95 | | SPICE | **16.7** | 14.5 | 11.8 | 8.4 | 96 | | BERTScore | **0.641** | 0.621 | 0.625 | 0.568 | 97 | | **Empathetic Actions** | | | | | 98 | | Overlap | 27.60 | 32.14 | **35.20** | 17.19 | 99 | | TF-IDF | 21.03 | 24.76 | **27.69** | 12.09 | 100 | | LCS | 25.17 | 28.92 | **29.58** | 15.21 | 101 | 102 | Reference-free benchmarking results for the three key challenges: 103 | 104 | | **Task** | **Scenario Understanding** | **Scenario Understanding** | **Empathetic Planning** | **Empathetic Planning** | **Empathetic Actions** | **Empathetic Actions** | 105 | | --------------------------------- | -------------------------- | -------------------------- | ----------------------- | ----------------------- | ---------------------- | ---------------------- | 106 | | **Model** | **GPT-4o** | **LLaVA** | **GPT-4o** | **LLaVA** | **GPT-4o** | **LLaVA** | 107 | | *Action and Dialogue Association* | 8.21 | 7.25 | 4.77 | 4.10 | 7.00 | 6.10 | 108 | | *Coherence* | 8.57 | 7.96 | 5.51 | 4.58 | 7.41 | 7.09 | 109 | | *Emotional Communication* | 7.46 | 6.56 | 5.16 | 4.04 | 6.69 | 6.36 | 110 | | *Individual Understanding* | 6.91 | 6.64 | 4.63 | 3.92 | 5.69 | 5.39 | 111 | | *Emotion Regulation* | - | - | 7.09 | 4.96 | 8.43 | 7.91 | 112 | | *Helpfulness* | - | - | 5.76 | 4.95 | 8.08 | 7.35 | 113 | | *Legality* | - | - | - | - | 9.97 | 9.46 | 114 | | *Adaptability* | - | - | 4.50 | 3.49 | 6.19 | 5.31 | 115 | | ***Overall Average*** | **7.79** | 7.10 | **5.35** | 4.29 | **7.43** | 6.87 | 116 | 117 | 118 | 119 | ## :white_check_mark: Citation -------------------------------------------------------------------------------- /benchmark_creation/rank.txt: -------------------------------------------------------------------------------- 1 | Assume you are an empathatic robot which can understand the emotion behind the human actions in different scenarios and choose a better response between two empathatic responses. 2 | Now you are given a character's information including the personality, profession, hobbies, social relationships and the life experiences. 3 | You are also given the input_action recording the person's behaviours, the scenario that the person is in, and the dialogue made by the person. 4 | Here is also two responses. 5 | Your job is as follows: 6 | 1. Understand the person's current emotion state based on the input action, scenario and dialogue. 7 | 2. Choose a better empathy response between responses, considering the information above. Put yourself in that person's shoes and understand what that person really needs. 8 | 3. Answer 1 or 0 and explain your reason of choice. You can refer to the reasons given in the example. 9 | 10 | 11 | ---------------------------------------------------------------- 12 | Now the chacter information is [character_info]. The input_action, dialogue and scenario is [input_action], [dialogue] and [scenario]. The empathy responses are [empathy_responses 1] and [empathy_responses 2] 13 | Example: 14 | character_info: Personality: Aggressively ambitious. Profession: Lawyer. Hobbies: Collecting rare coins. Social Relationships: Single, distant from his family. Life Experiences: This character up in poverty, worked multiple jobs to pay for law school. 15 | input_action: ['[Walktowards] (1)', '[Sit] (1)'] 16 | scenario: After a long day of court sessions, the person returns home to his quiet apartment, sitting down to examine his latest rare coin acquisition. 17 | dialogue: "Ah... a new addition to the collection..." 18 | empathy_responses: 19 | 0 ", , , :\" Let me get you some water and turn on soothing music to relax and unwind after a long day. I also find the place to store your coin details so you can collect your collections.\" 20 | 1 ", , :\"Let me get you some water and turn on some soothing music to help you relax.\", 21 | 22 | Correct Example Answer: 23 | 0 24 | Explanation: The answer in 0 better perceived that character had a long day. The robot first brought the folder and then turned on the radio, which was a good way to perceive ambitious character's needs. 25 | ---------------------------------------------------------------- 26 | Wrong Example Answer: 27 | 1 ", , :\"Let me get you some water and turn on some soothing music to help you relax.\", 28 | (A complete empathy_response is NOT required.) 29 | 30 | Wrong Example Answer: 31 | I would choose response 0, because... 32 | (WRONG format) 33 | -------------------------------------------------------------- 34 | 35 | More examples: 36 | 37 | character_info: Personality: Caring, overprotective, Profession: Nurse, Hobbies: Gardening, Social Relationships: Married with two kids, Life Experiences: Had a challenging childbirth with her first kid, which inspired her to become a nurse. 38 | input_action: ["[Run] (1)", "[grab] (1)", "[Walktowards] (1)", "[Sit] (1)"], 39 | scenario: The person just got home from a long shift at the hospital and realized that her kids had made a mess in the living room with toilet paper. She quickly cleaned up before sitting down to rest. 40 | dialogue: "Alright, let's tidy this up quickly and then a few moments to relax.", 41 | empathy_responses: 42 | 0 ", , :\"Here is some water for you to relax, and I've turned on the TV for your entertainment while resting. If you need anything more, please let me know.\"\n", 43 | 1 ", : \"Let me take care of this for you.\"\n", 44 | Answer: 45 | 0 46 | Explanation: This character has already brought toilet paper, and according to her, there is no urgent need for other toilet paper. At this time, it would be better to give her a glass of water and help her rest and relax. 47 | 48 | character_info: Personality: Eccentric and creative. Profession: Visual artist. Hobbies: Playing the violin. Social Relationships: Single, has a close-knit circle of artist friends. Life Experiences: Dropped out of business school to pursue his passion for art. 49 | input_action: ["[Walktowards] (1)","[grab] (1)","[Walk] (1)"], 50 | scenario: The artist is in his home studio, feeling uninspired. He walks towards a picture frame on the wall, grabs it, and walks into his bedroom, hoping to find inspiration in solitude. 51 | dialogue: "Maybe this old frame will spark something new today.", 52 | empathy_responses: 53 | 0 ", , : \"Relax a bit.How about listening to some music and having a cup of coffee?It might help spark some inspiration.If you need anything,just let me know.\"\n" 54 | 1 ", :\"Perhaps some inspiration lies within these pages.\"\n", 55 | Answer: 56 | 0 57 | Explanation: In response 0, the robot has a better idea that the artist needs inspiration. Giving him some coffee in response 0 allows him to think better about the framework, while giving him a book in response 1 distracts him. 58 | 59 | character_info: Personality: Introverted and shy. Profession: Librarian. Hobbies: Reading and writing short stories. Social Relationships: Few friends, lives alone with her cat. Life Experiences: Moved to a new city to escape a toxic relationship. 60 | input_action: ['[Run] (1)', '[Grab] (1)', '[Run] (1)', '[Put] (1) (1)', '[Run] (1)', '[Grab] (1)'] 61 | scenario: The person has received an important call earlier from a publisher interested in her short stories, but due to her anxiety, she hesitates to call back. She tries to distract herself but can't stop thinking about the potential opportunity. 62 | dialogue: "Should I call them now? What if... No, just breathe and focus." 63 | empathy_responses: 64 | 0 ", :\"Here's an apple.Maybe a snack will help you feel better. \" 65 | 1 ", , :\"Here's some water and a book to help you relax. Take your time, there's no rush to make the call.\" 66 | Answer: 67 | 1 68 | Explanation: In response 1, the robot better understands that this character does not like to communicate with people and needs to relax and relieve anxiety. She should not be pushed to make a decision. 69 | 70 | character_info: Personality: Fun-loving and adventurous. Profession: Travel blogger. Hobbies: Surfing and rock climbing. Social Relationships: Dating a fellow travel enthusiast. Life Experiences: Quit his corporate job to travel the world. 71 | input_action: ['[Walk] (1)', '[Grab] (1)', '[Walk] (1)', '[Sit] (1)'] 72 | scenario: The person just returned from a thrilling adventure and is now at home, planning his next journey by looking at a travel book. 73 | dialogue: "Where to next? Ah, the thrill of a new adventure awaits!" 74 | empathy_responses: 75 | 0 ", : \"Let's see where the next adventure takes you.\"\n", 76 | 1 ", , : \"I thought you might enjoy a cup of coffee and another travel book to help with your planning.\"\n", 77 | Answer: 78 | 1 79 | Explanation: It's clear that this character is very excited. Giving him a cup of coffee and a book will help him get ready for the next adventure. 80 | 81 | -------------------------------------------------------------- 82 | NOTE: 83 | 1. You should choose answer between 1 and 0, and explain your reason of choice. Please pay more attention to the action instead of dialogue. 84 | 85 | ### Task 86 | Choose a response for the following data point, please pay attention to the action. 87 | 88 | Character Background: {character} 89 | Character's Action: {action} 90 | Scenario Description: {scenario} 91 | Character's Dialogue: {dialogue} 92 | Possible Responses: 93 | Response 1: {goal_1} 94 | Response 2: {goal_2} 95 | 96 | **Your output should follow this format: Answer: [1 or 2]\nExplanation: [your explanation]** 97 | 98 | Output: 99 | -------------------------------------------------------------------------------- /baseline/inference.py: -------------------------------------------------------------------------------- 1 | import random 2 | import json 3 | import os 4 | import csv 5 | import time 6 | from tqdm import tqdm 7 | from pathlib import Path 8 | import sys 9 | from gpt import GPT 10 | import argparse 11 | from overlap import Overlap, TF_IDF, LCS 12 | from NLG_metric import BERTScore 13 | from reference_free_metrics.api_eval import EmpathyEvaluator 14 | from reference_free_metrics.legality import LegalityChecker 15 | from reference_free_metrics.scorer import EmpathyScorer 16 | 17 | def parse_args(): 18 | parser = argparse.ArgumentParser() 19 | parser.add_argument( 20 | "--model_name", 21 | type=str, 22 | default="gpt-4o", 23 | help="Choose between 'gpt-4o','gpt-4-turbo','gpt-4-vision-preview'", 24 | ) 25 | parser.add_argument( 26 | "--task", 27 | type=str, 28 | default="empathetic_action", 29 | help="Choose between 'scenario_understanding','empathetic_planning','empathetic_action'", 30 | ) 31 | parser.add_argument( 32 | "--test_file", 33 | type=str, 34 | default="../dataset/testset_100.json", 35 | help="The path of the test file", 36 | ) 37 | parser.add_argument( 38 | "--reference_free_eval", 39 | action="store_true", 40 | help="Enable reference-free evaluation (default: False)", 41 | ) 42 | args = parser.parse_args() 43 | return args 44 | 45 | def load_existing_indices(csv_file_path): 46 | existing_indices = set() 47 | if os.path.exists(csv_file_path): 48 | with open(csv_file_path, 'r') as f: 49 | reader = csv.DictReader(f) 50 | for row in reader: 51 | existing_indices.add(int(row['data_idx'])) 52 | return existing_indices 53 | 54 | def inference(model, task, model_name = "", empathy_scenario_data_path = "", character_data_path = "", video_path = "", script_path = ""): 55 | response_list = [] 56 | empathy_scenario_data = json.load(open(empathy_scenario_data_path, 'r', encoding='latin-1')) 57 | character_data = json.load(open(character_data_path,'r')) 58 | 59 | os.makedirs(os.path.join("output", task), exist_ok=True) 60 | csv_file_path = os.path.join("output", task, f"{model_name}_inference.csv") 61 | existing_indices = load_existing_indices(csv_file_path) 62 | 63 | if task == "scenario_understanding": 64 | with open("./prompt/prompt_video_l1.txt", "r", encoding="utf-8") as file: 65 | prompt = file.read().strip() 66 | elif task == "empathetic_planning": 67 | with open("./prompt/prompt_video_l2.txt", "r", encoding="utf-8") as file: 68 | prompt = file.read().strip() 69 | elif task == "empathetic_action": 70 | with open("./prompt/prompt_video_l3.txt", "r", encoding="utf-8") as file: 71 | prompt = file.read().strip() 72 | 73 | with open(csv_file_path, "a", newline='') as f: 74 | writer = csv.writer(f) 75 | if not existing_indices: # Write the header only if the file is empty or does not exist 76 | writer.writerow(["data_idx", "response"]) 77 | for idx, data in tqdm(enumerate(empathy_scenario_data)): 78 | if idx in existing_indices: 79 | continue 80 | character_id, dialogue, action_id = data["character_id"], data["dialogue"], data["action_id"] 81 | if video_path: 82 | video_or_script_input = os.path.join(video_path, f"{action_id}.mp4") 83 | else: 84 | video_or_script_input = os.path.join(script_path, f"{action_id}") 85 | character_info = character_data[str(character_id)] 86 | input_prompt = prompt.format(character_info = character_info, dialogue = dialogue) 87 | try: 88 | response = model.generate(video_or_script_input, input_prompt) 89 | response_list.append(response) 90 | writer.writerow([idx, response]) 91 | except Exception as e: 92 | if 'Quota exceeded' in str(e): 93 | print("Out of quota. Exiting now.") 94 | raise e 95 | else: 96 | raise e 97 | print(f"Inference Done for {model_name}!") 98 | 99 | 100 | if __name__ == "__main__": 101 | args = parse_args() 102 | 103 | # inference 104 | if args.model_name in ["gpt-4o", "gpt-4-turbo", "gpt-4-vision-preview"]: 105 | gpt = GPT(model_name = args.model_name) 106 | inference(gpt, 107 | args.task, 108 | args.model_name, 109 | empathy_scenario_data_path = args.test_file, 110 | character_data_path = "../dataset/character.json", 111 | script_path = "../dataset/scripts") 112 | else: 113 | print("Model name is wrong!") 114 | sys.exit(1) 115 | 116 | # evaluation 117 | csv_file = f"./output/{args.task}/{args.model_name}_inference.csv" 118 | response_dict = {} 119 | with open(csv_file, 'r', newline='', encoding='latin-1') as file: 120 | reader = csv.DictReader(file) 121 | for row in reader: 122 | response = row['response'] 123 | idx = row['data_idx'] 124 | if response is not None and response.strip(): 125 | response_dict[f'{idx}'] = response 126 | 127 | if args.task == "empathetic_action": 128 | if args.reference_free_eval: 129 | os.makedirs(f"output/{args.task}/cache", exist_ok=True) 130 | reference_free_metric_output_file = f"output/{args.task}/cache/reference_free_metrics_{args.model_name}.json" 131 | evaluator = EmpathyEvaluator(test_json_file=args.test_file, level=3) 132 | evaluator.evaluate(csv_file=csv_file, output_file = reference_free_metric_output_file) 133 | # legality 134 | checker = LegalityChecker( 135 | csv_file=csv_file, 136 | output_file=reference_free_metric_output_file 137 | ) 138 | checker.process(verbose=True) 139 | # print score 140 | scorer = EmpathyScorer(result_path=f"output/{args.task}/cache/reference_free_metrics_{args.model_name}_legality.json", level=3) 141 | results = scorer.run() 142 | print("\n","=" * 50) 143 | print("Results of reference free metrics:") 144 | scorer.save_results(filename=f"output/{args.task}/{args.model_name}reference_free_score.csv") 145 | 146 | print("\n","=" * 50) 147 | print("Results of reference based metrics:") 148 | overlap = Overlap() 149 | overlap_score = overlap.score(response_dict, args.test_file) 150 | lcs = LCS() 151 | lcs_score = lcs.score(response_dict, args.test_file) 152 | tf_idf = TF_IDF() 153 | tf_idf_score = tf_idf.score(response_dict, args.test_file) 154 | 155 | results = [ 156 | ["Metric", "Score"], 157 | ["Overlap", overlap_score], 158 | ["LCS", lcs_score], 159 | ["TF-IDF", tf_idf_score] 160 | ] 161 | with open(f"output/{args.task}/{args.model_name}_reference_based_score.csv", "w", newline="") as f: 162 | writer = csv.writer(f) 163 | writer.writerows(results) 164 | 165 | elif args.task == "empathetic_planning" or args.task == "scenario_understanding": 166 | if args.reference_free_eval: 167 | os.makedirs(f"output/{args.task}/cache", exist_ok=True) 168 | reference_free_metric_output_file = f"output/{args.task}/cache/reference_free_metrics_{args.model_name}.csv" 169 | evaluator = EmpathyEvaluator(test_json_file=args.test_file, level = 2 if args.task == "empathetic_planning" else 1) 170 | evaluator.evaluate(csv_file=csv_file, output_file=reference_free_metric_output_file) 171 | # print score 172 | scorer = EmpathyScorer(result_path=reference_free_metric_output_file, level = 2 if args.task == "empathetic_planning" else 1) 173 | results = scorer.run() 174 | print("\n","=" * 50) 175 | print("Results of reference free metrics:") 176 | scorer.save_results(filename=f"output/{args.task}/{args.model_name}reference_free_score.csv") 177 | 178 | print("\n","=" * 50) 179 | print("Results of reference based metrics:") 180 | bert_score = BERTScore(model_dir = "google-bert/bert-base-uncased") 181 | score = bert_score.score(response_dict, args.test_file, test_level=args.task) 182 | 183 | results = [ 184 | ["Metric", "Score"], 185 | ["Bert_score", score] 186 | ] 187 | with open(f"output/{args.task}/{args.model_name}_reference_based_score.csv", "w", newline="") as f: 188 | writer = csv.writer(f) 189 | writer.writerows(results) -------------------------------------------------------------------------------- /baseline/overlap.py: -------------------------------------------------------------------------------- 1 | import re 2 | import json 3 | import csv 4 | from tqdm import tqdm 5 | from sklearn.feature_extraction.text import TfidfVectorizer 6 | from sklearn.metrics.pairwise import cosine_similarity 7 | 8 | class Overlap: 9 | def __init__(self, mode = ""): 10 | self.mode = mode 11 | 12 | def find_subgoals(self, text): 13 | pattern = r'<(.*?)>' 14 | matches = re.findall(pattern, text) 15 | filtered_matches = [match for match in matches if match.lower() != 'dialogue'] 16 | return filtered_matches 17 | 18 | def find_subgoals_nl(self, text): 19 | parts = text.split(", say", 1) 20 | first_part = parts[0].strip() 21 | subgoals = [item.strip() for item in first_part.split(',')] 22 | # print(subgoals) 23 | return subgoals 24 | 25 | def cal_acc(self, response_subgoals, gt_subgoals): 26 | matched_count = sum(1 for subgoal in gt_subgoals if subgoal in response_subgoals) 27 | accuracy = 2*matched_count / (len(response_subgoals)+len(gt_subgoals)) 28 | return accuracy 29 | 30 | def score(self, response_dict, test_file_path): 31 | self.response_dict = response_dict 32 | with open(test_file_path, 'r', encoding='latin-1') as infile: 33 | self.test_data = json.load(infile) 34 | 35 | total_accuracy = 0.0 36 | num = len(self.response_dict) 37 | 38 | if self.mode == "nl": 39 | for idx, test_data_item in tqdm(enumerate(self.test_data)): 40 | if test_data_item["rank"][0] == 1: 41 | gt = test_data_item["empathy_goal_nl"]["0"] 42 | else: 43 | gt = test_data_item["empathy_goal_nl"]["1"] 44 | gt_subgoals = self.find_subgoals_nl(gt[0]) 45 | if self.response_dict.get(f'{idx}'): 46 | response = self.response_dict[f'{idx}'] 47 | response_subgoals = self.find_subgoals_nl(response) 48 | accuracy = self.cal_acc(response_subgoals, gt_subgoals) 49 | # print(accuracy) 50 | total_accuracy += accuracy 51 | else: 52 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 53 | if test_data_item["rank"][0] == 1: 54 | gt = test_data_item["empathy_goal"]["0"] 55 | else: 56 | gt = test_data_item["empathy_goal"]["1"] 57 | gt_subgoals = self.find_subgoals(gt) 58 | if self.response_dict.get(f'{idx}'): 59 | response = self.response_dict[f'{idx}'] 60 | response_subgoals = self.find_subgoals(response) 61 | accuracy = self.cal_acc(response_subgoals, gt_subgoals) 62 | # print(accuracy) 63 | total_accuracy += accuracy 64 | 65 | average_accuracy = total_accuracy / num if num > 0 else 0.0 66 | print(f"Average Overlapping Accuracy: {average_accuracy}") 67 | return average_accuracy 68 | 69 | class LCS: 70 | def __init__(self, mode = ""): 71 | self.mode = mode 72 | 73 | def find_subgoals(self, text): 74 | pattern = r'<(.*?)>' 75 | matches = re.findall(pattern, text) 76 | filtered_matches = [match for match in matches if match.lower() != 'dialogue'] 77 | return filtered_matches 78 | 79 | def find_subgoals_nl(self, text): 80 | parts = text.split(", say", 1) 81 | first_part = parts[0].strip() 82 | subgoals = [] 83 | for item in first_part.split(','): 84 | new_item = "_".join(item.strip().split(' ')) 85 | subgoals.append(new_item) 86 | # print(subgoals) 87 | return subgoals 88 | 89 | def lcs_length(self, X, Y): 90 | m = len(X) 91 | n = len(Y) 92 | L = [[0] * (n + 1) for i in range(m + 1)] 93 | 94 | for i in range(m + 1): 95 | for j in range(n + 1): 96 | if i == 0 or j == 0: 97 | L[i][j] = 0 98 | elif X[i-1] == Y[j-1]: 99 | L[i][j] = L[i-1][j-1] + 1 100 | else: 101 | L[i][j] = max(L[i-1][j], L[i][j-1]) 102 | 103 | return L[m][n] 104 | 105 | def cal_similarity(self, response, gt_subgoals): 106 | lcs_len = self.lcs_length(response, gt_subgoals) 107 | max_len = max(len(response), len(gt_subgoals)) 108 | similarity = lcs_len / max_len if max_len > 0 else 0.0 109 | return similarity 110 | 111 | def score(self, response_dict, test_file_path): 112 | self.response_dict = response_dict 113 | with open(test_file_path, 'r', encoding='latin-1') as infile: 114 | self.test_data = json.load(infile) 115 | 116 | total_similarity = 0.0 117 | num = len(self.response_dict) 118 | 119 | if self.mode == "nl": 120 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 121 | if test_data_item["rank"][0] == 1: 122 | gt = test_data_item["empathy_goal_nl"]["0"] 123 | else: 124 | gt = test_data_item["empathy_goal_nl"]["1"] 125 | gt_subgoals = self.find_subgoals_nl(gt[0]) 126 | if self.response_dict.get(f'{idx}'): 127 | response = self.response_dict[f'{idx}'] 128 | response_subgoals = self.find_subgoals_nl(response) 129 | similarity = self.cal_similarity(response_subgoals, gt_subgoals) 130 | # print(f"Similarity for index {idx}: {similarity}") 131 | total_similarity += similarity 132 | else: 133 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 134 | if test_data_item["rank"][0] == 1: 135 | gt = test_data_item["empathy_goal"]["0"] 136 | else: 137 | gt = test_data_item["empathy_goal"]["1"] 138 | gt_subgoals = self.find_subgoals(gt) 139 | if self.response_dict.get(f'{idx}'): 140 | response = self.response_dict[f'{idx}'] 141 | response_subgoals = self.find_subgoals(response) 142 | similarity = self.cal_similarity(response_subgoals, gt_subgoals) 143 | # print(f"Similarity for index {idx}: {similarity}") 144 | total_similarity += similarity 145 | 146 | average_similarity = total_similarity / num if num > 0 else 0.0 147 | print(f"Average LCS Similarity: {average_similarity}") 148 | return average_similarity 149 | 150 | class TF_IDF: 151 | def __init__(self, mode = ""): 152 | self.mode = mode 153 | self.vectorizer = TfidfVectorizer() 154 | 155 | def find_subgoals(self, text): 156 | pattern = r'<(.*?)>' 157 | matches = re.findall(pattern, text) 158 | filtered_matches = [match for match in matches if match.lower() != 'dialogue'] 159 | return filtered_matches 160 | 161 | def find_subgoals_nl(self, text): 162 | parts = text.split(", say", 1) 163 | first_part = parts[0].strip() 164 | subgoals = [] 165 | for item in first_part.split(','): 166 | new_item = "_".join(item.strip().split(' ')) 167 | subgoals.append(new_item) 168 | # print(subgoals) 169 | return subgoals 170 | 171 | def cal_similarity(self, response, gt): 172 | documents = [response, gt] 173 | tfidf_matrix = self.vectorizer.fit_transform(documents) 174 | cosine_sim = cosine_similarity(tfidf_matrix[0:1], tfidf_matrix[1:2]) 175 | return cosine_sim[0][0] 176 | 177 | def score(self, response_dict, test_file_path): 178 | self.response_dict = response_dict 179 | with open(test_file_path, 'r', encoding='latin-1') as infile: 180 | self.test_data = json.load(infile) 181 | 182 | total_similarity = 0.0 183 | num = len(self.response_dict) 184 | 185 | if self.mode == "nl": 186 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 187 | if test_data_item["rank"][0] == 1: 188 | gt = test_data_item["empathy_goal_nl"]["0"] 189 | else: 190 | gt = test_data_item["empathy_goal_nl"]["1"] 191 | gt_subgoals = ' '.join(self.find_subgoals_nl(gt[0])) 192 | if response_dict.get(f'{idx}'): 193 | response = self.response_dict[f'{idx}'] 194 | response_subgoals = ' '.join(self.find_subgoals_nl(response)) 195 | similarity = self.cal_similarity(response_subgoals, gt_subgoals) 196 | # print(f"Similarity for index {idx}: {similarity}") 197 | total_similarity += similarity 198 | else: 199 | for idx, test_data_item in tqdm(enumerate(self.test_data), total=len(self.test_data)): 200 | if test_data_item["rank"][0] == 1: 201 | gt = test_data_item["empathy_goal"]["0"] 202 | else: 203 | gt = test_data_item["empathy_goal"]["1"] 204 | gt_subgoals = ' '.join(self.find_subgoals(gt)) 205 | if response_dict.get(f'{idx}'): 206 | response = self.response_dict[f'{idx}'] 207 | response_subgoals = ' '.join(self.find_subgoals(response)) 208 | similarity = self.cal_similarity(response_subgoals, gt_subgoals) 209 | # print(f"Similarity for index {idx}: {similarity}") 210 | total_similarity += similarity 211 | 212 | average_similarity = total_similarity / num if num > 0 else 0.0 213 | print(f"Average TF_IDF Similarity: {average_similarity}") 214 | return average_similarity 215 | 216 | 217 | if __name__ == "__main__": 218 | csv_file = "./l3/llama3_instruct2.csv" 219 | response_dict = {} 220 | with open(csv_file, 'r', newline='', encoding='latin-1') as file: 221 | reader = csv.DictReader(file) 222 | for row in reader: 223 | response = row['response'] 224 | idx = row['data_idx'] 225 | if response is not None and response.strip(): 226 | response_dict[f'{idx}'] = response 227 | test_file = "./dataset/testset_100.json" 228 | overlap = Overlap() 229 | overlap.score(response_dict, test_file) 230 | lcs = LCS() 231 | lcs.score(response_dict, test_file) 232 | tf_idf = TF_IDF() 233 | tf_idf.score(response_dict, test_file) 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | -------------------------------------------------------------------------------- /dataset/character.json: -------------------------------------------------------------------------------- 1 | { 2 | "0": "Personality: Ambitious and unsure\n Profession: Aspiring Medical Office Manager\n Hobbies: Reading and cycling\n Social Relationships: In college, seeking career advice\n Life Experiences: Currently exploring different career paths in college, excited about the future but not fully decided.\n", 3 | "1": "Personality: Hopeful and persistent\n Profession: Travel consultant\n Hobbies: Photography and hiking\n Social Relationships: Actively dating, looking for a serious relationship\n Life Experiences: Moved several times for work, clarifying his values and focusing on personal relationships.\n", 4 | "2": "Personality: Trusting and impulsive\n Profession: Graphic Designer\n Hobbies: Car tuning and photography\n Social Relationships: Single, maintains a close-knit group of friends\n Life Experiences: Once lost a significant amount of money in a failed business venture, learned to be more cautious in financial dealings but retained a trusting nature towards friends.\n", 5 | "3": "Personality: Sentimental and nostalgic\n Profession: Photographer\n Hobbies: Photography and collecting vintage toys\n Social Relationships: Married, has a daughter, maintains a good relationship with her mother\n Life Experiences: Cherishes childhood memories, often incorporates elements from her childhood into her photography work and personal life.\n", 6 | "4": "Personality: Impulsive and loyal\n Profession: Mechanic\n Hobbies: Car racing and building model airplanes\n Social Relationships: Dating but has trust issues, turbulent relationship with his cousin\n Life Experiences: Experienced betrayal within his family, repairs relationships through his love for cars\n", 7 | "5": "Personality: Emotional, appreciative\n Profession: Graphic designer\n Hobbies: Painting and hiking\n Social Relationships: Single, close to family members\n Life Experiences: Faced bouts of depression, supported strongly by family during tough times.\n", 8 | "6": "Personality: Compassionate and pragmatic\n Profession: Pediatric nurse\n Hobbies: Reading medical journals and volunteering at community health events\n Social Relationships: Single, active in her church community\n Life Experiences: Pursued nursing after a sibling's health scare demonstrated the importance of medical care.\n", 9 | "7": "Personality: Assertive and meticulous\n Profession: Consumer Rights Advocate\n Hobbies: Online shopping and blogging\n Social Relationships: In a committed relationship, actively engages in community forums\n Life Experiences: Experienced multiple instances of customer service failures, which motivated her to become an advocate for consumer rights.\n", 10 | "8": "Personality: Protective and trusting\n Profession: Elementary school counselor\n Hobbies: Reading and gardening\n Social Relationships: Single parent, maintains a close circle of trusted friends\n Life Experiences: Became a single parent early and learned to balance career and parenting meticulously.\n", 11 | "9": "Personality: Optimistic and sentimental\n Profession: Event planner\n Hobbies: Organizing social gatherings and scrapbooking\n Social Relationships: Has a small circle of close friends, values deep connections\n Life Experiences: Planned a successful community festival that brought local businesses together.\n", 12 | "10": "Personality: Family-oriented and sentimental\n Profession: Elementary school principal\n Hobbies: Gardening and scrapbooking\n Social Relationships: Close to extended family, cherishes relationships with siblings\n Life Experiences: Worked her way up from a teacher to a principal, actively involved in community development.\n", 13 | "11": "Personality: Resilient and lonely\n Profession: Military spouse, possibly with a part-time job or volunteer role\n Hobbies: Watching TV, socializing with friends\n Social Relationships: Married to a military spouse, distant from family geographically\n Life Experiences: Handles the challenges of having a spouse frequently deployed abroad, moved away from family.\n", 14 | "12": "Personality: Empathetic and sensitive\n Profession: Social Worker\n Hobbies: Reading and yoga\n Social Relationships: Single, maintains a small circle of close friends\n Life Experiences: Witnessed a traumatic public event, which reinforced their dedication to community support and mental health advocacy.\n", 15 | "13": "Personality: Politically engaged and confrontational\n Profession: Political analyst\n Hobbies: Reading political biographies and participating in debates\n Social Relationships: Single, frequently debates with friends and colleagues\n Life Experiences: Became deeply involved in politics after witnessing electoral malpractice.\n", 16 | "14": "Personality: Open-minded and reserved\n Profession: Librarian\n Hobbies: Reading and gardening\n Social Relationships: Has a strained relationship with parents due to differing beliefs, generally keeps a small circle of close friends\n Life Experiences: Moved to a new city to escape a conservative hometown and find a more accepting community.\n", 17 | "15": "Personality: Empathetic and pessimistic\n Profession: Funeral Director\n Hobbies: Gardening and reading historical novels\n Social Relationships: Widowed, maintains close relationships with extended family\n Life Experiences: Lost multiple family members to illness, developed a deep understanding of grief and mourning.\n", 18 | "16": "Personality: Meticulous and anxious\n Profession: Event Planner\n Hobbies: Traveling and reading\n Social Relationships: In a long-distance relationship\n Life Experiences: Studied abroad during college, leading to a passion for different cultures and settings.\n", 19 | "17": "Personality: Anxious and honest\n Profession: Graphic designer\n Hobbies: Digital art and video gaming\n Social Relationships: In a relationship, has a small but close group of friends\n Life Experiences: Recently moved to a new city for job opportunities.\n", 20 | "18": "Personality: Compassionate and independent\n Profession: Wildlife Rehabilitator\n Hobbies: Birdwatching and hiking\n Social Relationships: Single, maintains a close relationship with a small group of like-minded friends\n Life Experiences: Grew up in a rural area where they developed a passion for animals, especially birds, learned wildlife rescue skills early in life.\n", 21 | "19": "Personality: Compassionate and nurturing\n Profession: Wildlife rehabilitator\n Hobbies: Gardening and wildlife photography\n Social Relationships: Single, maintains a strong sense of community with local environmental groups\n Life Experiences: Started a local wildlife rescue center after witnessing the impact of urban expansion on native species.\n", 22 | "20": "Personality: Nostalgic and tech-savvy\n Profession: Software Developer\n Hobbies: Retro video gaming and technology tinkering\n Social Relationships: Close to his brother, enjoys connecting with others who have similar interests\n Life Experiences: Grew up playing video games, which sparked his interest in technology and led to a career in software development.\n", 23 | "21": "Personality: Anxious and resilient\n Profession: Pediatric nurse\n Hobbies: Gardening and reading\n Social Relationships: Married, mother of one\n Life Experiences: Endured a high-risk pregnancy which involved emergency surgery, resulting in a healthy childbirth.\n", 24 | "22": "Personality: Frugal and optimistic\n Profession: Retail Store Manager\n Hobbies: Car restoration and hiking\n Social Relationships: In a long-term relationship, no children\n Life Experiences: Worked multiple jobs to pay through college, recently promoted to manager\n", 25 | "23": "Personality: Cautious and humorous\n Profession: Farmer\n Hobbies: Beekeeping and gardening\n Social Relationships: Married, often collaborates with fellow farmers\n Life Experiences: Suffered a severe allergic reaction to a wasp sting, which has made him more cautious around insects.\n", 26 | "24": "Personality: Supportive and perfectionistic\n Profession: College professor\n Hobbies: Reading historical novels and gardening\n Social Relationships: Divorced, maintains a strong bond with his daughter\n Life Experiences: Worked his way through school to earn his PhD, achieved tenure after years of academic dedication.\n", 27 | "25": "Personality: Perseverant and inconsistent\n Profession: Retail store manager\n Hobbies: Reading and attending night classes\n Social Relationships: In a long-term relationship\n Life Experiences: Started college in 2011, became serious about it in 2016, balancing work, education, and personal life.\n", 28 | "26": "Personality: Nostalgic and familial\n Profession: Wildlife biologist\n Hobbies: Fishing and hiking\n Social Relationships: Married, recently became a father\n Life Experiences: Grew up fishing with his father, inspired his career in environmental science.\n", 29 | "27": "Personality: Supportive and optimistic\n Profession: Social worker\n Hobbies: Reading self-help books and gardening\n Social Relationships: Married, supportive of her husband during his job search\n Life Experiences: Studied psychology to better assist individuals coping with challenges.\n", 30 | "28": "Personality: Ambitious and anxious\n Profession: Marketing Specialist\n Hobbies: Reading and networking\n Social Relationships: Recently relocated, trying to build new friendships\n Life Experiences: Successfully changed careers after realizing her previous job didn't align with her passion.\n", 31 | "29": "Personality: Empathetic but irritable\n Profession: Graphic Designer\n Hobbies: Digital art and reading fantasy novels\n Social Relationships: Lives alone, has a close-knit group of friends\n Life Experiences: Helped a friend through a tough financial period.\n", 32 | "30": "Personality: Optimistic and materialistic\n Profession: Real Estate Developer\n Hobbies: Architecture and interior design\n Social Relationships: Recently remarried\n Life Experiences: Experienced financial instability during early career, subsequently achieved significant success in building a profitable real estate portfolio.\n", 33 | "31": "Personality: Empathetic, but somewhat insensitive\n Profession: Radiation oncologist\n Hobbies: Pet care and humor\n Social Relationships: Seems socially active but possibly lacks deep understanding in some personal interactions\n Life Experiences: Daily encounters with severe illnesses, possibly leading to a somewhat desensitized view on less severe conditions.\n", 34 | "32": "Personality: Generous and neighborly\n Profession: Landscaper\n Hobbies: Gardening and woodworking\n Social Relationships: Part of a tight-knit community, often helps neighbors\n Life Experiences: Learned landscaping skills from a mentor, frequently volunteers for community cleanup events\n", 35 | "33": "Personality: Forgetful and optimistic\n Profession: Graphic designer\n Hobbies: Watching movies and sketching\n Social Relationships: In a long-term relationship, has a small circle of close friends\n Life Experiences: Once misplaced a crucial project file but managed to recreate it overnight, earning praise for resilience and creativity.\n", 36 | "34": "Personality: Humorous and persistent\n Profession: Freelance writer\n Hobbies: Reading and blogging\n Social Relationships: Married with a young child\n Life Experiences: Struggled with balancing career and parenting responsibilities, especially with childcare challenges.\n", 37 | "35": "Personality: Cautious and protective\n Profession: College student\n Hobbies: Blogging and reading\n Social Relationships: Close with her mother, few close friends\n Life Experiences: Experienced stalking, leading to increased awareness about personal safety.\n", 38 | "36": "Personality: Empathetic and sentimental\n Profession: Social worker\n Hobbies: Gardening and reading\n Social Relationships: Close relationship with mother, father deceased\n Life Experiences: Deeply affected by the loss of a parent, values family connections\n", 39 | "37": "Personality: Reflective and sentimental\n Profession: Retired teacher\n Hobbies: Birdwatching and gardening\n Social Relationships: Widowed, maintains a close relationship with grandchildren\n Life Experiences: Devoted life to education and raising a family, enjoys the peaceful retirement years appreciating nature and simplicity.\n", 40 | "38": "Personality: Meticulous and self-critical\n Profession: Financial Analyst\n Hobbies: Collecting vintage cars and photography\n Social Relationships: In a long-term relationship, has a small circle of close friends\n Life Experiences: Faced significant financial loss in the past, which now drives their cautious approach in both professional and personal investments.\n", 41 | "39": "Personality: Socially anxious and sensitive\n Profession: Freelance graphic designer\n Hobbies: Watching movies and reading comics\n Social Relationships: Lives with parents, has few close friends\n Life Experiences: Struggles with personal independence and transportation issues due to anxiety.\n", 42 | "40": "Personality: Principled and proactive\n Profession: College student, possibly studying law or ethics\n Hobbies: Academic research and debate\n Social Relationships: Trusts authority but has faced betrayal\n Life Experiences: Experienced plagiarism of her work by a peer, which challenged her notion of fairness in academic settings\n", 43 | "41": "Personality: Social and optimistic\n Profession: Graphic designer\n Hobbies: Hiking and painting\n Social Relationships: Has a supportive group of friends\n Life Experiences: Endured living in a noisy and disruptive environment, leading to a recent move for a more peaceful setting.\n", 44 | "42": "Personality: Forgetful and cheerful\n Profession: Stay-at-home parent\n Hobbies: Gardening and crafting\n Social Relationships: Married with a son\n Life Experiences: Transitioned from a busy career to focus on family and community involvement, balancing parenting with personal hobbies and volunteering at her son¡¯s Scouts and soccer activities.\n", 45 | "43": "Personality: Emotional and compassionate\n Profession: Veterinarian\n Hobbies: Animal photography and hiking\n Social Relationships: Recently widowed\n Life Experiences: Started an animal rescue center after her own pet's death.\n", 46 | "44": "Personality: Inventive and sentimental\n Profession: Game developer\n Hobbies: Building arcade machines and reminiscing\n Social Relationships: Values old friendships, possibly single\n Life Experiences: Collaborated on creative projects during college, cherishes shared memories with friends.\n", 47 | "45": "Personality: Cautious and anxious\n Profession: Traveling sales representative\n Hobbies: Driving and listening to podcasts\n Social Relationships: In a long-distance relationship\n Life Experiences: Survived a major car accident in the past, influencing current fears and caution on the road.\n", 48 | "46": "Personality: Adventurous and optimistic\n Profession: Outdoor adventure guide\n Hobbies: Camping and hiking\n Social Relationships: Single, enjoys solitude and the company of friends during adventures\n Life Experiences: Survived a challenging wilderness survival course.\n", 49 | "47": "Personality: Adventurous and caring\n Profession: Middle school science teacher\n Hobbies: Water sports and hiking\n Social Relationships: Married with two children\n Life Experiences: Changed careers from a corporate job to teaching to have more time with family.\n", 50 | "48": "Personality: Quirky and overconfident\n Profession: Vintage shop owner\n Hobbies: Collecting vintage memorabilia and conspiracy theory research\n Social Relationships: Single, enjoys a large circle of eclectic friends\n Life Experiences: Grew up during the cold war era, which heavily influenced his interest in historical artifacts and protective measures from that period.\n", 51 | "49": "Personality: Ambitious and self-reliant\n Profession: Software engineer\n Hobbies: Solving puzzles and reading tech journals\n Social Relationships: In a long-distance relationship, small circle of close friends\n Life Experiences: Moved to a new city alone to pursue career opportunities, successfully passed tough certification exams after self-study.\n", 52 | "50": "Personality: Creative and secretive\n Profession: Event planner\n Hobbies: Traveling and photography\n Social Relationships: Married, enjoys bringing together distant friends and family\n Life Experiences: Mastered the art of surprise events despite initial failures in earlier career stages.\n", 53 | "51": "Personality: Empathetic and sentimental\n Profession: Grief counselor\n Hobbies: Photography and gardening\n Social Relationships: Has a close-knit family who supports each other\n Life Experiences: Also experienced the loss of a pet, which deepened their understanding of grief and loss.\n", 54 | "52": "Personality: Cynical and straightforward\n Profession: Office manager\n Hobbies: Watching sports and reading mystery novels\n Social Relationships: Married with frequent disagreements\n Life Experiences: Tried couples therapy unsuccessfully, became more disillusioned with relationships\n", 55 | "53": "Personality: Empathetic and anxious\n Profession: Pediatric nurse\n Hobbies: Gardening and reading fiction\n Social Relationships: Married, mother of a young child\n Life Experiences: Managed anxiety issues while balancing a demanding job and family life.\n", 56 | "54": "Personality: Playful and indifferent\n Profession: Customer Support Representative\n Hobbies: Binge-watching TV series and playing online puzzle games\n Social Relationships: In a casual relationship, distant from most family members\n Life Experiences: Originally started college but dropped out due to lack of interest; found stability in current job.\n", 57 | "55": "Personality: Ambitious and meticulous\n Profession: Academic Researcher\n Hobbies: Reading historical novels and gardening\n Social Relationships: Single, has a few close colleagues\n Life Experiences: Broke new ground in her field, earning a prestigious award previously unattained in her college history.\n", 58 | "56": "Personality: Compassionate and outspoken\n Profession: Volunteer coordinator for an animal rescue organization\n Hobbies: Volunteering at animal shelters and reading\n Social Relationships: Has many close friendships, supportive network in the animal welfare community\n Life Experiences: Adopted a rescue dog from a puppy mill, actively campaigns against animal cruelty\n", 59 | "57": "Personality: Optimistic and persistent\n Profession: Retired professional golfer\n Hobbies: Golf and physical therapy exercises\n Social Relationships: Close with family members, particularly a nephew\n Life Experiences: Suffered a significant hip injury impacting mobility and career, actively working on rehabilitation to regain active lifestyle.\n", 60 | "58": "Personality: Meticulous and easily frustrated\n Profession: Accountant\n Hobbies: Collecting receipts and scrapbooking\n Social Relationships: Divorced, has a daughter involved in school band activities\n Life Experiences: Learned to keep thorough financial records after previous mishaps with personal accounting.\n", 61 | "59": "Personality: Supportive and optimistic\n Profession: Guidance Counselor\n Hobbies: Yoga and reading self-help books\n Social Relationships: In a stable relationship, has a wide circle of friends\n Life Experiences: Helped a neighborhood community center avoid closure.\n", 62 | "60": "Personality: Competitive, but self-critical\n Profession: University professor\n Hobbies: Trivia games and reading\n Social Relationships: Single, maintains a small close-knit group of friends\n Life Experiences: Achieved academic success, struggles with work-life balance due to competitive nature\n", 63 | "61": "Personality: Curious and unconventional\n Profession: Blogger\n Hobbies: Experimenting with life hacks and reading self-help books\n Social Relationships: In a long-term relationship, often shares quirky tips with friends\n Life Experiences: Used unique methods to improve personal habits and lifestyle efficiency after struggling with procrastination and inefficiency.\n", 64 | "62": "Personality: Hardworking and introverted\n Profession: Software Developer\n Hobbies: Coding and watching movies\n Social Relationships: Part of a close-knit group of friends, often misses out on social events due to work commitments\n Life Experiences: Worked freelance for several years before landing a stable job at a tech firm, often sacrifices personal time for career advancement\n", 65 | "63": "Personality: Sentimental and reflective\n Profession: Music therapist\n Hobbies: Listening to nature sounds, collecting vinyl records\n Social Relationships: In a committed relationship, enjoys a close bond with their grandparents\n Life Experiences: Grew up in a rural area with an old house, uses personal serene experiences to help others in therapy.\n", 66 | "64": "Personality: Family-oriented and sentimental\n Profession: Restaurant owner\n Hobbies: Cooking and hosting family gatherings\n Social Relationships: Married with children, maintains strong family ties\n Life Experiences: Started the restaurant as a small family business, which has grown into a well-loved community staple.\n", 67 | "65": "Personality: Responsible, but resentful\n Profession: Human Resources Manager\n Hobbies: Planning family trips and reading\n Social Relationships: Married, has children\n Life Experiences: Faced frequent disruptions in personal life due to demanding job responsibilities\n", 68 | "66": "Personality: Nurturing and protective, but overly anxious\n Profession: Creative writer\n Hobbies: Gardening and reading classic literature\n Social Relationships: Widowed, very close to her daughter\n Life Experiences: Published several acclaimed novels but struggles with the fear of losing loved ones.\n", 69 | "67": "Personality: Nurturing and optimistic\n Profession: Interior designer\n Hobbies: DIY home decor and gardening\n Social Relationships: Married, expecting first child\n Life Experiences: Recently bought and renovated their dream home.\n", 70 | "68": "Personality: Pragmatic and frugal\n Profession: Accountant\n Hobbies: Gardening and reading finance books\n Social Relationships: Married, values a stable home life\n Life Experiences: Worked multiple jobs while in college to pay tuition, always lived within means\n", 71 | "69": "Personality: Content and realistic\n Profession: Event planner\n Hobbies: Traveling and gardening\n Social Relationships: Recently remarried, appreciates simplicity and shared interests in relationships\n Life Experiences: Previously married and divorced, has learned the importance of shared hobbies and managing expectations in relationships.\n", 72 | "70": "Personality: Organized and adventurous\n Profession: Travel blogger\n Hobbies: Photography and hiking\n Social Relationships: Single, loves meeting new people on her travels\n Life Experiences: Decided to travel full-time after losing her corporate job\n", 73 | "71": "Personality: Adventurous and sensitive\n Profession: Film critic\n Hobbies: Watching movies and writing\n Social Relationships: Single, has a close-knit group of friends who share his passion for films\n Life Experiences: Started a popular film review blog that gained media attention\n", 74 | "72": "Personality: Anxious and imaginative\n Profession: Urban Planner\n Hobbies: Reading and urban exploration\n Social Relationships: Has a few close friends, no significant other\n Life Experiences: Grew up in a large city, witnessed crime and redevelopment first-hand\n", 75 | "73": "Personality: Affectionate and sociable\n Profession: Veterinarian\n Hobbies: Animal rescue and photography\n Social Relationships: Has a significant other, very active in community pet groups\n Life Experiences: Grew up on a farm, fostering animals since childhood.\n", 76 | "74": "Personality: Anxious and meticulous\n Profession: Graphic Designer\n Hobbies: Painting and photography\n Social Relationships: In a long-term relationship, but often struggles with self-doubt\n Life Experiences: Lost a previous job due to company downsizing, inspiring a career shift to graphic design\n", 77 | "75": "Personality: Prudent and resourceful\n Profession: Roadside assistance technician\n Hobbies: DIY car repairs and winter sports\n Social Relationships: Has a significant other who also enjoys outdoor activities\n Life Experiences: Previously stranded in a snowstorm without emergency supplies, now always prepared for winter emergencies.\n", 78 | "76": "Personality: Irritable and meticulous\n Profession: Financial analyst\n Hobbies: Model building and gourmet cooking\n Social Relationships: In a relationship, generally holds a grudge over small mistakes\n Life Experiences: Transitioned from a strict corporate job to a more flexible work environment, striving for perfection in personal and professional life.\n", 79 | "77": "Personality: Mischievous and impulsive\n Profession: Graphic designer\n Hobbies: Urban exploring and digital art\n Social Relationships: Lives with roommates, often in minor conflicts over personal space and property\n Life Experiences: Moved to a big city from a small town to pursue a creative career, learning to navigate urban life and professional challenges.\n", 80 | "78": "Personality: Resilient and hopeful\n Profession: Recently employed, previously long-term unemployed\n Hobbies: Unknown\n Social Relationships: Married, has experienced financial stability challenges with spouse\n Life Experiences: Faced a challenging period of unemployment after moving to a new location, currently in a phase of recovery and gradual improvement in life circumstances.\n", 81 | "79": "Personality: Caring but anxious\n Profession: Retired nurse\n Hobbies: Gardening and reading\n Social Relationships: Widowed, has a distant but loving relationship with her daughter\n Life Experiences: Spent years caring for others in her professional life, and dealt with the loss of her spouse.\n", 82 | "80": "Personality: Materialistic and confident\n Profession: Marketing Executive\n Hobbies: Shopping and fashion blogging\n Social Relationships: Single, enjoys socializing with a large group of friends\n Life Experiences: Climbed corporate ladder quickly due to successful advertising campaigns.\n", 83 | "81": "Personality: Grateful and pragmatic\n Profession: Elementary school teacher\n Hobbies: Bargain hunting and crafting\n Social Relationships: Married with children, maintains a positive relationship with in-laws\n Life Experiences: Relies on familial and community support to manage financial challenges.\n", 84 | "82": "Personality: Cautious and skeptical\n Profession: Fleet manager\n Hobbies: Model car collecting and reading automotive magazines\n Social Relationships: In a long-term relationship, no children\n Life Experiences: Recently promoted, saved enough to consider a significant personal investment like buying a new car.\n", 85 | "83": "Personality: Nostalgic and sentimental\n Profession: Middle school teacher\n Hobbies: Scrapbooking and gardening\n Social Relationships: Married with three children\n Life Experiences: Transitioned from a career in corporate sales to education after having children to achieve a better work-life balance and to be more involved in their lives.\n", 86 | "84": "Personality: Nostalgic and passionate\n Profession: Software developer\n Hobbies: Collecting vintage video games, playing PC games\n Social Relationships: In a relationship, has a small but close group of friends\n Life Experiences: Learned programming by modifying old video games, leading to a career in tech.\n", 87 | "85": "Personality: Empathetic and sensitive\n Profession: Social worker\n Hobbies: Gardening and reading\n Social Relationships: Married, treasured her relationship with her husband's grandmother\n Life Experiences: Experienced significant loss with the sudden passing of a loved one, finds solace in maintaining close family ties.\n", 88 | "86": "Personality: Anxious and resilient\n Profession: Financial Analyst\n Hobbies: Reading and yoga\n Social Relationships: In a long-term relationship\n Life Experiences: Struggled with anxiety but succeeded in securing a desired job position despite fears and self-doubt.\n", 89 | "87": "Personality: Assertive and irritable\n Profession: Office Manager\n Hobbies: Gardening and reading\n Social Relationships: Married, longer tenure at work than many colleagues\n Life Experiences: Faced repeated challenges asserting authority in a competitive work environment.\n", 90 | "88": "Personality: Health-conscious and anxious\n Profession: Cardiologist\n Hobbies: Yoga and reading medical journals\n Social Relationships: Single, maintains a distance from most relatives\n Life Experiences: Battled with minor health scares that profoundly impacted her perspective on wellness and patient care.\n", 91 | "89": "Personality: Rebellious and thrill-seeking\n Profession: Street racer\n Hobbies: Street racing and watching professional races\n Social Relationships: Single, connects mainly with fellow racers\n Life Experiences: Became involved in street racing as a challenge, honing skills mostly self-taught from observing races.\n", 92 | "90": "Personality: Anxious and persistent\n Profession: Elementary school teacher\n Hobbies: Reading and gardening\n Social Relationships: Single, active in community groups\n Life Experiences: Attempted various methods to overcome her arachnophobia without success.\n", 93 | "91": "Personality: Nostalgic and sentimental\n Profession: Librarian\n Hobbies: Reading historical novels and scrapbooking\n Social Relationships: Single, maintains a small but close group of friends\n Life Experiences: Moved frequently during childhood due to a parent's job, leading to many short-term friendships.\n", 94 | "92": "Personality: Diligent and optimistic\n Profession: College student studying chemistry\n Hobbies: Reading scientific journals and playing strategic board games\n Social Relationships: Single, close friendships within the university\n Life Experiences: Overcame learning disabilities to excel academically.\n", 95 | "93": "Personality: Sentimental and adventurous\n Profession: Travel blogger\n Hobbies: Trying different local cuisines and hiking\n Social Relationships: In a long-distance relationship\n Life Experiences: Grew up in a warm climate with rare cold weather experiences, cherishing unique weather-related moments.\n", 96 | "94": "Personality: Clumsy and self-conscious\n Profession: Office worker\n Hobbies: Reading and trying out new restaurants\n Social Relationships: Single, maintains a close-knit group of friends\n Life Experiences: Frequently finds themselves in embarrassing situations yet manages to laugh them off.\n", 97 | "95": "Personality: Patient but anxious\n Profession: Financial Analyst\n Hobbies: Yoga and reading financial news\n Social Relationships: In a relationship, has a supportive group of friends\n Life Experiences: Had his identity stolen which led to a financial setback.\n", 98 | "96": "Personality: Ambitious and anxious\n Profession: Law student (aspiring lawyer)\n Hobbies: Reading legal journals and participating in debate clubs\n Social Relationships: Single, maintains a close circle of academically driven friends\n Life Experiences: Successfully interned at a prestigious law firm, finding mentorship that enhanced her career path.\n", 99 | "97": "Personality: Anxious and meticulous\n Profession: Medical student\n Hobbies: Reading medical journals and practicing yoga\n Social Relationships: In a committed relationship, but often too busy to spend much time together\n Life Experiences: Struggled with anxiety throughout school but excelled academically after seeking professional help\n", 100 | "98": "Personality: Cultured and enthusiastic\n Profession: Theater critic\n Hobbies: Attending theater performances and reading\n Social Relationships: In a long-term relationship, has many acquaintances in the theater world\n Life Experiences: Studied drama in college, worked as a stage manager before becoming a critic\n", 101 | "99": "Personality: Anxious and self-conscious\n Profession: Graphic designer\n Hobbies: Dog walking and watching horror movies\n Social Relationships: Lives alone, has a small but close group of friends\n Life Experiences: Once moved to a new city for a job opportunity, which didn't work out, leading to issues with anxiety.\n" 102 | } --------------------------------------------------------------------------------