├── LICENSE ├── Mermaid_KnowledgeGraph_Implementation └── toolkit │ ├── DataSet_Augmentation_Toolkit_InUse.png │ ├── Entries │ ├── entry_1_1.png │ ├── entry_1_2.png │ ├── entry_1_3.png │ ├── entry_1_4.png │ ├── entry_1_5.png │ └── entry_1_6.png │ ├── OpenAI_Compatible_temperature_diversity_dataset_toolkit.py │ ├── README.md │ ├── output.json │ └── prompt.txt ├── Meta-A-Wrapper-Around-OpenAI.mov ├── README.md └── opera_BDKLCRC9IW.gif /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/DataSet_Augmentation_Toolkit_InUse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/DataSet_Augmentation_Toolkit_InUse.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_1.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_2.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_3.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_4.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_5.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Mermaid_KnowledgeGraph_Implementation/toolkit/Entries/entry_1_6.png -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/OpenAI_Compatible_temperature_diversity_dataset_toolkit.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import os 4 | import requests 5 | import subprocess 6 | import tempfile 7 | 8 | ##### MermaidDiagramGenerator Class ##### 9 | class MermaidDiagramGenerator: 10 | def __init__(self, theme='dark', background='transparent'): 11 | self._theme = theme 12 | self._background = background 13 | self._entries_dir = os.path.join(os.getcwd(), 'Entries') 14 | os.makedirs(self._entries_dir, exist_ok=True) 15 | 16 | def convert_to_image(self, mermaid_code, entry_number, output_number): 17 | clean_code = self._remove_mermaid_block_markers(mermaid_code) 18 | output_filename = f"entry_{entry_number}_{output_number}.png" 19 | output_path = os.path.join(self._entries_dir, output_filename) 20 | self._generate_image_from_code(clean_code, output_path) 21 | return output_path 22 | 23 | def _remove_mermaid_block_markers(self, code): 24 | code_lines = code.strip().splitlines() 25 | if code_lines[0].startswith("```mermaid") and code_lines[-1] == "```": 26 | return "\n".join(code_lines[1:-1]).strip() 27 | return code 28 | 29 | def _generate_image_from_code(self, mermaid_code, output_path): 30 | with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file: 31 | temp_file.write(mermaid_code) 32 | input_path = temp_file.name 33 | result = subprocess.run(["mmdc", "-i", input_path, "-o", output_path, "-t", self._theme, "-b", self._background], shell=True, check=False) 34 | os.remove(input_path) 35 | if result.returncode != 0: 36 | raise ValueError("Mermaid diagram generation failed.") 37 | 38 | ##### Script to Generate Responses and Validate Mermaid Diagrams ##### 39 | def read_input(input_source): 40 | if os.path.isfile(input_source): 41 | filename, file_extension = os.path.splitext(input_source) 42 | if file_extension == '.json': 43 | with open(input_source, 'r') as file: 44 | return json.load(file) 45 | elif file_extension == '.txt': 46 | with open(input_source, 'r') as file: 47 | return [{"input": file.read()}] 48 | else: 49 | return [{"input": input_source}] 50 | 51 | def generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs): 52 | prompt_template = f"{prompt}\n\n```mermaid\n" 53 | 54 | url = "http://127.0.0.1:5000/v1/completions" 55 | headers = {"Content-Type": "application/json"} 56 | dataset_entries = [] 57 | 58 | for output_number, temp in enumerate(base_temperatures, start=1): 59 | while True: 60 | data = { 61 | "prompt": prompt_template, 62 | "max_tokens": 4096, 63 | "temperature": temp, 64 | "top_p": 1.0, 65 | "seed": -1, 66 | "top_k": 4, 67 | "repetition_penalty": 1.0, 68 | "guidance_scale": 1.0, 69 | "typical_p": 1.0, 70 | "stream": stream, 71 | } 72 | 73 | response = requests.post(url, headers=headers, json=data, verify=False) 74 | response_text = response.json()['choices'][0]['text'].strip() 75 | 76 | if response_text.endswith("```"): # Check if response ends with ``` 77 | response_text = response_text[:-3].strip() # Remove ``` from the end 78 | 79 | if response_text not in unique_outputs: 80 | try: 81 | image_path = generator.convert_to_image(response_text, entry_number, output_number) 82 | print(f"Mermaid diagram generated at: {image_path}") 83 | unique_outputs.add(response_text) 84 | break 85 | except ValueError as e: 86 | print(f"Validation failed, retrying... Error: {e}") 87 | else: 88 | temp += 0.1 # Adjust temperature if output is not unique 89 | 90 | dataset_entry = { 91 | "input": prompt, 92 | "output": f"```mermaid\n{response_text}\n```", 93 | "temperature": temp 94 | } 95 | dataset_entries.append(dataset_entry) 96 | 97 | return dataset_entries 98 | 99 | def generate_unique_responses(input_data, base_temperatures, stream, generator): 100 | all_entries = [] 101 | unique_outputs = set() 102 | 103 | for entry_number, entry in enumerate(input_data, start=1): 104 | prompt = entry.get("input", "") 105 | if prompt: 106 | entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs) 107 | all_entries.extend(entries) # Extend the list with new entries 108 | 109 | return all_entries 110 | 111 | def main(input_source, stream=False): 112 | generator = MermaidDiagramGenerator() 113 | input_data = read_input(input_source) 114 | base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input 115 | output_file = "output.json" 116 | 117 | all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator) 118 | 119 | # Write all entries to the JSON file at once 120 | with open(output_file, "w") as f: 121 | json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file 122 | 123 | if __name__ == "__main__": 124 | parser = argparse.ArgumentParser(description="Generate unique responses and validate Mermaid diagrams.") 125 | parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.') 126 | parser.add_argument('--stream', action='store_true', help='Use streaming responses.') 127 | args = parser.parse_args() 128 | 129 | main(args.input_source, args.stream) 130 | -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/README.md: -------------------------------------------------------------------------------- 1 | # Language Model Diversification Toolkit 2 | 3 | ## Overview 4 | The Language Model Diversification Toolkit empowers the development of more adaptable and robust language models by facilitating the creation of diversified training datasets. Leveraging advanced temperature-controlled text generation and incorporating mechanisms for the validation of complex data structures such as Mermaid diagrams, this toolkit is pivotal in enhancing model generalization capabilities across a broad spectrum of NLP tasks. 5 | 6 | ## Key Features 7 | - **Temperature-Controlled Text Generation**: Dynamically generates text over a range of temperatures, ensuring a richly diverse dataset. 8 | - **Mermaid Diagram Integration**: Validates and generates Mermaid diagrams, adding a structural and visual dimension to text-based datasets. 9 | - **Unique Output Assurance**: Implements retry mechanisms to guarantee the uniqueness of generated outputs, thereby improving dataset quality and consistency. 10 | - **Incremental Dataset Construction**: Writes outputs to the dataset file in real-time, ensuring no data loss and enabling immediate review and use of generated data. 11 | 12 | ## Installation 13 | This toolkit requires Python 3.8 or later and npm for installing the Mermaid CLI tool. Follow these steps to set up your environment: 14 | 15 | 1. **Clone the Repository**: 16 | ```bash 17 | git clone 18 | ``` 19 | 20 | 2. **Install Python Dependencies**: 21 | Navigate to the toolkit's directory and install the required Python packages: 22 | ``` 23 | requests==2.25.1 24 | argparse==1.4.0 25 | ``` 26 | 27 | 3. **Install Mermaid CLI**: 28 | The Mermaid CLI tool is necessary for diagram generation and validation: 29 | ```bash 30 | npm install -g @mermaid-js/mermaid-cli 31 | ``` 32 | 33 | ## Usage 34 | Execute the toolkit script with your chosen input source. The toolkit accepts inputs in various formats: plain text, `.txt`, or `.json` files containing multiple prompts. Optional streaming can be enabled for real-time response generation. 35 | 36 | ```bash 37 | python main_script.py "path/to/your_input_file.json" 38 | ``` 39 | 40 | ```bash 41 | python main_script.py "path/to/your_input_file.txt" 42 | ``` 43 | 44 | ```bash 45 | python main_script.py "input as plain text" 46 | ``` 47 | 48 | Outputs, including diversified text entries and associated Mermaid diagrams (if enabled), are saved in an `output.json` file and /Entries. 49 | 50 | ## Contribution to NLP Research and Development 51 | The Language Model Diversification Toolkit is designed not just as a utility for dataset enhancement but as a contribution to the broader NLP research community. It embodies the practical application of theoretical insights from cognitive linguistics and computational modeling, providing a scalable solution for one of the most pressing challenges in NLP: developing language models that truly understand and adapt to the dynamic nature of human language. -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/output.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 4 | "output": "```mermaid\ngraph TD\n A[Start] --> B[Initialize MermaidDiagramGenerator]\n B --> C[Read Input Data]\n C --> D{Input is File?}\n D --> |Yes| E[Load JSON Data]\n D --> |No| F[Create Single Prompt Entry]\n \n E --> G[Generate Response for Each Prompt]\n F --> G\n \n subgraph Generate Response\n direction TB\n \n G --> H[Get Base Temperatures]\n H --> I[Loop Through Each Temperature]\n I --> J[Send Request & Get Response]\n J --> K{Response Unique?}\n K --> |Yes| L[Add Entry to List]\n K --> |No| M[Increment Temperature]\n M --> I\n \n L --> N[Write Mermaid Image if PNG]\n N --> O[Return Entry List]\n \n end\n \n O --> P[Write Entries to JSON File]\n P --> Q[End]\n```", 5 | "temperature": 0.5 6 | }, 7 | { 8 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 9 | "output": "```mermaid\ngraph TD\n A[Start] --> B[Define MermaidDiagramGenerator Class]\n\n B --> C[Define _init_ method]\n C --> D[Initialize theme and background attributes]\n C --> E[Initialize entries directory path]\n\n B --> F[Define convert_to_image method]\n F --> G[Remove Mermaid code block markers]\n G --> H[Generate image using mmdc]\n H --> I{Check for errors}\n I --> J[Return image path]\n\n C --> K[Define generate_response method]\n K --> L{For each base temperature}\n L --> M[Send prompt with temperature]\n M --> N[Get response text]\n N --> O{Response ends with ```?}\n O --> P[Remove code block markers]\n P --> Q[Check if output is unique]\n Q --> R[Add output to dataset entry]\n R --> S[Add dataset entry to list]\n S --> T{Base temperatures exhausted?}\n T --> U[Adjust temperature and retry]\n U --> L\n\n K --> V[Define generate_unique_responses method]\n V --> W[Read input source]\n W --> X[Loop through prompts]\n X --> Y[Generate unique responses]\n Y --> Z[Write output to JSON file]\n\n Z --> AA[End]\n```", 10 | "temperature": 0.6 11 | }, 12 | { 13 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 14 | "output": "```mermaid\ngraph TD\n A[Start] --> B[Parse Arguments]\n B --> C{Input Source}\n C --> |.json file| D[Read JSON File]\n C --> |.txt file| E[Read Text File]\n C --> |String| F[Create Entry]\n D --> G[Generate Responses]\n E --> G\n F --> G\n \n subgraph Generate Responses\n direction TB\n \n G --> H{For Each Entry}\n H -->|No| I[Done]\n H -->|Yes| J[Get Base Temperatures]\n J --> K[Call Generate Response]\n \n subgraph Generate Response\n direction TB\n \n K --> L[Post Request]\n L --> M[Get Response]\n M --> N{Output Valid?}\n N --> |No| O[Increment Temperature]\n O --> K\n N --> |Yes| P[Save Output]\n \n end\n \n P --> Q[Add Entry to List]\n Q --> H\n \n end\n \n I --> R[Write Entries to File]\n R --> S[End]\n```", 15 | "temperature": 0.7 16 | }, 17 | { 18 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 19 | "output": "```mermaid\ngraph TB;\n A[Start] --> B[Define MermaidDiagramGenerator Class];\n B --> C[Define convert_to_image method];\n C --> D[Define _remove_mermaid_block_markers helper method];\n D --> E[Define _generate_image_from_code method];\n E --> F[Define generate_response method];\n F --> G[Read input source argument];\n G --> H[Generate unique responses for each input];\n H --> I[Write all entries to JSON file];\n I --> J[Main entry point];\n J --> K[Parse input source argument];\n K --> L[Call main function];\n L --> M[End];\n```", 20 | "temperature": 0.8 21 | }, 22 | { 23 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 24 | "output": "```mermaid\ngraph TD\n A[Start Script] --> B[Define MermaidDiagramGenerator Class]\n B --> C[Define read_input function]\n C --> D[Define generate_response function]\n D --> E[Define generate_unique_responses function]\n E --> F[Define main function]\n \n F --> G[Initialize MermaidDiagramGenerator object]\n G --> H[Read input source]\n H --> I{Input file or string?}\n I --> |Input file| J[Load JSON input]\n J --> K[Generate responses]\n I --> |Input string| L[Convert input string to dataset entry]\n L --> K\n \n K --> M[Write output JSON file]\n M --> N[End Script]\n```", 25 | "temperature": 0.9 26 | }, 27 | { 28 | "input": "import argparse\nimport json\nimport os\nimport requests\nimport subprocess\nimport tempfile\n\n##### MermaidDiagramGenerator Class #####\nclass MermaidDiagramGenerator:\n def __init__(self, theme='dark', background='transparent'):\n self._theme = theme\n self._background = background\n self._entries_dir = os.path.join(os.getcwd(), 'Entries')\n os.makedirs(self._entries_dir, exist_ok=True)\n\n def convert_to_image(self, mermaid_code, entry_number, output_number):\n clean_code = self._remove_mermaid_block_markers(mermaid_code)\n output_filename = f\"entry_{entry_number}_{output_number}.png\"\n output_path = os.path.join(self._entries_dir, output_filename)\n self._generate_image_from_code(clean_code, output_path)\n return output_path\n\n def _remove_mermaid_block_markers(self, code):\n code_lines = code.strip().splitlines()\n if code_lines[0].startswith(\"```mermaid\") and code_lines[-1] == \"```\":\n return \"\\n\".join(code_lines[1:-1]).strip()\n return code\n\n def _generate_image_from_code(self, mermaid_code, output_path):\n with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file:\n temp_file.write(mermaid_code)\n input_path = temp_file.name\n result = subprocess.run([\"mmdc\", \"-i\", input_path, \"-o\", output_path, \"-t\", self._theme, \"-b\", self._background], shell=True, check=False)\n os.remove(input_path)\n if result.returncode != 0:\n raise ValueError(\"Mermaid diagram generation failed.\")\n\n##### Script to Generate Responses and Validate Mermaid Diagrams #####\ndef read_input(input_source):\n if os.path.isfile(input_source):\n filename, file_extension = os.path.splitext(input_source)\n if file_extension == '.json':\n with open(input_source, 'r') as file:\n return json.load(file)\n elif file_extension == '.txt':\n with open(input_source, 'r') as file:\n return [{\"input\": file.read()}]\n else:\n return [{\"input\": input_source}]\n\ndef generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs):\n prompt_template = f\"{prompt}\\n```mermaid\\n\"\n\n url = \"http://127.0.0.1:5000/v1/completions\"\n headers = {\"Content-Type\": \"application/json\"}\n dataset_entries = []\n\n for output_number, temp in enumerate(base_temperatures, start=1):\n while True:\n data = {\n \"prompt\": prompt_template,\n \"max_tokens\": 4096,\n \"temperature\": temp,\n \"top_p\": 1.0,\n \"seed\": -1,\n \"top_k\": 4,\n \"repetition_penalty\": 1.0,\n \"guidance_scale\": 1.0,\n \"typical_p\": 1.0,\n \"stream\": stream,\n }\n\n response = requests.post(url, headers=headers, json=data, verify=False)\n response_text = response.json()['choices'][0]['text'].strip()\n\n if response_text.endswith(\"```\"): # Check if response ends with ```\n response_text = response_text[:-3].strip() # Remove ``` from the end\n\n if response_text not in unique_outputs:\n try:\n image_path = generator.convert_to_image(response_text, entry_number, output_number)\n print(f\"Mermaid diagram generated at: {image_path}\")\n unique_outputs.add(response_text)\n break\n except ValueError as e:\n print(f\"Validation failed, retrying... Error: {e}\")\n else:\n temp += 0.1 # Adjust temperature if output is not unique\n\n dataset_entry = {\n \"input\": prompt,\n \"output\": f\"```mermaid\\n{response_text}\\n```\",\n \"temperature\": temp\n }\n dataset_entries.append(dataset_entry)\n\n return dataset_entries\n\ndef generate_unique_responses(input_data, base_temperatures, stream, generator):\n all_entries = []\n unique_outputs = set()\n\n for entry_number, entry in enumerate(input_data, start=1):\n prompt = entry.get(\"input\", \"\")\n if prompt:\n entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs)\n all_entries.extend(entries) # Extend the list with new entries\n\n return all_entries\n\ndef main(input_source, stream=False):\n generator = MermaidDiagramGenerator()\n input_data = read_input(input_source)\n base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input\n output_file = \"output.json\"\n\n all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator)\n\n # Write all entries to the JSON file at once\n with open(output_file, \"w\") as f:\n json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Generate unique responses and validate Mermaid diagrams.\")\n parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.')\n parser.add_argument('--stream', action='store_true', help='Use streaming responses.')\n args = parser.parse_args()\n \n main(args.input_source, args.stream)\n", 29 | "output": "```mermaid\ngraph TB;\n A[Start] --> B[Initialize MermaidDiagramGenerator];\n B --> C[Read input source];\n C --> D{Input file or string?};\n D --> |Input file| E[Load JSON file];\n D --> |Multi-line string| F[Create single input entry];\n E --> G[Generate responses for each input];\n F --> G;\n \n subgraph Generate Response\n direction TB;\n \n G --> H[Temperature loop];\n H --> I[Send prompt to LLM];\n I --> J{Get valid response?};\n J --> |No| K[Adjust temperature];\n K --> H;\n J --> |Yes| L[Convert response to image];\n \n L --> M[Add entry to output list];\n M --> N[Write output file];\n \n end\n \n N --> O[End];\n```", 30 | "temperature": 1.0 31 | } 32 | ] -------------------------------------------------------------------------------- /Mermaid_KnowledgeGraph_Implementation/toolkit/prompt.txt: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import os 4 | import requests 5 | import subprocess 6 | import tempfile 7 | 8 | ##### MermaidDiagramGenerator Class ##### 9 | class MermaidDiagramGenerator: 10 | def __init__(self, theme='dark', background='transparent'): 11 | self._theme = theme 12 | self._background = background 13 | self._entries_dir = os.path.join(os.getcwd(), 'Entries') 14 | os.makedirs(self._entries_dir, exist_ok=True) 15 | 16 | def convert_to_image(self, mermaid_code, entry_number, output_number): 17 | clean_code = self._remove_mermaid_block_markers(mermaid_code) 18 | output_filename = f"entry_{entry_number}_{output_number}.png" 19 | output_path = os.path.join(self._entries_dir, output_filename) 20 | self._generate_image_from_code(clean_code, output_path) 21 | return output_path 22 | 23 | def _remove_mermaid_block_markers(self, code): 24 | code_lines = code.strip().splitlines() 25 | if code_lines[0].startswith("```mermaid") and code_lines[-1] == "```": 26 | return "\n".join(code_lines[1:-1]).strip() 27 | return code 28 | 29 | def _generate_image_from_code(self, mermaid_code, output_path): 30 | with tempfile.NamedTemporaryFile(delete=False, mode='w', suffix='.mmd') as temp_file: 31 | temp_file.write(mermaid_code) 32 | input_path = temp_file.name 33 | result = subprocess.run(["mmdc", "-i", input_path, "-o", output_path, "-t", self._theme, "-b", self._background], shell=True, check=False) 34 | os.remove(input_path) 35 | if result.returncode != 0: 36 | raise ValueError("Mermaid diagram generation failed.") 37 | 38 | ##### Script to Generate Responses and Validate Mermaid Diagrams ##### 39 | def read_input(input_source): 40 | if os.path.isfile(input_source): 41 | filename, file_extension = os.path.splitext(input_source) 42 | if file_extension == '.json': 43 | with open(input_source, 'r') as file: 44 | return json.load(file) 45 | elif file_extension == '.txt': 46 | with open(input_source, 'r') as file: 47 | return [{"input": file.read()}] 48 | else: 49 | return [{"input": input_source}] 50 | 51 | def generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs): 52 | prompt_template = f"{prompt}\n```mermaid\n" 53 | 54 | url = "http://127.0.0.1:5000/v1/completions" 55 | headers = {"Content-Type": "application/json"} 56 | dataset_entries = [] 57 | 58 | for output_number, temp in enumerate(base_temperatures, start=1): 59 | while True: 60 | data = { 61 | "prompt": prompt_template, 62 | "max_tokens": 4096, 63 | "temperature": temp, 64 | "top_p": 1.0, 65 | "seed": -1, 66 | "top_k": 4, 67 | "repetition_penalty": 1.0, 68 | "guidance_scale": 1.0, 69 | "typical_p": 1.0, 70 | "stream": stream, 71 | } 72 | 73 | response = requests.post(url, headers=headers, json=data, verify=False) 74 | response_text = response.json()['choices'][0]['text'].strip() 75 | 76 | if response_text.endswith("```"): # Check if response ends with ``` 77 | response_text = response_text[:-3].strip() # Remove ``` from the end 78 | 79 | if response_text not in unique_outputs: 80 | try: 81 | image_path = generator.convert_to_image(response_text, entry_number, output_number) 82 | print(f"Mermaid diagram generated at: {image_path}") 83 | unique_outputs.add(response_text) 84 | break 85 | except ValueError as e: 86 | print(f"Validation failed, retrying... Error: {e}") 87 | else: 88 | temp += 0.1 # Adjust temperature if output is not unique 89 | 90 | dataset_entry = { 91 | "input": prompt, 92 | "output": f"```mermaid\n{response_text}\n```", 93 | "temperature": temp 94 | } 95 | dataset_entries.append(dataset_entry) 96 | 97 | return dataset_entries 98 | 99 | def generate_unique_responses(input_data, base_temperatures, stream, generator): 100 | all_entries = [] 101 | unique_outputs = set() 102 | 103 | for entry_number, entry in enumerate(input_data, start=1): 104 | prompt = entry.get("input", "") 105 | if prompt: 106 | entries = generate_response(prompt, base_temperatures, stream, generator, entry_number, unique_outputs) 107 | all_entries.extend(entries) # Extend the list with new entries 108 | 109 | return all_entries 110 | 111 | def main(input_source, stream=False): 112 | generator = MermaidDiagramGenerator() 113 | input_data = read_input(input_source) 114 | base_temperatures = [i / 10 for i in range(5, 11)] # Adjusted for batch of unique outputs per input 115 | output_file = "output.json" 116 | 117 | all_entries = generate_unique_responses(input_data, base_temperatures, stream, generator) 118 | 119 | # Write all entries to the JSON file at once 120 | with open(output_file, "w") as f: 121 | json.dump(all_entries, f, indent=4) # Dump the entire list of entries into the file 122 | 123 | if __name__ == "__main__": 124 | parser = argparse.ArgumentParser(description="Generate unique responses and validate Mermaid diagrams.") 125 | parser.add_argument('input_source', type=str, help='A multi-line string, path to a .txt file, or a .json file with prompts.') 126 | parser.add_argument('--stream', action='store_true', help='Use streaming responses.') 127 | args = parser.parse_args() 128 | 129 | main(args.input_source, args.stream) 130 | -------------------------------------------------------------------------------- /Meta-A-Wrapper-Around-OpenAI.mov: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/Meta-A-Wrapper-Around-OpenAI.mov -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ```python 2 | import json 3 | import argparse 4 | from transformers import AutoModelForCausalLM, AutoTokenizer 5 | 6 | class TransformerTextGenerator: 7 | def __init__(self, model_name_or_path): 8 | self.model = AutoModelForCausalLM.from_pretrained(model_name_or_path, revision="main") 9 | self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) 10 | 11 | def generate_text_output(self, prompt, temperature): 12 | input_ids = self.tokenizer(prompt, return_tensors='pt').input_ids.cuda() 13 | output = self.model.generate(inputs=input_ids, temperature=temperature, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512) 14 | return self.tokenizer.decode(output[0]) 15 | 16 | def main(): 17 | parser = argparse.ArgumentParser(description="Generate text outputs using a transformer model.") 18 | parser.add_argument("--input_file", required=True, help="Path to the input JSON file containing input entries.") 19 | parser.add_argument("--output_file", required=True, help="Path to save the output JSON file with generated text outputs.") 20 | parser.add_argument("--creative", action="store_true", help="Flag to enable creative mode with varied temperature from 0.5 to 1.0.") 21 | parser.add_argument("--factual", action="store_true", help="Flag to enable factual mode with varied temperature from 0.5 to 0.0.") 22 | args = parser.parse_args() 23 | 24 | if args.creative: 25 | temperature_range = [i / 10 for i in range(5, 11)] # Temperature range from 0.5 to 1.0 26 | elif args.factual: 27 | temperature_range = [i / 10 for i in range(5, -1, -1)] # Temperature range from 0.5 to 0.0 with decrement of 0.1 28 | else: 29 | print("Please specify either --creative or --factual mode.") 30 | return 31 | 32 | text_generator = TransformerTextGenerator("TheBloke/CapybaraHermes-2.5-Mistral-7B-GPTQ") 33 | text_outputs = [] 34 | 35 | with open(args.input_file, "r") as f: 36 | input_data = json.load(f) 37 | 38 | for entry in input_data: 39 | input_text = entry.get("input", "") 40 | entry_outputs = [] 41 | for temperature in temperature_range: 42 | text_output = text_generator.generate_text_output(input_text, temperature) 43 | entry_outputs.append({"temperature": temperature, "text_output": text_output}) 44 | text_outputs.append({"input": input_text, "outputs": entry_outputs}) 45 | 46 | with open(args.output_file, "w") as f: 47 | json.dump(text_outputs, f, indent=4) 48 | 49 | if __name__ == "__main__": 50 | main() 51 | ``` 52 | 53 | **Research Paper: Exploring Temperature Variation for Tailored Language Model Training** 54 | 55 | *By Troy Andrew Schultz* 56 | 57 | **Abstract:** 58 | This study delves into the impact of temperature variation on language model training, aiming to develop specialized models optimized for creative expression and factual precision. Through systematic experimentation, I investigate how adjusting temperature settings influences the diversity and accuracy of text generation in language models. 59 | 60 | **Introduction:** 61 | Driven by curiosity and a desire to unlock the potential of language models, I embarked on a quest to uncover the intricate relationship between temperature variation and model performance. By fine-tuning temperature settings, I aimed to develop models capable of meeting diverse linguistic needs, ranging from creative storytelling to factual reporting. 62 | 63 | **Methodology:** 64 | Employing the Hugging Face Transformers library, I trained separate language models tailored for creative expression and factual precision. By adjusting temperature settings within a defined range during training and inference, I sought to optimize model outputs for specific tasks and requirements. 65 | 66 | **Results:** 67 | My experiments demonstrated the effectiveness of temperature variation in shaping the outputs of language models. Enabling creative mode with a temperature range from 0.5 to 1.0 enhanced response diversity, fostering imaginative and contextually rich text generation. Conversely, factual mode, with a temperature range from 0.5 to 0.0, yielded precise and evidence-based responses, ideal for tasks requiring factual accuracy. 68 | 69 | **Discussion:** 70 | The findings suggest that temperature variation serves as a powerful tool in tailoring language model outputs to specific linguistic tasks. By providing control over temperature settings, I empowered users to fine-tune models for creativity or factual precision, depending on the application at hand. 71 | 72 | **Conclusion:** 73 | This research sheds light on the nuanced relationship between temperature variation and text generation in language models. By embracing experimentation and intuitive exploration, I pave the way for the development of specialized models capable of adapting to diverse linguistic needs and improving generalization across different tasks. 74 | 75 | **Acknowledgments:** 76 | I extend my gratitude to the open-source community and the developers of the Hugging Face Transformers library for their invaluable contributions to natural language processing research. 77 | 78 | # Exploring Temperature Variation for Tailored Language Model Training 79 | 80 | ## Introduction 81 | This repository contains code for exploring the impact of temperature variation on language model training. By adjusting temperature settings within a defined range, we aim to develop specialized language models optimized for creative expression and factual precision. 82 | -------------------------------------------------------------------------------- /opera_BDKLCRC9IW.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TroyDoesAI/AI_Research/7defe01133543bd5f1478e6d9422c7a83be67099/opera_BDKLCRC9IW.gif --------------------------------------------------------------------------------