├── assets ├── screenshot.jpg ├── screenshot_vlc.jpg └── screenshot_in_out.jpg ├── requirements.txt ├── LICENSE ├── readme.md ├── .gitignore └── main.py /assets/screenshot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Tophness/ChatGPT-PC-Controller/HEAD/assets/screenshot.jpg -------------------------------------------------------------------------------- /assets/screenshot_vlc.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Tophness/ChatGPT-PC-Controller/HEAD/assets/screenshot_vlc.jpg -------------------------------------------------------------------------------- /assets/screenshot_in_out.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Tophness/ChatGPT-PC-Controller/HEAD/assets/screenshot_in_out.jpg -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | PyAutoIt==0.6.5 2 | git+https://github.com/Tophness/OpenAIAPIGrabber 3 | git+https://github.com/Tophness/pyautoit -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Chris Malone 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # ChatGPT PC Controller 2 | 3 | This application lets ChatGPT directly control your PC using Python and AutoIt to hook into the Win32 API and Windows Runtime UI. 4 | 5 | It can control the mouse and keyboard, read and click windows control elements, run any command, move, minimize and close windows, read and write to clipboard and more. 6 | 7 | It utilizes the main OpenAI website's ChatGPT API underneath, instead of requiring costly and limited API keys. 8 | 9 | ## Screenshots 10 | ![ChatGPT PC Controller](assets/screenshot.jpg?raw=true) 11 | - Here is is creating a command to read the text of my open notepad window, automatically replying back to ChatGPT with it's contents, and ChatGPT generating a new command back to windows based on that: 12 | ![ChatGPT PC Controller performing input and output ](assets/screenshot_in_out.jpg?raw=true) 13 | - Newer syntax 26/06/2023: 14 | ![ChatGPT PC Controller performing input and output ](assets/screenshot_vlc.jpg?raw=true) 15 | 16 | ## Prerequisites 17 | 18 | Before running the application, make sure you have the following prerequisites installed: 19 | 20 | 1. Python 3.x 21 | 22 | 2. Git 23 | 24 | ## Installation 25 | 26 | 1. Clone the repository: 27 | 28 | ```shell 29 | git clone https://github.com/Tophness/ChatGPT-PC-Controller.git 30 | ``` 31 | 32 | 2. Change to the project directory: 33 | 34 | ```shell 35 | cd ChatGPT-PC-Controller 36 | ``` 37 | 38 | 3. Install the prerequisite Python libraries: 39 | 40 | ```shell 41 | pip install -r requirements.txt 42 | ``` 43 | 44 | ## Usage 45 | 46 | To use the program, follow these steps: 47 | 48 | 1. Open a terminal or command prompt and navigate to the project directory. 49 | 50 | 2. Run the `main.py` file: 51 | 52 | ```shell 53 | python main.py 54 | ``` 55 | 56 | 3. Enter a goal for what you want ChatGPT to do to control your PC. 57 | 58 | 4. ChatGPT will generate a set of control commands. 59 | 60 | 5. Confirm whether you want to run it. 61 | 62 | ## Settings 63 | - Set config.ini's 'Unattended mode' to True if you like to live dangerously. It will run in an infinite loop allowing ChatGPT to make it's own decisions and explore everything it opens and reads on your computer without asking. 64 | - You can also edit the 'Preprompt' setting to adjust the prompt that gets prepended to your command. 65 | 66 | ## Contributing 67 | 68 | Contributions are welcome! If you find any issues or want to enhance the functionality of this application, feel free to open an issue or submit a pull request. 69 | 70 | ## License 71 | 72 | This project is licensed under the [MIT License](LICENSE). 73 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Generated 2 | chromedriver/ 3 | chrome-win32/ 4 | chrome-win64/ 5 | config.ini 6 | messages.yaml 7 | 8 | # Shortcuts 9 | run.bat 10 | 11 | # Byte-compiled / optimized / DLL files 12 | __pycache__/ 13 | *.py[cod] 14 | *$py.class 15 | 16 | # C extensions 17 | *.so 18 | 19 | # Distribution / packaging 20 | .Python 21 | build/ 22 | develop-eggs/ 23 | dist/ 24 | downloads/ 25 | eggs/ 26 | .eggs/ 27 | lib/ 28 | lib64/ 29 | parts/ 30 | sdist/ 31 | var/ 32 | wheels/ 33 | share/python-wheels/ 34 | *.egg-info/ 35 | .installed.cfg 36 | *.egg 37 | MANIFEST 38 | 39 | # PyInstaller 40 | # Usually these files are written by a python script from a template 41 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 42 | *.manifest 43 | *.spec 44 | 45 | # Installer logs 46 | pip-log.txt 47 | pip-delete-this-directory.txt 48 | 49 | # Unit test / coverage reports 50 | htmlcov/ 51 | .tox/ 52 | .nox/ 53 | .coverage 54 | .coverage.* 55 | .cache 56 | nosetests.xml 57 | coverage.xml 58 | *.cover 59 | *.py,cover 60 | .hypothesis/ 61 | .pytest_cache/ 62 | cover/ 63 | 64 | # Translations 65 | *.mo 66 | *.pot 67 | 68 | # Django stuff: 69 | *.log 70 | local_settings.py 71 | db.sqlite3 72 | db.sqlite3-journal 73 | 74 | # Flask stuff: 75 | instance/ 76 | .webassets-cache 77 | 78 | # Scrapy stuff: 79 | .scrapy 80 | 81 | # Sphinx documentation 82 | docs/_build/ 83 | 84 | # PyBuilder 85 | .pybuilder/ 86 | target/ 87 | 88 | # Jupyter Notebook 89 | .ipynb_checkpoints 90 | 91 | # IPython 92 | profile_default/ 93 | ipython_config.py 94 | 95 | # pyenv 96 | # For a library or package, you might want to ignore these files since the code is 97 | # intended to run in multiple environments; otherwise, check them in: 98 | # .python-version 99 | 100 | # pipenv 101 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 102 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 103 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 104 | # install all needed dependencies. 105 | #Pipfile.lock 106 | 107 | # poetry 108 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 109 | # This is especially recommended for binary packages to ensure reproducibility, and is more 110 | # commonly ignored for libraries. 111 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 112 | #poetry.lock 113 | 114 | # pdm 115 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 116 | #pdm.lock 117 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 118 | # in version control. 119 | # https://pdm.fming.dev/#use-with-ide 120 | .pdm.toml 121 | 122 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 123 | __pypackages__/ 124 | 125 | # Celery stuff 126 | celerybeat-schedule 127 | celerybeat.pid 128 | 129 | # SageMath parsed files 130 | *.sage.py 131 | 132 | # Environments 133 | .env 134 | .venv 135 | env/ 136 | venv/ 137 | ENV/ 138 | env.bak/ 139 | venv.bak/ 140 | 141 | # Spyder project settings 142 | .spyderproject 143 | .spyproject 144 | 145 | # Rope project settings 146 | .ropeproject 147 | 148 | # mkdocs documentation 149 | /site 150 | 151 | # mypy 152 | .mypy_cache/ 153 | .dmypy.json 154 | dmypy.json 155 | 156 | # Pyre type checker 157 | .pyre/ 158 | 159 | # pytype static type analyzer 160 | .pytype/ 161 | 162 | # Cython debug symbols 163 | cython_debug/ 164 | 165 | # PyCharm 166 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 167 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 168 | # and can be added to the global gitignore or merged into this file. For a more nuclear 169 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 170 | #.idea/ 171 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | import autoit 2 | import sys 3 | import ast 4 | from OpenAIAPIGrabber.chat import OpenAIChat 5 | import re 6 | import configparser 7 | import os 8 | import time 9 | 10 | config_file = 'config.ini' 11 | config = None 12 | unattended = False 13 | preprompt = '''You are directly controlling a windows PC using autoit functions. 14 | I will reply back with the result of any commands that return a value and you can use that to decide what commands to generate next. 15 | Output only the functions to run and nothing else. Don't set any variables. Don't use window handles. Don't comment anything. 16 | Here is an example to type hello world in notepad. 17 | `Run('notepad.exe') 18 | WinWaitActive('[CLASS:Notepad]','',3) 19 | ControlSend('[CLASS:Notepad]','','[CLASS:Edit1]','hello world')` 20 | Now generate commands to''' 21 | 22 | def load_config(): 23 | global config 24 | config = configparser.ConfigParser() 25 | config.read(config_file) 26 | controller_section = None 27 | if('Controller' in config): 28 | controller_section = config['Controller'] 29 | else: 30 | save_config() 31 | controller_section = config['Controller'] 32 | if(controller_section): 33 | global unattended, preprompt 34 | unattended = controller_section.getboolean('Unattended') 35 | preprompt = controller_section.get('Preprompt') 36 | 37 | def save_config(): 38 | global config 39 | config['Controller'] = { 40 | 'Unattended': unattended, 41 | 'Preprompt': preprompt 42 | } 43 | with open(config_file, 'w') as configfile: 44 | config.write(configfile) 45 | 46 | def extract_code_blocks(code_string): 47 | code_block_regex = r"```(?:\w+\n)?([\w\W]*?)```|`(?:\w+\n)?([\w\W]*?)`" 48 | matches = re.findall(code_block_regex, code_string) 49 | matches = [match[0] if match[0] else match[1] for match in matches] 50 | functions = [] 51 | 52 | for match in matches: 53 | match = match.strip() 54 | if '(' in match and ')' in match: 55 | functions.append(match) 56 | if(len(functions) > 0): 57 | return functions 58 | else: 59 | return code_string 60 | 61 | def correct_file_path(arg): 62 | if(not isinstance(arg, str)): 63 | return arg 64 | if arg.startswith("'\"") and arg.endswith("\"'"): 65 | arg = arg.replace("'\"", "'").replace("\"'", "'") 66 | if arg.startswith('"'): 67 | arg = arg.replace('"', "'") 68 | if arg.endswith('"'): 69 | arg = arg.replace('"', "'") 70 | drive, path = os.path.splitdrive(arg.replace("'","").replace('"',"")) 71 | if(drive): 72 | if(os.path.exists(arg)): 73 | return arg 74 | elif("\\" in arg and not "\\\\" in arg): 75 | arg = arg.replace("\\", "/") 76 | if(os.path.exists(arg)): 77 | return arg 78 | return arg 79 | 80 | def extract_file_path(cmd_string): 81 | cmds = re.findall(r'\((.*?)\)', cmd_string) 82 | argsnew = [] 83 | for args in cmds: 84 | arg_list = [correct_file_path(arg.strip()) for arg in args.split(',')] 85 | argsnew.append('(' + ', '.join(arg_list) + ')') 86 | cmd_string_new = re.sub(r'\((.*?)\)', lambda x: argsnew.pop(0), cmd_string) 87 | return cmd_string_new 88 | 89 | def convert_function_call(cmd_string): 90 | cmd_string = extract_file_path(cmd_string) 91 | tree = ast.parse(cmd_string.strip()) 92 | function_call = next(node for node in ast.walk(tree) if isinstance(node, ast.Expr) and isinstance(node.value, ast.Call)) 93 | function_name = function_call.value.func.id 94 | func = None 95 | if(function_name == 'Sleep'): 96 | func = time.sleep 97 | else: 98 | func = getattr(autoit, function_name) 99 | if func: 100 | args = [ast.literal_eval(arg) for arg in function_call.value.args] 101 | if(function_name == 'Sleep'): 102 | args[0] = args[0] / 1000 103 | try: 104 | funcResult = func(*args) 105 | return funcResult 106 | except Exception as e: 107 | errorMsg = "Call to " + function_name + " with arguments " + ",".join(str(x) for x in args) + " returned error: " + str(e) 108 | print(errorMsg) 109 | return errorMsg 110 | else: 111 | raise ValueError("Invalid function name: " + function_name) 112 | 113 | def remove_variable_assignment(text): 114 | if hasattr(text, '__len__') and (not isinstance(text, str)): 115 | text = "\n".join(text) 116 | lines = text.split("\n") 117 | functions = [] 118 | for line in lines: 119 | match = re.search(r'(\b\w+\(.*?\))', line) 120 | if match: 121 | functions.append(match.group(1).strip()) 122 | else: 123 | functions.append(line) 124 | return functions 125 | 126 | def execute_commands(cmds_string): 127 | funcData = [] 128 | commands = remove_variable_assignment(cmds_string) 129 | for cmd in commands: 130 | returnData = str(convert_function_call(cmd)) 131 | if(returnData): funcData.append('Call to ' + cmd + ' returned: ' + returnData) 132 | if(len(funcData) > 0): 133 | return "\n".join(funcData) + "\nWhat function do you want to execute next?" 134 | else: 135 | return None 136 | 137 | def getCmd(chat, prompt, reply=False): 138 | chatResult = None 139 | if(reply): 140 | chatResult = chat.replyLast(prompt) 141 | else: 142 | chatResult = chat.start(preprompt + " " + prompt) 143 | if(chatResult): 144 | if hasattr(chatResult, '__len__') and (not isinstance(chatResult, str)): 145 | chatResult = str(chatResult[0]) 146 | chatResult = extract_code_blocks(chatResult) 147 | if (not unattended): 148 | print('Going to execute:') 149 | commands = remove_variable_assignment(chatResult) 150 | for cmd in commands: 151 | print(cmd) 152 | confirmation = input("\nProceed? (y/n): ") 153 | if confirmation.lower() != "y": 154 | chat.deleteLast() 155 | print("Operation cancelled.") 156 | return 157 | cmdResult = execute_commands(chatResult) 158 | if(cmdResult): 159 | getCmd(chat, cmdResult, True) 160 | elif(unattended): 161 | getCmd(chat, "what function do you want to execute next?", True) 162 | else: 163 | chat.deleteLast() 164 | 165 | if __name__ == "__main__": 166 | load_config() 167 | try: 168 | cmd_string = sys.argv[1] 169 | except IndexError: 170 | cmd_string = input("Enter a task: ") 171 | getCmd(OpenAIChat(), cmd_string) --------------------------------------------------------------------------------