├── .gitignore ├── LICENSE ├── README.md ├── clean.sh ├── env.example ├── go.sh ├── output └── output-sample.html ├── prepare.sh ├── repl.py ├── requirements.txt └── yt-sum.py /.gitignore: -------------------------------------------------------------------------------- 1 | env.py 2 | __pycache__ 3 | output/output.html 4 | storage -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Yamir Encarnacion 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # yt-sum.py - YouTube Video Summarizer 2 | 3 | This is a Python script that summarizes a youtube video from a YouTube URL 4 | using llamaindex. 5 | 6 | ## Prerequisites 7 | 8 | OpenAI API Key 9 | 10 | Python 3.6 or later. 11 | 12 | ## Installation 13 | 14 | Before running the script, you need to install the required Python libraries. 15 | 16 | 1. Open a terminal. 17 | 2. Navigate to the project directory where `requirements.txt` is located. 18 | 3. Run the following command: 19 | 20 | ```bash 21 | pip install -r requirements.txt 22 | ``` 23 | 24 | This will install the required libraries 25 | 26 | Also 27 | 28 | ```bash 29 | cp env.example env.py 30 | ``` 31 | 32 | And enter your openai API Key into the variable OPENAI_API_KEY in env.py 33 | 34 | ## Usage 35 | 36 | To summarize a YouTube video, you need to provide the YouTube URL as an argument 37 | when you run the go.sh script. 38 | 39 | Here is are some examples of how to run the script: 40 | 41 | ```bash 42 | ./go.sh https://www.youtube.com/watch?v=wbiEGHjlE4Y 43 | 44 | ./go.sh https://www.youtube.com/watch?v=-hxeDjAxvJ8 45 | ``` 46 | 47 | Running a command from like the above will generate an output.html 48 | file in the ./output directory. 49 | 50 | You can then ask additional questions about the video by running: 51 | 52 | ```bash 53 | python3 repl.py 54 | ``` 55 | 56 | The output of the repl gets appended to output.html 57 | 58 | ## Other: 59 | 60 | The script is a work in progress. If you pass a url with the character & 61 | on *nix, it will give an error 62 | (e.g., https://www.youtube.com/watch?v=-hxeDjAxvJ8&t=108s ). 63 | This is a *nix thing I will correct at some later point in time. 64 | 65 | ## Inspiration for the script 66 | 67 | The prompts that follow were copied from https://github.com/daveshap/Quickly_Extract_Science_Papers which has an MIT license. 68 | ``` 69 | >>> 70 | can you give me a very clear explanation of the core assertions, implications, and mechanics elucidated in this paper? 71 | <<< 72 | --- 73 | >>> 74 | can you explain the value of this in basic terms? Like you're talking to a ceo. so what? what's the bottom line here? 75 | <<< 76 | --- 77 | can you give me an analogy or metaphor that will help explain this to a broad audience 78 | <<< 79 | ``` -------------------------------------------------------------------------------- /clean.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -f clean.sh~ 4 | rm -rf __pycache__ 5 | rm -rf output/output.html 6 | rm -rf storage 7 | rm -f env.py~ 8 | rm -f go.sh~ 9 | rm -f prepare.sh~ 10 | rm -f README.md~ 11 | rm -f requirements.txt~ 12 | rm -f yt-sum.py~ 13 | rm -f env.example~ 14 | rm -f .gitignore~ 15 | rm -f repl.py~ 16 | -------------------------------------------------------------------------------- /env.example: -------------------------------------------------------------------------------- 1 | OPENAI_API_KEY = "" 2 | -------------------------------------------------------------------------------- /go.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf output/output.html 4 | rm -rf storage 5 | 6 | python3 yt-sum.py "$@" 7 | -------------------------------------------------------------------------------- /output/output-sample.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | -hxeDjAxvJ8 6 | 17 | 18 | 19 |

Embedded YouTube Video

20 | 21 |

what is this video about?

22 |

23 | This video is about the potential of using a chat interface with an LLM (Language Learning Machine) to provide prompts and advice to help people interpret and understand their lives more deeply. It also discusses the concept of prompt engineering and how LLMs could be used to write prompts. Finally, it mentions the controversy surrounding Marc Andreessen's Wikipedia page in 1993 and how this could potentially drive engagement up.

24 |

can you give me a very clear explanation of the core assertions, implications, and mechanics elucidated in this video?

25 |

26 | The core assertions, implications, and mechanics elucidated in this video are that the world is malleable and can be reconfigured around a person if they have a clear goal and the drive and passion to pursue it. Marc Andreessen suggests that LLMs (Life-Long Machines) could be used to provide a continuous feed of information to help people interpret and understand their lives more deeply, and could even provide advice on mundane tasks such as making coffee or where to go for lunch. He also suggests that LLMs could be used to write prompts to drive engagement, and that this could be tested today.

27 |

can you explain the value of this in basic terms? Like you're talking to a ceo. so what? what's the bottom line here?

28 |

29 | The bottom line is that startups have the ability to innovate quickly and create new products and services that can disrupt the market. They don't have the same resources as larger companies, but they can still make a big impact. This is why it's important to support startups and give them the resources they need to succeed.

30 |

can you give me an analogy or metaphor that will help explain this to a broad audience

31 |

32 | A feedback loop is like a game of catch. You throw the ball, and the other person catches it and throws it back. The ball is the information, and the two people are the different parts of the feedback loop. The game of catch is a two-way process, and the media is the field in which the game is played. Just like changing the field can change the game, changing the media environment can change reality.

33 |

according to this video what is the relationship between thr philosophers stone and programmers or developers or engineers or software

34 | According to this video, the relationship between the philosopher's stone and programmers, developers, engineers, or software is that software is like a modern philosopher's stone, which transmutes labor into capital. This is a concept that would have been inconceivable to Carl Marx, as he believed that labor and capital were two separate entities. Software engineers are able to create value out of thin air, purely from human thought, which is a magical and special concept.

35 | 36 | -------------------------------------------------------------------------------- /prepare.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mkdir output 4 | # sudo apt install -y tesseract-ocr 5 | # sudo apt-get install -y poppler-utils 6 | -------------------------------------------------------------------------------- /repl.py: -------------------------------------------------------------------------------- 1 | from env import OPENAI_API_KEY 2 | from llama_index import StorageContext, load_index_from_storage 3 | from bs4 import BeautifulSoup 4 | import os 5 | import openai 6 | 7 | os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY 8 | openai.api_key = os.getenv("OPENAI_API_KEY") 9 | 10 | if openai.api_key is None: 11 | raise Exception('OPENAI_API_KEY environment variable not set') 12 | sys.exit(1) 13 | 14 | def append_to_output(question, answer): 15 | # Open the HTML file and turn it into a BeautifulSoup object 16 | with open("./output/output.html", 'r') as f: 17 | contents = f.read() 18 | 19 | soup = BeautifulSoup(contents, 'html.parser') 20 | 21 | # Find the body tag in the HTML 22 | body = soup.find('body') 23 | 24 | # Create the new h2 tag and p tag 25 | new_h2 = soup.new_tag('h2') 26 | new_h2.string = question 27 | new_p = soup.new_tag('p') 28 | new_p.string = answer 29 | 30 | # Append these tags to the body 31 | body.append(new_h2) 32 | body.append(new_p) 33 | 34 | # Write the changes back to the file 35 | with open("./output/output.html", "w") as f: 36 | f.write(str(soup)) 37 | 38 | 39 | def process_input(user_input): 40 | storage_context = StorageContext.from_defaults(persist_dir="./storage") 41 | index = load_index_from_storage(storage_context) 42 | 43 | 44 | query_engine = index.as_query_engine() 45 | 46 | question_01 = user_input 47 | response_01 = query_engine.query(question_01) 48 | 49 | response = [ {question_01: response_01.response} ] 50 | 51 | return response 52 | 53 | while True: # Start an infinite loop 54 | try: 55 | # Read the input 56 | user_input = input('Ask something: ') 57 | 58 | # Process the input 59 | response = process_input(user_input) 60 | 61 | answer = list(response[0].values())[0] 62 | 63 | # Print the output 64 | print(f">>>{answer}") 65 | print("<<<\n") 66 | append_to_output(user_input,answer) 67 | except KeyboardInterrupt: 68 | print("REPL terminated by user.") 69 | break 70 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | youtube-transcript-api==0.6.1 2 | argparse==1.4.0 3 | llama-index==0.7.4 4 | pdf2image==1.16.3 5 | pytesseract==0.3.10 6 | beautifulsoup4==4.12.2 7 | openai 8 | -------------------------------------------------------------------------------- /yt-sum.py: -------------------------------------------------------------------------------- 1 | from youtube_transcript_api import YouTubeTranscriptApi 2 | import argparse 3 | from urllib.parse import urlparse, parse_qs 4 | from env import OPENAI_API_KEY 5 | import os 6 | from llama_index import download_loader, GPTVectorStoreIndex 7 | from pathlib import Path 8 | import tempfile 9 | from llama_index import ServiceContext 10 | from llama_index import StorageContext, load_index_from_storage 11 | import sys 12 | import openai 13 | 14 | os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY 15 | openai.api_key = os.getenv("OPENAI_API_KEY") 16 | 17 | if openai.api_key is None: 18 | raise Exception('OPENAI_API_KEY environment variable not set') 19 | sys.exit(1) 20 | 21 | def extract_video_id(url): 22 | # Parse the URL 23 | query = urlparse(url) 24 | 25 | # Extract the video ID from the query parameters 26 | video_id = parse_qs(query.query).get('v') 27 | 28 | if video_id: 29 | return video_id[0] 30 | else: 31 | return None 32 | 33 | def get_video_transcript(video_id): 34 | transcript = YouTubeTranscriptApi.get_transcript(video_id) 35 | 36 | return transcript 37 | 38 | def index_persist(transcript): 39 | UnstructuredReader = download_loader("UnstructuredReader", refresh_cache=True) 40 | loader = UnstructuredReader() 41 | 42 | # Create a temporary file and write the string data to it 43 | with tempfile.NamedTemporaryFile(delete=False, mode='w') as temp: 44 | for item in transcript: 45 | text = item['text'] 46 | temp.write(text +'\n') 47 | temp_file_path = Path(temp.name) 48 | 49 | # Load the data from the temporary file 50 | data = loader.load_data(file=temp_file_path, split_documents=False) 51 | 52 | service_context = ServiceContext.from_defaults(chunk_size_limit=512) 53 | 54 | cur_index = GPTVectorStoreIndex.from_documents(data, service_context=service_context) 55 | 56 | cur_index.storage_context.persist() 57 | 58 | 59 | def get_summary(): 60 | storage_context = StorageContext.from_defaults(persist_dir="./storage") 61 | 62 | index = load_index_from_storage(storage_context) 63 | 64 | query_engine = index.as_query_engine() 65 | 66 | 67 | question_01 = "what is this video about?" 68 | response_01 = query_engine.query(question_01) 69 | 70 | question_02 = "can you give me a very clear explanation of the core assertions, implications, and mechanics elucidated in this video?" 71 | response_02 = query_engine.query(question_02) 72 | 73 | question_03 = "can you explain the value of this in basic terms? Like you're talking to a ceo. so what? what's the bottom line here?" 74 | response_03 = query_engine.query(question_03) 75 | 76 | question_04 = "can you give me an analogy or metaphor that will help explain this to a broad audience" 77 | response_04 = query_engine.query(question_04) 78 | 79 | response = [ {question_01: response_01.response}, 80 | {question_02: response_02.response}, 81 | {question_03: response_03.response}, 82 | {question_04: response_04.response} 83 | ] 84 | 85 | return response 86 | 87 | def write_response_to_file(video_id, response): 88 | # Starting the HTML string 89 | html_string = f""" 90 | 91 | 92 | {video_id} 93 | 104 | 105 | 106 | """ 107 | html_string += f'

Embedded YouTube Video

\n\n' 108 | 109 | # Appending each question and response 110 | for r in response: 111 | for question, answer in r.items(): 112 | html_string += f"

{question}

\n

{answer}

\n" 113 | 114 | # Closing the HTML string 115 | html_string += """ 116 | 117 | 118 | """ 119 | 120 | # Make directory if it doesn't exist 121 | if not os.path.exists('./output'): 122 | os.makedirs('./output') 123 | 124 | # Writing the HTML string to a file 125 | with open(os.path.expanduser("./output/output.html"), "w") as file: 126 | file.write(html_string) 127 | 128 | def main(): 129 | # Initialize the ArgumentParser object 130 | parser = argparse.ArgumentParser(description='Summarize a YouTube video from URL.') 131 | 132 | # Add an argument for the YouTube URL 133 | parser.add_argument('url', type=str, help='The YouTube video URL.') 134 | 135 | # Parse the arguments 136 | args = parser.parse_args() 137 | 138 | # Extract the video ID 139 | video_id = extract_video_id(args.url) 140 | 141 | # Print the video ID 142 | if video_id: 143 | print(f'Video ID: {video_id}') 144 | else: 145 | print('Invalid URL or no video ID found.') 146 | sys.exit(1) 147 | 148 | # check if something already stored in ./storage 149 | # if not persist index 150 | try: 151 | storage_context = StorageContext.from_defaults(persist_dir="./storage") 152 | del storage_context 153 | except FileNotFoundError as e: 154 | transcript = get_video_transcript(video_id) 155 | if not transcript: 156 | print('Invalid transcript or transcript found.') 157 | sys.exit(1) 158 | 159 | index_persist(transcript) 160 | 161 | response = get_summary() 162 | 163 | write_response_to_file(video_id, response) 164 | 165 | if __name__ == "__main__": 166 | main() 167 | --------------------------------------------------------------------------------