├── .gitignore ├── Tutorial-Notebook-Olympics-App └── README.md ├── requirements.txt ├── src ├── logo.jpeg └── index-creation.png ├── README.md └── app.py /.gitignore: -------------------------------------------------------------------------------- 1 | .env 2 | __pycache__/ -------------------------------------------------------------------------------- /Tutorial-Notebook-Olympics-App/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | streamlit 2 | twelvelabs 3 | requests 4 | python-dotenv 5 | apscheduler 6 | -------------------------------------------------------------------------------- /src/logo.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Hrishikesh332/TwelveLabs-Olympics-App/HEAD/src/logo.jpeg -------------------------------------------------------------------------------- /src/index-creation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Hrishikesh332/TwelveLabs-Olympics-App/HEAD/src/index-creation.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |
2 |
3 |

Olympics Video Classification 4 |

5 | Categorize Olympic sports using video clips with Twelve Labs 6 |
7 | Explore the docs » 8 |
9 |
10 | View Demo 11 | · 12 | Report Bug 13 | · 14 | Request Feature 15 |

16 |

17 | 18 | 19 | 20 |
21 | Table of Contents 22 |
    23 |
  1. About
    2. 24 |
  2. Features
    3. 25 |
  3. Tech Stack
    4. 26 |
  4. Instructions on running project locally
    5. 27 |
  5. Feedback
    6. 28 |
  6. Feedback
    7. 29 |
30 |
31 | 32 | ------ 33 | 34 | ## About 35 | 36 | The Olympics Video Clips Classification Application is a powerful tool designed to categorize various Olympic sports using video clips. By leveraging Twelve Labs' Marengo 2.6 Embedding Model, this application provides accurate classification of Olympic sports based on visual content, conversation, and text in video. 37 | 38 | 39 | ## Demonstration 40 | 41 | Try the Application Now - 42 | 43 | 57 | Olympics Classification App 58 | 59 | 60 | 61 | [![Watch the video](https://img.youtube.com/vi/9f2mScVn5ck/hqdefault.jpg)](https://youtu.be/9f2mScVn5ck) 62 | 63 | 64 | ## Features 65 | 66 | 🏅 **Olympic Sports Classification**: Accurately categorize various Olympic sports from video clips. 67 | 68 | 🎥 **Video Analysis**: Twelve Labs' Marengo 2.6 Embedding Model for comprehensive video analysis. 69 | 70 | 🚀️**Custom Classes**:Custom Classes for the more personalized categorization. 71 | 72 | ## Tech Stack 73 | 74 | **Frontend** - Streamlit 75 | 76 | **Backend** - Python 77 | 78 | **AI Technologies** - Twelve Labs Marengo 2.6 (Embedding Model) 79 | 80 | **Deployment** - Streamlit Cloud 81 | 82 | 83 | ## Instructions on running project locally: 84 | 85 | To get started with the Olympics Video Clips Classification Application, follow these steps - 86 | 87 | Clone the project 88 | 89 | Step 1 - 90 | 91 | ```bash 92 | git clone https://github.com/Hrishikesh332/TwelveLabs-Olympics-App.git 93 | ``` 94 | 95 | Step 2 - 96 | 97 | Install dependencies: 98 | 99 | ```bash 100 | cd TwelveLabs-Interview-App 101 | 102 | pip install -r requirements.txt 103 | ``` 104 | 105 | Step 3 - 106 | 107 | Set up your Twelve Labs account - 108 | 109 | Create an account on the Twelve Labs Portal 110 | Navigate to the Twelve Labs Playground 111 | Create a new index and select Marengo 2.6 as the Embedding Model 112 | Upload Olympic video clips to your index 113 | 114 | ![index-creation](https://github.com/Hrishikesh332/TwelveLabs-Olympics-App/blob/main/src/index-creation.png) 115 | 116 | Step 4 - 117 | 118 | Get your API Key from the [Twelve Labs Dashboard](https://playground.twelvelabs.io/dashboard/api-key) 119 | Find your INDEX_ID in the URL of your created [index](https://playground.twelvelabs.io/indexes/{index_id}) 120 | 121 | Step 5 - 122 | Configure the application with your API credentials. 123 | 124 | Step 6 - 125 | 126 | Run the Streamlit application 127 | 128 | ```bash 129 | streamlit run app.py 130 | ``` 131 | 132 | Step 7 - 133 | 134 | Run the Server - 135 | 136 | ```bash 137 | http://localhost:8501/ 138 | ``` 139 | 140 | ## Usecases 141 | 142 | 🔍**Video Search Enginen:** Create a searchable database of video content, allowing users to find specific scenes or topics within large video collections. 143 | 144 | 🎥**Security Footage Analyzer** 145 | Detect and categorize specific events or behaviors in security camera footage. 146 | 147 | 💃 **Dance Move Classifier** Identify and categorize different dance styles or specific moves from dance videos. 148 | 149 | 150 | ## Feedback 151 | 152 | If you have any feedback, please reach out to us at **hriskikesh.yadav332@gmail.com** 153 | 154 | 155 | ## License 156 | 157 | [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) 158 | 159 | -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- 1 | import streamlit as st 2 | from twelvelabs import TwelveLabs 3 | from twelvelabs import SearchItem, SearchItemClipsItem, SearchResults 4 | import requests 5 | import os 6 | from dotenv import load_dotenv 7 | import threading 8 | from apscheduler.schedulers.background import BackgroundScheduler 9 | import time 10 | 11 | 12 | load_dotenv() 13 | 14 | def keep_alive(): 15 | print("Keep-alive ping executed at:", time.strftime("%Y-%m-%d %H:%M:%S")) 16 | try: 17 | app_url = os.getenv("STREAMLIT_APP_URL", "http://twelvelabs-olympics-app.streamlit.app/") 18 | requests.get(app_url, timeout=10) 19 | print("Successfully pinged the app") 20 | except Exception as e: 21 | print(f"Keep-alive ping failed: {str(e)}") 22 | 23 | 24 | scheduler = BackgroundScheduler() 25 | scheduler.add_job(keep_alive, 'interval', minutes=10) 26 | scheduler.start() 27 | 28 | API_KEY = os.getenv("API_KEY") 29 | BASE_URL = "https://api.twelvelabs.io/v1.2" 30 | 31 | INDEX_ID = os.getenv("INDEX_ID") 32 | 33 | client = TwelveLabs(api_key=API_KEY) 34 | 35 | page_element = """ 36 | 54 | """ 55 | st.markdown(page_element, unsafe_allow_html=True) 56 | 57 | 58 | @st.cache_data 59 | def get_initial_classes(): 60 | return [ 61 | {"name": "AquaticSports", "prompts": ["swimming competition", "diving event", "water polo match", "synchronized swimming", "open water swimming"]}, 62 | {"name": "AthleticEvents", "prompts": ["track and field", "marathon running", "long jump competition", "javelin throw", "high jump event"]}, 63 | {"name": "GymnasticsEvents", "prompts": ["artistic gymnastics", "rhythmic gymnastics", "trampoline gymnastics", "balance beam routine", "floor exercise performance"]}, 64 | {"name": "CombatSports", "prompts": ["boxing match", "judo competition", "wrestling bout", "taekwondo fight", "fencing duel"]}, 65 | {"name": "TeamSports", "prompts": ["basketball game", "volleyball match", "football (soccer) match", "handball game", "field hockey competition"]}, 66 | {"name": "CyclingSports", "prompts": ["road cycling race", "track cycling event", "mountain bike competition", "BMX racing", "cycling time trial"]}, 67 | {"name": "RacquetSports", "prompts": ["tennis match", "badminton game", "table tennis competition", "squash game", "tennis doubles match"]}, 68 | {"name": "RowingAndSailing", "prompts": ["rowing competition", "sailing race", "canoe sprint", "kayak event", "windsurfing competition"]} 69 | ] 70 | 71 | def get_custom_classes(): 72 | if 'custom_classes' not in st.session_state: 73 | st.session_state.custom_classes = [] 74 | return st.session_state.custom_classes 75 | 76 | def add_custom_class(name, prompts): 77 | custom_classes = get_custom_classes() 78 | custom_classes.append({"name": name, "prompts": prompts}) 79 | st.session_state.custom_classes = custom_classes 80 | st.session_state.new_class_added = True 81 | 82 | def search_videos(selected_prompts, selected_class_names): 83 | results_by_prompt = {} 84 | 85 | for i, prompt in enumerate(selected_prompts): 86 | try: 87 | class_name = next((class_name for class_name in selected_class_names 88 | for cls in get_initial_classes() + get_custom_classes() 89 | if cls["name"] == class_name and prompt in cls["prompts"]), "Unknown") 90 | 91 | result = client.search.create( 92 | index_id=INDEX_ID, 93 | search_options=["visual", "audio"], 94 | query_text=prompt, 95 | group_by="video", 96 | threshold="medium", 97 | operator="or", 98 | page_limit=5, 99 | sort_option="score" 100 | ) 101 | 102 | print(f"Search response for prompt '{prompt}':") 103 | print(f" Total results: {result.page_info.total_results}") 104 | print(f" Index ID: {result.search_pool.index_id}") 105 | print(f" Total count in pool: {result.search_pool.total_count}") 106 | 107 | if result.data and len(result.data) > 0: 108 | print(f" First result type: {type(result.data[0])}") 109 | if isinstance(result.data[0], SearchItem) and result.data[0].clips: 110 | clip = result.data[0].clips[0] 111 | print(f" Sample clip data: score={clip.score}, start={clip.start}, end={clip.end}") 112 | print(f" Confidence type: {type(clip.confidence)}") 113 | print(f" Confidence value: {clip.confidence}") 114 | 115 | results_by_prompt[prompt] = { 116 | "class_name": class_name, 117 | "result": result 118 | } 119 | except Exception as e: 120 | st.error(f"API Error for prompt '{prompt}': {str(e)}") 121 | print(f"Exception details: {type(e).__name__}: {str(e)}") 122 | 123 | return results_by_prompt 124 | 125 | def get_video_urls(video_ids): 126 | base_url = f"https://api.twelvelabs.io/v1.3/indexes/{INDEX_ID}/videos/{{}}" 127 | headers = {"x-api-key": API_KEY, "Content-Type": "application/json"} 128 | video_urls = {} 129 | 130 | for video_id in video_ids: 131 | try: 132 | response = requests.get(base_url.format(video_id), headers=headers) 133 | response.raise_for_status() 134 | data = response.json() 135 | if 'hls' in data and 'video_url' in data['hls']: 136 | video_urls[video_id] = data['hls']['video_url'] 137 | else: 138 | st.warning(f"No video URL found for video ID: {video_id}") 139 | except requests.exceptions.RequestException as e: 140 | st.error(f"Failed to get data for video ID: {video_id}. Error: {str(e)}") 141 | 142 | return video_urls 143 | 144 | def render_video(video_url, key): 145 | hls_player = f""" 146 | 147 |
148 | 149 |
150 | 168 | """ 169 | st.components.v1.html(hls_player, height=400) 170 | 171 | 172 | def main(): 173 | 174 | st.markdown(""" 175 | 269 | """, unsafe_allow_html=True) 270 | 271 | st.markdown('

Olympics Classification w/t Twelve Labs

', unsafe_allow_html=True) 272 | 273 | CLASSES = get_initial_classes() + get_custom_classes() 274 | 275 | 276 | tab1, tab2 = st.tabs(["Search Videos", "Add Custom Class"]) 277 | 278 | with tab1: 279 | st.markdown('

Search Videos

', unsafe_allow_html=True) 280 | with st.container(): 281 | class_names = [cls["name"] for cls in CLASSES] 282 | selected_classes = st.multiselect("Choose one or more Olympic sports categories:", class_names) 283 | 284 | if st.button("Search Videos", key="search_button"): 285 | if selected_classes: 286 | with st.spinner("Searching videos..."): 287 | selected_prompts = [] 288 | for cls in CLASSES: 289 | if cls["name"] in selected_classes: 290 | selected_prompts.extend(cls["prompts"]) 291 | 292 | results_by_prompt = search_videos(selected_prompts, selected_classes) 293 | 294 | video_ids = set() 295 | for prompt_data in results_by_prompt.values(): 296 | result = prompt_data["result"] 297 | for item in result.data: 298 | if isinstance(item, SearchItem): 299 | video_ids.add(item.id) 300 | else: 301 | video_ids.add(item.video_id) 302 | 303 | video_urls = get_video_urls(list(video_ids)) 304 | 305 | if not video_ids: 306 | st.warning("No videos found matching your search criteria.") 307 | else: 308 | st.success(f"Found {len(video_ids)} unique videos across {len(selected_prompts)} search prompts") 309 | 310 | results_by_class = {} 311 | for prompt, prompt_data in results_by_prompt.items(): 312 | class_name = prompt_data["class_name"] 313 | if class_name not in results_by_class: 314 | results_by_class[class_name] = [] 315 | results_by_class[class_name].append({ 316 | "prompt": prompt, 317 | "result": prompt_data["result"] 318 | }) 319 | 320 | for class_name, class_results in results_by_class.items(): 321 | st.markdown(f'
{class_name}
', unsafe_allow_html=True) 322 | 323 | for prompt_result in class_results: 324 | prompt = prompt_result["prompt"] 325 | result = prompt_result["result"] 326 | 327 | st.markdown(f'
Results for: "{prompt}"
', unsafe_allow_html=True) 328 | 329 | video_count = 0 330 | for item in result.data: 331 | if isinstance(item, SearchItem): 332 | video_id = item.id 333 | if not item.clips: 334 | continue 335 | 336 | with st.expander(f"Video {video_count+1}: {video_id}", expanded=(video_count == 0)): 337 | st.markdown('
', unsafe_allow_html=True) 338 | 339 | for i, clip in enumerate(item.clips[:3]): # Limit to top 3 clips 340 | confidence_class = "confidence-high" if clip.confidence == "high" else "confidence-medium" if clip.confidence == "medium" else "confidence-low" 341 | 342 | st.markdown(f""" 343 |
344 | Clip {i+1}: {float(clip.start):.1f}s - {float(clip.end):.1f}s | 345 | Score: {float(clip.score):.1f} | 346 | Confidence: {clip.confidence} 347 |
348 | """, unsafe_allow_html=True) 349 | 350 | if video_id in video_urls: 351 | render_video(video_urls[video_id], f"{class_name}-{prompt}-{video_count}") 352 | else: 353 | st.warning("Video URL not available. Unable to render video.") 354 | 355 | st.markdown('
', unsafe_allow_html=True) 356 | 357 | video_count += 1 358 | if video_count >= 3: 359 | break 360 | 361 | if video_count == 0: 362 | st.info(f"No videos found for prompt: {prompt}") 363 | else: 364 | st.warning("Please select at least one class.") 365 | st.markdown('', unsafe_allow_html=True) 366 | 367 | with tab2: 368 | st.markdown('

Add Custom Class

', unsafe_allow_html=True) 369 | with st.container(): 370 | custom_class_name = st.text_input("Enter custom class name") 371 | custom_class_prompts = st.text_input("Enter custom class prompts (comma-separated)") 372 | if st.button("Add Custom Class"): 373 | if custom_class_name and custom_class_prompts: 374 | prompts_list = [p.strip() for p in custom_class_prompts.split(',')] 375 | add_custom_class(custom_class_name, prompts_list) 376 | st.success(f"Custom class '{custom_class_name}' added successfully!") 377 | st.rerun() 378 | else: 379 | st.warning("Please enter both class name and prompts.") 380 | st.markdown('', unsafe_allow_html=True) 381 | 382 | if st.session_state.get('new_class_added', False): 383 | st.session_state.new_class_added = False 384 | st.rerun() 385 | 386 | if __name__ == "__main__": 387 | try: 388 | main() 389 | except Exception as e: 390 | st.error(f"An error occurred: {str(e)}") 391 | 392 | import atexit 393 | atexit.register(lambda: scheduler.shutdown()) 394 | --------------------------------------------------------------------------------