├── LICENSE ├── README.md ├── download.py ├── readme ├── external_dataset.md ├── personpath22.png ├── personpath22_small.gif ├── public_detection.md └── results.md └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![alt text](readme/personpath22.png) 2 | 3 | # Large scale Real-world Multi-Person Tracking 4 | 5 | This repo contains utility codes for the PersonPath22 dataset introduced in the 6 | following ECCV 2022 paper. 7 | 8 | > [**Large Scale Real-world Multi-Person Tracking**](https://www.amazon.science/publications/large-scale-real-world-multi-person-tracking), 9 | > Bing Shuai, Alessandro Bergamo, Uta Buechler, Andrew Berneshawi, Alyssa Boden and Joseph Tighe 10 | 11 | 12 | @inproceedings{personpath22, 13 | title={Large Scale Real-world Multi-Person Tracking}, 14 | author={Shuai, Bing and Bergamo, Alessandro and Buechler, Uta and Berneshawi, Andrew and Boden, Alyssa and Tighe, Joseph}, 15 | booktitle={ECCV}, 16 | year={2022}, 17 | organization={Springer} 18 | } 19 | 20 | 21 | ![alt text](readme/personpath22_small.gif) 22 | **We ask all researchers to re-download the videos / annotations, if your data copy was downloaded before 10/21/2022 (MM/DD/YYYY).** 23 | 24 | [PersonPath22 Homepage](https://amazon-science.github.io/tracking-dataset/personpath22.html)) 25 | 26 | 27 | ## Leaderboard 28 | We encourage all researchers report their evaluation results on the [leaderboard](https://paperswithcode.com/sota/multi-object-tracking-on-personpath22) 29 | on paperswithcode.com. 30 | 31 | Please refer to [Results](readme/results.md) to find the raw predictions for models that are used in the paper. 32 | 33 | ## Requirements 34 | The code should be run with python 3. We use the aws cli to help download the data, please first run 35 | `python3 -m pip install -r requirements.txt` to install it. 36 | 37 | The code also requires ffmpeg to process some videos, please install it for your respective system 38 | and make sure ffmpeg and ffprobe are available in the command line path. Use your package manager 39 | or find other download options here: https://ffmpeg.org/download.html 40 | 41 | ## Data 42 | Data can be downloaded using the download.py script in this folder. Simply run: 43 | `python3 download.py` 44 | It will automatically download the dataset videos and annotations and extract them under 45 | REPO_ROOT/dataset/raw_data REPO_ROOT/dataset/annotations respectively. 46 | 47 | There are a few videos from [external datasets](readme/external_dataset.md) that we don't have right to re-distribute, 48 | we also provide the following means to smoothly download and process them. 49 | 50 | ## Annotation Format 51 | Annotations are provided in the gluoncv motion dataset format: 52 | https://github.com/dmlc/gluon-cv/tree/master/gluoncv/torch/data/gluoncv_motion_dataset 53 | 54 | Annotations are provided with visible and amodal bounding boxes in anno_visible_2022.json and anno_amodal_2022.json 55 | respectively. 56 | 57 | ## Public Detection 58 | Please refer to [Public detection](readme/public_detection.md) for details 59 | 60 | ## Version 61 | [Updated 10/21/2022] Bugs were fixed related to Pixabay and MEVA videos in the early downloading script. 62 | 63 | [Updated 09/28/2023] Official pre-trained detection results for "public detection" evaluation protocol. 64 | 65 | [To Appear] Official script to convert gluoncv motion dataset format to MOTChallenge and MSCOCO dataset format. 66 | 67 | [To Appear] The full FPS annotation. (The existing annotation only includes key frames which are sampled at 5FPS.) 68 | 69 | 70 | ## License 71 | This provided code is licensed under the Apache-2.0 License. 72 | 73 | The retrieved videos and their annotations are licensed under the CC-BY-NC-4.0 License. 74 | 75 | -------------------------------------------------------------------------------- /download.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import requests 4 | import shutil 5 | import subprocess 6 | import sys 7 | import tempfile 8 | import xml.etree.ElementTree as XMLTree 9 | import zipfile 10 | 11 | from pathlib import Path 12 | from tqdm import tqdm 13 | 14 | FFMPEG_BASE_CMD = ["ffmpeg", "-y", "-hide_banner", "-loglevel", "error"] 15 | 16 | 17 | def copy_to_target(src_path, tgt_path): 18 | # Try to get the video fps as a test before copying 19 | get_vid_fps(src_path) 20 | src_path = Path(src_path) 21 | src_tmp_path = src_path.with_name(f"{src_path.stem}_copy{src_path.suffix}") 22 | shutil.copy2(src_path, src_tmp_path) 23 | src_tmp_path.rename(tgt_path) 24 | 25 | 26 | def run_with_tmp_dir(cmd_fn, out_path): 27 | if isinstance(cmd_fn, list): 28 | cmd = cmd_fn 29 | def cmd_fn(tmp_file_path): 30 | fullcmd = cmd + [tmp_file_path] 31 | return subprocess.run(fullcmd, check=True) 32 | with tempfile.TemporaryDirectory() as tmp_path: 33 | tmp_file_path = Path(tmp_path, Path(out_path).name) 34 | ret = cmd_fn(tmp_file_path) 35 | subprocess.run(["mv", tmp_file_path, out_path], check=True) 36 | return ret 37 | 38 | 39 | def write_img_files_to_vid_ffmpeg(out_file, in_files, fps): 40 | input_str = "'\nfile '".join([str(p) for p in in_files]) 41 | input_str = "file '" + input_str + "'\n" 42 | 43 | cmd = [*FFMPEG_BASE_CMD, 44 | "-f", "concat", "-safe", "0", "-r", str(fps), "-i", "/dev/stdin", 45 | "-vf", "pad=ceil(iw/2)*2:ceil(ih/2)*2"] 46 | 47 | def run_cmd(tmp_file_path): 48 | fullcmd = cmd + [tmp_file_path] 49 | return subprocess.run(fullcmd, input=input_str.encode('utf-8'), check=True) 50 | 51 | return run_with_tmp_dir(run_cmd, out_file) 52 | 53 | 54 | def crop_vid_ffmpeg(vid_in_path, crop_out_path, ts_ms_start, ts_ms_end, fps=None, end_pad_ms=1): 55 | # Pad end time with 1 since end times are exclusive according to this SO post: https://stackoverflow.com/a/52916510 56 | t1_secs, t2_secs = [t / 1000.0 for t in (ts_ms_start, ts_ms_end + end_pad_ms)] 57 | # Hack to not provide a start seek if t1_secs is 0 due to an ffmpeg issue where it can't seek to 0 on some videos 58 | start_seek_args = ["-ss", "{:.3f}".format(t1_secs)] if t1_secs else [] 59 | fps_args = ["-r", str(fps)] if fps else [] 60 | cmd = [ 61 | *FFMPEG_BASE_CMD, 62 | *start_seek_args, 63 | "-i", vid_in_path, 64 | "-t", "{:.3f}".format(t2_secs - t1_secs), 65 | *fps_args, 66 | "-strict", "experimental", 67 | ] 68 | 69 | return run_with_tmp_dir(cmd, crop_out_path) 70 | 71 | 72 | def convert_vid_ffmpeg(in_path, out_path, crf=None, copy=False, fps=None, fps_filter=False): 73 | # muxing queue size bug workaround: https://stackoverflow.com/questions/49686244/ffmpeg-too-many-packets-buffered-for-output-stream-01 74 | crf_args = ["-crf", str(crf)] if crf else [] 75 | copy_args = ["-c", "copy"] if copy else [] 76 | if fps: 77 | if fps_filter: 78 | fps_args = ["-filter:v", f"fps={fps}"] 79 | else: 80 | fps_args = ["-r", str(fps)] 81 | else: 82 | fps_args = [] 83 | cmd = [*FFMPEG_BASE_CMD, "-i", in_path, *crf_args, *copy_args, *fps_args, "-max_muxing_queue_size", "99999"] 84 | return run_with_tmp_dir(cmd, out_path) 85 | 86 | 87 | def get_vid_fps(vid_path): 88 | probe_json = subprocess.run( 89 | ["ffprobe", "-v", "error", "-select_streams", "v", "-of", "json", "-show_entries", 90 | "stream=r_frame_rate", vid_path], stdout=subprocess.PIPE, check=True) 91 | probe_obj = json.loads(probe_json.stdout) 92 | fps_str = probe_obj["streams"][0]["r_frame_rate"] 93 | fps_numer, fps_denom = (int(s) for s in fps_str.split("/")) 94 | fps = fps_numer / fps_denom 95 | return fps 96 | 97 | 98 | def get_pixabay_vids(pixabay_mapping, videos_root): 99 | pixabay_url_home = "https://pixabay.com" 100 | pixabay_download_template = pixabay_url_home + "/videos/download/video-{}_source.mp4?attachment" 101 | 102 | out_dir = videos_root / "pixabay" 103 | out_dir.mkdir(exist_ok=True) 104 | 105 | print(f"Downloading Pixabay") 106 | 107 | # Approach from: https://stackoverflow.com/a/63770219 108 | headers = { 109 | 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 110 | "Accept-Language": "en-US,en;q=0.9", 111 | "Cache-Control": "no-cache", 112 | } 113 | 114 | session = requests.Session() 115 | session.headers.update(headers) 116 | 117 | # --- get cookies --- 118 | r = session.get(pixabay_url_home) 119 | 120 | # --- get videos --- 121 | for pixa_id, uid in tqdm(pixabay_mapping.items()): 122 | try: 123 | download_out_path = out_dir / (pixa_id + ".mp4") 124 | final_vid_path = videos_root / uid 125 | if not download_out_path.exists(): 126 | print(f"Downloading {download_out_path}") 127 | pixa_vid_num = pixa_id.rsplit("-", 1)[-1] 128 | url = pixabay_download_template.format(pixa_vid_num) 129 | response = session.get(url) 130 | with open(download_out_path, "wb") as f: 131 | f.write(response.content) 132 | 133 | fps = get_vid_fps(download_out_path) 134 | if fps > 31: 135 | new_fps = fps / 2 136 | new_fps_path = download_out_path.with_name(f"{download_out_path.stem}_newfps{download_out_path.suffix}") 137 | if not new_fps_path.exists(): 138 | print(f"Downsampling fps of {download_out_path}") 139 | # The fps filter seems to align properly with the orig video while -r has slight misalignments 140 | convert_vid_ffmpeg(download_out_path, new_fps_path, fps=new_fps, fps_filter=True) 141 | else: 142 | new_fps_path = download_out_path 143 | 144 | copy_to_target(new_fps_path, final_vid_path) 145 | except Exception as e: 146 | print(f"Failed to process pixabay video: {pixa_id}, for uid: {uid}, exception: {e}", file=sys.stderr) 147 | 148 | 149 | def get_pathtrack_vids(pathtrack_mapping, videos_root): 150 | pathtrack_download_url = "https://data.vision.ee.ethz.ch/daid/MOT/pathtrack_release_v1.0.zip" 151 | 152 | out_dir = videos_root / "pathtrack" 153 | out_dir.mkdir(exist_ok=True) 154 | 155 | print(f"Downloading PathTrack") 156 | 157 | zip_out_path = out_dir / "pathtrack.zip" 158 | extracted_path = out_dir / "pathtrack_release" 159 | if not extracted_path.exists(): 160 | if not zip_out_path.exists(): 161 | download = True 162 | else: 163 | try: 164 | print(f"Testing zip: {zip_out_path}") 165 | with zipfile.ZipFile(zip_out_path) as zip_ref: 166 | download = zip_ref.testzip() is not None 167 | except zipfile.BadZipFile as e: 168 | download = True 169 | if download: 170 | # Streaming as described here: https://stackoverflow.com/a/37573701 171 | print(f"Downloading pathtrack to {zip_out_path}") 172 | print(f"Please note this zip is over 30GB and can take hours to download") 173 | response = requests.get(pathtrack_download_url, stream=True) 174 | total_size_in_bytes = int(response.headers.get('content-length', 0)) 175 | block_size = 1024 * 1024 * 10 176 | progress_bar = tqdm(total=total_size_in_bytes, unit='iB', unit_scale=True) 177 | with open(zip_out_path, 'wb') as f: 178 | for data in response.iter_content(block_size): 179 | progress_bar.update(len(data)) 180 | f.write(data) 181 | progress_bar.close() 182 | if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes: 183 | raise Exception("Error downloading pathtrack") 184 | print(f"Extracting pathtrack {zip_out_path}", flush=True) 185 | with zipfile.ZipFile(zip_out_path) as zip_ref: 186 | zip_ref.extractall(zip_out_path.parent) 187 | 188 | for patht_id, uid in tqdm(pathtrack_mapping.items()): 189 | try: 190 | patht_root = extracted_path / "train" / patht_id 191 | if not patht_root.exists(): 192 | patht_root = extracted_path / "test" / patht_id 193 | ffmpeg_out_file = patht_root / (patht_id + ".mp4") 194 | final_vid_path = videos_root / uid 195 | 196 | if not ffmpeg_out_file.exists(): 197 | info_path = patht_root / "info.xml" 198 | info_xml = XMLTree.parse(info_path) 199 | info_root = info_xml.getroot().find("doc") 200 | def get_text(tag): 201 | return info_root.find(tag).text 202 | fps = float(get_text('fps')) 203 | 204 | img_path = patht_root / "img1" 205 | input_paths = sorted(img_path.glob("*.jpg")) 206 | write_img_files_to_vid_ffmpeg(ffmpeg_out_file, input_paths, fps) 207 | copy_to_target(ffmpeg_out_file, final_vid_path) 208 | except Exception as e: 209 | print(f"Failed to process pathtrack video: {patht_id}, for uid: {uid}, exception: {e}", file=sys.stderr) 210 | 211 | 212 | def get_virat_vids(virat_id_to_uid_and_urlid, videos_root): 213 | # virat_page = "https://data.kitware.com/#collection/56f56db28d777f753209ba9f/folder/56f581ce8d777f753209ca43" 214 | # e.g. download link https://data.kitware.com/api/v1/item/56f587f18d777f753209cc33/download 215 | # be careful of the 'file' urls instead of 'item', they work but the code is actually different 216 | virat_download_template = "https://data.kitware.com/api/v1/item/{}/download" 217 | 218 | out_dir = videos_root / "virat" 219 | out_dir.mkdir(exist_ok=True) 220 | 221 | print(f"Downloading Virat") 222 | 223 | for virat_id, (uid, down_code) in tqdm(virat_id_to_uid_and_urlid.items()): 224 | try: 225 | download_out_path = out_dir / virat_id 226 | final_vid_path = videos_root / uid 227 | if not download_out_path.exists(): 228 | print(f"Downloading {download_out_path}") 229 | url = virat_download_template.format(down_code) 230 | response = requests.get(url) 231 | with open(download_out_path, "wb") as f: 232 | f.write(response.content) 233 | copy_to_target(download_out_path, final_vid_path) 234 | except Exception as e: 235 | print(f"Failed to process virat video: {virat_id}, for uid: {uid}, exception: {e}", file=sys.stderr) 236 | 237 | 238 | def get_meva_vids(meva_mapping, videos_root): 239 | # meva_s3_url = "s3://mevadata-public-01/drops-123-r13/" 240 | # aws s3 ls s3://mevadata-public-01/drops-123-r13/2018-03-07/16/2018-03-07.16-50-00.16-55-00.bus.G331.r13.avi 241 | meva_download_template = "s3://mevadata-public-01/drops-123-r13/{date}/{hour}/{fname}" 242 | 243 | out_dir = videos_root / "meva" 244 | out_dir.mkdir(exist_ok=True) 245 | 246 | print(f"Downloading Meva") 247 | 248 | for meva_id, uid in tqdm(meva_mapping.items()): 249 | try: 250 | meva_fname_base, fname_timestamp = meva_id.rsplit("_", 1) 251 | ts_ms_start, ts_ms_end = [int(x) for x in Path(fname_timestamp).stem.split("-")] 252 | meva_url_fname = meva_fname_base + ".r13.avi" 253 | download_out_path = out_dir / (meva_url_fname) 254 | final_vid_path = videos_root / uid 255 | if not download_out_path.exists(): 256 | # print(f"Downloading {download_out_path}") 257 | meva_date = meva_url_fname[:10] 258 | meva_hour = meva_url_fname[20:22] # Don't use [11:13], that's not the hour used actually 259 | url = meva_download_template.format(date=meva_date, hour=meva_hour, fname=meva_url_fname) 260 | subprocess.run([sys.executable, 261 | "-m", "awscli", "s3", "cp", "--no-sign-request", url, download_out_path], check=True) 262 | 263 | mp4_out_path = download_out_path.with_suffix(".mp4") 264 | # mp4_out_path = download_out_path 265 | if not mp4_out_path.exists(): 266 | print(f"Converting {download_out_path.name} to mp4") 267 | # Convert to mp4 first or there are some weird time issues (e.g. only 2:59 instead of 3:00) 268 | convert_vid_ffmpeg(download_out_path, mp4_out_path, copy=True) 269 | # convert_vid_ffmpeg(download_out_path, mp4_out_path, crf=17) 270 | 271 | crop_out_path = out_dir / meva_id 272 | if not crop_out_path.exists(): 273 | print(f"Cropping {mp4_out_path.name} to {crop_out_path.name}") 274 | # Original dataset meva videos were cropped with 15 fps 275 | crop_vid_ffmpeg(mp4_out_path, crop_out_path, ts_ms_start, ts_ms_end, fps=15, end_pad_ms=0) 276 | copy_to_target(crop_out_path, final_vid_path) 277 | except Exception as e: 278 | print(f"Failed to process meva video: {meva_id}, for uid: {uid}, exception: {e}", file=sys.stderr) 279 | 280 | 281 | virat_id_to_uid_and_urlid = { 282 | "VIRAT_S_010201_04_000374_000469.mp4": ("uid_vid_00144.mp4", "56f587f18d777f753209cc33"), 283 | "VIRAT_S_010203_06_000620_000760.mp4": ("uid_vid_00145.mp4", "56f587fd8d777f753209cc63"), 284 | "VIRAT_S_010203_08_000895_000975.mp4": ("uid_vid_00146.mp4", "56f588008d777f753209cc69"), 285 | "VIRAT_S_010004_06_000547_000715.mp4": ("uid_vid_00147.mp4", "56f587a38d777f753209cb58"), 286 | "VIRAT_S_010208_06_000671_000744.mp4": ("uid_vid_00148.mp4", "56f588208d777f753209ccdb"), 287 | "VIRAT_S_010200_07_000748_000837.mp4": ("uid_vid_00149.mp4", "56f587ec8d777f753209cc1b"), 288 | "VIRAT_S_010004_08_000811_000920.mp4": ("uid_vid_00150.mp4", "56f587a68d777f753209cb5e"), 289 | "VIRAT_S_010200_00_000060_000218.mp4": ("uid_vid_00151.mp4", "56f587e78d777f753209cc06"), 290 | "VIRAT_S_010203_07_000775_000869.mp4": ("uid_vid_00152.mp4", "56f587ff8d777f753209cc66"), 291 | } 292 | 293 | external_video_id_mapping = { 294 | "Pixabay": { 295 | "lord-street-liverpool-city-28889": "uid_vid_00162.mp4", 296 | "people-alley-street-ukraine-bike-39836": "uid_vid_00232.mp4", 297 | "india-street-busy-rickshaw-people-3175": "uid_vid_00233.mp4", 298 | "lima-building-street-people-32242": "uid_vid_00234.mp4", 299 | "street-people-city-urban-walking-33572": "uid_vid_00235.mp4", 300 | "thailand-bangkok-airport-aircraft-30872": "uid_vid_00236.mp4" 301 | }, 302 | "PathTrack": { 303 | "yZde6KWbWuM_0_7": "uid_vid_00118.mp4", 304 | "1ifHYTW_AU0_674_680": "uid_vid_00119.mp4", 305 | "OabhIVN8Pps_1_22": "uid_vid_00120.mp4", 306 | "2DiQUX11YaY_4_11": "uid_vid_00121.mp4", 307 | "HlvCV1YtJFI_127_132": "uid_vid_00122.mp4", 308 | "Twfy_jWEDt4_40_46": "uid_vid_00123.mp4", 309 | "03tAll3Rnb8_145_160": "uid_vid_00124.mp4", 310 | "EjekbTQT2dw_129_155": "uid_vid_00125.mp4", 311 | "1k9qcUaw_ZY_92_106": "uid_vid_00126.mp4", 312 | "CcevP4Dgocw_55_66": "uid_vid_00127.mp4", 313 | "0Xtp77A4zF4_8_84": "uid_vid_00128.mp4", 314 | "D8RppPVMWAM_0_18": "uid_vid_00129.mp4", 315 | "Bqn_GzsrDqM_174_182": "uid_vid_00130.mp4", 316 | "CqusBYOcaA4_639_647": "uid_vid_00131.mp4", 317 | "xnHDu-oW7_Y_871_879": "uid_vid_00132.mp4", 318 | "6EWOHAs2CeY_116_127": "uid_vid_00133.mp4", 319 | "wt9UHXJ1kRU_0_15": "uid_vid_00134.mp4", 320 | "881uK5SDnFU_26_55": "uid_vid_00135.mp4", 321 | "L4H-nBgiFmg_212_226": "uid_vid_00136.mp4", 322 | "2_tnrvMIt2E_99_135": "uid_vid_00137.mp4", 323 | "HzOiJ87yNIg_237_262": "uid_vid_00140.mp4", 324 | "h2hwIDQkzBE_12_27": "uid_vid_00141.mp4", 325 | "WAUYbr5YE2U_0_13": "uid_vid_00142.mp4", 326 | "WKbN-iY7DSU_42_53": "uid_vid_00143.mp4" 327 | }, 328 | "Meva": { 329 | "2018-03-07.16-50-00.16-55-00.bus.G331_0-180000.mp4": "uid_vid_00102.mp4", 330 | "2018-03-15.15-10-00.15-15-00.bus.G331_180000-300300.mp4": "uid_vid_00103.mp4", 331 | "2018-03-13.17-40-00.17-41-19.school.G421_0-79500.mp4": "uid_vid_00104.mp4", 332 | "2018-03-09.10-15-01.10-20-01.school.G420_180000-300033.mp4": "uid_vid_00105.mp4", 333 | "2018-03-15.15-35-00.15-40-00.school.G421_180000-300066.mp4": "uid_vid_00106.mp4", 334 | "2018-03-07.16-50-00.16-55-00.bus.G331_180000-300166.mp4": "uid_vid_00107.mp4", 335 | "2018-03-13.15-55-00.16-00-00.school.G421_0-180000.mp4": "uid_vid_00108.mp4", 336 | "2018-03-07.16-55-01.17-00-00.bus.G508_0-180000.mp4": "uid_vid_00109.mp4", 337 | "2018-03-07.16-50-00.16-55-00.school.G421_180000-300066.mp4": "uid_vid_00110.mp4", 338 | "2018-03-11.16-35-01.16-40-01.school.G420_180000-300033.mp4": "uid_vid_00111.mp4", 339 | "2018-03-15.15-10-00.15-15-00.bus.G331_0-180000.mp4": "uid_vid_00112.mp4", 340 | "2018-03-15.15-35-00.15-40-00.school.G421_0-180000.mp4": "uid_vid_00113.mp4", 341 | "2018-03-13.15-55-00.16-00-00.school.G421_180000-300000.mp4": "uid_vid_00114.mp4", 342 | "2018-03-15.14-50-00.14-55-00.bus.G508_0-180000.mp4": "uid_vid_00115.mp4", 343 | "2018-03-15.14-50-00.14-55-00.bus.G508_180000-300000.mp4": "uid_vid_00116.mp4", 344 | "2018-03-07.16-50-00.16-55-00.school.G421_0-180000.mp4": "uid_vid_00117.mp4" 345 | }, 346 | "Virat": {k: v[0] for k, v in virat_id_to_uid_and_urlid.items()}, 347 | } 348 | 349 | 350 | def download_core_dataset(s3_base_url, dataset_dir): 351 | anno_files = ["anno_amodal.zip", "anno_visible.zip", "splits.json"] 352 | anno_subpaths = [Path("annotation", v) for v in anno_files] 353 | video_subpaths = ["raw_data/videos.zip"] 354 | core_subpaths = anno_subpaths + video_subpaths 355 | for subpath in core_subpaths: 356 | fullpath = dataset_dir / subpath 357 | if not fullpath.exists(): 358 | s3_url = os.path.join(s3_base_url, str(subpath)) 359 | print(f"Downloading {s3_url} to {fullpath}") 360 | cmd = [sys.executable, "-m", "awscli", "s3", "cp", "--no-sign-request", s3_url, fullpath] 361 | subprocess.run(cmd, check=True) 362 | if fullpath.suffix == ".zip": 363 | print(f"Extracting {fullpath}", flush=True) 364 | with zipfile.ZipFile(fullpath, 'r') as zip_ref: 365 | zip_ref.extractall(fullpath.parent) 366 | 367 | 368 | def get_missing_and_corrupt_videos(videos_root): 369 | expected_uid_nums = set(list(range(237)) + [99999]) - {138, 139} 370 | expected_uid_vid_paths = set(videos_root / f"uid_vid_{str(i).zfill(5)}.mp4" for i in expected_uid_nums) 371 | # uid_vid_paths = set(videos_root.glob("uid_vid_*.mp4")) 372 | missing_vids = set() 373 | corrupt_vids = set() 374 | for expected_uid_vid_path in tqdm(expected_uid_vid_paths): 375 | if expected_uid_vid_path.exists(): 376 | try: 377 | get_vid_fps(expected_uid_vid_path) 378 | except Exception as e: 379 | corrupt_vids.add(expected_uid_vid_path.name) 380 | else: 381 | missing_vids.add(expected_uid_vid_path.name) 382 | 383 | return missing_vids, corrupt_vids 384 | 385 | 386 | if __name__ == "__main__": 387 | S3_URL = "s3://tracking-dataset-eccv-2022/dataset/" 388 | 389 | script_dir = Path(__file__).parent.resolve() 390 | dataset_dir = script_dir / "dataset" / "personpath22" 391 | videos_root = dataset_dir / "raw_data" 392 | 393 | print(f"Downloading dataset under {dataset_dir}. Please note there are many videos and this can take hours to" 394 | f" complete.") 395 | 396 | download_core_dataset(S3_URL, dataset_dir) 397 | get_pixabay_vids(external_video_id_mapping["Pixabay"], videos_root) 398 | get_virat_vids(virat_id_to_uid_and_urlid, videos_root) 399 | get_meva_vids(external_video_id_mapping["Meva"], videos_root) 400 | get_pathtrack_vids(external_video_id_mapping["PathTrack"], videos_root) 401 | 402 | print("Checking videos...") 403 | missing_vids, corrupt_vids = get_missing_and_corrupt_videos(videos_root) 404 | if missing_vids or corrupt_vids: 405 | print(f"Warning!!! There are missing and/or corrupt videos under {videos_root}," 406 | f" missing: {missing_vids}, corrupt: {corrupt_vids}." 407 | f" Please check the above output in the script to see the respective errors and refer to README.md for" 408 | f" potential fixes.", file=sys.stderr) 409 | else: 410 | print(f"Check successful. There are temporary files under the videos root path {videos_root}," 411 | f" in the folders: virat, meva, pixabay, and pathtrack, you can remove these folders once you have" 412 | f" verified all 236 uid*.mp4 videos are present and working") 413 | -------------------------------------------------------------------------------- /readme/external_dataset.md: -------------------------------------------------------------------------------- 1 | ## External Datasets ## 2 | 3 | The videos from the below external datasets (pixabay, pathtrack, meva, and virat) should automatically be downloaded and 4 | processed by the download script. 5 | All video paths are under: `REPO_ROOT/dataset/personpath22/raw_data/` and the external datasets will first be downloaded 6 | under a subpath, e.g. `raw_data/pixabay/` , before being copied to their appropriate uid_vid_*.mp4 file. 7 | 8 | If however there is an exception, you might see something like: 9 | ``` 10 | Failed to process pixabay video: people-alley-street-ukraine-bike-39836, for uid: uid_vid_00232.mp4 exception: 11 | Command '['ffprobe', '-v', ....]' returned non-zero exit status 1. 12 | ``` 13 | Then a video may have failed to download correctly. You can delete that external dataset's respective subfolder 14 | under raw_data and try running download.py again, which may work if the error was temporary. 15 | If the error is not temporary, as can happen with pixabay for example (certain hosts seem to get blocked), then please 16 | follow the manual steps below for the appropriate external dataset. 17 | 18 | Note: If you are getting errors about ffprobe and ffmpeg not found then please make sure you have `ffmpeg` and `ffprobe` 19 | available on the command line. 20 | 21 | ### Manual steps: ### 22 | 23 | #### Pixabay: #### 24 | For the keys in the mapping below, append them to: `https://pixabay.com/videos/` and you can download the videos from 25 | the resulting page. E.g. https://pixabay.com/videos/lord-street-liverpool-city-28889 , there is a free download button 26 | on the right, please download at the 1920x1080 resolution. Move the downloaded files under the `raw_data/pixabay` folder 27 | with the same name as the key in the mapping + .mp4, (e.g. `lord-street-liverpool-city-28889.mp4`). Rerun the 28 | download script to have them processed and copied to the appropriate path. 29 | 30 | #### PathTrack: #### 31 | Download the whole dataset via this link: 32 | https://data.vision.ee.ethz.ch/daid/MOT/pathtrack_release_v1.0.zip 33 | , and copy this zip to the `raw_data/pathtrack/` folder. Run download.py again which should now extract and process it. 34 | If you are still getting errors make sure the zip is not corrupt. Note that we use python here to extract the zip for 35 | cross-platform compatibility, if you want to extract faster you can use your tool of choice to extract before running 36 | the download script again. 37 | 38 | #### Meva: #### 39 | For the keys in the mapping go to the appropriate s3 subpath from here "s3://mevadata-public-01/drops-123-r13/" 40 | For example for the first key 41 | `"2018-03-07.16-50-00.16-55-00.bus.G331_0-180000.mp4"` the s3 path is: 42 | `s3://mevadata-public-01/drops-123-r13/2018-03-07/16/2018-03-07.16-50-00.16-55-00.bus.G331.r13.avi` 43 | (remove the time interval _a-b and replace .mp4 with .r13.avi, use the date and ending hour from the filename as the 44 | parent folders). Some keys are mapping to the same source video (original video cropped into 2), so you will not need to 45 | download a video for every key. 46 | Download the videos and put them under the `raw_data/meva` folder. Rerun the download script to process and crop them. 47 | 48 | #### Virat: #### 49 | For the keys in the mapping look for them in this page to download them: https://data.kitware.com/#collection/56f56db28d777f753209ba9f/folder/56f581ce8d777f753209ca43 50 | (go to the bottom and press show more until all are showing). Put the downloaded videos in the `raw_data/virat` folder 51 | and rerun the download script to copy them to the correct uid path. 52 | 53 | ### Video id mapping: ### 54 | ```python 55 | { 56 | "Pixabay": { 57 | "lord-street-liverpool-city-28889": "uid_vid_00162.mp4", 58 | "people-alley-street-ukraine-bike-39836": "uid_vid_00232.mp4", 59 | "india-street-busy-rickshaw-people-3175": "uid_vid_00233.mp4", 60 | "lima-building-street-people-32242": "uid_vid_00234.mp4", 61 | "street-people-city-urban-walking-33572": "uid_vid_00235.mp4", 62 | "thailand-bangkok-airport-aircraft-30872": "uid_vid_00236.mp4" 63 | }, 64 | "PathTrack": { 65 | "yZde6KWbWuM_0_7": "uid_vid_00118.mp4", 66 | "1ifHYTW_AU0_674_680": "uid_vid_00119.mp4", 67 | "OabhIVN8Pps_1_22": "uid_vid_00120.mp4", 68 | "2DiQUX11YaY_4_11": "uid_vid_00121.mp4", 69 | "HlvCV1YtJFI_127_132": "uid_vid_00122.mp4", 70 | "Twfy_jWEDt4_40_46": "uid_vid_00123.mp4", 71 | "03tAll3Rnb8_145_160": "uid_vid_00124.mp4", 72 | "EjekbTQT2dw_129_155": "uid_vid_00125.mp4", 73 | "1k9qcUaw_ZY_92_106": "uid_vid_00126.mp4", 74 | "CcevP4Dgocw_55_66": "uid_vid_00127.mp4", 75 | "0Xtp77A4zF4_8_84": "uid_vid_00128.mp4", 76 | "D8RppPVMWAM_0_18": "uid_vid_00129.mp4", 77 | "Bqn_GzsrDqM_174_182": "uid_vid_00130.mp4", 78 | "CqusBYOcaA4_639_647": "uid_vid_00131.mp4", 79 | "xnHDu-oW7_Y_871_879": "uid_vid_00132.mp4", 80 | "6EWOHAs2CeY_116_127": "uid_vid_00133.mp4", 81 | "wt9UHXJ1kRU_0_15": "uid_vid_00134.mp4", 82 | "881uK5SDnFU_26_55": "uid_vid_00135.mp4", 83 | "L4H-nBgiFmg_212_226": "uid_vid_00136.mp4", 84 | "2_tnrvMIt2E_99_135": "uid_vid_00137.mp4", 85 | "SM4dmuL8KKw_13_28": "uid_vid_00138.mp4", 86 | "iSJqd8z1v20_0_13": "uid_vid_00139.mp4", 87 | "HzOiJ87yNIg_237_262": "uid_vid_00140.mp4", 88 | "h2hwIDQkzBE_12_27": "uid_vid_00141.mp4", 89 | "WAUYbr5YE2U_0_13": "uid_vid_00142.mp4", 90 | "WKbN-iY7DSU_42_53": "uid_vid_00143.mp4" 91 | }, 92 | "Meva": { 93 | "2018-03-07.16-50-00.16-55-00.bus.G331_0-180000.mp4": "uid_vid_00102.mp4", 94 | "2018-03-15.15-10-00.15-15-00.bus.G331_180000-300300.mp4": "uid_vid_00103.mp4", 95 | "2018-03-13.17-40-00.17-41-19.school.G421_0-79500.mp4": "uid_vid_00104.mp4", 96 | "2018-03-09.10-15-01.10-20-01.school.G420_180000-300033.mp4": "uid_vid_00105.mp4", 97 | "2018-03-15.15-35-00.15-40-00.school.G421_180000-300066.mp4": "uid_vid_00106.mp4", 98 | "2018-03-07.16-50-00.16-55-00.bus.G331_180000-300166.mp4": "uid_vid_00107.mp4", 99 | "2018-03-13.15-55-00.16-00-00.school.G421_0-180000.mp4": "uid_vid_00108.mp4", 100 | "2018-03-07.16-55-01.17-00-00.bus.G508_0-180000.mp4": "uid_vid_00109.mp4", 101 | "2018-03-07.16-50-00.16-55-00.school.G421_180000-300066.mp4": "uid_vid_00110.mp4", 102 | "2018-03-11.16-35-01.16-40-01.school.G420_180000-300033.mp4": "uid_vid_00111.mp4", 103 | "2018-03-15.15-10-00.15-15-00.bus.G331_0-180000.mp4": "uid_vid_00112.mp4", 104 | "2018-03-15.15-35-00.15-40-00.school.G421_0-180000.mp4": "uid_vid_00113.mp4", 105 | "2018-03-13.15-55-00.16-00-00.school.G421_180000-300000.mp4": "uid_vid_00114.mp4", 106 | "2018-03-15.14-50-00.14-55-00.bus.G508_0-180000.mp4": "uid_vid_00115.mp4", 107 | "2018-03-15.14-50-00.14-55-00.bus.G508_180000-300000.mp4": "uid_vid_00116.mp4", 108 | "2018-03-07.16-50-00.16-55-00.school.G421_0-180000.mp4": "uid_vid_00117.mp4" 109 | }, 110 | "Virat": { 111 | "VIRAT_S_010201_04_000374_000469.mp4": "uid_vid_00144.mp4", 112 | "VIRAT_S_010203_06_000620_000760.mp4": "uid_vid_00145.mp4", 113 | "VIRAT_S_010203_08_000895_000975.mp4": "uid_vid_00146.mp4", 114 | "VIRAT_S_010004_06_000547_000715.mp4": "uid_vid_00147.mp4", 115 | "VIRAT_S_010208_06_000671_000744.mp4": "uid_vid_00148.mp4", 116 | "VIRAT_S_010200_07_000748_000837.mp4": "uid_vid_00149.mp4", 117 | "VIRAT_S_010004_08_000811_000920.mp4": "uid_vid_00150.mp4", 118 | "VIRAT_S_010200_00_000060_000218.mp4": "uid_vid_00151.mp4", 119 | "VIRAT_S_010203_07_000775_000869.mp4": "uid_vid_00152.mp4" 120 | } 121 | } 122 | ``` -------------------------------------------------------------------------------- /readme/personpath22.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/tracking-dataset/2f9c757f847e172ce375372913bdbb3a47074e4f/readme/personpath22.png -------------------------------------------------------------------------------- /readme/personpath22_small.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-science/tracking-dataset/2f9c757f847e172ce375372913bdbb3a47074e4f/readme/personpath22_small.gif -------------------------------------------------------------------------------- /readme/public_detection.md: -------------------------------------------------------------------------------- 1 | ## Public detection 2 | To standardize the "public detection" evaluation protocol, we provide a pre-trained person detection model. 3 | 4 | It is a Fully Convolutional Object Detection (FCOS) model that is trained on CrowdHuman train and validation dataset. It achieves 81.39% AP @ IOU = 0.5 AND 45.56% AP @ IOU = 0.75. 5 | 6 | We encourage all researchers report their model's results with the provided detection results. The detection results can be downloaded with [MOTChallenge format](https://aws-cv-sci-motion-public.s3.us-west-2.amazonaws.com/PersonPath22/public_detection_fcos/mot_format.zip) and [GluonCV format](https://aws-cv-sci-motion-public.s3.us-west-2.amazonaws.com/PersonPath22/public_detection_fcos/amazon_format.zip). -------------------------------------------------------------------------------- /readme/results.md: -------------------------------------------------------------------------------- 1 | ## Results 2 | 3 | We provide the raw predictions that are reported in the paper. The predictions are provided in MOTChallenge format. 4 | 5 | Methods | IDF1 | MOTA | FP | FN | IDsw | Raw predictions 6 | ----- |------|------|---------|-------|-------| ----- 7 | CenterTrack | 46.4 | 59.3 | 24340 | 71550 | 10319 | [URL]() 8 | SiamMOT | 53.7 | 67.5 | 13217 | 62543 | 8942 | [URL]() 9 | FairMOT | 61.1 | 61.8 | 14540 | 80034 | 5095 | [URL]() 10 | IDFree | 63.1 | 68.6 | 9218 | 66573 | 6148 | [URL]() 11 | TrackFormer | 57.1 | 69.7 | 23138 | 47303 | 8633 | [URL]() 12 | ByteTrack | 66.8 | 75.4 | 17214 | 40902 | 5931 | [URL](https://aws-cv-sci-motion-public.s3.us-west-2.amazonaws.com/PersonPath22/bytetrack_results.tar) 13 | 14 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | awscli 2 | requests 3 | tqdm 4 | --------------------------------------------------------------------------------