├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── data └── sweep_audio │ ├── 10ms_sweep.wav │ ├── 1s_sweep.wav │ ├── 3ms_10times_per_second.wav │ ├── 3ms_sweep.wav │ └── 5ms_sweep.wav ├── getEchoes.py └── visualEchoes_teaser.png /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. 4 | Please read the [full text](https://code.fb.com/codeofconduct/) 5 | so that you can understand what actions will and will not be tolerated. 6 | 7 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to VisualEchoes 2 | We want to make contributing to this project as easy and transparent as 3 | possible. 4 | 5 | ## Pull Requests 6 | We actively welcome your pull requests. 7 | 8 | 1. Fork the repo and create your branch from `master`. 9 | 2. If you've added code that should be tested, add tests. 10 | 3. If you've changed APIs, update the documentation. 11 | 4. Ensure the test suite passes. 12 | 5. Make sure your code lints. 13 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 14 | 15 | ## Contributor License Agreement ("CLA") 16 | In order to accept your pull request, we need you to submit a CLA. You only need 17 | to do this once to work on any of Facebook's open source projects. 18 | 19 | Complete your CLA here: 20 | 21 | ## Issues 22 | We use GitHub issues to track public bugs. Please ensure your description is 23 | clear and has sufficient instructions to be able to reproduce the issue. 24 | 25 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 26 | disclosure of security bugs. In those cases, please go through the process 27 | outlined on that page and do not file a public issue. 28 | 29 | ## Coding Style 30 | * 80 character line length 31 | 32 | ## License 33 | By contributing to Listen-to-Look, you agree that your contributions will be licensed 34 | under the LICENSE file in the root directory of this source tree. 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Attribution 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution 4.0 International Public License 58 | 59 | By exercising the Licensed Rights (defined below), You accept and agree 60 | to be bound by the terms and conditions of this Creative Commons 61 | Attribution 4.0 International Public License ("Public License"). To the 62 | extent this Public License may be interpreted as a contract, You are 63 | granted the Licensed Rights in consideration of Your acceptance of 64 | these terms and conditions, and the Licensor grants You such rights in 65 | consideration of benefits the Licensor receives from making the 66 | Licensed Material available under these terms and conditions. 67 | 68 | Section 1 -- Definitions. 69 | 70 | a. Adapted Material means material subject to Copyright and Similar 71 | Rights that is derived from or based upon the Licensed Material 72 | and in which the Licensed Material is translated, altered, 73 | arranged, transformed, or otherwise modified in a manner requiring 74 | permission under the Copyright and Similar Rights held by the 75 | Licensor. For purposes of this Public License, where the Licensed 76 | Material is a musical work, performance, or sound recording, 77 | Adapted Material is always produced where the Licensed Material is 78 | synched in timed relation with a moving image. 79 | 80 | b. Adapter's License means the license You apply to Your Copyright 81 | and Similar Rights in Your contributions to Adapted Material in 82 | accordance with the terms and conditions of this Public License. 83 | 84 | c. Copyright and Similar Rights means copyright and/or similar rights 85 | closely related to copyright including, without limitation, 86 | performance, broadcast, sound recording, and Sui Generis Database 87 | Rights, without regard to how the rights are labeled or 88 | categorized. For purposes of this Public License, the rights 89 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 90 | Rights. 91 | 92 | d. Effective Technological Measures means those measures that, in the 93 | absence of proper authority, may not be circumvented under laws 94 | fulfilling obligations under Article 11 of the WIPO Copyright 95 | Treaty adopted on December 20, 1996, and/or similar international 96 | agreements. 97 | 98 | e. Exceptions and Limitations means fair use, fair dealing, and/or 99 | any other exception or limitation to Copyright and Similar Rights 100 | that applies to Your use of the Licensed Material. 101 | 102 | f. Licensed Material means the artistic or literary work, database, 103 | or other material to which the Licensor applied this Public 104 | License. 105 | 106 | g. Licensed Rights means the rights granted to You subject to the 107 | terms and conditions of this Public License, which are limited to 108 | all Copyright and Similar Rights that apply to Your use of the 109 | Licensed Material and that the Licensor has authority to license. 110 | 111 | h. Licensor means the individual(s) or entity(ies) granting rights 112 | under this Public License. 113 | 114 | i. Share means to provide material to the public by any means or 115 | process that requires permission under the Licensed Rights, such 116 | as reproduction, public display, public performance, distribution, 117 | dissemination, communication, or importation, and to make material 118 | available to the public including in ways that members of the 119 | public may access the material from a place and at a time 120 | individually chosen by them. 121 | 122 | j. Sui Generis Database Rights means rights other than copyright 123 | resulting from Directive 96/9/EC of the European Parliament and of 124 | the Council of 11 March 1996 on the legal protection of databases, 125 | as amended and/or succeeded, as well as other essentially 126 | equivalent rights anywhere in the world. 127 | 128 | k. You means the individual or entity exercising the Licensed Rights 129 | under this Public License. Your has a corresponding meaning. 130 | 131 | Section 2 -- Scope. 132 | 133 | a. License grant. 134 | 135 | 1. Subject to the terms and conditions of this Public License, 136 | the Licensor hereby grants You a worldwide, royalty-free, 137 | non-sublicensable, non-exclusive, irrevocable license to 138 | exercise the Licensed Rights in the Licensed Material to: 139 | 140 | a. reproduce and Share the Licensed Material, in whole or 141 | in part; and 142 | 143 | b. produce, reproduce, and Share Adapted Material. 144 | 145 | 2. Exceptions and Limitations. For the avoidance of doubt, where 146 | Exceptions and Limitations apply to Your use, this Public 147 | License does not apply, and You do not need to comply with 148 | its terms and conditions. 149 | 150 | 3. Term. The term of this Public License is specified in Section 151 | 6(a). 152 | 153 | 4. Media and formats; technical modifications allowed. The 154 | Licensor authorizes You to exercise the Licensed Rights in 155 | all media and formats whether now known or hereafter created, 156 | and to make technical modifications necessary to do so. The 157 | Licensor waives and/or agrees not to assert any right or 158 | authority to forbid You from making technical modifications 159 | necessary to exercise the Licensed Rights, including 160 | technical modifications necessary to circumvent Effective 161 | Technological Measures. For purposes of this Public License, 162 | simply making modifications authorized by this Section 2(a) 163 | (4) never produces Adapted Material. 164 | 165 | 5. Downstream recipients. 166 | 167 | a. Offer from the Licensor -- Licensed Material. Every 168 | recipient of the Licensed Material automatically 169 | receives an offer from the Licensor to exercise the 170 | Licensed Rights under the terms and conditions of this 171 | Public License. 172 | 173 | b. No downstream restrictions. You may not offer or impose 174 | any additional or different terms or conditions on, or 175 | apply any Effective Technological Measures to, the 176 | Licensed Material if doing so restricts exercise of the 177 | Licensed Rights by any recipient of the Licensed 178 | Material. 179 | 180 | 6. No endorsement. Nothing in this Public License constitutes or 181 | may be construed as permission to assert or imply that You 182 | are, or that Your use of the Licensed Material is, connected 183 | with, or sponsored, endorsed, or granted official status by, 184 | the Licensor or others designated to receive attribution as 185 | provided in Section 3(a)(1)(A)(i). 186 | 187 | b. Other rights. 188 | 189 | 1. Moral rights, such as the right of integrity, are not 190 | licensed under this Public License, nor are publicity, 191 | privacy, and/or other similar personality rights; however, to 192 | the extent possible, the Licensor waives and/or agrees not to 193 | assert any such rights held by the Licensor to the limited 194 | extent necessary to allow You to exercise the Licensed 195 | Rights, but not otherwise. 196 | 197 | 2. Patent and trademark rights are not licensed under this 198 | Public License. 199 | 200 | 3. To the extent possible, the Licensor waives any right to 201 | collect royalties from You for the exercise of the Licensed 202 | Rights, whether directly or through a collecting society 203 | under any voluntary or waivable statutory or compulsory 204 | licensing scheme. In all other cases the Licensor expressly 205 | reserves any right to collect such royalties. 206 | 207 | Section 3 -- License Conditions. 208 | 209 | Your exercise of the Licensed Rights is expressly made subject to the 210 | following conditions. 211 | 212 | a. Attribution. 213 | 214 | 1. If You Share the Licensed Material (including in modified 215 | form), You must: 216 | 217 | a. retain the following if it is supplied by the Licensor 218 | with the Licensed Material: 219 | 220 | i. identification of the creator(s) of the Licensed 221 | Material and any others designated to receive 222 | attribution, in any reasonable manner requested by 223 | the Licensor (including by pseudonym if 224 | designated); 225 | 226 | ii. a copyright notice; 227 | 228 | iii. a notice that refers to this Public License; 229 | 230 | iv. a notice that refers to the disclaimer of 231 | warranties; 232 | 233 | v. a URI or hyperlink to the Licensed Material to the 234 | extent reasonably practicable; 235 | 236 | b. indicate if You modified the Licensed Material and 237 | retain an indication of any previous modifications; and 238 | 239 | c. indicate the Licensed Material is licensed under this 240 | Public License, and include the text of, or the URI or 241 | hyperlink to, this Public License. 242 | 243 | 2. You may satisfy the conditions in Section 3(a)(1) in any 244 | reasonable manner based on the medium, means, and context in 245 | which You Share the Licensed Material. For example, it may be 246 | reasonable to satisfy the conditions by providing a URI or 247 | hyperlink to a resource that includes the required 248 | information. 249 | 250 | 3. If requested by the Licensor, You must remove any of the 251 | information required by Section 3(a)(1)(A) to the extent 252 | reasonably practicable. 253 | 254 | 4. If You Share Adapted Material You produce, the Adapter's 255 | License You apply must not prevent recipients of the Adapted 256 | Material from complying with this Public License. 257 | 258 | Section 4 -- Sui Generis Database Rights. 259 | 260 | Where the Licensed Rights include Sui Generis Database Rights that 261 | apply to Your use of the Licensed Material: 262 | 263 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 264 | to extract, reuse, reproduce, and Share all or a substantial 265 | portion of the contents of the database; 266 | 267 | b. if You include all or a substantial portion of the database 268 | contents in a database in which You have Sui Generis Database 269 | Rights, then the database in which You have Sui Generis Database 270 | Rights (but not its individual contents) is Adapted Material; and 271 | 272 | c. You must comply with the conditions in Section 3(a) if You Share 273 | all or a substantial portion of the contents of the database. 274 | 275 | For the avoidance of doubt, this Section 4 supplements and does not 276 | replace Your obligations under this Public License where the Licensed 277 | Rights include other Copyright and Similar Rights. 278 | 279 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 280 | 281 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 282 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 283 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 284 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 285 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 286 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 287 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 288 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 289 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 290 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 291 | 292 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 293 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 294 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 295 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 296 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 297 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 298 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 299 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 300 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 301 | 302 | c. The disclaimer of warranties and limitation of liability provided 303 | above shall be interpreted in a manner that, to the extent 304 | possible, most closely approximates an absolute disclaimer and 305 | waiver of all liability. 306 | 307 | Section 6 -- Term and Termination. 308 | 309 | a. This Public License applies for the term of the Copyright and 310 | Similar Rights licensed here. However, if You fail to comply with 311 | this Public License, then Your rights under this Public License 312 | terminate automatically. 313 | 314 | b. Where Your right to use the Licensed Material has terminated under 315 | Section 6(a), it reinstates: 316 | 317 | 1. automatically as of the date the violation is cured, provided 318 | it is cured within 30 days of Your discovery of the 319 | violation; or 320 | 321 | 2. upon express reinstatement by the Licensor. 322 | 323 | For the avoidance of doubt, this Section 6(b) does not affect any 324 | right the Licensor may have to seek remedies for Your violations 325 | of this Public License. 326 | 327 | c. For the avoidance of doubt, the Licensor may also offer the 328 | Licensed Material under separate terms or conditions or stop 329 | distributing the Licensed Material at any time; however, doing so 330 | will not terminate this Public License. 331 | 332 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 333 | License. 334 | 335 | Section 7 -- Other Terms and Conditions. 336 | 337 | a. The Licensor shall not be bound by any additional or different 338 | terms or conditions communicated by You unless expressly agreed. 339 | 340 | b. Any arrangements, understandings, or agreements regarding the 341 | Licensed Material not stated herein are separate from and 342 | independent of the terms and conditions of this Public License. 343 | 344 | Section 8 -- Interpretation. 345 | 346 | a. For the avoidance of doubt, this Public License does not, and 347 | shall not be interpreted to, reduce, limit, restrict, or impose 348 | conditions on any use of the Licensed Material that could lawfully 349 | be made without permission under this Public License. 350 | 351 | b. To the extent possible, if any provision of this Public License is 352 | deemed unenforceable, it shall be automatically reformed to the 353 | minimum extent necessary to make it enforceable. If the provision 354 | cannot be reformed, it shall be severed from this Public License 355 | without affecting the enforceability of the remaining terms and 356 | conditions. 357 | 358 | c. No term or condition of this Public License will be waived and no 359 | failure to comply consented to unless expressly agreed to by the 360 | Licensor. 361 | 362 | d. Nothing in this Public License constitutes or may be interpreted 363 | as a limitation upon, or waiver of, any privileges and immunities 364 | that apply to the Licensor or You, including from the legal 365 | processes of any jurisdiction or authority. 366 | 367 | ======================================================================= 368 | 369 | Creative Commons is not a party to its public licenses. 370 | Notwithstanding, Creative Commons may elect to apply one of its public 371 | licenses to material it publishes and in those instances will be 372 | considered the "Licensor." Except for the limited purpose of indicating 373 | that material is shared under a Creative Commons public license or as 374 | otherwise permitted by the Creative Commons policies published at 375 | creativecommons.org/policies, Creative Commons does not authorize the 376 | use of the trademark "Creative Commons" or any other trademark or logo 377 | of Creative Commons without its prior written consent including, 378 | without limitation, in connection with any unauthorized modifications 379 | to any of its public licenses or any other arrangements, 380 | understandings, or agreements concerning use of licensed material. For 381 | the avoidance of doubt, this paragraph does not form part of the public 382 | licenses. 383 | 384 | Creative Commons may be contacted at creativecommons.org. 385 | 386 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## VisualEchoes: Spatial Image Representation Learning through Echolocation 2 | This repository contains the RGB / Depth / Echo data used for spatial image representations learning in our ECCV 2020 paper [VisualEchoes](https://arxiv.org/abs/2005.01616). [[Project Page]](http://vision.cs.utexas.edu/projects/visualEchoes/) 3 | 4 | 5 | 6 |
7 | 8 | [VisualEchoes: Spatial Image Representation Learning through Echolocation](https://arxiv.org/abs/2005.01616) 9 | [Ruohan Gao](https://www.cs.utexas.edu/~rhgao/)1,3, [Changan Chen](https://changan.io/)1,3, [Ziad Al-Halah](https://www.cs.utexas.edu/~ziad/)1, [Carl Schissler](https://scholar.google.com/citations?user=YcL_y8AAAAAJ&hl=en)2, [Kristen Grauman](http://www.cs.utexas.edu/~grauman/)1,3
10 | 1UT Austin, 2Facebook Reality Lab, 3Facebook AI Research 11 | In European Conference on Computer Vision (**ECCV**), 2020 12 | 13 |
14 | 15 | If you find our data or project useful in your research, please cite: 16 | 17 | @inproceedings{gao2020visualechoes, 18 | title = {VisualEchoes: Spatial Image Representation Learning through Echolocation}, 19 | author = {Gao, Ruohan and Chen, Changan and Al-Halab, Ziad and Schissler, Carl and Grauman, Kristen}, 20 | booktitle = {ECCV}, 21 | year = {2020} 22 | } 23 | 24 | ### Demo Video of Echolocation Simulation 25 | We show an example of the agent navigating in one replica scene and performing echolocation. The agent emits 3ms chirp signals from 20Hz to 20kHz and receives echo responses from the room. Echoes resulting from the emitted chirps reflect the scene geometry. 26 | 27 | [](https://youtu.be/MobPvUf6F20) 28 | 29 | ### Overview of VisualEchoes Dataset 30 | We provide the (RGB, Depth, Echo) data generated using [habitat-sim](https://github.com/facebookresearch/habitat-sim) and [sound-spaces](https://github.com/facebookresearch/sound-spaces) on [Replica](https://github.com/facebookresearch/Replica-Dataset) dataset. The source audio "chirp" we use is a sweep signal from 20Hz - 20kHz (the human-audible range) within a duration of 3ms (3ms_sweep.wav). The echoes are obtained by convolving the 1s audio (with the sweep signal in the first 3ms) with the corresponding binaural echolocation room impuse responses (RIRs) for each of the four orientations (0°, 90°, 180°, 270°) at the agents' locations. When the agent emits sound from its position, convolving the emitted omnidirectional audio with the corresponding binaural RIR generates the binaural echo responses from the environment the agent hears when facing each orientation. 31 | 32 | ### Data Download 33 | 1. The VisualEchoes dataset contains the RGB images, depth maps, echo responses generated from Replica dataset. Run the commands below to download the rgb-depth pairs (4 different resolutions), echoes, room impuse responses used for echolocation for the 1,740 navigable locations x 4 orientations = 6,960 agent states used in the paper. 34 | ``` 35 | # rgb-depth pairs of 4 different resolutions 36 | # dictionary is in the format of {scene:{(location, orientation): {'rgb':rgb_image, 'depth':depth_map}}} 37 | wget http://dl.fbaipublicfiles.com/VisualEchoes/rgb_depth/scene_observations_128.pkl 38 | wget http://dl.fbaipublicfiles.com/VisualEchoes/rgb_depth/scene_observations_140.pkl 39 | wget http://dl.fbaipublicfiles.com/VisualEchoes/rgb_depth/scene_observations_256.pkl 40 | wget http://dl.fbaipublicfiles.com/VisualEchoes/rgb_depth/scene_observations_280.pkl 41 | 42 | # echo responses for the 3ms sweep signal at all navigable locations 43 | # ├── echoes_navigable 44 | # │ └── [scene] (scene name) 45 | # │ └── [sweep_sound] (name of the source signal) 46 | # │ └── [angle] (agent's orientation) 47 | # │ └── location_index.wav (agent's location) 48 | wget http://dl.fbaipublicfiles.com/VisualEchoes/echoes_navigable.tar.gz 49 | 50 | # echolocation room impulse response for all navigable locations 51 | # ├── echolocation_RIRs_navigable 52 | # │ └── [scene] (scene name) 53 | # │ └── [angle] (agent's orientation) 54 | # │ └── location_index.wav (agent's location) 55 | wget http://dl.fbaipublicfiles.com/VisualEchoes/echolocation_RIRs_navigable.tar.gz 56 | ``` 57 | 58 | 2. We also provide the echoes and binaural echolocation room impulse responses for all locations of the 18 environments in Replica. 59 | ``` 60 | # echo responses for the 3ms sweep signal at all locations 61 | wget http://dl.fbaipublicfiles.com/VisualEchoes/echoes_all.tar.gz 62 | 63 | # echolocation room impulse response for all locations 64 | wget http://dl.fbaipublicfiles.com/VisualEchoes/echolocation_RIRs_all.tar.gz 65 | ``` 66 | 67 | ### Echolocation 68 | The data/sweep_audio/ directory contains the 3ms sweep signal we use for echolocation in our paper as well as some other types of sweep signals. To generate echo responses for other types of 1s signals of your choice, place the directory of echolocation room impluse response properly under your data directory and use the getEchoes.py script to perform echolocation. 69 | ``` 70 | python getEchoes.py 71 | ``` 72 | 73 | 74 | ### Acknowlegements 75 | The RGB and depth data are generated using [habitat-sim](https://github.com/facebookresearch/habitat-sim) on [Replica](https://github.com/facebookresearch/Replica-Dataset) dataset. The binaural RIRs we use for echolocation is a subset of the binaural RIRs from [sound-spaces](https://github.com/facebookresearch/sound-spaces). 76 | 77 | ### Licence 78 | The VisualEchoes dataset is CC BY 4.0 licensed, as found in the LICENSE file. Please also refer to the licence files for [habitat-sim](https://github.com/facebookresearch/habitat-sim), [Replica](https://github.com/facebookresearch/Replica-Dataset), and [sound-spaces](https://github.com/facebookresearch/sound-spaces). 79 | -------------------------------------------------------------------------------- /data/sweep_audio/10ms_sweep.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/data/sweep_audio/10ms_sweep.wav -------------------------------------------------------------------------------- /data/sweep_audio/1s_sweep.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/data/sweep_audio/1s_sweep.wav -------------------------------------------------------------------------------- /data/sweep_audio/3ms_10times_per_second.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/data/sweep_audio/3ms_10times_per_second.wav -------------------------------------------------------------------------------- /data/sweep_audio/3ms_sweep.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/data/sweep_audio/3ms_sweep.wav -------------------------------------------------------------------------------- /data/sweep_audio/5ms_sweep.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/data/sweep_audio/5ms_sweep.wav -------------------------------------------------------------------------------- /getEchoes.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) Facebook, Inc. and its affiliates. 4 | # All rights reserved. 5 | # 6 | # This source code is licensed under the license found in the 7 | # LICENSE file in the root directory of this source tree. 8 | 9 | # Acknowledgement: Part of the codes are adapted from Unnat Jain 10 | 11 | import os 12 | import numpy as np 13 | import librosa 14 | from scipy.io import wavfile 15 | from scipy.signal import fftconvolve 16 | from tqdm import tqdm 17 | import soundfile as sf 18 | 19 | FREQ = 44100 20 | AMPLITUDE_SCALING_FACTOR = 10 21 | 22 | def convolve_data( 23 | scene_rir_dir, 24 | scene_echoes_dir, 25 | source_audio_file, 26 | amplitude_scaling_factor 27 | ): 28 | sig, fs = sf.read(source_audio_file, dtype='int16') 29 | sig = librosa.util.fix_length(sig, FREQ) 30 | assert fs == FREQ 31 | 32 | for angle in os.listdir(scene_rir_dir): 33 | angle_scene_rir_dir = os.path.join(scene_rir_dir, angle) 34 | angle_scene_echoes_dir = os.path.join(scene_echoes_dir, angle) 35 | os.makedirs(angle_scene_echoes_dir, exist_ok=True) 36 | 37 | rir_files = os.listdir(angle_scene_rir_dir) 38 | last_rir_file = None 39 | for rir_file_name in tqdm(rir_files): 40 | rir_file = os.path.join(angle_scene_rir_dir, rir_file_name) 41 | echo_file = os.path.join(angle_scene_echoes_dir, rir_file_name) 42 | if os.path.exists(echo_file): 43 | last_rir_file = rir_file 44 | continue 45 | try: 46 | fs_imp, sig_imp = wavfile.read(rir_file) 47 | except ValueError: 48 | print('{} has wrong file format. Using {} as substitute.'.format(rir_file, last_rir_file)) 49 | fs_imp, sig_imp = wavfile.read(last_rir_file) 50 | 51 | imp_full_length = np.zeros((FREQ,2)) 52 | imp_full_length[:sig_imp.shape[0]-128, :] = sig_imp[128:, :] #remove the first 127 zero samples 53 | sig_imp = imp_full_length 54 | assert fs_imp == FREQ 55 | output_sig = [] 56 | for ch_i in range(sig_imp.shape[-1]): 57 | output_sig.append(fftconvolve(sig, sig_imp[:, ch_i] / amplitude_scaling_factor, mode="full")) 58 | output_sig = np.array(output_sig).T 59 | output_sig = output_sig[:FREQ,:] 60 | output_sig_pcm = np.round(output_sig).astype("int16") 61 | wavfile.write(echo_file, FREQ, output_sig_pcm) 62 | last_rir_file = rir_file 63 | 64 | def main(): 65 | data_root_dir = '/YOUR_DATA_ROOT/' 66 | echolocation_RIR_dir = os.path.join(data_root_dir, 'echolocation_RIRs_navigable') 67 | echoes_dir = os.path.join(data_root_dir, 'echoes_navigable') 68 | sound_dir = os.path.join(data_root_dir, 'sweep_audio') 69 | sound_files = ['3ms_sweep.wav'] 70 | 71 | scenes = ['apartment_0', 'apartment_1', 'apartment_2', 'frl_apartment_0', 'frl_apartment_1', 'frl_apartment_2', 72 | 'frl_apartment_3', 'frl_apartment_4', 'frl_apartment_5', 'office_0', 'office_1', 'office_2', 73 | 'office_3', 'office_4', 'hotel_0', 'room_0', 'room_1', 'room_2'] 74 | 75 | for sound_file in sound_files: 76 | sound = sound_file.split('.')[0] 77 | source_audio_file = os.path.join(sound_dir, sound_file) 78 | for scene in scenes: 79 | scene_rir_dir = os.path.join(echolocation_RIR_dir, scene) 80 | scene_echoes_dir = os.path.join(echoes_dir, scene, sound) 81 | os.makedirs(scene_echoes_dir, exist_ok=True) 82 | convolve_data(scene_rir_dir, scene_echoes_dir, source_audio_file, AMPLITUDE_SCALING_FACTOR) 83 | 84 | if __name__ == '__main__': 85 | main() 86 | -------------------------------------------------------------------------------- /visualEchoes_teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/facebookresearch/VisualEchoes/1c2dbf4300fe4f8294b7435e510dd5e2d56ff7d2/visualEchoes_teaser.png --------------------------------------------------------------------------------