├── ExTrack_GUI.py ├── LICENSE ├── License └── cc-by-sa-4.0.txt ├── README.md ├── Tutorials ├── Fitting_methods.ipynb ├── Tutorial_ExTrack.ipynb ├── __init__.py ├── automated_fitting.py ├── dataset.zip ├── example_tracks.csv ├── example_tracks.xml ├── images │ ├── img1.png │ ├── img2.png │ └── imgdir ├── simulated_tracks.xml ├── tracks.csv └── tracks.xml ├── extrack ├── __init__.py ├── auto_fitting.py ├── exporters.py ├── histograms.py ├── old_tracking.py ├── readers.py ├── refined_loc_old.py ├── refined_localization.py ├── simulate_tracks.py ├── tracking.py ├── tracking_0.py ├── version.py └── visualization.py ├── pyproject.toml ├── requirements.txt ├── setup.py ├── simulated_tracks.xml └── todo.txt /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 van Teeffelen Lab 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /License/cc-by-sa-4.0.txt: -------------------------------------------------------------------------------- 1 | Creative Commons Attribution-ShareAlike 4.0 International Creative Commons 2 | Corporation ("Creative Commons") is not a law firm and does not provide legal 3 | services or legal advice. Distribution of Creative Commons public licenses 4 | does not create a lawyer-client or other relationship. Creative Commons makes 5 | its licenses and related information available on an "as-is" basis. Creative 6 | Commons gives no warranties regarding its licenses, any material licensed 7 | under their terms and conditions, or any related information. Creative Commons 8 | disclaims all liability for damages resulting from their use to the fullest 9 | extent possible. 10 | 11 | Using Creative Commons Public Licenses 12 | 13 | Creative Commons public licenses provide a standard set of terms and conditions 14 | that creators and other rights holders may use to share original works of 15 | authorship and other material subject to copyright and certain other rights 16 | specified in the public license below. The following considerations are for 17 | informational purposes only, are not exhaustive, and do not form part of our 18 | licenses. 19 | 20 | Considerations for licensors: Our public licenses are intended for use by 21 | those authorized to give the public permission to use material in ways otherwise 22 | restricted by copyright and certain other rights. Our licenses are irrevocable. 23 | Licensors should read and understand the terms and conditions of the license 24 | they choose before applying it. Licensors should also secure all rights necessary 25 | before applying our licenses so that the public can reuse the material as 26 | expected. Licensors should clearly mark any material not subject to the license. 27 | This includes other CC-licensed material, or material used under an exception 28 | or limitation to copyright. More considerations for licensors : wiki.creativecommons.org/Considerations_for_licensors 29 | 30 | Considerations for the public: By using one of our public licenses, a licensor 31 | grants the public permission to use the licensed material under specified 32 | terms and conditions. If the licensor's permission is not necessary for any 33 | reason–for example, because of any applicable exception or limitation to copyright–then 34 | that use is not regulated by the license. Our licenses grant only permissions 35 | under copyright and certain other rights that a licensor has authority to 36 | grant. Use of the licensed material may still be restricted for other reasons, 37 | including because others have copyright or other rights in the material. A 38 | licensor may make special requests, such as asking that all changes be marked 39 | or described. 40 | 41 | Although not required by our licenses, you are encouraged to respect those 42 | requests where reasonable. More considerations for the public : wiki.creativecommons.org/Considerations_for_licensees 43 | 44 | Creative Commons Attribution-ShareAlike 4.0 International Public License 45 | 46 | By exercising the Licensed Rights (defined below), You accept and agree to 47 | be bound by the terms and conditions of this Creative Commons Attribution-ShareAlike 48 | 4.0 International Public License ("Public License"). To the extent this Public 49 | License may be interpreted as a contract, You are granted the Licensed Rights 50 | in consideration of Your acceptance of these terms and conditions, and the 51 | Licensor grants You such rights in consideration of benefits the Licensor 52 | receives from making the Licensed Material available under these terms and 53 | conditions. 54 | 55 | Section 1 – Definitions. 56 | 57 | a. Adapted Material means material subject to Copyright and Similar Rights 58 | that is derived from or based upon the Licensed Material and in which the 59 | Licensed Material is translated, altered, arranged, transformed, or otherwise 60 | modified in a manner requiring permission under the Copyright and Similar 61 | Rights held by the Licensor. For purposes of this Public License, where the 62 | Licensed Material is a musical work, performance, or sound recording, Adapted 63 | Material is always produced where the Licensed Material is synched in timed 64 | relation with a moving image. 65 | 66 | b. Adapter's License means the license You apply to Your Copyright and Similar 67 | Rights in Your contributions to Adapted Material in accordance with the terms 68 | and conditions of this Public License. 69 | 70 | c. BY-SA Compatible License means a license listed at creativecommons.org/compatiblelicenses, 71 | approved by Creative Commons as essentially the equivalent of this Public 72 | License. 73 | 74 | d. Copyright and Similar Rights means copyright and/or similar rights closely 75 | related to copyright including, without limitation, performance, broadcast, 76 | sound recording, and Sui Generis Database Rights, without regard to how the 77 | rights are labeled or categorized. For purposes of this Public License, the 78 | rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 79 | 80 | e. Effective Technological Measures means those measures that, in the absence 81 | of proper authority, may not be circumvented under laws fulfilling obligations 82 | under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, 83 | and/or similar international agreements. 84 | 85 | f. Exceptions and Limitations means fair use, fair dealing, and/or any other 86 | exception or limitation to Copyright and Similar Rights that applies to Your 87 | use of the Licensed Material. 88 | 89 | g. License Elements means the license attributes listed in the name of a Creative 90 | Commons Public License. The License Elements of this Public License are Attribution 91 | and ShareAlike. 92 | 93 | h. Licensed Material means the artistic or literary work, database, or other 94 | material to which the Licensor applied this Public License. 95 | 96 | i. Licensed Rights means the rights granted to You subject to the terms and 97 | conditions of this Public License, which are limited to all Copyright and 98 | Similar Rights that apply to Your use of the Licensed Material and that the 99 | Licensor has authority to license. 100 | 101 | j. Licensor means the individual(s) or entity(ies) granting rights under this 102 | Public License. 103 | 104 | k. Share means to provide material to the public by any means or process that 105 | requires permission under the Licensed Rights, such as reproduction, public 106 | display, public performance, distribution, dissemination, communication, or 107 | importation, and to make material available to the public including in ways 108 | that members of the public may access the material from a place and at a time 109 | individually chosen by them. 110 | 111 | l. Sui Generis Database Rights means rights other than copyright resulting 112 | from Directive 96/9/EC of the European Parliament and of the Council of 11 113 | March 1996 on the legal protection of databases, as amended and/or succeeded, 114 | as well as other essentially equivalent rights anywhere in the world. 115 | 116 | m. You means the individual or entity exercising the Licensed Rights under 117 | this Public License. Your has a corresponding meaning. 118 | 119 | Section 2 – Scope. 120 | 121 | a. License grant. 122 | 123 | 1. Subject to the terms and conditions of this Public License, the Licensor 124 | hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, 125 | irrevocable license to exercise the Licensed Rights in the Licensed Material 126 | to: 127 | 128 | A. reproduce and Share the Licensed Material, in whole or in part; and 129 | 130 | B. produce, reproduce, and Share Adapted Material. 131 | 132 | 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions 133 | and Limitations apply to Your use, this Public License does not apply, and 134 | You do not need to comply with its terms and conditions. 135 | 136 | 3. Term. The term of this Public License is specified in Section 6(a). 137 | 138 | 4. Media and formats; technical modifications allowed. The Licensor authorizes 139 | You to exercise the Licensed Rights in all media and formats whether now known 140 | or hereafter created, and to make technical modifications necessary to do 141 | so. The Licensor waives and/or agrees not to assert any right or authority 142 | to forbid You from making technical modifications necessary to exercise the 143 | Licensed Rights, including technical modifications necessary to circumvent 144 | Effective Technological Measures. For purposes of this Public License, simply 145 | making modifications authorized by this Section 2(a)(4) never produces Adapted 146 | Material. 147 | 148 | 5. Downstream recipients. 149 | 150 | A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed 151 | Material automatically receives an offer from the Licensor to exercise the 152 | Licensed Rights under the terms and conditions of this Public License. 153 | 154 | B. Additional offer from the Licensor – Adapted Material. Every recipient 155 | of Adapted Material from You automatically receives an offer from the Licensor 156 | to exercise the Licensed Rights in the Adapted Material under the conditions 157 | of the Adapter's License You apply. 158 | 159 | C. No downstream restrictions. You may not offer or impose any additional 160 | or different terms or conditions on, or apply any Effective Technological 161 | Measures to, the Licensed Material if doing so restricts exercise of the Licensed 162 | Rights by any recipient of the Licensed Material. 163 | 164 | 6. No endorsement. Nothing in this Public License constitutes or may be construed 165 | as permission to assert or imply that You are, or that Your use of the Licensed 166 | Material is, connected with, or sponsored, endorsed, or granted official status 167 | by, the Licensor or others designated to receive attribution as provided in 168 | Section 3(a)(1)(A)(i). 169 | 170 | b. Other rights. 171 | 172 | 1. Moral rights, such as the right of integrity, are not licensed under this 173 | Public License, nor are publicity, privacy, and/or other similar personality 174 | rights; however, to the extent possible, the Licensor waives and/or agrees 175 | not to assert any such rights held by the Licensor to the limited extent necessary 176 | to allow You to exercise the Licensed Rights, but not otherwise. 177 | 178 | 2. Patent and trademark rights are not licensed under this Public License. 179 | 180 | 3. To the extent possible, the Licensor waives any right to collect royalties 181 | from You for the exercise of the Licensed Rights, whether directly or through 182 | a collecting society under any voluntary or waivable statutory or compulsory 183 | licensing scheme. In all other cases the Licensor expressly reserves any right 184 | to collect such royalties. 185 | 186 | Section 3 – License Conditions. 187 | 188 | Your exercise of the Licensed Rights is expressly made subject to the following 189 | conditions. 190 | 191 | a. Attribution. 192 | 193 | 1. If You Share the Licensed Material (including in modified form), You must: 194 | 195 | A. retain the following if it is supplied by the Licensor with the Licensed 196 | Material: 197 | 198 | i. identification of the creator(s) of the Licensed Material and any others 199 | designated to receive attribution, in any reasonable manner requested by the 200 | Licensor (including by pseudonym if designated); 201 | 202 | ii. a copyright notice; 203 | 204 | iii. a notice that refers to this Public License; 205 | 206 | iv. a notice that refers to the disclaimer of warranties; 207 | 208 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 209 | 210 | B. indicate if You modified the Licensed Material and retain an indication 211 | of any previous modifications; and 212 | 213 | C. indicate the Licensed Material is licensed under this Public License, and 214 | include the text of, or the URI or hyperlink to, this Public License. 215 | 216 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner 217 | based on the medium, means, and context in which You Share the Licensed Material. 218 | For example, it may be reasonable to satisfy the conditions by providing a 219 | URI or hyperlink to a resource that includes the required information. 220 | 221 | 3. If requested by the Licensor, You must remove any of the information required 222 | by Section 3(a)(1)(A) to the extent reasonably practicable. 223 | 224 | b. ShareAlike.In addition to the conditions in Section 3(a), if You Share 225 | Adapted Material You produce, the following conditions also apply. 226 | 227 | 1. The Adapter's License You apply must be a Creative Commons license with 228 | the same License Elements, this version or later, or a BY-SA Compatible License. 229 | 230 | 2. You must include the text of, or the URI or hyperlink to, the Adapter's 231 | License You apply. You may satisfy this condition in any reasonable manner 232 | based on the medium, means, and context in which You Share Adapted Material. 233 | 234 | 3. You may not offer or impose any additional or different terms or conditions 235 | on, or apply any Effective Technological Measures to, Adapted Material that 236 | restrict exercise of the rights granted under the Adapter's License You apply. 237 | 238 | Section 4 – Sui Generis Database Rights. 239 | 240 | Where the Licensed Rights include Sui Generis Database Rights that apply to 241 | Your use of the Licensed Material: 242 | 243 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, 244 | reuse, reproduce, and Share all or a substantial portion of the contents of 245 | the database; 246 | 247 | b. if You include all or a substantial portion of the database contents in 248 | a database in which You have Sui Generis Database Rights, then the database 249 | in which You have Sui Generis Database Rights (but not its individual contents) 250 | is Adapted Material, including for purposes of Section 3(b); and 251 | 252 | c. You must comply with the conditions in Section 3(a) if You Share all or 253 | a substantial portion of the contents of the database. 254 | 255 | For the avoidance of doubt, this Section 4 supplements and does not replace 256 | Your obligations under this Public License where the Licensed Rights include 257 | other Copyright and Similar Rights. 258 | 259 | Section 5 – Disclaimer of Warranties and Limitation of Liability. 260 | 261 | a. Unless otherwise separately undertaken by the Licensor, to the extent possible, 262 | the Licensor offers the Licensed Material as-is and as-available, and makes 263 | no representations or warranties of any kind concerning the Licensed Material, 264 | whether express, implied, statutory, or other. This includes, without limitation, 265 | warranties of title, merchantability, fitness for a particular purpose, non-infringement, 266 | absence of latent or other defects, accuracy, or the presence or absence of 267 | errors, whether or not known or discoverable. Where disclaimers of warranties 268 | are not allowed in full or in part, this disclaimer may not apply to You. 269 | 270 | b. To the extent possible, in no event will the Licensor be liable to You 271 | on any legal theory (including, without limitation, negligence) or otherwise 272 | for any direct, special, indirect, incidental, consequential, punitive, exemplary, 273 | or other losses, costs, expenses, or damages arising out of this Public License 274 | or use of the Licensed Material, even if the Licensor has been advised of 275 | the possibility of such losses, costs, expenses, or damages. Where a limitation 276 | of liability is not allowed in full or in part, this limitation may not apply 277 | to You. 278 | 279 | c. The disclaimer of warranties and limitation of liability provided above 280 | shall be interpreted in a manner that, to the extent possible, most closely 281 | approximates an absolute disclaimer and waiver of all liability. 282 | 283 | Section 6 – Term and Termination. 284 | 285 | a. This Public License applies for the term of the Copyright and Similar Rights 286 | licensed here. However, if You fail to comply with this Public License, then 287 | Your rights under this Public License terminate automatically. 288 | 289 | b. Where Your right to use the Licensed Material has terminated under Section 290 | 6(a), it reinstates: 291 | 292 | 1. automatically as of the date the violation is cured, provided it is cured 293 | within 30 days of Your discovery of the violation; or 294 | 295 | 2. upon express reinstatement by the Licensor. 296 | 297 | c. For the avoidance of doubt, this Section 6(b) does not affect any right 298 | the Licensor may have to seek remedies for Your violations of this Public 299 | License. 300 | 301 | d. For the avoidance of doubt, the Licensor may also offer the Licensed Material 302 | under separate terms or conditions or stop distributing the Licensed Material 303 | at any time; however, doing so will not terminate this Public License. 304 | 305 | e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 306 | 307 | Section 7 – Other Terms and Conditions. 308 | 309 | a. The Licensor shall not be bound by any additional or different terms or 310 | conditions communicated by You unless expressly agreed. 311 | 312 | b. Any arrangements, understandings, or agreements regarding the Licensed 313 | Material not stated herein are separate from and independent of the terms 314 | and conditions of this Public License. 315 | 316 | Section 8 – Interpretation. 317 | 318 | a. For the avoidance of doubt, this Public License does not, and shall not 319 | be interpreted to, reduce, limit, restrict, or impose conditions on any use 320 | of the Licensed Material that could lawfully be made without permission under 321 | this Public License. 322 | 323 | b. To the extent possible, if any provision of this Public License is deemed 324 | unenforceable, it shall be automatically reformed to the minimum extent necessary 325 | to make it enforceable. If the provision cannot be reformed, it shall be severed 326 | from this Public License without affecting the enforceability of the remaining 327 | terms and conditions. 328 | 329 | c. No term or condition of this Public License will be waived and no failure 330 | to comply consented to unless expressly agreed to by the Licensor. 331 | 332 | d. Nothing in this Public License constitutes or may be interpreted as a limitation 333 | upon, or waiver of, any privileges and immunities that apply to the Licensor 334 | or You, including from the legal processes of any jurisdiction or authority. 335 | 336 | Creative Commons is not a party to its public licenses. Notwithstanding, Creative 337 | Commons may elect to apply one of its public licenses to material it publishes 338 | and in those instances will be considered the "Licensor." The text of the 339 | Creative Commons public licenses is dedicated to the public domain under the 340 | CC0 Public Domain Dedication. Except for the limited purpose of indicating 341 | that material is shared under a Creative Commons public license or as otherwise 342 | permitted by the Creative Commons policies published at creativecommons.org/policies, 343 | Creative Commons does not authorize the use of the trademark "Creative Commons" 344 | or any other trademark or logo of Creative Commons without its prior written 345 | consent including, without limitation, in connection with any unauthorized 346 | modifications to any of its public licenses or any other arrangements, understandings, 347 | or agreements concerning use of licensed material. For the avoidance of doubt, 348 | this paragraph does not form part of the public licenses. 349 | 350 | Creative Commons may be contacted at creativecommons.org. 351 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ExTrack 2 | ------- 3 | 4 | This repository contains the necessary scripts to run the method ExTrack. ExTrack is a method to detemine kinetics of particles able to transition between different motion states. It can assess diffusion coefficients, transition rates, localization error as well as annotating the probability for any track to be in each state for every time points. It can produce histograms of durations in each state to highlight none-markovian transition kinetics. Eventually it can be used to refine the localization precision of tracks by considering the most likely positions which is especially efficient when the particle do not move. 5 | 6 | More details on the methods are available in the Journal of Cell Biology https://rupress.org/jcb/article/222/5/e202208059/213911/ExTrack-characterizes-transition-kinetics-and. 7 | 8 | ExTrack has been designed and implemented by François Simon in the laboratory of Sven van Teeffelen at University of Montreal. ExTrack is primarely implemented as a python package and as a stand-alone software. The stand alone version of ExTrack can be download at https://zenodo.org/records/15133436. See the wiki https://github.com/vanTeeffelenLab/ExTrack/wiki or the pdf ExTrack_GUI_manual.pdf (not implemented yet) in this repository for detailed informations on how to use the software. Currently supported OS: Windows. 9 | 10 | See the Wiki section for more information on how to install and use ExTrack (python package and stand-alone software). 11 | 12 | https://pypi.org/project/extrack/ 13 | 14 | # Dependencies 15 | 16 | - numpy 17 | - lmfit 18 | - xmltodict 19 | - matplotlib 20 | - pandas 21 | 22 | Optional: jupyter, cupy 23 | 24 | # Installation (from pip) 25 | 26 | (needs to be run in anaconda prompt for anaconda users on windows) 27 | 28 | ## Install dependencies 29 | 30 | `pip install numpy lmfit xmltodict matplotlib pandas` 31 | 32 | ## Install ExTrack 33 | 34 | `pip install extrack` 35 | 36 | https://pypi.org/project/extrack/ 37 | 38 | the current version (1.5) has working but oudated version of the position refinement method. It may only work for 2-state models. This will be updated as soon as possible. 39 | 40 | ## Input file format 41 | 42 | ExTrack can deal with tracks saved with TrackMate xml format or csv format by using the integrated readers https://github.com/vanTeeffelenLab/ExTrack/wiki/Loading-data-sets. 43 | 44 | # Installation from this GitHub repository 45 | 46 | ## From Unix/Mac: 47 | 48 | `sudo apt install git` (if git is not installed) 49 | 50 | `git clone https://github.com/vanTeeffelenLab/ExTrack.git` 51 | 52 | `cd ExTrack` 53 | 54 | `sudo python setup.py install` 55 | 56 | ## From Windows using anaconda prompt: 57 | 58 | Need to install git if not already installed. 59 | 60 | `git clone https://github.com/vanTeeffelenLab/ExTrack.git` One can also just manually download the package if git is not installed. Once extracted the folder may be named ExTrack-main 61 | 62 | `cd ExTrack` or cd `ExTrack-main` 63 | 64 | `python setup.py install` from the ExTrack directory 65 | 66 | # Tutorial 67 | 68 | Tutorials for the python package of ExTrack are available. 69 | 70 | A first tutorial allows the user to have an overview of all the possibilities of the different modules of ExTrack (https://github.com/vanTeeffelenLab/ExTrack/blob/main/Tutorials/Tutorial_ExTrack.ipynb). This jupyter notebook tutorial shows the whole pipeline: 71 | - Loading data sets (https://github.com/vanTeeffelenLab/ExTrack/wiki/Loading-data-sets). 72 | - Initialize parameters of the model (https://github.com/vanTeeffelenLab/ExTrack/wiki/Parameters-for-fitting). 73 | - Fitting. 74 | - Probabilistic state annotation. 75 | - Histograms of state duration. 76 | - Position refinement. 77 | - Saving results. 78 | 79 | from loading data sets to saving results 80 | at these location: 81 | - tests/test_extrack.py 82 | - or Tutorials/tutorial_extrack.ipynb 83 | 84 | These contain the most important modules in a comprehensive framework. We recommand following the tutorial tutorial_extrack.ipynb which uses Jupyter notebook as it is more didactic. One has to install jupyter to use it: `pip install jupyter` in the anaconda prompt for conda users. 85 | 86 | # Usage 87 | ## Units 88 | The distance units of the parameters (input and output) are the same unit as the units of the tracks. Our initial parameters are chosen to work for micron units but initial parameters can be changed to match other units. The rate parameters are rates per frame. Rates per second can be inferred from the rates per frame by dividing them by the time in between frames. 89 | 90 | ## Main functions 91 | 92 | extrack.tracking.param_fitting : performs the fit to infer the parameters of a given data set. 93 | 94 | extrack.visualization.visualize_states_durations : plot histograms of the duration in each state. 95 | 96 | extrack.tracking.predict_Bs : predicts the states of the tracks. 97 | 98 | ## Extra functions 99 | 100 | extrack.simulate_tracks.sim_FOV : allows to simulate tracks. 101 | 102 | extrack.exporters.extrack_2_pandas : turn the outputs from ExTrack to a pandas dataframe. outputed dataframe can be save with dataframe.to_csv(save_path) 103 | 104 | extrack.exporters.save_extrack_2_xml : save extrack data to xml file (trackmate format). 105 | 106 | extrack.visualization.visualize_tracks : show all tracks in a single plot. 107 | 108 | extrack.visualization.plot_tracks : show the longest tracks on separated plots 109 | 110 | ## Caveats 111 | 112 | # References 113 | 114 | # License 115 | This program is released under the GNU General Public License version 3 or upper (GPLv3+). 116 | 117 | This program is free software: you can redistribute it and/or modify 118 | it under the terms of the GNU General Public License as published by 119 | the Free Software Foundation, either version 3 of the License, or 120 | (at your option) any later version. 121 | 122 | This program is distributed in the hope that it will be useful, 123 | but WITHOUT ANY WARRANTY; without even the implied warranty of 124 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 125 | GNU General Public License for more details. 126 | 127 | You should have received a copy of the GNU General Public License 128 | along with this program. If not, see . 129 | 130 | # Parallelization 131 | 132 | Multiple CPU Parallelization can be performed in get_2DSPT_params with the argument worker the number of cores used for the job (equal to 1 by default). 133 | Warning: Do not work on windows. 134 | 135 | GPU parallelization used to be available but may not be compatible with the current CPU parallelization, GPU parallelization uses the package cupy which can be installed as described here : https://github.com/cupy/cupy. The cupy version will depend on your cuda version which itself must be compatible with your GPU driver and GPU. Usage of cupy requires a change in the module extrack/tracking (line 4) : GPU_computing = True 136 | 137 | # Graphical User interface of ExTrack 138 | 139 | The Graphical User interface of ExTrack can be used with the script ExTrack_GUI.py. 140 | 141 | # Stand-alone Version 142 | 143 | The stand alone version of ExTrack can be download at https://zenodo.org/records/15133436. See the wiki https://github.com/vanTeeffelenLab/ExTrack/wiki or the pdf ExTrack_GUI_manual.pdf (not implemented yet) in this repository for detailed informations on how to use the software (on going). 144 | Currently supported OS: Windows 145 | 146 | To create a stand-alone version of ExTrack for your own OS, you can follow the following steps: 147 | 1) pip install pyinstaller 148 | 2) pyinstaller --onedir path\ExTrack_GUI.py 149 | 3) Copy the .ddl files starting with mkl into the dist\ExTrack_GUI\_internal (the mkl files can be found in C:\Users\Franc\anaconda3\Library\bin in my case) 150 | 4) execute dist\ExTrack_GUI.exe to run the stand alone software 151 | 152 | # Authors 153 | François Simon 154 | 155 | # Bugs/suggestions 156 | Sent an email at simon.francois \at protonmail.com 157 | -------------------------------------------------------------------------------- /Tutorials/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Tutorials/automated_fitting.py: -------------------------------------------------------------------------------- 1 | import extrack 2 | import numpy as np 3 | from matplotlib import pyplot as plt 4 | import os 5 | from glob import glob 6 | 7 | dt = 0.3 8 | 9 | datafolder = '/mnt/c/Users/username/path/dataset' 10 | SAVEDIR = '/mnt/c/Users/username/path/Res' 11 | workers = 5 12 | 13 | if not os.path.exists(SAVEDIR): 14 | os.mkdir(SAVEDIR) 15 | 16 | exps = glob(datafolder + '/*') 17 | print(exps) 18 | 19 | # performs a fitting for each replicate represented by the folder name (Exp1 or Exp2 for instance). 20 | for exp in exps: 21 | paths = glob(exp + '/*.xml') # collect all paths from the replicate 22 | 23 | if len(paths) ==0: 24 | raise ValueError('Wrong path, no xml file were found') 25 | 26 | lengths = np.arange(2,20) 27 | 28 | all_tracks, frames, opt_metrics = extrack.readers.read_trackmate_xml(paths, 29 | lengths=lengths, 30 | dist_th = 0.4, 31 | frames_boundaries = [0, 10000], 32 | remove_no_disp = True, 33 | opt_metrics_names = [], # Name of the optional metrics to catch 34 | opt_metrics_types = None) 35 | 36 | for l in list(all_tracks.keys()): 37 | if len(all_tracks[l])==0: 38 | del all_tracks[l] 39 | 40 | for l in all_tracks: 41 | print(all_tracks[l].shape) 42 | 43 | params = extrack.tracking.generate_params(nb_states = 2, 44 | LocErr_type = 1, 45 | nb_dims = 2, # only matters if LocErr_type == 2. 46 | LocErr_bounds = [0.01, 0.05], # the initial guess on LocErr will be the geometric mean of the boundaries. 47 | D_max = 1, # maximal diffusion coefficient allowed. 48 | Fractions_bounds = [0.001, 0.99], 49 | estimated_LocErr = [0.022], 50 | estimated_Ds = [0.0001, 0.03], # D will be arbitrary spaced from 0 to D_max if None, otherwise input a list of Ds for each state from state 0 to nb_states - 1. 51 | estimated_Fs = [0.3,0.7], # fractions will be equal if None, otherwise input a list of fractions for each state from state 0 to nb_states - 1. 52 | estimated_transition_rates = 0.1, # transition rate per step. example [0.1,0.05,0.03,0.07,0.2,0.2] for a 3-state model. 53 | ) 54 | 55 | res = {} 56 | for param in params: 57 | res[param] = [] 58 | res['residual'] = [] 59 | 60 | for k in range(3): 61 | # We run multiple iterations to make sure about the convergence. 62 | model_fit = extrack.tracking.param_fitting(all_tracks = all_tracks, 63 | dt = dt, 64 | params = params, 65 | nb_states = 2, 66 | nb_substeps = 1, 67 | cell_dims = [0.3], 68 | frame_len = 9, 69 | verbose = 0, 70 | workers = workers, # increase the number of CPU workers for faster computing, do not work on windows or mac (keep to 1) 71 | input_LocErr = None, 72 | steady_state = False, 73 | threshold = 0.1, 74 | max_nb_states = 200, 75 | method = 'bfgs') 76 | 77 | params = model_fit.params 78 | print(model_fit.residual[0]) 79 | 80 | for param in params: 81 | res[param].append(params[param].value) 82 | res['residual'].append(model_fit.residual[0]) 83 | 84 | np.save(SAVEDIR + exp.split('/')[-1] + '.npy', res, allow_pickle=True) 85 | 86 | print(res) 87 | 88 | -------------------------------------------------------------------------------- /Tutorials/dataset.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vanTeeffelenLab/ExTrack/e612eb50c25ef07758127cce0fe479039bdb0dfc/Tutorials/dataset.zip -------------------------------------------------------------------------------- /Tutorials/images/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vanTeeffelenLab/ExTrack/e612eb50c25ef07758127cce0fe479039bdb0dfc/Tutorials/images/img1.png -------------------------------------------------------------------------------- /Tutorials/images/img2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vanTeeffelenLab/ExTrack/e612eb50c25ef07758127cce0fe479039bdb0dfc/Tutorials/images/img2.png -------------------------------------------------------------------------------- /Tutorials/images/imgdir: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Tutorials/tracks.xml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vanTeeffelenLab/ExTrack/e612eb50c25ef07758127cce0fe479039bdb0dfc/Tutorials/tracks.xml -------------------------------------------------------------------------------- /extrack/__init__.py: -------------------------------------------------------------------------------- 1 | from extrack import tracking 2 | from extrack import old_tracking 3 | from extrack.version import __version__ 4 | print('version:', __version__) 5 | #from extrack import auto_fitting 6 | from extrack import exporters 7 | from extrack import histograms 8 | from extrack import readers 9 | from extrack import refined_localization 10 | from extrack import simulate_tracks 11 | from extrack import visualization 12 | -------------------------------------------------------------------------------- /extrack/auto_fitting.py: -------------------------------------------------------------------------------- 1 | from extrack.tracking import predict_Bs, get_2DSPT_params 2 | import numpy as np 3 | 4 | def fit_2states(all_tracks, 5 | dt, 6 | steady_state = True, 7 | cell_dims = [], 8 | estimated_vals = {'LocErr' : 0.025, 'D0' : 1e-20, 'D1' : 0.05, 'F0' : 0.45, 'p01' : 0.05, 'p10' : 0.05, 'pBL': 0.1}, 9 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'F0' : True, 'p01' : True, 'p10' : True, 'pBL': True}): 10 | 11 | model_fit = get_2DSPT_params(all_tracks, dt, nb_substeps = 1, nb_states = 2, do_frame = 1,frame_len = 4,cell_dims = cell_dims, verbose = 0, vary_params = vary_params, estimated_vals = estimated_vals, steady_state = steady_state) 12 | 13 | estimated_vals = [model_fit.params['LocErr'].value, model_fit.params['D0'].value, model_fit.params['D1'].value, model_fit.params['F0'].value, model_fit.params['p01'].value, model_fit.params['p10'].value] 14 | tr_freq = estimated_vals[3]*estimated_vals[4] + (1-estimated_vals[3])*estimated_vals[5] 15 | DLR = (2*dt*estimated_vals[2])**0.5/estimated_vals[0] 16 | 17 | frame_lens = [6,6,5] 18 | nb_substeps = 1 19 | 20 | if DLR < 1.5: 21 | frame_len = 8 22 | nb_substeps = 1 23 | elif DLR < 5: 24 | if tr_freq > 0.15: 25 | nb_substeps = 2 26 | if tr_freq < 0.15: 27 | nb_substeps = 1 28 | frame_len = frame_lens[nb_substeps-1] 29 | else : 30 | frame_lens = [6,6,5] 31 | if tr_freq < 0.15: 32 | nb_substeps = 1 33 | if tr_freq > 0.15: 34 | nb_substeps = 2 35 | if tr_freq > 0.3: 36 | nb_substeps = 3 37 | frame_len = frame_lens[nb_substeps-1] 38 | 39 | keep_running = 1 40 | res_val = 0 41 | for kk in range(40): 42 | if keep_running: 43 | estimated_vals = { 'LocErr' : model_fit.params['LocErr'], 'D0' : model_fit.params['D0'], 'D1' : model_fit.params['D1'], 'F0' : model_fit.params['F0'], 'p01' : model_fit.params['p01'], 'p10' : model_fit.params['p10']} 44 | model_fit = get_2DSPT_params(all_tracks, dt, nb_substeps = nb_substeps, nb_states = 2, do_frame = 1,frame_len = frame_len,cell_dims = cell_dims, verbose = 0, vary_params = vary_params, estimated_vals = estimated_vals, steady_state = True) 45 | if res_val - 0.1 > model_fit.residual: 46 | res_val = model_fit.residual 47 | else: 48 | keep_running = 0 49 | 50 | q = [param + ' = ' + str(np.round(model_fit.params[param].value, 4)) for param in model_fit.params] 51 | print(model_fit.residual[0], q) 52 | 53 | preds = predict_Bs(all_tracks, dt, model_fit.params, 2, frame_len = 12) 54 | return model_fit, preds 55 | 56 | def fit_3states(all_tracks, 57 | dt, 58 | steady_state = True, 59 | vary_params = { 'LocErr' : True, 'D0' : True, 'D1' : True, 'D2' : True, 'F0' : True, 'F1' : True, 'p01' : True, 'p02' : True, 'p10' : True,'p12' : True,'p20' : True, 'p21' : True, 'pBL': True}, 60 | estimated_vals = { 'LocErr' : 0.023, 'D0' : 1e-20, 'D1' : 0.02, 'D2' : 1, 'F0' : 0.33, 'F1' : 0.33, 'p01' : 0.1, 'p02' : 0.1, 'p10' :0.1, 'p12' : 0.1, 'p20' :0.1, 'p21' :0.1, 'pBL': 0.1}, 61 | min_values = { 'LocErr' : 0.007, 'D0' : 1e-20, 'D1' : 0.0000001, 'D2' : 0.000001, 'F0' : 0.001, 'F1' : 0.001, 'p01' : 0.001, 'p02' : 0.001, 'p10' :0.001, 'p12' : 0.001, 'p20' :0.001, 'p21' :0.001, 'pBL': 0.001}, 62 | max_values = { 'LocErr' : 0.6, 'D0' : 1e-20, 'D1' : 1, 'D2' : 10, 'F0' : 0.999, 'F1' : 0.999, 'p01' : 0.5, 'p02' : 0.5, 'p10' : 0.5, 'p12' : 0.5, 'p20' : 0.5, 'p21' : 0.5, 'pBL': 0.3}): 63 | ''' 64 | all_tracks is a dict of numpy arrays of tracks, each numpy array have the following dimensions : dim 0 = track ID, dim 1 = time position, dim 2 = x and y positions 65 | dt is a scalar : the time per frame 66 | steady_state = True assume fractions are determined by the rates, fractions and rates are independent if False 67 | the other optional lists allow to fit or fix parameter fitting (vary_params), chose values (estimated_vals), 68 | and limits for the fit (min_values and max_values) 69 | ''' 70 | model_fit = get_2DSPT_params(all_tracks, 71 | dt, 72 | nb_substeps = 1, 73 | nb_states = 3, 74 | do_frame = 1, 75 | frame_len = 5, 76 | verbose = 1, 77 | cell_dims = cell_dims, 78 | steady_state = steady_state, 79 | vary_params = vary_params, 80 | estimated_vals = estimated_vals, 81 | min_values = min_values, 82 | max_values = max_values) 83 | 84 | keep_running = 1 85 | res_val = 0 86 | for kk in range(40): 87 | if keep_running: 88 | estimated_vals = { 'LocErr' : model_fit.params['LocErr'], 'D0' : model_fit.params['D0'], 'D1' : model_fit.params['D1'], 'D2' : model_fit.params['D2'], 'F0' : model_fit.params['F0'], 'F1' : model_fit.params['F1'], 'p01' : model_fit.params['p01'], 'p02' : model_fit.params['p02'], 'p10' :model_fit.params['p10'], 'p12' :model_fit.params['p12'], 'p20' :model_fit.params['p20'], 'p21' :model_fit.params['p21']} 89 | model_fit = get_2DSPT_params(all_tracks, 90 | dt, 91 | nb_substeps = 1, 92 | nb_states = 3, 93 | do_frame = 1, 94 | frame_len = 5, 95 | verbose = 1, 96 | cell_dims = cell_dims, 97 | steady_state = steady_state, 98 | vary_params = vary_params, 99 | estimated_vals = estimated_vals, 100 | min_values = min_values, 101 | max_values = max_values) 102 | 103 | q = [param + ' = ' + str(np.round(model_fit.params[param].value, 4)) for param in model_fit.params] 104 | print(model_fit.residual[0], q) 105 | 106 | if res_val - 0.1 > model_fit.residual: 107 | res_val = model_fit.residual 108 | else: 109 | keep_running = 0 110 | 111 | preds = predict_Bs(all_tracks, dt, params = model_fit.params, nb_states = 3, frame_len = 7) 112 | return model_fit, preds 113 | 114 | 115 | 116 | -------------------------------------------------------------------------------- /extrack/exporters.py: -------------------------------------------------------------------------------- 1 | #def dict_pred_to_df_pred(all_Cs, all_Bs): 2 | import numpy as np 3 | import pickle 4 | import json 5 | import pandas as pd 6 | 7 | def save_params(params, path = '.', fmt = 'json', file_name = 'params'): 8 | save_params = {} 9 | [save_params.update({param : params[param].value}) for param in params] 10 | ''' 11 | available formats : json, npy, csv 12 | ''' 13 | if fmt == 'npy': 14 | np.save(path + '/' + file_name, save_params) 15 | elif fmt == 'pkl': 16 | with open(path + '/' + file_name + ".pkl", "wb") as tf: 17 | pickle.dump(save_params,tf) 18 | elif fmt == 'json': 19 | with open(path + '/' + file_name + ".json", "w") as tf: 20 | json.dump(save_params,tf) 21 | elif fmt == 'csv': 22 | with open(path + '/' + file_name + ".csv", 'w') as tf: 23 | for key in save_params.keys(): 24 | tf.write("%s,%s\n"%(key,save_params[key])) 25 | else : 26 | raise ValueError("format not supported, use one of : 'json', 'pkl', 'npy', 'csv'") 27 | 28 | def extrack_2_matrix(all_Css, pred_Bss, dt, all_frames = None): 29 | row_ID = 0 30 | nb_pos = 0 31 | for len_ID in all_Css: 32 | all_Cs = all_Css[len_ID] 33 | nb_pos += all_Cs.shape[0]*all_Cs.shape[1] 34 | 35 | matrix = np.empty((nb_pos, 4+pred_Bss[list(pred_Bss.keys())[0]].shape[2])) 36 | #TRACK_ID,POSITION_X,POSITION_Y,POSITION_Z,POSITION_T,FRAME,PRED_0, PRED_1,(PRED_2 etc) 37 | track_ID = 0 38 | for len_ID in all_Css: 39 | all_Cs = all_Css[len_ID] 40 | pred_Bs = pred_Bss[len_ID] 41 | if all_frames != None: 42 | all_frame = all_frames[len_ID] 43 | else: 44 | all_frame = np.arange(all_Cs.shape[0]*all_Cs.shape[1]).reshape((all_Cs.shape[0],all_Cs.shape[1])) 45 | for track, preds, frames in zip(all_Cs, pred_Bs, all_frame): 46 | track_IDs = np.full(len(track),track_ID)[:,None] 47 | frames = frames[:,None] 48 | cur_track = np.concatenate((track, track_IDs,frames,preds ),1) 49 | 50 | matrix[row_ID:row_ID+cur_track.shape[0]] = cur_track 51 | row_ID += cur_track.shape[0] 52 | track_ID+=1 53 | return matrix 54 | 55 | # all_tracks = tracks 56 | # pred_Bs = preds 57 | 58 | def extrack_2_pandas(all_tracks, pred_Bs, frames = None, opt_metrics = {}): 59 | ''' 60 | turn outputs form ExTrack to a unique pandas DataFrame 61 | ''' 62 | if frames is None: 63 | frames = {} 64 | for l in all_tracks: 65 | frames[l] = np.repeat(np.array([np.arange(int(l))]), len(all_tracks[l]), axis = 0) 66 | 67 | track_list = [] 68 | frames_list = [] 69 | track_ID_list = [] 70 | opt_metrics_list = [] 71 | for metric in opt_metrics: 72 | opt_metrics_list.append([]) 73 | 74 | cur_nb_track = 0 75 | pred_Bs_list = [] 76 | for l in all_tracks: 77 | track_list = track_list + list(all_tracks[l].reshape(all_tracks[l].shape[0] * all_tracks[l].shape[1], 2)) 78 | frames_list = frames_list + list(frames[l].reshape(frames[l].shape[0] * frames[l].shape[1], 1)) 79 | track_ID_list = track_ID_list + list(np.repeat(np.arange(cur_nb_track,cur_nb_track+all_tracks[l].shape[0]),all_tracks[l].shape[1])) 80 | cur_nb_track += all_tracks[l].shape[0] 81 | 82 | for j, metric in enumerate(opt_metrics): 83 | opt_metrics_list[j] = opt_metrics_list[j] + list(opt_metrics[metric][l].reshape(opt_metrics[metric][l].shape[0] * opt_metrics[metric][l].shape[1], 1)) 84 | 85 | n = pred_Bs[l].shape[2] 86 | pred_Bs_list = pred_Bs_list + list(pred_Bs[l].reshape(pred_Bs[l].shape[0] * pred_Bs[l].shape[1], n)) 87 | 88 | all_data = np.concatenate((np.array(track_list), np.array(frames_list), np.array(track_ID_list)[:,None], np.array(pred_Bs_list)), axis = 1) 89 | for opt_metric in opt_metrics_list: 90 | all_data = np.concatenate((all_data, opt_metric), axis = 1) 91 | 92 | nb_dims = len(track_list[0]) 93 | colnames = ['POSITION_X', 'POSITION_Y', 'POSITION_Z'][:nb_dims] + ['FRAME', 'TRACK_ID'] 94 | for i in range(np.array(pred_Bs_list).shape[1]): 95 | colnames = colnames + ['pred_' + str(i)] 96 | for metric in opt_metrics: 97 | colnames = colnames + [metric] 98 | 99 | df = pd.DataFrame(data = all_data, index = np.arange(len(all_data)), columns = colnames) 100 | df['FRAME'] = df['FRAME'].astype(int) 101 | df['TRACK_ID'] = df['TRACK_ID'].astype(int) 102 | return df 103 | 104 | 105 | def extrack_2_pandas2(tracks, pred_Bs, frames = None, opt_metrics = {}): 106 | ''' 107 | turn outputs form ExTrack to a unique pandas DataFrame 108 | ''' 109 | if frames is None: 110 | frames = {} 111 | for l in tracks: 112 | frames[l] = np.repeat(np.array([np.arange(int(l))]), len(tracks[l]), axis = 0) 113 | 114 | n = 0 115 | for l in tracks: 116 | n+= tracks[l].shape[0]*tracks[l].shape[1] 117 | 118 | nb_dims = tracks[l].shape[2] 119 | 120 | nb_states = pred_Bs[list(pred_Bs.keys())[0]].shape[-1] 121 | 122 | flat_tracks = np.zeros((n, tracks[l].shape[2])) 123 | flat_frames = np.zeros((n, 1)) 124 | flat_Track_IDs = np.zeros((n, 1)) 125 | flat_opt_metrics = np.zeros((n, len(opt_metrics.keys()))) 126 | flat_preds = np.zeros((n, nb_states)) 127 | 128 | track_ID = 0 129 | k = 0 130 | for l in tracks: 131 | for i, (track, f, p) in enumerate(zip(tracks[l], frames[l], pred_Bs[l])): 132 | track_length = track.shape[0] 133 | flat_tracks[k:k+track_length] = track 134 | flat_frames[k:k+track_length] = f[:, None] 135 | flat_Track_IDs[k:k+track_length] = track_ID 136 | flat_preds[k:k+track_length] = p 137 | for j, metric in enumerate(opt_metrics): 138 | flat_opt_metrics[k:k+track_length, j] = opt_metrics[metric][l][i] 139 | k+=track_length 140 | track_ID+=1 141 | 142 | arr = np.concatenate((flat_tracks, flat_frames, flat_Track_IDs, flat_opt_metrics, flat_preds), axis = 1) 143 | columns = ['POSITION_X', 'POSITION_Y', 'POSITION_Z'][:nb_dims] + ['FRAME', 'TRACK_ID'] + list(opt_metrics.keys()) 144 | for i in range(nb_states): 145 | columns = columns + ['pred_' + str(i)] 146 | 147 | df = pd.DataFrame(data = arr, columns = columns) 148 | df['FRAME'] = df['FRAME'].astype(int) 149 | df['TRACK_ID'] = df['TRACK_ID'].astype(int) 150 | return df 151 | 152 | def save_extrack_2_CSV(path, all_tracks, pred_Bss, dt, all_frames = None): 153 | track_ID = 0 154 | 155 | preds_header_str_fmt = '' 156 | preds_str_fmt = '' 157 | for k in range(pred_Bss[list(pred_Bss.keys())[0]].shape[2]): 158 | preds_header_str_fmt = preds_header_str_fmt + 'PRED_%s,'%(k) 159 | preds_str_fmt = preds_str_fmt + ',%s' 160 | 161 | with open(path, 'w') as f: 162 | f.write('TRACK_ID,POSITION_X,POSITION_Y,POSITION_Z,POSITION_T,FRAME,%s\n'%(preds_header_str_fmt)) 163 | 164 | for len_ID in all_tracks: 165 | nb_dims = all_tracks[len_ID].shape[2] 166 | tracks = np.zeros((all_tracks[len_ID].shape[0], all_tracks[len_ID].shape[1], 3)) # create 3D tracks with values 0 in unknown dims 167 | tracks[:,:, :nb_dims] = all_tracks[len_ID] 168 | pred_Bs = pred_Bss[len_ID] 169 | if all_frames != None: 170 | all_frame = all_frames[len_ID] 171 | else: 172 | all_frame = np.arange(tracks.shape[0]*tracks.shape[1]).reshape((tracks.shape[0],tracks.shape[1])) 173 | for track, preds, frames in zip(tracks, pred_Bs, all_frame): 174 | track_ID+=1 175 | for pos, p, frame in zip(track, preds, frames): 176 | preds_str = preds_str_fmt%(tuple(p)) 177 | f.write('%s,%s,%s,%s,%s,%s%s\n'%(track_ID, pos[0], pos[1], pos[2], dt* frame*1000, frame, preds_str)) 178 | 179 | def save_extrack_2_xml(all_tracks, pred_Bss, params, path, dt, all_frames = None, opt_metrics = {}): 180 | track_ID = 0 181 | for len_ID in all_tracks: 182 | tracks = all_tracks[len_ID] 183 | track_ID += len(tracks) 184 | nb_dims = all_tracks[len_ID].shape[2] 185 | 186 | final_params = [] 187 | for param in params: 188 | if not '_' in param: 189 | final_params.append(param) 190 | Extrack_headers = 'ExTrack_results="' 191 | 192 | for param in final_params: 193 | Extrack_headers = Extrack_headers + param + "='" + str(np.round(params[param].value, 8)) +"' " 194 | Extrack_headers += '"' 195 | 196 | preds_str_fmt = '' 197 | for k in range(pred_Bss[list(pred_Bss.keys())[0]].shape[2]): 198 | preds_str_fmt = preds_str_fmt + ' pred_%s="%s"'%(k,'%s') 199 | 200 | opt_metrics_fmt = '' 201 | 202 | for m in opt_metrics: 203 | opt_metrics_fmt = opt_metrics_fmt + '%s="%s" '%(m,'%s') 204 | 205 | with open(path, 'w') as f: 206 | f.write('\n\n'%(track_ID, dt, Extrack_headers)) 207 | 208 | for len_ID in all_tracks: 209 | tracks = np.zeros((all_tracks[len_ID].shape[0], all_tracks[len_ID].shape[1], 3)) 210 | tracks[:,:, :nb_dims] = all_tracks[len_ID] 211 | pred_Bs = pred_Bss[len_ID] 212 | opt_met = np.empty((tracks.shape[0],tracks.shape[1],len(opt_metrics))) 213 | for i, m in enumerate(opt_metrics): 214 | opt_met[:,:,i] = opt_metrics[m][len_ID] 215 | opt_met.shape 216 | if all_frames != None: 217 | all_frame = all_frames[len_ID] 218 | else: 219 | all_frame = np.arange(tracks.shape[0]*tracks.shape[1]).reshape((tracks.shape[0],tracks.shape[1])) 220 | for i, (track, preds, frames) in enumerate(zip(tracks, pred_Bs, all_frame)): 221 | track_opt_met = opt_met[i] 222 | f.write(' \n'%(len_ID)) 223 | for pos, p, frame, track_opt_met in zip(track, preds, frames, track_opt_met): 224 | preds_str = preds_str_fmt%(tuple(p)) 225 | opt_metrics_str = opt_metrics_fmt%(tuple(track_opt_met)) 226 | f.write(' \n'%(frame,pos[0],pos[1],pos[2], preds_str, opt_metrics_str)) 227 | f.write(' \n') 228 | f.write('\n') 229 | 230 | 231 | def save_extrack_2_input_xml(all_tracks, pred_Bss, params, path, dt, all_frames = None, opt_metrics = {}): 232 | ''' 233 | xml format for vizualization with TrackMate using the plugin "Load a TrackMate file" 234 | ''' 235 | track_ID = 0 236 | for len_ID in all_tracks: 237 | tracks = all_tracks[len_ID] 238 | track_ID += len(tracks) 239 | nb_dims = all_tracks[len_ID].shape[2] 240 | 241 | final_params = [] 242 | for param in params: 243 | if not '_' in param: 244 | final_params.append(param) 245 | Extrack_headers = 'ExTrack_results="' 246 | 247 | for param in final_params: 248 | Extrack_headers = Extrack_headers + param + "='" + str(np.round(params[param].value, 8)) +"' " 249 | Extrack_headers += '"' 250 | 251 | preds_str_fmt = '' 252 | for k in range(pred_Bss[list(pred_Bss.keys())[0]].shape[2]): 253 | preds_str_fmt = preds_str_fmt + ' pred_%s="%s"'%(k,'%s') 254 | 255 | opt_metrics_fmt = '' 256 | 257 | for m in opt_metrics: 258 | opt_metrics_fmt = opt_metrics_fmt + '%s="%s" '%(m,'%s') 259 | 260 | with open(path, 'w', encoding="utf-8") as f: 261 | f.write('\n\n \n \n \n') 262 | f.write(' \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n') 263 | nspots = 0 264 | frames = [] 265 | new_all_frames = {} 266 | all_spot_IDs = {} 267 | for len_ID in all_tracks: 268 | new_all_frames[len_ID] = [] 269 | nspots = nspots + all_tracks[len_ID].shape[0] * all_tracks[len_ID].shape[1] 270 | if all_frames == None: 271 | frames = frames + list(np.arange(0, all_tracks[len_ID].shape[1])) 272 | new_all_frames[len_ID] = np.repeat(np.arange(0, all_tracks[len_ID].shape[1])[None], all_tracks[len_ID].shape[0], 0) 273 | else: 274 | for frame in all_frames[len_ID]: 275 | frames = frames + list(frame) # not tested 276 | if all_frames == None: 277 | all_frames = new_all_frames 278 | frames = np.unique(frames) 279 | f.write(' \n \n'%nspots) 280 | spot_ID = 0 281 | for len_ID in all_tracks: 282 | all_spot_IDs[len_ID] = np.zeros(all_frames[len_ID].shape).astype(int) 283 | for frame in frames: 284 | for len_ID in all_tracks: 285 | cur_frames = all_frames[len_ID] 286 | tracks = all_tracks[len_ID] 287 | for i, (track, fm) in enumerate(zip(tracks, cur_frames)): 288 | print(i) 289 | pos_ID = np.where(frame == fm)[0] 290 | if len(pos_ID)>0: 291 | 292 | pos_ID = pos_ID[0] 293 | pos = np.zeros((3)) 294 | pos[:len(track[pos_ID])] = track[pos_ID] 295 | all_spot_IDs[len_ID][i, pos_ID] = spot_ID 296 | f.write(' \n'%(spot_ID, spot_ID, frame*dt, pos[0], pos[1], frame,pos[2])) 297 | spot_ID = spot_ID + 1 298 | f.write(' \n \n \n') 299 | track_ID = 0 300 | all_track_IDs = [] 301 | for len_ID in all_tracks: 302 | tracks = all_tracks[len_ID] 303 | frames = all_frames[len_ID] 304 | spot_IDss = all_spot_IDs[len_ID] 305 | for track, frame, spot_IDs in zip(tracks, frames, spot_IDss): 306 | f.write(' \n'%(track_ID, track_ID, track_ID, track.shape[0], frame[-1] - frame[0] + 1 - track.shape[0], (frame[-1] - frame[0])*dt, frame[0]*dt, frame[-1]*dt)) 307 | (pos, fm, spot_ID) = track[0], frame[0], spot_IDs[0] 308 | previous_spot_ID = spot_ID 309 | for pos, fm, spot_ID in zip(track[1:], frame[1:], spot_IDs[1:]): 310 | f.write(' \n'%(previous_spot_ID, spot_ID, dt/2 + (track.shape[0]-1)*dt, pos[0], pos[1])) 311 | previous_spot_ID = spot_ID 312 | all_track_IDs.append(track_ID) 313 | track_ID = track_ID + 1 314 | f.write(' \n') 315 | f.write(' \n \n') 316 | for track_ID in all_track_IDs: 317 | f.write(' \n'%track_ID) 318 | f.write(' \n') 319 | f.write(' \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n {\n "name": "CurrentDisplaySettings",\n "spotUniformColor": "204, 51, 204, 255",\n "spotColorByType": "TRACKS",\n "spotColorByFeature": "TRACK_INDEX",\n "spotDisplayRadius": 0.1,\n "spotDisplayedAsRoi": true,\n "spotMin": 0.0,\n "spotMax": 10.0,\n "spotShowName": false,\n "trackMin": 0.0,\n "trackMax": 10.0,\n "trackColorByType": "TRACKS",\n "trackColorByFeature": "TRACK_INDEX",\n "trackUniformColor": "204, 204, 51, 255",\n "undefinedValueColor": "0, 0, 0, 255",\n "missingValueColor": "89, 89, 89, 255",\n "highlightColor": "51, 230, 51, 255",\n "trackDisplayMode": "LOCAL",\n "colormap": "Jet",\n "limitZDrawingDepth": false,\n "drawingZDepth": 10.0,\n "fadeTracks": true,\n "fadeTrackRange": 5,\n "useAntialiasing": true,\n "spotVisible": true,\n "trackVisible": true,\n "font": {\n "name": "Arial",\n "style": 1,\n "size": 12,\n "pointSize": 12.0,\n "fontSerializedDataVersion": 1\n },\n "lineThickness": 1.0,\n "selectionLineThickness": 4.0,\n "trackschemeBackgroundColor1": "128, 128, 128, 255",\n "trackschemeBackgroundColor2": "192, 192, 192, 255",\n "trackschemeForegroundColor": "0, 0, 0, 255",\n "trackschemeDecorationColor": "0, 0, 0, 255",\n "trackschemeFillBox": false,\n "spotFilled": false,\n "spotTransparencyAlpha": 1.0\n}\n\n') 320 | -------------------------------------------------------------------------------- /extrack/histograms.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | """ 5 | Created on Thu Nov 11 17:01:56 2021 6 | @author: francois 7 | """ 8 | 9 | import numpy as np 10 | from itertools import product 11 | 12 | GPU_computing = False 13 | 14 | if GPU_computing : 15 | import cupy as cp 16 | from cupy import asnumpy 17 | else : 18 | import numpy as cp 19 | def asnumpy(x): 20 | return np.array(x) 21 | 22 | import multiprocessing 23 | import scipy 24 | from extrack.tracking import extract_params, get_all_Bs, get_Ts_from_Bs, first_log_integrale_dif, log_integrale_dif 25 | 26 | def P_segment_len(Cs, LocErr, ds, Fs, TrMat, min_l = 3, pBL=0.1, isBL = 1, cell_dims = [0.5], nb_substeps=1, max_nb_states = 1000) : 27 | ''' 28 | compute the product of the integrals over Ri as previousily described 29 | work in log space to avoid overflow and underflow 30 | 31 | Cs : dim 0 = track ID, dim 1 : states, dim 2 : peaks postions through time, 32 | dim 3 : x, y position 33 | 34 | we process by steps, at each step we account for 1 more localization, we compute 35 | the canstant (LC), the mean (m_arr) and std (s2_arr) of of the normal distribution 36 | resulting from integration. 37 | 38 | each step is made of substeps if nb_substeps > 1, and we increase the matrix 39 | of possible Bs : cur_Bs accordingly 40 | 41 | to be able to process long tracks with good accuracy, for each track we fuse m_arr and s2_arr 42 | of sequences of states equal exept for the state 'frame_len' steps ago. 43 | ''' 44 | nb_Tracks = Cs.shape[0] 45 | nb_locs = Cs.shape[1] # number of localization per track 46 | nb_dims = Cs.shape[2] # number of spatial dimentions (x, y) or (x, y, z) 47 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 48 | Cs = cp.array(Cs) 49 | nb_states = TrMat.shape[0] 50 | Cs = Cs[:,:,::-1] 51 | LocErr = LocErr[:,None] 52 | LocErr = LocErr[:,:,::-1] # useful when single peak localization error is inputed, 53 | LocErr2 = LocErr**2 54 | if LocErr.shape[2] == 1: # combined to min(LocErr_index, nb_locs-current_step) it will select the right index 55 | LocErr_index = -1 56 | elif LocErr.shape[2] == nb_locs: 57 | LocErr_index = nb_locs 58 | else: 59 | raise ValueError("Localization error is not specified correctly, in case of unique localization error specify a float number in estimated_vals['LocErr'].\n If one localization error per dimension, specify a list or 1D array of elements the localization error for each dimension.\n If localization error is predetermined by another method for each position the argument input_LocErr should be a dict for each track length of the 3D arrays corresponding to all_tracks (can be obtained from the reader functions using the opt_colname argument)") 60 | 61 | cell_dims = np.array(cell_dims) 62 | cell_dims = cell_dims[cell_dims!=None] 63 | 64 | cur_Bs = get_all_Bs(nb_substeps + 1, nb_states)[None] 65 | 66 | TrMat = cp.array(TrMat.T) 67 | current_step = 1 68 | if nb_locs ==1: 69 | 70 | cur_states = get_all_Bs(1, nb_states)[None] #states of interest for the current displacement 71 | cur_Bs = get_all_Bs(2, nb_states)[None] 72 | 73 | cur_d2s = ds[cur_states]**2 74 | cur_d2s = (cur_d2s[:,:,1:] + cur_d2s[:,:,:-1]) / 2 # assuming a transition at the middle of the substeps 75 | 76 | # we can average the variances of displacements per step to get the actual std of displacements per step 77 | cur_d2s = cp.mean(cur_d2s, axis = 2) 78 | cur_d2s = cur_d2s[:,:,None] 79 | cur_d2s = cp.array(cur_d2s) 80 | 81 | sub_Bs = cur_Bs.copy()[:1,:cur_Bs.shape[1]//nb_states,:nb_substeps] # list of possible current states we can meet to compute the proba of staying in the FOV 82 | sub_ds = cp.mean(ds[sub_Bs]**2, axis = 2)**0.5 # corresponding list of d 83 | 84 | p_stay = np.ones(sub_ds.shape[-1]) 85 | for cell_len in cell_dims: 86 | xs = np.linspace(0+cell_len/2000,cell_len-cell_len/2000,1000) 87 | cur_p_stay = ((np.mean(scipy.stats.norm.cdf((cell_len-xs[:,None])/(sub_ds+1e-200)) - scipy.stats.norm.cdf(-xs[:,None]/(sub_ds+1e-200)),0))*2)/2 # proba to stay in the FOV for each of the possible cur Bs 88 | p_stay = p_stay*cur_p_stay 89 | Lp_stay = np.log(p_stay * (1-pBL)) # proba for the track to survive = both stay in the FOV and not bleach 90 | 91 | LL = Lp_stay[np.argmax(np.all(cur_Bs[:,None,:,:-1] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 92 | LP = np.zeros((LL.shape)) 93 | 94 | cur_Bs = np.repeat(cur_Bs,nb_Tracks,axis = 0) 95 | cur_Bs = cur_Bs[:,:,1:] 96 | 97 | else: 98 | cur_Bs = get_all_Bs(nb_substeps + 1, nb_states)[None] 99 | cur_Bs = np.repeat(cur_Bs, nb_Tracks, 0) 100 | 101 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) #states of interest for the current displacement 102 | cur_nb_Bs = cur_Bs.shape[1] 103 | # compute the vector of diffusion stds knowing the current states 104 | ds = cp.array(ds) 105 | Fs = cp.array(Fs) 106 | 107 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 108 | LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 109 | 110 | LP = LT + LF #+ compensate_leaving 111 | LL = np.zeros(LP.shape) 112 | # current log proba of seeing the track 113 | #LP = cp.repeat(LP, nb_Tracks, axis = 0) 114 | cur_d2s = ds[cur_states]**2 115 | cur_d2s = (cur_d2s[:,:,1:] + cur_d2s[:,:,:-1]) / 2 # assuming a transition at the middle of the substeps 116 | 117 | # we can average the variances of displacements per step to get the actual std of displacements per step 118 | cur_d2s = cp.mean(cur_d2s, axis = 2) 119 | cur_d2s = cur_d2s[:,:,None] 120 | cur_d2s = cp.array(cur_d2s) 121 | 122 | sub_Bs = cur_Bs.copy()[:1,:cur_Bs.shape[1]//nb_states,:nb_substeps] # list of possible current states we can meet to compute the proba of staying in the FOV 123 | sub_ds = cp.mean(ds[sub_Bs]**2, axis = 2)**0.5 # corresponding list of d 124 | 125 | p_stay = np.ones(sub_ds.shape[-1]) 126 | for cell_len in cell_dims: 127 | xs = np.linspace(0+cell_len/2000,cell_len-cell_len/2000,1000) 128 | cur_p_stay = ((np.mean(scipy.stats.norm.cdf((cell_len-xs[:,None])/(sub_ds+1e-200)) - scipy.stats.norm.cdf(-xs[:,None]/(sub_ds+1e-200)),0))) # proba to stay in the FOV for each of the possible cur Bs 129 | p_stay = p_stay*cur_p_stay 130 | Lp_stay = np.log(p_stay * (1-pBL)) # proba for the track to survive = both stay in the FOV and not bleach 131 | 132 | if current_step >= min_l: 133 | LL = LL + Lp_stay[np.argmax(np.all(cur_states[:,None,:,:-1] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 134 | 135 | # inject the first position to get the associated m_arr and s2_arr : 136 | m_arr, s2_arr = first_log_integrale_dif(Cs[:,:, nb_locs-current_step], LocErr2[:,:, min(LocErr_index, nb_locs-current_step)], cur_d2s) 137 | current_step += 1 138 | m_arr = cp.repeat(m_arr, cur_nb_Bs, axis = 1) 139 | removed_steps = 0 140 | 141 | while current_step <= nb_locs-1: 142 | # update cur_Bs to describe the states at the next step : 143 | #cur_Bs = get_all_Bs(current_step*nb_substeps+1 - removed_steps, nb_states)[None] 144 | #cur_Bs = all_Bs[:,:nb_states**(current_step*nb_substeps+1 - removed_steps),:current_step*nb_substeps+1 - removed_steps] 145 | for iii in range(nb_substeps): 146 | cur_Bs = np.concatenate((np.repeat(np.mod(np.arange(cur_Bs.shape[1]*nb_states),nb_states)[None,:,None], nb_Tracks,0),np.repeat(cur_Bs,nb_states,1)),-1) 147 | 148 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) 149 | # compute the vector of diffusion stds knowing the states at the current step 150 | cur_d2s = ds[cur_states]**2 151 | cur_d2s = (cur_d2s[:,:,1:] + cur_d2s[:,:,:-1]) / 2 # assuming a transition at the middle of the substeps 152 | 153 | # we can average the variances of displacements per step to get the actual std of displacements per step 154 | cur_d2s = cp.mean(cur_d2s, axis = 2) 155 | cur_d2s = cur_d2s[:,:,None] 156 | cur_d2s = cp.array(cur_d2s) 157 | LT = get_Ts_from_Bs(cur_states, TrMat) 158 | 159 | # repeat the previous matrix to account for the states variations due to the new position 160 | m_arr = cp.repeat(m_arr, nb_states**nb_substeps , axis = 1) 161 | s2_arr = cp.repeat(s2_arr, nb_states**nb_substeps, axis = 1) 162 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1) 163 | LL = cp.repeat(LL, nb_states**nb_substeps, axis = 1) 164 | # inject the next position to get the associated m_arr, s2_arr and Constant describing the integral of 3 normal laws : 165 | m_arr, s2_arr, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr2[:,:, min(LocErr_index, nb_locs-current_step)], cur_d2s, m_arr, s2_arr) 166 | #print('integral',time.time() - t0); t0 = time.time() 167 | if current_step >= min_l : 168 | LL = LL + Lp_stay[np.argmax(np.all(cur_states[:,None,:,:-1] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 169 | 170 | LP += LT + LC # current (log) constants associated with each track and sequences of states 171 | del LT, LC 172 | cur_nb_Bs = len(cur_Bs[0]) # current number of sequences of states 173 | 174 | ''''idea : the position and the state 6 steps ago should not impact too much the 175 | probability of the next position so the m_arr and s2_arr of tracks with the same 6 last 176 | states must be very similar, we can then fuse the parameters of the pairs of Bs 177 | which vary only for the last step (7) and sum their probas''' 178 | if current_step < nb_locs-1: 179 | if cur_nb_Bs > max_nb_states: 180 | 181 | #new_s2_arr = cp.array((s2_arr + LocErr2))[:,:,0] 182 | #log_integrated_term = -cp.log(2*np.pi*new_s2_arr) - cp.sum((Cs[:,:,nb_locs-current_step] - m_arr)**2,axis=2)/(2*new_s2_arr) 183 | new_s2_arr = cp.array((s2_arr + LocErr2[:,:, min(LocErr_index, nb_locs-current_step-1)])) 184 | log_integrated_term = cp.sum(-0.5*cp.log(2*np.pi*new_s2_arr) - (Cs[:,:,nb_locs-current_step-1] - m_arr)**2/(2*new_s2_arr),axis=2) 185 | LF = 0 #cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 186 | 187 | test_LP = LP + log_integrated_term + LF 188 | LP.shape 189 | if np.max(test_LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 190 | test_LP = test_LP - (np.max(test_LP)-600) 191 | 192 | P = np.exp(test_LP) 193 | 194 | argP = P.argsort() 195 | argP = argP[:,::-1] 196 | 197 | m_arr = np.take_along_axis(m_arr, argP[:,:,None], axis = 1)[:,:max_nb_states] 198 | s2_arr = np.take_along_axis(s2_arr, argP[:,:,None], axis = 1)[:,:max_nb_states] 199 | LP = np.take_along_axis(LP, argP, axis = 1)[:,:max_nb_states] 200 | LL = np.take_along_axis(LL, argP, axis = 1)[:,-max_nb_states:] 201 | cur_Bs = np.take_along_axis(cur_Bs, argP[:,:,None], axis = 1)[:,:max_nb_states] 202 | 203 | P = np.take_along_axis(P, argP, axis = 1)[:,:max_nb_states] 204 | 205 | cur_nb_Bs = cur_Bs.shape[1] 206 | removed_steps += 1 207 | 208 | current_step += 1 209 | 210 | 211 | if isBL: 212 | for iii in range(nb_substeps): 213 | cur_Bs = np.concatenate((np.repeat(np.mod(np.arange(cur_Bs.shape[1]*nb_states),nb_states)[None,:,None], nb_Tracks,0),np.repeat(cur_Bs,nb_states,1)),-1) 214 | 215 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) 216 | 217 | LT = get_Ts_from_Bs(cur_states, TrMat) 218 | #cur_states = cur_states[:,:,0] 219 | # repeat the previous matrix to account for the states variations due to the new position 220 | m_arr = cp.repeat(m_arr, nb_states**nb_substeps , axis = 1) 221 | s2_arr = cp.repeat(s2_arr, nb_states**nb_substeps, axis = 1) 222 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1)# + LT 223 | LL = cp.repeat(LL, nb_states**nb_substeps, axis = 1) 224 | #LL = Lp_stay[np.argmax(np.all(cur_states[:,None] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 225 | end_p_stay = p_stay[np.argmax(np.all(cur_states[:,None,:] == sub_Bs[:,:,None],-1),1)] 226 | 227 | np.all(cur_states[:,None,:] == sub_Bs[:,:,None],-1).shape 228 | cur_states[:,:,0] 229 | sub_Bs[:,:,None].shape 230 | end_p_stay 231 | LP.shape 232 | LL = LL + np.log(pBL + (1-end_p_stay) - pBL * (1-end_p_stay)) 233 | cur_Bs = cur_Bs[:,:,1:] 234 | #isBL = 0 235 | 236 | #new_s2_arr = cp.array((s2_arr + LocErr2))[:,:,0] 237 | #log_integrated_term = -cp.log(2*np.pi*new_s2_arr) - cp.sum((Cs[:,:,0] - m_arr)**2,axis=2)/(2*new_s2_arr) 238 | new_s2_arr = cp.array((s2_arr + LocErr2[:,:, min(LocErr_index, 0)])) 239 | log_integrated_term = cp.sum(-0.5*cp.log(2*np.pi*new_s2_arr) - (Cs[:,:,0] - m_arr)**2/(2*new_s2_arr),axis=2) 240 | 241 | #LF = cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 242 | #LF = cp.log(0.5) 243 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 244 | LP += log_integrated_term 245 | 246 | LP = LP 247 | if np.max(LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 248 | LP = LP - (np.max(LP, axis = 0, keepdims=True)-600) 249 | 250 | P = np.exp(LP+LL) 251 | #P = np.exp(LP) 252 | 253 | cur_nb_Bs = len(cur_Bs[0]) 254 | cur_pos = cur_Bs[:,:,0] 255 | cur_seg_len = np.ones((nb_Tracks, cur_nb_Bs)) 256 | seg_lens = np.zeros((nb_Tracks, cur_nb_Bs, nb_locs, nb_states)) # dims : track ID, sequence of states ID, position in the sequence, state of the sequence 257 | if nb_locs == 1: 258 | last_len = 1 259 | else: 260 | for k in range(1, nb_locs): 261 | is_Tr = cur_pos != cur_Bs[:,:,k] 262 | cur_seg_len = cur_seg_len + (is_Tr==0).astype(int) 263 | len_before_tr = (cur_seg_len * is_Tr.astype(int))[:,:,None] # cur_seg_len * is_Tr.astype(int)) selects the segments that stops so we can add them, if 0 : not transition if n : transition after n consecutive steps 264 | cat_states = (cur_pos[:,:,None] == np.arange(nb_states)[None,None]).astype(int) # position of the segment in categorcal format so it can fit seg_lens 265 | seg_lens[:,:,k-1] = len_before_tr * cat_states # add the segment len of the corresponding states 266 | cur_seg_len[is_Tr] = 1 267 | cur_pos = cur_Bs[:,:,k] 268 | last_len = (nb_locs - np.sum(seg_lens, (2,3)))[:,:,None] 269 | 270 | cat_states = (cur_pos[:,:,None] == np.arange(nb_states)[None,None]).astype(int) 271 | seg_lens[:,:,-1] = last_len * cat_states # include the lenght of the last position (bleaching or leaving the FOV) 272 | #seg_lens[:,:,-1] = 0 # do not include the lenght of the last position (bleaching or leaving the FOV) 273 | #seg_lens[:,:,0] = 0 # do not include the lenght of the last position (bleaching or leaving the FOV) 274 | #last_seg_lens = seg_lens[:,:,-1:] 275 | 276 | seg_len_hist = np.zeros((nb_locs-1, nb_states)) 277 | #last_seg_len_hist = np.zeros((nb_locs, nb_states)) 278 | 279 | for k in range(1, nb_locs): 280 | P_seg = ((P/np.sum(P, axis=1, keepdims=True))[:,:,None,None] * (seg_lens == k).astype(int)) 281 | #P_seg = np.sum(P_seg, (1,2)) / (np.sum(P_seg,(1,2,3))[:,None] +1e-300) # seemed good, normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 282 | P_seg = np.sum(P_seg, (1,2)) # seemed good, normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 283 | P_seg = np.sum(P_seg,0) 284 | seg_len_hist[k-1] = P_seg 285 | 286 | return LP, cur_Bs, seg_len_hist 287 | 288 | def pool_star_P_seg(args): 289 | return P_segment_len(*args)[-1] 290 | 291 | # all_tracks = tracks 292 | # params = lmfit_params 293 | 294 | def len_hist(all_tracks, 295 | params, 296 | dt, 297 | cell_dims=[0.5,None,None], 298 | nb_states=2, 299 | max_nb_states = 500, 300 | workers = 1, 301 | nb_substeps=1, 302 | input_LocErr = None 303 | ): 304 | ''' 305 | each probability can be multiplied to get a likelihood of the model knowing 306 | the parameters LocErr, D0 the diff coefficient of state 0 and F0 fraction of 307 | state 0, D1 the D coef at state 1, p01 the probability of transition from 308 | state 0 to 1 and p10 the proba of transition from state 1 to 0. 309 | here sum the logs(likelihood) to avoid too big numbers 310 | ''' 311 | min_l = np.min((np.array(list(all_tracks.keys()))).astype(int)) 312 | 313 | if type(input_LocErr) == dict: 314 | new_input_LocErr = [] 315 | for l in input_LocErr: 316 | new_input_LocErr.append(input_LocErr[l]) 317 | input_LocErr = new_input_LocErr 318 | 319 | if type(all_tracks) == dict: 320 | new_all_tracks = [] 321 | for l in all_tracks: 322 | new_all_tracks.append(all_tracks[l]) 323 | all_tracks = new_all_tracks 324 | 325 | LocErr, ds, Fs, TrMat, pBL = extract_params(params, dt, nb_states, nb_substeps, input_LocErr) 326 | 327 | Csss = [] 328 | sigss = [] 329 | isBLs = [] 330 | for k in range(len(all_tracks)): 331 | if k == len(all_tracks)-1: 332 | isBL = 0 # last position correspond to tracks which didn't disapear within maximum track length 333 | else: 334 | isBL = 1 335 | Css = all_tracks[k] 336 | if input_LocErr != None: 337 | sigs = LocErr[k] 338 | nb_max = 50 339 | for n in range(int(np.ceil(len(Css)/nb_max))): 340 | Csss.append(Css[n*nb_max:(n+1)*nb_max]) 341 | if input_LocErr != None: 342 | sigss.append(sigs[n*nb_max:(n+1)*nb_max]) 343 | if k == len(all_tracks)-1: 344 | isBLs.append(0) # last position correspond to tracks which didn't disapear within maximum track length 345 | else: 346 | isBLs.append(1) 347 | #Csss.reverse() 348 | #sigss.reverse() 349 | print('number of chunks:', len(isBLs)) 350 | 351 | args_prod = np.array(list(product(Csss, [0], [ds], [Fs], [TrMat], [min_l],[pBL], [0],[cell_dims], [nb_substeps], [max_nb_states])), dtype=object) 352 | args_prod[:, 7] = isBLs 353 | if input_LocErr != None: 354 | args_prod[:,1] = sigss 355 | else: 356 | args_prod[:,1] = LocErr 357 | 358 | Cs, LocErr, ds, Fs, TrMat, min_l, pBL,isBL, cell_dims, nb_substeps, max_nb_states = args_prod[0] 359 | 360 | if workers >= 2: 361 | with multiprocessing.Pool(workers) as pool: 362 | list_seg_len_hists = pool.map(pool_star_P_seg, args_prod) 363 | else: 364 | list_seg_len_hists = [] 365 | for k, args in enumerate(args_prod): 366 | 367 | list_seg_len_hists.append(pool_star_P_seg(args)) 368 | 369 | seg_len_hists = np.zeros((all_tracks[-1].shape[1],nb_states)) 370 | for seg_len_hist in list_seg_len_hists: 371 | seg_len_hists[:seg_len_hist.shape[0]] = seg_len_hists[:seg_len_hist.shape[0]] + seg_len_hist 372 | print('') 373 | return seg_len_hists 374 | 375 | ''' 376 | if type(all_tracks) == dict: 377 | new_all_tracks = [] 378 | for l in all_tracks: 379 | new_all_tracks.append(all_tracks[l]) 380 | all_tracks = new_all_tracks 381 | 382 | #seg_len_hists = np.zeros((all_tracks[-1].shape[1]+1,nb_states)) 383 | seg_len_hists = np.zeros((all_tracks[-1].shape[1],nb_states)) 384 | for k in range(len(all_tracks)): 385 | print('.', end='') 386 | if k == len(all_tracks)-1: 387 | isBL = 1 # last position correspond to tracks which didn't disapear within maximum track length 388 | else: 389 | isBL = 0 390 | 391 | Css = all_tracks[k] 392 | if len(Css) > 0: 393 | nb_max = 50 394 | for n in range(int(np.ceil(len(Css)/nb_max))): 395 | Csss = Css[n*nb_max:(n+1)*nb_max] 396 | LP, cur_Bs, seg_len_hist = P_segment_len(Csss, LocErr, ds, Fs, TrMat, min_l = min_l, pBL=pBL, isBL = isBL, cell_dims = cell_dims, nb_substeps=nb_substeps, max_nb_states = max_nb_states) 397 | isBL = 0 398 | seg_len_hists[:seg_len_hist.shape[0]] = seg_len_hists[:seg_len_hist.shape[0]] + seg_len_hist 399 | print('') 400 | return seg_len_hists 401 | ''' 402 | 403 | def ground_truth_hist(all_Bs, 404 | nb_states = 2, 405 | long_tracks = False, # return hist from long tracks only 406 | nb_steps_lim = 20): # if long_tracks = True minimum track length considered 407 | 408 | if long_tracks: 409 | for i, l in enumerate(list(all_Bs.keys())): 410 | if int(l) < nb_steps_lim: 411 | del all_Bs[l] 412 | seg_len_hists = np.zeros((np.max(np.array(list(all_Bs.keys())).astype(int)),nb_states)) 413 | 414 | for i, l in enumerate(all_Bs): 415 | cur_Bs = all_Bs[l][:,None] 416 | if len(cur_Bs)>0: 417 | if cur_Bs.shape[-1] == 1 : 418 | nb_Tracks = cur_Bs.shape[0] 419 | nb_locs = cur_Bs.shape[2] 420 | cur_nb_Bs = len(cur_Bs[0]) 421 | #seg_len_hists[0] = seg_len_hists[0] + np.sum(cur_Bs[:,0] == np.arange(nb_states)[None],0) 422 | seg_lens = np.zeros((nb_Tracks, cur_nb_Bs, nb_locs+1, nb_states)) # dims : track ID, sequence of states ID, position in the sequence, state of the sequence 423 | seg_lens[:,:,1,:] = (cur_Bs[:,:,0,None] == np.arange(nb_states)[None,None]).astype(float) 424 | if 1: 425 | nb_Tracks = cur_Bs.shape[0] 426 | nb_locs = cur_Bs.shape[2] 427 | cur_nb_Bs = len(cur_Bs[0]) 428 | cur_pos = cur_Bs[:,:,0] 429 | cur_seg_len = np.ones((nb_Tracks, cur_nb_Bs)) 430 | seg_lens = np.zeros((nb_Tracks, cur_nb_Bs, nb_locs+1, nb_states)) # dims : track ID, sequence of states ID, position in the sequence, state of the sequence 431 | for k in range(1, nb_locs): 432 | is_Tr = cur_pos != cur_Bs[:,:,k] 433 | cur_seg_len = cur_seg_len + (is_Tr==0).astype(int) 434 | len_before_tr = (cur_seg_len * is_Tr.astype(int))[:,:,None] # cur_seg_len * is_Tr.astype(int)) selects the segments that stops so we can add them, if 0 : not transition if n : transition after n consecutive steps 435 | cat_states = (cur_pos[:,:,None] == np.arange(nb_states)[None,None]).astype(int) # position of the segment in categorcal format so it can fit seg_lens 436 | seg_lens[:,:,k-1] = len_before_tr * cat_states # add the segment len of the corresponding states 437 | cur_seg_len[is_Tr] = 1 438 | cur_pos = cur_Bs[:,:,k] 439 | 440 | last_len = (nb_locs - np.sum(seg_lens, (2,3)))[:,:,None] 441 | cat_states = (cur_pos[:,:,None] == np.arange(nb_states)[None,None]).astype(int) 442 | seg_lens[:,:,-1] = last_len * cat_states 443 | seg_len_hist = np.zeros((nb_locs, nb_states)) 444 | 445 | for k in range(1, nb_locs+1): 446 | #P_seg = (P[:,:,None,None] * np.exp(-LL)[:,:,None,None] * (seg_lens == k).astype(int)) 447 | P_seg = (1 * (seg_lens == k).astype(int)) 448 | #np.sum(P_seg*np.exp(-LL)[:,:,None,None],(1,2,3))[:,None]==0 449 | #P_seg = np.sum(P_seg, (1,2)) / (np.sum(P_seg,(1,2,3))[:,None]+1e-300) # normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 450 | P_seg = np.sum(P_seg, (0,1,2)) # normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 451 | #P_seg = np.sum(P_seg, (1,2)) / (np.sum(P_seg,(1,2)) +1e-300) # normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 452 | #P_seg = np.sum(P_seg, (1,2)) # normalize with regard to the proba of the track without the proba of leaving the FOV (to sum up over different track lengths) 453 | seg_len_hist[k-1] = P_seg 454 | 455 | seg_len_hists[:seg_len_hist.shape[0]] = seg_len_hists[:seg_len_hist.shape[0]] + seg_len_hist 456 | 457 | return seg_len_hists 458 | -------------------------------------------------------------------------------- /extrack/old_tracking.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Thu Jun 23 18:31:26 2022 4 | 5 | @author: Franc 6 | """ 7 | 8 | import numpy as np 9 | 10 | GPU_computing = False 11 | 12 | if GPU_computing : 13 | import cupy as cp 14 | from cupy import asnumpy 15 | else : 16 | import numpy as cp 17 | def asnumpy(x): 18 | return np.array(x) 19 | 20 | import itertools 21 | import scipy 22 | from lmfit import minimize, Parameters 23 | 24 | import multiprocessing 25 | from itertools import product 26 | 27 | ''' 28 | Maximum likelihood to determine transition rates : 29 | We compute the probability of observing the tracks knowing the parameters : 30 | For this, we assum a given set of consecutive states ruling the displacement terms, 31 | we express the probability of having a given set of tracks and real positions. 32 | Next, we integrate reccursively over the possible real positions to find the probability 33 | of the track knowing the consecutive states. This recurrance can be performed as the 34 | integral of the product of normal laws is a constant time a normal law. 35 | In the end of the reccurance process we have a constant time normal law of which 36 | all the terms are known and then a value of the probability. 37 | We finally use the conditional probability principle over the possible set of states 38 | to compute the probability of having the tracks. 39 | ''' 40 | 41 | def ds_froms_states(ds, cur_states): 42 | cur_ds = ds[cur_states] 43 | cur_ds = (cur_ds[:,:,1:]**2 + cur_ds[:,:,:-1]**2)**0.5 / 2**0.5 # assuming a transition at the middle of the substeps 44 | # we can average the variances of displacements per step to get the actual std of displacements per step 45 | cur_ds = cp.mean(cur_ds**2, axis = 2)**0.5 46 | cur_ds = cur_ds[:,:,None] 47 | cur_ds = cp.array(cur_ds) 48 | return cur_ds 49 | 50 | def log_integrale_dif(Ci, l, cur_ds, Km, Ks): 51 | ''' 52 | integral of the 3 exponetional terms (localization error, diffusion, previous term) 53 | the integral over r1 of f_l(r1-c1)f_d(r1-r0)f_Ks(r1-Km) equals : 54 | np.exp(-((l**2+Ks**2)*r0**2+(-2*Km*l**2-2*Ks**2*c1)*r0+Km**2*l**2+(Km**2-2*c1*Km+c1**2)*d**2+Ks**2*c1**2)/((2*d**2+2*Ks**2)*l**2+2*Ks**2*d**2))/(2*np.pi*Ks*d*l*np.sqrt((d**2+Ks**2)*l**2+Ks**2*d**2)) 55 | which can be turned into the form Constant*fKs(r0 - newKm) where fKs is a normal law of std newKs 56 | the idea is to create a function of integral of integral of integral etc 57 | dim 0 : tracks 58 | dim 1 : possible sequences of states 59 | dim 2 : x,y (z) 60 | ''' 61 | newKm = (Km*l**2 + Ci*Ks**2)/(l**2+Ks**2) 62 | newKs = ((cur_ds**2*l**2 + cur_ds**2*Ks**2 + l**2*Ks**2)/(l**2 + Ks**2))**0.5 63 | logConstant = Km.shape[2]*cp.log(1/((l**2 + Ks[:,:,0]**2)**0.5*np.sqrt(2*np.pi))) + cp.sum((newKm**2/(2*newKs**2) - (Km**2*l**2 + Ks**2*Ci**2 + (Km-Ci)**2*cur_ds**2)/(2*newKs**2*(l**2 + Ks**2))), axis = 2) 64 | return newKm, newKs, logConstant 65 | 66 | def first_log_integrale_dif(Ci, LocErr, cur_ds): 67 | ''' 68 | convolution of 2 normal laws = normal law (mean = sum of means and variance = sum of variances) 69 | ''' 70 | Ks = (LocErr**2+cur_ds**2)**0.5 71 | Km = Ci 72 | return Km, Ks 73 | 74 | def P_Cs_inter_bound_stats(Cs, LocErr, ds, Fs, TrMat, pBL=0.1, isBL = 1, cell_dims = [0.5], nb_substeps=1, frame_len = 6, do_preds = 0, min_len = 3) : 75 | ''' 76 | compute the product of the integrals over Ri as previousily described 77 | work in log space to avoid overflow and underflow 78 | 79 | Cs : dim 0 = track ID, dim 1 : states, dim 2 : peaks postions through time, 80 | dim 3 : x, y position 81 | 82 | we process by steps, at each step we account for 1 more localization, we compute 83 | the canstant (LC), the mean (Km) and std (Ks) of of the normal distribution 84 | resulting from integration. 85 | 86 | each step is made of substeps if nb_substeps > 1, and we increase the matrix 87 | of possible Bs : cur_Bs accordingly 88 | 89 | to be able to process long tracks with good accuracy, for each track we fuse Km and Ks 90 | of sequences of states equal exept for the state 'frame_len' steps ago. 91 | ''' 92 | nb_Tracks = Cs.shape[0] 93 | nb_locs = Cs.shape[1] # number of localization per track 94 | nb_dims = Cs.shape[2] # number of spatial dimentions (x, y) or (x, y, z) 95 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 96 | Cs = cp.array(Cs) 97 | nb_states = TrMat.shape[0] 98 | Cs = Cs[:,:,::-1] 99 | 100 | if do_preds: 101 | preds = np.zeros((nb_Tracks, nb_locs, nb_states))-1 102 | else : 103 | preds = [] 104 | 105 | if nb_locs < 2: 106 | raise ValueError('minimal track length = 2, here track length = %s'%nb_locs) 107 | 108 | all_Bs = get_all_Bs(np.min([(nb_locs-1)*nb_substeps+1, frame_len + nb_substeps - 1]), nb_states)[None] 109 | 110 | TrMat = cp.array(TrMat.T) 111 | current_step = 1 112 | 113 | #cur_Bs = get_all_Bs(nb_substeps + 1, nb_states)[None] # get initial sequences of states 114 | cur_Bs = all_Bs[:,:nb_states**(nb_substeps + 1),:nb_substeps + 1] 115 | 116 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) #states of interest for the current displacement 117 | cur_nb_Bs = cur_Bs.shape[1] 118 | # compute the vector of diffusion stds knowing the current states 119 | ds = cp.array(ds) 120 | Fs = cp.array(Fs) 121 | 122 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 123 | LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 124 | 125 | LP = LT + LF #+ compensate_leaving 126 | # current log proba of seeing the track 127 | LP = cp.repeat(LP, nb_Tracks, axis = 0) 128 | cur_ds = ds[cur_states] 129 | cur_ds = (cur_ds[:,:,1:]**2 + cur_ds[:,:,:-1]**2)**0.5 / 2**0.5 # assuming a transition at the middle of the substeps 130 | # we can average the variances of displacements per step to get the actual std of displacements per step 131 | cur_ds = cp.mean(cur_ds**2, axis = 2)**0.5 132 | cur_ds = cur_ds[:,:,None] 133 | cur_ds = cp.array(cur_ds) 134 | 135 | sub_Bs = cur_Bs.copy()[:,:cur_Bs.shape[1]//nb_states,:nb_substeps] # list of possible current states we can meet to compute the proba of staying in the FOV 136 | sub_ds = cp.mean(ds[sub_Bs]**2, axis = 2)**0.5 # corresponding list of d 137 | sub_ds = asnumpy(sub_ds) 138 | 139 | p_stay = np.ones(sub_ds.shape[-1]) 140 | for cell_len in cell_dims: 141 | xs = np.linspace(0+cell_len/2000,cell_len-cell_len/2000,1000) 142 | cur_p_stay = ((cp.mean(scipy.stats.norm.cdf((cell_len-xs[:,None])/(sub_ds+1e-200)) - scipy.stats.norm.cdf(-xs[:,None]/(sub_ds+1e-200)),0))) # proba to stay in the FOV for each of the possible cur Bs 143 | p_stay = p_stay*cur_p_stay 144 | p_stay = cp.array(p_stay) 145 | Lp_stay = cp.log(p_stay * (1-pBL)) # proba for the track to survive = both stay in the FOV and not bleach 146 | 147 | # inject the first position to get the associated Km and Ks : 148 | Km, Ks = first_log_integrale_dif(Cs[:,:, nb_locs-current_step], LocErr, cur_ds) 149 | current_step += 1 150 | Km = cp.repeat(Km, cur_nb_Bs, axis = 1) 151 | removed_steps = 0 152 | #TrMat = np.array([[0.9,0.1],[0.2,0.8]]) 153 | while current_step <= nb_locs-1: 154 | # update cur_Bs to describe the states at the next step : 155 | #cur_Bs = get_all_Bs(current_step*nb_substeps+1 - removed_steps, nb_states)[None] 156 | cur_Bs = all_Bs[:,:nb_states**(current_step*nb_substeps+1 - removed_steps),:current_step*nb_substeps+1 - removed_steps] 157 | 158 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) 159 | # compute the vector of diffusion stds knowing the states at the current step 160 | cur_ds = ds[cur_states] 161 | cur_ds = (cur_ds[:,:,1:]**2 + cur_ds[:,:,:-1]**2)**0.5 / 2**0.5 # assuming a transition at the middle of the substeps 162 | cur_ds = cp.mean(cur_ds**2, axis = 2)**0.5 163 | cur_ds = cur_ds[:,:,None] 164 | LT = get_Ts_from_Bs(cur_states, TrMat) 165 | 166 | #np.arange(32)[None][:,TT] 167 | #np.arange(32)[None][:,1:][:,TT[:-1]] 168 | # repeat the previous matrix to account for the states variations due to the new position 169 | Km = cp.repeat(Km, nb_states**nb_substeps , axis = 1) 170 | Ks = cp.repeat(Ks, nb_states**nb_substeps, axis = 1) 171 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1) 172 | # inject the next position to get the associated Km, Ks and Constant describing the integral of 3 normal laws : 173 | Km, Ks, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr, cur_ds, Km, Ks) 174 | #print('integral',time.time() - t0); t0 = time.time() 175 | if current_step >= min_len : 176 | LL = Lp_stay[np.argmax(np.all(cur_states[:,None,:,:-1] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 177 | else: 178 | LL = 0 179 | 180 | LP += LT + LC + LL # current (log) constants associated with each track and sequences of states 181 | del LT, LC 182 | cur_nb_Bs = len(cur_Bs[0]) # current number of sequences of states 183 | 184 | ''''idea : the position and the state 6 steps ago should not impact too much the 185 | probability of the next position so the Km and Ks of tracks with the same 6 last 186 | states must be very similar, we can then fuse the parameters of the pairs of Bs 187 | which vary only for the last step (7) and sum their probas''' 188 | if current_step < nb_locs-1: 189 | while cur_nb_Bs >= nb_states**frame_len: 190 | if do_preds : 191 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 192 | log_integrated_term = -cp.log(2*np.pi*newKs**2) - cp.sum((Cs[:,:,nb_locs-current_step] - Km)**2,axis=2)/(2*newKs**2) 193 | LF = 0 #cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 194 | 195 | test_LP = LP + log_integrated_term + LF 196 | 197 | if np.max(test_LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 198 | test_LP = test_LP - (np.max(test_LP)-600) 199 | 200 | P = np.exp(test_LP) 201 | 202 | for state in range(nb_states): 203 | B_is_state = cur_Bs[:,:,-1] == state 204 | preds[:,nb_locs-current_step+frame_len-2, state] = asnumpy(np.sum(B_is_state*P,axis = 1)/np.sum(P,axis = 1)) 205 | cur_Bs = cur_Bs[:,:cur_nb_Bs//nb_states, :-1] 206 | Km, Ks, LP = fuse_tracks(Km, Ks, LP, cur_nb_Bs, nb_states) 207 | cur_nb_Bs = len(cur_Bs[0]) 208 | removed_steps += 1 209 | #print('frame',time.time() - t0) 210 | #print(current_step,time.time() - t0) 211 | current_step += 1 212 | 213 | all_Bs.shape 214 | cur_Bs.shape 215 | 216 | if not isBL: 217 | LL = 0 218 | else: 219 | cur_Bs = get_all_Bs(np.round(np.log(cur_nb_Bs)/np.log(nb_states)+nb_substeps).astype(int), nb_states)[None] 220 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) 221 | len(cur_Bs[0]) 222 | LT = get_Ts_from_Bs(cur_states, TrMat) 223 | #cur_states = cur_states[:,:,0] 224 | # repeat the previous matrix to account for the states variations due to the new position 225 | Km = cp.repeat(Km, nb_states**nb_substeps , axis = 1) 226 | Ks = cp.repeat(Ks, nb_states**nb_substeps, axis = 1) 227 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1) 228 | 229 | #LL = Lp_stay[np.argmax(np.all(cur_states[:,None] == sub_Bs[:,:,None],-1),1)] # pick the right proba of staying according to the current states 230 | #end_p_stay = p_stay[np.argmax(np.all(cur_states[:,None:,:-1] == sub_Bs[:,:,None],-1),1)] 231 | end_p_stay = p_stay[cur_states[:,None:,:-1]][:,:,0] 232 | end_p_stay.shape 233 | LL = cp.log(pBL + (1-end_p_stay) - pBL * (1-end_p_stay)) + LT 234 | 235 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 236 | log_integrated_term = -cp.log(2*np.pi*newKs**2) - cp.sum((Cs[:,:,0] - Km)**2,axis=2)/(2*newKs**2) 237 | #LF = cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 238 | #LF = cp.log(0.5) 239 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 240 | LP += log_integrated_term + LL 241 | 242 | pred_LP = LP 243 | if np.max(LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 244 | pred_LP = LP - (np.max(LP)-600) 245 | 246 | P = np.exp(pred_LP) 247 | if do_preds : 248 | for state in range(nb_states): 249 | B_is_state = cur_Bs[:,:] == state 250 | preds[:,0:frame_len, state] = asnumpy(np.sum(B_is_state[:,:,isBL:]*P[:,:,None],axis = 1)/np.sum(P[:,:,None],axis = 1)) # index isBL is here to remove the additional position infered to take leaving the FOV into account when isBL (when the track stops) 251 | preds = preds[:,::-1] 252 | return LP, cur_Bs, preds 253 | 254 | def fuse_tracks(Km, Ks, LP, cur_nb_Bs, nb_states = 2): 255 | ''' 256 | The probabilities of the pairs of tracks must be added 257 | I chose to define the updated Km and Ks as the weighted average (of the variance for Ks) 258 | but other methods may be better 259 | As I must divid by a sum of exponentials which can be equal to zero because of underflow 260 | I correct the values in the exponetial to keep the maximal exp value at 0 261 | ''' 262 | # cut the matrixes so the resulting matrices only vary for their last state 263 | I = cur_nb_Bs//nb_states 264 | LPk = [] 265 | Kmk = [] 266 | Ksk = [] 267 | for k in range(nb_states): 268 | LPk.append(LP[:, k*I:(k+1)*I])# LP of which the last state is k 269 | Kmk.append(Km[:, k*I:(k+1)*I])# Km of which the last state is k 270 | Ksk.append(Ks[:, k*I:(k+1)*I])# Ks of which the last state is k 271 | 272 | LPk = cp.array(LPk) 273 | Kmk = cp.array(Kmk) 274 | Ksk = cp.array(Ksk) 275 | 276 | maxLP = cp.max(LPk, axis = 0, keepdims = True) 277 | Pk = cp.exp(LPk - maxLP) 278 | 279 | #sum of the probas of the 2 corresponding matrices : 280 | SP = cp.sum(Pk, axis = 0, keepdims = True) 281 | ak = Pk/SP 282 | 283 | # update the parameters, this step is tricky as an approximation of a gaussian mixture by a simple gaussian 284 | Km = cp.sum(ak[:,:,:,None] * Kmk, axis=0) 285 | Ks = cp.sum((ak[:,:,:,None] * Ksk**2), axis=0)**0.5 286 | del ak 287 | LP = maxLP + np.log(SP) 288 | LP = LP[0] 289 | # cur_Bs = cur_Bs[:,:I, :-1] 290 | # np.mean(np.abs(Km0-Km1)) # to verify how far they are, I found a difference of 0.2nm for D = 0.1um2/s, LocErr=0.02um and 6 frames 291 | # np.mean(np.abs(Ks0-Ks1)) 292 | return Km, Ks, LP 293 | 294 | def get_all_Bs(nb_Cs, nb_states): 295 | ''' 296 | produces a matrix of the possible sequences of states 297 | ''' 298 | Bs_ID = np.arange(nb_states**nb_Cs) 299 | all_Bs = np.zeros((nb_states**nb_Cs, nb_Cs), int) 300 | 301 | for k in range(all_Bs.shape[1]): 302 | cur_row = np.mod(Bs_ID,nb_states**(k+1)) 303 | Bs_ID = (Bs_ID - cur_row) 304 | all_Bs[:,k] = cur_row//nb_states**k 305 | return all_Bs 306 | 307 | def get_Ts_from_Bs(all_Bs, TrMat): 308 | ''' 309 | compute the probability of the sequences of states according to the markov transition model 310 | ''' 311 | LT = cp.zeros((all_Bs.shape[:2])) 312 | # change from binary base 10 numbers to identify the consecutive states (from ternary if 3 states) 313 | for k in range(len(all_Bs[0,0])-1): 314 | LT += cp.log(TrMat[all_Bs[:,:,k], all_Bs[:,:,k+1]]) 315 | return LT 316 | 317 | def Proba_Cs(Cs, LocErr, ds, Fs, TrMat,pBL,isBL, cell_dims, nb_substeps, frame_len, min_len): 318 | ''' 319 | inputs the observed localizations and determine the probability of 320 | observing these data knowing the localization error, D the diffusion coef, 321 | pu the proba of unbinding per step and pb the proba of binding per step 322 | sum the proba of Cs inter Bs (calculated with P_Cs_inter_bound_stats) 323 | over all Bs to get the proba of Cs (knowing the initial position c0) 324 | ''' 325 | 326 | LP_CB, _, _ = P_Cs_inter_bound_stats(Cs, LocErr, ds, Fs, TrMat, pBL,isBL,cell_dims, nb_substeps, frame_len, do_preds = 0, min_len = min_len) 327 | # calculates P(C) the sum of P(C inter B) for each track 328 | max_LP = np.max(LP_CB, axis = 1, keepdims = True) 329 | LP_CB = LP_CB - max_LP 330 | max_LP = max_LP[:,0] 331 | P_CB = np.exp(LP_CB) 332 | P_C = cp.sum(P_CB, axis = 1) # sum over B 333 | LP_C = np.log(P_C) + max_LP # back to log proba of C without overflow due to exponential 334 | return LP_C 335 | 336 | def predict_Bs(all_tracks, 337 | dt, 338 | params, 339 | cell_dims=[1], 340 | nb_states=2, 341 | frame_len=12): 342 | ''' 343 | inputs the observed localizations and parameters and determines the proba 344 | of each localization to be in a given state. 345 | 346 | arguments: 347 | all_tracks: dict describing the tracks with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = x, y position. 348 | params: lmfit parameters used for the model. 349 | dt: time in between frames. 350 | cell_dims: dimension limits (um). estimated_vals, min_values, max_values should be changed accordingly to describe all states and transitions. 351 | nb_states: number of states. estimated_vals, min_values, max_values should be changed accordingly to describe all states and transitions. 352 | frame_len: number of frames for which the probability is perfectly computed. See method of the paper for more details. 353 | 354 | outputs: 355 | pred_Bs: dict describing the state probability of each track for each time position with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = state. 356 | extrack.visualization.visualize_states_durations 357 | ''' 358 | nb_substeps=1 # substeps should not impact the step labelling 359 | LocErr, ds, Fs, TrMat, pBL = extract_params(params, dt, nb_states, nb_substeps) 360 | all_pred_Bs = [] 361 | 362 | l_list = np.sort(np.array(list(all_tracks.keys())).astype(int)).astype(str) 363 | min_len = int(l_list[0]) 364 | max_len = int(l_list[-1]) 365 | 366 | for l in l_list: 367 | if len(all_tracks[l]) > 0: 368 | if int(l) == max_len: 369 | isBL = 1 370 | else: 371 | isBL = 0 372 | LP_Cs, trunkated_Bs, pred_Bs = P_Cs_inter_bound_stats(all_tracks[l], LocErr, ds, Fs, TrMat, pBL,isBL, cell_dims, nb_substeps = 1, frame_len = frame_len, do_preds = 1, min_len = min_len) 373 | all_pred_Bs.append(pred_Bs) 374 | 375 | all_pred_Bs_dict = {} 376 | for pred_Bs in all_pred_Bs: 377 | all_pred_Bs_dict[str(pred_Bs.shape[1])] = pred_Bs 378 | 379 | return all_pred_Bs_dict 380 | 381 | def extract_params(params, dt, nb_states, nb_substeps): 382 | ''' 383 | turn the parameters which differ deppending on the number of states into lists 384 | ds (diffusion lengths), Fs (fractions), TrMat (substep transiton matrix) 385 | ''' 386 | LocErr = params['LocErr'].value 387 | 388 | param_names = np.sort(list(params.keys())) 389 | 390 | Ds = [] 391 | Fs = [] 392 | for param in param_names: 393 | if param.startswith('D') and len(param)<3: 394 | Ds.append(params[param].value) 395 | elif param.startswith('F'): 396 | Fs.append(params[param].value) 397 | Ds = np.array(Ds) 398 | Fs = np.array(Fs) 399 | TrMat = np.zeros((len(Ds),len(Ds))) 400 | for param in params: 401 | if param == 'pBL': 402 | pBL = params[param].value 403 | elif param.startswith('p'): 404 | i = int(param[1]) 405 | j = int(param[2]) 406 | TrMat[i,j] = params[param].value 407 | 408 | TrMat = TrMat/nb_substeps 409 | 410 | 411 | TrMat = 1 - np.exp(-TrMat) 412 | TrMat[np.arange(len(Ds)), np.arange(len(Ds))] = 1-np.sum(TrMat,1) 413 | 414 | #print(TrMat) 415 | ds = np.sqrt(2*Ds*dt) 416 | return LocErr, ds, Fs, TrMat, pBL 417 | 418 | def pool_star_proba(args): 419 | return Proba_Cs(*args) 420 | 421 | def cum_Proba_Cs(params, all_tracks, dt, cell_dims, nb_states, nb_substeps, frame_len, verbose = 1, workers = 1): 422 | ''' 423 | each probability can be multiplied to get a likelihood of the model knowing 424 | the parameters LocErr, D0 the diff coefficient of state 0 and F0 fraction of 425 | state 0, D1 the D coef at state 1, p01 the probability of transition from 426 | state 0 to 1 and p10 the proba of transition from state 1 to 0. 427 | here sum the logs(likelihood) to avoid too big numbers 428 | ''' 429 | LocErr, ds, Fs, TrMat, pBL = extract_params(params, dt, nb_states, nb_substeps) 430 | min_len = all_tracks[0].shape[1] 431 | 432 | if np.all(TrMat>0) and np.all(Fs>0): 433 | Cum_P = 0 434 | Csss = [] 435 | isBLs = [] 436 | for k in range(len(all_tracks)): 437 | if k == len(all_tracks)-1: 438 | isBL = 0 # last position correspond to tracks which didn't disapear within maximum track length 439 | else: 440 | isBL = 1 441 | Css = all_tracks[k] 442 | nb_max = 50 443 | for n in range(int(np.ceil(len(Css)/nb_max))): 444 | Csss.append(Css[n*nb_max:(n+1)*nb_max]) 445 | if k == len(all_tracks)-1: 446 | isBLs.append(0) # last position correspond to tracks which didn't disapear within maximum track length 447 | else: 448 | isBLs.append(1) 449 | Csss.reverse() 450 | args_prod = np.array(list(product(Csss, [LocErr], [ds], [Fs], [TrMat],[pBL], [0],[cell_dims], [nb_substeps], [frame_len], [min_len]))) 451 | args_prod[:, 6] = isBLs 452 | 453 | if workers >= 2: 454 | with multiprocessing.Pool(workers) as pool: 455 | LP = pool.map(pool_star_proba, args_prod) 456 | else: 457 | LP = [] 458 | for args in args_prod: 459 | LP.append(pool_star_proba(args)) 460 | 461 | Cum_P += cp.sum(cp.concatenate(LP)) 462 | Cum_P = asnumpy(Cum_P) 463 | 464 | if verbose == 1: 465 | q = [param + ' = ' + str(np.round(params[param].value, 4)) for param in params] 466 | print(Cum_P, q) 467 | else: 468 | print('.', end='') 469 | out = - Cum_P # normalize by the number of tracks and number of displacements 470 | else: 471 | out = 1e100 472 | print('x',end='') 473 | if out == np.nan: 474 | out = 1e100 475 | print('inputs give nans') 476 | return out 477 | 478 | def get_params(nb_states = 2, 479 | steady_state = False, 480 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'F0' : True, 'p01' : True, 'p10' : True, 'pBL' : True}, 481 | estimated_vals = {'LocErr' : 0.025, 'D0' : 1e-20, 'D1' : 0.05, 'F0' : 0.45, 'p01' : 0.05, 'p10' : 0.05, 'pBL' : 0.1}, 482 | min_values = {'LocErr' : 0.007, 'D0' : 1e-12, 'D1' : 0.00001, 'F0' : 0.001, 'p01' : 0.01, 'p10' : 0.01, 'pBL' : 0.01}, 483 | max_values = {'LocErr' : 0.6, 'D0' : 1, 'D1' : 10, 'F0' : 0.999, 'p01' : 1., 'p10' : 1., 'pBL' : 0.99}): 484 | 485 | if nb_states == 2: 486 | if steady_state: 487 | print(estimated_vals) 488 | param_kwargs = [{'name' : 'D0', 'value' : estimated_vals['D0'], 'min' : min_values['D0'], 'max' : max_values['D0'], 'vary' : vary_params['D0']}, 489 | {'name' : 'D1_minus_D0', 'value' : estimated_vals['D1'] - estimated_vals['D0'], 'min' : min_values['D1']-min_values['D0'], 'max' : max_values['D1'], 'vary' : vary_params['D1']}, 490 | {'name' : 'D1', 'expr' : 'D0 + D1_minus_D0'}, 491 | {'name' : 'LocErr', 'value' : estimated_vals['LocErr'], 'min' : min_values['LocErr'],'max' : max_values['LocErr'], 'vary' : vary_params['LocErr']}, 492 | {'name' : 'F0', 'value' : estimated_vals['F0'], 'min' : min_values['F0'], 'max' : max_values['F0'], 'vary' : vary_params['F0']}, 493 | {'name' : 'F1', 'expr' : '1 - F0'}, 494 | {'name' : 'p01', 'value' : estimated_vals['p01'], 'min' : min_values['p01'], 'max' : max_values['p01'], 'vary' : vary_params['p01']}, 495 | {'name' : 'p10', 'expr' : 'p01/(1/F0-1)'}, 496 | {'name' : 'pBL', 'value' : estimated_vals['pBL'], 'min' : min_values['pBL'], 'max' : max_values['pBL'], 'vary' : vary_params['pBL']}] 497 | else : 498 | param_kwargs = [{'name' : 'D0', 'value' : estimated_vals['D0'], 'min' : min_values['D0'], 'max' : max_values['D0'], 'vary' : vary_params['D0']}, 499 | {'name' : 'D1_minus_D0', 'value' : estimated_vals['D1'] - estimated_vals['D0'], 'min' : min_values['D1']-min_values['D0'], 'max' : max_values['D1'], 'vary' : vary_params['D1']}, 500 | {'name' : 'D1', 'expr' : 'D0 + D1_minus_D0' }, 501 | {'name' : 'LocErr', 'value' : estimated_vals['LocErr'], 'min' : min_values['LocErr'],'max' : max_values['LocErr'], 'vary' : vary_params['LocErr']}, 502 | {'name' : 'F0', 'value' : estimated_vals['F0'], 'min' : min_values['F0'], 'max' : max_values['F0'], 'vary' : vary_params['F0']}, 503 | {'name' : 'F1', 'expr' : '1 - F0'}, 504 | {'name' : 'p01', 'value' : estimated_vals['p01'], 'min' : min_values['p01'], 'max' : max_values['p01'], 'vary' : vary_params['p01']}, 505 | {'name' : 'p10', 'value' : estimated_vals['p10'], 'min' : min_values['p10'], 'max' : max_values['p10'], 'vary' : vary_params['p10']}, 506 | {'name' : 'pBL', 'value' : estimated_vals['pBL'], 'min' : min_values['pBL'], 'max' : max_values['pBL'], 'vary' : vary_params['pBL']}] 507 | 508 | elif nb_states == 3: 509 | if not (len(min_values) == 13 and len(max_values) == 13 and len(estimated_vals) == 13 and len(vary_params) == 13): 510 | raise ValueError('estimated_vals, min_values, max_values and vary_params should all containing 13 parameters for a 3 states model') 511 | 512 | if steady_state: 513 | param_kwargs = [{'name' : 'LocErr', 'value' : estimated_vals['LocErr'], 'min' : min_values['LocErr'], 'max' : max_values['LocErr'] , 'vary' : vary_params['LocErr']}, 514 | {'name' : 'D0', 'value' : estimated_vals['D0'], 'min' : min_values['D0'], 'max' : 0.3, 'brute_step' : 0.04, 'vary' : vary_params['D0']}, 515 | {'name' : 'D1_minus_D0', 'value' : estimated_vals['D1'] - estimated_vals['D0'], 'min' : 0, 'max' : max_values['D1'], 'brute_step' : 0.04, 'vary' : vary_params['D1']}, 516 | {'name' : 'D1', 'expr' : 'D0+D1_minus_D0'}, 517 | {'name' : 'D2_minus_D1', 'value' : estimated_vals['D2'] - estimated_vals['D1'], 'min' : 0, 'max' : max_values['D2'], 'vary' : vary_params['D2']}, 518 | {'name' : 'D2', 'expr' : 'D1+D2_minus_D1'}, 519 | {'name' : 'p01', 'value' : estimated_vals['p01'], 'min' : min_values['p01'], 'max' : max_values['p01'], 'vary' : vary_params['p01']}, 520 | {'name' : 'p02', 'value' : estimated_vals['p02'], 'min' : min_values['p02'], 'max' : max_values['p02'], 'vary' : vary_params['p02']}, 521 | {'name' : 'p10', 'value' : estimated_vals['p10'], 'min' : min_values['p10'], 'max' : max_values['p10'], 'vary' : vary_params['p10']}, 522 | {'name' : 'p12', 'value' : estimated_vals['p12'], 'min' : min_values['p12'], 'max' : max_values['p12'], 'vary' : vary_params['p12']}, 523 | {'name' : 'p20', 'value' : estimated_vals['p20'], 'min' : min_values['p20'], 'max' : max_values['p20'], 'vary' : vary_params['p20']}, 524 | {'name' : 'p21', 'value' : estimated_vals['p21'], 'min' : min_values['p21'], 'max' : max_values['p21'], 'vary' : vary_params['p21']}, 525 | {'name' : 'F0', 'expr' : '(p10*(p21+p20)+p20*p12)/((p01)*(p12 + p21) + p02*(p10 + p12 + p21) + p01*p20 + p21*p10 + p20*(p10+p12))'}, 526 | {'name' : 'F1', 'expr' : '(F0*p01 + (1-F0)*p21)/(p10 + p12 + p21)'}, 527 | {'name' : 'F2', 'expr' : '1-F0-F1'}, 528 | {'name' : 'pBL', 'value' : estimated_vals['pBL'], 'min' : min_values['pBL'], 'max' : max_values['pBL'], 'vary' : vary_params['pBL']}] 529 | else: 530 | param_kwargs = [{'name' : 'LocErr', 'value' : estimated_vals['LocErr'], 'min' : min_values['LocErr'], 'max' : max_values['LocErr'] , 'vary' : vary_params['LocErr']}, 531 | {'name' : 'D0', 'value' : estimated_vals['D0'], 'min' : min_values['D0'], 'max' : 0.3, 'brute_step' : 0.04, 'vary' : vary_params['D0']}, 532 | {'name' : 'D1_minus_D0', 'value' : estimated_vals['D1'] - estimated_vals['D0'], 'min' : 0, 'max' : max_values['D1'], 'brute_step' : 0.04, 'vary' : vary_params['D1']}, 533 | {'name' : 'D1', 'expr' : 'D0+D1_minus_D0'}, 534 | {'name' : 'D2_minus_D1', 'value' : estimated_vals['D2'] - estimated_vals['D1'], 'min' : 0, 'max' : max_values['D2'], 'vary' : vary_params['D2']}, 535 | {'name' : 'D2', 'expr' : 'D1+D2_minus_D1'}, 536 | {'name' : 'p01', 'value' : estimated_vals['p01'], 'min' : min_values['p01'], 'max' : max_values['p01'], 'vary' : vary_params['p01']}, 537 | {'name' : 'p02', 'value' : estimated_vals['p02'], 'min' : min_values['p02'], 'max' : max_values['p02'], 'vary' : vary_params['p02']}, 538 | {'name' : 'p10', 'value' : estimated_vals['p10'], 'min' : min_values['p10'], 'max' : max_values['p10'], 'vary' : vary_params['p10']}, 539 | {'name' : 'p12', 'value' : estimated_vals['p12'], 'min' : min_values['p12'], 'max' : max_values['p12'], 'vary' : vary_params['p12']}, 540 | {'name' : 'p20', 'value' : estimated_vals['p20'], 'min' : min_values['p20'], 'max' : max_values['p20'], 'vary' : vary_params['p20']}, 541 | {'name' : 'p21', 'value' : estimated_vals['p21'], 'min' : min_values['p21'], 'max' : max_values['p21'], 'vary' : vary_params['p21']}, 542 | {'name' : 'F0', 'value' : estimated_vals['F0'], 'min' : min_values['F0'], 'max' : max_values['F0'], 'vary' : vary_params['F0']}, 543 | {'name' : 'F1_minus_F0', 'value' : (estimated_vals['F1'])/(1-estimated_vals['F0']), 'min' : min_values['F1'], 'max' : max_values['F1'], 'vary' : vary_params['F1']}, 544 | {'name' : 'F1', 'expr' : 'F1_minus_F0*(1-F0)'}, 545 | {'name' : 'F2', 'expr' : '1-F0-F1'}, 546 | {'name' : 'pBL', 'value' : estimated_vals['pBL'], 'min' : min_values['pBL'], 'max' : max_values['pBL'], 'vary' : vary_params['pBL']}] 547 | else : 548 | param_kwargs = [{'name' : 'LocErr', 'value' : estimated_vals['LocErr'], 'min' : min_values['LocErr'], 'max' : max_values['LocErr'] , 'vary' : vary_params['LocErr']}] 549 | Ds = [] 550 | Fs = [] 551 | for param in list(vary_params.keys()): 552 | if param.startswith('D'): 553 | Ds.append(param) 554 | if param.startswith('F'): 555 | Fs.append(param) 556 | param_kwargs.append({'name' : 'D0', 'value' : estimated_vals['D0'], 'min' : min_values['D0'], 'max' : 0.3, 'brute_step' : 0.04, 'vary' : vary_params['D0']}) 557 | last_D = 'D0' 558 | sum_Ds = estimated_vals['D0'] 559 | expr = 'D0' 560 | for D in Ds[1:]: 561 | param_kwargs.append({'name' : D + '_minus_' + last_D, 'value' : estimated_vals[D] - sum_Ds, 'min' : 0, 'max' : max_values[D] , 'vary' : vary_params[D]}) 562 | expr = expr + '+' + D + '_minus_' + last_D 563 | param_kwargs.append({'name' : D, 'expr' : expr}) 564 | last_D = D 565 | sum_Ds += estimated_vals[D] 566 | 567 | param_kwargs.append({'name' : 'F0', 'value' : estimated_vals['F0'], 'min' : min_values['F0'], 'max' : 0.3, 'brute_step' : 0.04, 'vary' : vary_params['F0']}) 568 | frac = 1-estimated_vals['F0'] 569 | expr = '1-F0' 570 | 571 | for F in Fs[1:len(Ds)-1]: 572 | param_kwargs.append({'name' : F , 'value' : estimated_vals[F], 'min' : 0.001, 'max' : 0.99 , 'vary' : vary_params[F]}) 573 | frac = frac - 1 574 | expr = expr + '-' + F 575 | param_kwargs.append({'name' : 'F'+str(len(Ds)-1), 'expr' : expr}) 576 | 577 | for param in list(vary_params.keys()): 578 | if param.startswith('p'): 579 | param_kwargs.append({'name' : param, 'value' : estimated_vals[param], 'min' : min_values[param], 'max' : max_values[param] , 'vary' : vary_params[param]}) 580 | 581 | params = Parameters() 582 | [params.add(**param_kwargs[k]) for k in range(len(param_kwargs))] 583 | return params 584 | 585 | def get_2DSPT_params(all_tracks, 586 | dt, 587 | nb_substeps = 1, 588 | nb_states = 2, 589 | frame_len = 8, 590 | verbose = 1, 591 | workers = 1, 592 | method = 'powell', 593 | steady_state = False, 594 | cell_dims = [1], # list of dimensions limit for the field of view (FOV) of the cell in um, a membrane protein in a typical e-coli cell in tirf would have a cell_dims = [0.5,3], in case of cytosolic protein one should imput the depth of the FOV e.g. [0.3] for tirf or [0.8] for hilo 595 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'F0' : True, 'p01' : True, 'p10' : True, 'pBL' : True}, 596 | estimated_vals = {'LocErr' : 0.025, 'D0' : 1e-20, 'D1' : 0.05, 'F0' : 0.45, 'p01' : 0.05, 'p10' : 0.05, 'pBL' : 0.1}, 597 | min_values = {'LocErr' : 0.007, 'D0' : 1e-12, 'D1' : 0.00001, 'F0' : 0.001, 'p01' : 0.001, 'p10' : 0.001, 'pBL' : 0.001}, 598 | max_values = {'LocErr' : 0.6, 'D0' : 1, 'D1' : 10, 'F0' : 0.999, 'p01' : 1., 'p10' : 1., 'pBL' : 0.99}): 599 | ''' 600 | fitting the parameters to the data set 601 | 602 | arguments: 603 | all_tracks: dict describing the tracks with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = x, y position. 604 | dt: time in between frames. 605 | cell_dims: dimension limits (um). 606 | nb_substeps: number of virtual transition steps in between consecutive 2 positions. 607 | nb_states: number of states. estimated_vals, min_values, max_values should be changed accordingly to describe all states and transitions. 608 | frame_len: number of frames for which the probability is perfectly computed. See method of the paper for more details. 609 | verbose: if 1, print the intermediate values for each iteration of the fit. 610 | steady_state: True if tracks are considered at steady state (fractions independent of time), this is most likely not true as tracks join and leave the FOV. 611 | vary_params: dict specifying if each parameters should be changed (True) or not (False). 612 | estimated_vals: initial values of the fit. (stay constant if parameter fixed by vary_params). estimated_vals must be in between min_values and max_values even if fixed. 613 | min_values: minimal values for the fit. 614 | max_values: maximal values for the fit. 615 | 616 | outputs: 617 | model_fit: lmfit model 618 | 619 | in case of 3 states models vary_params, estimated_vals, min_values and max_values can be replaced : 620 | 621 | vary_params = {'LocErr' : True, 'D0' : False, 'D1' : True, 'D2' : True, 'F0' : True, 'F1' : True, 'p01' : True, 'p02' : True, 'p10' : True,'p12' : True,'p20' : True, 'p21' : True, 'pBL' : True}, 622 | estimated_vals = {'LocErr' : 0.023, 'D0' : 1e-20, 'D1' : 0.02, 'D2' : 0.1, 'F0' : 0.33, 'F1' : 0.33, 'p01' : 0.1, 'p02' : 0.1, 'p10' :0.1, 'p12' : 0.1, 'p20' :0.1, 'p21' :0.1, 'pBL' : 0.1}, 623 | min_values = {'LocErr' : 0.007, 'D0' : 1e-20, 'D1' : 0.0000001, 'D2' : 0.000001, 'F0' : 0.001, 'F1' : 0.001, 'p01' : 0.001, 'p02' : 0.001, 'p10' :0.001, 'p12' : 0.001, 'p20' :0.001, 'p21' :0.001, 'pBL' : 0.001}, 624 | max_values = {'LocErr' : 0.6, 'D0' : 1e-20, 'D1' : 1, 'D2' : 10, 'F0' : 0.999, 'F1' : 0.999, 'p01' : 1, 'p02' : 1, 'p10' : 1, 'p12' : 1, 'p20' : 1, 'p21' : 1, 'pBL' : 0.99} 625 | 626 | in case of 4 states models : 627 | 628 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'D2' : True, 'D3' : True, 'F0' : True, 'F1' : True, 'F2' : True, 'p01' : True, 'p02' : True, 'p03' : True, 'p10' : True, 'p12' : True, 'p13' : True, 'p20' :True, 'p21' :True, 'p23' : True, 'p30' :True, 'p31' :True, 'p32' : True, 'pBL' : True} 629 | estimated_vals = {'LocErr' : 0.023, 'D0' : 1e-20, 'D1' : 0.02, 'D2' : 0.1, 'D3' : 0.5, 'F0' : 0.1, 'F1' : 0.2, 'F2' : 0.3, 'p01' : 0.1, 'p02' : 0.1, 'p03' : 0.1, 'p10' :0.1, 'p12' : 0.1, 'p13' : 0.1, 'p20' :0.1, 'p21' :0.1, 'p23' : 0.1, 'p30' :0.1, 'p31' :0.1, 'p32' : 0.1, 'pBL' : 0.1} 630 | min_values = {'LocErr' : 0.005, 'D0' : 0, 'D1' : 0, 'D2' : 0.001, 'D3' : 0.001, 'F0' : 0.001, 'F1' : 0.001, 'F2' : 0.001, 'p01' : 0.001, 'p02' : 0.001, 'p03' : 0.001, 'p10' :0.001, 'p12' : 0.001, 'p13' : 0.001, 'p20' :0.001, 'p21' :0.001, 'p23' : 0.001, 'p30' :0.001, 'p31' :0.001, 'p32' : 0.001, 'pBL' : 0.001} 631 | max_values = {'LocErr' : 0.023, 'D0' : 1, 'D1' : 1, 'D2' : 10, 'D3' : 100, 'F0' : 0.999, 'F1' : 0.999, 'F2' : 0.999, 'p01' : 1, 'p02' : 1, 'p03' : 1, 'p10' :1, 'p12' : 1, 'p13' : 1, 'p20' : 1, 'p21' : 1, 'p23' : 1, 'p30' : 1, 'p31' : 1, 'p32' : 1, 'pBL' : 0.99} 632 | ''' 633 | if nb_states == 2: 634 | if not (len(min_values) == 7 and len(max_values) == 7 and len(estimated_vals) == 7 and len(vary_params) == 7): 635 | raise ValueError('estimated_vals, min_values, max_values and vary_params should all containing 7 parameters') 636 | elif nb_states == 3: 637 | if len(vary_params) != 13: 638 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'D2' : True, 'F0' : True, 'F1' : True, 'p01' : True, 'p02' : True, 'p10' : True,'p12' : True,'p20' : True, 'p21' : True, 'pBL' : True}, 639 | if len(estimated_vals) != 13: 640 | estimated_vals = {'LocErr' : 0.023, 'D0' : 1e-20, 'D1' : 0.02, 'D2' : 0.1, 'F0' : 0.33, 'F1' : 0.33, 'p01' : 0.1, 'p02' : 0.1, 'p10' :0.1, 'p12' : 0.1, 'p20' :0.1, 'p21' :0.1, 'pBL' : 0.1}, 641 | if len(min_values) != 13: 642 | min_values = {'LocErr' : 0.007, 'D0' : 1e-20, 'D1' : 0.0000001, 'D2' : 0.000001, 'F0' : 0.001, 'F1' : 0.001, 'p01' : 0.001, 'p02' : 0.001, 'p10' :0.001, 'p12' : 0.001, 'p20' :0.001, 'p21' :0.001, 'pBL' : 0.001}, 643 | if len(max_values) != 13: 644 | max_values = {'LocErr' : 0.023, 'D0' : 1, 'D1' : 1, 'D2' : 10, 'F0' : 0.999, 'F1' : 0.999, 'p01' : 1, 'p02' : 1, 'p10' :1, 'p12' : 1, 'p20' : 1, 'p21' : 1, 'pBL' : 0.99} 645 | elif nb_states == 4: 646 | if len(vary_params) != 21: 647 | vary_params = {'LocErr' : True, 'D0' : True, 'D1' : True, 'D2' : True, 'D3' : True, 'F0' : True, 'F1' : True, 'F2' : True, 'p01' : True, 'p02' : True, 'p03' : True, 'p10' : True, 'p12' : True, 'p13' : True, 'p20' :True, 'p21' :True, 'p23' : True, 'p30' :True, 'p31' :True, 'p32' : True, 'pBL' : True} 648 | if len(estimated_vals) != 21: 649 | estimated_vals = {'LocErr' : 0.023, 'D0' : 1e-20, 'D1' : 0.02, 'D2' : 0.1, 'D3' : 0.5, 'F0' : 0.1, 'F1' : 0.2, 'F2' : 0.3, 'p01' : 0.1, 'p02' : 0.1, 'p03' : 0.1, 'p10' :0.1, 'p12' : 0.1, 'p13' : 0.1, 'p20' :0.1, 'p21' :0.1, 'p23' : 0.1, 'p30' :0.1, 'p31' :0.1, 'p32' : 0.1, 'pBL' : 0.1} 650 | if len(min_values) != 21: 651 | min_values = {'LocErr' : 0.005, 'D0' : 0, 'D1' : 0, 'D2' : 0.001, 'D3' : 0.001, 'F0' : 0.001, 'F1' : 0.001, 'F2' : 0.001, 'p01' : 0.001, 'p02' : 0.001, 'p03' : 0.001, 'p10' :0.001, 'p12' : 0.001, 'p13' : 0.001, 'p20' :0.001, 'p21' :0.001, 'p23' : 0.001, 'p30' :0.001, 'p31' :0.001, 'p32' : 0.001, 'pBL' : 0.001} 652 | if len(max_values) != 21: 653 | max_values = {'LocErr' : 0.023, 'D0' : 1, 'D1' : 1, 'D2' : 10, 'D3' : 100, 'F0' : 0.999, 'F1' : 0.999, 'F2' : 0.999, 'p01' : 1, 'p02' : 1, 'p03' : 1, 'p10' :1, 'p12' : 1, 'p13' : 1, 'p20' : 1, 'p21' : 1, 'p23' : 1, 'p30' : 1, 'p31' : 1, 'p32' : 1, 'pBL' : 0.99} 654 | else: 655 | if len(vary_params) == 7 or len(estimated_vals) == 7 or len(min_values) == 7 or len(max_values) == 7: 656 | raise ValueError('vary_params, estimated_vals, min_values and max_values have to be correctly specified if more than 4 states') 657 | 658 | if not str(all_tracks.__class__) == "": 659 | raise ValueError('all_tracks should be a dictionary of arrays with n there number of steps as keys') 660 | 661 | params = get_params(nb_states, steady_state, vary_params, estimated_vals, min_values, max_values) 662 | 663 | print(params) 664 | 665 | l_list = np.sort(np.array(list(all_tracks.keys())).astype(int)).astype(str) 666 | sorted_tracks = [] 667 | for l in l_list: 668 | if len(all_tracks[l]) > 0 : 669 | sorted_tracks.append(all_tracks[l]) 670 | all_tracks = sorted_tracks 671 | 672 | fit = minimize(cum_Proba_Cs, params, args=(all_tracks, dt, cell_dims, nb_states, nb_substeps, frame_len, verbose, workers), method = method, nan_policy = 'propagate') 673 | 674 | return fit -------------------------------------------------------------------------------- /extrack/readers.py: -------------------------------------------------------------------------------- 1 | import xmltodict 2 | import numpy as np 3 | import pandas as pd 4 | 5 | def read_trackmate_xml(paths, # path (string specifying the path of the file or list of paths in case of multiple files. 6 | lengths=np.arange(5,40), # track lengths kept. 7 | dist_th = 0.5, # maximum distance between consecutive peaks of a track. 8 | frames_boundaries = [-np.inf, np.inf], # min and max frame values allowed for peak detection 9 | remove_no_disp = True, # removes tracks showing no motion if True. 10 | opt_metrics_names = ['t', 'x'], # e.g. ['pred_0', 'pred_1'], 11 | opt_metrics_types = [int, 'float64'] # will assume 'float64' type if none, otherwise specify a list of same length as opt_metrics_names: e.g. ['float64','float64'] 12 | ): 13 | """ 14 | Converts xml output from trackmate to a list of arrays of tracks 15 | each element of the list is an array composed of several tracks (dim 0), of a fix number of position (dim 1) 16 | with x and y coordinates (dim 2) 17 | path : path to xml file 18 | lengths : lengths used for the arrays, list of intergers, tracks with n localization will be added 19 | to the array of length n if n is in the list, if n > max(lenghs)=k the k first steps of the track will be added. 20 | dist_th : maximum distance allowed to connect consecutive peaks 21 | start_frame : first frame considered 22 | """ 23 | if type(paths) == type(''): 24 | paths = [paths] 25 | traces = {} 26 | frames = {} 27 | opt_metrics = {} 28 | for m in opt_metrics_names: 29 | opt_metrics[m] = {} 30 | for l in lengths: 31 | frames[str(l)] = [] 32 | for m in opt_metrics_names: 33 | opt_metrics[m][str(l)] = [] 34 | 35 | if opt_metrics_types == None: 36 | opt_metrics_types = ['float64']*len(opt_metrics_names) 37 | 38 | for path in paths: 39 | data = xmltodict.parse(open(path, 'r').read(), encoding='utf-8') 40 | # Checks 41 | spaceunit = data['Tracks']['@spaceUnits'] 42 | #if spaceunit not in ('micron', 'um', 'µm', 'µm'): 43 | # raise IOError("Spatial unit not recognized: {}".format(spaceunit)) 44 | #if data['Tracks']['@timeUnits'] != 'ms': 45 | # raise IOError("Time unit not recognized") 46 | 47 | # parameters 48 | framerate = float(data['Tracks']['@frameInterval'])/1000. # framerate in ms 49 | 50 | for i, particle in enumerate(data['Tracks']['particle']): 51 | try: 52 | track = [(float(d['@x']), float(d['@y']), float(d['@t'])*framerate, int(d['@t']), i) for d in particle['detection']] 53 | opt_met = np.empty((int(particle['@nSpots']), len(opt_metrics_names)), dtype = 'object') 54 | for k, d in enumerate(particle['detection']): 55 | for j, m in enumerate(opt_metrics_names): 56 | opt_met[k, j] = d['@'+opt_metrics_names[j]] 57 | 58 | track = np.array(track) 59 | if remove_no_disp: 60 | no_zero_disp = np.min((track[1:,0] - track[:-1,0])**2) * np.min((track[1:,1] - track[:-1,1])**2) 61 | else: 62 | no_zero_disp = True 63 | 64 | dists = np.sum((track[1:, :2] - track[:-1, :2])**2, axis = 1)**0.5 65 | if no_zero_disp and track[0, 3] >= frames_boundaries[0] and track[0, 3] <= frames_boundaries[1] and np.all(dists np.max(lengths): 73 | l = np.max(lengths) 74 | traces[str(l)].append(track[:l, 0:2]) 75 | frames[str(l)].append(track[:l, 3]) 76 | for k, m in enumerate(opt_metrics_names): 77 | opt_metrics[m][str(l)].append(opt_met[:l, k]) 78 | except : 79 | print('problem with data on path :', path) 80 | raise e 81 | 82 | for l in list(traces.keys()): 83 | if len(traces[l])>0: 84 | traces[l] = np.array(traces[l]) 85 | frames[l] = np.array(frames[l]) 86 | for k, m in enumerate(opt_metrics_names): 87 | cur_opt_met = np.array(opt_metrics[m][l]) 88 | try: 89 | cur_opt_met = cur_opt_met.astype(opt_metrics_types[k]) 90 | except : 91 | print('Error of type with the optional metric:', m) 92 | opt_metrics[m][l] = cur_opt_met 93 | else: 94 | del traces[l], frames[l] 95 | for k, m in enumerate(opt_metrics_names): 96 | del opt_metrics[m][l] 97 | 98 | return traces, frames, opt_metrics 99 | 100 | def read_table(paths, # path of the file to read or list of paths to read multiple files. 101 | lengths = np.arange(5,40), # number of positions per track accepted (take the first position if longer than max 102 | dist_th = np.inf, # maximum distance allowed for consecutive positions 103 | frames_boundaries = [-np.inf, np.inf], # min and max frame values allowed for peak detection 104 | fmt = 'csv', # format of the document to be red, 'csv' or 'pkl', one can also just specify a separator e.g. ' '. 105 | colnames = ['POSITION_X', 'POSITION_Y', 'FRAME', 'TRACK_ID'], # if multiple columns are required to identify a track, the string used to identify the track ID can be replaced by a list of strings represening the column names e.g. ['TRACK_ID', 'Movie_ID'] 106 | opt_colnames = [], # list of additional metrics to collect e.g. ['QUALITY', 'ID'] 107 | remove_no_disp = True): 108 | 109 | if type(paths) == str or type(paths) == np.str_: 110 | paths = [paths] 111 | 112 | tracks = {} 113 | frames = {} 114 | opt_metrics = {} 115 | for m in opt_colnames: 116 | opt_metrics[m] = {} 117 | nb_peaks = 0 118 | for l in lengths: 119 | tracks[str(l)] = [] 120 | frames[str(l)] = [] 121 | for m in opt_colnames: 122 | opt_metrics[m][str(l)] = [] 123 | 124 | for path in paths: 125 | 126 | if fmt == 'csv': 127 | data = pd.read_csv(path, sep=',') 128 | elif fmt == 'pkl': 129 | data = pd.read_pickle(path) 130 | else: 131 | data = pd.read_csv(path, sep = fmt) 132 | 133 | if not (type(colnames[3]) == str or type(colnames[3]) == np.str_): 134 | # in this case we remove the NA values for simplicity 135 | None_ID = (data[colnames[3]] == 'None') + pd.isna(data[colnames[3]]) 136 | data = data.drop(data[np.any(None_ID,1)].index) 137 | 138 | new_ID = data[colnames[3][0]].astype(str) 139 | 140 | for k in range(1,len(colnames[3])): 141 | new_ID = new_ID + '_' + data[colnames[3][k]].astype(str) 142 | data['unique_ID'] = new_ID 143 | colnames[3] = 'unique_ID' 144 | try: 145 | # in this case, peaks without an ID are assumed alone and are added a unique ID, only works if ID are integers 146 | None_ID = (data[colnames[3]] == 'None' ) + pd.isna(data[colnames[3]]) 147 | max_ID = np.max(data[colnames[3]][(data[colnames[3]] != 'None' ) * (pd.isna(data[colnames[3]]) == False)].astype(int)) 148 | data.loc[None_ID, colnames[3]] = np.arange(max_ID+1, max_ID+1 + np.sum(None_ID)) 149 | except: 150 | None_ID = (data[colnames[3]] == 'None' ) + pd.isna(data[colnames[3]]) 151 | data = data.drop(data[None_ID].index) 152 | 153 | data = data[colnames + opt_colnames] 154 | 155 | zero_disp_tracks = 0 156 | 157 | try: 158 | for ID, track in data.groupby(colnames[3]): 159 | 160 | track = track.sort_values(colnames[2], axis = 0) 161 | track_mat = track.values[:,:3].astype('float64') 162 | dists2 = (track_mat[1:, :2] - track_mat[:-1, :2])**2 163 | if remove_no_disp: 164 | if np.mean(dists2==0)>0.05: 165 | continue 166 | dists = np.sum(dists2, axis = 1)**0.5 167 | if track_mat[0, 2] >= frames_boundaries[0] and track_mat[0, 2] <= frames_boundaries[1] : #and np.all(distsdist_th): 169 | 170 | if np.any([len(track_mat)]*len(lengths) == np.array(lengths)): 171 | l = len(track) 172 | tracks[str(l)].append(track_mat[:, 0:2]) 173 | frames[str(l)].append(track_mat[:, 2]) 174 | for m in opt_colnames: 175 | opt_metrics[m][str(l)].append(track[m].values) 176 | 177 | elif len(track_mat) > np.max(lengths): 178 | l = np.max(lengths) 179 | tracks[str(l)].append(track_mat[:l, 0:2]) 180 | frames[str(l)].append(track_mat[:l, 2]) 181 | for m in opt_colnames: 182 | opt_metrics[m][str(l)].append(track[m].values[:l]) 183 | 184 | elif len(track_mat) < np.max(lengths) and len(track_mat) > np.min(lengths) : # in case where lengths between min(lengths) and max(lentghs) are not all present: 185 | l_idx = np.argmin(np.floor(len(track_mat) / lengths))-1 186 | l = lengths[l_idx] 187 | tracks[str(l)].append(track_mat[:l, 0:2]) 188 | frames[str(l)].append(track_mat[:l, 2]) 189 | except : 190 | print('problem with file :', path) 191 | 192 | for l in list(tracks.keys()): 193 | if len(tracks[str(l)])>0: 194 | print(l) 195 | tracks[str(l)] = np.array(tracks[str(l)]) 196 | frames[str(l)] = np.array(frames[str(l)]) 197 | for m in opt_colnames: 198 | opt_metrics[m][str(l)] = np.array(opt_metrics[m][str(l)]) 199 | else: 200 | del tracks[str(l)], frames[str(l)] 201 | for k, m in enumerate(opt_colnames): 202 | del opt_metrics[m][str(l)] 203 | 204 | if zero_disp_tracks and not remove_no_disp: 205 | print('Warning: some tracks show no displacements. To be checked if normal or not. These tracks can be removed with remove_no_disp = True') 206 | return tracks, frames, opt_metrics 207 | -------------------------------------------------------------------------------- /extrack/refined_loc_old.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Fri Aug 13 15:31:38 2021 5 | 6 | @author: francois 7 | """ 8 | GPU_computing = 0 9 | 10 | import numpy as np 11 | 12 | if GPU_computing : 13 | import cupy as cp 14 | from cupy import asnumpy 15 | else: 16 | # if CPU computing : 17 | import numpy as cp 18 | def asnumpy(x): 19 | return cp.array(x) 20 | try: 21 | from matplotlib import pyplot as plt 22 | import imageio 23 | except: 24 | pass 25 | 26 | from extrack.old_tracking import extract_params, predict_Bs, P_Cs_inter_bound_stats, log_integrale_dif, first_log_integrale_dif, ds_froms_states, fuse_tracks, get_all_Bs, get_Ts_from_Bs 27 | from extrack.tracking import P_Cs_inter_bound_stats 28 | from extrack.exporters import extrack_2_matrix 29 | from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas 30 | 31 | 32 | def prod_2GaussPDF(sigma1,sigma2, mu1, mu2): 33 | sigma = ((sigma1**2*sigma2**2)/(sigma1**2+sigma2**2))**0.5 34 | mu = (mu1*sigma2**2 + mu2*sigma1**2)/(sigma1**2 + sigma2**2) 35 | LK = np.sum(-0.5*np.log(2*np.pi*(sigma1**2 + sigma2**2)) -(mu1-mu2)**2/(2*(sigma1**2 + sigma2**2)),-1) 36 | return sigma, mu, LK 37 | 38 | def prod_3GaussPDF(sigma1,sigma2,sigma3, mu1, mu2, mu3): 39 | sigma, mu, LK = prod_2GaussPDF(sigma1,sigma2, mu1, mu2) 40 | sigma, mu, LK2 = prod_2GaussPDF(sigma,sigma3, mu, mu3) 41 | LK = LK + LK2 42 | return sigma, mu, LK 43 | 44 | def gaussian(x, sig, mu): 45 | return np.product(1/(2*np.pi*sig**2)**0.5 * np.exp(-(x-mu)**2/(2*sig**2)), -1) 46 | 47 | def get_LC_Km_Ks(Cs, LocErr, ds, Fs, TrMat, nb_substeps=1, frame_len = 4): 48 | ''' 49 | variation of the main function to extract LC, Km and Ks for all positions 50 | ''' 51 | nb_Tracks = len(Cs) 52 | nb_locs = len(Cs[0]) # number of localization per track 53 | nb_dims = len(Cs[0,0]) # number of spatial dimentions (x, y) or (x, y, z) 54 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 55 | Cs = cp.array(Cs) 56 | 57 | nb_states = TrMat.shape[0] 58 | all_Km = [] 59 | all_Ks = [] 60 | all_LP = [] 61 | 62 | preds = np.zeros((nb_Tracks, nb_locs, nb_states))-1 63 | 64 | TrMat = cp.array(TrMat) # transition matrix of the markovian process 65 | current_step = 1 66 | 67 | cur_Bs = get_all_Bs(nb_substeps + 1, nb_states) # get initial sequences of states 68 | cur_Bs = cur_Bs[None,:,:] # include dim for different tracks 69 | 70 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) #states of interest for the current displacement 71 | cur_nb_Bs = cur_Bs.shape[1] 72 | 73 | # compute the vector of diffusion stds knowing the current states 74 | ds = cp.array(ds) 75 | Fs = cp.array(Fs) 76 | 77 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 78 | LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 79 | 80 | LP = LT # current log proba of seeing the track 81 | LP = cp.repeat(LP, nb_Tracks, axis = 0) 82 | 83 | cur_ds = ds_froms_states(ds, cur_states) 84 | 85 | # inject the first position to get the associated Km and Ks : 86 | Km, Ks = first_log_integrale_dif(Cs[:,:, nb_locs-current_step], LocErr, cur_ds) 87 | 88 | Ks = cp.repeat(Ks, nb_Tracks, axis = 0) 89 | 90 | current_step += 1 91 | Km = cp.repeat(Km, cur_nb_Bs, axis = 1) 92 | removed_steps = 0 93 | 94 | all_Km.append(Km) 95 | all_Ks.append(Ks) 96 | all_LP.append(LP) 97 | 98 | while current_step <= nb_locs-1: 99 | # update cur_Bs to describe the states at the next step : 100 | cur_Bs = get_all_Bs(current_step*nb_substeps+1 - removed_steps, nb_states)[None] 101 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) 102 | # compute the vector of diffusion stds knowing the states at the current step 103 | 104 | cur_ds = ds_froms_states(ds, cur_states) 105 | 106 | LT = get_Ts_from_Bs(cur_states, TrMat) 107 | 108 | # repeat the previous matrix to account for the states variations due to the new position 109 | Km = cp.repeat(Km, nb_states**nb_substeps , axis = 1) 110 | Ks = cp.repeat(Ks, nb_states**nb_substeps, axis = 1) 111 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1) 112 | # inject the next position to get the associated Km, Ks and Constant describing the integral of 3 normal laws : 113 | Km, Ks, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr, cur_ds, Km, Ks) 114 | #print('integral',time.time() - t0); t0 = time.time() 115 | LP += LT + LC # current (log) constants associated with each track and sequences of states 116 | del LT, LC 117 | cur_nb_Bs = len(cur_Bs[0]) # current number of sequences of states 118 | 119 | ''''idea : the position and the state 6 steps ago should not impact too much the 120 | probability of the next position so the Km and Ks of tracks with the same 6 last 121 | states must be very similar, we can then fuse the parameters of the pairs of Bs 122 | which vary only for the last step (7) and sum their probas''' 123 | if current_step < nb_locs-1: 124 | while cur_nb_Bs >= nb_states**frame_len: 125 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 126 | log_integrated_term = -cp.log(2*cp.pi*newKs**2) - cp.sum((Cs[:,:,nb_locs-current_step] - Km)**2,axis=2)/(2*newKs**2) 127 | LF = 0 #cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 128 | 129 | test_LP = LP + log_integrated_term + LF 130 | 131 | if cp.max(test_LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 132 | test_LP = test_LP - (cp.max(test_LP)-600) 133 | 134 | P = cp.exp(test_LP) 135 | for state in range(nb_states): 136 | B_is_state = cur_Bs[:,:,-1] == state 137 | preds[:,nb_locs-current_step+frame_len-2, state] = asnumpy(cp.sum(B_is_state*P,axis = 1)/cp.sum(P,axis = 1)) 138 | cur_Bs.shape 139 | cur_Bs = cur_Bs[:,:cur_nb_Bs//nb_states, :-1] 140 | Km, Ks, LP = fuse_tracks(Km, Ks, LP, cur_nb_Bs, nb_states) 141 | cur_nb_Bs = len(cur_Bs[0]) 142 | removed_steps += 1 143 | 144 | all_Km.append(Km) 145 | all_Ks.append(Ks) 146 | all_LP.append(LP) 147 | 148 | current_step += 1 149 | 150 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 151 | log_integrated_term = -cp.log(2*cp.pi*newKs**2) - cp.sum((Cs[:,:,0] - Km)**2,axis=2)/(2*newKs**2) 152 | LF = cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 153 | #LF = cp.log(0.5) 154 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 155 | LP += log_integrated_term + LF 156 | 157 | pred_LP = LP 158 | if cp.max(LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 159 | pred_LP = LP - (cp.max(LP)-600) 160 | 161 | P = cp.exp(pred_LP) 162 | for state in range(nb_states): 163 | B_is_state = cur_Bs[:,:] == state 164 | preds[:,0:frame_len, state] = asnumpy(cp.sum(B_is_state*P[:,:,None],axis = 1)/cp.sum(P[:,:,None],axis = 1)) 165 | return LP, cur_Bs, preds, all_Km, all_Ks, all_LP 166 | 167 | def get_pos_PDF(Cs, LocErr, ds, Fs, TrMat, frame_len = 7): 168 | ds = cp.array(ds) 169 | Cs = cp.array(Cs) 170 | # get Km, Ks and LC forward 171 | LP1, final_Bs1, preds1, all_Km1, all_Ks1, all_LP1 = get_LC_Km_Ks(Cs, LocErr, ds, Fs, TrMat, nb_substeps=1, frame_len = frame_len) 172 | #get Km, Ks and LC backward 173 | TrMat2 = np.copy(TrMat).T # transpose the matrix for the backward transitions 174 | Cs2 = Cs[:,::-1,:] # inverse the time steps 175 | LP2, final_Bs2, preds2, all_Km2, all_Ks2, all_LP2 = get_LC_Km_Ks(Cs2, LocErr, ds, cp.ones(TrMat2.shape[0],)/TrMat2.shape[0], TrMat2, nb_substeps=1, frame_len = frame_len) # we set a neutral Fs so it doesn't get counted twice 176 | 177 | # do the approximation for the first position, product of 2 gaussian PDF, (integrated term and localization error) 178 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks1[-1], Cs[:,None,0], all_Km1[-1]) 179 | 180 | LP = all_LP1[-1] + LC 181 | all_pos_means = [mu] 182 | all_pos_stds = [sig] 183 | all_pos_weights = [LP] 184 | all_pos_Bs = [final_Bs1] 185 | 186 | for k in range(1,Cs.shape[1]-1): 187 | ''' 188 | we take the corresponding Km1, Ks1, LP1, Km2, Ks2, LP2 189 | which are the corresponding stds and means of the resulting 190 | PDF surrounding the position k 191 | with localization uncertainty, we have 3 gaussians to compress to 1 gaussian * K 192 | This has to be done for all combinations of set of consective states before and after.step k 193 | to do so we set dim 1 as dim for consecutive states computed by the forward proba and 194 | dim 2 for sets of states computed by the backward proba. 195 | 196 | ''' 197 | LP1 = all_LP1[-1-k][:,:,None] 198 | Km1 = all_Km1[-1-k][:,:,None] 199 | Ks1 = all_Ks1[-1-k][:,:,None] 200 | LP2 = all_LP2[k-1][:,None] 201 | Km2 = all_Km2[k-1][:,None] 202 | Ks2 = all_Ks2[k-1][:,None] 203 | 204 | nb_Bs1 = Ks1.shape[1] 205 | nb_Bs2 = Ks2.shape[2] 206 | nb_tracks = Km1.shape[0] 207 | nb_dims = Km1.shape[3] 208 | nb_states = TrMat.shape[0] 209 | LP2.shape 210 | Bs2_len = np.min([k+1, frame_len-1]) 211 | cur_Bs2 = get_all_Bs(Bs2_len, nb_states) 212 | # we must reorder the metrics so the Bs from the backward terms correspond to the forward terms 213 | indexes = cp.sum(cur_Bs2 * nb_states**cp.arange(Bs2_len)[::-1][None],1).astype(int) 214 | 215 | LP2 = LP2[:,:,indexes] 216 | Km2 = Km2[:,:,indexes] 217 | Ks2 = Ks2[:,:,indexes] 218 | Km2.shape 219 | # we must associate only forward and backward metrics that share the same state at position k as followed : 220 | slice_len = nb_Bs1//nb_states 221 | new_LP1 = LP1[:,0:slice_len:nb_states] 222 | new_Km1 = Km1[:,0:slice_len:nb_states] 223 | new_Ks1 = Ks1[:,0:slice_len:nb_states] 224 | for i in range(1,nb_states): 225 | new_LP1 = cp.concatenate((new_LP1,LP1[:,i*slice_len+i:(i+1)*slice_len:nb_states]), 1) 226 | new_Km1 = cp.concatenate((new_Km1,Km1[:,i*slice_len+i:(i+1)*slice_len:nb_states]), 1) 227 | new_Ks1 = cp.concatenate((new_Ks1,Ks1[:,i*slice_len+i:(i+1)*slice_len:nb_states]), 1) 228 | 229 | LP1 = new_LP1 230 | Km1 = new_Km1 231 | Ks1 = new_Ks1 232 | 233 | cur_nb_pos = np.round((np.log(nb_Bs2)+np.log(nb_Bs1//nb_states))/np.log(nb_states)).astype(int) 234 | cur_Bs = get_all_Bs(cur_nb_pos, nb_states)[None] 235 | 236 | sig, mu, LC = prod_3GaussPDF(Ks1,LocErr,Ks2, Km1, Cs[:,None,None,k], Km2) 237 | LP = LP1 + LP2 + LC 238 | sig.shape 239 | sig = sig.reshape((nb_tracks,(nb_Bs1//nb_states)*nb_Bs2,1)) 240 | mu = mu.reshape((nb_tracks,(nb_Bs1//nb_states)*nb_Bs2, nb_dims)) 241 | LP = LP.reshape((nb_tracks,(nb_Bs1//nb_states)*nb_Bs2)) 242 | 243 | all_pos_means.append(mu) 244 | all_pos_stds.append(sig) 245 | all_pos_weights.append(LP) 246 | all_pos_Bs.append(cur_Bs) 247 | 248 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks2[-1], Cs[:,None,-1], all_Km2[-1]) 249 | LP = all_LP2[-1] + LC 250 | 251 | cur_Bs2 = get_all_Bs(Bs2_len, nb_states) 252 | all_pos_means.append(mu) 253 | all_pos_stds.append(sig) 254 | all_pos_weights.append(LP) 255 | all_pos_Bs.append(final_Bs2) 256 | 257 | return all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs 258 | 259 | def position_refinement(all_tracks, LocErr, ds, Fs, TrMat, frame_len = 7): 260 | all_mus = {} 261 | all_sigmas = {} 262 | for l in all_tracks.keys(): 263 | Cs = all_tracks[l] 264 | all_mus[l] = np.zeros((Cs.shape[0], int(l), Cs.shape[2])) 265 | all_sigmas[l] = np.zeros((Cs.shape[0], int(l))) 266 | all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs = get_pos_PDF(Cs, LocErr, ds, Fs, TrMat, frame_len = 7) 267 | #best_mus, best_sigs, best_Bs = get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds) 268 | for k, (pos_means, pos_stds, pos_weights) in enumerate(zip(all_pos_means, all_pos_stds, all_pos_weights)): 269 | all_mus[l][:, k] = np.sum(pos_weights[:,:,None]*pos_means, 1) / np.sum(pos_weights, 1)[:,None] 270 | all_sigmas[l][:, k] = (np.sum(pos_weights[:,:]*pos_stds[:,:,0]**2, 1) / np.sum(pos_weights, 1))**0.5 271 | return all_mus, all_sigmas 272 | 273 | def get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds): 274 | nb_Bs = [] 275 | nb_pos = len(all_pos_weights) 276 | for weights in all_pos_weights: 277 | nb_Bs.append(weights.shape[1]) 278 | nb_Bs = np.max(nb_Bs) 279 | nb_states = (np.max(all_pos_Bs[0])+1).astype(int) 280 | max_frame_len = (np.log(nb_Bs) // np.log(nb_states)).astype(int) 281 | mid_frame_pos = ((max_frame_len-0.1)//2).astype(int) # -0.1 is used for mid_frame_pos to be the good index for both odd and pair numbers 282 | mid_frame_pos = np.max([1,mid_frame_pos]) 283 | best_Bs = [] 284 | best_mus = [] 285 | best_sigs = [] 286 | for k, (weights, Bs, mus, sigs) in enumerate(zip(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds)): 287 | if k <= nb_pos/2 : 288 | idx = np.min([k, mid_frame_pos]) 289 | else : 290 | idx = np.max([-mid_frame_pos-1, k - nb_pos]) 291 | best_args = np.argmax(weights, 1) 292 | best_Bs.append(Bs[0,best_args][:,idx]) 293 | best_sigs.append(sigs[[cp.arange(len(mus)), best_args]]) 294 | best_mus.append(mus[[cp.arange(len(mus)), best_args]]) 295 | best_Bs = cp.array(best_Bs).T.astype(int) 296 | best_sigs = cp.transpose(cp.array(best_sigs), (1,0,2)) 297 | best_mus = cp.transpose(cp.array(best_mus), (1,0,2)) 298 | return asnumpy(best_mus), asnumpy(best_sigs), asnumpy(best_Bs) 299 | 300 | def save_gifs(Cs, all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs, gif_pathnames = './tracks', lim = None, nb_pix = 200, fps=1): 301 | try: 302 | plt 303 | imageio 304 | except: 305 | raise ImportError('matplotlib and imageio has to be installed to use save_gifs') 306 | best_mus, best_sigs, best_Bs = get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds) 307 | for ID in range(len(Cs)): 308 | all_images = [] 309 | Cs_offset = np.mean(Cs[ID], 0) 310 | Cs[ID] = Cs[ID] 311 | 312 | if lim == None: 313 | cur_lim = np.max(np.abs(Cs[ID]))*1.1 314 | else: 315 | cur_lim = lim 316 | pix_size = nb_pix / (2*cur_lim) 317 | for k in range(len(all_pos_means)): 318 | 319 | sig = asnumpy(all_pos_stds[k]) 320 | mu = asnumpy(all_pos_means[k]) 321 | LP = asnumpy(all_pos_weights[k]) 322 | mu.shape 323 | fig = plt.figure() 324 | plt.plot((Cs[ID, :,1] - Cs_offset[1] + cur_lim)*pix_size-0.5, (Cs[ID, :,0]- Cs_offset[0]+cur_lim)*pix_size-0.5) 325 | plt.scatter((best_mus[ID, :,1] - Cs_offset[1] + cur_lim)*pix_size-0.5, (best_mus[ID, :,0] - Cs_offset[0]+cur_lim)*pix_size-0.5, c='r', s=3) 326 | best_mus.shape 327 | 328 | P_xs = gaussian(np.linspace(-cur_lim,cur_lim,nb_pix)[None,:,None], sig[ID][:,:,None], mu[ID][:,:1,None] - Cs_offset[0]) * np.exp(LP[ID]-np.max(LP[ID]))[:,None] 329 | P_ys = gaussian(np.linspace(-cur_lim,cur_lim,nb_pix)[None,:,None], sig[ID][:,:,None], mu[ID][:,1:,None] - Cs_offset[1]) * np.exp(LP[ID]-np.max(LP[ID]))[:,None] 330 | 331 | heatmap = np.sum(P_xs[:,:,None]*P_ys[:,None] * np.exp(LP[ID]-np.max(LP[ID]))[:,None,None],0) 332 | 333 | heatmap = heatmap/np.max(heatmap) 334 | plt.imshow(heatmap) 335 | plt.xticks(np.linspace(0,nb_pix-1, 5), np.round(np.linspace(-cur_lim,cur_lim, 5), 2)) 336 | plt.yticks(np.linspace(0,nb_pix-1, 5), np.round(np.linspace(-cur_lim,cur_lim, 5), 2)) 337 | canvas = FigureCanvas(fig) 338 | canvas.draw() 339 | s, (width, height) = canvas.print_to_buffer() 340 | image = np.fromstring(s, dtype='uint8').reshape((height, width, 4)) 341 | 342 | all_images.append(image) 343 | plt.close() 344 | 345 | imageio.mimsave(gif_pathnames + str(ID)+'.gif', all_images,fps=fps) 346 | 347 | 348 | def get_LC_Km_Ks_fixed_Bs(Cs, LocErr, ds, Fs, TrMat, Bs): 349 | ''' 350 | variation of the main function to extract LC, Km and Ks for all positions 351 | ''' 352 | nb_Tracks = len(Cs) 353 | nb_locs = len(Cs[0]) # number of localization per track 354 | nb_dims = len(Cs[0,0]) # number of spatial dimentions (x, y) or (x, y, z) 355 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 356 | Cs = cp.array(Cs) 357 | 358 | all_Km = [] 359 | all_Ks = [] 360 | all_LP = [] 361 | 362 | TrMat = cp.array(TrMat) # transition matrix of the markovian process 363 | current_step = 1 364 | 365 | cur_states = Bs[:,:,-2:].astype(int) #states of interest for the current displacement 366 | 367 | # compute the vector of diffusion stds knowing the current states 368 | ds = cp.array(ds) 369 | Fs = cp.array(Fs) 370 | 371 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 372 | LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 373 | 374 | LP = LT # current log proba of seeing the track 375 | LP = cp.repeat(LP, nb_Tracks, axis = 0) 376 | 377 | cur_ds = ds_froms_states(ds, cur_states) 378 | 379 | # inject the first position to get the associated Km and Ks : 380 | Km, Ks = first_log_integrale_dif(Cs[:,:, nb_locs-current_step], LocErr, cur_ds) 381 | all_Km.append(Km) 382 | all_Ks.append(Ks) 383 | all_LP.append(LP) 384 | current_step += 1 385 | 386 | while current_step <= nb_locs-1: 387 | # update cur_Bs to describe the states at the next step : 388 | cur_states = Bs[:,:,-current_step-1:-current_step+1].astype(int) 389 | # compute the vector of diffusion stds knowing the states at the current step 390 | cur_ds = ds_froms_states(ds, cur_states) 391 | LT = get_Ts_from_Bs(cur_states, TrMat) 392 | 393 | # inject the next position to get the associated Km, Ks and Constant describing the integral of 3 normal laws : 394 | Km, Ks, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr, cur_ds, Km, Ks) 395 | #print('integral',time.time() - t0); t0 = time.time() 396 | LP += LT + LC # current (log) constants associated with each track and sequences of states 397 | del LT, LC 398 | 399 | all_Km.append(Km) 400 | all_Ks.append(Ks) 401 | all_LP.append(LP) 402 | 403 | current_step += 1 404 | 405 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 406 | log_integrated_term = -cp.log(2*np.pi*newKs**2) - cp.sum((Cs[:,:,0] - Km)**2,axis=2)/(2*newKs**2) 407 | LF = cp.log(Fs[Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 408 | #LF = cp.log(0.5) 409 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 410 | LP += log_integrated_term + LF 411 | 412 | all_Ks = cp.array(all_Ks)[:,0,0] 413 | all_Km = cp.array(all_Km)[:,0,0] 414 | all_LP = cp.array(all_LP)[:,0,0] 415 | return all_Km, all_Ks, all_LP 416 | 417 | def get_pos_PDF_fixedBs(Cs, LocErr, ds, Fs, TrMat, Bs): 418 | ''' 419 | get mu and sigma for each position given inputed Bs, 420 | ideally used for a single track with its most likely set of states 421 | ''' 422 | ds = np.array(ds) 423 | Cs = cp.array(Cs) 424 | # get Km, Ks and LC forward 425 | all_Km1, all_Ks1, all_LP1 = get_LC_Km_Ks_fixed_Bs(Cs, LocErr, ds, Fs, TrMat, Bs) 426 | #get Km, Ks and LC backward 427 | TrMat2 = np.copy(TrMat).T # transpose the matrix for the backward transitions 428 | Cs2 = Cs[:,::-1,:] # inverse the time steps 429 | all_Km2, all_Ks2, all_LP2 = get_LC_Km_Ks_fixed_Bs(Cs2, LocErr, ds, cp.ones(TrMat2.shape[0],)/TrMat2.shape[0], TrMat2, Bs[:,:,::-1]) 430 | # do the approximation for the first position, product of 2 gaussian PDF, (integrated term and localization error) 431 | 432 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks1[-1], Cs[:,0], all_Km1[-1]) 433 | np 434 | all_pos_means = [mu] 435 | all_pos_stds = [sig[None]] 436 | Cs.shape 437 | for k in range(1,Cs.shape[1]-1): 438 | 439 | Km1 = all_Km1[-k][None] 440 | Ks1 = all_Ks1[-1-k][None] 441 | Km2 = all_Km2[k-1][None] 442 | Ks2 = all_Ks2[k-1][None] 443 | 444 | sig, mu, LC = prod_3GaussPDF(Ks1,LocErr,Ks2, Km1, Cs[:,k], Km2) 445 | 446 | all_pos_means.append(mu) 447 | all_pos_stds.append(sig) 448 | 449 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks2[-1], Cs[:,-1], all_Km2[-1]) 450 | 451 | all_pos_means.append(mu) 452 | all_pos_stds.append(sig[None]) 453 | return cp.array(all_pos_means)[:,0], cp.array(all_pos_stds)[:,0] 454 | 455 | def get_global_sigs_mus(all_pos_means, all_pos_stds, all_pos_weights, idx = 0): 456 | w_sigs = [] 457 | w_mus = [] 458 | for mus, sigs, LC in zip(all_pos_means, all_pos_stds, all_pos_weights): 459 | mus = mus[idx] 460 | sigs = sigs[idx] 461 | LC = LC[idx] 462 | LC = LC - np.max(LC, keepdims = True) 463 | #sigs = sigs[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 464 | #mus = mus[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 465 | #LC = LC[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 466 | w_sigs.append(np.sum(np.exp(LC[:,None])**2 * sigs) / np.sum(np.exp(LC[:,None])**2)) 467 | w_mus.append(np.sum(np.exp(LC[:,None]) * mus,0) / np.sum(np.exp(LC[:,None]),0)) 468 | return np.array(w_mus), np.array(w_sigs) 469 | 470 | def full_extrack_2_matrix(all_tracks, params, dt, all_frames = None, cell_dims = [1,None,None], nb_states = 2, frame_len = 15): 471 | nb_dims = list(all_tracks.items())[0][1].shape[2] 472 | pred_Bss = predict_Bs(all_tracks, dt, params, nb_states=nb_states, frame_len=frame_len, cell_dims = cell_dims) 473 | 474 | DATA = extrack_2_matrix(all_tracks, pred_Bss, dt, all_frames = all_frames) 475 | DATA = np.concatenate((DATA, np.empty((DATA.shape[0], nb_dims+1))),1) 476 | LocErr, ds, Fs, TrMat = extract_params(params, dt, nb_states, nb_substeps = 1) 477 | for ID in np.unique(DATA[:,2]): 478 | track = DATA[DATA[:,2]==ID,:nb_dims][None] 479 | all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs = get_pos_PDF(track, LocErr, ds, Fs, TrMat, frame_len = frame_len//2+3) 480 | w_mus, w_sigs = get_global_sigs_mus(all_pos_means, all_pos_stds, all_pos_weights, idx = 0) 481 | DATA[DATA[:,2]==ID,-1] = w_sigs 482 | DATA[DATA[:,2]==ID,-nb_dims-1:-1] = w_mus 483 | return DATA 484 | 485 | def get_best_estimates(Cs, LocErr, ds, Fs, TrMat, frame_len = 10): 486 | all_mus = [] 487 | all_sigs = [] 488 | for track in Cs: 489 | a,b, preds = P_Cs_inter_bound_stats(track[None], LocErr, ds, Fs, TrMat, nb_substeps=1, do_frame = 1, frame_len = frame_len, do_preds = 1) 490 | Bs = np.argmax(preds, 2)[None] 491 | mus, sigs = get_pos_PDF_fixedBs(Cs, LocErr, ds, Fs, TrMat, Bs) 492 | all_mus.append(mus) 493 | all_sigs.append(sigs) 494 | return mus, sigs 495 | 496 | 497 | 498 | def do_gifs_from_params(all_tracks, params, dt, gif_pathnames = './tracks', frame_len = 9, nb_states = 2, nb_pix = 200, fps = 1): 499 | for Cs in all_tracks: 500 | LocErr, ds, Fs, TrMat = extract_params(params, dt, nb_states, nb_substeps = 1) 501 | all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs = get_pos_PDF(Cs, LocErr, ds, Fs, TrMat, frame_len = frame_len) 502 | save_gifs(Cs, all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs, gif_pathnames = gif_pathnames + '_' + str(len(Cs[0])) + '_pos', lim = None, nb_pix = nb_pix, fps=fps) 503 | -------------------------------------------------------------------------------- /extrack/refined_localization.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Fri Aug 13 15:31:38 2021 5 | 6 | @author: francois 7 | """ 8 | GPU_computing = 0 9 | 10 | import numpy as np 11 | 12 | if GPU_computing : 13 | import cupy as cp 14 | from cupy import asnumpy 15 | else: 16 | # if CPU computing : 17 | import numpy as cp 18 | def asnumpy(x): 19 | return cp.array(x) 20 | try: 21 | from matplotlib import pyplot as plt 22 | import imageio 23 | except: 24 | pass 25 | 26 | #from extrack.old_tracking import extract_params, predict_Bs, P_Cs_inter_bound_stats, log_integrale_dif, first_log_integrale_dif, ds_froms_states, fuse_tracks, get_all_Bs, get_Ts_from_Bs 27 | from extrack.tracking_0 import extract_params, predict_Bs, P_Cs_inter_bound_stats, log_integrale_dif, first_log_integrale_dif, ds_froms_states, fuse_tracks, get_all_Bs, get_Ts_from_Bs 28 | from extrack.tracking import fuse_tracks_th 29 | from extrack.tracking_0 import P_Cs_inter_bound_stats 30 | from extrack.exporters import extrack_2_matrix 31 | from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas 32 | 33 | def prod_2GaussPDF(sigma1,sigma2, mu1, mu2): 34 | sigma = ((sigma1**2*sigma2**2)/(sigma1**2+sigma2**2))**0.5 35 | mu = (mu1*sigma2**2 + mu2*sigma1**2)/(sigma1**2 + sigma2**2) 36 | LK = np.sum(-0.5*np.log(2*np.pi*(sigma1**2 + sigma2**2)) -(mu1-mu2)**2/(2*(sigma1**2 + sigma2**2)),-1) 37 | return sigma, mu, LK 38 | 39 | def prod_3GaussPDF(sigma1,sigma2,sigma3, mu1, mu2, mu3): 40 | sigma, mu, LK = prod_2GaussPDF(sigma1,sigma2, mu1, mu2) 41 | sigma, mu, LK2 = prod_2GaussPDF(sigma,sigma3, mu, mu3) 42 | LK = LK + LK2 43 | return sigma, mu, LK 44 | 45 | def gaussian(x, sig, mu): 46 | return np.product(1/(2*np.pi*sig**2)**0.5 * np.exp(-(x-mu)**2/(2*sig**2)), -1) 47 | 48 | def get_LC_Km_Ks(Cs, LocErr, ds, Fs, TrMat, nb_substeps=1, frame_len = 4, threshold = 0.2, max_nb_states = 1000): 49 | ''' 50 | variation of the main function to extract LC, Km and Ks for all positions 51 | ''' 52 | nb_Tracks = len(Cs) 53 | nb_locs = len(Cs[0]) # number of localization per track 54 | nb_dims = len(Cs[0,0]) # number of spatial dimentions (x, y) or (x, y, z) 55 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 56 | Cs = cp.array(Cs) 57 | 58 | nb_states = TrMat.shape[0] 59 | all_Km = [] 60 | all_Ks = [] 61 | all_LP = [] 62 | all_cur_Bs = [] 63 | 64 | LocErr = LocErr[:,None] 65 | LocErr = LocErr[:,:,::-1] # useful when single peak localization error is inputed, 66 | LocErr2 = LocErr**2 67 | if LocErr.shape[2] == 1: # combined to min(LocErr_index, nb_locs-current_step) it will select the right index 68 | LocErr_index = -1 69 | elif LocErr.shape[2] == nb_locs: 70 | LocErr_index = nb_locs 71 | else: 72 | raise ValueError("Localization error is not specified correctly, in case of unique localization error specify a float number in estimated_vals['LocErr'].\n If one localization error per dimension, specify a list or 1D array of elements the localization error for each dimension.\n If localization error is predetermined by another method for each position the argument input_LocErr should be a dict for each track length of the 3D arrays corresponding to all_tracks (can be obtained from the reader functions using the opt_colname argument)") 73 | 74 | preds = np.zeros((nb_Tracks, nb_locs, nb_states))-1 75 | 76 | TrMat = cp.array(TrMat) # transition matrix of the markovian process 77 | current_step = 1 78 | 79 | cur_Bs = get_all_Bs(nb_substeps + 1, nb_states)[None] # get initial sequences of states 80 | cur_Bs_cat = (cur_Bs[:,:,:,None] == np.arange(nb_states)[None,None,None,:]).astype('float64') 81 | 82 | cur_states = cur_Bs[:,:,0:nb_substeps+1].astype(int) #states of interest for the current displacement 83 | cur_nb_Bs = cur_Bs.shape[1] 84 | 85 | sub_Bs = get_all_Bs(nb_substeps, nb_states)[None] 86 | TrMat = cp.array(TrMat.T) 87 | 88 | # compute the vector of diffusion stds knowing the current states 89 | ds = cp.array(ds) 90 | ds2 = ds**2 91 | Fs = cp.array(Fs) 92 | 93 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 94 | #LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 95 | 96 | LP = LT # current log proba of seeing the track 97 | LP = cp.repeat(LP, nb_Tracks, axis = 0) 98 | 99 | cur_d2s = ds2[cur_states] 100 | cur_d2s = (cur_d2s[:,:,1:] + cur_d2s[:,:,:-1]) / 2 # assuming a transition at the middle of the substeps 101 | 102 | # we can average the variances of displacements per step to get the actual std of displacements per step 103 | cur_d2s = cp.mean(cur_d2s, axis = 2) 104 | cur_d2s = cur_d2s[:,:,None] 105 | cur_d2s = cp.array(cur_d2s) 106 | 107 | # inject the first position to get the associated Km and Ks : 108 | Km, Ks = first_log_integrale_dif(Cs[:,:, nb_locs - current_step], LocErr2[:,:,min(LocErr_index,nb_locs-current_step)], cur_d2s) 109 | 110 | if len(Ks)==1: 111 | Ks = cp.repeat(Ks, nb_Tracks, axis = 0) 112 | 113 | current_step += 1 114 | Km = cp.repeat(Km, cur_nb_Bs, axis = 1) 115 | removed_steps = 0 116 | 117 | all_Km.append(Km) 118 | all_Ks.append(Ks**0.5) 119 | all_LP.append(LP) 120 | all_cur_Bs.append(cur_Bs) 121 | 122 | while current_step <= nb_locs-1: 123 | # update cur_Bs to describe the states at the next step : 124 | for iii in range(nb_substeps): 125 | #cur_Bs = np.concatenate((np.repeat(np.mod(np.arange(cur_Bs.shape[1]*nb_states),nb_states)[None,:,None], nb_Tracks, 0), np.repeat(cur_Bs,nb_states,1)),-1) 126 | cur_Bs = np.concatenate((np.mod(np.arange(cur_Bs.shape[1]*nb_states),nb_states)[None,:,None], np.repeat(cur_Bs,nb_states,1)),-1) 127 | new_states = np.repeat(np.mod(np.arange(cur_Bs_cat.shape[1]*nb_states, dtype = 'int8'),nb_states)[None,:,None,None] == np.arange(nb_states, dtype = 'int8')[None,None,None], cur_Bs_cat.shape[0], 0).astype('int8') 128 | cur_Bs_cat = np.concatenate((new_states, np.repeat(cur_Bs_cat,nb_states,1)),-2) 129 | 130 | cur_states = cur_Bs[:1,:,0:nb_substeps+1].astype(int) 131 | # compute the vector of diffusion stds knowing the states at the current step 132 | 133 | cur_d2s = ds2[cur_states] 134 | cur_d2s = (cur_d2s[:,:,1:] + cur_d2s[:,:,:-1]) / 2 # assuming a transition at the middle of the substeps 135 | cur_d2s = cp.mean(cur_d2s, axis = 2) 136 | cur_d2s = cur_d2s[:,:,None] 137 | cur_d2s = cp.array(cur_d2s) 138 | 139 | LT = get_Ts_from_Bs(cur_states, TrMat) 140 | 141 | # repeat the previous matrix to account for the states variations due to the new position 142 | Km = cp.repeat(Km, nb_states**nb_substeps , axis = 1) 143 | Ks = cp.repeat(Ks, nb_states**nb_substeps, axis = 1) 144 | LP = cp.repeat(LP, nb_states**nb_substeps, axis = 1) 145 | # inject the next position to get the associated Km, Ks and Constant describing the integral of 3 normal laws : 146 | Km, Ks, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr2[:,:,min(LocErr_index,nb_locs-current_step)], cur_d2s, Km, Ks) 147 | 148 | #print('integral',time.time() - t0); t0 = time.time() 149 | LP += LT + LC # current (log) constants associated with each track and sequences of states 150 | del LT, LC 151 | cur_nb_Bs = len(cur_Bs[0]) # current number of sequences of states 152 | 153 | cur_nb_Bs = len(cur_Bs[0]) # current number of sequences of states 154 | 155 | if cur_nb_Bs > max_nb_states: 156 | threshold = threshold*1.2 157 | 158 | '''idea : the position and the state 6 steps ago should not impact too much the 159 | probability of the next position so the Km and Ks of tracks with the same 6 last 160 | states must be very similar, we can then fuse the parameters of the pairs of Bs 161 | which vary only for the last step (7) and sum their probas''' 162 | 163 | 164 | if current_step < nb_locs-1: # do not fuse sequences at the last step as it doesn't improves speed. 165 | Km, Ks, LP, cur_Bs, cur_Bs_cat = fuse_tracks_th(Km, 166 | Ks, 167 | LP, 168 | cur_Bs, 169 | cur_Bs_cat, 170 | nb_Tracks, 171 | nb_states = nb_states, 172 | nb_dims = nb_dims, 173 | do_preds = 1, 174 | threshold = threshold, 175 | frame_len = frame_len) # threshold on values normalized by sigma. 176 | 177 | cur_nb_Bs = len(cur_Bs[0]) 178 | #print(current_step, m_arr.shape) 179 | removed_steps += 1 180 | 181 | current_step += 1 182 | #print(current_step) 183 | all_Km.append(Km) 184 | all_Ks.append(Ks**0.5) 185 | all_LP.append(LP) 186 | all_cur_Bs.append(cur_Bs) 187 | 188 | newKs = cp.array((Ks + LocErr2[:,:, min(LocErr_index, nb_locs-current_step)])) 189 | log_integrated_term = cp.sum(-0.5*cp.log(2*np.pi*newKs) - (Cs[:,:,0] - Km)**2/(2*newKs),axis=2) 190 | LF = cp.log(Fs[cur_Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 191 | #LF = cp.log(0.5) 192 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 193 | LP += log_integrated_term + LF 194 | 195 | pred_LP = LP 196 | if cp.max(LP)>600: # avoid overflow of exponentials, mechanically also reduces the weight of longest tracks 197 | pred_LP = LP - (cp.max(LP)-600) 198 | 199 | P = np.exp(pred_LP) 200 | sum_P = np.sum(P, axis = 1, keepdims = True)[:,:,None] 201 | preds = np.sum(P[:,:,None,None]*cur_Bs_cat, axis = 1) / sum_P 202 | preds = preds[:,::-1] 203 | 204 | return LP, cur_Bs, all_cur_Bs, preds, all_Km, all_Ks, all_LP 205 | 206 | # LocErr = cur_LocErr 207 | def get_pos_PDF(Cs, LocErr, ds, Fs, TrMat, frame_len = 7, threshold = 0.2, max_nb_states = 1000): 208 | nb_substeps = 1 209 | ds = cp.array(ds) 210 | Cs = cp.array(Cs) 211 | 212 | LocErr.shape 213 | # get Km, Ks and LC forward 214 | LP1, final_Bs1, all_cur_Bs1, preds1, all_Km1, all_Ks1, all_LP1 = get_LC_Km_Ks(Cs, LocErr, ds, Fs, TrMat, nb_substeps, frame_len, threshold, max_nb_states) 215 | #get Km, Ks and LC backward 216 | TrMat2 = np.copy(TrMat).T # transpose the matrix for the backward transitions 217 | Cs2 = Cs[:,::-1,:] # inverse the time steps 218 | LP2, final_Bs2, all_cur_Bs2, preds2, all_Km2, all_Ks2, all_LP2 = get_LC_Km_Ks(Cs2, LocErr, ds, cp.ones(TrMat2.shape[0],)/TrMat2.shape[0], TrMat2, nb_substeps, frame_len, threshold, max_nb_states) # we set a neutral Fs so it doesn't get counted twice 219 | 220 | # do the approximation for the first position, product of 2 gaussian PDF, (integrated term and localization error) 221 | sig, mu, LC = prod_2GaussPDF(LocErr[:,None,0], all_Ks1[-1], Cs[:,None,0], all_Km1[-1]) 222 | 223 | LP = all_LP1[-1] + LC 224 | all_pos_means = [mu] 225 | all_pos_stds = [sig] 226 | all_pos_weights = [LP] 227 | all_pos_Bs = [final_Bs1] 228 | 229 | for k in range(1,Cs.shape[1]-1): 230 | ''' 231 | we take the corresponding Km1, Ks1, LP1, Km2, Ks2, LP2 232 | which are the corresponding stds and means of the resulting 233 | PDF surrounding the position k 234 | with localization uncertainty, we have 3 gaussians to compress to 1 gaussian * K 235 | This has to be done for all combinations of set of consective states before and after.step k 236 | to do so we set dim 1 as dim for consecutive states computed by the forward proba and 237 | dim 2 for sets of states computed by the backward proba. 238 | ''' 239 | LP1 = all_LP1[-1-k][:,:,None] 240 | Km1 = all_Km1[-1-k][:,:,None] 241 | Ks1 = all_Ks1[-1-k][:,:,None] 242 | cur_Bs1 = all_cur_Bs1[-1-k][0,:,0] 243 | 244 | LP2 = all_LP2[k-1][:,None] 245 | Km2 = all_Km2[k-1][:,None] 246 | Ks2 = all_Ks2[k-1][:,None] 247 | cur_Bs2 = all_cur_Bs2[k-1][0,:,0] 248 | 249 | nb_Bs1 = Ks1.shape[1] 250 | nb_Bs2 = Ks2.shape[2] 251 | nb_tracks = Km1.shape[0] 252 | nb_dims = Km1.shape[3] 253 | nb_states = TrMat.shape[0] 254 | 255 | mu = np.zeros((nb_tracks, 0, Km2.shape[-1])) 256 | sig = np.zeros((nb_tracks, 0, Ks2.shape[-1])) 257 | LP = np.zeros((nb_tracks, 0)) 258 | 259 | Bs2_len = np.min([k+1, frame_len-1]) 260 | # we must reorder the metrics so the Bs from the backward terms correspond to the forward terms 261 | for state in range(nb_states): 262 | 263 | sub_Ks1 = Ks1[:,np.where(cur_Bs1==state)[0]] 264 | sub_Ks2 = Ks2[:,:,np.where(cur_Bs2==state)[0]] 265 | sub_Km1 = Km1[:,np.where(cur_Bs1==state)[0]] 266 | sub_Km2 = Km2[:,:,np.where(cur_Bs2==state)[0]] 267 | sub_LP1 = LP1[:,np.where(cur_Bs1==state)[0]] 268 | sub_LP2 = LP2[:,:,np.where(cur_Bs2==state)[0]] 269 | 270 | if LocErr.shape[1]>1: 271 | cur_LocErr = LocErr[:,None,None,k] 272 | else: 273 | cur_LocErr = LocErr[:,None] 274 | 275 | sub_sig, sub_mu, sub_LC = prod_3GaussPDF(sub_Ks1, cur_LocErr, sub_Ks2, sub_Km1, Cs[:,None,None,k], sub_Km2) 276 | 277 | sub_LP = sub_LP1 + sub_LP2 + sub_LC 278 | 279 | sub_sig = sub_sig.reshape((nb_tracks,sub_sig.shape[1]*sub_sig.shape[2],1)) 280 | sub_mu = sub_mu.reshape((nb_tracks,sub_mu.shape[1]*sub_mu.shape[2], nb_dims)) 281 | sub_LP = sub_LP.reshape((nb_tracks,sub_LP.shape[1]*sub_LP.shape[2])) 282 | 283 | mu = np.concatenate((mu, sub_mu), axis = 1) 284 | sig = np.concatenate((sig, sub_sig), axis = 1) 285 | LP = np.concatenate((LP, sub_LP), axis = 1) 286 | 287 | all_pos_means.append(mu) 288 | all_pos_stds.append(sig) 289 | all_pos_weights.append(LP) 290 | 291 | sig, mu, LC = prod_2GaussPDF(LocErr[:,None,-1], all_Ks2[-1], Cs[:,None,-1], all_Km2[-1]) 292 | LP = all_LP2[-1] + LC 293 | 294 | all_pos_means.append(mu) 295 | all_pos_stds.append(sig) 296 | all_pos_weights.append(LP) 297 | 298 | return all_pos_means, all_pos_stds, all_pos_weights 299 | 300 | # all_tracks = tracks 301 | # LocErr = input_LocErr 302 | # LocErr = LocErr[0] 303 | 304 | def position_refinement(all_tracks, LocErr, ds, Fs, TrMat, frame_len = 7, threshold = 0.1, max_nb_states = 1000): 305 | if type(LocErr) == float or type(LocErr) == np.float64 or type(LocErr) == np.float32: 306 | LocErr = np.array([[[LocErr]]]) 307 | LocErr_type = 'array' 308 | cur_LocErr = LocErr 309 | 310 | elif type(LocErr) == np.ndarray: 311 | LocErr_type = 'array' 312 | if len(LocErr.shape) == 1: 313 | LocErr = LocErr[None, None] 314 | if len(LocErr.shape) == 2: 315 | LocErr = LocErr[None] 316 | cur_LocErr = LocErr 317 | 318 | elif type(LocErr) == dict: 319 | LocErr_type = 'dict' 320 | else: 321 | LocErr_type = 'other' 322 | print('LocErr_type', LocErr_type) 323 | 324 | all_mus = {} 325 | all_sigmas = {} 326 | for l in all_tracks.keys(): 327 | Cs = all_tracks[l] 328 | if LocErr_type == 'dict': 329 | cur_LocErr = LocErr[l] 330 | all_mus[l] = np.zeros((Cs.shape[0], int(l), Cs.shape[2])) 331 | all_sigmas[l] = np.zeros((Cs.shape[0], int(l))) 332 | all_pos_means, all_pos_stds, all_pos_weights = get_pos_PDF(Cs, cur_LocErr, ds, Fs, TrMat, frame_len, threshold, max_nb_states) 333 | #best_mus, best_sigs, best_Bs = get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds) 334 | for k, (pos_means, pos_stds, pos_weights) in enumerate(zip(all_pos_means, all_pos_stds, all_pos_weights)): 335 | P = np.exp(pos_weights - np.max(pos_weights, 1, keepdims = True)) 336 | all_mus[l][:, k] = np.sum(P[:,:,None]*pos_means, 1) / np.sum(P, 1)[:,None] 337 | all_sigmas[l][:, k] = (np.sum(P[:,:]*pos_stds[:,:,0]**2, 1) / np.sum(P, 1))**0.5 338 | return all_mus, all_sigmas 339 | 340 | def get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds): 341 | nb_Bs = [] 342 | nb_pos = len(all_pos_weights) 343 | for weights in all_pos_weights: 344 | nb_Bs.append(weights.shape[1]) 345 | nb_Bs = np.max(nb_Bs) 346 | nb_states = (np.max(all_pos_Bs[0])+1).astype(int) 347 | max_frame_len = (np.log(nb_Bs) // np.log(nb_states)).astype(int) 348 | mid_frame_pos = ((max_frame_len-0.1)//2).astype(int) # -0.1 is used for mid_frame_pos to be the good index for both odd and pair numbers 349 | mid_frame_pos = np.max([1,mid_frame_pos]) 350 | best_Bs = [] 351 | best_mus = [] 352 | best_sigs = [] 353 | for k, (weights, Bs, mus, sigs) in enumerate(zip(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds)): 354 | if k <= nb_pos/2 : 355 | idx = np.min([k, mid_frame_pos]) 356 | else: 357 | idx = np.max([-mid_frame_pos-1, k - nb_pos]) 358 | best_args = np.argmax(weights, 1) 359 | best_Bs.append(Bs[0,best_args][:,idx]) 360 | best_sigs.append(sigs[[cp.arange(len(mus)), best_args]]) 361 | best_mus.append(mus[[cp.arange(len(mus)), best_args]]) 362 | best_Bs = cp.array(best_Bs).T.astype(int) 363 | best_sigs = cp.transpose(cp.array(best_sigs), (1,0,2)) 364 | best_mus = cp.transpose(cp.array(best_mus), (1,0,2)) 365 | return asnumpy(best_mus), asnumpy(best_sigs), asnumpy(best_Bs) 366 | 367 | def save_gifs(Cs, all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs, gif_pathnames = './tracks', lim = None, nb_pix = 200, fps=1): 368 | try: 369 | plt 370 | imageio 371 | except: 372 | raise ImportError('matplotlib and imageio has to be installed to use save_gifs') 373 | best_mus, best_sigs, best_Bs = get_all_estimates(all_pos_weights, all_pos_Bs, all_pos_means, all_pos_stds) 374 | for ID in range(len(Cs)): 375 | all_images = [] 376 | Cs_offset = np.mean(Cs[ID], 0) 377 | Cs[ID] = Cs[ID] 378 | 379 | if lim == None: 380 | cur_lim = np.max(np.abs(Cs[ID]))*1.1 381 | else: 382 | cur_lim = lim 383 | pix_size = nb_pix / (2*cur_lim) 384 | for k in range(len(all_pos_means)): 385 | 386 | sig = asnumpy(all_pos_stds[k]) 387 | mu = asnumpy(all_pos_means[k]) 388 | LP = asnumpy(all_pos_weights[k]) 389 | 390 | fig = plt.figure() 391 | plt.plot((Cs[ID, :,1] - Cs_offset[1] + cur_lim)*pix_size-0.5, (Cs[ID, :,0]- Cs_offset[0]+cur_lim)*pix_size-0.5) 392 | plt.scatter((best_mus[ID, :,1] - Cs_offset[1] + cur_lim)*pix_size-0.5, (best_mus[ID, :,0] - Cs_offset[0]+cur_lim)*pix_size-0.5, c='r', s=3) 393 | 394 | P_xs = gaussian(np.linspace(-cur_lim,cur_lim,nb_pix)[None,:,None], sig[ID][:,:,None], mu[ID][:,:1,None] - Cs_offset[0]) * np.exp(LP[ID]-np.max(LP[ID]))[:,None] 395 | P_ys = gaussian(np.linspace(-cur_lim,cur_lim,nb_pix)[None,:,None], sig[ID][:,:,None], mu[ID][:,1:,None] - Cs_offset[1]) * np.exp(LP[ID]-np.max(LP[ID]))[:,None] 396 | 397 | heatmap = np.sum(P_xs[:,:,None]*P_ys[:,None] * np.exp(LP[ID]-np.max(LP[ID]))[:,None,None],0) 398 | 399 | heatmap = heatmap/np.max(heatmap) 400 | plt.imshow(heatmap) 401 | plt.xticks(np.linspace(0,nb_pix-1, 5), np.round(np.linspace(-cur_lim,cur_lim, 5), 2)) 402 | plt.yticks(np.linspace(0,nb_pix-1, 5), np.round(np.linspace(-cur_lim,cur_lim, 5), 2)) 403 | canvas = FigureCanvas(fig) 404 | canvas.draw() 405 | s, (width, height) = canvas.print_to_buffer() 406 | image = np.fromstring(s, dtype='uint8').reshape((height, width, 4)) 407 | 408 | all_images.append(image) 409 | plt.close() 410 | 411 | imageio.mimsave(gif_pathnames + str(ID)+'.gif', all_images,fps=fps) 412 | 413 | 414 | def get_LC_Km_Ks_fixed_Bs(Cs, LocErr, ds, Fs, TrMat, Bs): 415 | ''' 416 | variation of the main function to extract LC, Km and Ks for all positions 417 | ''' 418 | nb_Tracks = len(Cs) 419 | nb_locs = len(Cs[0]) # number of localization per track 420 | nb_dims = len(Cs[0,0]) # number of spatial dimentions (x, y) or (x, y, z) 421 | Cs = Cs.reshape((nb_Tracks,1,nb_locs, nb_dims)) 422 | Cs = cp.array(Cs) 423 | 424 | all_Km = [] 425 | all_Ks = [] 426 | all_LP = [] 427 | 428 | TrMat = cp.array(TrMat) # transition matrix of the markovian process 429 | current_step = 1 430 | 431 | cur_states = Bs[:,:,-2:].astype(int) #states of interest for the current displacement 432 | 433 | # compute the vector of diffusion stds knowing the current states 434 | ds = cp.array(ds) 435 | Fs = cp.array(Fs) 436 | 437 | LT = get_Ts_from_Bs(cur_states, TrMat) # Log proba of transitions per step 438 | LF = cp.log(Fs[cur_states[:,:,-1]]) # Log proba of finishing/starting in a given state (fractions) 439 | 440 | LP = LT # current log proba of seeing the track 441 | LP = cp.repeat(LP, nb_Tracks, axis = 0) 442 | 443 | cur_ds = ds_froms_states(ds, cur_states) 444 | 445 | # inject the first position to get the associated Km and Ks : 446 | Km, Ks = first_log_integrale_dif(Cs[:,:, nb_locs-current_step], LocErr, cur_ds) 447 | all_Km.append(Km) 448 | all_Ks.append(Ks) 449 | all_LP.append(LP) 450 | current_step += 1 451 | 452 | while current_step <= nb_locs-1: 453 | # update cur_Bs to describe the states at the next step : 454 | cur_states = Bs[:,:,-current_step-1:-current_step+1].astype(int) 455 | # compute the vector of diffusion stds knowing the states at the current step 456 | cur_ds = ds_froms_states(ds, cur_states) 457 | LT = get_Ts_from_Bs(cur_states, TrMat) 458 | 459 | # inject the next position to get the associated Km, Ks and Constant describing the integral of 3 normal laws : 460 | Km, Ks, LC = log_integrale_dif(Cs[:,:,nb_locs-current_step], LocErr, cur_ds, Km, Ks) 461 | #print('integral',time.time() - t0); t0 = time.time() 462 | LP += LT + LC # current (log) constants associated with each track and sequences of states 463 | del LT, LC 464 | 465 | all_Km.append(Km) 466 | all_Ks.append(Ks) 467 | all_LP.append(LP) 468 | 469 | current_step += 1 470 | 471 | newKs = cp.array((Ks**2 + LocErr**2)**0.5)[:,:,0] 472 | log_integrated_term = -cp.log(2*np.pi*newKs**2) - cp.sum((Cs[:,:,0] - Km)**2,axis=2)/(2*newKs**2) 473 | LF = cp.log(Fs[Bs[:,:,0].astype(int)]) # Log proba of starting in a given state (fractions) 474 | #LF = cp.log(0.5) 475 | # cp.mean(cp.log(Fs[cur_Bs[:,:,:].astype(int)]), 2) # Log proba of starting in a given state (fractions) 476 | LP += log_integrated_term + LF 477 | 478 | all_Ks = cp.array(all_Ks)[:,0,0] 479 | all_Km = cp.array(all_Km)[:,0,0] 480 | all_LP = cp.array(all_LP)[:,0,0] 481 | return all_Km, all_Ks, all_LP 482 | 483 | def get_pos_PDF_fixedBs(Cs, LocErr, ds, Fs, TrMat, Bs): 484 | ''' 485 | get mu and sigma for each position given inputed Bs, 486 | ideally used for a single track with its most likely set of states 487 | ''' 488 | ds = np.array(ds) 489 | Cs = cp.array(Cs) 490 | # get Km, Ks and LC forward 491 | all_Km1, all_Ks1, all_LP1 = get_LC_Km_Ks_fixed_Bs(Cs, LocErr, ds, Fs, TrMat, Bs) 492 | #get Km, Ks and LC backward 493 | TrMat2 = np.copy(TrMat).T # transpose the matrix for the backward transitions 494 | Cs2 = Cs[:,::-1,:] # inverse the time steps 495 | all_Km2, all_Ks2, all_LP2 = get_LC_Km_Ks_fixed_Bs(Cs2, LocErr, ds, cp.ones(TrMat2.shape[0],)/TrMat2.shape[0], TrMat2, Bs[:,:,::-1]) 496 | # do the approximation for the first position, product of 2 gaussian PDF, (integrated term and localization error) 497 | 498 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks1[-1], Cs[:,0], all_Km1[-1]) 499 | np 500 | all_pos_means = [mu] 501 | all_pos_stds = [sig[None]] 502 | 503 | for k in range(1,Cs.shape[1]-1): 504 | 505 | Km1 = all_Km1[-k][None] 506 | Ks1 = all_Ks1[-1-k][None] 507 | Km2 = all_Km2[k-1][None] 508 | Ks2 = all_Ks2[k-1][None] 509 | 510 | sig, mu, LC = prod_3GaussPDF(Ks1,LocErr,Ks2, Km1, Cs[:,k], Km2) 511 | 512 | all_pos_means.append(mu) 513 | all_pos_stds.append(sig) 514 | 515 | sig, mu, LC = prod_2GaussPDF(LocErr,all_Ks2[-1], Cs[:,-1], all_Km2[-1]) 516 | 517 | all_pos_means.append(mu) 518 | all_pos_stds.append(sig[None]) 519 | return cp.array(all_pos_means)[:,0], cp.array(all_pos_stds)[:,0] 520 | 521 | def get_global_sigs_mus(all_pos_means, all_pos_stds, all_pos_weights, idx = 0): 522 | w_sigs = [] 523 | w_mus = [] 524 | for mus, sigs, LC in zip(all_pos_means, all_pos_stds, all_pos_weights): 525 | mus = mus[idx] 526 | sigs = sigs[idx] 527 | LC = LC[idx] 528 | LC = LC - np.max(LC, keepdims = True) 529 | #sigs = sigs[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 530 | #mus = mus[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 531 | #LC = LC[LC > np.max(LC) + np.log(1e-5)] # remove the unlikely set of states 532 | w_sigs.append(np.sum(np.exp(LC[:,None])**2 * sigs) / np.sum(np.exp(LC[:,None])**2)) 533 | w_mus.append(np.sum(np.exp(LC[:,None]) * mus,0) / np.sum(np.exp(LC[:,None]),0)) 534 | return np.array(w_mus), np.array(w_sigs) 535 | 536 | def full_extrack_2_matrix(all_tracks, params, dt, all_frames = None, cell_dims = [1,None,None], nb_states = 2, frame_len = 15): 537 | nb_dims = list(all_tracks.items())[0][1].shape[2] 538 | pred_Bss = predict_Bs(all_tracks, dt, params, nb_states=nb_states, frame_len=frame_len, cell_dims = cell_dims) 539 | 540 | DATA = extrack_2_matrix(all_tracks, pred_Bss, dt, all_frames = all_frames) 541 | DATA = np.concatenate((DATA, np.empty((DATA.shape[0], nb_dims+1))),1) 542 | LocErr, ds, Fs, TrMat = extract_params(params, dt, nb_states, nb_substeps = 1) 543 | for ID in np.unique(DATA[:,2]): 544 | track = DATA[DATA[:,2]==ID,:nb_dims][None] 545 | all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs = get_pos_PDF(track, LocErr, ds, Fs, TrMat, frame_len = frame_len//2+3) 546 | w_mus, w_sigs = get_global_sigs_mus(all_pos_means, all_pos_stds, all_pos_weights, idx = 0) 547 | DATA[DATA[:,2]==ID,-1] = w_sigs 548 | DATA[DATA[:,2]==ID,-nb_dims-1:-1] = w_mus 549 | return DATA 550 | 551 | def get_best_estimates(Cs, LocErr, ds, Fs, TrMat, frame_len = 10): 552 | all_mus = [] 553 | all_sigs = [] 554 | for track in Cs: 555 | a,b, preds = P_Cs_inter_bound_stats(track[None], LocErr, ds, Fs, TrMat, nb_substeps=1, do_frame = 1, frame_len = frame_len, do_preds = 1) 556 | Bs = np.argmax(preds, 2)[None] 557 | mus, sigs = get_pos_PDF_fixedBs(Cs, LocErr, ds, Fs, TrMat, Bs) 558 | all_mus.append(mus) 559 | all_sigs.append(sigs) 560 | return mus, sigs 561 | 562 | def do_gifs_from_params(all_tracks, params, dt, gif_pathnames = './tracks', frame_len = 9, nb_states = 2, nb_pix = 200, fps = 1): 563 | for Cs in all_tracks: 564 | LocErr, ds, Fs, TrMat = extract_params(params, dt, nb_states, nb_substeps = 1) 565 | all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs = get_pos_PDF(Cs, LocErr, ds, Fs, TrMat, frame_len = frame_len) 566 | save_gifs(Cs, all_pos_means, all_pos_stds, all_pos_weights, all_pos_Bs, gif_pathnames = gif_pathnames + '_' + str(len(Cs[0])) + '_pos', lim = None, nb_pix = nb_pix, fps=fps) 567 | -------------------------------------------------------------------------------- /extrack/simulate_tracks.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Fri Jan 8 14:16:58 2021 5 | 6 | @author: mcm 7 | """ 8 | 9 | import numpy as np 10 | 11 | def markovian_process(TrMat, initial_fractions, nb_tracks, track_len): 12 | nb_states = len(TrMat) 13 | cumMat = np.cumsum(TrMat, 1) 14 | cumF = np.cumsum(initial_fractions) 15 | states = np.zeros((nb_tracks, track_len)).astype(int) 16 | randoms = np.random.rand(nb_tracks, track_len) 17 | for s in range(nb_states -1): 18 | states[:,0] += (randoms[:,0]>cumF[s]).astype(int) 19 | for k in range(1,track_len): 20 | for s in range(nb_states -1): 21 | states[:,k] += (randoms[:,k] > cumMat[states[:,k-1]][:,s]).astype(int) 22 | return states 23 | 24 | def get_fractions_from_TrMat(TrMat): 25 | if len(TrMat) == 2: 26 | p01 = TrMat[0,1] 27 | p10 = TrMat[1,0] 28 | initial_fractions = np.array([p10 / (p10+p01), p01 / (p10+p01)]) 29 | elif len(TrMat) == 3: 30 | p01 = TrMat[0,1] 31 | p02 = TrMat[0,2] 32 | p10 = TrMat[1,0] 33 | p12 = TrMat[1,2] 34 | p20 = TrMat[2,0] 35 | p21 = TrMat[2,1] 36 | F0 = (p10*(p21+p20)+p20*p12)/((p01)*(p12 + p21) + p02*(p10 + p12 + p21) + p01*p20 + p21*p10 + p20*(p10+p12)) 37 | F1 = (F0*p01 + (1-F0)*p21)/(p10 + p12 + p21) 38 | initial_fractions = np.array([F0, F1, 1-F0-F1]) 39 | else: 40 | ''' 41 | if more states lets just run the transition process until equilibrium 42 | ''' 43 | A0 = np.ones(len(TrMat))/len(TrMat) 44 | k = 0 45 | prev_A = A0 46 | A = np.dot(A0, TrMat) 47 | while not np.all(prev_A == A): 48 | k += 1 49 | prev_A = A 50 | A = np.dot(A, TrMat) 51 | if k > 10000000: 52 | raise Exception("We could't find a steady state, convergence didn't occur within %s steps, \ncurrent fractions : %s"%(k, A)) 53 | initial_fractions = A 54 | return initial_fractions 55 | 56 | def sim_noBias(track_lengths = [7,8,9,10,11], # create arrays of tracks of specified number of localizations 57 | track_nb_dist = [1000, 800, 700, 600, 550], # list of number of tracks per array 58 | LocErr = 0.02, # Localization error in um 59 | Ds = [0, 0.05], # diffusion coef of each state in um^2/s 60 | TrMat = np.array([[0.9,0.1], [0.2,0.8]]), # transition matrix e.g. np.array([[1-p01,p01], [p10,1-p10]]) 61 | initial_fractions = None, # fraction in each states at the beginning of the tracks, if None assums steady state determined by TrMat 62 | dt = 0.02, # time in between positions in s 63 | nb_dims = 2 # number of spatial dimensions : 2 for (x,y) and 3 for (x,y,z) 64 | ): 65 | ''' 66 | create tracks with specified kinetics 67 | outputs the tracks and the actual hidden states using the ExTrack format (dict of np.arrays) 68 | any states number 'n' can be producted as long as len(Ds) = len(initial_fractions) and TrMat.shape = (n,n) 69 | ''' 70 | 71 | Ds = np.array(Ds) 72 | TrMat = np.array(TrMat) 73 | nb_sub_steps = 30 74 | 75 | if initial_fractions is None: 76 | initial_fractions = get_fractions_from_TrMat(TrMat) 77 | 78 | nb_states = len(TrMat) 79 | sub_dt = dt/nb_sub_steps 80 | 81 | TrSubMat = TrMat / nb_sub_steps 82 | TrSubMat[np.arange(nb_states),np.arange(nb_states)] = 0 83 | TrSubMat[np.arange(nb_states),np.arange(nb_states)] = 1 - np.sum(TrSubMat,1) 84 | 85 | all_Css = [] 86 | all_Bss = [] 87 | 88 | for nb_tracks, track_len in zip(track_nb_dist, track_lengths): 89 | print(nb_tracks, track_len) 90 | 91 | states = markovian_process(TrSubMat, 92 | initial_fractions = initial_fractions, 93 | track_len =(track_len-1) * nb_sub_steps + 1, 94 | nb_tracks = nb_tracks) 95 | # determines displacements then MSDs from stats 96 | positions = np.random.normal(0, 1, (nb_tracks, (track_len-1) * nb_sub_steps+1, nb_dims)) * np.sqrt(2*Ds*sub_dt)[states[:,:, None]] 97 | positions = np.cumsum(positions, 1) 98 | 99 | positions = positions + np.random.normal(0, LocErr, (nb_tracks, (track_len-1) * nb_sub_steps + 1, nb_dims)) 100 | 101 | all_Css.append(positions[:,np.arange(0,(track_len-1) * nb_sub_steps +1, nb_sub_steps)]) 102 | all_Bss.append(states[:,np.arange(0,(track_len-1) * nb_sub_steps +1, nb_sub_steps)]) 103 | 104 | all_Css_dict = {} 105 | all_Bss_dict = {} 106 | for Cs, Bs in zip(all_Css, all_Bss): 107 | l = str(Cs.shape[1]) 108 | all_Css_dict[l] = Cs 109 | all_Bss_dict[l] = Bs 110 | 111 | return all_Css_dict, all_Bss_dict 112 | 113 | def is_in_FOV(positions, cell_dims): 114 | inFOV = np.ones((len(positions)+1,)) == 1 115 | for i, l in enumerate(cell_dims): 116 | if not l == None: 117 | cur_inFOV = (positions[:,i] < l) * (positions[:,i] > 0) 118 | cur_inFOV = np.concatenate((cur_inFOV,[False])) 119 | inFOV = inFOV*cur_inFOV 120 | return inFOV 121 | 122 | 123 | def sim_FOV(nb_tracks=10000, 124 | max_track_len=40, 125 | min_track_len = 2, 126 | LocErr=0.02, # localization error in x, y and z (even if not used) 127 | Ds = np.array([0,0.05]), 128 | nb_dims = 2, 129 | initial_fractions = np.array([0.6,0.4]), 130 | TrMat = np.array([[0.9,0.1],[0.1,0.9]]), 131 | LocErr_std = 0, 132 | dt = 0.02, 133 | pBL = 0.1, 134 | cell_dims = [0.5,None,None]): # dimension limits in x, y and z respectively 135 | ''' 136 | simulate tracks able to come and leave from the field of view : 137 | 138 | nb_tracks: number of tracks simulated. 139 | max_track_len: number of steps simulated per track. 140 | LocErr: standard deviation of the localization error. 141 | Ds: 1D array of the diffusion coefs for each state. 142 | TrMat: transition array per step (lines: state at time n, cols: states at time n+1). 143 | dt: time in between frames. 144 | pBL: probability of bleaching per step. 145 | cell_dims: dimension limits in x, y and z respectively. x, y dimension limits are useful when tracking membrane proteins in tirf when the particles leave the field of view from the sides of the cells. z dimension is relevant for cytoplasmic proteins which call leave from the z axis. Consider the particle can leave from both ends of each axis: multiply axis limit by 2 to aproximate tracks leaving from one end. 146 | min_track_len: minimal track length for the track to be considered. 147 | 148 | outputs: 149 | all_tracks: dict describing the tracks with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = x, y position. 150 | all_Bs: dict descibing the true states of tracks with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = x, y position. 151 | ''' 152 | 153 | if type(LocErr) != np.ndarray: 154 | LocErr = np.array(LocErr) 155 | if LocErr.shape == (): 156 | LocErr = LocErr[None] 157 | 158 | nb_sub_steps = 20 159 | nb_strobo_frames = 1 160 | nb_states = len(TrMat) 161 | sub_dt = dt/nb_sub_steps 162 | cell_dims = np.array(cell_dims) 163 | 164 | TrSubMat = TrMat / nb_sub_steps 165 | TrSubMat[np.arange(nb_states),np.arange(nb_states)] = 0 166 | TrSubMat[np.arange(nb_states),np.arange(nb_states)] = 1 - np.sum(TrSubMat,1) 167 | 168 | all_Css = [[]]*(max_track_len - min_track_len + 1) 169 | all_Bss = [[]]*(max_track_len - min_track_len + 1) 170 | all_sigs = [[]]*(max_track_len - min_track_len + 1) 171 | 172 | nb_tracks = 2**np.sum(cell_dims!=None)*nb_tracks 173 | states = markovian_process(TrSubMat, initial_fractions, nb_tracks, (max_track_len) * nb_sub_steps) 174 | cell_dims0 = np.copy(cell_dims) 175 | cell_dims0[cell_dims0==None] = 1 176 | 177 | for state in states: 178 | 179 | cur_track_len = max_track_len 180 | positions = np.zeros(((max_track_len) * nb_sub_steps, 3)) 181 | positions[0,:] = 2*np.random.rand(3)*cell_dims0-cell_dims0 182 | positions[1:] = np.random.normal(0, 1, ((max_track_len) * nb_sub_steps - 1, 3))* np.sqrt(2*Ds*sub_dt)[state[:-1, None]] 183 | state = state[np.arange(0,(max_track_len-1) * nb_sub_steps +1, nb_sub_steps)] 184 | positions = np.cumsum(positions, 0) 185 | positions = positions.reshape((max_track_len, nb_sub_steps, 3)) 186 | positions = positions[:,:nb_strobo_frames] 187 | positions = np.mean(positions,axis = 1) 188 | 189 | inFOV = is_in_FOV(positions, cell_dims) 190 | inFOV = np.concatenate((inFOV,[False])) 191 | while np.any(inFOV): 192 | if inFOV[0] == False: 193 | positions = positions[np.argmax(inFOV):] 194 | state = state[np.argmax(inFOV):] 195 | inFOV = inFOV[np.argmax(inFOV):] 196 | 197 | cur_sub_track = positions[:np.argmin(inFOV)] 198 | cur_sub_state = state[:np.argmin(inFOV)] 199 | 200 | pBLs = np.random.rand((len(cur_sub_track))) 201 | pBLs = pBLs < pBL 202 | if not np.all(pBLs==0): 203 | cur_sub_track = cur_sub_track[:np.argmax(pBLs)+1] 204 | cur_sub_state = cur_sub_state[:np.argmax(pBLs)+1] 205 | inFOV = np.array([False]) 206 | 207 | k = 2 / (LocErr_std**2+1e-20) # compute the parameter of the chi2 distribution with results in the specified standard deviation 208 | cur_sub_sigmas = np.random.chisquare(k, (len(cur_sub_track), 3)) * LocErr[None] / k 209 | cur_sub_errors = np.random.normal(0, cur_sub_sigmas, (len(cur_sub_track), 3)) 210 | cur_real_pos = np.copy(cur_sub_track) 211 | cur_sub_track = cur_sub_track + cur_sub_errors 212 | #cur_sub_track = cur_sub_track + np.random.normal(0, std_spurious, (len(cur_sub_track), 3)) * (np.random.rand(len(cur_sub_track),1)0: 231 | print(str(all_Css[k].shape[1])+' pos :',str(len(all_Css[k]))+', ', end = '') 232 | print('') 233 | 234 | all_Css_dict = {} 235 | all_Bss_dict = {} 236 | all_sigs_dict = {} 237 | for Cs, Bs, sigs in zip(all_Css, all_Bss, all_sigs): 238 | if Cs.shape[0] > 0: 239 | l = str(Cs.shape[1]) 240 | all_Css_dict[l] = Cs 241 | all_Bss_dict[l] = Bs 242 | all_sigs_dict[l] = sigs 243 | 244 | return all_Css_dict, all_Bss_dict, all_sigs_dict 245 | -------------------------------------------------------------------------------- /extrack/version.py: -------------------------------------------------------------------------------- 1 | __version__ = '1.6.1' 2 | -------------------------------------------------------------------------------- /extrack/visualization.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from matplotlib import pyplot as plt 3 | from extrack.histograms import len_hist 4 | from matplotlib import cm 5 | 6 | def visualize_states_durations(all_tracks, 7 | params, 8 | dt, 9 | cell_dims = [1,None,None], 10 | nb_states = 2, 11 | max_nb_states = 500, 12 | workers = 1, 13 | long_tracks = True, 14 | nb_steps_lim = 20, 15 | steps = False, 16 | input_LocErr = None): 17 | ''' 18 | arguments: 19 | all_tracks: dict describing the tracks with track length as keys (number of time positions, e.g. '23') of 3D arrays: dim 0 = track, dim 1 = time position, dim 2 = x, y position. 20 | params: lmfit parameters used for the model. 21 | dt: time in between frames. 22 | cell_dims: dimension limits (um). estimated_vals, min_values, max_values should be changed accordingly to describe all states and transitions. 23 | max_nb_states: maximum number of sequences kept (most likely sequences). 24 | nb_steps_lim: upper limit of the plot in the x axis (number of steps) 25 | long_tracks: if True only selects tracks longer than nb_steps_lim 26 | steps: x axis in seconds if False or in number of steps if False. 27 | 28 | outputs: 29 | plot of all tracks (preferencially input a single movie) 30 | ''' 31 | len_hists = len_hist(all_tracks, 32 | params, 33 | dt, 34 | cell_dims=cell_dims, 35 | nb_states=nb_states, 36 | workers = workers, 37 | nb_substeps=1, 38 | max_nb_states = max_nb_states, 39 | input_LocErr = input_LocErr) 40 | 41 | if steps: 42 | step_type = 'step' 43 | dt = 1 44 | else: 45 | step_type = 's' 46 | 47 | plt.figure(figsize = (3,3)) 48 | for k, hist in enumerate(len_hists.T): 49 | plt.plot(np.arange(1,len(hist)+1)*dt, hist/np.sum(hist), label='state %s'%k) 50 | 51 | plt.legend() 52 | plt.yscale('log') 53 | plt.grid() 54 | plt.xlim([0,nb_steps_lim*dt]) 55 | plt.ylim([0.001,0.5]) 56 | plt.xlabel('state duration (%s)'%(step_type)) 57 | plt.ylabel('fraction') 58 | plt.tight_layout() 59 | return len_hists 60 | 61 | def visualize_tracks(DATA, 62 | track_length_range = [10,np.inf], 63 | figsize = (5,5)): 64 | ''' 65 | DATA: dataframe outputed by extrack.exporters.extrack_2_pandas 66 | track_length_range: range of tracks ploted. plotting too many tracks may make it crash 67 | figsize: size of the figure plotted 68 | ''' 69 | nb_states = 0 70 | for param in list(DATA.keys()): 71 | if param.find('pred')+1: 72 | nb_states += 1 73 | 74 | plt.figure(figsize = figsize) 75 | DATA['X'] 76 | for ID in np.unique(DATA['track_ID'])[::-1]: 77 | if np.mod(ID, 20)==0: 78 | print('.', end = '') 79 | #print(ID) 80 | track = DATA[DATA['track_ID'] ==ID ] 81 | if track_length_range[0] < len(track) > track_length_range[0]: 82 | if nb_states == 2 : 83 | pred = track['pred_1'] 84 | pred = cm.brg(pred*0.5) 85 | else: 86 | pred = track[['pred_2', 'pred_1', 'pred_0']].values 87 | 88 | plt.plot(track['X'], track['Y'], 'k:', alpha = 0.2) 89 | plt.scatter(track['X'], track['Y'], c = pred, s=3) 90 | plt.gca().set_aspect('equal', adjustable='datalim') 91 | #plt.scatter(track['X'], track['X'], marker = 'x', c='k', s=5, alpha = 0.5) 92 | 93 | def plot_tracks(DATA, 94 | max_track_length = 50, 95 | nb_subplots = [5,5], 96 | figsize = (10,10), 97 | lim = 0.4 ): 98 | '''' 99 | DATA: dataframe outputed by extrack.exporters.extrack_2_pandas. 100 | max_track_length: maximum track length to be outputed, it will plot the longest tracks respecting this criteria. 101 | nb_subplots: number of lines and columns of subplots. 102 | figsize: size of the figure plotted 103 | ''' 104 | nb_states = 0 105 | for param in list(DATA.keys()): 106 | if param.find('pred')+1: 107 | nb_states += 1 108 | 109 | plt.figure(figsize=figsize) 110 | 111 | for ID in np.unique(DATA['track_ID'])[::-1]: 112 | track = DATA[DATA['track_ID'] ==ID] 113 | if len(track) > max_track_length: 114 | DATA.drop((DATA[DATA['track_ID'] == ID]).index, inplace=True) 115 | 116 | for k, ID in enumerate(np.unique(DATA['track_ID'])[::-1][:np.product(nb_subplots)]): 117 | plt.subplot(nb_subplots[0], nb_subplots[1], k+1) 118 | 119 | track = DATA[DATA['track_ID'] ==ID ] 120 | if nb_states == 2 : 121 | pred = track['pred_1'] 122 | pred = cm.brg(pred*0.5) 123 | else: 124 | pred = track[['pred_2', 'pred_1', 'pred_0']].values 125 | 126 | plt.plot(track['X'], track['Y'], 'k:', alpha = 0.2) 127 | plt.scatter(track['X'], track['Y'], c = pred, s=3) 128 | plt.xlim([np.mean(track['X']) - lim, np.mean(track['X']) + lim]) 129 | plt.ylim([np.mean(track['Y']) - lim, np.mean(track['Y']) + lim]) 130 | plt.gca().set_aspect('equal', adjustable='box') 131 | plt.yticks(fontsize = 6) 132 | plt.xticks(fontsize = 6) 133 | print('') 134 | plt.tight_layout(h_pad = 1, w_pad = 1) 135 | 136 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "setuptools>=42", 4 | "wheel" 5 | ] 6 | build-backend = "setuptools.build_meta" 7 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | lmfit>=0.9.7 2 | xmltodict 3 | matplotlib 4 | pandas 5 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.md", "r", encoding="utf-8") as fh: 4 | long_description = fh.read() 5 | 6 | exec(open("extrack/version.py").read()) 7 | 8 | setuptools.setup( 9 | name="extrack", 10 | version=__version__, 11 | author="Francois Simon", 12 | author_email="simon.francois@protonmail.com", 13 | description="SPT kinetic modelling and states annotation of tracks", 14 | long_description=long_description, 15 | long_description_content_type="text/markdown", 16 | url="https://github.com/FrancoisSimon/ExTrack-python3", 17 | project_urls={ 18 | "Bug Tracker": "https://github.com/FrancoisSimon/ExTrack-python3", 19 | }, 20 | classifiers=[ 21 | "Programming Language :: Python :: 3", 22 | "License :: OSI Approved :: MIT License", 23 | "Operating System :: OS Independent", 24 | "Intended Audience :: Science/Research", 25 | "Development Status :: 3 - Alpha" 26 | ], 27 | package_dir={"": "."}, 28 | packages=setuptools.find_packages(where="."), 29 | install_requires=['lmfit', 'xmltodict', 'pandas', 'matplotlib'], 30 | python_requires=">=3.6", 31 | ) 32 | -------------------------------------------------------------------------------- /todo.txt: -------------------------------------------------------------------------------- 1 | repare histogram module 2 | parallelization for the state prediction module 3 | --------------------------------------------------------------------------------