├── AngleCam_val.png ├── CITATION.cff ├── LICENSE ├── README.md ├── code_manuscript ├── 00_convert_brinno_avi_to_jpeg_timeseriesprediction.R.R ├── 00_convert_brinno_avi_to_jpeg_training.R ├── 01_labelling_leaf_angles.R ├── 03_fit_beta_distributions_to_leaf_angles.R ├── 08_train_AngleCam.R ├── 09_TLSLeAF_output_to_single_plants.R ├── 10_predict_with_AngleCam_on_TLS_reference.R ├── 11_compare_TLSLeAF_AngleCam.R ├── 12_predict_with_AngleCam_on_timeseries.R ├── 14_evaluate_AngleCam_test_train.R └── 15_evaluate_AngleCam_timeseries.R ├── code_run_AngleCam ├── cnn_reqs.R ├── run_AngleCam.R └── run_AngleCam.py ├── illustrations_small.png ├── result_small_mod.gif └── tlsleaf_anglecam_comparison.png /AngleCam_val.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tejakattenborn/AngleCAM/73743fc56638ea4e9323367ff85f6b3e7dcb30aa/AngleCam_val.png -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | message: "If you use this software, please cite it as below." 3 | authors: 4 | - family-names: "Kattenborn" 5 | given-names: "Teja" 6 | orcid: "https://orcid.org/0000-0001-7381-3828" 7 | title: "AngleCam (vers. 2023-06-05)" 8 | version: 0.4 9 | doi: 10.5281/zenodo.8008012 10 | date-released: 2023-06-05 11 | url: "https://github.com/tejakattenborn/AngleCAM" 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AngleCAM 2 | 3 | ## Contents 4 | 5 | [Introduction](#Introduction) 6 | 7 | [Approach and evaluation](#approach-and-evaluation) 8 | 9 | [Use AngleCam and how to contribute](#Use-AngleCam-and-how-to-contribute) 10 | 11 | 12 | ## Introduction 13 | 14 | 15 | Vertical leaf surface angles are relevant for various applications (e.g., modelling radiative transfer in plants, producitvity, or Earth systems). Measuring or tracking leaf angles through time is, hence, of vital importance for various disciplines. AngleCam is a deep learning-based method to predict leaf angle distributions from horizontal photographs. AngleCam can be applied on single photographs or time series. AngleCam was evaluated over various species, growth forms and for predicting time series over several months at 3-min intervals under field conditions. 16 | The underlying CNN models of AngleCam were trained with the Brinno TLC-200 Pro camera. The latter is low-cost and outdoor ready and enables to acquire time series up to several month. AngleCam may also be applicable to other cameras as long as their properties are comparable (Field of view, etc.).
17 | 18 |
19 | 20 | ![diurnal](https://github.com/tejakattenborn/AngleCAM/blob/main/result_small_mod.gif) 21 | 22 | *Animation of the AngleCam input (image frames) and output (leaf angle estimates) for a Acer pseudoplatanus crown. Note that here the output (leaf angle distribution) is for simplicity integrated to average leaf angles. The animation shows that during a sunny day Tilia cordata tends to oscilate its leaf angles. The estimates show a relatively high variance, mostly caused by small changes in leaf orientation due to wind. Despite considerable variation in illumination conditions, the predictions show a relatively stable course during the day.* 23 | 24 | ## Approach and evaluation 25 | 26 | AngleCam is based on Convolutional Neural Networks (at current stage with TensorFlow and the EfficientNet backbone). We trained the networks with several thousands reference samples that were generated from visual interpretation of invidiual image frames. For each image frame, we sampled 20 leaves, which were then converted to a leaf angle distribution (beta distribution). The CNN models were, hence, trained to predict a leaf angle distribution for each individual image. The model accuracy was estimated from independent holdouts. Additionally, we performed a independent validation using terrestrial laser scanning and the [TLSLeAF method by Atticus Stovall](https://github.com/aestovall/TLSLeAF). 27 | 28 | ![val](https://github.com/tejakattenborn/AngleCAM/blob/main/AngleCam_val.png) 29 | 30 | *Model evaluation based on training data, test data and terrestrial laser scanning. A manuscript describing the method and its evaluation is currently in review.* 31 | 32 | ![tls validation](https://github.com/tejakattenborn/AngleCAM/blob/main/tlsleaf_anglecam_comparison.png) 33 | 34 | *Comparison of AngleCam and TLSLeAF for predicting leaf angle distributions.* 35 | 36 | 37 | ## Use AngleCam and how to contribute 38 | 39 | * R- and Python-Scripts for running AngleCam can be found in [code_run_AngleCam](https://github.com/tejakattenborn/AngleCAM/tree/main/code_run_AngleCam). 40 | 41 | * The mandatory model object (hdf5) and example data can be downloaded from [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8008113.svg)](https://doi.org/10.5281/zenodo.8008113) 42 | 43 | * Further details on the method and its evaluation are published (2022) in Methods in Ecology and Evolution [![DOI](https://img.shields.io/badge/DOI-10.1111%252F2041----210X.13968-red)](https://doi.org/10.1111/2041-210X.13968) 44 | 45 | 46 | 47 | The code requires a running TensorFlow installation (see script for some help ressources). 48 | Please contact me if you find any bugs or have problems getting the models running: 49 | [https://geosense.uni-freiburg.de/en ](https://uni-freiburg.de/enr-geosense/team/kattenborn/) 50 | 51 | Current evaluations indicate the transferability of the approach across scence conditions, species and plant forms. However, we cannot eventually state how well the models perform on your datasets (which may be composed of very different species, leaf forms or cameras). Additional labels (reference data) may be helpful to tune the model towards you application scenario. A [R-script](https://github.com/tejakattenborn/AngleCAM/blob/main/code_manuscript/01_labelling_leaf_angles.R) for producing new reference data is included in this repository. We would be very thankful if you would share these samples with us, so we can continuously improve the model performance and transferability. In return, we provide you a model object that was optimized for your data. AngleCam is truly in a alpha-phase and it success also depends on your help. Contributors will ofcourse be involved in upcoming analysis and scientific output. 52 | 53 | ## Acknowledgements 54 | 55 | * Many thanks for funding to the German Centre for Integrative Biodiversity Research (iDiv) and Leipzig University. 56 | 57 | * Thanks to Marco Bobbinger for setting up the example. 58 | -------------------------------------------------------------------------------- /code_manuscript/00_convert_brinno_avi_to_jpeg_timeseriesprediction.R.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script converts the .avi files from the Brinno-TLC200Pro to image frames in .png. 4 | # the frames per second (fps) define how many images should be converted (NULL --> all frames in the .avi-file) 5 | # the time stamp will be extracted from the imagery footer and the latter will be removed. The timestamp will be included in the image filename. 6 | # The script will query a xlsx file with two columns, which indicate if a file is usable (should be converted) and if the camera was installed upside-down. 7 | 8 | require(abind) 9 | require(av) 10 | library(magick) 11 | require(ggplot2) 12 | library(tesseract) 13 | require(xlsx) 14 | 15 | dir = "F:/data/data_brinno/data_2021_lak_brinno/data_brinno_lak2" 16 | setwd(dir) 17 | 18 | #fps = 0.5 # 1 pic every 10 seconds of the video. framerate of video = 1 pic per minute 19 | fps = 5 #NULL to get all images 20 | paste0("approx. pics per avi-file: ", (12*60) / (15 / fps)) # 15 = framerate in avi-file 21 | paste0("every ", 15/fps, " minutes on picture") # 15 = framerate in avi-file 22 | 23 | # initialize individual folders where the .avi files are located in 24 | treeindivids = list.dirs(getwd(), recursive = F); treeindivids 25 | 26 | for(treeindivid in 1:length(treeindivids)){ 27 | setwd(treeindivids[treeindivid]) 28 | seqs = list.files(pattern = "AVI") # load sequences 29 | meta = read.xlsx("1_manual_meta.xlsx", sheetIndex = 1) 30 | 31 | for(sequence in which(meta$usable == 1)){ # only apply loop for those videos that are "usable" 32 | 33 | # define output directory 34 | destdir = paste0(dir, "_timeseries") 35 | destdir_temp = paste0(destdir, "/temp") 36 | 37 | # create folders 38 | unlink(destdir_temp, recursive=TRUE) # clean target directory (if existent already) 39 | dir.create(destdir_temp) # create target directory # av_video_images will create a folder on its own 40 | 41 | # convert avi to jpeg 42 | av_video_images(seqs[sequence], destdir = destdir_temp, format = "png", fps = fps) # to png, jpeg induces artefacts 43 | 44 | # read time stamp from jpeg 45 | lim_top = 690 46 | lim_bottom = 720 47 | lim_left = 395 48 | lim_right = 900 49 | 50 | eng = tesseract(options = list(tessedit_char_whitelist = "0123456789/: "), language = "eng") 51 | 52 | all_pics = list.files(destdir_temp, full.names = T, pattern = "png") 53 | date_time = list() 54 | 55 | for(pic in 1:length(all_pics)){ 56 | input <- image_read(all_pics[pic]) %>%# image_convert(type = 'Grayscale') %>% 57 | .[[1]] %>% 58 | as.numeric() 59 | input_cut = input[c(lim_top:lim_bottom),c(lim_left:lim_right),] # crop time stamp section 60 | input_cut = (abs(input_cut-1)) # convert to black font on white background 61 | input_cut[input_cut <= 0.3] = 0 # set threshold 62 | input_cut[input_cut > 0.3] = 1 # set threshold 63 | input_cut[1:15,,] = 1 # insert upper margin; enhances character detection 64 | input_cut = abind(input_cut, input_cut[1:12,,], along=1) # insert lower margin; enhances character detection 65 | 66 | date_time[[pic]] = input_cut %>% image_read() %>% image_resize("280x") %>% tesseract::ocr(engine = eng) # resizing enahnces character detection 67 | } 68 | 69 | date_time = do.call(rbind, date_time) 70 | # filter frequent wrong data identification 71 | err = which(substr(date_time, 13,14) == "38") 72 | date_time[err,] = paste0(substr(date_time[err,], 1,12),"30", substr(date_time[err,], 15, nchar(date_time[1,]))) 73 | err = which(substr(date_time, 5,14) == "2021/89/89" | substr(date_time, 5,14) == "2021/89/09" | substr(date_time, 5,14) == "2021/09/89") 74 | date_time[err,] = paste0(substr(date_time[err,], 1,4),"2021/09/09", substr(date_time[err,], 15, nchar(date_time[1,]))) 75 | # remove unecessary chars 76 | date_time = apply(date_time, 2, substr, start=5, stop=23) 77 | date_time = as.POSIXct(date_time, tz = "Europe/Berlin", format = "%Y/%m/%d %H:%M:%OS") # check summer / winter time 78 | # remove outliers due to wrong date/time estimation 79 | outliers = which(is.na(date_time)) # in case PSIXct cannot reveal date time format (e.g., if estimated day in month exceeds total days of that month) 80 | outliers = c(outliers, which(abs(scale(date_time[-outliers])) > 2)) 81 | if(length(outliers) > 0) 82 | { 83 | date_time = date_time[-outliers] 84 | file.remove(all_pics[outliers]) 85 | all_pics = all_pics[-outliers] 86 | } 87 | 88 | # cut date / time area and rotate if necessary 89 | for(pic in 1:length(all_pics)){ 90 | input <- image_read(all_pics[pic]) %>%# image_convert(type = 'Grayscale') %>% 91 | .[[1]] %>% 92 | as.numeric() 93 | input_cut = input[c(1:lim_top),,] # crop time stamp section 94 | if(meta$upsidedown[sequence] == 1){ 95 | input_cut %>% image_read() %>% image_rotate(degrees = 180) %>% image_write(path = all_pics[pic]) 96 | }else{ 97 | input_cut %>% image_read() %>% image_write(path = all_pics[pic]) 98 | } 99 | } 100 | # rename and move files to output directory 101 | file.rename(list.files(destdir_temp, full.names = T), paste0(destdir, "/",substr(basename(dir), 13, 16), "_",basename(treeindivids[treeindivid]),"_",format(date_time, "%Y-%m-%d_%H-%M-%S"), ".png")) 102 | } 103 | } 104 | 105 | 106 | -------------------------------------------------------------------------------- /code_manuscript/00_convert_brinno_avi_to_jpeg_training.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script converts the .avi files from the Brinno-TLC200Pro to image frames in .png. 4 | # the frames per second (fps) define how many images should be converted (NULL --> all frames in the .avi-file) 5 | # the time stamp will be extracted from the imagery footer and the latter will be removed. 6 | # The script will query a xlsx file with two columns, which indicate if a file is usable (should be converted) and if the camera was installed upside-down 7 | 8 | require(abind) 9 | require(av) 10 | library(magick) 11 | require(ggplot2) 12 | library(tesseract) 13 | require(xlsx) 14 | 15 | dir = "INSERT DIR" 16 | setwd(dir) 17 | 18 | #fps = 0.5 # 1 pic every 10 seconds of the video. framerate of video = 1 pic per minute 19 | #fps = NULL #NULL to get all images 20 | fps = 0.3 21 | paste0("approx. pics per avi-file: ", (12*60) / (15 / fps)) # 15 = framerate in avi-file 22 | 23 | # initialize individual folders where the .avi files are located in 24 | treeindivids = list.dirs(getwd(), recursive = F); treeindivids 25 | 26 | for(treeindivid in 1:length(treeindivids)){ 27 | setwd(treeindivids[treeindivid]) 28 | seqs = list.files(pattern = "AVI") # load sequences 29 | meta = read.xlsx("1_manual_meta.xlsx", sheetIndex = 1) 30 | 31 | for(sequence in which(meta$usable == 1)){ # only apply loop for those videos that are "usable" 32 | 33 | # create folders 34 | destdir = substr(seqs[sequence], 1, nchar(seqs[sequence])-4) 35 | unlink(destdir, recursive=TRUE) # clean target directory (if existent already) 36 | #dir.create(destdir) # create target directory # av_video_images will create a folder on its own 37 | 38 | # convert avi to jpeg 39 | av_video_images(seqs[sequence], destdir = destdir, format = "png", fps = fps) # to png, jpeg induces artefacts 40 | file.rename(list.files(destdir, full.names = T), paste0(destdir, "/", basename(getwd()), "_", destdir, "_", list.files(destdir))) 41 | 42 | # read time stamp from jpeg 43 | lim_top = 690 44 | lim_bottom = 720 45 | lim_left = 395 46 | lim_right = 900 47 | 48 | eng = tesseract(options = list(tessedit_char_whitelist = "0123456789/: "), language = "eng") 49 | 50 | all_pics = list.files(destdir, full.names = T, pattern = "png") 51 | date_time = list() 52 | 53 | for(pic in 1:length(all_pics)){ 54 | input <- image_read(all_pics[pic]) %>%# image_convert(type = 'Grayscale') %>% 55 | .[[1]] %>% 56 | as.numeric() 57 | input_cut = input[c(lim_top:lim_bottom),c(lim_left:lim_right),] # crop time stamp section 58 | input_cut = (abs(input_cut-1)) # convert to black font on white background 59 | input_cut[input_cut <= 0.3] = 0 # set threshold 60 | input_cut[input_cut > 0.3] = 1 # set threshold 61 | input_cut[1:15,,] = 1 # insert upper margin; enhances character detection 62 | input_cut = abind(input_cut, input_cut[1:12,,], along=1) # insert lower margin; enhances character detection 63 | 64 | date_time[[pic]] = input_cut %>% image_read() %>% image_resize("280x") %>% tesseract::ocr(engine = eng) # resizing enahnces character detection 65 | } 66 | 67 | date_time = do.call(rbind, date_time) 68 | # filter frequent wrong data identification 69 | err = which(substr(date_time, 5,14) == "2021/89/89" | substr(date_time, 5,14) == "2021/89/09" | substr(date_time, 5,14) == "2021/09/89") 70 | date_time[err,] = paste0(substr(date_time[err,], 1,4),"2021/09/09", substr(date_time[err,], 15, nchar(date_time[1,]))) 71 | # remove unecessary chars 72 | date_time = apply(date_time, 2, substr, start=5, stop=23) 73 | 74 | # date / time in R: https://www.stat.berkeley.edu/~s133/dates.html 75 | date_time = as.POSIXct(date_time, tz = "Europe/Berlin", format = "%Y/%m/%d %H:%M:%OS") # check summer / winter time 76 | # remove outliers due to wrong date/time estimation 77 | outliers = which(is.na(date_time)) # in case PSIXct cannot reveal date time format (e.g., if estimated day in month exceeds total days of that month) 78 | outliers = c(outliers, which(abs(scale(date_time[-outliers])) > 2)) 79 | if(length(outliers) > 0) 80 | { 81 | date_time = date_time[-outliers] 82 | file.remove(all_pics[outliers]) 83 | all_pics = all_pics[-outliers] 84 | } 85 | 86 | # export date / time files 87 | save(date_time, file = paste0(destdir,"/", "1_", basename(getwd()), "_", destdir, "_1_date_time.Rdata")) 88 | pdf(file = paste0(destdir,"/", "1_", basename(getwd()), "_", destdir, "_1_date_time_check.pdf")) 89 | plot(date_time, xlab="pic index") 90 | dev.off() 91 | 92 | # -------- cut date / time area and rotate if necessary 93 | 94 | for(pic in 1:length(all_pics)){ 95 | input <- image_read(all_pics[pic]) %>%# image_convert(type = 'Grayscale') %>% 96 | .[[1]] %>% 97 | as.numeric() 98 | input_cut = input[c(1:lim_top),,] # crop time stamp section 99 | if(meta$upsidedown[sequence] == 1){ 100 | input_cut %>% image_read() %>% image_rotate(degrees = 180) %>% image_write(path = all_pics[pic]) 101 | }else{ 102 | input_cut %>% image_read() %>% image_write(path = all_pics[pic]) 103 | } 104 | } 105 | } 106 | } 107 | 108 | -------------------------------------------------------------------------------- /code_manuscript/01_labelling_leaf_angles.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # the script enables to label vertical angles for individual leaves from imagery. 4 | # The script will iterate over all images in the target folder (randomly) 5 | # After selecting a leaf, one can enter the estimated angle and a rolling index (the latter has not been used yet). 6 | # The red marks are meant to help extracting a balanced samples over the image area by taking the closest leaf for each mark. This is not always possible (depends on the image) 7 | # the samples are exported to a .csv file after no_samples has been reached (amount of samples) and the next image will be plotted. 8 | 9 | require(raster) 10 | require(scales) # for easy scaling of coordinate values 11 | require(gstat) 12 | require(dplyr) 13 | require(ggplot2) 14 | require(fitdistrplus) 15 | 16 | # set working directory 17 | setwd("INSERT DIR") 18 | 19 | # settings 20 | no_samples = 20 # how many samples per pic=? 21 | 22 | # load imagery 23 | pics = list.files(pattern=".png"); pics 24 | #pics = pics[c(which(grepl("kranzer", pics)), which(grepl("raw", pics)))] 25 | pics = pics[c(which(grepl("raw", pics)))] 26 | 27 | # start labelling procedure from here. 28 | # Stop the labelling procedure with ESC (press 3x) 29 | # The labelling procedure can be restarted from here (without running the code above) 30 | 31 | for(ii in sample(1:length(pics))){ 32 | 33 | if(file.exists(paste0(sub('\\.jpg$', '', pics[ii]), ".csv") ) == F){ 34 | pic = stack(pics[ii]) 35 | 36 | ysamp = seq(0+100, dim(pic)[1]-100, length.out = 5) 37 | xsamp = seq(0+100, dim(pic)[2]-100, length.out = 6) 38 | samp = expand.grid(xsamp, ysamp) 39 | 40 | #x11() 41 | plotRGB(pic) 42 | points(samp, cex = 5, col="red", pch=3) 43 | obs_all = data.frame(id = NA, x=NA, Y=NA, angle = NA, rolling = NA) 44 | 45 | i = 1 46 | while(i <= no_samples){ 47 | obs = locator(1, type = "p", pch=15, col = "pink") 48 | obs = unlist(obs) 49 | points(obs[1], obs[2], pch=19, col="turquoise", cex=2.5) 50 | obs = c(obs, as.numeric(readline("Enter estimated leaf angle:"))) 51 | obs = c(obs, as.numeric(readline("Enter estimated leaf rolling:"))) 52 | points(obs[1], obs[2], pch=19, col="yellow", cex=2.5) 53 | obs = c(i, obs) # add ID 54 | obs_all = rbind(obs_all, obs) 55 | print(paste0("Sample ", i, " of ", no_samples)) 56 | i = i+1 57 | } 58 | 59 | obs_all = obs_all[complete.cases(obs_all),] 60 | write.csv(obs_all, row.names = F, file = paste0(sub('\\.png$', '', pics[ii]), ".csv")) 61 | print(paste0("finished picture ", ii, "; ", pics[ii])) 62 | dev.off() 63 | } 64 | } 65 | 66 | -------------------------------------------------------------------------------- /code_manuscript/03_fit_beta_distributions_to_leaf_angles.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script fits beta distributions leaf angle samples. The script will iteratate over individual leaf samples sets (where one sample set corresponds to one image). 4 | # the mle method results in error estimates for both beta coefficients (alpha, beta). These error estimates are then used to augment the labels (50 variants per default) 5 | # The augmented labels are used in the CNN training (model regularization), where in each epoch a random variant is used as reference. 6 | # The script exports the fitted beta distribution and the 50 variants as .csv file that will be loaded during CNN training. 7 | 8 | require(fitdistrplus) 9 | 10 | # set working directory 11 | setwd("INSERT DIR") 12 | 13 | #fraction of the standard deviation (0-1) 14 | sd_frac = 0.2 # 0.50 15 | sim_per_pic = 50 16 | angle_resolution = 45 # angle resolution 17 | scaling_factor = 10 #used to rescale the values; to low numbers will slow down NN leanring, to high values will result in too fast updates 18 | ncp = 0 # ncp corrects for bias, just for testing 19 | 20 | # read all files 21 | samples_files = list.files(pattern=".csv", recursive = TRUE); samples_files 22 | samples_files = samples_files[-grep("sim", samples_files)]; samples_files 23 | 24 | #samples_files = samples_files[-grep("results", samples_files)]; samples_files 25 | samples = lapply(samples_files, read.csv, sep=",", header = T) 26 | 27 | # #just for repairing / error checking 28 | # for(i in 1:length(samples_files)){ 29 | # read.csv(samples_files[i]) 30 | # } 31 | # samples_files[i];i 32 | 33 | 34 | # fit distribution to all data 35 | for(i in 1:length(samples)){ 36 | 37 | # produce multiple simulations per pic (n = sim_per_pic) 38 | # one file for each pic with original distribution (n = 1) in the first row and simulated distributions with errors (n = sim_per_pic) in subsequent rows 39 | # in read from data read text file for each pic containing the simulations 40 | # use the simulations for training and the original rows for validation 41 | 42 | # load file and fit beta dist 43 | obs_all = samples[[i]] 44 | obs_angle_scaled = obs_all$angle/90 # from degree (0-90) to 0-1 45 | obs_angle_scaled[obs_angle_scaled == 0] = 0.00001 # dbeta cannot be fitted to zeros. Replace zeros with small value. 46 | obs_angle_scaled = obs_angle_scaled - 0.000001 # ...also cannot fit 1, so we have to subtract marginal value 47 | dist = fitdist(obs_angle_scaled, "beta", method = "mle") 48 | 49 | # predict distribution without fitted coefficients (onwards reference) 50 | beta = dbeta(seq(0, 1, length.out = angle_resolution), dist$estimate[1], dist$estimate[2], ncp = ncp, log = FALSE) 51 | beta[which(beta==Inf)] = 0 52 | beta = beta/sum(beta)*scaling_factor 53 | 54 | # predict multiple distributions incorporating the estimated errors (sd); onwards used for response based data augmentation 55 | sims = matrix(NA, nrow = sim_per_pic, ncol = angle_resolution) 56 | for(ii in 1:sim_per_pic){ 57 | beta1 = dbeta(seq(0, 1, length.out = angle_resolution), dist$estimate[1] + rnorm(1, 0, dist$sd[1])*sd_frac, dist$estimate[2] + rnorm(1, 0, dist$sd[1])*sd_frac, ncp = 0, log = FALSE) 58 | beta1[which(beta1==Inf)] = 0 59 | #beta1 = beta1*is.finite(beta1) # remove inf (sometimes happens...) 60 | sims[ii,] = beta1/sum(beta1)*scaling_factor 61 | } 62 | sims = sims[complete.cases(sims),-c(1,angle_resolution)] # remove NA, i.e. cases where no beta could be simulated (sampled values out of range), and remove first and last column since they always will be 0 63 | all = rbind(beta[-c(1,angle_resolution)], sims) 64 | # scale from 0-1 to 0-0.9 (degree * 0.01; 90 degree --> 0.9) 65 | all = all*90*0.01 66 | # write beta distribution variants to file 67 | write.table(all, paste0(sub(pattern = "(.*)\\..*$", replacement = "\\1", samples_files[i]), "_sim.csv"), row.names = F, col.names = F) 68 | } 69 | 70 | 71 | -------------------------------------------------------------------------------- /code_manuscript/08_train_AngleCam.R: -------------------------------------------------------------------------------- 1 | 2 | 3 | # -------- Description 4 | # This script trains CNN models using the pairs of input imagery and leaf angle distributions. 5 | # The training includes a data augmentation on both the predictors (imagery) and the response (leaf angle distributions). 6 | # The augmentation of the imagery includes color modifications, geometric operations and selecting different parts of the imagery 7 | # The augmentation of the leaf angle distribution is based on the variants of the fitted beta distributions (one variant is randmoly selected during the training process) 8 | # The data is loaded on the fly from harddist (tfdataset input pipeline). The input pipeline will act differently on the training and the test data (no augmentation on test data). 9 | # different (pretrained) backbones may be used 10 | # all model objects and are written to checkpoints and can restored from there after training (per default the best model as determined by the test dataset will be loaded). 11 | 12 | # the first row of each angle_distribution.csv file should contain the "original" data that will be used for testing. 13 | # additional ressources: 14 | # Guide to transfer learning: https://keras.io/guides/transfer_learning/ 15 | # Available models: https://keras.io/api/applications/ 16 | 17 | library(reticulate) 18 | reticulate::use_condaenv("rtf2", required = TRUE) 19 | reticulate::conda_binary() # "/net/home/tkattenborn/.local/share/r-miniconda/bin/conda" 20 | #install_tensorflow(version = "2.5.0") 21 | library(tensorflow) 22 | 23 | # check tensorflow 24 | hello <- tf$constant("Hello") 25 | print(hello) 26 | 27 | #initialize GPU 28 | gpu <- tf$config$experimental$get_visible_devices('GPU')[[1]] 29 | tf$config$experimental$set_memory_growth(device = gpu, enable = TRUE) 30 | # limit GPU RAM usage 31 | #tf$config$gpu$set_per_process_memory_growth() 32 | 33 | library(keras) 34 | library(dplyr) 35 | library(tfdatasets) 36 | 37 | # model config ---------------------------------------------------------------- 38 | 39 | workdir = "INSERT DIR" 40 | setwd(workdir) 41 | 42 | # model output directory 43 | outdir = "cnn_angle_model_efficientnet_b7_custlay_v2_do02/" 44 | # define directory for model checkpoints 45 | checkpoint_dir = paste0(outdir, "checkpoints_cnn_angle_predict") 46 | overwrite = FALSE # overwrite previous model output? 47 | 48 | # create output directories 49 | if(!dir.exists(outdir)){ 50 | dir.create(outdir, recursive = TRUE) 51 | } 52 | if(overwrite == TRUE) { 53 | unlink(list.files(outdir, full.names = TRUE)) 54 | } 55 | if(length(list.files(outdir)) > 0 & overwrite == FALSE) { 56 | stop(paste0("Can't overwrite files in ", outdir, " -> set 'overwrite = TRUE'")) 57 | } 58 | 59 | # image settings 60 | xres = 512L # max 1280 with brinno 61 | yres = 512L # max 690 with brinno 62 | no_bands = 3L # RGB 63 | no_test_samples = 200L # number of samples used for model selection / testing 64 | angle_resolution = 45L-2L # intervals of the leaf angle distributions 65 | 66 | # hyperparameters 67 | batch_size <- 10L # 8 for 256 * 256 68 | num_epochs <- 200L 69 | 70 | # Loading Data ---------------------------------------------------------------- 71 | 72 | # list data with reference data 73 | path_ref = list.files("data_brinno_lak_training_samples_withlabels", full.names = T, pattern = "sim.csv", recursive = T) 74 | path_img = list.files("data_brinno_lak_training_samples_withlabels", full.names = T, pattern = ".png", recursive = T) 75 | 76 | #match images with available reference data 77 | matches = match(sub("_sim.csv.*", "", path_ref), sub(".png.*", "", path_img)) 78 | path_img = path_img[matches] 79 | 80 | #just for identifying corrupt files 81 | # for(i in 1:length(path_ref)){ 82 | # read.table(path_ref[i]) 83 | # } 84 | # path_ref[i];i 85 | 86 | # prepare and standardize reference data samples 87 | ref = lapply(path_ref, read.table) 88 | ref_min_no = min(do.call(rbind, lapply(ref, dim))[,1]); ref_min_no # minimum number of valid simulations (tf dataset requires equal-sized data) 89 | # convert to equal-sized double 90 | for(i in 1:length(ref)){ 91 | ref[[i]] = array_reshape(as.matrix(ref[[i]][1:ref_min_no,]), dim=c(ref_min_no, angle_resolution)) 92 | } 93 | 94 | # split test data and save to disk 95 | set.seed(1234) 96 | testIdx = sample(x = 1:length(path_img), size = no_test_samples, replace = F) 97 | test_img = path_img[testIdx] 98 | test_ref = ref[testIdx] 99 | train_img = path_img[-testIdx] 100 | train_ref = ref[-testIdx] 101 | 102 | save(test_img, file = paste0(outdir, "test_img.RData"), overwrite = T) 103 | save(test_ref, file = paste0(outdir, "test_ref.RData"), overwrite = T) 104 | save(train_img, file = paste0(outdir, "train_img.RData"), overwrite = T) 105 | save(train_ref, file = paste0(outdir, "train_ref.RData"), overwrite = T) 106 | 107 | # if restore 108 | load(file = paste0(outdir, "test_img.RData")) 109 | load(file = paste0(outdir, "test_ref.RData")) 110 | load(file = paste0(outdir, "train_img.RData")) 111 | load(file = paste0(outdir, "train_ref.RData")) 112 | 113 | train_data = tibble(img = train_img, ref = train_ref) 114 | test_data = tibble(img = test_img, ref = test_ref) 115 | 116 | head(train_data) 117 | dim(train_data) 118 | train_data$img 119 | 120 | # tfdatasets input pipeline ----------------------------------------------- 121 | 122 | create_dataset <- function(data, 123 | train, # logical. TRUE for augmentation of training data 124 | batch, # numeric. multiplied by number of available gpus since batches will be split between gpus 125 | epochs, 126 | aug_rad, 127 | aug_geo, 128 | shuffle, # logical. default TRUE, set FALSE for test data 129 | dataset_size){ # numeric. number of samples per epoch the model will be trained on 130 | # the data will be shuffled in each epoch 131 | if(shuffle){ 132 | dataset = data %>% 133 | tensor_slices_dataset() %>% 134 | dataset_shuffle(buffer_size = length(data$img), reshuffle_each_iteration = TRUE) 135 | } else { 136 | dataset = data %>% 137 | tensor_slices_dataset() 138 | } 139 | 140 | # Apply data augmentation for training data 141 | if(train){ 142 | dataset = dataset %>% 143 | dataset_map(~.x %>% purrr::list_modify( # read files and decode png 144 | #img = tf$image$decode_png(tf$io$read_file(.x$img), channels = no_bands) 145 | img = tf$image$decode_jpeg(tf$io$read_file(.x$img), channels = no_bands, try_recover_truncated = TRUE, acceptable_fraction=0.3) %>% 146 | tf$image$convert_image_dtype(dtype = tf$float32), #%>% 147 | ref = .x$ref[sample(2:ref_min_no, 1),] # sample from beta distributions with random errors 148 | ), num_parallel_calls = NULL)# %>% 149 | }else{ 150 | dataset = dataset %>% 151 | dataset_map(~.x %>% purrr::list_modify( # read files and decode png 152 | #img = tf$image$decode_png(tf$io$read_file(.x$img), channels = no_bands) 153 | img = tf$image$decode_jpeg(tf$io$read_file(.x$img), channels = no_bands, try_recover_truncated = TRUE, acceptable_fraction=0.3) %>% 154 | tf$image$convert_image_dtype(dtype = tf$float32), #%>% 155 | ref = .x$ref[1,] # sample from original beta distributions 156 | ), num_parallel_calls = NULL)# %>% 157 | } 158 | 159 | # geometric modifications 160 | if(aug_geo) { 161 | dataset = dataset %>% 162 | dataset_map(~.x %>% purrr::list_modify( # randomly flip left/right 163 | img = tf$image$random_flip_left_right(.x$img, seed = 1L) %>% 164 | tf$image$random_crop(size = c(552L, 1024L, 3L)) %>% # will crop within image, original height = 690 *0.8, width = 1280*0,8 165 | tf$keras$preprocessing$image$smart_resize(size=c(xres, yres)) 166 | )) 167 | }else{ 168 | dataset = dataset %>% 169 | dataset_map(~.x %>% purrr::list_modify( # randomly flip left/right 170 | img = tf$keras$preprocessing$image$smart_resize(.x$img, size=c(xres, yres)) 171 | )) 172 | } 173 | # radiometric modifications 174 | if(aug_rad) { 175 | dataset = dataset %>% 176 | dataset_map(~.x %>% purrr::list_modify( # randomly assign brightness, contrast and saturation to images 177 | img = tf$image$random_brightness(.x$img, max_delta = 0.1, seed = 1L) %>% 178 | tf$image$random_contrast(lower = 0.8, upper = 1.2, seed = 2L) %>% 179 | tf$image$random_saturation(lower = 0.8, upper = 1.2, seed = 3L) %>% # requires 3 chnl -> with useDSM chnl = 4 180 | tf$clip_by_value(0, 1) # clip the values into [0,1] range. 181 | )) 182 | } 183 | if(train) { 184 | dataset = dataset %>% 185 | dataset_repeat(count = epochs) 186 | } 187 | dataset = dataset %>% dataset_batch(batch, drop_remainder = TRUE) %>% 188 | dataset_map(unname) %>% 189 | dataset_prefetch(buffer_size = tf$data$experimental$AUTOTUNE) 190 | #dataset_prefetch_to_device("/gpu:0", buffer_size = tf$data$experimental$AUTOTUNE) 191 | } 192 | 193 | dataset_size <- length(train_data$img) 194 | 195 | train_dataset <- create_dataset(train_data, batch = batch_size, epochs = num_epochs, 196 | shuffle = T, train = T, aug_geo = T, aug_rad = T) 197 | test_dataset <- create_dataset(test_data, batch = 1, epochs = 1, 198 | shuffle = F, train = F, aug_geo = F, aug_rad = F) 199 | 200 | # test train dataset pipeline 201 | dataset_iter = reticulate::as_iterator(train_dataset) 202 | example = dataset_iter %>% reticulate::iter_next() 203 | #example 204 | par(mfrow=c(1,2)) 205 | plot(as.raster(as.array(example[[1]][1,,,1:3]))) 206 | plot(seq(0,90, length.out = angle_resolution),as.numeric(example[[2]][1,]), type="l") 207 | 208 | # test test dataset pipeline 209 | dataset_iter = reticulate::as_iterator(test_dataset) 210 | example = dataset_iter %>% reticulate::iter_next() 211 | example 212 | par(mfrow=c(1,2)) 213 | plot(as.raster(as.array(example[[1]][1,,,1:3]))) 214 | plot(seq(0,90, length.out = angle_resolution),as.numeric(example[[2]][1,]), type="l") 215 | 216 | 217 | 218 | # Definition of the CNN structure (some alternatives are included; commented) ----------------------------------------------- 219 | 220 | base_model <- application_efficientnet_b7( 221 | input_shape = c(xres, yres, no_bands), 222 | include_top = FALSE, 223 | drop_connect_rate=0.1, # 0.2 is default 224 | #include_preprocessing=True, 225 | pooling = NULL 226 | ) 227 | 228 | # base_model <- application_resnet50_v2(weights = NULL, # = 'imagenet', 229 | # include_top = FALSE, 230 | # input_shape = c(xres, yres, no_bands), 231 | # pooling = NULL 232 | # ) 233 | 234 | # base_model <- application_resnet152_v2(weights = NULL, # = 'imagenet', 235 | # include_top = FALSE, 236 | # input_shape = c(xres, yres, no_bands), 237 | # pooling = NULL 238 | # ) 239 | 240 | # base_model <- application_mobilenet( 241 | # input_shape = c(xres, yres, no_bands), 242 | # alpha = 1, 243 | # depth_multiplier = 1L, 244 | # dropout = 0.001, 245 | # include_top = FALSE, 246 | # weights = "imagenet", 247 | # pooling = NULL 248 | # ) 249 | 250 | # # custom layers v1 251 | # predictions <- base_model$output %>% 252 | # layer_global_average_pooling_2d() %>% 253 | # layer_dropout(rate = 0.5) %>% 254 | # layer_dense(units = 512L, activation = 'relu') %>% 255 | # layer_dropout(rate = 0.5) %>% 256 | # layer_dense(units = 256L, activation = 'relu') %>% 257 | # layer_dropout(rate = 0.5) %>% 258 | # # layer_dense(units = 128L, activation = 'relu') %>% 259 | # # layer_dropout(rate = 0.5) %>% 260 | # layer_dense(units = 64L, activation = 'relu') %>% 261 | # layer_dense(units = angle_resolution, activation = 'linear') 262 | 263 | # # custom layers v2 264 | predictions <- base_model$output %>% 265 | layer_global_average_pooling_2d() %>% 266 | layer_dense(units = angle_resolution, activation = 'linear') 267 | 268 | # # custom layers v3 269 | # predictions <- base_model$output %>% 270 | # layer_global_max_pooling_2d() %>% 271 | # layer_dense(units = angle_resolution, activation = 'linear') 272 | 273 | # merge base model and custom layers 274 | model <- keras_model(inputs = base_model$input, outputs = predictions) 275 | 276 | # compile 277 | model %>% compile(optimizer = optimizer_adam(learning_rate = 0.0001), loss = 'mse') # mse or mae? 278 | 279 | 280 | # Model Training ----------------------------------------------- 281 | 282 | # create / clear output directory 283 | checkpoint_dir <- paste0(outdir, "checkpoints/") 284 | unlink(checkpoint_dir, recursive = TRUE) 285 | dir.create(checkpoint_dir, recursive = TRUE) 286 | filepath <- file.path(checkpoint_dir, "weights.{epoch:02d}-{val_loss:.5f}.hdf5") 287 | 288 | # define checkpoint saving 289 | ckpt_callback <- callback_model_checkpoint(filepath = filepath, 290 | monitor = "val_loss", 291 | save_weights_only = FALSE, 292 | save_best_only = TRUE, 293 | verbose = 1, 294 | mode = "auto", 295 | save_freq = "epoch") 296 | csv_callback <- callback_csv_logger(filename = paste0(outdir, "/epoch_results.csv")) 297 | 298 | #start training 299 | history <- model %>% fit(x = train_dataset, 300 | epochs = num_epochs, 301 | steps_per_epoch = dataset_size/batch_size, 302 | callbacks = list(ckpt_callback, 303 | csv_callback, 304 | callback_terminate_on_naan()), 305 | validation_data = test_dataset) 306 | 307 | 308 | # load best models from all epochs 309 | checkpoint_dir <- paste0(outdir, "checkpoints/") 310 | models = list.files(checkpoint_dir) 311 | models_best = which.min(substr(models, nchar(models)-11,nchar(models)-5)) 312 | model = load_model_hdf5(paste0(checkpoint_dir, models[models_best]), compile = FALSE) 313 | 314 | 315 | ### Export predictions for sampled train and test 316 | 317 | pred_test = model %>% predict(test_dataset) 318 | write.table(pred_test, file = paste0(outdir, "test_pred.csv"), row.names = F, col.names = F) 319 | 320 | # create train dataset without augmentation, shuffling, etc... 321 | train_dataset2 <- create_dataset(train_data, batch = 1, epochs = 1, 322 | shuffle = F, train = F, aug_geo = F, aug_rad = F) 323 | pred_train = model %>% predict(train_dataset2) 324 | write.table(pred_train, file = paste0(outdir, "train_pred.csv"), row.names = F, col.names = F) 325 | 326 | ### Export reference data for sampled train and test 327 | 328 | test_ref = lapply(test_ref,'[',1,) 329 | train_ref = lapply(train_ref,'[',1,) 330 | write.table(test_ref, file = paste0(outdir, "test_ref.csv"), row.names = F, col.names = F) 331 | write.table(train_ref, file = paste0(outdir, "train_ref.csv"), row.names = F, col.names = F) 332 | -------------------------------------------------------------------------------- /code_manuscript/09_TLSLeAF_output_to_single_plants.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script was used to extract individual plants from the TLSLeAF output. The TLSLeAF output was created from a TLS scan and included the estimated surface angles. 4 | # To extract individual plants from the TLSLeAF output, we manually segmented individual plants from the raw point cloud (saved as individual files). 5 | # The point clouds of the individual plants were onwards cleaned for artefacts and filtered for noise. 6 | # TLSLeAF output (with angle estimates) was then subsetted for each individual plant by applying a maximum distance threshold (distance to the cleaned point cloud subsets of the raw data). 7 | 8 | library(data.table) 9 | require(dplyr) 10 | 11 | setwd("F:/data/data_brinno/data_2022_brinno_tls_validation/2022_01_tls/") 12 | deci = 4 # precision to remove (keep points) 13 | #plyr::round_any(test$Y[1:5], 0.005, f = ceiling) # could be an alternative to round 14 | 15 | # load TLS scan with estimated angles (from TLSleAF) 16 | input_dirty_1 = list.files(full.names = T, pattern = "BG_scan_1_angles")[1];input_dirty_1 17 | input_dirty_2 = list.files(full.names = T, pattern = "BG_scan_2_angles")[1];input_dirty_2 18 | # load cleaned segments 19 | input_clean = list.files("2022_01_tls_single_plants", full.names = T, pattern = ".txt"); input_clean 20 | 21 | dirty_1 = fread(input_dirty_1, sep = " ") 22 | dirty_2 = fread(input_dirty_2, sep = " ") 23 | colnames(dirty_1)[1:3] = c("X", "Y", "Z") 24 | dirty_1$X = round(dirty_1$X, deci) 25 | dirty_1$Y = round(dirty_1$Y, deci) 26 | dirty_1$Z = round(dirty_1$Z, deci) 27 | colnames(dirty_2)[1:3] = c("X", "Y", "Z") 28 | dirty_2$X = round(dirty_2$X, deci) 29 | dirty_2$Y = round(dirty_2$Y, deci) 30 | dirty_2$Z = round(dirty_2$Z, deci) 31 | 32 | # iterate over all individual plants 33 | for(i in 1:length(input_clean)){ 34 | 35 | clean = fread(input_clean[i], sep = " ") 36 | colnames(clean)[1:3] = c("X", "Y", "Z") 37 | 38 | # backup coordinates 39 | back_X = round(clean$X, 5) 40 | back_Y = round(clean$Y, 5) 41 | back_Z = round(clean$Z, 5) 42 | 43 | #round coordinates (just for matching) 44 | clean$X = round(clean$X ,deci) 45 | clean$Y = round(clean$Y, deci) 46 | clean$Z = round(clean$Z, deci) 47 | 48 | if(i<12){ 49 | cleaned = left_join(clean, dirty_1, by = c("X", "Y", "Z")) 50 | }else{ 51 | cleaned = left_join(clean, dirty_2, by = c("X", "Y", "Z")) 52 | } 53 | 54 | # restore original coordinates 55 | cleaned$X = back_X 56 | cleaned$Y = back_Y 57 | cleaned$Z = back_Z 58 | 59 | cleaned = cleaned[complete.cases(cleaned)] 60 | write.csv(cleaned, row.names = F, file = paste0("2022_01_tls_single_plants_with_angle/", 61 | substr(basename(input_clean[i]), 1, nchar(basename(input_clean[i]))-4), 62 | "_angle.txt")) 63 | print(i) 64 | flush.console() 65 | } 66 | 67 | 68 | 69 | 70 | -------------------------------------------------------------------------------- /code_manuscript/10_predict_with_AngleCam_on_TLS_reference.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script applies AngleCam on the data of the botanical garden (TLS comparison) 4 | # The script assumes that the model was compiled 5 | 6 | checkpoint_dir <- paste0(outdir, "checkpoints/") 7 | models = list.files(checkpoint_dir) 8 | models_best = which.min(substr(models, nchar(models)-11,nchar(models)-5)) 9 | model = load_model_hdf5(paste0(checkpoint_dir, models[models_best]), compile = FALSE) 10 | 11 | path_tls_pics = "INSERT DIR" 12 | filename = "tls_predictions_2022_efficientnet_b7_custlay_v2_do01.RData" 13 | 14 | paths = list.dirs(path_tls_pics, recursive = F)[grepl("brinno", basename(list.dirs(path_tls_pics, recursive = F)))] 15 | paths 16 | 17 | tls_preds = list() 18 | for(i in 1:length(paths)){ 19 | tls_images = list.files(paths[i], full.names = T) 20 | 21 | tls_data = tibble(img = tls_images) 22 | tls_dataset <- create_dataset(tls_data, batch = 1, epochs = 1, 23 | shuffle = F, train = F, aug_geo = F, aug_rad = F) 24 | tls_preds[[i]] = model %>% predict(tls_dataset) 25 | } 26 | 27 | names(tls_preds) = basename(paths) 28 | save(tls_preds, file = paste0(path_tls_pics, filename)) 29 | 30 | # test botgarden pics vs. botgarden labelling 31 | 32 | botselect = grepl("raw", basename(path_img)) 33 | path_img_bot = path_img[botselect] 34 | ref_bot = ref[botselect] 35 | 36 | path_img_bot_data = tibble(img = path_img_bot) 37 | path_img_bot_dataset <- create_dataset(path_img_bot_data, batch = 1, epochs = 1, 38 | shuffle = F, train = F, aug_geo = F, aug_rad = F) 39 | path_img_bot_pred = model %>% predict(path_img_bot_dataset) 40 | 41 | # plot predictions (just for testing) 42 | i=1 43 | plot(seq(0,90, length.out= ncol(path_img_bot_pred)), path_img_bot_pred[i,], ylim= c( 0,0.8)) 44 | lines(seq(0,90, length.out= ncol(path_img_bot_pred)), ref_bot[[i]][1,]) 45 | i = i+1 46 | -------------------------------------------------------------------------------- /code_manuscript/11_compare_TLSLeAF_AngleCam.R: -------------------------------------------------------------------------------- 1 | 2 | 3 | # -------- Description 4 | # This script compares TLSLeAF outputs (leaf surface angles) for individual plants with leaf angle estimatates from the AngleCam method 5 | # Leaf angles are compared on the basis of leaf angle distributions and average leaf angles 6 | # The leaf angle distributions for TLSLeAF were derived by aggregating a distribution over all points. 7 | 8 | 9 | require(data.table) 10 | #devtools::install_bitbucket("derpylz/babyplots.git") 11 | require(babyplots) 12 | library(viridis) # https://cran.r-project.org/web/packages/viridis/vignettes/intro-to-viridis.html 13 | require(ggplot2) 14 | 15 | setwd("F:/data/data_brinno/") 16 | 17 | angle_resolution = 44 18 | scaling_factor = 10 #used to rescale the values to the range predicted by AngleCam; values were rescaled, since low values will slow the convergence of the CNN models for AngleCam (see respective script) 19 | 20 | load("INSERT DIR") 21 | brinno_freq = tls_preds # the AngleCam predictions were named (tls_preds; i.e. predictions on the TLS dataset) 22 | 23 | ### clean predictions (remove outliers) 24 | cleaner = function(x){ 25 | errors = which(x < 0 | x > 0.75) 26 | x[errors] = 0 27 | return(x) 28 | } 29 | brinno_freq = lapply(brinno_freq, cleaner) 30 | 31 | ### load and preprocess TLS 32 | dir_tls = "data_2022_brinno_tls_validation/2022_01_tls/2022_01_tls_single_plants_with_angle/" 33 | 34 | tls_files = list.files(dir_tls, full.names = T, pattern = "_angle.txt") 35 | tls_files 36 | 37 | 38 | ### calculate / compare leaf angle distributions ----------------------------- 39 | 40 | # conversion to leaf angle distributions 41 | breaks = seq(0, 90, length.out = angle_resolution) 42 | to_interval = function(x){ 43 | to_breaks = cut(tls$dip, breaks, right=FALSE) 44 | to_freq = as.numeric(table(to_breaks)) 45 | to_freq = to_freq/sum(to_freq)*scaling_factor 46 | } 47 | 48 | tls_freq = matrix(NA, nrow = length(tls_files), ncol = angle_resolution-1) 49 | rownames(tls_freq) = basename(tls_files) 50 | for(i in 1:length(tls_files)){ 51 | tls = fread(tls_files[i]) 52 | tls_freq[i,] = to_interval(tls$dip) 53 | } 54 | 55 | res_p = c() 56 | res_w = c() 57 | nmae_range = c() 58 | nmae_sd = c() 59 | range_1 = c() 60 | range_2 = c() 61 | 62 | for(i in 1:nrow(tls_freq)){ 63 | tls = tls_freq[i,] 64 | brinno = colMeans(brinno_freq[[i]]) 65 | res_p[i] <- as.numeric(stats::wilcox.test(x = tls, y= brinno, data = comp, paired = T)$p.value) 66 | res_w[i] <- as.numeric(stats::wilcox.test(x = tls, y= brinno, data = comp, paired = T)$statistic) 67 | nmae_range[i] = mean((tls - brinno) / diff(range(tls_freq))) * 100 68 | nmae_sd[i] = mean((tls - brinno) / sd(tls)) * 100 69 | range_1[i] = diff(range(tls)) 70 | range_2[i] = diff(range(tls_freq)) 71 | } 72 | 73 | # check output 74 | nmae_range 75 | nmae_sd 76 | mean(nmae_range) 77 | sd(nmae_sd) 78 | range_1 79 | range_2 80 | 81 | # calc nrmse 82 | sqrt(mean((tls - brinno)^2)) / diff(range(tls)) 83 | 84 | round(unlist(res_p), 4) 85 | nrow(tls_freq) 86 | length(which(round(unlist(res_p), 4)>0.05)) 87 | res_w 88 | plot(tls) 89 | lines(brinno) 90 | 91 | # export plots 92 | plot(1, ylim = c(0,0.8), xlim = c(0,90)) 93 | for(ii in 1:nrow(brinno_freq[[i]])){ 94 | lines(breaks[-1], brinno_freq[[i]][ii,], col = rgb(0.5, 0.5, 0.5, 0.3), lwd= 3) 95 | } 96 | lines(breaks[-1], tls_freq[i,], col = rgb(0.5, 0.5, 0.7, 1), lwd = 7) 97 | tls_test = fread(tls_files[i]) 98 | t = pointCloud(cbind(tls_test$X, tls_test$Y, tls_test$Z), colorBy = 'values', colorVar = tls_test$dip, upAxis = "+z", 99 | name = "test", 100 | size = 2, 101 | colorScale = "custom", #viridis(90, direction = -1),# "viridis", #"custom", # "viridis", 102 | customColorScale = viridis(90, direction = -1), 103 | #customColorScale = c("lightblue", "cyan", "yellow"), 104 | backgroundColor = "#ffffffff", 105 | showAxes = T, 106 | turntable = F) 107 | t 108 | i; names(brinno_freq)[i]; basename(tls_files[i]) 109 | i = i+1 110 | 111 | plot(1, ylim = c(0,0.8), xlim = c(0,90)) 112 | for(ii in 1:nrow(brinno_freq[[i]])){ 113 | lines(breaks[-1], brinno_freq[[i]][ii,], col = rgb(0.5, 0.5, 0.5, 0.3), lwd= 3) 114 | } 115 | lines(breaks[-1], tls_freq[i,], col = rgb(0.5, 0.5, 0.7, 1), lwd = 7) 116 | 117 | line_color = "black" 118 | line_size = 1 119 | col_brinno = "cyan3" # "deeppink3" 120 | col_tls = "yellow2"#"darkolivegreen" # "lightseagreen" 121 | 122 | for(i in 1:length(tls_files)){ 123 | 124 | df = data.frame(breaker = c(0,breaks[-1],90), 125 | tls = c(0, tls_freq[i,],0), 126 | brinno = c(0, colMeans(brinno_freq[[i]]),0)) 127 | mean_tls = sum(breaks[-1] * tls_freq[i,])/10 128 | mean_brinno = sum(breaks[-1] * colMeans(brinno_freq[[i]]))/10 129 | 130 | p <- ggplot(data=df, aes(x=breaker)) + 131 | geom_polygon(aes(y=tls), fill=col_tls, colour = line_color, size = line_size, alpha=0.3) + 132 | geom_polygon(aes(y=brinno), fill=col_brinno, colour = line_color, size = line_size, alpha=0.3) + 133 | xlab("leaf angle [°]") + ylab("density") + 134 | theme_minimal() 135 | ggsave(filename = paste0("data_2022_brinno_tls_validation/2022_01_results/compare_density_tls_plant_", sprintf("%02d", i), ".pdf"), 136 | width = 3, height = 3, p) 137 | } 138 | 139 | 140 | ### calculate / compare average leaf angles ----------------------------- 141 | 142 | # derive average leaf angle from leaf angle distributions 143 | avg_angle = function(x){ 144 | sum(breaks[-1] * x/10) 145 | } 146 | avg_tls = apply(tls_freq, 1, avg_angle) 147 | avg_brinno = c() 148 | for(i in 1:length(brinno_freq)){ 149 | avg_brinno[i] = mean(apply(brinno_freq[[i]], 1, avg_angle)) 150 | } 151 | 152 | # correlation (pearson) 153 | cor.test(avg_brinno, avg_tls)$estimate^2 154 | # linear model comparison 155 | lm(avg_tls ~ avg_brinno) 156 | 157 | avg = data.frame(avg_tls = avg_tls, avg_brinno = avg_brinno) 158 | 159 | # plot fit 160 | lims = c(25, 75) 161 | plot = ggplot(avg, aes(x=avg_brinno, y=avg_tls)) + 162 | geom_abline(slope = 1, linetype = "dashed", color="grey", size=1) + 163 | geom_point() + 164 | geom_smooth(method=lm, se=FALSE, col = "cyan3") + 165 | scale_x_continuous(limits = lims) + 166 | scale_y_continuous(limits = lims) + 167 | coord_fixed() + 168 | annotate(geom="text", x=50, y=30, label = paste0("R² = ", round(as.numeric(cor.test(avg$avg_tls, avg$avg_brinno)$estimate^2), 2))) + 169 | xlab("Avg. angle Brinno") + 170 | ylab("Avg. angle TLS") + 171 | theme_classic() 172 | plot 173 | ggsave(filename = "data_2022_brinno_tls_validation/2022_01_results/compare_avg_brinni_tls.pdf", 174 | width = 3, height = 3, plot) 175 | 176 | -------------------------------------------------------------------------------- /code_manuscript/12_predict_with_AngleCam_on_timeseries.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script applies AngleCam on image time series 4 | # The script assumes that the model was compiled 5 | # The file name included the camera-id and the timestamp. The file names were exported with the predictions, so that at a later stage the predictions were already linked with the acquisition time. 6 | 7 | checkpoint_dir <- paste0(outdir, "checkpoints/") 8 | models = list.files(checkpoint_dir) 9 | models_best = which.min(substr(models, nchar(models)-11,nchar(models)-5)) 10 | model = load_model_hdf5(paste0(checkpoint_dir, models[models_best]), compile = FALSE) 11 | 12 | # load image data 13 | time_series_no = 3 # our dataset comprised multiple time series as segments (1 June - July, 2 July - August, 3 August - September). 14 | t_series = list.files(path = paste0("/net/home/tkattenborn/data_brinno/data_2021_lak_brinno/data_brinno_lak",time_series_no,"_timeseries"), 15 | full.names = T, pattern = ".png", recursive = F) 16 | t_series_size = lapply(t_series, file.size) 17 | 18 | # query individual cameras (the predictions are performed and saved for each camera individually) 19 | t_series_trees = unique(substr(basename(t_series), 6, 13)); t_series_trees 20 | 21 | # predict any write to file 22 | for(i in 1:length(t_series_trees)){ 23 | t_series_sub = grepl(t_series_trees[i], t_series) 24 | imgdata = tibble(img = t_series[which(t_series_sub)]) 25 | imgdataset <- create_dataset(imgdata, batch = 1, epochs = 1, 26 | shuffle = F, train = F, aug_geo = F, aug_rad = F) 27 | pred_imgdataset = model %>% predict(imgdataset) 28 | rownames(pred_imgdataset) = basename(t_series[which(t_series_sub)]) 29 | write.csv(pred_imgdataset, file = paste0("/net/home/tkattenborn/data_brinno/data_2021_lak_brinno/data_brinno_lak",time_series_no,"_timeseries_prediction/", t_series_trees[i], ".csv"), 30 | row.names = T) 31 | } 32 | 33 | -------------------------------------------------------------------------------- /code_manuscript/14_evaluate_AngleCam_test_train.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script estimates the accuracy (R2) of AngleCam for the training and the test data based on the average leaf angle (determined from the predicted leaf angle distribution) 4 | # The script assumes that the model was compiled 5 | 6 | checkpoint_dir <- paste0(outdir, "checkpoints/") 7 | models = list.files(checkpoint_dir) 8 | models_best = which.min(substr(models, nchar(models)-11,nchar(models)-5)) 9 | model = load_model_hdf5(paste0(checkpoint_dir, models[models_best]), compile = FALSE) 10 | 11 | require(zoo) 12 | library(fpp2) 13 | 14 | setwd("G:/My Drive/paper/paper_2022_time_lapse_angles") 15 | 16 | # load the test and train reference data and predictions corresponding to the model 17 | model_version = "cnn_angle_model_efficientnet_b7_custlay_v2_do02/" 18 | test_pred = read.csv(paste0(model_version,"test_pred.csv"), sep=" ", row.names = NULL, header = FALSE) 19 | test_ref = t(read.csv(paste0(model_version,"test_ref.csv"), sep=" ", row.names = NULL, header = FALSE)) 20 | train_pred = read.csv(paste0(model_version,"train_pred.csv"), sep=" ", row.names = NULL, header = FALSE) 21 | train_ref = t(read.csv(paste0(model_version,"train_ref.csv"), sep=" ", row.names = NULL, header = FALSE)) 22 | load(paste0(model_version, "test_img.RData")) 23 | load(paste0(model_version, "train_img.RData")) 24 | 25 | # calculate the average leaf angle from the leaf angle distribution 26 | angle_res = ncol(test_pred) 27 | mean_angle = function(pred){ 28 | sum(pred/10*seq(0,90, length.out = angle_res)) 29 | } 30 | test_pred_avg = apply(test_pred, 1, mean_angle) 31 | test_ref_avg = apply(test_ref, 1, mean_angle) 32 | train_pred_avg = apply(train_pred, 1, mean_angle) 33 | train_ref_avg = apply(train_ref, 1, mean_angle) 34 | 35 | #remove outliers 36 | errr = which(train_pred_avg < 0) 37 | train_pred_avg = train_pred_avg[-errr] 38 | train_ref_avg = train_ref_avg[-errr] 39 | train_img = train_img[-errr] 40 | 41 | test_dat = data.frame(test_pred_avg = test_pred_avg, test_ref_avg = test_ref_avg) 42 | train_dat = data.frame(train_pred_avg = train_pred_avg,train_ref_avg = train_ref_avg) 43 | 44 | crane0_train = which(substr(basename(train_img), 1,3) %in% c("hom", "kra", "raw")) 45 | crane0_test = which(substr(basename(test_img), 1,3) %in% c("hom", "kra", "raw")) 46 | 47 | ### accuracy all data 48 | round(cor.test(test_dat$test_pred_avg, test_dat$test_ref_avg)$estimate^2, 2) 49 | 50 | ### accuracy crane 51 | round(cor.test(train_dat$train_pred_avg[-crane0_train], train_dat$train_ref_avg[-crane0_train])$estimate^2, 2) 52 | round(cor.test(test_dat$test_pred_avg[-crane0_test], test_dat$test_ref_avg[-crane0_test])$estimate^2, 2) 53 | 54 | ### accuracy potpurri 55 | round(cor.test(train_dat$train_pred_avg[crane0_train], train_dat$train_ref_avg[crane0_train])$estimate^2, 2) 56 | round(cor.test(test_dat$test_pred_avg[crane0_test], test_dat$test_ref_avg[crane0_test])$estimate^2, 2) 57 | 58 | # x and y limits 59 | lims = c(5,70) 60 | 61 | #scatterplot test 62 | plot_test = ggplot(test_dat, aes(x=test_pred_avg, y=test_ref_avg)) + 63 | geom_abline(slope = 1, linetype = "dashed", color="grey", size=1) + 64 | geom_point() + 65 | geom_smooth(method=lm, se=FALSE, col = "cyan3") + 66 | #geom_abline(slope = 1) + 67 | scale_x_continuous(limits = lims) + 68 | scale_y_continuous(limits = lims) + 69 | coord_fixed() + 70 | ylab("Avg. angle reference") + 71 | xlab("Avg. angle prediction") + 72 | annotate(geom="text", x=50, y=20, label=paste0("R² = ", round(cor.test(test_dat$test_pred_avg, test_dat$test_ref_avg)$estimate^2, 2)))+ 73 | theme_classic() 74 | plot_test 75 | ggsave(filename = paste0(model_version, "compare_pred_vs_ref_test.pdf"), 76 | width = 3, height = 3, plot_test) 77 | 78 | #scatterplot train 79 | plot_train = ggplot(train_dat, aes(x=train_pred_avg, y=train_ref_avg)) + 80 | geom_abline(slope = 1, linetype = "dashed", color="grey", size=1) + 81 | geom_point() + 82 | geom_smooth(method=lm, se=FALSE, col = "cyan3") + 83 | scale_x_continuous(limits = lims) + 84 | scale_y_continuous(limits = lims) + 85 | coord_fixed() + 86 | ylab("Avg. angle reference") + 87 | xlab("Avg. angle prediction") + 88 | annotate(geom="text", x=50, y=20, label=paste0("R² = ", round(cor.test(train_dat$train_pred_avg, train_dat$train_ref_avg)$estimate^2, 2)))+ 89 | theme_classic() 90 | plot_train 91 | ggsave(filename = paste0(model_version, "compare_pred_vs_ref_train.pdf"), 92 | width = 3, height = 3, plot_train) 93 | -------------------------------------------------------------------------------- /code_manuscript/15_evaluate_AngleCam_timeseries.R: -------------------------------------------------------------------------------- 1 | 2 | # -------- Description 3 | # This script includes several analysis based on the predicted time series derived from the 19 cameras placed at the Leipzig Canopy Crane 4 | # > plotting of time series 5 | # > correlation of leaf angle estimates from different cameras (cameras mounted on top vs within crowns) 6 | # > simulation of effects on canopy reflectance (simulated with PROSAIL-5B) determined by leaf angle variation through time 7 | # > calculating random forest(rf, x = environmental variables, y = mean_average_leaf_angle and sd_average_leaf_angle); calculation of rf variable importance 8 | # > plotting of time series of environmental variables and the rf variable importance 9 | 10 | 11 | require(zoo) 12 | library(fpp2) 13 | require(xlsx) 14 | require(readxl) 15 | require(ggplot2) 16 | require(data.table) 17 | library(ggpubr) 18 | require(hsdar) 19 | require(randomForest) 20 | require(permimp) 21 | require(foreach) 22 | require(doParallel) 23 | 24 | 25 | setwd("INSERT DIR") 26 | 27 | # Loading Data ---------------------------------------------------------------- 28 | 29 | cam_setup = read.xlsx("data_brinno_lak_timetable.xlsx", sheetIndex = 1) 30 | 31 | dlak_1 = "data_brinno_lak1_timeseries_prediction/" 32 | flak_1 = list.files(dlak_1, full.names = T, pattern = ".csv"); flak_1 33 | lak_1 = lapply(flak_1, read.csv, row.names = 1) 34 | 35 | dlak_2 = "data_brinno_lak2_timeseries_prediction/" 36 | flak_2 = list.files(dlak_2, full.names = T, pattern = ".csv"); flak_2 37 | lak_2 = lapply(flak_2, read.csv, row.names = 1) 38 | 39 | dlak_3 = "data_brinno_lak3_timeseries_prediction/" 40 | flak_3 = list.files(dlak_3, full.names = T, pattern = ".csv"); flak_3 41 | lak_3 = lapply(flak_3, read.csv, row.names = 1) 42 | 43 | angle_res = ncol(lak_1[[1]]) 44 | 45 | ### insert date time to data freame 46 | for(i in 1:length(lak_1)){ 47 | lak_1[[i]] = data.frame(datetime = as.POSIXct(paste0("2021", substr(rownames(lak_1[[i]]), 19, 33)), tz = "Europe/Berlin", format = "%Y-%m-%d_%H-%M-%OS"), lak_1[[i]]) 48 | } 49 | for(i in 1:length(lak_2)){ 50 | lak_2[[i]] = data.frame(datetime = as.POSIXct(paste0("2021", substr(rownames(lak_2[[i]]), 19, 33)), tz = "Europe/Berlin", format = "%Y-%m-%d_%H-%M-%OS"), lak_2[[i]]) 51 | } 52 | for(i in 1:length(lak_3)){ 53 | lak_3[[i]] = data.frame(datetime = as.POSIXct(paste0("2021", substr(rownames(lak_3[[i]]), 19, 33)), tz = "Europe/Berlin", format = "%Y-%m-%d_%H-%M-%OS"), lak_3[[i]]) 54 | } 55 | names(lak_1) = substr(basename(flak_1),0, 8) 56 | names(lak_2) = substr(basename(flak_2),0, 8) 57 | names(lak_3) = substr(basename(flak_3),0, 8) 58 | 59 | 60 | ### convert to avg angle 61 | mean_angle = function(pred){ 62 | sum(pred/10*seq(0,90, length.out = angle_res)) 63 | } 64 | lak_1_avg = list() 65 | for(i in 1:length(lak_1)){ 66 | lak_1_avg[[i]] = data.frame(date = lak_1[[i]][,1], avg = apply(lak_1[[i]][,-1], 1, mean_angle)) 67 | } 68 | lak_2_avg = list() 69 | for(i in 1:length(lak_2)){ 70 | lak_2_avg[[i]] = data.frame(date = lak_2[[i]][,1], avg = apply(lak_2[[i]][,-1], 1, mean_angle)) 71 | } 72 | lak_3_avg = list() 73 | for(i in 1:length(lak_3)){ 74 | lak_3_avg[[i]] = data.frame(date = lak_3[[i]][,1], avg = apply(lak_3[[i]][,-1], 1, mean_angle)) 75 | } 76 | 77 | 78 | ### filter daytimes 79 | filter_time = function(x){ 80 | new_dat = list() 81 | for(i in 1:length(x)){ 82 | filtered = which((format(x[[i]]$date, "%H-%M")>"09-00") & (format(x[[i]]$date, "%H-%M")<"19-00")) 83 | new_dat[[i]] = x[[i]][filtered,] 84 | } 85 | return(new_dat) 86 | } 87 | 88 | lak_1_avg = filter_time(lak_1_avg) 89 | lak_2_avg = filter_time(lak_2_avg) 90 | lak_3_avg = filter_time(lak_3_avg) 91 | 92 | # add names 93 | names(lak_1_avg) = names(lak_1) 94 | names(lak_2_avg) = names(lak_2) 95 | names(lak_3_avg) = names(lak_3) 96 | 97 | 98 | library(scales) 99 | library(dplyr) 100 | 101 | windows = 5*60 # minutes to hours 102 | 103 | 104 | ### rolling functions 105 | 106 | applyrollmean = function(data, camera){ 107 | dat = getElement(data, camera) 108 | if(is.null(dat)){ 109 | return(NULL) 110 | }else{ 111 | every.hour <- data.frame(date=seq(min(dat$date), max(dat$date), by="1 min")) 112 | dat_merg = full_join(x = dat, y= every.hour, by = "date") 113 | dat_merg_2 = dat_merg[order(dat_merg$date),] 114 | rmean = frollmean(dat_merg_2$avg, n = windows, fill = NA, na.rm = T, align = "center") 115 | rmean = data.frame(date = dat_merg_2$date, avg = rmean) 116 | return(rmean) 117 | } 118 | } 119 | 120 | applyrollsd = function(data, camera){ 121 | dat = getElement(data, camera) 122 | if(is.null(dat)){ 123 | return(NULL) 124 | }else{ 125 | every.hour <- data.frame(date=seq(min(dat$date), max(dat$date), by="1 min")) 126 | dat_merg = full_join(x = dat, y= every.hour, by = "date") 127 | dat_merg_2 = dat_merg[order(dat_merg$date),] 128 | rsd = frollapply(dat_merg_2$avg, n = windows, sd, fill = NA, na.rm = T, align = "center") 129 | rsd = data.frame(date = dat_merg_2$date, sd = rsd) 130 | return(rsd) 131 | } 132 | } 133 | 134 | vec_slope = function(x, na.rm){ 135 | if(length(which(is.na(x))) == length(x)){ 136 | return(NaN) 137 | }else{ 138 | x_vec = 1:length(x) 139 | return(as.numeric(lm(x ~ x_vec)$coefficients[2])) 140 | } 141 | } 142 | 143 | applyrollslope = function(data, camera){ 144 | dat = getElement(data, camera) 145 | if(is.null(dat)){ 146 | return(NULL) 147 | }else{ 148 | every.hour <- data.frame(date=seq(min(dat$date), max(dat$date), by="1 min")) 149 | dat_merg = full_join(x = dat, y= every.hour, by = "date") 150 | dat_merg_2 = dat_merg[order(dat_merg$date),] 151 | rslope = frollapply(dat_merg_2$avg, n = windows, vec_slope, fill = NA, na.rm = T, align = "center") 152 | rslope = data.frame(date = dat_merg_2$date, slope = rslope) 153 | return(rslope) 154 | } 155 | } 156 | 157 | 158 | 159 | 160 | # plot time series ---------------------------------------------------------------- 161 | 162 | xlims <- as.POSIXct(strptime(c(min(lak_1_avg$b_346_o1$date), max(lak_3_avg$b_346_o1$date)), 163 | format = "%Y-%m-%d %H:%M")) 164 | ylims = c(10, 61) 165 | plot_avg = TRUE 166 | times_col = "darkcyan" # "cyan3" #"deeppink3" 167 | 168 | for(i in 1:nrow(cam_setup)){ 169 | 170 | # plot time series 171 | camera = cam_setup[i,1] 172 | plot = ggplot(data = getElement(lak_1_avg, camera), 173 | aes(x = date, y = avg)) + 174 | geom_point(size = 0.5, alpha = 0.2, colour = "black", shape=16) + 175 | #geom_point(size = 0.5, alpha = 0.2, colour = "black", shape=16) + 176 | geom_point(data = getElement(lak_2_avg, camera), size = 0.5, alpha = 0.2, colour = "black", shape=16) + 177 | geom_point(data = getElement(lak_3_avg, camera), size = 0.5, alpha = 0.2, colour = "black", shape=16) + 178 | geom_vline(xintercept = min(getElement(lak_2_avg, camera)$date), linetype="dashed", color = "lightgrey", size=0.7) + 179 | geom_vline(xintercept = min(getElement(lak_3_avg, camera)$date), linetype="dashed", color = "lightgrey", size=0.7) + 180 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 181 | breaks = date_breaks("10 days"), 182 | limits = xlims, 183 | expand = c(0, 0)) + 184 | scale_y_continuous(limits = ylims) + 185 | xlab(NULL) +#xlab("Date") + 186 | ylab("Avg. angle [°]") + 187 | theme_classic() 188 | 189 | # add rolling mean 190 | if(plot_avg == TRUE){ 191 | if(is.null(getElement(lak_1_avg, camera))==FALSE){ 192 | rmean1 = applyrollmean(lak_1_avg, camera) 193 | plot = plot + geom_line(data = rmean1, aes(x = date, y = avg), colour = times_col) 194 | } 195 | if(is.null(getElement(lak_2_avg, camera))==FALSE){ 196 | rmean2 = applyrollmean(lak_2_avg, camera) 197 | plot = plot + geom_line(data = rmean2, aes(x = date, y = avg), colour = times_col) 198 | } 199 | if(is.null(getElement(lak_3_avg, camera))==FALSE){ 200 | rmean3 = applyrollmean(lak_3_avg, camera) 201 | plot = plot + geom_line(data = rmean3, aes(x = date, y = avg), colour = times_col) 202 | } 203 | } 204 | 205 | # add camera label 206 | if(substr(camera, 7,7) == "o"){ 207 | crown = paste0("top crown ", substr(camera, 8,8)) 208 | } 209 | if(substr(camera, 7,7) == "u"){ 210 | crown = paste0("within crown ", substr(camera, 8,8)) 211 | } 212 | if(substr(camera, 1,1) == "l"){ 213 | crown = paste0("Tilia cordata, ", crown) 214 | } 215 | if(substr(camera, 1,1) == "b"){ 216 | crown = paste0("Acer pseudoplantus, ", crown) 217 | } 218 | 219 | annotations <- data.frame( 220 | xpos = c(xlims[1] + 1*24*60*60), 221 | ypos = c(Inf), 222 | annotateText = crown, 223 | hjustvar = c(0) , 224 | vjustvar = c(1)) 225 | plot= plot + geom_text(data=annotations, 226 | aes(x=xpos,y=ypos,hjust=hjustvar,vjust=vjustvar,label=annotateText), size = 5) 227 | ggsave(filename = paste0("plots_timeseries_prediction/", camera, ".png"), 228 | width = 12, height = 2.1, plot) 229 | } 230 | 231 | 232 | 233 | 234 | # calculate correlation of leaf angle time series (average leaf angle) among different cameras ---------------------------------------------------------------- 235 | 236 | # function to calculate the correlation of two cameras 237 | calc_cor = function(camera_1, camera_2){ 238 | 239 | c1 = data.frame(avg = c(getElement(lak_1_avg, camera_1)$avg, getElement(lak_2_avg, camera_1)$avg, getElement(lak_3_avg, camera_1)$avg), 240 | date = c(getElement(lak_1_avg, camera_1)$date, getElement(lak_2_avg, camera_1)$date, getElement(lak_3_avg, camera_1)$date)) 241 | c2 = data.frame(avg = c(getElement(lak_1_avg, camera_2)$avg, getElement(lak_2_avg, camera_2)$avg, getElement(lak_3_avg, camera_2)$avg), 242 | date = c(getElement(lak_1_avg, camera_2)$date, getElement(lak_2_avg, camera_2)$date, getElement(lak_3_avg, camera_2)$date)) 243 | d <- function(x,y){ 244 | distance = abs(x-y) 245 | if(distance[which.min(distance)]< 180){ 246 | return(which.min(distance)) 247 | } 248 | } 249 | idx <- sapply( c1$date, function(x) d(x,c2$date)) # find matches (closest time stamp) 250 | return(cor.test(c1$avg, c2$avg[idx])$estimate) 251 | } 252 | 253 | # calculate correlation between cameras of same level (top / within) 254 | cor_within = c(calc_cor("b_346_u1", "b_346_u2"), calc_cor("b_513_u1", "b_513_u2"), calc_cor("l_343_u1", "l_343_u2"), 255 | calc_cor("l_439_u1", "l_439_u2")) 256 | cor_top = c(calc_cor("b_346_o1", "b_346_o2"), calc_cor("b_513_o1", "b_513_o2"), calc_cor("l_343_o1", "l_343_o2"),calc_cor("l_439_o1", "l_439_o2"), 257 | calc_cor("b_346_o1", "b_346_o3"), calc_cor("l_343_o1", "l_343_o3"),calc_cor("l_439_o1", "l_439_o3"), 258 | calc_cor("b_346_o2", "b_346_o3"), calc_cor("l_343_o2", "l_343_o3"),calc_cor("l_439_o2", "l_439_o3")) 259 | 260 | mean(cor_within)^2 261 | mean(cor_top)^2 262 | 263 | 264 | 265 | 266 | # simulate leaf angle effect on canopy reflectance ---------------------------------------------------------------- 267 | 268 | #camera = "l_439_o2" 269 | camera = "b_513_o2" 270 | 271 | dat = rbind(getElement(lak_1_avg, camera), getElement(lak_2_avg, camera), getElement(lak_3_avg, camera)) 272 | parameter <- data.frame(lidfa = dat$avg) 273 | spectra <- PROSAIL(parameterList = parameter, TypeLidf = 0, LAI = 3, Cab = 30, Car = 8, lidfb = 0) 274 | 275 | mean_ref = apply(spectra(spectra), 2, mean) 276 | sd_ref = apply(spectra(spectra), 2, sd) 277 | q05_ref = apply(spectra(spectra), 2, quantile, probs = 0.01) 278 | q95_ref = apply(spectra(spectra), 2, quantile, probs = 0.91) 279 | min_ref = apply(spectra(spectra), 2, min) 280 | max_ref = apply(spectra(spectra), 2, max) 281 | 282 | spec_sim = data.frame(min = spectra(PROSAIL(lidfa = quantile(dat$avg, 0.01), TypeLidf = 0, LAI = 3, Cab = 30, Car = 8, lidfb = 0))[1,], 283 | max = spectra(PROSAIL(lidfa = quantile(dat$avg, 0.99), TypeLidf = 0, LAI = 3, Cab = 30, Car = 8, lidfb = 0))[1,], 284 | mean = spectra(PROSAIL(lidfa = mean(dat$avg), TypeLidf = 0, LAI = 3, Cab = 30, Car = 8, lidfb = 0))[1,], 285 | wavelength = 400:2500) 286 | 287 | plot= ggplot(data=spec_sim, aes(x=wavelength, y=max)) + 288 | geom_line(colour = "cyan3", alpha=0.5) + 289 | geom_line(aes(y=mean), colour = "black") + 290 | geom_line(aes(y=min), colour = "cyan3", alpha=0.5) + 291 | #geom_ribbon(data=spec_sim(x, 2 <= x & x <= 3), 292 | geom_ribbon(aes(ymin=min,ymax=max), fill="cyan3", alpha=0.5) + 293 | geom_line(aes(y=mean), colour = "black") + 294 | xlab("wavelength [nm]") + 295 | ylab("reflectance [0-1]") + 296 | theme_classic() 297 | plot 298 | ggsave(filename = paste0("plots_timeseries_prediction/simulated_spectral_variation_",camera, ".pdf"), 299 | width = 4, height = 3, plot) 300 | 301 | # mean change in reflectance 302 | mean((spec_sim$min - spec_sim$max) / spec_sim$mean) 303 | 304 | #effect on NDVI 305 | red = c(630:690) 306 | nir = c(760:900) 307 | mean_red = rowMeans(spectra(spectra)[,red-400]) 308 | mean_nir = rowMeans(spectra(spectra)[,nir-400]) 309 | ndvi = (mean_nir-mean_red)/(mean_nir+mean_red) 310 | plot(ndvi) 311 | 312 | 313 | 314 | 315 | # random forest (leaf angles vs. environment) ---------------------------------------------------------------- 316 | 317 | ### load environmental data (weather + soil) 318 | envpath = "F:/data/data_brinno/data_2021_lak_environment/" 319 | # wheather; temp / humidity top of canopy 320 | weather = read_excel(paste0(envpath,"180927-upls-31_KranWetter_1642102879_TD.xlsx")) 321 | weather$date = as.POSIXct(paste(weather$`YYYY-MM-DD`, weather$Time), tz = "Europe/Berlin", format = "%Y-%m-%d %H:%M:%OS") 322 | soil = read_excel(paste0(envpath,"soil_moisture_2020_2021.xlsx")) 323 | soil$date = as.POSIXct(soil$Label, tz = "Europe/Berlin", format = "%d. %m. %Y %H:%M:%OS") 324 | weather = left_join(weather, soil[, c("date", "BF2")], "date") 325 | 326 | set.seed(1234) 327 | ntree = 1000 328 | mtry = 3 329 | n_cores = 6 330 | train_bins = c(5,10) # first value no of training bins, second value number of total bins 331 | 332 | weather2 = weather # simply a backup since the data is being modified in the following 333 | weather2$date = format(weather$date, "%Y-%m-%d %H:%M") 334 | 335 | # select predictors 336 | variable_set_avg = c("avg", "SPN1_Total", "Regen" , "RegenDau" ,"Ltemp" , "Ldruck" , "relFeuchte" , "WiGe", "BF2") 337 | variable_set_sd = c("sd", "SPN1_Total", "Regen" , "RegenDau" ,"Ltemp" , "Ldruck" , "relFeuchte" , "WiGe", "BF2") 338 | # initialize output files 339 | rf_results = data.frame(camera = NA, rf_avg_r2 = NA, rf_sd_r2 = NA) 340 | rf_results_varimp_avg = data.frame(matrix(NA, ncol=length(variable_set_avg[-1]))) 341 | names(rf_results_varimp_avg) = variable_set_avg[-1] 342 | rf_results_varimp_sd = data.frame(matrix(NA, ncol=length(variable_set_sd[-1]))) 343 | names(rf_results_varimp_sd) = variable_set_sd[-1] 344 | 345 | #for(i in 1:6){ 346 | 347 | for(i in 1:length(cam_setup[,1])){ 348 | camera = cam_setup[i,1] 349 | rf_results[i,1] = camera 350 | 351 | # with rolling mean of average leaf angle 352 | dat1_avg = rbind(applyrollmean(lak_1_avg, camera), applyrollmean(lak_2_avg, camera), applyrollmean(lak_3_avg, camera)) 353 | dat2_avg = rbind(applyrollmean(lak_1_avg, camera), applyrollmean(lak_2_avg, camera), applyrollmean(lak_3_avg, camera)) 354 | dat3_avg = rbind(applyrollmean(lak_1_avg, camera), applyrollmean(lak_2_avg, camera), applyrollmean(lak_3_avg, camera)) 355 | dat1_avg$date = format(dat1_avg$date, "%Y-%m-%d %H:%M") 356 | dat2_avg$date = format(dat2_avg$date, "%Y-%m-%d %H:%M") 357 | dat3_avg$date = format(dat3_avg$date, "%Y-%m-%d %H:%M") 358 | 359 | # with rolling sd of average leaf angle 360 | dat1_sd = rbind(applyrollsd(lak_1_avg, camera), applyrollsd(lak_2_avg, camera), applyrollsd(lak_3_avg, camera)) 361 | dat2_sd = rbind(applyrollsd(lak_1_avg, camera), applyrollsd(lak_2_avg, camera), applyrollsd(lak_3_avg, camera)) 362 | dat3_sd = rbind(applyrollsd(lak_1_avg, camera), applyrollsd(lak_2_avg, camera), applyrollsd(lak_3_avg, camera)) 363 | dat1_sd$date = format(dat1_sd$date, "%Y-%m-%d %H:%M") 364 | dat2_sd$date = format(dat2_sd$date, "%Y-%m-%d %H:%M") 365 | dat3_sd$date = format(dat3_sd$date, "%Y-%m-%d %H:%M") 366 | 367 | datall1_avg = inner_join(dat1_avg, weather2, by = "date") 368 | datall2_avg = inner_join(dat2_avg, weather2, by = "date") 369 | datall3_avg = inner_join(dat3_avg, weather2, by = "date") 370 | dat_avg = rbind(datall1_avg, datall2_avg, datall3_avg) 371 | 372 | datall1_sd = inner_join(dat1_sd, weather2, by = "date") 373 | datall2_sd = inner_join(dat2_sd, weather2, by = "date") 374 | datall3_sd = inner_join(dat3_sd, weather2, by = "date") 375 | dat_sd = rbind(datall1_sd, datall2_sd, datall3_sd) 376 | 377 | dat_avg$date = as.POSIXct(dat_avg$date, tz = "Europe/Berlin", format = "%Y-%m-%d %H:%M") 378 | dat_avg$date = format(dat_avg$date, "%H:%M") 379 | dat_avg = dat_avg[complete.cases(dat_avg),] 380 | 381 | dat_sd$date = as.POSIXct(dat_sd$date, tz = "Europe/Berlin", format = "%Y-%m-%d %H:%M") 382 | dat_sd$date = format(dat_sd$date, "%H:%M") 383 | dat_sd = dat_sd[complete.cases(dat_sd),] 384 | #dat$date = as.numeric(paste0(as.numeric(substr(dat$date, 0,2)),".", 60/as.numeric(substr(dat$date, 4,5))*10)) 385 | 386 | dat_avg = as.matrix(dat_avg[variable_set_avg]) 387 | dat_sd = as.matrix(dat_sd[variable_set_sd]) 388 | 389 | # binned sampling (to avoid temporal autocorrelation) 390 | cutmarks_avg = seq(1, nrow(dat_avg), length.out = train_bins[2]+1) 391 | groups_avg = .bincode(1:nrow(dat_avg), cutmarks_avg, right = TRUE, include.lowest = TRUE) 392 | #samp_id_avg = sample(unique(groups_avg), train_bins[1]) # define training samples 393 | samp_id_avg = which(unique(groups_avg) %% 2 == 1) 394 | cutmarks_sd = seq(1, nrow(dat_sd), length.out = train_bins[2]+1) 395 | groups_sd = .bincode(1:nrow(dat_sd), cutmarks_sd, right = TRUE, include.lowest = TRUE) 396 | #samp_id_sd = sample(unique(groups_sd), train_bins[1]) # define training samples 397 | samp_id_sd = which(unique(groups_sd) %% 2 == 1) 398 | 399 | dat_avg_train = data.frame(dat_avg[which(groups_avg %in% samp_id_avg),]) 400 | dat_sd_train = data.frame(dat_sd[which(groups_sd %in% samp_id_sd),]) 401 | dat_avg_test = data.frame(dat_avg[-which(groups_avg %in% samp_id_avg),]) 402 | dat_sd_test = data.frame(dat_sd[-which(groups_sd %in% samp_id_sd),]) 403 | 404 | 405 | # train raindom forest 406 | cl <- parallel::makeCluster(n_cores) 407 | doParallel::registerDoParallel(cl) 408 | rf_avg <- foreach(ntree=rep(ntree/n_cores, n_cores), .combine=randomForest::combine, 409 | .multicombine=TRUE, .packages='randomForest') %dopar% { 410 | randomForest(avg ~ ., data = dat_avg_train, ntree = ntree, localImp = TRUE, type = "regression", mtry = mtry, conditionalTree = F, 411 | keep.forest = TRUE, keep.inbag = TRUE) 412 | } 413 | rf_sd <- foreach(ntree=rep(ntree/n_cores, n_cores), .combine=randomForest::combine, 414 | .multicombine=TRUE, .packages='randomForest') %dopar% { 415 | randomForest(sd ~ ., data = dat_sd_train, ntree = ntree, importance = TRUE, type = "regression", mtry = mtry, conditionalTree = F, 416 | keep.forest = TRUE, keep.inbag = TRUE) 417 | } 418 | stopCluster(cl) 419 | 420 | # calculate importance (non conditional) 421 | # https://www.r-bloggers.com/2018/06/be-aware-of-bias-in-rf-variable-importance-metrics/ 422 | # rf_results_varimp_avg[i,] = rf_avg$importance[,1] 423 | # rf_results_varimp_sd[i,] = rf_sd$importance[,1] 424 | 425 | # predict on test data 426 | predicted_avg = predict(rf_avg, dat_avg_test) 427 | predicted_sd = predict(rf_sd, dat_sd_test) 428 | 429 | # calulcate R2 430 | rf_results$rf_avg_r2[i] = cor.test(dat_avg_test[,1], predicted_avg)$estimate^2 431 | rf_results$rf_sd_r2[i] = cor.test(dat_sd_test[,1], predicted_sd)$estimate^2 432 | 433 | # https://cran.r-project.org/web/packages/permimp/vignettes/permimp-package.html 434 | rf_results_varimp_avg[i,] = permimp(rf_avg, conditional = T, scaled = F, do_check = F)$values 435 | rf_results_varimp_sd[i,] = permimp(rf_sd, conditional = T, scaled = F, do_check = F)$values 436 | 437 | rf_eval_avg = data.frame(observed = dat_avg_test[,1], predicted = predicted_avg) 438 | rf_eval_sd = data.frame(observed = dat_sd_test[,1], predicted = predicted_sd) 439 | 440 | write.csv(rf_results, file = "rf_model_r2.csv") 441 | write.csv(rf_results_varimp_avg, file = "rf_model_varimp_avg_permip.csv") 442 | write.csv(rf_results_varimp_sd, file = "rf_model_varimp_sd_permip.csv") 443 | 444 | if(substr(camera, 7, 7) == "u"){ 445 | haribo = "darkorchid4" 446 | }else{ 447 | haribo = "cyan4" 448 | } 449 | 450 | # plots for moving mean (average leaf angle) 451 | plot = ggplot(rf_eval_avg, aes(x=observed, y=predicted)) + 452 | geom_abline(slope = 1, linetype = "dashed", color="grey", size=1) + 453 | geom_point(alpha = 0.1, colour = haribo, shape=16, size = 2) + 454 | scale_x_continuous(limits = c(min(rf_eval_avg),quantile(unlist(rf_eval_avg), 0.99))) + 455 | scale_y_continuous(limits = c(min(rf_eval_avg),quantile(unlist(rf_eval_avg), 0.99))) + 456 | coord_fixed() + 457 | annotate(geom="text", x=quantile(unlist(rf_eval_avg), 0.05), y=quantile(unlist(rf_eval_avg), 0.98), label = paste0("R² = ", round(as.numeric(cor.test(rf_eval_avg$observed, rf_eval_avg$predicted)$estimate^2), 3))) + 458 | xlab("observed avg [°]") + 459 | ylab("predicted avg [°]") + 460 | theme_classic() 461 | ggsave(filename = paste0("plots_timeseries_prediction/rf_model_avg_",camera, ".png"), 462 | width = 3, height = 3.1, plot) 463 | 464 | # plots for moving sd (average leaf angle) 465 | plot = ggplot(rf_eval_sd, aes(x=observed, y=predicted)) + 466 | geom_abline(slope = 1, linetype = "dashed", color="grey", size=1) + 467 | geom_point(alpha = 0.1, colour = haribo, shape=16, size = 2) + 468 | scale_x_continuous(limits = c(min(rf_eval_sd),quantile(unlist(rf_eval_sd), 0.99))) + 469 | scale_y_continuous(limits = c(min(rf_eval_sd),quantile(unlist(rf_eval_sd), 0.99))) + 470 | coord_fixed() + 471 | annotate(geom="text", x=quantile(unlist(rf_eval_sd), 0.05), y=quantile(unlist(rf_eval_sd), 0.98), label = paste0("R² = ", round(as.numeric(cor.test(rf_eval_sd$observed, rf_eval_sd$predicted)$estimate^2), 3))) + 472 | xlab("observed sd [°]") + 473 | ylab("predicted sd [°]") + 474 | theme_classic() 475 | ggsave(filename = paste0("plots_timeseries_prediction/rf_model_sd_",camera, ".png"), 476 | width = 3, height = 3.1, plot) 477 | } 478 | 479 | 480 | rf_results_varimp_avg = abs(rf_results_varimp_avg) 481 | rf_results_varimp_sd = abs(rf_results_varimp_sd) 482 | 483 | colMeans(rf_results_varimp_avg)/sum(colMeans(rf_results_varimp_avg)) * 100 484 | colMeans(rf_results_varimp_sd)/sum(colMeans(rf_results_varimp_sd)) * 100 485 | 486 | rf_results_varimp_avg 487 | rf_results_varimp_sd 488 | 489 | ## summary rf avg 490 | # for Tilia 491 | round(mean(rf_results$rf_avg_r2[which(substr(rf_results$camera, 1,1) == "l")]),2) 492 | # for Acer 493 | round(mean(rf_results$rf_avg_r2[which(substr(rf_results$camera, 1,1) == "b")]),2) 494 | # top 495 | round(mean(rf_results$rf_avg_r2[which(substr(rf_results$camera, 7,7) == "o")]),2) 496 | # within 497 | round(mean(rf_results$rf_avg_r2[which(substr(rf_results$camera, 7,7) == "u")]),2) 498 | 499 | ## summary rf sd 500 | # for Tilia 501 | round(mean(rf_results$rf_sd_r2[which(substr(rf_results$camera, 1,1) == "l")]),2) 502 | # for Acer 503 | round(mean(rf_results$rf_sd_r2[which(substr(rf_results$camera, 1,1) == "b")]),2) 504 | # top 505 | round(mean(rf_results$rf_sd_r2[which(substr(rf_results$camera, 7,7) == "o")]),2) 506 | # within 507 | round(mean(rf_results$rf_sd_r2[which(substr(rf_results$camera, 7,7) == "u")]),2) 508 | 509 | ## varimp avg 510 | round(colMeans(rf_results_varimp_avg),2) 511 | 512 | ## varimp sd 513 | round(colMeans(rf_results_varimp_sd),2) 514 | 515 | 516 | 517 | 518 | 519 | # plot environmental variables and rf variable importance ---------------------------------------------------------------- 520 | 521 | order(weather$date[-nrow(weather)]) 522 | weather$Ltemp = as.numeric(weather$Ltemp) 523 | weather$Ldruck = as.numeric(weather$Ldruck) 524 | weather$SPN1_Diff = as.numeric(weather$SPN1_Diff) 525 | weather$SPN1_Total = as.numeric(weather$SPN1_Total) 526 | weather$Regen = as.numeric(weather$Regen) 527 | weather$RegenDau = as.numeric(weather$RegenDau) 528 | weather$relFeuchte = as.numeric(weather$relFeuchte) 529 | weather$WiGe = as.numeric(weather$WiGe) 530 | weather$wg_peak = as.numeric(weather$wg_peak) 531 | weather$BF2 = as.numeric(weather$BF2) 532 | 533 | anno_bg = "white" 534 | alpha_bg = 0.7 535 | vj = 0 536 | hj = 1 537 | anno_date = min(weather$date, na.rm=T)+55*24*60*60 538 | 539 | rf_varimp_avg = read.csv(file = "rf_model_varimp_avg_permip.csv", row.names = 1) # format to keep zeros for label consitency 540 | rf_varimp_sd = read.csv(file = "rf_model_varimp_sd_permip.csv", row.names = 1) 541 | rf_varimp_avg = format(round(colMeans(rf_varimp_avg)/sum(colMeans(rf_varimp_avg)) * 100, 2), nsmall = 1) 542 | rf_varimp_sd = format(round(colMeans(rf_varimp_sd)/sum(colMeans(rf_varimp_sd)) * 100, 2), nsmall = 1) 543 | rf_varimp_avg 544 | rf_varimp_sd 545 | 546 | # Temp 547 | ylims = c(10, 31) 548 | plot_temp = ggplot(data=weather, aes(x=date, y=Ltemp)) + 549 | geom_line(colour = "red3") + 550 | ylab("temp. [°C]") + 551 | xlab(NULL) + 552 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 553 | breaks = date_breaks("10 days"), 554 | limits = xlims, 555 | expand = c(0, 0)) + 556 | scale_y_continuous(limits = ylims) + 557 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 558 | label = paste0("Var imp mean: ", rf_varimp_avg[4], " Var imp sd: ", rf_varimp_sd[4])), 559 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 560 | theme_classic() 561 | plot_temp 562 | 563 | geom_label(aes(x = anno_date, y = ylims[2], label = "Here is a line"), fill = "green") 564 | 565 | # Airpressure 566 | ylims = c(988, 1010) 567 | plot_pressure = ggplot(data=weather, aes(x=date, y=Ldruck)) + 568 | geom_line(colour = "cyan3") + 569 | ylab("air press. [hPa]") + 570 | xlab(NULL) + 571 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 572 | breaks = date_breaks("10 days"), 573 | limits = xlims, 574 | expand = c(0, 0)) + 575 | scale_y_continuous(limits = ylims) + 576 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 577 | label = paste0("Var imp mean: ", rf_varimp_avg[5], " Var imp sd: ", rf_varimp_sd[5])), 578 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 579 | theme_classic() 580 | plot_pressure 581 | 582 | # Regen 583 | ylims = c(0.01, 4) 584 | plot_rain = ggplot(data=weather, aes(x=date, y=Regen)) + 585 | #geom_segment(colour = "cyan3") + 586 | geom_line(colour = "blue2") + 587 | #geom_bar(stat="identity") + 588 | ylab("rain [mm]") + 589 | xlab(NULL) + 590 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 591 | breaks = date_breaks("10 days"), 592 | limits = xlims, 593 | expand = c(0, 0)) + 594 | scale_y_continuous(limits = ylims) + 595 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 596 | label = paste0("Var imp mean: ", rf_varimp_avg[2], " Var imp sd: ", rf_varimp_sd[2])), 597 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 598 | theme_classic() 599 | plot_rain 600 | 601 | # solar irradiance 602 | ylims = c(min(weather$SPN1_Total, na.rm=T), max(weather$SPN1_Total, na.rm=T)) 603 | plot_solar = ggplot(data=weather, aes(x=date, y=SPN1_Total)) + 604 | geom_line(colour = "orange2") + 605 | ylab("rad. [W/m²]") + 606 | xlab(NULL) + 607 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 608 | breaks = date_breaks("10 days"), 609 | limits = xlims, 610 | expand = c(0, 0)) + 611 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 612 | label = paste0("Var imp mean: ", rf_varimp_avg[1], " Var imp sd: ", rf_varimp_sd[1])), 613 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 614 | # scale_y_continuous(limits = ylims) + 615 | theme_classic() 616 | plot_solar 617 | 618 | # humidity 619 | ylims = c(min(weather$relFeuchte, na.rm=T), max(weather$relFeuchte, na.rm=T)) 620 | plot_hum = ggplot(data=weather, aes(x=date, y=relFeuchte)) + 621 | geom_line(colour = "green4") + 622 | ylab("rel. hum. %]") + 623 | xlab(NULL) + 624 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 625 | breaks = date_breaks("10 days"), 626 | limits = xlims, 627 | expand = c(0, 0)) + 628 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 629 | label = paste0("Var imp mean: ", rf_varimp_avg[6], " Var imp sd: ", rf_varimp_sd[6])), 630 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 631 | # scale_y_continuous(limits = ylims) + 632 | theme_classic() 633 | plot_hum 634 | 635 | ylims = c(0, 6) 636 | plot_wind = ggplot(data=weather, aes(x=date, y=WiGe)) + 637 | geom_line(colour = "deeppink3") + 638 | ylab("wind [m/s]") + 639 | xlab(NULL) + 640 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 641 | breaks = date_breaks("10 days"), 642 | limits = xlims, 643 | expand = c(0, 0)) + 644 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 645 | label = paste0("Var imp mean: ", rf_varimp_avg[7], " Var imp sd: ", rf_varimp_sd[7])), 646 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 647 | scale_y_continuous(limits = ylims) + 648 | theme_classic() 649 | plot_wind 650 | 651 | ylims = c(0, 600) 652 | plot_raind = ggplot(data=weather, aes(x=date, y=RegenDau)) + 653 | geom_line(colour = "purple") + 654 | ylab("rain dur. [s]") + 655 | xlab(NULL) + 656 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 657 | breaks = date_breaks("10 days"), 658 | limits = xlims, 659 | expand = c(0, 0)) + 660 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 661 | label = paste0("Var imp mean: ", rf_varimp_avg[3], " Var imp sd: ", rf_varimp_sd[3])), 662 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 663 | scale_y_continuous(limits = ylims) + 664 | theme_classic() 665 | plot_raind 666 | 667 | 668 | ylims = c(0, 30) 669 | plot_windpeak = ggplot(data=weather, aes(x=date, y=wg_peak)) + 670 | geom_line(colour = "deeppink3") + 671 | ylab("wind [m/s]") + 672 | xlab(NULL) + 673 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 674 | breaks = date_breaks("10 days"), 675 | limits = xlims, 676 | expand = c(0, 0)) + 677 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 678 | label = paste0("Var imp mean: ", rf_varimp_avg[7], " Var imp sd: ", rf_varimp_sd[7])), 679 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 680 | scale_y_continuous(limits = ylims) + 681 | theme_classic() 682 | plot_windpeak 683 | 684 | ylims = c(435, 540) 685 | plot_soil = ggplot(data=weather, aes(x=date, y=BF2)) + 686 | geom_line(colour = "deeppink3") + 687 | ylab("soil [m³/mm]") + 688 | xlab(NULL) + 689 | scale_x_datetime(labels = date_format("%d-%m-%y", tz = "Europe/Berlin"), 690 | breaks = date_breaks("10 days"), 691 | limits = xlims, 692 | expand = c(0, 0)) + 693 | geom_label(data = data.frame(x = anno_date, y = ylims[1], 694 | label = paste0("Var imp mean: ", rf_varimp_avg[8], " Var imp sd: ", rf_varimp_sd[8])), 695 | aes(x = x, y = y, label = label), fill=anno_bg, alpha = alpha_bg, label.size = 0.0, vjust = vj, just = hj) + 696 | scale_y_continuous(limits = ylims) + 697 | theme_classic() 698 | plot_soil 699 | 700 | 701 | figure <- ggarrange(plot_solar + theme(plot.margin = unit(c(0,0,0,0), "cm")), 702 | plot_rain + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.55), "cm")), 703 | plot_raind + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.18), "cm")), 704 | plot_temp + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.38), "cm")), 705 | plot_pressure + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0), "cm")), 706 | plot_hum + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.38), "cm")), 707 | plot_wind + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.55), "cm")), 708 | plot_soil + rremove("xlab") + theme(plot.margin = unit(c(0,0,0,0.18), "cm")), 709 | #labels = c("A", "B", "C"), 710 | ncol = 1, nrow = 8, align = "h") 711 | figure = figure + theme(plot.margin = ggplot2::margin(0.1,0.3,0.1,0.1, "cm")) 712 | figure 713 | ggsave(filename = paste0("plots_timeseries_prediction/environmental_variables.pdf"), 714 | width = 12, height = 7, figure) 715 | 716 | 717 | 718 | 719 | 720 | 721 | 722 | 723 | 724 | 725 | -------------------------------------------------------------------------------- /code_run_AngleCam/cnn_reqs.R: -------------------------------------------------------------------------------- 1 | 2 | 3 | # -------- Description 4 | # This script trains CNN models using the pairs of input imagery and leaf angle distributions. 5 | # The training includes a data augmentation on both the predictors (imagery) and the response (leaf angle distributions). 6 | # The augmentation of the imagery includes color modifications, geometric operations and selecting different parts of the imagery 7 | # The augmentation of the leaf angle distribution is based on the variants of the fitted beta distributions (one variant is randmoly selected during the training process) 8 | # The data is loaded on the fly from harddist (tfdataset input pipeline). The input pipeline will act differently on the training and the test data (no augmentation on test data). 9 | # different (pretrained) backbones may be used 10 | # all model objects and are written to checkpoints and can restored from there after training (per default the best model as determined by the test dataset will be loaded). 11 | 12 | 13 | # image settings 14 | xres = 600L # max 1280 with brinno 15 | yres = 600L # max690 with brinno 16 | no_bands = 3L 17 | angle_resolution = 45L-2L 18 | 19 | 20 | # hyperparameters 21 | batch_size <- 9L # 8 for 256 * 256 22 | num_epochs <- 500L 23 | 24 | 25 | # tfdatasets input pipeline ----------------------------------------------- 26 | 27 | create_dataset <- function(data, 28 | train, # logical. TRUE for augmentation of training data 29 | batch, # numeric. multiplied by number of available gpus since batches will be split between gpus 30 | epochs, 31 | aug_rad, 32 | aug_geo, 33 | shuffle, # logical. default TRUE, set FALSE for test data 34 | dataset_size){ # numeric. number of samples per epoch the model will be trained on 35 | # the data will be shuffled in each epoch 36 | if(shuffle){ 37 | dataset = data %>% 38 | tensor_slices_dataset() %>% 39 | dataset_shuffle(buffer_size = length(data$img), reshuffle_each_iteration = TRUE) 40 | } else { 41 | dataset = data %>% 42 | tensor_slices_dataset() 43 | } 44 | 45 | # Apply data augmentation for training data 46 | if(train){ 47 | dataset = dataset %>% 48 | dataset_map(~.x %>% purrr::list_modify( # read files and decode png 49 | #img = tf$image$decode_png(tf$io$read_file(.x$img), channels = no_bands) 50 | img = tf$image$decode_jpeg(tf$io$read_file(.x$img), channels = no_bands, try_recover_truncated = TRUE, acceptable_fraction=0.3) %>% 51 | tf$image$convert_image_dtype(dtype = tf$float32), #%>% 52 | ref = .x$ref[sample(2:ref_min_no, 1),] # sample from beta distributions with random errors 53 | ), num_parallel_calls = NULL)# %>% 54 | }else{ 55 | dataset = dataset %>% 56 | dataset_map(~.x %>% purrr::list_modify( # read files and decode png 57 | #img = tf$image$decode_png(tf$io$read_file(.x$img), channels = no_bands) 58 | img = tf$image$decode_jpeg(tf$io$read_file(.x$img), channels = no_bands, try_recover_truncated = TRUE, acceptable_fraction=0.3) %>% 59 | tf$image$convert_image_dtype(dtype = tf$float32), #%>% 60 | ref = .x$ref[1,] # sample from original beta distributions 61 | ), num_parallel_calls = NULL)# %>% 62 | } 63 | 64 | # geometric modifications 65 | if(aug_geo) { 66 | dataset = dataset %>% 67 | dataset_map(~.x %>% purrr::list_modify( # randomly flip left/right 68 | img = tf$image$random_flip_left_right(.x$img, seed = 1L) %>% 69 | tf$image$random_crop(size = c(552L, 1024L, 3L)) %>% # will crop within image, original height = 690 *0.8, width = 1280*0,8 70 | tf$keras$preprocessing$image$smart_resize(size=c(xres, yres)) 71 | )) 72 | }else{ 73 | dataset = dataset %>% 74 | dataset_map(~.x %>% purrr::list_modify( # randomly flip left/right 75 | img = tf$keras$preprocessing$image$smart_resize(.x$img, size=c(xres, yres)) 76 | )) 77 | } 78 | # radiometric modifications 79 | if(aug_rad) { 80 | dataset = dataset %>% 81 | dataset_map(~.x %>% purrr::list_modify( # randomly assign brightness, contrast and saturation to images 82 | img = tf$image$random_brightness(.x$img, max_delta = 0.1, seed = 1L) %>% 83 | tf$image$random_contrast(lower = 0.8, upper = 1.2, seed = 2L) %>% 84 | tf$image$random_saturation(lower = 0.8, upper = 1.2, seed = 3L) %>% # requires 3 chnl -> with useDSM chnl = 4 85 | tf$clip_by_value(0, 1) # clip the values into [0,1] range. 86 | )) 87 | } 88 | if(train) { 89 | dataset = dataset %>% 90 | dataset_repeat(count = epochs) 91 | } 92 | dataset = dataset %>% dataset_batch(batch, drop_remainder = TRUE) %>% 93 | dataset_map(unname) %>% 94 | dataset_prefetch(buffer_size = tf$data$experimental$AUTOTUNE) 95 | #dataset_prefetch_to_device("/gpu:0", buffer_size = tf$data$experimental$AUTOTUNE) 96 | } 97 | 98 | 99 | 100 | # Definition of the CNN structure ----------------------------------------------- 101 | # 102 | # base_model <- tf$keras$applications$EfficientNetV2L( 103 | # input_shape = c(xres, yres, no_bands), 104 | # include_top = FALSE, 105 | # include_preprocessing = FALSE, 106 | # weights = NULL, 107 | # pooling = NULL 108 | # ) 109 | # 110 | # # custom layers v2 111 | # predictions <- base_model$output %>% 112 | # layer_global_average_pooling_2d() %>% 113 | # layer_dropout(rate = 0.1) %>% 114 | # layer_dense(units = angle_resolution, activation = 'linear') 115 | # 116 | # # merge base model and custom layers 117 | # model <- keras_model(inputs = base_model$input, outputs = predictions) 118 | # 119 | # # compile 120 | # model %>% compile(optimizer = optimizer_adam(learning_rate = 0.0001), loss = 'mse') # mse or mae? -------------------------------------------------------------------------------- /code_run_AngleCam/run_AngleCam.R: -------------------------------------------------------------------------------- 1 | 2 | ### ---------------------------------------------------------------------------------------------- 3 | ### ---- AngleCam -------------------------------------------------------------------------------- 4 | ### ---------------------------------------------------------------------------------------------- 5 | 6 | # This is the alpha version of AngleCam (2023-06-05) 7 | # AngleCam estimates leaf angle distributions from conventional RGB photographs 8 | # for details see: https://github.com/tejakattenborn/AngleCAM 9 | # AngleCam can be applied on horizontally acquired imagery with a minumum image size of 600x600 px. 10 | # Input imagery can be of different size; the pipeline will resample them on the fly to 600x600 px. 11 | # Find below two examples for visualizing estimates as leaf angle distribution or average leaf angle 12 | 13 | # -------TensorFlow------------------------------------------------------------------------------ 14 | 15 | # Sketch and some info on installing TensorFlow (sometimes tricky): 16 | # devtools::install_github("rstudio/tensorflow") 17 | # install_tensorflow(version = "2.11.0") 18 | # > official github repository: https://github.com/rstudio/tensorflow 19 | # > General info on installing Tensorflow: https://tensorflow.rstudio.com/installation/ 20 | # > running TensorFlow with GPU support is recommended but not mandatory (CPU also works fine, but slower) 21 | # > Hints for TensorFlow with GPU support: https://tensorflow.rstudio.com/installation/gpu/local_gpu/ 22 | # > hello <- tf$constant("Hello"); print(hello) # check TensorFlow installation 23 | 24 | # ------Dependencies-------------------------------------------------------------------------------- 25 | 26 | library(reticulate) 27 | reticulate::use_condaenv(condaenv = "tfr", required = TRUE) 28 | 29 | library(tensorflow) 30 | library(keras) 31 | library(dplyr) 32 | library(tfdatasets) 33 | require(purrr) 34 | 35 | setwd("INSERT DIR") 36 | 37 | ### GPU settings (only needed if tensorflow is run on GPU, recommended but not mandatory; CPU also works) 38 | gpu_no = 1 # GPU 1 or 2? (in case of multiple GPUs) 39 | gpu_no = tf$config$list_physical_devices()[c(1, gpu_no +1)] 40 | tf$config$set_visible_devices(gpu_no) 41 | 42 | ### compile the model and initialize the tfdatasets pipeline 43 | source("cnn_reqs.R") 44 | # load model weights 45 | model <- load_model_hdf5("AngleCam_efficientnet_V2L_14-03-2023.hdf5", compile = FALSE) 46 | 47 | # -------AngleCam prediction---------------------------------------------------------------------- 48 | 49 | ### load image data and create tfdataset (the imagery will not be loaded to RAM but processed on the fly; there are, hence, no limits regarding dataset size) 50 | imagepaths <- list.files(path = "example_dataset_timeseries_tilia_cordata", full.names = T, pattern = ".png", recursive = F); paste0(length(imagepaths), " images found.") 51 | imgdataset <- create_dataset(tibble(img = imagepaths), batch = 1, epochs = 1, shuffle = F, train = F, aug_geo = F, aug_rad = F) 52 | 53 | ### apply the model to the imagery and write output to file 54 | pred_imgdataset <- model %>% predict(imgdataset) 55 | pred_imgdataset <- pred_imgdataset/10 56 | rownames(pred_imgdataset) <- basename(imagepaths) 57 | write.csv(pred_imgdataset, file = "AngleCam_predictions_example_dataset_timeseries_tilia_cordata.csv", row.names = T) 58 | 59 | #--------simple visulization-------------------------------------------------------------------- 60 | 61 | ### plot leaf angle distributions over time 62 | for(i in 1:nrow(pred_imgdataset)){ 63 | if (i%%10 == 0) { 64 | #lines(seq(0,90, length.out = 43), pred_imgdataset[i,]) 65 | plot(seq(0,90, length.out = 43), pred_imgdataset[i,], ylim=c(0,0.07), xlim=c(0,90), type="l", main = substr(rownames(pred_imgdataset)[i], 15, 33), ylab = "density", xlab = "leaf angle [deg]") 66 | Sys.sleep(0.6) 67 | } 68 | } 69 | 70 | ### plot average leaf angles over time 71 | # derive average leaf angle from leaf angle distributions 72 | mean_angle = function(pred){sum(pred*seq(0,90, length.out = ncol(pred_imgdataset)))} 73 | avg_angle <- apply(pred_imgdataset, 1, mean_angle) 74 | # derive date from filename 75 | dates = as.POSIXct(substr(rownames(pred_imgdataset), 15, 33), tz = "Europe/Berlin", format = "%Y-%m-%d_%H-%M-%OS") 76 | # plot leaf angles for one day 77 | plot(dates, avg_angle, ylab = "Average leaf angle [deg]", xlab = "time (CEST)") 78 | 79 | 80 | -------------------------------------------------------------------------------- /code_run_AngleCam/run_AngleCam.py: -------------------------------------------------------------------------------- 1 | # Code Description: 2 | 3 | # In this code, the parameters and their explanations are as follows: 4 | 5 | # - `directory`: The directory path where the image data is located. It can be set to the desired directory path. 6 | # - `model_path`: The file path to the model weights file (.hdf5) that will be loaded for inference. 7 | # - `imagepaths`: A list of image file paths to be processed. These files should be located in the "wildcam12_test" folder within the directory. 8 | 9 | # The code performs the following steps: 10 | 11 | # 1. Loads the model weights using `keras.models.load_model`. 12 | # 2. Defines a function `get_image_taken_datetime` to extract the image taken date and time from the metadata using PIL. 13 | # 3. Loads the image data from the specified folder using `glob.glob` and stores the image file paths in `imagepaths`. 14 | # 4. Processes and resizes the images using OpenCV, scaling the pixel values between 0 and 1. 15 | # 5. Applies the loaded model to the processed images and saves the predictions to a CSV file. 16 | # 6. Extracts the date and time metadata from the images using the `get_image_taken_datetime` function. 17 | # 7. Calculates the average angle from the prediction dataset. 18 | # 8. Creates a DataFrame (`df`) containing the extracted dates, average angles, and file names. 19 | # 9. Saves the DataFrame to a CSV file. 20 | # 10. Plots the average leaf angles over time using Matplotlib and saves the plot as an image. 21 | 22 | # Please make sure to modify the relevant parameters and file paths according to your specific setup. 23 | 24 | import tensorflow as tf 25 | import keras 26 | import pandas as pd 27 | import glob 28 | import os 29 | os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE" 30 | import cv2 31 | import matplotlib.pyplot as plt 32 | import numpy as np 33 | #from PIL import Image 34 | #from PIL.ExifTags import TAGS 35 | 36 | # Set the directory where the image data are located 37 | directory = os.getcwd() 38 | print("Working directory: " + str(directory)) 39 | 40 | # Load the model weights 41 | model_path = "G:/My Drive/1_coding/anglecam/1_AngleCam_application/v2/AngleCam_efficientnet_V2L_14-03-2023.hdf5" 42 | model = keras.models.load_model(model_path) 43 | 44 | # Check if the model is loaded correctly 45 | if model: 46 | print("Model loaded successfully.") 47 | else: 48 | print("Failed to load the model.") 49 | 50 | 51 | # Set GPU settings if running TensorFlow on GPU 52 | gpus = tf.config.experimental.list_physical_devices('GPU') 53 | if gpus: 54 | tf.config.experimental.set_memory_growth(gpus[0], True) 55 | 56 | # Load the image data 57 | imagepaths = glob.glob(os.path.join(directory, "example_dataset_timeseries_tilia_cordata", "*.png")) 58 | print("Taking following imagepath: " + str(imagepaths)) 59 | print(f"{len(imagepaths)} images found.") 60 | 61 | # Process and resize the images 62 | processed_images = [] 63 | for imagepath in imagepaths: 64 | image = cv2.imread(imagepath) 65 | resized_image = cv2.resize(image, (600, 600)) 66 | processed_image = resized_image.astype(np.float32) / 255.0 67 | processed_images.append(processed_image) 68 | 69 | 70 | imgdataset = tf.data.Dataset.from_tensor_slices(processed_images) 71 | 72 | # Apply the model to the imagery and write output to file 73 | pred_imgdataset = model.predict(imgdataset.batch(1)) 74 | pred_imgdataset = pred_imgdataset / 10 75 | pred_imgdataset = pd.DataFrame(pred_imgdataset, index=[os.path.basename(path) for path in imagepaths]) 76 | pred_imgdataset.to_csv(os.path.join(directory, "AngleCam_prediction_table.csv")) 77 | 78 | print("Column titles in pred_imgdataset:") 79 | print(pred_imgdataset.columns) 80 | 81 | # Extract the date and time metadata from the images 82 | file_names = [] 83 | for imagepath in imagepaths: 84 | file_names.append(os.path.basename(imagepath)) 85 | 86 | dates = [] 87 | for filename in file_names: 88 | dates.append(pd.to_datetime(filename[14:33], format="%Y-%m-%d_%H-%M-%S", utc=True)) 89 | 90 | 91 | # Convert filenames to datetime objects 92 | dates = [datetime.datetime.strptime(filename[-19:-9], "%Y-%m-%d") for filename in filenames] 93 | 94 | def mean_angle(row): 95 | pred = row.iloc[1:] 96 | length = len(pred) 97 | angles = np.linspace(0, 90, num=length) 98 | result = np.sum(pred * angles) 99 | return result 100 | 101 | avg_angle = pred_imgdataset.apply(mean_angle, axis=1) 102 | 103 | print("Average angle: " + str(avg_angle)) 104 | data = {"dates": dates, "avg_angle": avg_angle, "file_name": file_names} 105 | df = pd.DataFrame(data) 106 | 107 | # Save the data to a CSV file 108 | df.to_csv(os.path.join(directory, "leaf_angle_data.csv"), index=False) 109 | 110 | # Plot average leaf angles over time 111 | plt.scatter(dates, avg_angle) 112 | plt.xlabel("Time (CEST)") 113 | plt.ylabel("Average leaf angle [deg]") 114 | plt.title("Average Leaf Angle over Time") 115 | plt.savefig("leaf_angle_plot.png") 116 | 117 | -------------------------------------------------------------------------------- /illustrations_small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tejakattenborn/AngleCAM/73743fc56638ea4e9323367ff85f6b3e7dcb30aa/illustrations_small.png -------------------------------------------------------------------------------- /result_small_mod.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tejakattenborn/AngleCAM/73743fc56638ea4e9323367ff85f6b3e7dcb30aa/result_small_mod.gif -------------------------------------------------------------------------------- /tlsleaf_anglecam_comparison.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tejakattenborn/AngleCAM/73743fc56638ea4e9323367ff85f6b3e7dcb30aa/tlsleaf_anglecam_comparison.png --------------------------------------------------------------------------------