├── LICENSE.md ├── README.md ├── data └── processed │ └── .gitkeep ├── inference ├── .gitkeep └── README.md ├── logs └── .gitkeep ├── reports ├── .gitkeep └── figures │ ├── .gitkeep │ ├── Model_Architecture.png │ └── Qualitative_Results.png ├── requirements.txt ├── src ├── README.md ├── __init__.py ├── config.py ├── data │ ├── .gitkeep │ └── dataloader.py ├── models │ ├── .gitkeep │ ├── __init__.py │ ├── loss.py │ ├── network.py │ ├── test_model.py │ └── train_model.py └── utils.py └── weights ├── .gitkeep └── README.md /LICENSE.md: -------------------------------------------------------------------------------- 1 | Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 2 | International 3 | 4 | Creative Commons Corporation ("Creative Commons") is not a law firm and 5 | does not provide legal services or legal advice. Distribution of 6 | Creative Commons public licenses does not create a lawyer-client or 7 | other relationship. Creative Commons makes its licenses and related 8 | information available on an "as-is" basis. Creative Commons gives no 9 | warranties regarding its licenses, any material licensed under their 10 | terms and conditions, or any related information. Creative Commons 11 | disclaims all liability for damages resulting from their use to the 12 | fullest extent possible. 13 | 14 | Using Creative Commons Public Licenses 15 | 16 | Creative Commons public licenses provide a standard set of terms and 17 | conditions that creators and other rights holders may use to share 18 | original works of authorship and other material subject to copyright and 19 | certain other rights specified in the public license below. The 20 | following considerations are for informational purposes only, are not 21 | exhaustive, and do not form part of our licenses. 22 | 23 | Considerations for licensors: Our public licenses are intended for use 24 | by those authorized to give the public permission to use material in 25 | ways otherwise restricted by copyright and certain other rights. Our 26 | licenses are irrevocable. Licensors should read and understand the terms 27 | and conditions of the license they choose before applying it. Licensors 28 | should also secure all rights necessary before applying our licenses so 29 | that the public can reuse the material as expected. Licensors should 30 | clearly mark any material not subject to the license. This includes 31 | other CC-licensed material, or material used under an exception or 32 | limitation to copyright. More considerations for licensors : 33 | wiki.creativecommons.org/Considerations\_for\_licensors 34 | 35 | Considerations for the public: By using one of our public licenses, a 36 | licensor grants the public permission to use the licensed material under 37 | specified terms and conditions. If the licensor's permission is not 38 | necessary for any reason–for example, because of any applicable 39 | exception or limitation to copyright–then that use is not regulated by 40 | the license. Our licenses grant only permissions under copyright and 41 | certain other rights that a licensor has authority to grant. Use of the 42 | licensed material may still be restricted for other reasons, including 43 | because others have copyright or other rights in the material. A 44 | licensor may make special requests, such as asking that all changes be 45 | marked or described. Although not required by our licenses, you are 46 | encouraged to respect those requests where reasonable. More 47 | considerations for the public : 48 | wiki.creativecommons.org/Considerations\_for\_licensees 49 | 50 | Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 51 | International Public License 52 | 53 | By exercising the Licensed Rights (defined below), You accept and agree 54 | to be bound by the terms and conditions of this Creative Commons 55 | Attribution-NonCommercial-NoDerivatives 4.0 International Public License 56 | ("Public License"). To the extent this Public License may be interpreted 57 | as a contract, You are granted the Licensed Rights in consideration of 58 | Your acceptance of these terms and conditions, and the Licensor grants 59 | You such rights in consideration of benefits the Licensor receives from 60 | making the Licensed Material available under these terms and conditions. 61 | 62 | - Section 1 – Definitions. 63 | 64 | - a. Adapted Material means material subject to Copyright and 65 | Similar Rights that is derived from or based upon the Licensed 66 | Material and in which the Licensed Material is translated, 67 | altered, arranged, transformed, or otherwise modified in a 68 | manner requiring permission under the Copyright and Similar 69 | Rights held by the Licensor. For purposes of this Public 70 | License, where the Licensed Material is a musical work, 71 | performance, or sound recording, Adapted Material is always 72 | produced where the Licensed Material is synched in timed 73 | relation with a moving image. 74 | - b. Copyright and Similar Rights means copyright and/or similar 75 | rights closely related to copyright including, without 76 | limitation, performance, broadcast, sound recording, and Sui 77 | Generis Database Rights, without regard to how the rights are 78 | labeled or categorized. For purposes of this Public License, the 79 | rights specified in Section 2(b)(1)-(2) are not Copyright and 80 | Similar Rights. 81 | - c. Effective Technological Measures means those measures that, 82 | in the absence of proper authority, may not be circumvented 83 | under laws fulfilling obligations under Article 11 of the WIPO 84 | Copyright Treaty adopted on December 20, 1996, and/or similar 85 | international agreements. 86 | - d. Exceptions and Limitations means fair use, fair dealing, 87 | and/or any other exception or limitation to Copyright and 88 | Similar Rights that applies to Your use of the Licensed 89 | Material. 90 | - e. Licensed Material means the artistic or literary work, 91 | database, or other material to which the Licensor applied this 92 | Public License. 93 | - f. Licensed Rights means the rights granted to You subject to 94 | the terms and conditions of this Public License, which are 95 | limited to all Copyright and Similar Rights that apply to Your 96 | use of the Licensed Material and that the Licensor has authority 97 | to license. 98 | - g. Licensor means the individual(s) or entity(ies) granting 99 | rights under this Public License. 100 | - h. NonCommercial means not primarily intended for or directed 101 | towards commercial advantage or monetary compensation. For 102 | purposes of this Public License, the exchange of the Licensed 103 | Material for other material subject to Copyright and Similar 104 | Rights by digital file-sharing or similar means is NonCommercial 105 | provided there is no payment of monetary compensation in 106 | connection with the exchange. 107 | - i. Share means to provide material to the public by any means or 108 | process that requires permission under the Licensed Rights, such 109 | as reproduction, public display, public performance, 110 | distribution, dissemination, communication, or importation, and 111 | to make material available to the public including in ways that 112 | members of the public may access the material from a place and 113 | at a time individually chosen by them. 114 | - j. Sui Generis Database Rights means rights other than copyright 115 | resulting from Directive 96/9/EC of the European Parliament and 116 | of the Council of 11 March 1996 on the legal protection of 117 | databases, as amended and/or succeeded, as well as other 118 | essentially equivalent rights anywhere in the world. 119 | - k. You means the individual or entity exercising the Licensed 120 | Rights under this Public License. Your has a corresponding 121 | meaning. 122 | 123 | - Section 2 – Scope. 124 | 125 | - a. License grant. 126 | - 1. Subject to the terms and conditions of this Public 127 | License, the Licensor hereby grants You a worldwide, 128 | royalty-free, non-sublicensable, non-exclusive, irrevocable 129 | license to exercise the Licensed Rights in the Licensed 130 | Material to: 131 | - A. reproduce and Share the Licensed Material, in whole 132 | or in part, for NonCommercial purposes only; and 133 | - B. produce and reproduce, but not Share, Adapted 134 | Material for NonCommercial purposes only. 135 | - 2. Exceptions and Limitations. For the avoidance of doubt, 136 | where Exceptions and Limitations apply to Your use, this 137 | Public License does not apply, and You do not need to comply 138 | with its terms and conditions. 139 | - 3. Term. The term of this Public License is specified in 140 | Section 6(a). 141 | - 4. Media and formats; technical modifications allowed. The 142 | Licensor authorizes You to exercise the Licensed Rights in 143 | all media and formats whether now known or hereafter 144 | created, and to make technical modifications necessary to do 145 | so. The Licensor waives and/or agrees not to assert any 146 | right or authority to forbid You from making technical 147 | modifications necessary to exercise the Licensed Rights, 148 | including technical modifications necessary to circumvent 149 | Effective Technological Measures. For purposes of this 150 | Public License, simply making modifications authorized by 151 | this Section 2(a)(4) never produces Adapted Material. 152 | - 5. Downstream recipients. 153 | - A. Offer from the Licensor – Licensed Material. Every 154 | recipient of the Licensed Material automatically 155 | receives an offer from the Licensor to exercise the 156 | Licensed Rights under the terms and conditions of this 157 | Public License. 158 | - B. No downstream restrictions. You may not offer or 159 | impose any additional or different terms or conditions 160 | on, or apply any Effective Technological Measures to, 161 | the Licensed Material if doing so restricts exercise of 162 | the Licensed Rights by any recipient of the Licensed 163 | Material. 164 | - 6. No endorsement. Nothing in this Public License 165 | constitutes or may be construed as permission to assert or 166 | imply that You are, or that Your use of the Licensed 167 | Material is, connected with, or sponsored, endorsed, or 168 | granted official status by, the Licensor or others 169 | designated to receive attribution as provided in Section 170 | 3(a)(1)(A)(i). 171 | - b. Other rights. 172 | - 1. Moral rights, such as the right of integrity, are not 173 | licensed under this Public License, nor are publicity, 174 | privacy, and/or other similar personality rights; however, 175 | to the extent possible, the Licensor waives and/or agrees 176 | not to assert any such rights held by the Licensor to the 177 | limited extent necessary to allow You to exercise the 178 | Licensed Rights, but not otherwise. 179 | - 2. Patent and trademark rights are not licensed under this 180 | Public License. 181 | - 3. To the extent possible, the Licensor waives any right to 182 | collect royalties from You for the exercise of the Licensed 183 | Rights, whether directly or through a collecting society 184 | under any voluntary or waivable statutory or compulsory 185 | licensing scheme. In all other cases the Licensor expressly 186 | reserves any right to collect such royalties, including when 187 | the Licensed Material is used other than for NonCommercial 188 | purposes. 189 | 190 | - Section 3 – License Conditions. 191 | 192 | Your exercise of the Licensed Rights is expressly made subject to 193 | the following conditions. 194 | 195 | - a. Attribution. 196 | - 1. If You Share the Licensed Material, You must: 197 | 198 | - A. retain the following if it is supplied by the 199 | Licensor with the Licensed Material: 200 | - i. identification of the creator(s) of the Licensed 201 | Material and any others designated to receive 202 | attribution, in any reasonable manner requested by 203 | the Licensor (including by pseudonym if designated); 204 | - ii. a copyright notice; 205 | - iii. a notice that refers to this Public License; 206 | - iv. a notice that refers to the disclaimer of 207 | warranties; 208 | - v. a URI or hyperlink to the Licensed Material to 209 | the extent reasonably practicable; 210 | - B. indicate if You modified the Licensed Material and 211 | retain an indication of any previous modifications; and 212 | - C. indicate the Licensed Material is licensed under this 213 | Public License, and include the text of, or the URI or 214 | hyperlink to, this Public License. 215 | 216 | For the avoidance of doubt, You do not have permission under 217 | this Public License to Share Adapted Material. 218 | 219 | - 2. You may satisfy the conditions in Section 3(a)(1) in any 220 | reasonable manner based on the medium, means, and context in 221 | which You Share the Licensed Material. For example, it may 222 | be reasonable to satisfy the conditions by providing a URI 223 | or hyperlink to a resource that includes the required 224 | information. 225 | - 3. If requested by the Licensor, You must remove any of the 226 | information required by Section 3(a)(1)(A) to the extent 227 | reasonably practicable. 228 | 229 | - Section 4 – Sui Generis Database Rights. 230 | 231 | Where the Licensed Rights include Sui Generis Database Rights that 232 | apply to Your use of the Licensed Material: 233 | 234 | - a. for the avoidance of doubt, Section 2(a)(1) grants You the 235 | right to extract, reuse, reproduce, and Share all or a 236 | substantial portion of the contents of the database for 237 | NonCommercial purposes only and provided You do not Share 238 | Adapted Material; 239 | - b. if You include all or a substantial portion of the database 240 | contents in a database in which You have Sui Generis Database 241 | Rights, then the database in which You have Sui Generis Database 242 | Rights (but not its individual contents) is Adapted Material; 243 | and 244 | - c. You must comply with the conditions in Section 3(a) if You 245 | Share all or a substantial portion of the contents of the 246 | database. 247 | 248 | For the avoidance of doubt, this Section 4 supplements and does not 249 | replace Your obligations under this Public License where the 250 | Licensed Rights include other Copyright and Similar Rights. 251 | 252 | - Section 5 – Disclaimer of Warranties and Limitation of Liability. 253 | 254 | - a. Unless otherwise separately undertaken by the Licensor, to 255 | the extent possible, the Licensor offers the Licensed Material 256 | as-is and as-available, and makes no representations or 257 | warranties of any kind concerning the Licensed Material, whether 258 | express, implied, statutory, or other. This includes, without 259 | limitation, warranties of title, merchantability, fitness for a 260 | particular purpose, non-infringement, absence of latent or other 261 | defects, accuracy, or the presence or absence of errors, whether 262 | or not known or discoverable. Where disclaimers of warranties 263 | are not allowed in full or in part, this disclaimer may not 264 | apply to You. 265 | - b. To the extent possible, in no event will the Licensor be 266 | liable to You on any legal theory (including, without 267 | limitation, negligence) or otherwise for any direct, special, 268 | indirect, incidental, consequential, punitive, exemplary, or 269 | other losses, costs, expenses, or damages arising out of this 270 | Public License or use of the Licensed Material, even if the 271 | Licensor has been advised of the possibility of such losses, 272 | costs, expenses, or damages. Where a limitation of liability is 273 | not allowed in full or in part, this limitation may not apply to 274 | You. 275 | - c. The disclaimer of warranties and limitation of liability 276 | provided above shall be interpreted in a manner that, to the 277 | extent possible, most closely approximates an absolute 278 | disclaimer and waiver of all liability. 279 | 280 | - Section 6 – Term and Termination. 281 | 282 | - a. This Public License applies for the term of the Copyright and 283 | Similar Rights licensed here. However, if You fail to comply 284 | with this Public License, then Your rights under this Public 285 | License terminate automatically. 286 | - b. Where Your right to use the Licensed Material has terminated 287 | under Section 6(a), it reinstates: 288 | 289 | - 1. automatically as of the date the violation is cured, 290 | provided it is cured within 30 days of Your discovery of the 291 | violation; or 292 | - 2. upon express reinstatement by the Licensor. 293 | 294 | For the avoidance of doubt, this Section 6(b) does not affect 295 | any right the Licensor may have to seek remedies for Your 296 | violations of this Public License. 297 | 298 | - c. For the avoidance of doubt, the Licensor may also offer the 299 | Licensed Material under separate terms or conditions or stop 300 | distributing the Licensed Material at any time; however, doing 301 | so will not terminate this Public License. 302 | - d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 303 | License. 304 | 305 | - Section 7 – Other Terms and Conditions. 306 | 307 | - a. The Licensor shall not be bound by any additional or 308 | different terms or conditions communicated by You unless 309 | expressly agreed. 310 | - b. Any arrangements, understandings, or agreements regarding the 311 | Licensed Material not stated herein are separate from and 312 | independent of the terms and conditions of this Public License. 313 | 314 | - Section 8 – Interpretation. 315 | 316 | - a. For the avoidance of doubt, this Public License does not, and 317 | shall not be interpreted to, reduce, limit, restrict, or impose 318 | conditions on any use of the Licensed Material that could 319 | lawfully be made without permission under this Public License. 320 | - b. To the extent possible, if any provision of this Public 321 | License is deemed unenforceable, it shall be automatically 322 | reformed to the minimum extent necessary to make it enforceable. 323 | If the provision cannot be reformed, it shall be severed from 324 | this Public License without affecting the enforceability of the 325 | remaining terms and conditions. 326 | - c. No term or condition of this Public License will be waived 327 | and no failure to comply consented to unless expressly agreed to 328 | by the Licensor. 329 | - d. Nothing in this Public License constitutes or may be 330 | interpreted as a limitation upon, or waiver of, any privileges 331 | and immunities that apply to the Licensor or You, including from 332 | the legal processes of any jurisdiction or authority. 333 | 334 | Creative Commons is not a party to its public licenses. Notwithstanding, 335 | Creative Commons may elect to apply one of its public licenses to 336 | material it publishes and in those instances will be considered the 337 | "Licensor." The text of the Creative Commons public licenses is 338 | dedicated to the public domain under the CC0 Public Domain Dedication. 339 | Except for the limited purpose of indicating that material is shared 340 | under a Creative Commons public license or as otherwise permitted by the 341 | Creative Commons policies published at creativecommons.org/policies, 342 | Creative Commons does not authorize the use of the trademark "Creative 343 | Commons" or any other trademark or logo of Creative Commons without its 344 | prior written consent including, without limitation, in connection with 345 | any unauthorized modifications to any of its public licenses or any 346 | other arrangements, understandings, or agreements concerning use of 347 | licensed material. For the avoidance of doubt, this paragraph does not 348 | form part of the public licenses. 349 | 350 | Creative Commons may be contacted at creativecommons.org. 351 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/aclnet-an-attention-and-clustering-based/semantic-segmentation-on-swimseg)](https://paperswithcode.com/sota/semantic-segmentation-on-swimseg?p=aclnet-an-attention-and-clustering-based) 2 | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/aclnet-an-attention-and-clustering-based/semantic-segmentation-on-swinseg)](https://paperswithcode.com/sota/semantic-segmentation-on-swinseg?p=aclnet-an-attention-and-clustering-based) 3 | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/aclnet-an-attention-and-clustering-based/semantic-segmentation-on-swinyseg)](https://paperswithcode.com/sota/semantic-segmentation-on-swinyseg?p=aclnet-an-attention-and-clustering-based) 4 | 5 | ACLNet: An Attention and Clustering-based Cloud Segmentation Network 6 | 7 | ============================== 8 | 9 | This repository contains the source code of our paper, ACLNet (accepted for publication in Remote Sensing Letters). 10 | 11 | We propose a novel deep learning model named ACLNet, for cloud segmentation 12 | from ground images. ACLNet uses both deep neural network and machine learning 13 | (ML) algorithm to extract complementary features. Specifically, it uses EfficientNetB0 as the backbone, "atrous spatial pyramid pooling" (ASPP) to learn at multiple receptive fields, and "global attention module" (GAM) to extract fine-grained details from the image. ACLNet also uses k-means clustering to extract cloud boundaries more precisely. 14 | 15 | 16 | 17 | ## Sample Results 18 |
19 | 20 |

21 | 22 | Project Organization 23 | ------------ 24 | 25 | ├── LICENSE 26 | ├── README.md <- The top-level README for developers using this project. 27 | ├── data 28 | │   └── processed <- The final, canonical data sets for modeling. 29 | │ 30 | ├── requirements.txt <- The requirements file for reproducing the analysis environment. 31 | │ 32 | ├── src <- Source code for use in this project. 33 | │   ├── __init__.py <- Makes src a Python module 34 | │ ├── config.py <- All configuration params 35 | | ├── util.py <- All utilities functions 36 | │ │ 37 | │   ├── data <- Script to generate data in required format 38 | │   │   └── dataloader.py 39 | │ │ 40 | │   ├── models <- Scripts to train models and then use trained models to make 41 | │ │ │ predictions and test performance. 42 | │   │   ├── train_model.py 43 | | | ├── test_model.py 44 | | | ├── network.py 45 | | | └── loss.py 46 | ├── inference <- Inference directory, where predicted masks are stored. 47 | ├── logs <- Logs directory for saving terminal output. 48 | ├── weights <- Weights directory for saving checkpoints. 49 | -------- 50 | 51 | ## Get Started 52 |
53 | Dependencies: 54 | 55 | ``` 56 | pip install -r requirements.txt 57 | ``` 58 | 59 | ### (Optional) Conda Environment Configuration 60 | 61 | First, create a conda environment 62 | ```bash 63 | conda create -n aclnet # python=3 64 | conda activate aclnet 65 | ``` 66 | 67 | Now, add dependencies 68 | 69 | Now, you can install the required packages. 70 | ```bash 71 | pip install -r requirements.txt 72 | ``` 73 | 74 | ### Dataset 75 | 76 | We have used SWINySEG dataset which can be downloaded from SWINySEG for training and SWIMSEG, and SWINSEG for inference which can be downloaded from SWIMSEG and SWINSEG respectively. Download, and unzip dataset in ```data/processed/``` directory. 77 | 78 | ### Training 79 | 80 | change the hyperparameters and configuration parameters according to need in ```src/config.py```. 81 | 82 | To train aclnet, Run following command from ```/src``` directory. 83 | 84 | ```python models/train_model.py``` 85 | 86 | All the trained checkpoints for pre-training as well as full model training will be saved in ```/weights.``` 87 | 88 | Above command will train aclnet for given number of epochs in ```src/config.py```. 89 | 90 | ### Test performance 91 | 92 | To test aclnet with trained model, Run following command from ```/src``` directory. 93 | 94 | ```python models/test_model.py ``` 95 | 96 | Above command will generate Precision, Recall, F1-Score, Error rate, Matthews Correlation Coefficient (MCC) and ROC AUC Curve. ROC-AUC Curve will be saved in ```inference/``` directory. 97 | 98 | ## Citation 99 | ``` 100 | @article{makwana2022aclnet, 101 | title={ACLNet: an attention and clustering-based cloud segmentation network}, 102 | author={Makwana, Dhruv and Nag, Subhrajit and Susladkar, Onkar and Deshmukh, Gayatri and Teja R, Sai Chandra and Mittal, Sparsh and Mohan, C Krishna}, 103 | journal={Remote Sensing Letters}, 104 | volume={13}, 105 | number={9}, 106 | pages={865--875}, 107 | year={2022}, 108 | publisher={Taylor \& Francis} 109 | } 110 | ``` 111 | ## License 112 |
113 | CC BY-NC-ND 4.0 114 | -------------------------------------------------------------------------------- /data/processed/.gitkeep: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /inference/.gitkeep: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /inference/README.md: -------------------------------------------------------------------------------- 1 | ## Inference 2 | 3 | ROC AUC Curve will be stored here after running ```test_model.py``` from ```/src``` directory. 4 | -------------------------------------------------------------------------------- /logs/.gitkeep: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /reports/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/reports/.gitkeep -------------------------------------------------------------------------------- /reports/figures/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/reports/figures/.gitkeep -------------------------------------------------------------------------------- /reports/figures/Model_Architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/reports/figures/Model_Architecture.png -------------------------------------------------------------------------------- /reports/figures/Qualitative_Results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/reports/figures/Qualitative_Results.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==1.1.0 2 | albumentations==1.2.0 3 | astunparse==1.6.3 4 | cachetools==5.2.0 5 | certifi @ file:///C:/Windows/TEMP/abs_e9b7158a-aa56-4a5b-87b6-c00d295b01fanefpc8_o/croots/recipe/certifi_1655968940823/work/certifi 6 | charset-normalizer==2.0.12 7 | colorama==0.4.5 8 | cycler==0.11.0 9 | flatbuffers==1.12 10 | fonttools==4.33.3 11 | gast==0.4.0 12 | google-auth==2.8.0 13 | google-auth-oauthlib==0.4.6 14 | google-pasta==0.2.0 15 | grpcio==1.47.0 16 | h5py==3.7.0 17 | idna==3.3 18 | imageio==2.19.3 19 | joblib==1.1.0 20 | keras==2.9.0 21 | Keras-Preprocessing==1.1.2 22 | kiwisolver==1.4.3 23 | libclang==14.0.1 24 | Markdown==3.3.7 25 | matplotlib==3.5.2 26 | networkx==2.8.4 27 | numpy==1.23.0 28 | oauthlib==3.2.0 29 | opencv-python-headless==4.6.0.66 30 | opt-einsum==3.3.0 31 | packaging==21.3 32 | Pillow==9.1.1 33 | protobuf==3.19.4 34 | pyasn1==0.4.8 35 | pyasn1-modules==0.2.8 36 | pyparsing==3.0.9 37 | python-dateutil==2.8.2 38 | PyWavelets==1.3.0 39 | PyYAML==6.0 40 | qudida==0.0.4 41 | requests==2.28.0 42 | requests-oauthlib==1.3.1 43 | rsa==4.8 44 | scikit-image==0.18.3 45 | scikit-learn==1.1.1 46 | scipy==1.8.1 47 | six==1.16.0 48 | sklearn==0.0 49 | tensorboard==2.9.1 50 | tensorboard-data-server==0.6.1 51 | tensorboard-plugin-wit==1.8.1 52 | tensorflow-estimator==2.9.0 53 | tensorflow-gpu==2.9.1 54 | tensorflow-io-gcs-filesystem==0.26.0 55 | termcolor==1.1.0 56 | threadpoolctl==3.1.0 57 | tifffile==2022.5.4 58 | tqdm==4.64.0 59 | typing_extensions==4.2.0 60 | urllib3==1.26.9 61 | Werkzeug==2.1.2 62 | wincertstore==0.2 63 | wrapt==1.14.1 64 | -------------------------------------------------------------------------------- /src/README.md: -------------------------------------------------------------------------------- 1 | ## Source Code: 2 | 3 | ### Training 4 | 5 | change the hyperparameters and configuration parameters according to need in ```src/config.py```. 6 | 7 | To train aclnet, Run following command from ```/src``` directory. 8 | 9 | ```python models/train_model.py``` 10 | 11 | All the trained checkpoints for pre-training as well as full model training will be saved in ```/weights.``` 12 | 13 | Above command will train aclnet for given number of epochs in ```src/config.py```. 14 | 15 | ### Test performance 16 | 17 | To test aclnet with trained model, Run following command from ```/src``` directory. 18 | 19 | ```python models/test_model.py ``` 20 | 21 | Above command will generate Precision, Recall, F1-Score, Error rate, Matthews Correlation Coefficient (MCC) and ROC AUC Curve. ROC-AUC Curve will be saved in ```inference/``` directory. -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/src/__init__.py -------------------------------------------------------------------------------- /src/config.py: -------------------------------------------------------------------------------- 1 | IMAGES_DIR = "../data/processed/swinyseg/images" 2 | MASKS_DIR = "../data/processed/swinyseg/GTmaps" 3 | RESIZE_SIZE = (300, 300) 4 | CROP_SIZE = (288, 288) 5 | LOG_DIR = "../logs" 6 | WEIGHTS_DIR = "../weights" 7 | INFERENCE_DIR = "../inference" 8 | NUM_CLASSES = 2 9 | BATCH_SIZE = 2 10 | INITIAL_LEARNING_RATE = 0.0001 11 | EPOCHS = 300 12 | SEED = 42 13 | TEST_SIZE = 0.2 14 | 15 | COLOR_VALUES = {0: [0, 0, 0], 16 | 1: [255, 255, 255]} -------------------------------------------------------------------------------- /src/data/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/src/data/.gitkeep -------------------------------------------------------------------------------- /src/data/dataloader.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import sys 3 | sys.path.append("../src") 4 | 5 | from sklearn.model_selection import train_test_split 6 | import albumentations as A 7 | import tensorflow as tf 8 | from config import * 9 | from utils import * 10 | import numpy as np 11 | import cv2 12 | import os 13 | 14 | train_augment = A.Compose([ 15 | A.RandomCrop(height = CROP_SIZE[0], width = CROP_SIZE[1], p = 1), 16 | A.CLAHE(p = 0.5), 17 | A.RandomGamma(p = 0.5), 18 | A.OneOf([ 19 | A.VerticalFlip(p = 0.25), 20 | A.HorizontalFlip(p = 0.25), 21 | A.Transpose(p = 0.25), 22 | A.RandomRotate90(p = 0.25), 23 | ], p = 1.0), 24 | A.OneOf([ 25 | A.GridDistortion(p = 0.3), 26 | A.OpticalDistortion(distort_limit = 2, shift_limit = 0.5, p = 0.3), 27 | A.ElasticTransform(p = 0.3, alpha = 120, sigma = 120 * 0.05, alpha_affine = 120 * 0.03), 28 | ], p = 1.0), 29 | A.OneOf([ 30 | A.RandomBrightness(p = 0.25), 31 | A.RandomContrast(p = 0.25), 32 | ], p = 1.0), 33 | ]) 34 | 35 | test_augment = A.RandomCrop(height = CROP_SIZE[0], width = CROP_SIZE[1], p = 1.0) 36 | 37 | class ACLNetDataloader(tf.keras.utils.Sequence): 38 | """ Dataloader class to iterate over the data for 39 | segmentation""" 40 | def __init__(self, batch_size, resize_size, crop_size, input_img_paths, target_img_paths, data_type): 41 | self.batch_size = batch_size 42 | self.resize_size = resize_size 43 | self.crop_size = crop_size 44 | self.input_img_paths = input_img_paths 45 | self.target_img_paths = target_img_paths 46 | self.data_type = data_type 47 | self.train_transform = train_augment 48 | self.test_transform = test_augment 49 | 50 | def __len__(self): 51 | return len(self.target_img_paths) // self.batch_size 52 | 53 | def rgbToOnehot(self, rgb_mat, color_dict = COLOR_VALUES): 54 | num_classes = len(color_dict) 55 | shape = rgb_mat.shape[:2]+(num_classes,) 56 | mat = np.zeros( shape, dtype=np.float32) 57 | for i, _ in enumerate(color_dict): 58 | mat[:, :, i] = np.all(rgb_mat.reshape((-1, 3)) == color_dict[i], axis = 1).reshape(shape[:2]) 59 | return mat 60 | 61 | def KNNOutput(self, image): 62 | Z = image.reshape((-1,3)) 63 | Z = np.float32(Z) 64 | criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1.0) 65 | K = 2 66 | _, label, center=cv2.kmeans(Z, K, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS) 67 | center = np.uint8(center) 68 | res = center[label.flatten()] 69 | result = res.reshape((image.shape)) 70 | return result 71 | 72 | def __getitem__(self, idx): 73 | """Returns tuple (input, target) correspond to batch #idx.""" 74 | i = idx * self.batch_size 75 | batch_input_img_paths = self.input_img_paths[i : i + self.batch_size] 76 | batch_target_img_paths = self.target_img_paths[i : i + self.batch_size] 77 | x = np.zeros((self.batch_size,) + self.crop_size + (3,), dtype = "float32") 78 | y = np.zeros((self.batch_size,) + self.crop_size + (2,), dtype = "float32") 79 | z = np.zeros((self.batch_size,) + self.crop_size + (3,), dtype = "float32") 80 | 81 | for j, (input_image, input_mask) in enumerate(zip(batch_input_img_paths, batch_target_img_paths)): 82 | image = cv2.imread(input_image) 83 | image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) 84 | image = cv2.resize(image, self.resize_size) 85 | 86 | mask = cv2.imread(input_mask) 87 | mask = cv2.cvtColor(mask, cv2.COLOR_BGR2RGB) 88 | mask = cv2.resize(mask, self.resize_size) 89 | 90 | if self.data_type == "Train": 91 | augment = self.train_transform(image = image, mask = mask) 92 | else: 93 | augment = self.test_transform(image = image, mask = mask) 94 | image = augment['image'] 95 | mask = augment['mask'] 96 | 97 | knn_image = self.KNNOutput(image) 98 | knn_image = knn_image.astype("float32")/255 99 | mask = self.rgbToOnehot(mask).astype('float32') 100 | mask = mask.astype('float32') 101 | x[j] = image.astype('float32') 102 | y[j] = mask 103 | z[j] = knn_image 104 | 105 | return (x, z), y 106 | 107 | def getDataLoader(batch_size): 108 | """ Create dataloader and return dataloader object which can be used with 109 | model.fit 110 | """ 111 | input_img_paths = sorted([os.path.join(IMAGES_DIR, x) for x in os.listdir(IMAGES_DIR)]) 112 | target_img_paths = sorted([os.path.join(MASKS_DIR, x) for x in os.listdir(MASKS_DIR)]) 113 | 114 | X_train, X_test, y_train, y_test = train_test_split(input_img_paths, target_img_paths, test_size = TEST_SIZE, random_state = SEED) 115 | 116 | trainGen = ACLNetDataloader(batch_size = batch_size, resize_size = RESIZE_SIZE, crop_size = CROP_SIZE, input_img_paths = X_train, target_img_paths = y_train, data_type = "Train") 117 | testGen = ACLNetDataloader(batch_size = batch_size, resize_size = RESIZE_SIZE, crop_size = CROP_SIZE, input_img_paths = X_test, target_img_paths = y_test, data_type = "Test") 118 | 119 | return trainGen, testGen -------------------------------------------------------------------------------- /src/models/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/src/models/.gitkeep -------------------------------------------------------------------------------- /src/models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ckmvigil/ACLNet/c0cfc816ff22b9652d5d166f80dc6aa0e961e29e/src/models/__init__.py -------------------------------------------------------------------------------- /src/models/loss.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import tensorflow as tf 3 | 4 | def diceCoef(y_true, y_pred, smooth=tf.keras.backend.epsilon()): 5 | y_true_f = tf.keras.backend.flatten(y_true) 6 | y_pred_f = tf.keras.backend.flatten(y_pred) 7 | intersection = tf.keras.backend.sum(y_true_f * y_pred_f) 8 | return (2. * intersection + smooth) / (tf.keras.backend.sum(y_true_f * y_true_f) + tf.keras.backend.sum(y_pred_f * y_pred_f) + smooth) 9 | 10 | def diceCoefLoss(y_true, y_pred): 11 | return 1.0 - diceCoef(y_true, y_pred) 12 | 13 | def bceDiceLoss(y_true, y_pred): 14 | loss = tf.keras.losses.binary_crossentropy(y_true, y_pred) + diceCoefLoss(y_true, y_pred) 15 | return loss 16 | -------------------------------------------------------------------------------- /src/models/network.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import sys 3 | sys.path.append("../src") 4 | 5 | import tensorflow as tf 6 | from config import * 7 | 8 | def convBlock( 9 | block_input, 10 | num_filters = 256, 11 | kernel_size = 3, 12 | dilation_rate = 1, 13 | padding = "same", 14 | use_bias = False, 15 | ): 16 | x = tf.keras.layers.Conv2D( 17 | num_filters, 18 | kernel_size = kernel_size, 19 | dilation_rate = dilation_rate, 20 | padding = "same", 21 | use_bias = use_bias, 22 | kernel_initializer = tf.keras.initializers.HeNormal(), 23 | )(block_input) 24 | x = tf.keras.layers.BatchNormalization()(x) 25 | return tf.keras.layers.Activation("relu")(x) 26 | 27 | def DilatedSpatialPyramidPooling(dspp_input): 28 | dims = dspp_input.shape 29 | x = tf.keras.layers.AveragePooling2D(pool_size = (dims[-3], dims[-2]))(dspp_input) 30 | x = convBlock(x, kernel_size = 1, use_bias = True) 31 | out_pool = tf.keras.layers.UpSampling2D( 32 | size=(dims[-3] // x.shape[1], dims[-2] // x.shape[2]), interpolation = "bilinear", 33 | )(x) 34 | 35 | out_1 = convBlock(dspp_input, kernel_size = 1, dilation_rate = 1) 36 | out_6 = convBlock(dspp_input, kernel_size = 3, dilation_rate = 6) 37 | out_12 = convBlock(dspp_input, kernel_size = 3, dilation_rate = 12) 38 | out_18 = convBlock(dspp_input, kernel_size = 3, dilation_rate = 18) 39 | 40 | x = tf.keras.layers.Concatenate(axis=-1)([out_pool, out_1, out_6, out_12, out_18]) 41 | output = convBlock(x, kernel_size = 1) 42 | return output 43 | 44 | def getModel(): 45 | """ ACLNet network 46 | """ 47 | input_1 = tf.keras.layers.Input(shape = CROP_SIZE + (3,)) 48 | input_2 = tf.keras.layers.Input(shape = CROP_SIZE + (3,)) 49 | backbone = tf.keras.applications.EfficientNetB0( 50 | weights = "imagenet", include_top = False, input_tensor = input_1 51 | ) 52 | 53 | x = backbone.get_layer("block6a_expand_activation").output 54 | x = DilatedSpatialPyramidPooling(x) 55 | 56 | attn = tf.keras.layers.Resizing(18, 18)(input_1) 57 | attn = tf.keras.layers.Conv2D(256, 1, 1, 'same', use_bias=False)(attn) 58 | attn = tf.keras.layers.Activation('softmax')(attn) 59 | 60 | x = tf.keras.layers.Multiply()([x, attn]) 61 | input_a = tf.keras.layers.UpSampling2D( 62 | size = (CROP_SIZE[0] // 4 // x.shape[1], CROP_SIZE[1] // 4 // x.shape[2]), 63 | interpolation = "bilinear", 64 | )(x) 65 | input_b = backbone.get_layer("block3a_expand_activation").output 66 | input_b = convBlock(input_b, num_filters = 48, kernel_size = 1) 67 | 68 | knn_image = convBlock(input_2, kernel_size = 1) 69 | 70 | x = tf.keras.layers.Concatenate(axis=-1)([input_a, input_b]) 71 | x = convBlock(x) 72 | x = convBlock(x) 73 | x = tf.keras.layers.UpSampling2D( 74 | size = (CROP_SIZE[0] // x.shape[1], CROP_SIZE[1] // x.shape[2]), 75 | interpolation = "bilinear", 76 | )(x) 77 | x += knn_image 78 | x = tf.keras.layers.Conv2D(NUM_CLASSES, kernel_size = (1, 1), padding = "same")(x) 79 | out = tf.keras.layers.Activation('sigmoid', name='segmentation')(x) 80 | 81 | model = tf.keras.models.Model(inputs = [input_1, input_2], outputs = [out]) 82 | return model -------------------------------------------------------------------------------- /src/models/test_model.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import sys 3 | sys.path.append("../src") 4 | sys.path.append("../src/data/") 5 | 6 | from sklearn.metrics import matthews_corrcoef 7 | from sklearn.metrics import roc_curve, auc 8 | import matplotlib.pyplot as plt 9 | import tensorflow as tf 10 | from dataloader import * 11 | from tqdm import tqdm 12 | from config import * 13 | from utils import * 14 | from loss import * 15 | import numpy as np 16 | import logging 17 | import os 18 | 19 | def main(): 20 | """ testing performance of model. 21 | """ 22 | trainGen, testGen = getDataLoader(batch_size = 1) 23 | model = tf.keras.models.load_model(os.path.join(WEIGHTS_DIR, 'ACLNet_Best.h5'), custom_objects={"diceCoef":diceCoef, "bceDiceLoss":bceDiceLoss}) 24 | model.evaluate(testGen) 25 | original, prediction = [], [] 26 | with tqdm(total = int(6468*TEST_SIZE)) as pbar: 27 | for data in testGen: 28 | image, mask = data 29 | seg = model.predict(image) 30 | original.append(mask[0].argmax(-1)) 31 | prediction.append(seg[0].argmax(-1)) 32 | pbar.update(1) 33 | 34 | original = np.array(original) 35 | prediction = np.array(prediction) 36 | 37 | precisions, recalls, f1scores, error_rates = [], [], [], [] 38 | with tqdm(total = len(original)) as pbar: 39 | for orig, pred in zip(original, prediction): 40 | try: 41 | precision, recall, f1score, error_rate = score_card(pred, orig) 42 | precisions.append(precision) 43 | recalls.append(recall) 44 | f1scores.append(f1score) 45 | error_rates.append(error_rate) 46 | except: 47 | print('skipped') 48 | pbar.update(1) 49 | 50 | logging.info("[Info] Precision: {}".format(precisions)) 51 | logging.info("[Info] Recall: {}".format(recalls)) 52 | logging.info("[Info] F1-Score: {}".format(f1scores)) 53 | logging.info("[Info] Error Rate: {}".format(error_rates)) 54 | 55 | logging.info("[Info] Matthews Correlation Coefficient (MCC): ") 56 | logging.info(matthews_corrcoef(original.ravel(), prediction.ravel())) 57 | 58 | logging.info("[Info] ROC_AUC Curve: ") 59 | fpr, tpr, thresholds = roc_curve(original.ravel(), prediction.ravel()) 60 | auc_score = auc(fpr, tpr) 61 | 62 | fig, ax = plt.subplots(1,1) 63 | ax.plot(fpr, tpr, label='ROC curve WSCN (area = %0.4f)' % auc_score) 64 | ax.plot([0, 1], [0, 1], 'k--') 65 | ax.set_xlim([0.0, 1.0]) 66 | ax.set_ylim([0.0, 1.05]) 67 | ax.set_xlabel('False Positive Rate') 68 | ax.set_ylabel('True Positive Rate') 69 | ax.set_title('Receiver Operating Characteristic') 70 | ax.legend(loc="lower right") 71 | 72 | plt.savefig(os.path.join(INFERENCE_DIR, "ROC_Curve_WSCN.pdf")) 73 | 74 | if __name__ == '__main__': 75 | logging.basicConfig(level = logging.INFO, filename = os.path.join(LOG_DIR, 'app.log'), format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', filemode='w') 76 | 77 | sys.stdout = LoggerWriter(logging.info) 78 | sys.stderr = LoggerWriter(logging.error) 79 | 80 | main() 81 | -------------------------------------------------------------------------------- /src/models/train_model.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import sys 3 | sys.path.append("../src") 4 | sys.path.append("../src/data/") 5 | 6 | import tensorflow as tf 7 | from dataloader import * 8 | from network import * 9 | from loss import * 10 | from config import * 11 | from utils import * 12 | import logging 13 | import os 14 | 15 | def main(): 16 | """ training WaferSegClassNet model 17 | """ 18 | logger = logging.getLogger(__name__) 19 | logger.info("[Info] Getting DataLoader") 20 | trainGen, testGen = getDataLoader(batch_size=BATCH_SIZE) 21 | logger.info("[Info] Creating Network") 22 | model = getModel() 23 | logger.info("[Info] Summary of model \n") 24 | logger.info(model.summary()) 25 | 26 | model.compile(optimizer = tf.keras.optimizers.Adam(INITIAL_LEARNING_RATE), loss = bceDiceLoss, metrics = [diceCoef]) 27 | callbacks = [ 28 | tf.keras.callbacks.ModelCheckpoint(os.path.join(WEIGHTS_DIR, 'ACLNet_Best.h5'), monitor = 'val_diceCoef', mode="max", verbose = 1, save_best_only = True, save_weights_only = False), 29 | tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_diceCoef', mode="max", factor = 0.1, patience = 20, min_lr = 0.00001) 30 | ] 31 | model.fit(trainGen, validation_data = testGen, epochs = EPOCHS, verbose = 1, callbacks = callbacks) 32 | logger.info("[Info] Training Finished") 33 | 34 | if __name__ == '__main__': 35 | logging.basicConfig(level = logging.INFO, filename = os.path.join(LOG_DIR, 'app.log'), format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', filemode='w') 36 | 37 | sys.stdout = LoggerWriter(logging.info) 38 | sys.stderr = LoggerWriter(logging.error) 39 | 40 | main() 41 | -------------------------------------------------------------------------------- /src/utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | class LoggerWriter: 4 | def __init__(self, logfct): 5 | self.logfct = logfct 6 | self.buf = [] 7 | 8 | def write(self, msg): 9 | if msg.endswith('\n'): 10 | self.buf.append(msg.rstrip('\n')) 11 | self.logfct(''.join(self.buf)) 12 | self.buf = [] 13 | else: 14 | self.buf.append(msg) 15 | 16 | def flush(self): 17 | pass 18 | 19 | def score_card(input_map, groundtruth_image, threshold = 0.5): 20 | 21 | binary_map = input_map 22 | binary_map[binary_map < threshold ] = 0 23 | binary_map[binary_map == threshold ] = 0 24 | binary_map[binary_map > threshold ] = 1 25 | [rows,cols] = groundtruth_image.shape 26 | 27 | TP = 0 28 | FP = 0 29 | TN = 0 30 | FN = 0 31 | 32 | for i in range(rows): 33 | for j in range(cols): 34 | if (groundtruth_image[i,j]==1 and binary_map[i,j]==1): #TP condition 35 | TP = TP + 1 36 | elif ((groundtruth_image[i,j]==0) and (binary_map[i,j]==1)): #FP condition 37 | FP = FP + 1 38 | elif ((groundtruth_image[i,j]==0) and (binary_map[i,j]==0)): #TN condition 39 | TN = TN + 1 40 | elif ((groundtruth_image[i,j]==1) and (binary_map[i,j]==0)): #FN condition 41 | FN = FN + 1 42 | 43 | precision = float(TP) / float(TP + FP) 44 | recall = float(TP) / float(TP + FN) 45 | fScore = float(2 * precision * recall)/float(precision + recall) 46 | 47 | error_count = 0 48 | for i in range(rows): 49 | for j in range(cols): 50 | if (groundtruth_image[i, j] != binary_map[i, j]): 51 | error_count = error_count + 1 52 | error_rate = float(error_count) / float(rows * cols) 53 | 54 | return (precision, recall, fScore, error_rate) -------------------------------------------------------------------------------- /weights/.gitkeep: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /weights/README.md: -------------------------------------------------------------------------------- 1 | ## Weights: 2 | 3 | All the trained checkpoints for model training will be saved here. 4 | --------------------------------------------------------------------------------