├── .gitignore ├── LICENSE ├── MODELS.md ├── README.md ├── configs ├── example.json ├── example_val.json ├── fcn_example.json ├── tfcn_example.json ├── tfcn_example_real_eval.json ├── tfcn_example_real_pred.json ├── tfcn_example_real_train.json └── tfcn_example_real_train_and_eval.json ├── eoflow ├── __init__.py ├── base │ ├── __init__.py │ ├── base_input.py │ ├── base_model.py │ ├── base_task.py │ └── configuration.py ├── execute.py ├── input │ ├── __init__.py │ ├── eopatch.py │ ├── hdf5.py │ ├── numpy.py │ ├── operations.py │ └── random.py ├── models │ ├── __init__.py │ ├── callbacks.py │ ├── classification_base.py │ ├── classification_temp_nets.py │ ├── layers.py │ ├── losses.py │ ├── metrics.py │ ├── pse_tae_layers.py │ ├── segmentation_base.py │ ├── segmentation_unets.py │ └── transformer_encoder_layers.py ├── tasks │ ├── __init__.py │ ├── evaluate.py │ ├── predict.py │ └── train.py └── utils │ ├── __init__.py │ ├── tf_utils.py │ └── utils.py ├── examples ├── exporting_data.ipynb ├── input.py ├── models.py └── notebook.ipynb ├── figures ├── fcn-architecture.png ├── rfcn-architecture.png └── tfcn-architecture.png ├── requirements-dev.txt ├── requirements.txt ├── setup.py └── tests ├── test_layers.py ├── test_losses.py └── test_metrics.py /.gitignore: -------------------------------------------------------------------------------- 1 | # pyc files 2 | *.pyc 3 | 4 | # VS files 5 | .vscode/ 6 | 7 | # temp output folder 8 | temp/ 9 | 10 | # pycharm files 11 | .idea/ 12 | 13 | # egg folder 14 | *.egg-info/ 15 | 16 | # Jupyter checkpoints 17 | .ipynb_checkpoints -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | MIT License 3 | 4 | Copyright (c) 2017-2020 Matej Aleksandrov, Matej Batič, Matic Lubej, Grega Milčinski (Sinergise) 5 | Copyright (c) 2017-2020 Devis Peressutti, Jernej Puc, Anže Zupanc, Lojze Žust, Jovan Višnjić (Sinergise) 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy 8 | of this software and associated documentation files (the "Software"), to deal 9 | in the Software without restriction, including without limitation the rights 10 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | copies of the Software, and to permit persons to whom the Software is 12 | furnished to do so, subject to the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be included in all 15 | copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | SOFTWARE. 24 | -------------------------------------------------------------------------------- /MODELS.md: -------------------------------------------------------------------------------- 1 | # Models 2 | 3 | ## Fully-Convolutional-Network (FCN) 4 | 5 | The vanilla architecture as in the following figure is implemented. Convolutions are run along spatial dimensions of input tensor, which is supposed to have `[M, H, W, D]` shape, where M is the mini-batch size, and H, W and D are the height, width and number of bands (i.e. depth) of the input image tensor. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size. 6 | 7 | ![FCN](./figures/fcn-architecture.png "FCN") 8 | 9 | An example training script is provided. To run it execute the `configs/fcn_example.json` configuration: 10 | ``` 11 | python -m eoflow.execute configs/fcn_example.json 12 | ``` 13 | 14 | The example configuration can be used as a base to run your own experiments. 15 | 16 | ## Temporal Fully-Convolutional-Network (TFCN) 17 | 18 | Similarly to the RFCN, the TFCN works with time-series of input shape `[M, T, H, W, D]`. This network performs 3d convolutions along the tempo-spatial dimensions, i.e. the convolutional kernels are 3d `k x k x k`. As default, the temporal dimension is not pooled. For temporal pooling, enough time-frames need to be available in the input tensors. At the bottom of the TFCN and along the skip connections, a 1d convolution along the temporal dimension is performed to linearly combine the temporal features. The resulting tensors are 4d of shape `[M, H, W, D]`. The decoding path is as in FCN. 19 | 20 | ![TFCN](./figures/tfcn-architecture.png "TFCN") 21 | 22 | An example training script is provided. To run it execute the `configs/tfcn_example.json` configuration: 23 | ``` 24 | python -m eoflow.execute configs/tfcn_example.json 25 | ``` 26 | 27 | The example configuration can be used as a base to run your own experiments. 28 | 29 | ## Recurrent Fully-Convolutional-Network (RFCN) 30 | 31 | A recurrent version of the **FCN** is implemented as in below figure. The input tensor in this case is 5d with shape `[M, T, H, W, D]`, where `T` is the number of temporal acquisitions. As for the FCN, the 2d convolutions operate along the `H` and `W` dimensions. The recurrent layers are applied along the skip connections and the bottom layers to model the temporal relationship between the features extracted by the 2d convolutions. The output of the recurrent layers is a 4d tensor of shape `[M, H, W, D]` (the height, width and depth of the tensors will vary along the network). The decoding path is as in **FCN**. The 2d convolutions perform a `VALID` convolution, therefore the output tensor size is smaller than the input size. 32 | 33 | ![RFCN](./figures/rfcn-architecture.png "RFCN") 34 | 35 | An example training script is provided. To run it execute the `configs/rfcn_example.json` configuration: 36 | ``` 37 | python -m eoflow.execute configs/rfcn_example.json 38 | ``` 39 | 40 | The example configuration can be used as a base to run your own experiments. 41 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EOFlow 2 | 3 | This repository provides code and examples for creation of Earth Observation (EO) projects using TensorFlow. The code uses TensorFlow 2.0 with Keras as the main model building API. 4 | 5 | Common model architectures, layers, and input methods for EO tasks are provided in the package `eoflow`. Custom models and input methods can also be implemented building on top of the provided abstract classes. This package aims at seamlessly integrate with [`eo-learn`](https://github.com/sentinel-hub/eo-learn), and favours both creation of models for prototypying as well as production of EO applications. 6 | 7 | Architectures and examples for land cover and crop classification using time-series derived from satellite images are provided. 8 | 9 | ## Installation 10 | 11 | The package can be installed by running the following command. 12 | ``` 13 | $ pip install git+https://github.com/sentinel-hub/eo-flow 14 | ``` 15 | 16 | You can also install the package from source. Clone the repository and run the following command in the root directory of the project. 17 | ``` 18 | $ pip install . 19 | ``` 20 | 21 | ## Getting started 22 | 23 | The `eoflow` package can be used in two ways. For best control over the workflow and faster prototyping, the package can be used programmatically (in code). The [example notebook](examples/notebook.ipynb) should help you get started with that. It demonstrates how to prepare a dataset pipeline, train the model, evaluate the model and make predictions using the trained model. 24 | 25 | An alternate way of using `eoflow` is by writing configuration `json` files and running them using `eoflow`'s execute script. Configuration files specify and configure the task (training, evaluation, etc.) and contain the configurations of the model and input methods. Example configurations are provided in the `configs` directory. Once a configuration file is created it can be run using the execute command. 26 | 27 | A simple example can be run using the following command. More advanced configurations are also provided. 28 | ``` 29 | $ python -m eoflow.execute configs/example.json 30 | ``` 31 | 32 | This will create an output folder `temp/experiment` containing the tensorboard logs and model checkpoints. 33 | 34 | To visualize the logs in TensorBoard, run 35 | ``` 36 | $ tensorboard --logdir=temp/experiment 37 | ``` 38 | 39 | ## Writing custom code 40 | 41 | To get started with writing custom models and input methods for `eoflow` take a look at the example implementations ([`examples` folder](examples/)). Custom classes use schemas to define the configuration parameters in order to work with the execute script and configuration files. Since eoflow builds on top of TF2 and Keras, model building is very similar. 42 | 43 | ## Package structure 44 | 45 | The subpackages of `eoflow` are as follows: 46 | * `base`: this directory contains the abstract classes to build models, inputs and tasks. Any useful abstract class should go in this folder. 47 | * `models`: classes implementing the TF models (e.g. Fully-Convolutional-Network, GANs, seq2seq, ...). These classes inherit and implement the `BaseModel` abstract class. The module also contains custom losses, metrics and layers. 48 | * `tasks`: classes handling the configurable actions that can be applied to each TF model, when using the execute script. These actions may include training, inference, exporting the model, validation, etc. The tasks inherit the `BaseTask` abstract class. 49 | * `input`: building blocks and helper methods for loading the input data (EOPatch, numpy arrays, etc.) into a tensoflow Dataset and applying different transformations (data augmentation, patch extraction) 50 | * `utils`: collection of utility functions 51 | 52 | ### Examples and scripts 53 | 54 | Project also contains other folders: 55 | * `configs`: folder containing example configurations for different models. Config parameters are stored in .json files. Results of an experiment should be reproducible by re-running the same config file. Config files specify the whole workflow (model, task, data input if required). 56 | * `examples`: folder containing example implementations of custom models and input functions. Also contains a jupyter notebook example. 57 | 58 | ## Implemented architectures 59 | 60 | Segmentation models for land cover semantic segmentation: 61 | * **Fully-Convolutional-Network (FCN, a.k.a. U-net)**, vanilla implementation of method described in this [paper](https://arxiv.org/abs/1505.04597). This network expects 2D MSI images as inputs and predicts 2D label maps as output. 62 | * **Temporal FCN**, where the whole time-series is considered as a 3D MSI volume and convolutions are performed along the temporal dimension as well spatial dimension. The output of the network is a 2D label map as in previous cases. More details can be found in this [paper](https://www.researchgate.net/publication/333262625_Spatio-Temporal_Deep_Learning_An_Application_to_Land_Cover_Classification). 63 | * **ResUNet-a**, architecture proposed in Diakogiannis et al. ["ResUNet-a: A deep learning framework for semantic segmetnation of remotely sensed data"](https://www.sciencedirect.com/science/article/abs/pii/S0924271620300149). Original `mxnet` implementation can be found [here](https://github.com/feevos/resuneta). 64 | 65 | Classification models for crop classification using time-series: 66 | * **TCN**: Implementation of the TCN network taken from the [keras-TCN implementation by Philippe Remy](https://github.com/philipperemy/keras-tcn). 67 | * **TempCNN**: Implementation of the TempCNN network taken from the [temporalCNN implementation of Charlotte Pelletier](https://github.com/charlotte-pel/temporalCNN). 68 | * **Recurrent NN**: Implementation of (bidirectional) Recurrent Neural Networks for the classification of time-series. Implementation allows to use either `SimpleRNN`, `GRU` or `LSTM` layers as building blocks of the architecture. 69 | * **TransformerEncoder**: Implementation of a time-series classification architecture based on [self-attention](https://arxiv.org/abs/1706.03762) layers. This implementation follows [this PyTorch implementation of Marc Russwurm](https://github.com/MarcCoru/crop-type-mapping). 70 | * **PSE+TAE**: Implementation of the Pixel-Set Encoder and temporal Self-attention proposed in Garnot V. _et al._ 71 | ["Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention"](https://hal.archives-ouvertes.fr/hal-02879223/document). 72 | This implementation is adapted from the [Pytorch version](https://github.com/VSainteuf/pytorch-psetae). 73 | 74 | Descriptions and examples of semantic segmentation architectures are available [here](MODELS.md). 75 | -------------------------------------------------------------------------------- /configs/example.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "examples.models.ExampleModel", 4 | "config": { 5 | "output_size": 10, 6 | "hidden_units": 256, 7 | "learning_rate": 0.001 8 | } 9 | }, 10 | "task": { 11 | "classname": "eoflow.tasks.TrainTask", 12 | "config": { 13 | "num_epochs": 10, 14 | "model_directory": "./temp/experiment", 15 | "input_config":{ 16 | "classname": "examples.input.ExampleInput", 17 | "config": { 18 | "input_shape": [512], 19 | "num_classes": 10, 20 | "batch_size": 20, 21 | "batches_per_epoch": 200 22 | } 23 | }, 24 | "iterations_per_epoch": 200, 25 | "save_steps": 400, 26 | "summary_steps": 50 27 | } 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /configs/example_val.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "examples.models.ExampleModel", 4 | "config": { 5 | "output_size": 10, 6 | "hidden_units": 256, 7 | "learning_rate": 0.01 8 | } 9 | }, 10 | "task": { 11 | "classname": "eoflow.tasks.TrainAndEvaluateTask", 12 | "config": { 13 | "num_epochs": 10, 14 | "iterations_per_epoch": 1000, 15 | "model_directory": "./temp/experiment", 16 | "train_input_config":{ 17 | "classname": "examples.input.ExampleInput", 18 | "config": { 19 | "input_shape": [512], 20 | "num_classes": 10, 21 | "batch_size": 20, 22 | "batches_per_epoch": 200 23 | } 24 | }, 25 | "val_input_config":{ 26 | "classname": "examples.input.ExampleInput", 27 | "config": { 28 | "input_shape": [512], 29 | "num_classes": 10, 30 | "batch_size": 1, 31 | "batches_per_epoch": 200 32 | } 33 | }, 34 | "save_steps": 400, 35 | "summary_steps": 50 36 | } 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /configs/fcn_example.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.FCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 3, 8 | "keep_prob": 0.8, 9 | "features_root": 32, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "VALID", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "loss": "focal_loss", 20 | "metrics": ["accuracy"] 21 | } 22 | }, 23 | "task": { 24 | "classname": "eoflow.tasks.TrainTask", 25 | "config": { 26 | "num_epochs": 2, 27 | "model_directory": "./temp/experiment_fcn", 28 | "input_config":{ 29 | "classname": "eoflow.input.random.RandomSegmentationInput", 30 | "config": { 31 | "input_shape": [128, 128, 13], 32 | "output_shape": [128, 128], 33 | "num_classes": 3, 34 | "batch_size": 2, 35 | "batches_per_epoch": 200 36 | } 37 | }, 38 | "iterations_per_epoch": 50, 39 | "save_steps": 100, 40 | "summary_steps": 50 41 | } 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /configs/tfcn_example.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.TFCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 2, 8 | "keep_prob": 0.8, 9 | "features_root": 16, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "VALID", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "pool_time": false, 20 | "single_encoding_conv": true, 21 | "conv_size_reduce": 3, 22 | "loss": "cross_entropy" 23 | } 24 | }, 25 | "task": { 26 | "classname": "eoflow.tasks.TrainTask", 27 | "config": { 28 | "num_epochs": 2, 29 | "model_directory": "./temp/experiment_tfcn", 30 | "input_config":{ 31 | "classname": "eoflow.input.random.RandomSegmentationInput", 32 | "config": { 33 | "input_shape": [10, 128, 128, 13], 34 | "output_shape": [128, 128], 35 | "num_classes": 2, 36 | "batch_size": 1, 37 | "batches_per_epoch": 100 38 | } 39 | }, 40 | "save_steps": 100, 41 | "summary_steps": 50 42 | } 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /configs/tfcn_example_real_eval.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.TFCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 10, 8 | "keep_prob": 0.8, 9 | "features_root": 16, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "VALID", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "pool_time": false, 20 | "single_encoding_conv": true, 21 | "conv_size_reduce": 3 22 | } 23 | }, 24 | "task": { 25 | "classname": "eoflow.tasks.EvaluateTask", 26 | "config": { 27 | "model_directory": "./temp/tfcn_svn_lulc", 28 | "input_config":{ 29 | "classname": "examples.input.EOPatchInputExample", 30 | "config": { 31 | "data_dir": "/storage/jupyter/data/svn_lulc/data/train-val-linear-v01-aws/test", 32 | "input_feature_type": "data", 33 | "input_feature_name": "FEATURES", 34 | "input_feature_axis": [1,2], 35 | "input_feature_shape": [23, -1, -1, 9], 36 | "labels_feature_type": "mask_timeless", 37 | "labels_feature_name": "LULC_RABA", 38 | "labels_feature_axis": [0,1], 39 | "labels_feature_shape": [-1, -1, 1], 40 | "patch_size": [128, 128], 41 | "interleave_size": 3, 42 | "batch_size": 1, 43 | "num_classes": 10, 44 | "cache_file": "./temp/tfcn_svn_lulc/dataset/cache_val", 45 | "num_subpatches": 5 46 | } 47 | } 48 | } 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /configs/tfcn_example_real_pred.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.TFCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 10, 8 | "keep_prob": 0.8, 9 | "features_root": 16, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "VALID", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "pool_time": false, 20 | "single_encoding_conv": true, 21 | "conv_size_reduce": 3 22 | } 23 | }, 24 | "task": { 25 | "classname": "eoflow.tasks.PredictTask", 26 | "config": { 27 | "model_directory": "./temp/tfcn_svn_lulc", 28 | "input_config":{ 29 | "classname": "examples.input.EOPatchInputExample", 30 | "config": { 31 | "data_dir": "/storage/jupyter/data/svn_lulc/data/train-val-linear-v01-aws/test", 32 | "input_feature_type": "data", 33 | "input_feature_name": "FEATURES", 34 | "input_feature_axis": [1,2], 35 | "input_feature_shape": [23, -1, -1, 9], 36 | "labels_feature_type": "mask_timeless", 37 | "labels_feature_name": "LULC_RABA", 38 | "labels_feature_axis": [0,1], 39 | "labels_feature_shape": [-1, -1, 1], 40 | "patch_size": [128, 128], 41 | "interleave_size": 3, 42 | "batch_size": 1, 43 | "num_classes": 10, 44 | "cache_file": "./temp/tfcn_svn_lulc/dataset/cache_val", 45 | "num_subpatches": 5 46 | } 47 | } 48 | } 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /configs/tfcn_example_real_train.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.TFCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 10, 8 | "keep_prob": 0.8, 9 | "features_root": 16, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "SAME", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "pool_time": false, 20 | "single_encoding_conv": true, 21 | "conv_size_reduce": 3, 22 | "loss": "focal_loss", 23 | "metrics": ["accuracy", "iou"], 24 | "prediction_visualization": true 25 | } 26 | }, 27 | "task": { 28 | "classname": "eoflow.tasks.TrainTask", 29 | "config": { 30 | "num_epochs": 50, 31 | "iterations_per_epoch": 10, 32 | "model_directory": "./temp/tfcn_svn_lulc", 33 | "input_config":{ 34 | "classname": "examples.input.EOPatchInputExample", 35 | "config": { 36 | "data_dir": "/home/devis/Desktop/train-val-linear-v01-aws/train", 37 | "input_feature_type": "data", 38 | "input_feature_name": "FEATURES", 39 | "input_feature_axis": [1,2], 40 | "input_feature_shape": [23, -1, -1, 9], 41 | "labels_feature_type": "mask_timeless", 42 | "labels_feature_name": "LULC_RABA", 43 | "labels_feature_axis": [0,1], 44 | "labels_feature_shape": [-1, -1, 1], 45 | "patch_size": [128, 128], 46 | "interleave_size": 3, 47 | "batch_size": 2, 48 | "num_classes": 10, 49 | "cache_file": "./temp/tfcn_svn_lulc/dataset/cache", 50 | "num_subpatches": 5, 51 | "data_augmentation": true 52 | } 53 | }, 54 | "save_steps": 100, 55 | "summary_steps": 1 56 | } 57 | } 58 | } 59 | -------------------------------------------------------------------------------- /configs/tfcn_example_real_train_and_eval.json: -------------------------------------------------------------------------------- 1 | { 2 | "model": { 3 | "classname": "eoflow.models.TFCNModel", 4 | "config": { 5 | "learning_rate": 0.0001, 6 | "n_layers": 3, 7 | "n_classes": 10, 8 | "keep_prob": 0.8, 9 | "features_root": 16, 10 | "conv_size": 3, 11 | "conv_stride": 1, 12 | "deconv_size": 2, 13 | "add_dropout": true, 14 | "add_batch_norm": false, 15 | "bias_init": 0.0, 16 | "padding": "SAME", 17 | "pool_size": 2, 18 | "pool_stride": 2, 19 | "pool_time": false, 20 | "single_encoding_conv": true, 21 | "conv_size_reduce": 3, 22 | "loss": "focal_loss", 23 | "metrics": ["accuracy", "iou"], 24 | "prediction_visualization": true 25 | } 26 | }, 27 | "task": { 28 | "classname": "eoflow.tasks.TrainAndEvaluateTask", 29 | "config": { 30 | "num_epochs": 100, 31 | "iterations_per_epoch": 10, 32 | "model_directory": "./temp/tfcn_svn_lulc", 33 | "train_input_config":{ 34 | "classname": "examples.input.EOPatchInputExample", 35 | "config": { 36 | "data_dir": "/home/devis/Desktop/train-val-linear-v01-aws/train", 37 | "input_feature_type": "data", 38 | "input_feature_name": "FEATURES", 39 | "input_feature_axis": [1,2], 40 | "input_feature_shape": [23, -1, -1, 9], 41 | "labels_feature_type": "mask_timeless", 42 | "labels_feature_name": "LULC_RABA", 43 | "labels_feature_axis": [0,1], 44 | "labels_feature_shape": [-1, -1, 1], 45 | "patch_size": [128, 128], 46 | "interleave_size": 3, 47 | "batch_size": 2, 48 | "num_classes": 10, 49 | "cache_file": "./temp/tfcn_svn_lulc/dataset/cache_train", 50 | "num_subpatches": 5, 51 | "data_augmentation": true 52 | } 53 | }, 54 | "val_input_config":{ 55 | "classname": "examples.input.EOPatchInputExample", 56 | "config": { 57 | "data_dir": "/home/devis/Desktop/train-val-linear-v01-aws/test", 58 | "input_feature_type": "data", 59 | "input_feature_name": "FEATURES", 60 | "input_feature_axis": [1,2], 61 | "input_feature_shape": [23, -1, -1, 9], 62 | "labels_feature_type": "mask_timeless", 63 | "labels_feature_name": "LULC_RABA", 64 | "labels_feature_axis": [0,1], 65 | "labels_feature_shape": [-1, -1, 1], 66 | "patch_size": [128, 128], 67 | "interleave_size": 3, 68 | "batch_size": 2, 69 | "num_classes": 10, 70 | "cache_file": "./temp/tfcn_svn_lulc/dataset/cache_val", 71 | "num_subpatches": 5 72 | } 73 | }, 74 | "save_steps": 10, 75 | "summary_steps": 1 76 | } 77 | } 78 | } 79 | -------------------------------------------------------------------------------- /eoflow/__init__.py: -------------------------------------------------------------------------------- 1 | name = 'eoflow' 2 | -------------------------------------------------------------------------------- /eoflow/base/__init__.py: -------------------------------------------------------------------------------- 1 | from .configuration import Configurable 2 | from .base_input import BaseInput 3 | from .base_task import BaseTask 4 | from .base_model import BaseModel 5 | -------------------------------------------------------------------------------- /eoflow/base/base_input.py: -------------------------------------------------------------------------------- 1 | from . import Configurable 2 | 3 | 4 | class BaseInput(Configurable): 5 | def get_dataset(self): 6 | """Builds and returns a tensorflow Dataset object for reading the data.""" 7 | 8 | raise NotImplementedError 9 | -------------------------------------------------------------------------------- /eoflow/base/base_model.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import tensorflow as tf 4 | 5 | from . import Configurable 6 | 7 | 8 | class BaseModel(tf.keras.Model, Configurable): 9 | def __init__(self, config_specs): 10 | tf.keras.Model.__init__(self) 11 | Configurable.__init__(self, config_specs) 12 | 13 | self.net = None 14 | 15 | self.init_model() 16 | 17 | def init_model(self): 18 | """ Called on __init__. Keras model initialization. Create model here if does not require the inputs shape """ 19 | pass 20 | 21 | def build(self, inputs_shape): 22 | """ Keras method. Called once to build the model. Build the model here if the input shape is required. """ 23 | pass 24 | 25 | def call(self, inputs, training=False): 26 | """ Runs the model with inputs. """ 27 | pass 28 | 29 | def prepare(self, optimizer=None, loss=None, metrics=None, **kwargs): 30 | """ Prepares the model for training and evaluation. This method should create the 31 | optimizer, loss and metric functions and call the compile method of the model. The model 32 | should provide the defaults for the optimizer, loss and metrics, which can be overriden 33 | with custom arguments. """ 34 | 35 | raise NotImplementedError 36 | 37 | def load_latest(self, model_directory): 38 | """ Loads weights from the latest checkpoint in the model directory. """ 39 | 40 | checkpoints_path = os.path.join(model_directory, 'checkpoints', 'model.ckpt') 41 | 42 | return self.load_weights(checkpoints_path).expect_partial() 43 | 44 | def _fit(self, 45 | dataset, 46 | num_epochs, 47 | model_directory, 48 | iterations_per_epoch, 49 | val_dataset=None, 50 | save_steps=100, 51 | summary_steps='epoch', 52 | callbacks=[], 53 | **kwargs): 54 | """ Trains and evaluates the model on a given dataset, saving the model and recording summaries. """ 55 | logs_path = os.path.join(model_directory, 'logs') 56 | checkpoints_path = os.path.join(model_directory, 'checkpoints', 'model.ckpt') 57 | 58 | # Tensorboard callback 59 | tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=logs_path, 60 | update_freq=summary_steps, 61 | profile_batch=0) 62 | 63 | # Checkpoint saving callback 64 | checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(checkpoints_path, 65 | save_best_only=True, 66 | save_freq=save_steps, 67 | save_weights_only=True) 68 | return self.fit(dataset, 69 | validation_data=val_dataset, 70 | epochs=num_epochs, 71 | steps_per_epoch=iterations_per_epoch, 72 | callbacks=[tensorboard_callback, checkpoint_callback] + callbacks, 73 | **kwargs) 74 | 75 | def train(self, 76 | dataset, 77 | num_epochs, 78 | model_directory, 79 | iterations_per_epoch=None, 80 | save_steps=100, 81 | summary_steps='epoch', 82 | callbacks=[], 83 | **kwargs): 84 | """ Trains the model on a given dataset. Takes care of saving the model and recording summaries. 85 | 86 | :param dataset: A tf.data Dataset containing the input training data. 87 | The dataset must be of shape (features, labels) where features and labels contain the data 88 | in the shape required by the model. 89 | :type dataset: tf.data.Dataset 90 | :param num_epochs: Number of epochs. One epoch is equal to one pass over the dataset. 91 | :type num_epochs: int 92 | :param model_directory: Output directory, where the model checkpoints and summaries are saved. 93 | :type model_directory: str 94 | :param iterations_per_epoch: Number of training steps to make every epoch. 95 | Training dataset is repeated automatically when the end is reached. 96 | :type iterations_per_epoch: int 97 | :param save_steps: Number of steps between saving model checkpoints. 98 | :type save_steps: int 99 | :param summary_steps: Number of steps between recording summaries. 100 | :type summary_steps: str or int 101 | :param callbacks: Customised Keras callbacks to use during training/evaluation 102 | :type callbacks: tf.keras.callbacks 103 | 104 | Other keyword parameters are passed to the Model.fit method. 105 | """ 106 | 107 | return self._fit(dataset if iterations_per_epoch is None else dataset.repeat(), 108 | num_epochs=num_epochs, 109 | model_directory=model_directory, 110 | iterations_per_epoch=iterations_per_epoch, 111 | save_steps=save_steps, 112 | summary_steps=summary_steps, 113 | callbacks=callbacks, 114 | **kwargs) 115 | 116 | def train_and_evaluate(self, 117 | train_dataset, 118 | val_dataset, 119 | num_epochs, 120 | iterations_per_epoch, 121 | model_directory, 122 | save_steps=100, summary_steps='epoch', callbacks=[], **kwargs): 123 | """ Trains the model on a given dataset. At the end of each epoch an evaluation is performed on the provided 124 | validation dataset. Takes care of saving the model and recording summaries. 125 | 126 | :param train_dataset: A tf.data Dataset containing the input training data. 127 | The dataset must be of shape (features, labels) where features and labels contain the data 128 | in the shape required by the model. 129 | :type train_dataset: tf.data.Dataset 130 | :param val_dataset: Same as for `train_dataset`, but for the validation data. 131 | :type val_dataset: tf.data.Dataset 132 | :param num_epochs: Number of epochs. Epoch size is independent from the dataset size. 133 | :type num_epochs: int 134 | :param iterations_per_epoch: Number of training steps to make every epoch. 135 | Training dataset is repeated automatically when the end is reached. 136 | :type iterations_per_epoch: int 137 | :param model_directory: Output directory, where the model checkpoints and summaries are saved. 138 | :type model_directory: str 139 | :param save_steps: Number of steps between saving model checkpoints. 140 | :type save_steps: int 141 | :param summary_steps: Number of steps between recodring summaries. 142 | :type summary_steps: str or int 143 | :param callbacks: Customised Keras callbacks to use during training/evaluation 144 | :type callbacks: tf.keras.callbacks 145 | 146 | Other keyword parameters are passed to the Model.fit method. 147 | """ 148 | return self._fit(train_dataset.repeat(), 149 | num_epochs, 150 | model_directory, 151 | iterations_per_epoch, 152 | val_dataset=val_dataset, 153 | save_steps=save_steps, 154 | summary_steps=summary_steps, 155 | callbacks=callbacks, 156 | **kwargs) 157 | -------------------------------------------------------------------------------- /eoflow/base/base_task.py: -------------------------------------------------------------------------------- 1 | from . import Configurable, BaseInput 2 | from ..utils import parse_classname 3 | 4 | 5 | class BaseTask(Configurable): 6 | def __init__(self, model, config_specs): 7 | super().__init__(config_specs) 8 | 9 | self.model = model 10 | 11 | @staticmethod 12 | def parse_input(input_config): 13 | """ Builds the input dataset using the provided configuration. """ 14 | 15 | classname, config = input_config.classname, input_config.config 16 | 17 | cls = parse_classname(classname) 18 | if not issubclass(cls, BaseInput): 19 | raise ValueError("Data input class does not inherit from BaseInput.") 20 | 21 | model_input = cls(config) 22 | 23 | return model_input.get_dataset() 24 | 25 | def run(self): 26 | """Executes the task.""" 27 | 28 | raise NotImplementedError 29 | -------------------------------------------------------------------------------- /eoflow/base/configuration.py: -------------------------------------------------------------------------------- 1 | from abc import ABC 2 | import inspect 3 | import json 4 | 5 | from marshmallow import Schema, fields, EXCLUDE 6 | from munch import Munch 7 | 8 | 9 | def dict_to_munch(obj): 10 | """ Recursively convert a dict to Munch. (there is a Munch.from_dict method, but it's not python3 compatible) 11 | """ 12 | if isinstance(obj, list): 13 | return [dict_to_munch(element) for element in obj] 14 | if isinstance(obj, dict): 15 | return Munch({k: dict_to_munch(v) for k, v in obj.items()}) 16 | return obj 17 | 18 | 19 | class ObjectConfiguration(Schema): 20 | classname = fields.String(required=True, description="Class to instantiate.") 21 | config = fields.Dict(required=True, descripton="Configuration used for instantiation of the class.") 22 | 23 | 24 | class Configurable(ABC): 25 | """ Base class for all configurable objects. 26 | """ 27 | 28 | def __init__(self, config_specs): 29 | self.schema = self.initialize_schema() 30 | self.config = self._prepare_config(config_specs) 31 | 32 | @classmethod 33 | def initialize_schema(cls): 34 | """ A Schema should be provided as an internal class of any class that inherits from Configurable. 35 | This method finds the Schema by traversing the inheritance tree. If no Schema is provided or inherited 36 | an error is raised. 37 | """ 38 | for item in vars(cls).values(): 39 | if inspect.isclass(item) and issubclass(item, Schema): 40 | return item() 41 | 42 | if len(cls.__bases__) > 1: 43 | raise RuntimeError('Class does not have a defined schema however it inherits from multiple ' 44 | 'classes. Which one should schema be inherited from?') 45 | 46 | parent_class = cls.__bases__[0] 47 | 48 | if parent_class is Configurable: 49 | raise NotImplementedError('Configuration schema not provided.') 50 | 51 | return parent_class.initialize_schema() 52 | 53 | def _prepare_config(self, config_specs): 54 | """ Collects and validates configuration dictionary 55 | """ 56 | 57 | # if config_specs is a path 58 | if isinstance(config_specs, str): 59 | with open(config_specs, 'r') as config_file: 60 | config_specs = json.load(config_file) 61 | 62 | return Config(self.schema.load(config_specs, unknown=EXCLUDE)) 63 | 64 | def show_config(self): 65 | print(json.dumps(self.config, indent=4)) 66 | 67 | 68 | class Config(Munch): 69 | """ Config object used for automatic object creation from a dict. 70 | """ 71 | def __init__(self, config): 72 | config = dict_to_munch(config) 73 | 74 | super().__init__(config) 75 | -------------------------------------------------------------------------------- /eoflow/execute.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | 4 | from marshmallow import Schema, fields 5 | 6 | from .base import BaseModel, BaseTask 7 | from .base.configuration import ObjectConfiguration, Config 8 | from .utils import parse_classname 9 | 10 | 11 | class ExecutionConfig(Schema): 12 | model = fields.Nested(ObjectConfiguration, required=True, description='Model configuration') 13 | task = fields.Nested(ObjectConfiguration, required=True, description='Task configuration') 14 | 15 | 16 | def execute(config_file): 17 | """Executes a workflow defined in a config file.""" 18 | 19 | with open(config_file) as file: 20 | config = json.load(file) 21 | 22 | config = Config(ExecutionConfig().load(config)) 23 | 24 | # Parse model config 25 | model_cls = parse_classname(config.model.classname) 26 | if not issubclass(model_cls, BaseModel): 27 | raise ValueError("Model class does not inherit from BaseModel.") 28 | model = model_cls(config.model.config) 29 | 30 | # Parse task config 31 | task_cls = parse_classname(config.task.classname) 32 | if not issubclass(task_cls, BaseTask): 33 | raise ValueError("Task class does not inherit from BaseTask.") 34 | task = task_cls(model, config.task.config) 35 | 36 | # Run task 37 | task.run() 38 | 39 | 40 | if __name__ == '__main__': 41 | parser = argparse.ArgumentParser(description='Executes a workflow described in a provided config file.') 42 | 43 | parser.add_argument('config_file', type=str, help='Path to the configuration file.') 44 | 45 | args = parser.parse_args() 46 | 47 | execute(config_file=args.config_file) 48 | -------------------------------------------------------------------------------- /eoflow/input/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sentinel-hub/eo-flow/70a426fe3ab07d8ec096e06af4db7e445af1e740/eoflow/input/__init__.py -------------------------------------------------------------------------------- /eoflow/input/eopatch.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import tensorflow as tf 4 | from marshmallow import fields, Schema 5 | from marshmallow.validate import OneOf 6 | from eolearn.core import EOPatch, FeatureType 7 | 8 | from ..base import BaseInput 9 | 10 | _valid_types = [t.value for t in FeatureType] 11 | 12 | 13 | def eopatch_dataset(root_dir_or_list, features_data, fill_na=None): 14 | """ Creates a tf dataset with features from saved EOPatches. 15 | 16 | :param root_dir_or_list: Root directory containing eopatches or a list of eopatch directories. 17 | :type root_dir_or_list: str or list(str) 18 | :param features_data: List of tuples containing data about features to extract. 19 | Tuple structure: (feature_type, feature_name, out_feature_name, feature_dtype, feature_shape) 20 | :type features_data: (str, str, str, np.dtype, tuple) 21 | :param fill_na: Value with wich to replace nan values. No replacement is done if None. 22 | :type fill_na: int 23 | """ 24 | 25 | if isinstance(root_dir_or_list, str): 26 | file_pattern = os.path.join(root_dir_or_list, '*') 27 | dataset = tf.data.Dataset.list_files(file_pattern) 28 | else: 29 | dataset = tf.data.Dataset.from_tensor_slices(root_dir_or_list) 30 | 31 | def _read_patch(path): 32 | """ TF op for reading an eopatch at a given path. """ 33 | def _func(path): 34 | path = path.numpy().decode('utf-8') 35 | 36 | # Load only relevant features 37 | features = [(data[0], data[1]) for data in features_data] 38 | patch = EOPatch.load(path, features=features) 39 | 40 | data = [] 41 | for feat_type, feat_name, out_name, dtype, shape in features_data: 42 | arr = patch[feat_type][feat_name].astype(dtype) 43 | 44 | if fill_na is not None: 45 | arr[np.isnan(arr)] = fill_na 46 | 47 | data.append(arr) 48 | 49 | return data 50 | 51 | out_types = [tf.as_dtype(data[3]) for data in features_data] 52 | data = tf.py_function(_func, [path], out_types) 53 | 54 | out_data = {} 55 | for f_data, feature in zip(features_data, data): 56 | feat_type, feat_name, out_name, dtype, shape = f_data 57 | feature.set_shape(shape) 58 | out_data[out_name] = feature 59 | 60 | return out_data 61 | 62 | dataset = dataset.map(_read_patch) 63 | return dataset 64 | 65 | 66 | class EOPatchSegmentationInput(BaseInput): 67 | """ An input method for basic EOPatch reading. Reads features and segmentation labels. For more complex behaviour 68 | (subpatch extraction, data augmentation, caching, ...) create your own input method (see examples). 69 | """ 70 | 71 | class _Schema(Schema): 72 | data_dir = fields.String(description="The directory containing EOPatches.", required=True) 73 | 74 | input_feature_type = fields.String(description="Feature type of the input feature.", required=True, validate=OneOf(_valid_types)) 75 | input_feature_name = fields.String(description="Name of the input feature.", required=True) 76 | input_feature_axis = fields.List(fields.Int, description="Height and width axis for the input features", required=True, example=[1,2]) 77 | input_feature_shape = fields.List(fields.Int, description="Shape of the input feature. Use -1 for unknown dimesnions.", 78 | required=True, example=[-1, 100, 100, 3]) 79 | 80 | labels_feature_type = fields.String(description="Feature type of the labels feature.", required=True, validate=OneOf(_valid_types)) 81 | labels_feature_name = fields.String(description="Name of the labels feature.", required=True) 82 | labels_feature_axis = fields.List(fields.Int, description="Height and width axis for the labels", required=True, example=[1,2]) 83 | labels_feature_shape = fields.List(fields.Int, description="Shape of the labels feature. Use -1 for unknown dimesnions.", 84 | required=True, example=[-1, 100, 100, 3]) 85 | 86 | batch_size = fields.Int(description="Number of examples in a batch.", required=True, example=20) 87 | num_classes = fields.Int(description="Number of classes. Used for one-hot encoding.", required=True, example=2) 88 | 89 | def _parse_shape(self, shape): 90 | shape = [None if s < 0 else s for s in shape] 91 | return shape 92 | 93 | def get_dataset(self): 94 | cfg = self.config 95 | 96 | # Create a tf.data.Dataset from EOPatches 97 | features_data = [ 98 | (cfg.input_feature_type, cfg.input_feature_name, 'features', np.float32, self._parse_shape(cfg.input_feature_shape)), 99 | (cfg.labels_feature_type, cfg.labels_feature_name, 'labels', np.int64, self._parse_shape(cfg.labels_feature_shape)) 100 | ] 101 | dataset = eopatch_dataset(self.config.data_dir, features_data, fill_na=-2) 102 | 103 | # One-hot encode labels and return tuple 104 | def _prepare_data(data): 105 | features = data['features'] 106 | labels = data['labels'][...,0] 107 | 108 | labels_oh = tf.one_hot(labels, depth=self.config.num_classes) 109 | 110 | return features, labels_oh 111 | 112 | dataset = dataset.map(_prepare_data) 113 | 114 | # Create batches 115 | dataset = dataset.batch(self.config.batch_size) 116 | 117 | return dataset 118 | -------------------------------------------------------------------------------- /eoflow/input/hdf5.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import h5py 4 | import tensorflow as tf 5 | 6 | def hdf5_dataset(path, features): 7 | """ Creates a tf.data.Dataset from a hdf5 file 8 | 9 | :param path: path to the hdf5 file, 10 | :type path: str 11 | :param features: dict of (`dataset` -> `feature_name`) mappings, where `dataset` is the dataset name in the hdf5 file 12 | and `feature_name` is the name of the feature it is saved to. 13 | :type features: dict 14 | 15 | :return: dataset containing examples merged from files 16 | :rtype: tf.data.Dataset 17 | """ 18 | 19 | fields = list(features.keys()) 20 | feature_names = [features[f] for f in features] 21 | 22 | # Reads dataset row by row 23 | def _generator(): 24 | with h5py.File(path, 'r') as file: 25 | datasets = [file[field] for field in fields] 26 | for row in zip(*datasets): 27 | yield row 28 | 29 | # Converts a database of tuples to database of dicts 30 | def _to_dict(*features): 31 | return {name: feat for name, feat in zip(feature_names, features)} 32 | 33 | # Reads hdf5 metadata (types and shapes) 34 | with h5py.File(path, 'r') as file: 35 | datasets = [file[field] for field in fields] 36 | 37 | types = tuple(ds.dtype for ds in datasets) 38 | shapes = tuple(ds.shape[1:] for ds in datasets) 39 | 40 | # Create dataset 41 | ds = tf.data.Dataset.from_generator(_generator, types, shapes) 42 | ds = ds.map(_to_dict) 43 | 44 | return ds 45 | -------------------------------------------------------------------------------- /eoflow/input/numpy.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | 6 | def numpy_dataset(np_array_dict): 7 | """ Creates a tf.data Dataset from a dict of numpy arrays. """ 8 | 9 | # Unpack 10 | feature_names = list(np_array_dict.keys()) 11 | np_arrays = [np_array_dict[name] for name in feature_names] 12 | 13 | # Check that arrays match in the first dimension 14 | n_samples = np_arrays[0].shape[0] 15 | assert all(n_samples == arr.shape[0] for arr in np_arrays) 16 | 17 | # Extract types and shapes form np arrays 18 | types = tuple(arr.dtype for arr in np_arrays) 19 | shapes = tuple(arr.shape[1:] for arr in np_arrays) 20 | 21 | def _generator(): 22 | # Iterate through the first dimension of arrays 23 | for slices in zip(*np_arrays): 24 | yield slices 25 | 26 | ds = tf.data.Dataset.from_generator(_generator, types, shapes) 27 | ds = ds.take(n_samples) 28 | 29 | # Converts a database of tuples to database of dicts 30 | def _to_dict(*features): 31 | return {name: feat for name, feat in zip(feature_names, features)} 32 | 33 | ds = ds.map(_to_dict) 34 | 35 | return ds 36 | 37 | def _read_numpy_file(file_path, fields): 38 | """ Reads a single npz file. """ 39 | 40 | data = np.load(file_path) 41 | np_arrays = [data[f] for f in fields] 42 | 43 | # Check that arrays match in the first dimension 44 | n_samples = np_arrays[0].shape[0] 45 | assert all(n_samples == arr.shape[0] for arr in np_arrays) 46 | 47 | return tuple(np_arrays) 48 | 49 | def npz_dir_dataset(file_dir_or_list, features, num_parallel=5): 50 | """ Creates a tf.data.Dataset from a directory containing numpy .npz files. Files are loaded 51 | lazily when needed. `num_parallel` files are read in parallel and interleaved together. 52 | 53 | :param file_dir_or_list: directory containing .npz files or a list of paths to .npz files 54 | :type file_dir_or_list: str | list(str) 55 | :param features: dict of (`field` -> `feature_name`) mappings, where `field` is the field in the .npz array 56 | and `feature_name` is the name of the feature it is saved to. 57 | :type features: dict 58 | :param num_parallel: number of files to read in parallel and intereleave, defaults to 5 59 | :type num_parallel: int, optional 60 | 61 | :return: dataset containing examples merged from files 62 | :rtype: tf.data.Dataset 63 | """ 64 | 65 | files = file_dir_or_list 66 | 67 | # If dir, then list files 68 | if isinstance(file_dir_or_list, str): 69 | files = [os.path.join(file_dir_or_list, f) for f in os.listdir(file_dir_or_list)] 70 | 71 | fields = list(features.keys()) 72 | feature_names = [features[f] for f in features] 73 | 74 | # Read one file for shape info 75 | file = next(iter(files)) 76 | data = np.load(file) 77 | np_arrays = [data[f] for f in fields] 78 | 79 | # Read shape and type info 80 | types = tuple(arr.dtype for arr in np_arrays) 81 | shapes = tuple((None,) + arr.shape[1:] for arr in np_arrays) 82 | 83 | def _data_generator(files, fields): 84 | """ Returns samples from one file at a time. """ 85 | for f in files: 86 | yield _read_numpy_file(f, fields) 87 | 88 | # Converts a database of tuples to database of dicts 89 | def _to_dict(*features): 90 | return {name: feat for name, feat in zip(feature_names, features)} 91 | 92 | # Create dataset 93 | ds = tf.data.Dataset.from_generator(lambda:_data_generator(files, fields), types, shapes) 94 | 95 | # Prefetch needed amount of files for interleaving 96 | ds = ds.prefetch(num_parallel) 97 | 98 | # Unbatch and interleave 99 | ds = ds.interleave(lambda *x: tf.data.Dataset.from_tensor_slices(x), cycle_length=num_parallel) 100 | ds = ds.map(_to_dict) 101 | 102 | return ds 103 | -------------------------------------------------------------------------------- /eoflow/input/operations.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import tensorflow as tf 4 | 5 | from ..utils import create_dirs 6 | 7 | 8 | def extract_subpatches(patch_size, spatial_features_and_axis, random_sampling=False, num_random_samples=20, 9 | grid_overlap=0.2): 10 | """ Builds a TF op for building a dataset of subpatches from tensors. Subpatches sampling can be random or grid based. 11 | 12 | :param patch_size: Width and height of extracted patches 13 | :type patch_size: (int, int) 14 | :param spatial_features_and_axis: List of features from which subpatches are extracted and their height and width axis. 15 | Elements are tuples of (feature_name, (axis_h, axis_w)). 16 | :type spatial_features_and_axis: list of (string, (int, int)) 17 | :param random_sampling: If True random sampling is used. Else grid based sampling is used. 18 | :type random_sampling: bool 19 | :param num_random_samples: Defines the number of subpatches to sample, when random sampling is used. 20 | :type num_random_samples: int 21 | :param grid_overlap: Amount of overlap between subpatches extracted from a grid 22 | :type grid_overlap: float 23 | """ 24 | 25 | patch_w, patch_h = patch_size 26 | 27 | def _fn(data): 28 | feat_name_ref, axis_ref = spatial_features_and_axis[0] 29 | ay_ref, ax_ref = axis_ref 30 | # Get random coordinates 31 | 32 | def _py_get_random(image): 33 | x_space = image.shape[ax_ref]-patch_w 34 | if x_space > 0: 35 | x_rand = np.random.randint(x_space, size=num_random_samples, dtype=np.int64) 36 | else: 37 | x_rand = np.zeros(num_random_samples, np.int64) 38 | 39 | y_space = image.shape[ay_ref]-patch_h 40 | if y_space > 0: 41 | y_rand = np.random.randint(y_space, size=num_random_samples, dtype=np.int64) 42 | else: 43 | y_rand = np.zeros(num_random_samples, np.int64) 44 | 45 | return x_rand, y_rand 46 | 47 | # Get coordinates on a grid 48 | def _py_get_gridded(image): 49 | 50 | # alpha is overlaping ratio (w.r.t. patch size) 51 | alpha = grid_overlap 52 | 53 | img_height = image.shape[ay_ref] 54 | img_width = image.shape[ax_ref] 55 | 56 | # number of patches in x and y direction 57 | nx = int(np.ceil((img_width - alpha * patch_w) / (patch_w * (1 - alpha)))) 58 | ny = int(np.ceil((img_height - alpha * patch_h) / (patch_h * (1 - alpha)))) 59 | 60 | # total number of patches 61 | N = nx * ny 62 | # allocate output vectors (top-left patch coordinates) 63 | tl_x = np.zeros(N, dtype=np.int64) 64 | tl_y = np.zeros(N, dtype=np.int64) 65 | 66 | # calculate actual x and y coordinates 67 | for yi in range(ny): 68 | if yi == 0: 69 | # the highest patch has x0 = 0 70 | y_ = 0 71 | elif yi == ny - 1: 72 | # the lowest patch has y0 = H - patch_h 73 | y_ = img_height - patch_h 74 | else: 75 | # calculate top-left y coordinate and take into account overlaping, too 76 | y_ = np.round(yi * patch_h - yi * alpha * patch_h) 77 | 78 | for xi in range(nx): 79 | if xi == 0: 80 | # the left-most patch has x0 = 0 81 | x_ = 0 82 | elif xi == nx - 1: 83 | # the right-most patch has x0 = W - patch_w 84 | x_ = img_width - patch_w 85 | else: 86 | # calculate top-left x coordinate and take into account overlaping, too 87 | x_ = np.round(xi * patch_w - xi * alpha * patch_w) 88 | 89 | id = yi * nx + xi 90 | tl_x[id] = np.int64(x_) 91 | tl_y[id] = np.int64(y_) 92 | 93 | return tl_x, tl_y 94 | 95 | if random_sampling: 96 | x_samp, y_samp = tf.py_function(_py_get_random, [data[feat_name_ref]], [tf.int64, tf.int64]) 97 | else: 98 | x_samp, y_samp = tf.py_function(_py_get_gridded, [data[feat_name_ref]], [tf.int64, tf.int64]) 99 | 100 | def _py_get_patches(axis): 101 | ay, ax = axis 102 | # Extract patches for given coordinates 103 | 104 | def _func(image, x_samp, y_samp): 105 | patches = [] 106 | 107 | # Pad if necessary 108 | x_pad = max(0, patch_w - image.shape[ax]) 109 | y_pad = max(0, patch_h - image.shape[ay]) 110 | 111 | if x_pad > 0 or y_pad > 0: 112 | pad_x1 = x_pad//2 113 | pad_x2 = x_pad - pad_x1 114 | pad_y1 = y_pad//2 115 | pad_y2 = y_pad - pad_y1 116 | 117 | padding = [(0,0) for _ in range(image.ndim)] 118 | padding[ax] = (pad_x1,pad_x2) 119 | padding[ay] = (pad_y1,pad_y2) 120 | image = np.pad(image, padding, 'constant') 121 | 122 | # Extract patches 123 | for x, y in zip(x_samp, y_samp): 124 | # Slice on specified axis 125 | slicing = [slice(None) for _ in range(image.ndim)] 126 | slicing[ax] = slice(x, x+patch_w) 127 | slicing[ay] = slice(y, y+patch_h) 128 | 129 | patch = image[slicing] 130 | patches.append(patch) 131 | return np.stack(patches) 132 | 133 | return _func 134 | 135 | data_out = {} 136 | # TODO: repeat the rest of the data 137 | for feat_name, axis in spatial_features_and_axis: 138 | ay, ax = axis 139 | shape = data[feat_name].shape.as_list() 140 | patches = tf.py_function(_py_get_patches(axis), [data[feat_name], x_samp, y_samp], data[feat_name].dtype) 141 | 142 | # Update shape information 143 | shape[ax] = patch_w 144 | shape[ay] = patch_h 145 | shape = [None] + shape 146 | patches.set_shape(shape) 147 | 148 | data_out[feat_name] = patches 149 | 150 | # TODO: shuffle subpatches 151 | return tf.data.Dataset.from_tensor_slices(data_out) 152 | 153 | return _fn 154 | 155 | 156 | def augment_data(features_to_augment, brightness_delta=0.1, contrast_bounds=(0.9,1.1)): 157 | """ Builds a function that randomly augments features in specified ways. 158 | 159 | param features_to_augment: List of features to augment and which operations to perform on them. 160 | Each element is of shape (feature, list_of_operations). 161 | type features_to_augment: list of (str, list of str) 162 | param brightness_delta: Maximum brightness change. 163 | type brightness_delta: float 164 | param contrast_bounds: Upper and lower bounds of contrast multiplier. 165 | type contrast_bounds: (float, float) 166 | """ 167 | 168 | def _augment(data): 169 | contrast_lower, contrast_upper = contrast_bounds 170 | 171 | flip_lr_cond = tf.random.uniform(shape=[]) > 0.5 172 | flip_ud_cond = tf.random.uniform(shape=[]) > 0.5 173 | rot90_amount = tf.random.uniform(shape=[], maxval=4, dtype=tf.int32) 174 | 175 | # Available operations 176 | operations = { 177 | 'flip_left_right': lambda x: tf.cond(flip_lr_cond, lambda: tf.image.flip_left_right(x), lambda: x), 178 | 'flip_up_down': lambda x: tf.cond(flip_ud_cond, lambda: tf.image.flip_up_down(x), lambda: x), 179 | 'rotate': lambda x: tf.image.rot90(x, rot90_amount), 180 | 'brightness': lambda x: tf.image.random_brightness(x, brightness_delta), 181 | 'contrast': lambda x: tf.image.random_contrast(x, contrast_lower, contrast_upper) 182 | } 183 | 184 | for feature, ops in features_to_augment: 185 | # Apply specified ops to feature 186 | for op in ops: 187 | operation_fn = operations[op] 188 | data[feature] = operation_fn(data[feature]) 189 | 190 | return data 191 | 192 | return _augment 193 | 194 | 195 | def cache_dataset(dataset, path): 196 | """ Caches dataset into a file. Each element in the dataset will be computed only once. """ 197 | 198 | # Create dir if missing 199 | directory = os.path.dirname(path) 200 | create_dirs([directory]) 201 | 202 | # Cache 203 | dataset = dataset.cache(path) 204 | 205 | # Disable map and batch fusion to prevent a bug when caching 206 | options = tf.data.Options() 207 | options.experimental_optimization.map_and_batch_fusion = False 208 | dataset = dataset.with_options(options) 209 | 210 | return dataset 211 | -------------------------------------------------------------------------------- /eoflow/input/random.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | from marshmallow import fields, Schema 4 | 5 | from ..base import BaseInput 6 | 7 | 8 | class RandomClassificationInput(BaseInput): 9 | """ Class to create random batches for classification tasks. Can be used for prototyping. """ 10 | 11 | class _Schema(Schema): 12 | input_shape = fields.List(fields.Int, description="Shape of a single input example.", required=True, example=[784]) 13 | num_classes = fields.Int(description="Number of classes.", required=True, example=10) 14 | 15 | batch_size = fields.Int(description="Number of examples in a batch.", required=True, example=20) 16 | batches_per_epoch = fields.Int(required=True, description='Number of batches in epoch', example=20) 17 | 18 | def _generate_batch(self): 19 | for i in range(self.config.batches_per_epoch): 20 | input_shape = [self.config.batch_size] + self.config.input_shape 21 | input_data = np.random.rand(*input_shape) 22 | 23 | onehot = np.eye(self.config.num_classes) 24 | output_shape = [self.config.batch_size] 25 | classes = np.random.randint(self.config.num_classes, size=output_shape) 26 | labels = onehot[classes] 27 | 28 | yield input_data, labels 29 | 30 | def get_dataset(self): 31 | input_shape = [self.config.batch_size] + self.config.input_shape 32 | output_shape = [self.config.batch_size, self.config.num_classes] 33 | 34 | dataset = tf.data.Dataset.from_generator( 35 | self._generate_batch, 36 | (tf.float32, tf.float32), 37 | (tf.TensorShape(input_shape), tf.TensorShape(output_shape)) 38 | ) 39 | 40 | return dataset 41 | 42 | 43 | class RandomSegmentationInput(BaseInput): 44 | """ Class to create random batches for segmentation tasks. Can be used for prototyping. """ 45 | 46 | class _Schema(Schema): 47 | input_shape = fields.List(fields.Int, description="Shape of a single input example.", required=True, example=[512,512,3]) 48 | output_shape = fields.List(fields.Int, description="Shape of a single output mask.", required=True, example=[128,128]) 49 | num_classes = fields.Int(description="Number of segmentation classes.", required=True, example=10) 50 | 51 | batch_size = fields.Int(description="Number of examples in a batch.", required=True, example=20) 52 | batches_per_epoch = fields.Int(required=True, description='Number of batches in epoch', example=20) 53 | 54 | def _generate_batch(self): 55 | for i in range(self.config.batches_per_epoch): 56 | input_shape = [self.config.batch_size] + self.config.input_shape 57 | input_data = np.random.rand(*input_shape) 58 | 59 | onehot = np.eye(self.config.num_classes) 60 | output_shape = [self.config.batch_size] + self.config.output_shape 61 | classes = np.random.randint(self.config.num_classes, size=output_shape) 62 | labels = onehot[classes] 63 | 64 | yield input_data, labels 65 | 66 | def get_dataset(self): 67 | input_shape = [self.config.batch_size] + self.config.input_shape 68 | output_shape = [self.config.batch_size] + self.config.output_shape + [self.config.num_classes] 69 | 70 | dataset = tf.data.Dataset.from_generator( 71 | self._generate_batch, 72 | (tf.float32, tf.float32), 73 | (tf.TensorShape(input_shape), tf.TensorShape(output_shape)) 74 | ) 75 | 76 | return dataset 77 | -------------------------------------------------------------------------------- /eoflow/models/__init__.py: -------------------------------------------------------------------------------- 1 | from .segmentation_unets import FCNModel, TFCNModel 2 | from .classification_temp_nets import TCNModel, TempCNNModel, BiRNN, TransformerEncoder, PseTae 3 | -------------------------------------------------------------------------------- /eoflow/models/callbacks.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | import matplotlib as mpl 4 | import matplotlib.pyplot as plt 5 | 6 | from ..utils.tf_utils import plot_to_image 7 | 8 | 9 | class VisualizationCallback(tf.keras.callbacks.Callback): 10 | """ Keras Callback for saving prediction visualizations to TensorBoard. """ 11 | 12 | def __init__(self, val_images, log_dir, time_index=0, rgb_indices=[2, 1, 0]): 13 | """ 14 | :param val_images: Images to run predictions on. Tuple of (images, labels). 15 | :type val_images: (np.array, np.array) 16 | :param log_dir: Directory where the TensorBoard logs are written. 17 | :type log_dir: str 18 | :param time_index: Time index to use, when multiple time slices are available, defaults to 0 19 | :type time_index: int, optional 20 | :param rgb_indices: Indices for R, G and B bands in the input image, defaults to [0,1,2] 21 | :type rgb_indices: list, optional 22 | """ 23 | super().__init__() 24 | 25 | self.val_images = val_images 26 | self.time_index = time_index 27 | self.rgb_indices = rgb_indices 28 | 29 | self.file_writer = tf.summary.create_file_writer(log_dir) 30 | 31 | @staticmethod 32 | def plot_predictions(input_image, labels, predictions, n_classes): 33 | # TODO: fix figsize (too wide?) 34 | fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18, 5)) 35 | 36 | scaled_image = np.clip(input_image*2.5, 0., 1.) 37 | ax1.imshow(scaled_image) 38 | ax1.title.set_text('Input image') 39 | 40 | cnorm = mpl.colors.NoNorm() 41 | cmap = plt.cm.get_cmap('Set3', n_classes) 42 | 43 | ax2.imshow(labels, cmap=cmap, norm=cnorm) 44 | ax2.title.set_text('Labeled classes') 45 | 46 | img = ax3.imshow(predictions, cmap=cmap, norm=cnorm) 47 | ax3.title.set_text('Predicted classes') 48 | 49 | plt.colorbar(img, ax=[ax1, ax2, ax3], shrink=0.8, ticks=list(range(n_classes))) 50 | 51 | return fig 52 | 53 | def prediction_summaries(self, step): 54 | images, labels = self.val_images 55 | preds_raw = self.model.predict(images) 56 | 57 | pred_shape = tf.shape(preds_raw) 58 | 59 | # If temporal data only use time_index slice 60 | if images.ndim == 5: 61 | images = images[:, self.time_index, :, :, :] 62 | 63 | # Crop images and labels to output size 64 | labels = tf.image.resize_with_crop_or_pad(labels, pred_shape[1], pred_shape[2]) 65 | images = tf.image.resize_with_crop_or_pad(images, pred_shape[1], pred_shape[2]) 66 | 67 | # Take RGB values 68 | images = images.numpy()[..., self.rgb_indices] 69 | 70 | num_classes = labels.shape[-1] 71 | 72 | # Get class ids 73 | preds_raw = np.argmax(preds_raw, axis=-1) 74 | labels = np.argmax(labels, axis=-1) 75 | 76 | vis_images = [] 77 | for image_i, labels_i, pred_i in zip(images, labels, preds_raw): 78 | # Plot predictions and convert to image 79 | fig = self.plot_predictions(image_i, labels_i, pred_i, num_classes) 80 | img = plot_to_image(fig) 81 | 82 | vis_images.append(img) 83 | 84 | n_images = len(vis_images) 85 | vis_images = tf.concat(vis_images, axis=0) 86 | 87 | with self.file_writer.as_default(): 88 | tf.summary.image('predictions', vis_images, step=step, max_outputs=n_images) 89 | 90 | def on_epoch_end(self, epoch, logs=None): 91 | self.prediction_summaries(epoch) 92 | -------------------------------------------------------------------------------- /eoflow/models/classification_base.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | from marshmallow import Schema, fields 6 | from marshmallow.validate import OneOf, ContainsOnly 7 | 8 | from ..base import BaseModel 9 | 10 | from .losses import CategoricalCrossEntropy, CategoricalFocalLoss 11 | from .metrics import InitializableMetric 12 | 13 | logging.basicConfig(level=logging.INFO, 14 | format='%(asctime)s %(levelname)s %(message)s') 15 | 16 | 17 | # Available losses. Add keys with new losses here. 18 | classification_losses = { 19 | 'cross_entropy': CategoricalCrossEntropy, 20 | 'focal_loss': CategoricalFocalLoss 21 | } 22 | 23 | 24 | # Available metrics. Add keys with new metrics here. 25 | classification_metrics = { 26 | 'accuracy': tf.keras.metrics.CategoricalAccuracy(name='accuracy'), 27 | 'precision': tf.keras.metrics.Precision, 28 | 'recall': tf.keras.metrics.Recall 29 | } 30 | 31 | 32 | class BaseClassificationModel(BaseModel): 33 | """ Base for pixel-wise classification models. """ 34 | 35 | class _Schema(Schema): 36 | n_classes = fields.Int(required=True, description='Number of classes', example=2) 37 | learning_rate = fields.Float(missing=None, description='Learning rate used in training.', example=0.01) 38 | loss = fields.String(missing='cross_entropy', description='Loss function used for training.', 39 | validate=OneOf(classification_losses.keys())) 40 | metrics = fields.List(fields.String, missing=['accuracy'], 41 | description='List of metrics used for evaluation.', 42 | validate=ContainsOnly(classification_metrics.keys())) 43 | 44 | class_weights = fields.Dict(missing=None, description='Dictionary mapping class id with weight. ' 45 | 'If key for some labels is not specified, 1 is used.') 46 | 47 | def _prepare_class_weights(self): 48 | """ Utility function to parse class weights """ 49 | if self.config.class_weights is None: 50 | return np.ones(self.config.n_classes) 51 | return np.array([self.config.class_weights[iclass] if iclass in self.config.class_weights else 1.0 52 | for iclass in range(self.config.n_classes)]) 53 | 54 | def prepare(self, optimizer=None, loss=None, metrics=None, **kwargs): 55 | """ Prepares the model. Optimizer, loss and metrics are read using the following protocol: 56 | * If an argument is None, the default value is used from the configuration of the model. 57 | * If an argument is a key contained in segmentation specific losses/metrics, those are used. 58 | * Otherwise the argument is passed to `compile` as is. 59 | 60 | """ 61 | # Read defaults if None 62 | if optimizer is None: 63 | optimizer = tf.keras.optimizers.Adam(learning_rate=self.config.learning_rate) 64 | 65 | if loss is None: 66 | loss = self.config.loss 67 | 68 | if metrics is None: 69 | metrics = self.config.metrics 70 | 71 | class_weights = self._prepare_class_weights() 72 | 73 | # TODO: pass kwargs to loss from config 74 | loss = classification_losses[loss](from_logits=False, class_weights=class_weights) 75 | 76 | reported_metrics = [] 77 | for metric in metrics: 78 | 79 | if metric in classification_metrics: 80 | if metric in ['precision', 'recall']: 81 | reported_metrics += [classification_metrics[metric](top_k=1, 82 | class_id=class_id, 83 | name=f'{metric}_{class_id}') 84 | for class_id in range(self.config.n_classes)] 85 | continue 86 | else: 87 | metric = classification_metrics[metric] 88 | 89 | # Initialize initializable metrics 90 | if isinstance(metric, InitializableMetric): 91 | metric.init_from_config(self.config) 92 | 93 | reported_metrics.append(metric) 94 | 95 | self.compile(optimizer=optimizer, loss=loss, metrics=reported_metrics, **kwargs) 96 | 97 | # Override default method to add prediction visualization 98 | def train(self, 99 | dataset, 100 | num_epochs, 101 | model_directory, 102 | iterations_per_epoch=None, 103 | class_weights=None, 104 | callbacks=[], 105 | save_steps='epoch', 106 | summary_steps=1, **kwargs): 107 | 108 | super().train(dataset, num_epochs, model_directory, iterations_per_epoch, 109 | callbacks=callbacks, save_steps=save_steps, 110 | summary_steps=summary_steps, **kwargs) 111 | 112 | # Override default method to add prediction visualization 113 | def train_and_evaluate(self, 114 | train_dataset, 115 | val_dataset, 116 | num_epochs, 117 | iterations_per_epoch, 118 | model_directory, 119 | class_weights=None, 120 | save_steps=100, 121 | summary_steps=10, 122 | callbacks=[], **kwargs): 123 | 124 | super().train_and_evaluate(train_dataset, val_dataset, 125 | num_epochs, iterations_per_epoch, 126 | model_directory, 127 | save_steps=save_steps, summary_steps=summary_steps, 128 | callbacks=callbacks, **kwargs) 129 | -------------------------------------------------------------------------------- /eoflow/models/classification_temp_nets.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import tensorflow as tf 3 | from marshmallow import fields 4 | from marshmallow.validate import OneOf 5 | 6 | from tensorflow.keras.layers import SimpleRNN, LSTM, GRU 7 | from tensorflow.python.keras.utils.layer_utils import print_summary 8 | 9 | from .layers import ResidualBlock 10 | from .classification_base import BaseClassificationModel 11 | 12 | from . import transformer_encoder_layers 13 | from . import pse_tae_layers 14 | 15 | logging.basicConfig(level=logging.INFO, 16 | format='%(asctime)s %(levelname)s %(message)s') 17 | 18 | rnn_layers = dict(rnn=SimpleRNN, gru=GRU, lstm=LSTM) 19 | 20 | 21 | class TCNModel(BaseClassificationModel): 22 | """ Implementation of the TCN network taken form the keras-TCN implementation 23 | 24 | https://github.com/philipperemy/keras-tcn 25 | """ 26 | 27 | class TCNModelSchema(BaseClassificationModel._Schema): 28 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 29 | 30 | kernel_size = fields.Int(missing=2, description='Size of the convolution kernels.') 31 | nb_filters = fields.Int(missing=64, description='Number of convolutional filters.') 32 | nb_stacks = fields.Int(missing=1) 33 | dilations = fields.List(fields.Int, missing=[1, 2, 4, 8, 16, 32], description='Size of dilations used in the ' 34 | 'covolutional layers') 35 | padding = fields.String(missing='CAUSAL', validate=OneOf(['CAUSAL', 'SAME']), 36 | description='Padding type used in convolutions.') 37 | use_skip_connections = fields.Bool(missing=True, description='Flag to whether to use skip connections.') 38 | return_sequences = fields.Bool(missing=False, description='Flag to whether return sequences or not.') 39 | activation = fields.Str(missing='linear', description='Activation function used in final filters.') 40 | kernel_initializer = fields.Str(missing='he_normal', description='method to initialise kernel parameters.') 41 | 42 | use_batch_norm = fields.Bool(missing=False, description='Whether to use batch normalisation.') 43 | use_layer_norm = fields.Bool(missing=False, description='Whether to use layer normalisation.') 44 | 45 | def build(self, inputs_shape): 46 | """ Build TCN architecture 47 | 48 | The `inputs_shape` argument is a `(N, T, D)` tuple where `N` denotes the number of samples, `T` the number of 49 | time-frames, and `D` the number of channels 50 | """ 51 | x = tf.keras.layers.Input(inputs_shape[1:]) 52 | 53 | dropout_rate = 1 - self.config.keep_prob 54 | 55 | net = x 56 | 57 | net = tf.keras.layers.Conv1D(filters=self.config.nb_filters, 58 | kernel_size=1, 59 | padding=self.config.padding, 60 | kernel_initializer=self.config.kernel_initializer)(net) 61 | 62 | # list to hold all the member ResidualBlocks 63 | residual_blocks = list() 64 | skip_connections = list() 65 | 66 | total_num_blocks = self.config.nb_stacks * len(self.config.dilations) 67 | if not self.config.use_skip_connections: 68 | total_num_blocks += 1 # cheap way to do a false case for below 69 | 70 | for s in range(self.config.nb_stacks): 71 | for d in self.config.dilations: 72 | net, skip_out = ResidualBlock(dilation_rate=d, 73 | nb_filters=self.config.nb_filters, 74 | kernel_size=self.config.kernel_size, 75 | padding=self.config.padding, 76 | activation=self.config.activation, 77 | dropout_rate=dropout_rate, 78 | use_batch_norm=self.config.use_batch_norm, 79 | use_layer_norm=self.config.use_layer_norm, 80 | kernel_initializer=self.config.kernel_initializer, 81 | last_block=len(residual_blocks) + 1 == total_num_blocks, 82 | name=f'residual_block_{len(residual_blocks)}')(net) 83 | residual_blocks.append(net) 84 | skip_connections.append(skip_out) 85 | 86 | # Author: @karolbadowski. 87 | output_slice_index = int(net.shape.as_list()[1] / 2) \ 88 | if self.config.padding.lower() == 'same' else -1 89 | lambda_layer = tf.keras.layers.Lambda(lambda tt: tt[:, output_slice_index, :]) 90 | 91 | if self.config.use_skip_connections: 92 | net = tf.keras.layers.add(skip_connections) 93 | 94 | if not self.config.return_sequences: 95 | net = lambda_layer(net) 96 | 97 | net = tf.keras.layers.Dense(self.config.n_classes)(net) 98 | 99 | net = tf.keras.layers.Softmax()(net) 100 | 101 | self.net = tf.keras.Model(inputs=x, outputs=net) 102 | 103 | print_summary(self.net) 104 | 105 | def call(self, inputs, training=None): 106 | return self.net(inputs, training) 107 | 108 | 109 | class TempCNNModel(BaseClassificationModel): 110 | """ Implementation of the TempCNN network taken from the temporalCNN implementation 111 | 112 | https://github.com/charlotte-pel/temporalCNN 113 | """ 114 | 115 | class TempCNNModelSchema(BaseClassificationModel._Schema): 116 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 117 | 118 | kernel_size = fields.Int(missing=5, description='Size of the convolution kernels.') 119 | nb_conv_filters = fields.Int(missing=16, description='Number of convolutional filters.') 120 | nb_conv_stacks = fields.Int(missing=3, description='Number of convolutional blocks.') 121 | nb_conv_strides = fields.Int(missing=1, description='Value of convolutional strides.') 122 | nb_fc_neurons = fields.Int(missing=256, description='Number of Fully Connect neurons.') 123 | nb_fc_stacks = fields.Int(missing=1, description='Number of fully connected layers.') 124 | padding = fields.String(missing='SAME', validate=OneOf(['SAME']), 125 | description='Padding type used in convolutions.') 126 | activation = fields.Str(missing='relu', description='Activation function used in final filters.') 127 | kernel_initializer = fields.Str(missing='he_normal', description='Method to initialise kernel parameters.') 128 | kernel_regularizer = fields.Float(missing=1e-6, description='L2 regularization parameter.') 129 | 130 | use_batch_norm = fields.Bool(missing=False, description='Whether to use batch normalisation.') 131 | 132 | def build(self, inputs_shape): 133 | """ Build TCN architecture 134 | 135 | The `inputs_shape` argument is a `(N, T, D)` tuple where `N` denotes the number of samples, `T` the number of 136 | time-frames, and `D` the number of channels 137 | """ 138 | x = tf.keras.layers.Input(inputs_shape[1:]) 139 | 140 | dropout_rate = 1 - self.config.keep_prob 141 | 142 | net = x 143 | for _ in range(self.config.nb_conv_stacks): 144 | net = tf.keras.layers.Conv1D(filters=self.config.nb_conv_filters, 145 | kernel_size=self.config.kernel_size, 146 | strides=self.config.nb_conv_strides, 147 | padding=self.config.padding, 148 | kernel_initializer=self.config.kernel_initializer, 149 | kernel_regularizer=tf.keras.regularizers.l2(self.config.kernel_regularizer))(net) 150 | if self.config.use_batch_norm: 151 | net = tf.keras.layers.BatchNormalization(axis=-1)(net) 152 | 153 | net = tf.keras.layers.Activation(self.config.activation)(net) 154 | 155 | net = tf.keras.layers.Dropout(dropout_rate)(net) 156 | 157 | net = tf.keras.layers.Flatten()(net) 158 | 159 | for _ in range(self.config.nb_fc_stacks): 160 | net = tf.keras.layers.Dense(units=self.config.nb_fc_neurons, 161 | kernel_initializer=self.config.kernel_initializer, 162 | kernel_regularizer=tf.keras.regularizers.l2(self.config.kernel_regularizer))(net) 163 | if self.config.use_batch_norm: 164 | net = tf.keras.layers.BatchNormalization(axis=-1)(net) 165 | 166 | net = tf.keras.layers.Activation(self.config.activation)(net) 167 | 168 | net = tf.keras.layers.Dropout(dropout_rate)(net) 169 | 170 | net = tf.keras.layers.Dense(units=self.config.n_classes, 171 | kernel_initializer=self.config.kernel_initializer, 172 | kernel_regularizer=tf.keras.regularizers.l2(self.config.kernel_regularizer))(net) 173 | 174 | net = tf.keras.layers.Softmax()(net) 175 | 176 | self.net = tf.keras.Model(inputs=x, outputs=net) 177 | 178 | print_summary(self.net) 179 | 180 | def call(self, inputs, training=None): 181 | return self.net(inputs, training) 182 | 183 | 184 | class BiRNN(BaseClassificationModel): 185 | """ Implementation of a Bidirectional Recurrent Neural Network 186 | 187 | This implementation allows users to define which RNN layer to use, e.g. SimpleRNN, GRU or LSTM 188 | """ 189 | 190 | class BiRNNModelSchema(BaseClassificationModel._Schema): 191 | rnn_layer = fields.String(required=True, validate=OneOf(['rnn', 'lstm', 'gru']), 192 | description='Type of RNN layer to use') 193 | 194 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 195 | 196 | rnn_units = fields.Int(missing=64, description='Size of the convolution kernels.') 197 | rnn_blocks = fields.Int(missing=1, description='Number of LSTM blocks') 198 | bidirectional = fields.Bool(missing=True, description='Whether to use a bidirectional layer') 199 | 200 | activation = fields.Str(missing='linear', description='Activation function used in final dense filters.') 201 | kernel_initializer = fields.Str(missing='he_normal', description='Method to initialise kernel parameters.') 202 | kernel_regularizer = fields.Float(missing=1e-6, description='L2 regularization parameter.') 203 | 204 | layer_norm = fields.Bool(missing=True, description='Whether to apply layer normalization in the encoder.') 205 | batch_norm = fields.Bool(missing=False, description='Whether to use batch normalisation.') 206 | 207 | def _rnn_layer(self, last=False): 208 | """ Returns a RNN layer for current configuration. Use `last=True` for the last RNN layer. """ 209 | RNNLayer = rnn_layers[self.config.rnn_layer] 210 | dropout_rate = 1 - self.config.keep_prob 211 | 212 | layer = RNNLayer(units=self.config.rnn_units, 213 | dropout=dropout_rate, 214 | return_sequences=False if last else True) 215 | 216 | # Use bidirectional if specified 217 | if self.config.bidirectional: 218 | layer = tf.keras.layers.Bidirectional(layer) 219 | 220 | return layer 221 | 222 | def init_model(self): 223 | """ Creates the RNN model architecture. """ 224 | layers = [] 225 | if self.config.layer_norm: 226 | layer_norm = tf.keras.layers.LayerNormalization() 227 | layers.append(layer_norm) 228 | 229 | # RNN layers 230 | layers.extend([self._rnn_layer() for _ in range(self.config.rnn_blocks-1)]) 231 | layers.append(self._rnn_layer(last=True)) 232 | 233 | if self.config.batch_norm: 234 | batch_norm = tf.keras.layers.BatchNormalization() 235 | layers.append(batch_norm) 236 | 237 | if self.config.layer_norm: 238 | layer_norm = tf.keras.layers.LayerNormalization() 239 | layers.append(layer_norm) 240 | 241 | dense = tf.keras.layers.Dense(units=self.config.n_classes, 242 | activation=self.config.activation, 243 | kernel_initializer=self.config.kernel_initializer, 244 | kernel_regularizer=tf.keras.regularizers.l2(self.config.kernel_regularizer)) 245 | 246 | softmax = tf.keras.layers.Softmax() 247 | 248 | layers.append(dense) 249 | layers.append(softmax) 250 | 251 | self.net = tf.keras.Sequential(layers) 252 | 253 | def build(self, inputs_shape): 254 | self.net.build(inputs_shape) 255 | 256 | print_summary(self.net) 257 | 258 | def call(self, inputs, training=None): 259 | return self.net(inputs, training) 260 | 261 | 262 | class TransformerEncoder(BaseClassificationModel): 263 | """ Implementation of a self-attention classifier 264 | 265 | Code is based on the Pytorch implementation of Marc Russwurm https://github.com/MarcCoru/crop-type-mapping 266 | """ 267 | 268 | class TransformerEncoderSchema(BaseClassificationModel._Schema): 269 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 270 | 271 | num_heads = fields.Int(missing=8, description='Number of Attention heads.') 272 | num_layers = fields.Int(missing=4, description='Number of encoder layers.') 273 | num_dff = fields.Int(missing=512, description='Number of feed-forward neurons in point-wise MLP.') 274 | d_model = fields.Int(missing=128, description='Depth of model.') 275 | max_pos_enc = fields.Int(missing=24, description='Maximum length of positional encoding.') 276 | layer_norm = fields.Bool(missing=True, description='Whether to apply layer normalization in the encoder.') 277 | 278 | activation = fields.Str(missing='linear', description='Activation function used in final dense filters.') 279 | 280 | def init_model(self): 281 | 282 | self.encoder = transformer_encoder_layers.Encoder( 283 | num_layers=self.config.num_layers, 284 | d_model=self.config.d_model, 285 | num_heads=self.config.num_heads, 286 | dff=self.config.num_dff, 287 | maximum_position_encoding=self.config.max_pos_enc, 288 | layer_norm=self.config.layer_norm) 289 | 290 | self.dense = tf.keras.layers.Dense(units=self.config.n_classes, 291 | activation=self.config.activation) 292 | 293 | def build(self, inputs_shape): 294 | """ Build Transformer encoder architecture 295 | 296 | The `inputs_shape` argument is a `(N, T, D)` tuple where `N` denotes the number of samples, `T` the number of 297 | time-frames, and `D` the number of channels 298 | """ 299 | seq_len = inputs_shape[1] 300 | 301 | self.net = tf.keras.Sequential([ 302 | self.encoder, 303 | self.dense, 304 | tf.keras.layers.MaxPool1D(pool_size=seq_len), 305 | tf.keras.layers.Lambda(lambda x: tf.keras.backend.squeeze(x, axis=-2), name='squeeze'), 306 | tf.keras.layers.Softmax() 307 | ]) 308 | # Build the model, so we can print the summary 309 | self.net.build(inputs_shape) 310 | 311 | print_summary(self.net) 312 | 313 | def call(self, inputs, training=None, mask=None): 314 | return self.net(inputs, training, mask) 315 | 316 | 317 | class PseTae(BaseClassificationModel): 318 | """ Implementation of the Pixel-Set encoder + Temporal Attention Encoder sequence classifier 319 | 320 | Code is based on the Pytorch implementation of V. Sainte Fare Garnot et al. https://github.com/VSainteuf/pytorch-psetae 321 | """ 322 | 323 | class PseTaeSchema(BaseClassificationModel._Schema): 324 | mlp1 = fields.List(fields.Int, missing=[10, 32, 64], description='Number of units for each layer in mlp1.') 325 | pooling = fields.Str(missing='mean_std', description='Methods used for pooling. Seperated by underscore. (mean, std, max, min)') 326 | mlp2 = fields.List(fields.Int, missing=[132, 128], description='Number of units for each layer in mlp2.') 327 | 328 | num_heads = fields.Int(missing=4, description='Number of Attention heads.') 329 | num_dff = fields.Int(missing=32, description='Number of feed-forward neurons in point-wise MLP.') 330 | d_model = fields.Int(missing=None, description='Depth of model.') 331 | mlp3 = fields.List(fields.Int, missing=[512, 128, 128], description='Number of units for each layer in mlp3.') 332 | dropout = fields.Float(missing=0.2, description='Dropout rate for attention encoder.') 333 | T = fields.Float(missing=1000, description='Number of features for attention.') 334 | len_max_seq = fields.Int(missing=24, description='Number of features for attention.') 335 | mlp4 = fields.List(fields.Int, missing=[128, 64, 32], description='Number of units for each layer in mlp4. Last layer with n_classes is added automatically.') 336 | 337 | def init_model(self): 338 | # TODO: missing features from original PseTae: 339 | # * spatial encoder extra features (hand-made) 340 | # * spatial encoder masking 341 | 342 | self.spatial_encoder = pse_tae_layers.PixelSetEncoder( 343 | mlp1=self.config.mlp1, 344 | mlp2=self.config.mlp2, 345 | pooling=self.config.pooling) 346 | 347 | self.temporal_encoder = pse_tae_layers.TemporalAttentionEncoder( 348 | n_head=self.config.num_heads, 349 | d_k=self.config.num_dff, 350 | d_model=self.config.d_model, 351 | n_neurons=self.config.mlp3, 352 | dropout=self.config.dropout, 353 | T=self.config.T, 354 | len_max_seq=self.config.len_max_seq) 355 | 356 | mlp4_layers = [pse_tae_layers.LinearLayer(out_dim) for out_dim in self.config.mlp4] 357 | # Final layer (logits) 358 | mlp4_layers.append(pse_tae_layers.LinearLayer(self.config.n_classes, batch_norm=False, activation=False)) 359 | 360 | self.mlp4 = tf.keras.Sequential(mlp4_layers) 361 | 362 | def call(self, inputs, training=None, mask=None): 363 | 364 | out = self.spatial_encoder(inputs, training=training, mask=mask) 365 | out = self.temporal_encoder(out, training=training, mask=mask) 366 | out = self.mlp4(out, training=training, mask=mask) 367 | 368 | return out 369 | -------------------------------------------------------------------------------- /eoflow/models/layers.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | from tensorflow.keras.layers import Activation, SpatialDropout1D, Lambda, UpSampling2D, AveragePooling2D 4 | from tensorflow.keras.layers import Conv1D, BatchNormalization, LayerNormalization 5 | 6 | 7 | class ResidualBlock(tf.keras.layers.Layer): 8 | """ Code taken from keras-tcn implementation on available on 9 | https://github.com/philipperemy/keras-tcn/blob/master/tcn/tcn.py#L140 """ 10 | def __init__(self, 11 | dilation_rate, 12 | nb_filters, 13 | kernel_size, 14 | padding, 15 | activation='relu', 16 | dropout_rate=0, 17 | kernel_initializer='he_normal', 18 | use_batch_norm=False, 19 | use_layer_norm=False, 20 | last_block=True, 21 | **kwargs): 22 | 23 | """ Defines the residual block for the WaveNet TCN 24 | 25 | :param dilation_rate: The dilation power of 2 we are using for this residual block 26 | :param nb_filters: The number of convolutional filters to use in this block 27 | :param kernel_size: The size of the convolutional kernel 28 | :param padding: The padding used in the convolutional layers, 'same' or 'causal'. 29 | :param activation: The final activation used in o = Activation(x + F(x)) 30 | :param dropout_rate: Float between 0 and 1. Fraction of the input units to drop. 31 | :param kernel_initializer: Initializer for the kernel weights matrix (Conv1D). 32 | :param use_batch_norm: Whether to use batch normalization in the residual layers or not. 33 | :param use_layer_norm: Whether to use layer normalization in the residual layers or not. 34 | :param last_block: Whether to add a residual connection to the convolution layer or not. 35 | :param kwargs: Any initializers for Layer class. 36 | """ 37 | 38 | self.dilation_rate = dilation_rate 39 | self.nb_filters = nb_filters 40 | self.kernel_size = kernel_size 41 | self.padding = padding 42 | self.activation = activation 43 | self.dropout_rate = dropout_rate 44 | self.use_batch_norm = use_batch_norm 45 | self.use_layer_norm = use_layer_norm 46 | self.kernel_initializer = kernel_initializer 47 | self.last_block = last_block 48 | self.residual_layers = list() 49 | self.shape_match_conv = None 50 | self.res_output_shape = None 51 | self.final_activation = None 52 | 53 | super(ResidualBlock, self).__init__(**kwargs) 54 | 55 | def _add_and_activate_layer(self, layer): 56 | """Helper function for building layer 57 | Args: 58 | layer: Appends layer to internal layer list and builds it based on the current output 59 | shape of ResidualBlocK. Updates current output shape. 60 | """ 61 | self.residual_layers.append(layer) 62 | self.residual_layers[-1].build(self.res_output_shape) 63 | self.res_output_shape = self.residual_layers[-1].compute_output_shape(self.res_output_shape) 64 | 65 | def build(self, input_shape): 66 | 67 | with tf.keras.backend.name_scope(self.name): # name scope used to make sure weights get unique names 68 | self.res_output_shape = input_shape 69 | 70 | for k in range(2): 71 | name = f'conv1D_{k}' 72 | with tf.keras.backend.name_scope(name): # name scope used to make sure weights get unique names 73 | self._add_and_activate_layer(Conv1D(filters=self.nb_filters, 74 | kernel_size=self.kernel_size, 75 | dilation_rate=self.dilation_rate, 76 | padding=self.padding, 77 | name=name, 78 | kernel_initializer=self.kernel_initializer)) 79 | 80 | if self.use_batch_norm: 81 | self._add_and_activate_layer(BatchNormalization()) 82 | elif self.use_layer_norm: 83 | self._add_and_activate_layer(LayerNormalization()) 84 | 85 | self._add_and_activate_layer(Activation('relu')) 86 | self._add_and_activate_layer(SpatialDropout1D(rate=self.dropout_rate)) 87 | 88 | if not self.last_block: 89 | # 1x1 conv to match the shapes (channel dimension). 90 | name = f'conv1D_{k+1}' 91 | with tf.keras.backend.name_scope(name): 92 | # make and build this layer separately because it directly uses input_shape 93 | self.shape_match_conv = Conv1D(filters=self.nb_filters, 94 | kernel_size=1, 95 | padding='same', 96 | name=name, 97 | kernel_initializer=self.kernel_initializer) 98 | 99 | else: 100 | self.shape_match_conv = Lambda(lambda x: x, name='identity') 101 | 102 | self.shape_match_conv.build(input_shape) 103 | self.res_output_shape = self.shape_match_conv.compute_output_shape(input_shape) 104 | 105 | self.final_activation = Activation(self.activation) 106 | self.final_activation.build(self.res_output_shape) # probably isn't necessary 107 | 108 | # this is done to force keras to add the layers in the list to self._layers 109 | for layer in self.residual_layers: 110 | self.__setattr__(layer.name, layer) 111 | 112 | super(ResidualBlock, self).build(input_shape) # done to make sure self.built is set True 113 | 114 | def call(self, inputs, training=None): 115 | """ 116 | Returns: A tuple where the first element is the residual model tensor, and the second 117 | is the skip connection tensor. 118 | """ 119 | x = inputs 120 | for layer in self.residual_layers: 121 | if isinstance(layer, SpatialDropout1D): 122 | x = layer(x, training=training) 123 | else: 124 | x = layer(x) 125 | 126 | x2 = self.shape_match_conv(inputs) 127 | res_x = tf.keras.layers.add([x2, x]) 128 | return [self.final_activation(res_x), x] 129 | 130 | def compute_output_shape(self, input_shape): 131 | return [self.res_output_shape, self.res_output_shape] 132 | 133 | 134 | class Conv2D(tf.keras.layers.Layer): 135 | """ Multiple repetitions of 2d convolution, batch normalization and dropout layers. """ 136 | 137 | def __init__(self, filters, kernel_size=3, strides=1, dilation=1, padding='VALID', add_dropout=True, 138 | dropout_rate=0.2, activation='relu', batch_normalization=False, use_bias=True, num_repetitions=1): 139 | super().__init__() 140 | 141 | repetitions = [] 142 | 143 | for i in range(num_repetitions): 144 | layer = [] 145 | layer.append(tf.keras.layers.Conv2D( 146 | filters=filters, 147 | kernel_size=kernel_size, 148 | strides=strides, 149 | dilation_rate=dilation, 150 | padding=padding, 151 | use_bias=use_bias, 152 | activation=activation 153 | )) 154 | 155 | if batch_normalization: 156 | layer.append(tf.keras.layers.BatchNormalization()) 157 | 158 | if add_dropout: 159 | layer.append(tf.keras.layers.Dropout(rate=dropout_rate)) 160 | 161 | layer = tf.keras.Sequential(layer) 162 | 163 | repetitions.append(layer) 164 | 165 | self.combined_layer = tf.keras.Sequential(repetitions) 166 | 167 | def call(self, inputs, training=False): 168 | return self.combined_layer(inputs, training=training) 169 | 170 | 171 | class ResConv2D(tf.keras.layers.Layer): 172 | """ 173 | Layer of N residual convolutional blocks stacked in parallel 174 | 175 | This layer stacks in parallel a sequence of 2 2D convolutional layers and returns the addition of their output 176 | feature tensors with the input tensor. N number of convolutional blocks can be added together with different kernel 177 | size and dilation rate, which are specified as a list. If the inputs are not a list, the same parameters are used 178 | for all convolutional blocks. 179 | 180 | """ 181 | 182 | def __init__(self, filters, kernel_size=3, strides=1, dilation=1, padding='VALID', add_dropout=True, 183 | dropout_rate=0.2, activation='relu', use_bias=True, batch_normalization=False, num_parallel=1): 184 | super().__init__() 185 | 186 | if isinstance(kernel_size, list) and len(kernel_size) != num_parallel: 187 | raise ValueError('Number of specified kernel sizes needs to match num_parallel') 188 | 189 | if isinstance(dilation, list) and len(dilation) != num_parallel: 190 | raise ValueError('Number of specified dilation rate sizes needs to match num_parallel') 191 | 192 | kernel_list = kernel_size if isinstance(kernel_size, list) else [kernel_size]*num_parallel 193 | dilation_list = dilation if isinstance(dilation, list) else [dilation]*num_parallel 194 | 195 | self.convs = [Conv2D(filters, 196 | kernel_size=k, 197 | strides=strides, 198 | dilation=d, 199 | padding=padding, 200 | activation=activation, 201 | add_dropout=add_dropout, 202 | dropout_rate=dropout_rate, 203 | use_bias=use_bias, 204 | batch_normalization=batch_normalization, 205 | num_repetitions=2) for k, d in zip(kernel_list, dilation_list)] 206 | 207 | self.add = tf.keras.layers.Add() 208 | 209 | def call(self, inputs, training=False): 210 | outputs = [conv_layer(inputs, training=training) for conv_layer in self.convs] 211 | 212 | return self.add(outputs + [inputs]) 213 | 214 | 215 | class Conv3D(tf.keras.layers.Layer): 216 | """ Multiple repetitions of 3d convolution, batch normalization and dropout layers. """ 217 | 218 | def __init__(self, filters, kernel_size=3, strides=1, padding='VALID', add_dropout=True, dropout_rate=0.2, 219 | batch_normalization=False, use_bias=True, num_repetitions=1, convolve_time=True): 220 | super().__init__() 221 | 222 | repetitions = [] 223 | 224 | t_size = kernel_size if convolve_time else 1 225 | kernel_shape = (t_size, kernel_size, kernel_size) 226 | 227 | for i in range(num_repetitions): 228 | layer = [] 229 | layer.append(tf.keras.layers.Conv3D( 230 | filters=filters, 231 | kernel_size=kernel_shape, 232 | strides=strides, 233 | padding=padding, 234 | use_bias=use_bias, 235 | activation='relu' 236 | )) 237 | 238 | if batch_normalization: 239 | layer.append(tf.keras.layers.BatchNormalization()) 240 | 241 | if add_dropout: 242 | layer.append(tf.keras.layers.Dropout(rate=dropout_rate)) 243 | 244 | layer = tf.keras.Sequential(layer) 245 | 246 | repetitions.append(layer) 247 | 248 | self.combined_layer = tf.keras.Sequential(repetitions) 249 | 250 | def call(self, inputs, training=False): 251 | return self.combined_layer(inputs, training=training) 252 | 253 | 254 | class Deconv2D(tf.keras.layers.Layer): 255 | """ 2d transpose convolution with optional batch normalization. """ 256 | 257 | def __init__(self, filters, kernel_size=2, batch_normalization=False): 258 | super().__init__() 259 | 260 | layer = [] 261 | layer.append(tf.keras.layers.Conv2DTranspose( 262 | filters=filters, 263 | kernel_size=kernel_size, 264 | strides=kernel_size, 265 | padding='SAME', 266 | activation='relu' 267 | )) 268 | 269 | if batch_normalization: 270 | layer.append(tf.keras.layers.BatchNormalization()) 271 | 272 | self.layer = tf.keras.Sequential(layer) 273 | 274 | def call(self, inputs, training=None): 275 | return self.layer(inputs, training=training) 276 | 277 | 278 | class CropAndConcat(tf.keras.layers.Layer): 279 | """ Layer that crops the first tensor and concatenates it with the second. Used for skip connections. """ 280 | @staticmethod 281 | def call(x1, x2): 282 | # Crop x1 to shape of x2 283 | x2_shape = tf.shape(x2) 284 | x1_crop = tf.image.resize_with_crop_or_pad(x1, x2_shape[1], x2_shape[2]) 285 | 286 | # Concatenate along last dimension and return 287 | return tf.concat([x1_crop, x2], axis=-1) 288 | 289 | 290 | class MaxPool3D(tf.keras.layers.Layer): 291 | def __init__(self, kernel_size=2, strides=2, pool_time=False): 292 | super().__init__() 293 | 294 | tsize = kernel_size if pool_time else 1 295 | tstride = strides if pool_time else 1 296 | 297 | kernel_shape = (tsize, kernel_size, kernel_size) 298 | strides = (tstride, strides, strides) 299 | 300 | self.layer = tf.keras.layers.MaxPool3D( 301 | pool_size=kernel_shape, 302 | strides=strides, 303 | padding='SAME' 304 | ) 305 | 306 | def call(self, inputs, training=None): 307 | return self.layer(inputs, training=training) 308 | 309 | 310 | class Reduce3DTo2D(tf.keras.layers.Layer): 311 | """ Reduces 3d representations into 2d using 3d convolution over the whole time dimension. """ 312 | 313 | def __init__(self, filters, kernel_size=3, stride=1, add_dropout=False, dropout_rate=0.2): 314 | super().__init__() 315 | 316 | self.filters = filters 317 | self.kernel_size = kernel_size 318 | self.stride = stride 319 | self.add_dropout = add_dropout 320 | self.dropout_rate = dropout_rate 321 | self.layer = None 322 | 323 | def build(self, input_size): 324 | t_size = input_size[1] 325 | layer = [] 326 | layer.append(tf.keras.layers.Conv3D( 327 | self.filters, 328 | kernel_size=(t_size, self.kernel_size, self.kernel_size), 329 | strides=(1, self.stride, self.stride), 330 | padding='VALID', 331 | activation='relu' 332 | )) 333 | 334 | if self.add_dropout: 335 | layer.append(tf.keras.layers.Dropout(rate=self.dropout_rate)) 336 | 337 | self.layer = tf.keras.Sequential(layer) 338 | 339 | def call(self, inputs, training=None): 340 | r = self.layer(inputs, training=training) 341 | 342 | # Squeeze along temporal dimension 343 | return tf.squeeze(r, axis=[1]) 344 | 345 | 346 | class PyramidPoolingModule(tf.keras.layers.Layer): 347 | """ 348 | Implementation of the Pyramid Pooling Module 349 | 350 | Implementation taken from the following paper 351 | 352 | Zhao et al. - Pyramid Scene Parsing Network - https://arxiv.org/pdf/1612.01105.pdf 353 | 354 | PyTorch implementation https://github.com/hszhao/semseg/blob/master/model/pspnet.py 355 | """ 356 | def __init__(self, filters, bins=(1, 2, 4, 8), interpolation='bilinear', batch_normalization=False): 357 | super().__init__() 358 | 359 | self.filters = filters 360 | self.bins = bins 361 | self.batch_normalization = batch_normalization 362 | self.interpolation = interpolation 363 | self.layers = None 364 | 365 | def build(self, input_size): 366 | _, height, width, n_features = input_size 367 | 368 | layers = [] 369 | 370 | for bin_size in self.bins: 371 | 372 | size_factors = height // bin_size, width // bin_size 373 | 374 | layer = tf.keras.Sequential() 375 | layer.add(AveragePooling2D(pool_size=size_factors, 376 | padding='same')) 377 | layer.add(tf.keras.layers.Conv2D(filters=self.filters//len(self.bins), 378 | kernel_size=1, 379 | padding='same', 380 | use_bias=False)) 381 | if self.batch_normalization: 382 | layer.add(BatchNormalization()) 383 | layer.add(Activation('relu')) 384 | 385 | layer.add(UpSampling2D(size=size_factors, interpolation=self.interpolation)) 386 | 387 | layers.append(layer) 388 | 389 | self.layers = layers 390 | 391 | def call(self, inputs, training=None): 392 | """ Concatenate the output of the pooling layers, resampled to original size """ 393 | _, height, width, _ = inputs.shape 394 | 395 | outputs = [inputs] 396 | 397 | outputs += [layer(inputs, training=training) for layer in self.layers] 398 | 399 | return tf.concat(outputs, axis=-1) 400 | -------------------------------------------------------------------------------- /eoflow/models/losses.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.keras.losses import Loss, Reduction 3 | 4 | 5 | def cropped_loss(loss_fn): 6 | """ Wraps loss function. Crops the labels to match the logits size. """ 7 | 8 | def _loss_fn(labels, logits): 9 | logits_shape = tf.shape(logits) 10 | labels_crop = tf.image.resize_with_crop_or_pad(labels, logits_shape[1], logits_shape[2]) 11 | 12 | return loss_fn(labels_crop, logits) 13 | 14 | return _loss_fn 15 | 16 | 17 | class CategoricalCrossEntropy(Loss): 18 | """ Wrapper class for cross-entropy with class weights """ 19 | def __init__(self, from_logits=True, class_weights=None, reduction=Reduction.AUTO, name='FocalLoss'): 20 | """Categorical cross-entropy. 21 | 22 | :param from_logits: Whether predictions are logits or softmax, defaults to True 23 | :type from_logits: bool 24 | :param class_weights: Array of class weights to be applied to loss. Needs to be of `n_classes` length 25 | :type class_weights: np.array 26 | :param reduction: reduction to be used, defaults to Reduction.AUTO 27 | :type reduction: tf.keras.losses.Reduction, optional 28 | :param name: name of the loss, defaults to 'FocalLoss' 29 | :type name: str 30 | """ 31 | super().__init__(reduction=reduction, name=name) 32 | 33 | self.from_logits = from_logits 34 | self.class_weights = class_weights 35 | 36 | def call(self, y_true, y_pred): 37 | # Perform softmax 38 | if self.from_logits: 39 | y_pred = tf.nn.softmax(y_pred) 40 | 41 | # Clip the prediction value to prevent NaN's and Inf's 42 | epsilon = tf.keras.backend.epsilon() 43 | y_pred = tf.clip_by_value(y_pred, epsilon, 1.) 44 | 45 | # Calculate Cross Entropy 46 | loss = -y_true * tf.math.log(y_pred) 47 | 48 | # Multiply cross-entropy with class-wise weights 49 | if self.class_weights is not None: 50 | loss = tf.multiply(loss, self.class_weights) 51 | 52 | # Sum over classes 53 | loss = tf.reduce_sum(loss, axis=-1) 54 | 55 | return loss 56 | 57 | 58 | class CategoricalFocalLoss(Loss): 59 | """ Categorical version of focal loss. 60 | 61 | References: 62 | Official paper: https://arxiv.org/pdf/1708.02002.pdf 63 | Keras implementation: https://github.com/umbertogriffo/focal-loss-keras 64 | """ 65 | 66 | def __init__(self, gamma=2., alpha=.25, from_logits=True, class_weights=None, reduction=Reduction.AUTO, 67 | name='FocalLoss'): 68 | """Categorical version of focal loss. 69 | 70 | :param gamma: gamma value, defaults to 2. 71 | :type gamma: float 72 | :param alpha: alpha value, defaults to .25 73 | :type alpha: float 74 | :param from_logits: Whether predictions are logits or softmax, defaults to True 75 | :type from_logits: bool 76 | :param class_weights: Array of class weights to be applied to loss. Needs to be of `n_classes` length 77 | :type class_weights: np.array 78 | :param reduction: reduction to be used, defaults to Reduction.AUTO 79 | :type reduction: tf.keras.losses.Reduction, optional 80 | :param name: name of the loss, defaults to 'FocalLoss' 81 | :type name: str 82 | """ 83 | super().__init__(reduction=reduction, name=name) 84 | 85 | self.gamma = gamma 86 | self.alpha = alpha 87 | self.from_logits = from_logits 88 | self.class_weights = class_weights 89 | 90 | def call(self, y_true, y_pred): 91 | 92 | # Perform softmax 93 | if self.from_logits: 94 | y_pred = tf.nn.softmax(y_pred) 95 | 96 | # Clip the prediction value to prevent NaN's and Inf's 97 | epsilon = tf.keras.backend.epsilon() 98 | y_pred = tf.clip_by_value(y_pred, epsilon, 1.) 99 | 100 | # Calculate Cross Entropy 101 | cross_entropy = -y_true * tf.math.log(y_pred) 102 | 103 | # Calculate Focal Loss 104 | loss = self.alpha * tf.math.pow(1 - y_pred, self.gamma) * cross_entropy 105 | 106 | # Multiply focal loss with class-wise weights 107 | if self.class_weights is not None: 108 | loss = tf.multiply(cross_entropy, self.class_weights) 109 | 110 | # Sum over classes 111 | loss = tf.reduce_sum(loss, axis=-1) 112 | 113 | return loss 114 | 115 | 116 | class JaccardDistanceLoss(Loss): 117 | """ Implementation of the Jaccard distance, or Intersection over Union IoU loss. 118 | 119 | Jaccard = (|X & Y|)/ (|X|+ |Y| - |X & Y|) 120 | = sum(|A*B|)/(sum(|A|)+sum(|B|)-sum(|A*B|)) 121 | 122 | Implementation taken from https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/losses/jaccard.py 123 | """ 124 | def __init__(self, smooth=1, from_logits=True, class_weights=None, reduction=Reduction.AUTO, name='JaccardLoss'): 125 | """ Jaccard distance loss. 126 | 127 | :param smooth: Smoothing factor. Default is 1. 128 | :type smooth: int 129 | :param from_logits: Whether predictions are logits or softmax, defaults to True 130 | :type from_logits: bool 131 | :param class_weights: Array of class weights to be applied to loss. Needs to be of `n_classes` length 132 | :type class_weights: np.array 133 | :param reduction: reduction to be used, defaults to Reduction.AUTO 134 | :type reduction: tf.keras.losses.Reduction, optional 135 | :param name: name of the loss, defaults to 'JaccardLoss' 136 | :type name: str 137 | """ 138 | super().__init__(reduction=reduction, name=name) 139 | 140 | self.smooth = smooth 141 | self.from_logits = from_logits 142 | 143 | self.class_weights = class_weights 144 | 145 | def call(self, y_true, y_pred): 146 | 147 | # Perform softmax 148 | if self.from_logits: 149 | y_pred = tf.nn.softmax(y_pred) 150 | 151 | intersection = tf.reduce_sum(y_true * y_pred, axis=(1, 2)) 152 | 153 | sum_ = tf.reduce_sum(y_true + y_pred, axis=(1, 2)) 154 | 155 | jac = (intersection + self.smooth) / (sum_ - intersection + self.smooth) 156 | 157 | loss = (1 - jac) * self.smooth 158 | 159 | if self.class_weights is not None: 160 | loss = tf.multiply(loss, self.class_weights) 161 | 162 | loss = tf.reduce_sum(loss, axis=-1) 163 | 164 | return loss 165 | 166 | 167 | class TanimotoDistanceLoss(Loss): 168 | """ Implementation of the Tanimoto distance, which is modified version of the Jaccard distance. 169 | 170 | Tanimoto = (|X & Y|)/ (|X|^2+ |Y|^2 - |X & Y|) 171 | = sum(|A*B|)/(sum(|A|^2)+sum(|B|^2)-sum(|A*B|)) 172 | 173 | Implementation taken from 174 | https://github.com/feevos/resuneta/blob/145be5519ee4bec9a8cce9e887808b8df011f520/nn/loss/loss.py#L7 175 | """ 176 | def __init__(self, smooth=1.0e-5, from_logits=True, class_weights=None, 177 | reduction=Reduction.AUTO, normalise=False, name='TanimotoLoss'): 178 | """ Tanimoto distance loss. 179 | 180 | :param smooth: Smoothing factor. Default is 1.0e-5. 181 | :type smooth: float 182 | :param from_logits: Whether predictions are logits or softmax, defaults to True 183 | :type from_logits: bool 184 | :param class_weights: Array of class weights to be applied to loss. Needs to be of `n_classes` length 185 | :type class_weights: np.array 186 | :param reduction: Reduction to be used, defaults to Reduction.AUTO 187 | :type reduction: tf.keras.losses.Reduction, optional 188 | :param normalise: Whether to normalise loss by number of positive samples in class, defaults to `False` 189 | :type normalise: bool 190 | :param name: Name of the loss, defaults to 'TanimotoLoss' 191 | :type name: str 192 | """ 193 | super().__init__(reduction=reduction, name=name) 194 | 195 | self.smooth = smooth 196 | self.from_logits = from_logits 197 | self.normalise = normalise 198 | 199 | self.class_weights = class_weights 200 | 201 | def call(self, y_true, y_pred): 202 | 203 | # Perform softmax 204 | if self.from_logits: 205 | y_pred = tf.nn.softmax(y_pred) 206 | 207 | n_classes = y_true.shape[-1] 208 | 209 | volume = tf.reduce_mean(tf.reduce_sum(y_true, axis=(1, 2)), axis=0) \ 210 | if self.normalise else tf.ones(n_classes, dtype=tf.float32) 211 | 212 | weights = tf.math.reciprocal(tf.math.square(volume)) 213 | new_weights = tf.where(tf.math.is_inf(weights), tf.zeros_like(weights), weights) 214 | weights = tf.where(tf.math.is_inf(weights), tf.ones_like(weights) * tf.reduce_max(new_weights), weights) 215 | 216 | intersection = tf.reduce_sum(y_true * y_pred, axis=(1, 2)) 217 | 218 | sum_ = tf.reduce_sum(y_true * y_true + y_pred * y_pred, axis=(1, 2)) 219 | 220 | num_ = tf.multiply(intersection, weights) + self.smooth 221 | 222 | den_ = tf.multiply(sum_ - intersection, weights) + self.smooth 223 | 224 | tanimoto = num_ / den_ 225 | 226 | loss = (1 - tanimoto) 227 | 228 | if self.class_weights is not None: 229 | loss = tf.multiply(loss, self.class_weights) 230 | 231 | loss = tf.reduce_sum(loss, axis=-1) 232 | 233 | return loss 234 | -------------------------------------------------------------------------------- /eoflow/models/metrics.py: -------------------------------------------------------------------------------- 1 | import warnings 2 | 3 | from typing import Any, Callable, List 4 | 5 | from skimage import measure 6 | from scipy import ndimage 7 | 8 | import tensorflow as tf 9 | import tensorflow_addons as tfa 10 | import numpy as np 11 | 12 | 13 | class InitializableMetric(tf.keras.metrics.Metric): 14 | """ Metric that has to be initialized from model configuration. """ 15 | 16 | def __init__(self, *args, **kwargs): 17 | super().__init__(*args, **kwargs) 18 | self.initialized = False 19 | 20 | def init_from_config(self, model_config=None): 21 | """ Initializes the metric from configuration. """ 22 | 23 | self.initialized = True 24 | 25 | def assert_initialized(self): 26 | """ Checks if the metric is initialized. """ 27 | 28 | if not self.initialized: 29 | raise ValueError("InitializableMetric was not initialized before use.") 30 | 31 | 32 | class MeanIoU(InitializableMetric): 33 | """ Computes mean intersection over union metric for semantic segmentation. 34 | Wraps keras MeanIoU to work on logits. """ 35 | 36 | def __init__(self, default_max_classes=32, name='mean_iou'): 37 | """ Creates MeanIoU metric 38 | 39 | :param default_max_classes: Default value for max number of classes. Required by Keras MeanIoU. 40 | Must be greater or equal to the actual number of classes. 41 | Will not be used if n_classes is in model configuration. Defaults to 32. 42 | :type default_max_classes: int 43 | :param name: Name of the metric 44 | :type name: str 45 | """ 46 | 47 | super().__init__(name=name, dtype=tf.float32) 48 | self.default_max_classes = default_max_classes 49 | self.metric = None 50 | 51 | def init_from_config(self, model_config=None): 52 | super().init_from_config(model_config) 53 | 54 | if model_config is not None and 'n_classes' in model_config: 55 | self.metric = tf.keras.metrics.MeanIoU(num_classes=model_config['n_classes']) 56 | else: 57 | print("n_classes not found in model config or model config not provided. Using default max value.") 58 | self.metric = tf.keras.metrics.MeanIoU(num_classes=self.default_max_classes) 59 | 60 | def update_state(self, y_true, y_pred, sample_weight=None): 61 | self.assert_initialized() 62 | 63 | y_pred_c = tf.argmax(y_pred, axis=-1) 64 | y_true_c = tf.argmax(y_true, axis=-1) 65 | 66 | return self.metric.update_state(y_true_c, y_pred_c, sample_weight) 67 | 68 | def result(self): 69 | self.assert_initialized() 70 | 71 | return self.metric.result() 72 | 73 | def reset_states(self): 74 | self.assert_initialized() 75 | 76 | return self.metric.reset_states() 77 | 78 | def get_config(self): 79 | self.assert_initialized() 80 | 81 | return self.metric.get_config() 82 | 83 | 84 | class CroppedMetric(tf.keras.metrics.Metric): 85 | """ Wraps a metric. Crops the labels to match the logits size. """ 86 | 87 | def __init__(self, metric): 88 | super().__init__(name=metric.name, dtype=metric.dtype) 89 | self.metric = metric 90 | 91 | def update_state(self, y_true, y_pred, sample_weight=None): 92 | logits_shape = tf.shape(y_pred) 93 | labels_crop = tf.image.resize_with_crop_or_pad(y_true, logits_shape[1], logits_shape[2]) 94 | 95 | return self.metric.update_state(labels_crop, y_pred, sample_weight) 96 | 97 | def result(self): 98 | return self.metric.result() 99 | 100 | def reset_states(self): 101 | return self.metric.reset_states() 102 | 103 | def get_config(self): 104 | return self.metric.get_config() 105 | 106 | 107 | class MCCMetric(InitializableMetric): 108 | """ Computes Mathew Correlation Coefficient metric. Wraps metrics.MatthewsCorrelationCoefficient from 109 | tensorflow-addons, and reshapes the input (logits) into (m, n_classes) tensors. The logits are thresholded to get 110 | "one-hot encoded" values for (multi)class metrics """ 111 | 112 | def __init__(self, default_n_classes=2, default_threshold=0.5, name='mcc'): 113 | """ Creates MCCMetric metric 114 | 115 | :param default_n_classes: Default number of classes 116 | :type default_n_classes: int 117 | :param default_threshold: Default value for threshold 118 | :type default_threshold: float 119 | :param name: Name of the metric 120 | :type name: str 121 | """ 122 | 123 | super().__init__(name=name, dtype=tf.float32) 124 | self.metric = None 125 | self.default_n_classes = default_n_classes 126 | self.threshold = default_threshold 127 | 128 | def init_from_config(self, model_config=None): 129 | super().init_from_config(model_config) 130 | 131 | if model_config is not None and 'n_classes' in model_config: 132 | self.metric = tfa.metrics.MatthewsCorrelationCoefficient(num_classes=model_config['n_classes']) 133 | else: 134 | print("n_classes not found in model config or model config not provided. Using default max value.") 135 | self.metric = tfa.metrics.MatthewsCorrelationCoefficient(num_classes=self.default_n_classes) 136 | 137 | if model_config is not None and 'mcc_threshold' in model_config: 138 | self.threshold = model_config['mcc_threshold'] 139 | else: 140 | print(f"Using default value for threshold: {self.threshold}.") 141 | 142 | self.metric = tfa.metrics.MatthewsCorrelationCoefficient(num_classes=model_config['n_classes']) 143 | 144 | def update_state(self, y_true, y_pred, sample_weight=None): 145 | self.assert_initialized() 146 | 147 | n = tf.math.reduce_prod(tf.shape(y_pred)[:-1]) 148 | y_pred_c = tf.reshape(y_pred > self.threshold, (n, self.metric.num_classes)) 149 | y_true_c = tf.reshape(y_true, (n, self.metric.num_classes)) 150 | 151 | return self.metric.update_state(y_true_c, y_pred_c, sample_weight=sample_weight) 152 | 153 | def result(self): 154 | self.assert_initialized() 155 | 156 | return self.metric.result() 157 | 158 | def reset_states(self): 159 | self.assert_initialized() 160 | 161 | return self.metric.reset_states() 162 | 163 | def get_config(self): 164 | self.assert_initialized() 165 | 166 | return self.metric.get_config() 167 | 168 | 169 | class GeometricMetrics(InitializableMetric): 170 | """" 171 | Implementation of Geometric error metrics. Oversegmentation, Undersegmentation, Border, Fragmentation errors. 172 | 173 | The error metrics are based on a paper by C. Persello, A Novel Protocol for Accuracy Assessment in Classification of 174 | Very High Resolution Images (https://ieeexplore.ieee.org/document/5282610) 175 | """ 176 | 177 | @staticmethod 178 | def _detect_edges(im: np.ndarray, thr: float = 0) -> np.ndarray: 179 | """ Edge detection function using the sobel operator. """ 180 | sx = ndimage.sobel(im, axis=0, mode='constant') 181 | sy = ndimage.sobel(im, axis=1, mode='constant') 182 | sob = np.hypot(sx, sy) 183 | return sob > thr 184 | 185 | @staticmethod 186 | def _segmentation_error(intersection_area: float, object_area: float) -> float: 187 | return 1. - intersection_area / object_area 188 | 189 | @staticmethod 190 | def _intersection(mask1: np.ndarray, mask2: np.ndarray) -> float: 191 | return np.sum(np.logical_and(mask1, mask2)) 192 | 193 | def _border_err(self, border_ref_edge: np.ndarray, border_meas_edge: np.ndarray) -> float: 194 | ref_edge_size = np.sum(border_ref_edge) 195 | intersection = self._intersection(border_ref_edge, border_meas_edge) 196 | err = intersection / ref_edge_size if ref_edge_size != 0 else 0 197 | be = 1. - err 198 | return be 199 | 200 | def _fragmentation_err(self, r: int, reference_mask: np.ndarray) -> float: 201 | if r <= 1: 202 | return 0 203 | den = np.sum(reference_mask) - self.pixel_size 204 | err = (r - 1.) / den if den > 0 else 0 205 | return err 206 | 207 | @staticmethod 208 | def _validate_input(reference, measurement): 209 | if np.ndim(reference) != np.ndim(measurement): 210 | raise ValueError("Reference and measurement input shapes must match.") 211 | 212 | def __init__(self, pixel_size: int = 1, edge_func: Callable = None, **edge_func_params: Any): 213 | 214 | super().__init__(name='geometric_metrics', dtype=tf.float64) 215 | 216 | self.oversegmentation_error = [] 217 | self.undersegmentation_error = [] 218 | self.border_error = [] 219 | self.fragmentation_error = [] 220 | 221 | self.edge_func = self._detect_edges if edge_func is None else edge_func 222 | self.edge_func_params = edge_func_params 223 | self.pixel_size = pixel_size 224 | 225 | def update_state(self, reference: np.ndarray, measurement: np.ndarray, encode_reference: bool = True, 226 | background_value: int = 0) -> None: 227 | """ Calculate the error metrics for a measurement and reference arrays. For each . 228 | 229 | If encode_reference is set to True, connected components will be used to label objects in the reference and 230 | measurements. 231 | """ 232 | 233 | if not tf.executing_eagerly(): 234 | warnings.warn("Geometric metrics must be run with eager execution. If running as a compiled Keras model, " 235 | "enable eager execution with model.run_eagerly = True") 236 | 237 | reference = reference.numpy() if isinstance(reference, tf.Tensor) else reference 238 | measurement = measurement.numpy() if isinstance(reference, tf.Tensor) else measurement 239 | 240 | self._validate_input(reference, measurement) 241 | 242 | for ref, meas in zip(reference, measurement): 243 | ref = ref 244 | meas = meas 245 | 246 | if encode_reference: 247 | cc_reference = measure.label(ref, background=background_value) 248 | else: 249 | cc_reference = ref 250 | 251 | cc_measurement = measure.label(meas, background=background_value) 252 | components_reference = set(np.unique(cc_reference)).difference([background_value]) 253 | 254 | ref_edges = self.edge_func(cc_reference) 255 | meas_edges = self.edge_func(cc_measurement) 256 | for component in components_reference: 257 | reference_mask = cc_reference == component 258 | 259 | uniq, count = np.unique(cc_measurement[reference_mask & (cc_measurement != background_value)], 260 | return_counts=True) 261 | ref_area = np.sum(reference_mask) 262 | 263 | max_interecting_measurement = uniq[count.argmax()] if len(count) > 0 else background_value 264 | meas_mask = cc_measurement == max_interecting_measurement 265 | meas_area = np.count_nonzero(cc_measurement == max_interecting_measurement) 266 | intersection_area = count.max() if len(count) > 0 else 0 267 | 268 | self.oversegmentation_error.append(self._segmentation_error(intersection_area, ref_area)) 269 | self.undersegmentation_error.append(self._segmentation_error(intersection_area, meas_area)) 270 | border_ref_edge = ref_edges.squeeze() & reference_mask.squeeze() 271 | border_meas_edge = meas_edges.squeeze() & meas_mask.squeeze() 272 | 273 | self.border_error.append(self._border_err(border_ref_edge, border_meas_edge)) 274 | self.fragmentation_error.append(self._fragmentation_err(len(uniq), reference_mask)) 275 | 276 | def get_oversegmentation_error(self) -> float: 277 | """ Return oversegmentation error. """ 278 | return np.array(self.oversegmentation_error).mean() 279 | 280 | def get_undersegmentation_error(self) -> float: 281 | """ Return undersegmentation error. """ 282 | 283 | return np.array(self.undersegmentation_error).mean() 284 | 285 | def get_border_error(self) -> float: 286 | """ Return border error. """ 287 | 288 | return np.array(self.border_error).mean() 289 | 290 | def get_fragmentation_error(self) -> float: 291 | """ Return fragmentation error. """ 292 | 293 | return np.array(self.fragmentation_error).mean() 294 | 295 | def result(self) -> List[float]: 296 | """ Return a list of values representing oversegmentation, undersegmentation, border, fragmentation errors. """ 297 | 298 | return [self.get_oversegmentation_error(), 299 | self.get_undersegmentation_error(), 300 | self.get_border_error(), self.get_fragmentation_error()] 301 | 302 | def reset_states(self) -> None: 303 | """ Empty all the error arrays. """ 304 | self.oversegmentation_error = [] 305 | self.undersegmentation_error = [] 306 | self.border_error = [] 307 | self.fragmentation_error = [] 308 | -------------------------------------------------------------------------------- /eoflow/models/pse_tae_layers.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | import tensorflow.keras.layers as L 4 | 5 | from .transformer_encoder_layers import scaled_dot_product_attention, positional_encoding 6 | 7 | pooling_methods = { 8 | 'mean': tf.math.reduce_mean, 9 | 'std': tf.math.reduce_std, 10 | 'max': tf.math.reduce_max, 11 | 'min': tf.math.reduce_min 12 | } 13 | 14 | class PixelSetEncoder(tf.keras.layers.Layer): 15 | def __init__(self, mlp1=[10, 32, 64], mlp2=[64, 128], pooling='mean_std'): 16 | super().__init__() 17 | 18 | self.mlp1 = tf.keras.Sequential([LinearLayer(out_dim) for out_dim in mlp1]) 19 | 20 | pooling_methods = [SetPooling(method) for method in pooling.split('_')] 21 | self.pooling = SummaryConcatenate(pooling_methods, axis=-1) 22 | 23 | mlp2_layers = [LinearLayer(out_dim) for out_dim in mlp2[:-1]] 24 | mlp2_layers.append(LinearLayer(mlp2[-1], activation=False)) 25 | self.mlp2 = tf.keras.Sequential(mlp2_layers) 26 | 27 | self.encoder = tf.keras.Sequential([ 28 | self.mlp1, 29 | self.pooling, 30 | self.mlp2 31 | ]) 32 | 33 | def call(self, x, training=None, mask=None): 34 | return self.encoder(x, training=training, mask=mask) 35 | 36 | 37 | class MultiHeadAttention(tf.keras.layers.Layer): 38 | def __init__(self, n_head, d_k, name='multi_head_attention'): 39 | super().__init__(name=name) 40 | 41 | self.n_head = n_head 42 | self.d_k = d_k 43 | 44 | self.fc1_q = L.Dense(d_k * n_head, 45 | kernel_initializer=tf.random_normal_initializer(mean=0, stddev=np.sqrt(2.0 / d_k))) 46 | 47 | self.fc1_k = L.Dense(d_k * n_head, 48 | kernel_initializer=tf.random_normal_initializer(mean=0, stddev=np.sqrt(2.0 / d_k))) 49 | 50 | self.fc2 = tf.keras.Sequential([ 51 | L.BatchNormalization(), 52 | L.Dense(d_k) 53 | ]) 54 | 55 | def split_heads(self, x, batch_size): 56 | """Split the last dimension into (n_head, d_k). 57 | Transpose the result such that the shape is (batch_size, n_head, seq_len, d_k) 58 | """ 59 | 60 | x = tf.reshape(x, (batch_size, -1, self.n_head, self.d_k)) 61 | return tf.transpose(x, perm=[0, 2, 1, 3]) 62 | 63 | def call(self, q, k, v, training=None, mask=None): 64 | batch_size = tf.shape(q)[0] 65 | 66 | q = self.fc1_q(q) 67 | q = self.split_heads(q, batch_size) 68 | q = tf.reduce_mean(q, axis=2, keepdims=True) # MEAN query 69 | 70 | k = self.fc1_k(k) 71 | k = self.split_heads(k, batch_size) 72 | 73 | # Repeat n_head times 74 | v = tf.expand_dims(v, axis=1) 75 | v = tf.tile(v, (1, self.n_head, 1, 1)) 76 | 77 | output, attn = scaled_dot_product_attention(q, k, v, mask) 78 | 79 | output = tf.squeeze(output, axis=2) 80 | 81 | # Concat heads 82 | output = tf.reshape(output, (batch_size, -1)) 83 | 84 | return output 85 | 86 | class TemporalAttentionEncoder(tf.keras.layers.Layer): 87 | def __init__(self, n_head=4, d_k=32, d_model=None, n_neurons=[512, 128, 128], dropout=0.2, 88 | T=1000, len_max_seq=24, positions=None): 89 | super().__init__() 90 | 91 | 92 | self.positions = positions 93 | if self.positions is None: 94 | self.positions = len_max_seq + 1 95 | 96 | self.d_model = d_model 97 | self.T = T 98 | 99 | self.in_layer_norm = tf.keras.layers.LayerNormalization(name='in_layer_norm') 100 | 101 | self.inconv = None 102 | if d_model is not None: 103 | self.inconv = tf.keras.Sequential([ 104 | L.Conv1D(d_model, 1, name='inconv'), 105 | L.LayerNormalization(name='conv_layer_norm') 106 | ]) 107 | 108 | self.out_layer_norm = tf.keras.layers.LayerNormalization(name='out_layer_norm') 109 | 110 | self.attention_heads = MultiHeadAttention(n_head, d_k, name='attention_heads') 111 | 112 | mlp_layers = [LinearLayer(out_dim) for out_dim in n_neurons] 113 | self.mlp = tf.keras.Sequential(mlp_layers, name='mlp') 114 | 115 | self.dropout = L.Dropout(dropout) 116 | 117 | def build(self, input_shape): 118 | d_in = input_shape[-1] if self.d_model is None else self.d_model 119 | self.position_enc = positional_encoding(self.positions, d_in, T=self.T) 120 | 121 | def call(self, x, training=None, mask=None): 122 | seq_len = tf.shape(x)[1] 123 | 124 | x = self.in_layer_norm(x, training=training) 125 | 126 | if self.inconv is not None: 127 | x = self.inconv(x, training=training) 128 | 129 | pos_encoding = self.position_enc[:, :seq_len, :] 130 | if self.positions is None: 131 | pos_encoding = self.position_enc[:, 1:seq_len+1, :] 132 | 133 | enc_output = x + pos_encoding 134 | 135 | enc_output = self.attention_heads(enc_output, enc_output, enc_output, training=training, mask=mask) 136 | 137 | enc_output = self.mlp(enc_output, training=training) 138 | enc_output = self.dropout(enc_output, training=training) 139 | enc_output = self.out_layer_norm(enc_output, training=training) 140 | 141 | return enc_output 142 | 143 | def LinearLayer(out_dim, batch_norm=True, activation=True): 144 | """ Linear layer. """ 145 | 146 | layers = [L.Dense(out_dim)] 147 | 148 | if batch_norm: 149 | layers.append(L.BatchNormalization()) 150 | 151 | if activation: 152 | layers.append(L.ReLU()) 153 | 154 | return tf.keras.Sequential(layers) 155 | 156 | class SetPooling(tf.keras.layers.Layer): 157 | """ Pooling over the Set dimension using a specified pooling method. """ 158 | def __init__(self, pooling_method): 159 | super().__init__() 160 | 161 | self.pooling_method = pooling_methods[pooling_method] 162 | 163 | def call(self, x, training=None, mask=None): 164 | return self.pooling_method(x, axis=1) 165 | 166 | class SummaryConcatenate(tf.keras.layers.Layer): 167 | """ Runs multiple summary layers on a single input and concatenates them. """ 168 | def __init__(self, layers, axis=-1): 169 | super().__init__() 170 | 171 | self.layers = layers 172 | self.axis = axis 173 | 174 | def call(self, x, training=None, mask=None): 175 | layer_outputs = [layer(x, training=training, mask=mask) for layer in self.layers] 176 | return L.concatenate(layer_outputs, axis=self.axis) 177 | -------------------------------------------------------------------------------- /eoflow/models/segmentation_base.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | 4 | import numpy as np 5 | import tensorflow as tf 6 | from marshmallow import Schema, fields 7 | from marshmallow.validate import OneOf, ContainsOnly 8 | 9 | from ..base import BaseModel 10 | 11 | from .losses import CategoricalCrossEntropy, CategoricalFocalLoss, JaccardDistanceLoss, TanimotoDistanceLoss 12 | from .losses import cropped_loss 13 | from .metrics import MeanIoU, InitializableMetric, CroppedMetric, MCCMetric 14 | from .callbacks import VisualizationCallback 15 | 16 | logging.basicConfig(level=logging.INFO, 17 | format='%(asctime)s %(levelname)s %(message)s') 18 | 19 | 20 | # Available losses. Add keys with new losses here. 21 | segmentation_losses = { 22 | 'cross_entropy': CategoricalCrossEntropy, 23 | 'focal_loss': CategoricalFocalLoss, 24 | 'jaccard_loss': JaccardDistanceLoss, 25 | 'tanimoto_loss': TanimotoDistanceLoss 26 | } 27 | 28 | 29 | # Available metrics. Add keys with new metrics here. 30 | segmentation_metrics = { 31 | 'accuracy': lambda: tf.keras.metrics.CategoricalAccuracy(name='accuracy'), 32 | 'iou': lambda: MeanIoU(default_max_classes=32), 33 | 'precision': tf.keras.metrics.Precision, 34 | 'recall': tf.keras.metrics.Recall, 35 | 'mcc': MCCMetric 36 | } 37 | 38 | 39 | class BaseSegmentationModel(BaseModel): 40 | """ Base for segmentation models. """ 41 | 42 | class _Schema(Schema): 43 | n_classes = fields.Int(required=True, description='Number of classes', example=2) 44 | learning_rate = fields.Float(missing=None, description='Learning rate used in training.', example=0.01) 45 | loss = fields.String(missing='cross_entropy', description='Loss function used for training.', 46 | validate=OneOf(segmentation_losses.keys())) 47 | metrics = fields.List(fields.String, missing=['accuracy', 'iou'], 48 | description='List of metrics used for evaluation.', 49 | validate=ContainsOnly(segmentation_metrics.keys())) 50 | 51 | class_weights = fields.Dict(missing=None, description='Dictionary mapping class id with weight. ' 52 | 'If key for some labels is not specified, 1 is used.') 53 | 54 | prediction_visualization = fields.Bool(missing=False, description='Record prediction visualization summaries.') 55 | prediction_visualization_num = fields.Int(missing=5, 56 | description='Number of images used for prediction visualization.') 57 | 58 | def _prepare_class_weights(self): 59 | """ Utility function to parse class weights """ 60 | if self.config.class_weights is None: 61 | return np.ones(self.config.n_classes) 62 | return np.array([self.config.class_weights[iclass] if iclass in self.config.class_weights else 1.0 63 | for iclass in range(self.config.n_classes)]) 64 | 65 | def prepare(self, optimizer=None, loss=None, metrics=None, **kwargs): 66 | """ Prepares the model. Optimizer, loss and metrics are read using the following protocol: 67 | * If an argument is None, the default value is used from the configuration of the model. 68 | * If an argument is a key contained in segmentation specific losses/metrics, those are used. 69 | * Otherwise the argument is passed to `compile` as is. 70 | 71 | """ 72 | # Read defaults if None 73 | if optimizer is None: 74 | optimizer = tf.keras.optimizers.Adam(learning_rate=self.config.learning_rate) 75 | 76 | if loss is None: 77 | loss = self.config.loss 78 | 79 | if metrics is None: 80 | metrics = self.config.metrics 81 | 82 | class_weights = self._prepare_class_weights() 83 | 84 | # Wrap loss function 85 | # TODO: pass kwargs to loss from config 86 | if loss in segmentation_losses: 87 | loss = segmentation_losses[loss](from_logits=False, class_weights=class_weights) 88 | wrapped_loss = cropped_loss(loss) 89 | 90 | # Wrap metrics 91 | wrapped_metrics = [] 92 | for metric in metrics: 93 | 94 | if metric in segmentation_metrics: 95 | if metric in ['precision', 'recall']: 96 | wrapped_metrics += [CroppedMetric(segmentation_metrics[metric](top_k=1, 97 | class_id=class_id, 98 | name=f'{metric}_{class_id}')) 99 | for class_id in range(self.config.n_classes)] 100 | continue 101 | else: 102 | metric = segmentation_metrics[metric]() 103 | 104 | # Initialize initializable metrics 105 | if isinstance(metric, InitializableMetric): 106 | metric.init_from_config(self.config) 107 | 108 | wrapped_metric = CroppedMetric(metric) 109 | wrapped_metrics.append(wrapped_metric) 110 | 111 | self.compile(optimizer=optimizer, loss=wrapped_loss, metrics=wrapped_metrics, **kwargs) 112 | 113 | def _get_visualization_callback(self, dataset, log_dir): 114 | ds = dataset.unbatch().batch(self.config.prediction_visualization_num).take(1) 115 | data = next(iter(ds)) 116 | 117 | visualization_callback = VisualizationCallback(data, log_dir) 118 | return visualization_callback 119 | 120 | # Override default method to add prediction visualization 121 | def train(self, dataset, 122 | num_epochs, 123 | model_directory, 124 | iterations_per_epoch=None, 125 | class_weights=None, 126 | callbacks=[], 127 | save_steps='epoch', 128 | summary_steps=1, **kwargs): 129 | 130 | custom_callbacks = [] 131 | 132 | if self.config.prediction_visualization: 133 | log_dir = os.path.join(model_directory, 'logs', 'predictions') 134 | visualization_callback = self._get_visualization_callback(dataset, log_dir) 135 | custom_callbacks.append(visualization_callback) 136 | 137 | super().train(dataset, num_epochs, model_directory, iterations_per_epoch, 138 | callbacks=callbacks + custom_callbacks, save_steps=save_steps, 139 | summary_steps=summary_steps, **kwargs) 140 | 141 | # Override default method to add prediction visualization 142 | def train_and_evaluate(self, 143 | train_dataset, 144 | val_dataset, 145 | num_epochs, 146 | iterations_per_epoch, 147 | model_directory, 148 | class_weights=None, 149 | save_steps=100, 150 | summary_steps=10, 151 | callbacks=[], **kwargs): 152 | 153 | custom_callbacks = [] 154 | 155 | if self.config.prediction_visualization: 156 | log_dir = os.path.join(model_directory, 'logs', 'predictions') 157 | visualization_callback = self._get_visualization_callback(val_dataset, log_dir) 158 | custom_callbacks.append(visualization_callback) 159 | 160 | super().train_and_evaluate(train_dataset, val_dataset, 161 | num_epochs, iterations_per_epoch, 162 | model_directory, 163 | save_steps=save_steps, summary_steps=summary_steps, 164 | callbacks=callbacks + custom_callbacks, **kwargs) 165 | -------------------------------------------------------------------------------- /eoflow/models/segmentation_unets.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import tensorflow as tf 3 | from marshmallow import fields 4 | 5 | from .layers import Conv2D, Deconv2D, CropAndConcat, Conv3D, MaxPool3D, Reduce3DTo2D, ResConv2D, PyramidPoolingModule 6 | from .segmentation_base import BaseSegmentationModel 7 | 8 | logging.basicConfig(level=logging.INFO, 9 | format='%(asctime)s %(levelname)s %(message)s') 10 | 11 | 12 | class FCNModel(BaseSegmentationModel): 13 | """ Implementation of a vanilla Fully-Convolutional-Network (aka U-net) """ 14 | 15 | class FCNModelSchema(BaseSegmentationModel._Schema): 16 | n_layers = fields.Int(required=True, description='Number of layers of the FCN model', example=10) 17 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 18 | features_root = fields.Int(required=True, description='Number of features at the root level.', example=32) 19 | 20 | conv_size = fields.Int(missing=3, description='Size of the convolution kernels.') 21 | deconv_size = fields.Int(missing=2, description='Size of the deconvolution kernels.') 22 | conv_stride = fields.Int(missing=1, description='Stride used in convolutions.') 23 | dilation_rate = fields.List(fields.Int, missing=1, description='Dilation rate used in convolutions.') 24 | add_dropout = fields.Bool(missing=False, description='Add dropout to layers.') 25 | add_batch_norm = fields.Bool(missing=True, description='Add batch normalization to layers.') 26 | bias_init = fields.Float(missing=0.0, description='Bias initialization value.') 27 | use_bias = fields.Bool(missing=True, description='Add bias parameters to convolutional layer.') 28 | padding = fields.String(missing='VALID', description='Padding type used in convolutions.') 29 | 30 | pool_size = fields.Int(missing=2, description='Kernel size used in max pooling.') 31 | pool_stride = fields.Int(missing=2, description='Stride used in max pooling.') 32 | 33 | class_weights = fields.List(fields.Float, missing=None, description='Class weights used in training.') 34 | 35 | def build(self, inputs_shape): 36 | """Builds the net for input x.""" 37 | 38 | x = tf.keras.layers.Input(inputs_shape[1:]) 39 | dropout_rate = 1 - self.config.keep_prob 40 | 41 | # Encoding path 42 | # the number of features of the convolutional kernels is proportional to the square of the level 43 | # for instance, starting with 32 features at the first level (layer=0), there will be 64 features at layer=1 and 44 | # 128 features at layer=2 45 | net = x 46 | connection_outputs = [] 47 | for layer in range(self.config.n_layers): 48 | # compute number of features as a function of network depth level 49 | features = 2 ** layer * self.config.features_root 50 | 51 | # bank of two convolutional filters 52 | conv = Conv2D( 53 | filters=features, 54 | kernel_size=self.config.conv_size, 55 | strides=self.config.conv_stride, 56 | dilation=self.config.dilation_rate, 57 | add_dropout=self.config.add_dropout, 58 | dropout_rate=dropout_rate, 59 | batch_normalization=self.config.add_batch_norm, 60 | padding=self.config.padding, 61 | use_bias=self.config.use_bias, 62 | num_repetitions=2)(net) 63 | 64 | connection_outputs.append(conv) 65 | 66 | # max pooling operation 67 | net = tf.keras.layers.MaxPool2D( 68 | pool_size=self.config.pool_size, 69 | strides=self.config.pool_stride, 70 | padding='SAME')(conv) 71 | 72 | # bank of 2 convolutional filters at bottom of U-net. 73 | bottom = Conv2D( 74 | filters=2 ** self.config.n_layers * self.config.features_root, 75 | kernel_size=self.config.conv_size, 76 | strides=self.config.conv_stride, 77 | dilation=self.config.dilation_rate, 78 | add_dropout=self.config.add_dropout, 79 | dropout_rate=dropout_rate, 80 | batch_normalization=self.config.add_batch_norm, 81 | use_bias=self.config.use_bias, 82 | num_repetitions=2, 83 | padding=self.config.padding)(net) 84 | 85 | net = bottom 86 | # Decoding path 87 | # the decoding path mirrors the encoding path in terms of number of features per convolutional filter 88 | for layer in range(self.config.n_layers): 89 | # find corresponding level in decoding branch 90 | conterpart_layer = self.config.n_layers - 1 - layer 91 | # get same number of features as counterpart layer 92 | features = 2 ** conterpart_layer * self.config.features_root 93 | 94 | deconv = Deconv2D( 95 | filters=features, 96 | kernel_size=self.config.deconv_size, 97 | batch_normalization=self.config.add_batch_norm)(net) 98 | 99 | # # skip connections to concatenate features from encoding path 100 | cc = CropAndConcat()(connection_outputs[conterpart_layer], 101 | deconv) 102 | 103 | # bank of 2 convolutional filters 104 | net = Conv2D( 105 | filters=features, 106 | kernel_size=self.config.conv_size, 107 | strides=self.config.conv_stride, 108 | dilation=self.config.dilation_rate, 109 | add_dropout=self.config.add_dropout, 110 | dropout_rate=dropout_rate, 111 | batch_normalization=self.config.add_batch_norm, 112 | use_bias=self.config.use_bias, 113 | num_repetitions=2, 114 | padding=self.config.padding)(cc) 115 | 116 | # final 1x1 convolution corresponding to pixel-wise linear combination of feature channels 117 | logits = tf.keras.layers.Conv2D( 118 | filters=self.config.n_classes, 119 | kernel_size=1)(net) 120 | 121 | logits = tf.keras.layers.Softmax()(logits) 122 | 123 | self.net = tf.keras.Model(inputs=x, outputs=logits) 124 | 125 | def call(self, inputs, training=None): 126 | return self.net(inputs, training) 127 | 128 | 129 | class TFCNModel(BaseSegmentationModel): 130 | """ Implementation of a Temporal Fully-Convolutional-Network """ 131 | 132 | class TFCNModelSchema(BaseSegmentationModel._Schema): 133 | n_layers = fields.Int(required=True, description='Number of layers of the FCN model', example=10) 134 | keep_prob = fields.Float(required=True, description='Keep probability used in dropout layers.', example=0.5) 135 | features_root = fields.Int(required=True, description='Number of features at the root level.', example=32) 136 | 137 | conv_size = fields.Int(missing=3, description='Size of the convolution kernels.') 138 | deconv_size = fields.Int(missing=2, description='Size of the deconvolution kernels.') 139 | conv_size_reduce = fields.Int(missing=3, description='Size of the kernel for time reduction.') 140 | conv_stride = fields.Int(missing=1, description='Stride used in convolutions.') 141 | add_dropout = fields.Bool(missing=False, description='Add dropout to layers.') 142 | add_batch_norm = fields.Bool(missing=True, description='Add batch normalization to layers.') 143 | bias_init = fields.Float(missing=0.0, description='Bias initialization value.') 144 | use_bias = fields.Bool(missing=True, description='Add bias parameters to convolutional layer.') 145 | padding = fields.String(missing='VALID', description='Padding type used in convolutions.') 146 | single_encoding_conv = fields.Bool(missing=False, description="Whether to apply 1 or 2 banks of conv filters.") 147 | 148 | pool_size = fields.Int(missing=2, description='Kernel size used in max pooling.') 149 | pool_stride = fields.Int(missing=2, description='Stride used in max pooling.') 150 | pool_time = fields.Bool(missing=False, description='Operate pooling over time dimension.') 151 | 152 | class_weights = fields.List(fields.Float, missing=None, description='Class weights used in training.') 153 | 154 | def build(self, inputs_shape): 155 | 156 | x = tf.keras.layers.Input(inputs_shape[1:]) 157 | dropout_rate = 1 - self.config.keep_prob 158 | 159 | num_repetitions = 1 if self.config.single_encoding_conv else 2 160 | 161 | # encoding path 162 | net = x 163 | connection_outputs = [] 164 | for layer in range(self.config.n_layers): 165 | # compute number of features as a function of network depth level 166 | features = 2 ** layer * self.config.features_root 167 | # bank of one 3d convolutional filter; convolution is done along the temporal as well as spatial directions 168 | conv = Conv3D( 169 | features, 170 | kernel_size=self.config.conv_size, 171 | strides=self.config.conv_stride, 172 | add_dropout=self.config.add_dropout, 173 | dropout_rate=dropout_rate, 174 | batch_normalization=self.config.add_batch_norm, 175 | num_repetitions=num_repetitions, 176 | use_bias=self.config.use_bias, 177 | padding=self.config.padding)(net) 178 | 179 | connection_outputs.append(conv) 180 | # max pooling operation 181 | net = MaxPool3D( 182 | kernel_size=self.config.pool_size, 183 | strides=self.config.pool_stride, 184 | pool_time=self.config.pool_time)(conv) 185 | 186 | # Bank of 1 3d convolutional filter at bottom of FCN 187 | bottom = Conv3D( 188 | 2 ** self.config.n_layers * self.config.features_root, 189 | kernel_size=self.config.conv_size, 190 | strides=self.config.conv_stride, 191 | add_dropout=self.config.add_dropout, 192 | dropout_rate=dropout_rate, 193 | batch_normalization=self.config.add_batch_norm, 194 | num_repetitions=num_repetitions, 195 | padding=self.config.padding, 196 | use_bias=self.config.use_bias, 197 | convolve_time=(not self.config.pool_time))(net) 198 | 199 | # Reduce temporal dimension 200 | bottom = Reduce3DTo2D( 201 | 2 ** self.config.n_layers * self.config.features_root, 202 | kernel_size=self.config.conv_size_reduce, 203 | stride=self.config.conv_stride, 204 | add_dropout=self.config.add_dropout, 205 | dropout_rate=dropout_rate)(bottom) 206 | 207 | net = bottom 208 | # decoding path 209 | for layer in range(self.config.n_layers): 210 | # find corresponding level in decoding branch 211 | conterpart_layer = self.config.n_layers - 1 - layer 212 | # get same number of features as counterpart layer 213 | features = 2 ** conterpart_layer * self.config.features_root 214 | 215 | # transposed convolution to upsample tensors 216 | deconv = Deconv2D( 217 | filters=features, 218 | kernel_size=self.config.deconv_size, 219 | batch_normalization=self.config.add_batch_norm)(net) 220 | 221 | # skip connection with linear combination along time 222 | reduced = Reduce3DTo2D( 223 | features, 224 | kernel_size=self.config.conv_size_reduce, 225 | stride=self.config.conv_stride, 226 | add_dropout=self.config.add_dropout, 227 | dropout_rate=dropout_rate)(connection_outputs[conterpart_layer]) 228 | 229 | # crop and concatenate 230 | cc = CropAndConcat()(reduced, deconv) 231 | 232 | # bank of 2 convolutional layers as in standard FCN 233 | net = Conv2D( 234 | features, 235 | kernel_size=self.config.conv_size, 236 | strides=self.config.conv_stride, 237 | add_dropout=self.config.add_dropout, 238 | dropout_rate=dropout_rate, 239 | batch_normalization=self.config.add_batch_norm, 240 | padding=self.config.padding, 241 | use_bias=self.config.use_bias, 242 | num_repetitions=2)(cc) 243 | 244 | # final 1x1 convolution corresponding to pixel-wise linear combination of feature channels 245 | logits = tf.keras.layers.Conv2D( 246 | filters=self.config.n_classes, 247 | kernel_size=1)(net) 248 | 249 | logits = tf.keras.layers.Softmax()(logits) 250 | 251 | self.net = tf.keras.Model(inputs=x, outputs=logits) 252 | 253 | def call(self, inputs, training=None): 254 | return self.net(inputs, training) 255 | 256 | 257 | class ResUnetA(FCNModel): 258 | """ 259 | ResUnetA 260 | 261 | https://github.com/feevos/resuneta/tree/145be5519ee4bec9a8cce9e887808b8df011f520/models 262 | 263 | NOTE: The input to this network is a dictionary specifying input features and three output target images. This 264 | might require some modification to the functions used to automate training and evaluation. Get in touch through 265 | issues if this happens. 266 | 267 | TODO: build architecture from parameters as for FCn and TFCN 268 | 269 | """ 270 | 271 | def build(self, inputs_shape): 272 | """Builds the net for input x.""" 273 | x = tf.keras.layers.Input(shape=inputs_shape['features'][1:], name='features') 274 | dropout_rate = 1 - self.config.keep_prob 275 | 276 | # block 1 277 | initial_conv = Conv2D( 278 | filters=self.config.features_root, 279 | kernel_size=1, # 1x1 kernel 280 | strides=self.config.conv_stride, 281 | add_dropout=self.config.add_dropout, 282 | dropout_rate=dropout_rate, 283 | use_bias=self.config.use_bias, 284 | batch_normalization=True, 285 | padding=self.config.padding, 286 | num_repetitions=1)(x) 287 | 288 | # block 2 289 | resconv_1 = ResConv2D( 290 | filters=self.config.features_root, 291 | kernel_size=self.config.conv_size, 292 | dilation=[1, 3, 15, 31], 293 | strides=self.config.conv_stride, 294 | add_dropout=self.config.add_dropout, 295 | dropout_rate=dropout_rate, 296 | use_bias=self.config.use_bias, 297 | batch_normalization=self.config.add_batch_norm, 298 | padding=self.config.padding, 299 | num_parallel=4)(initial_conv) 300 | 301 | # block 3 302 | pool_1 = Conv2D( 303 | filters=2 * self.config.features_root, 304 | kernel_size=self.config.pool_size, 305 | strides=self.config.pool_stride, 306 | add_dropout=self.config.add_dropout, 307 | use_bias=self.config.use_bias, 308 | dropout_rate=dropout_rate, 309 | batch_normalization=self.config.add_batch_norm, 310 | padding='SAME', 311 | num_repetitions=1)(resconv_1) 312 | 313 | # block 4 314 | resconv_2 = ResConv2D( 315 | filters=2 * self.config.features_root, 316 | kernel_size=self.config.conv_size, 317 | dilation=[1, 3, 15, 31], 318 | strides=self.config.conv_stride, 319 | add_dropout=self.config.add_dropout, 320 | use_bias=self.config.use_bias, 321 | dropout_rate=dropout_rate, 322 | batch_normalization=self.config.add_batch_norm, 323 | padding=self.config.padding, 324 | num_parallel=4)(pool_1) 325 | 326 | # block 5 327 | pool_2 = Conv2D( 328 | filters=4 * self.config.features_root, 329 | kernel_size=self.config.pool_size, 330 | strides=self.config.pool_stride, 331 | add_dropout=self.config.add_dropout, 332 | use_bias=self.config.use_bias, 333 | dropout_rate=dropout_rate, 334 | batch_normalization=self.config.add_batch_norm, 335 | padding='SAME', 336 | num_repetitions=1)(resconv_2) 337 | 338 | # block 6 339 | resconv_3 = ResConv2D( 340 | filters=4 * self.config.features_root, 341 | kernel_size=self.config.conv_size, 342 | dilation=[1, 3, 15], 343 | strides=self.config.conv_stride, 344 | add_dropout=self.config.add_dropout, 345 | use_bias=self.config.use_bias, 346 | dropout_rate=dropout_rate, 347 | batch_normalization=self.config.add_batch_norm, 348 | padding=self.config.padding, 349 | num_parallel=3)(pool_2) 350 | 351 | # block 7 352 | pool_3 = Conv2D( 353 | filters=8 * self.config.features_root, 354 | kernel_size=self.config.pool_size, 355 | strides=self.config.pool_stride, 356 | add_dropout=self.config.add_dropout, 357 | use_bias=self.config.use_bias, 358 | dropout_rate=dropout_rate, 359 | batch_normalization=self.config.add_batch_norm, 360 | padding='SAME', 361 | num_repetitions=1)(resconv_3) 362 | 363 | # block 8 364 | resconv_4 = ResConv2D( 365 | filters=8 * self.config.features_root, 366 | kernel_size=self.config.conv_size, 367 | dilation=[1, 3, 15], 368 | strides=self.config.conv_stride, 369 | add_dropout=self.config.add_dropout, 370 | use_bias=self.config.use_bias, 371 | dropout_rate=dropout_rate, 372 | batch_normalization=self.config.add_batch_norm, 373 | padding=self.config.padding, 374 | num_parallel=3)(pool_3) 375 | 376 | # block 9 377 | pool_4 = Conv2D( 378 | filters=16 * self.config.features_root, 379 | kernel_size=self.config.pool_size, 380 | strides=self.config.pool_stride, 381 | add_dropout=self.config.add_dropout, 382 | use_bias=self.config.use_bias, 383 | dropout_rate=dropout_rate, 384 | batch_normalization=self.config.add_batch_norm, 385 | padding='SAME', 386 | num_repetitions=1)(resconv_4) 387 | 388 | # block 10 389 | resconv_5 = ResConv2D( 390 | filters=16 * self.config.features_root, 391 | kernel_size=self.config.conv_size, 392 | dilation=[1], 393 | strides=self.config.conv_stride, 394 | add_dropout=self.config.add_dropout, 395 | use_bias=self.config.use_bias, 396 | dropout_rate=dropout_rate, 397 | batch_normalization=self.config.add_batch_norm, 398 | padding=self.config.padding, 399 | num_parallel=1)(pool_4) 400 | 401 | # block 11 402 | pool_5 = Conv2D( 403 | filters=32 * self.config.features_root, 404 | kernel_size=self.config.pool_size, 405 | strides=self.config.pool_stride, 406 | add_dropout=self.config.add_dropout, 407 | use_bias=self.config.use_bias, 408 | dropout_rate=dropout_rate, 409 | batch_normalization=self.config.add_batch_norm, 410 | padding='SAME', 411 | num_repetitions=1)(resconv_5) 412 | 413 | # block 12 414 | resconv_6 = ResConv2D( 415 | filters=32 * self.config.features_root, 416 | kernel_size=self.config.conv_size, 417 | dilation=[1], 418 | strides=self.config.conv_stride, 419 | add_dropout=self.config.add_dropout, 420 | use_bias=self.config.use_bias, 421 | dropout_rate=dropout_rate, 422 | batch_normalization=self.config.add_batch_norm, 423 | padding=self.config.padding, 424 | num_parallel=1)(pool_5) 425 | 426 | # block 13 427 | ppm1 = PyramidPoolingModule(filters=32 * self.config.features_root, 428 | batch_normalization=True)(resconv_6) 429 | 430 | # block 14 431 | deconv_1 = Deconv2D( 432 | filters=32 * self.config.features_root, 433 | kernel_size=self.config.deconv_size, 434 | batch_normalization=self.config.add_batch_norm)(ppm1) 435 | 436 | # block 15 437 | concat_1 = CropAndConcat()(resconv_5, deconv_1) 438 | concat_1 = Conv2D( 439 | filters=16 * self.config.features_root, 440 | kernel_size=1, # 1x1 kernel 441 | strides=self.config.conv_stride, 442 | add_dropout=self.config.add_dropout, 443 | use_bias=self.config.use_bias, 444 | dropout_rate=dropout_rate, 445 | batch_normalization=True, # maybe 446 | padding=self.config.padding, 447 | num_repetitions=1)(concat_1) 448 | 449 | # block 16 450 | resconv_7 = ResConv2D( 451 | filters=16 * self.config.features_root, 452 | kernel_size=self.config.conv_size, 453 | dilation=[1], 454 | strides=self.config.conv_stride, 455 | add_dropout=self.config.add_dropout, 456 | use_bias=self.config.use_bias, 457 | dropout_rate=dropout_rate, 458 | batch_normalization=self.config.add_batch_norm, 459 | padding=self.config.padding, 460 | num_parallel=1)(concat_1) 461 | 462 | # block 17 463 | deconv_2 = Deconv2D( 464 | filters=16 * self.config.features_root, 465 | kernel_size=self.config.deconv_size, 466 | batch_normalization=self.config.add_batch_norm)(resconv_7) 467 | 468 | # block 18 469 | concat_2 = CropAndConcat()(resconv_4, deconv_2) 470 | concat_2 = Conv2D( 471 | filters=8 * self.config.features_root, 472 | kernel_size=1, # 1x1 kernel 473 | strides=self.config.conv_stride, 474 | add_dropout=self.config.add_dropout, 475 | use_bias=self.config.use_bias, 476 | dropout_rate=dropout_rate, 477 | batch_normalization=True, # maybe 478 | padding=self.config.padding, 479 | num_repetitions=1)(concat_2) 480 | 481 | # block 19 482 | resconv_8 = ResConv2D( 483 | filters=8 * self.config.features_root, 484 | kernel_size=self.config.conv_size, 485 | dilation=[1, 3, 15], 486 | strides=self.config.conv_stride, 487 | add_dropout=self.config.add_dropout, 488 | use_bias=self.config.use_bias, 489 | dropout_rate=dropout_rate, 490 | batch_normalization=self.config.add_batch_norm, 491 | padding=self.config.padding, 492 | num_parallel=3)(concat_2) 493 | 494 | # block 20 495 | deconv_3 = Deconv2D( 496 | filters=8 * self.config.features_root, 497 | kernel_size=self.config.deconv_size, 498 | batch_normalization=self.config.add_batch_norm)(resconv_8) 499 | 500 | # block 21 501 | concat_3 = CropAndConcat()(resconv_3, deconv_3) 502 | concat_3 = Conv2D( 503 | filters=4 * self.config.features_root, 504 | kernel_size=1, # 1x1 kernel 505 | strides=self.config.conv_stride, 506 | add_dropout=self.config.add_dropout, 507 | use_bias=self.config.use_bias, 508 | dropout_rate=dropout_rate, 509 | batch_normalization=True, 510 | padding=self.config.padding, 511 | num_repetitions=1)(concat_3) 512 | 513 | # block 22 514 | resconv_9 = ResConv2D( 515 | filters=4 * self.config.features_root, 516 | kernel_size=self.config.conv_size, 517 | dilation=[1, 3, 15], 518 | strides=self.config.conv_stride, 519 | add_dropout=self.config.add_dropout, 520 | use_bias=self.config.use_bias, 521 | dropout_rate=dropout_rate, 522 | batch_normalization=self.config.add_batch_norm, 523 | padding=self.config.padding, 524 | num_parallel=3)(concat_3) 525 | 526 | # block 23 527 | deconv_4 = Deconv2D( 528 | filters=4 * self.config.features_root, 529 | kernel_size=self.config.deconv_size, 530 | batch_normalization=self.config.add_batch_norm)(resconv_9) 531 | 532 | # block 24 533 | concat_4 = CropAndConcat()(resconv_2, deconv_4) 534 | concat_4 = Conv2D( 535 | filters=2 * self.config.features_root, 536 | kernel_size=1, # 1x1 kernel 537 | strides=self.config.conv_stride, 538 | add_dropout=self.config.add_dropout, 539 | use_bias=self.config.use_bias, 540 | dropout_rate=dropout_rate, 541 | batch_normalization=True, 542 | padding=self.config.padding, 543 | num_repetitions=1)(concat_4) 544 | 545 | # block 25 546 | resconv_10 = ResConv2D( 547 | filters=2 * self.config.features_root, 548 | kernel_size=self.config.conv_size, 549 | dilation=[1, 3, 15, 31], 550 | strides=self.config.conv_stride, 551 | add_dropout=self.config.add_dropout, 552 | use_bias=self.config.use_bias, 553 | dropout_rate=dropout_rate, 554 | batch_normalization=self.config.add_batch_norm, 555 | padding=self.config.padding, 556 | num_parallel=4)(concat_4) 557 | 558 | # block 26 559 | deconv_5 = Deconv2D( 560 | filters=2 * self.config.features_root, 561 | kernel_size=self.config.deconv_size, 562 | batch_normalization=self.config.add_batch_norm)(resconv_10) 563 | 564 | # block 27 565 | concat_5 = CropAndConcat()(resconv_1, deconv_5) 566 | concat_5 = Conv2D( 567 | filters=self.config.features_root, 568 | kernel_size=1, # 1x1 kernel 569 | strides=self.config.conv_stride, 570 | add_dropout=self.config.add_dropout, 571 | use_bias=self.config.use_bias, 572 | dropout_rate=dropout_rate, 573 | batch_normalization=True, 574 | padding=self.config.padding, 575 | num_repetitions=1)(concat_5) 576 | 577 | # block 28 578 | resconv_11 = ResConv2D( 579 | filters=self.config.features_root, 580 | kernel_size=self.config.conv_size, 581 | dilation=[1, 3, 15, 31], 582 | strides=self.config.conv_stride, 583 | add_dropout=self.config.add_dropout, 584 | use_bias=self.config.use_bias, 585 | dropout_rate=dropout_rate, 586 | batch_normalization=self.config.add_batch_norm, 587 | padding=self.config.padding, 588 | num_parallel=4)(concat_5) 589 | 590 | # block 29 591 | concat_6 = CropAndConcat()(initial_conv, resconv_11) 592 | concat_6 = Conv2D( 593 | filters=self.config.features_root, 594 | kernel_size=1, # 1x1 kernel 595 | strides=self.config.conv_stride, 596 | add_dropout=self.config.add_dropout, 597 | use_bias=self.config.use_bias, 598 | dropout_rate=dropout_rate, 599 | batch_normalization=True, 600 | padding=self.config.padding, 601 | num_repetitions=1)(concat_6) 602 | 603 | # block 30 604 | ppm2 = PyramidPoolingModule(filters=self.config.features_root, 605 | batch_normalization=True)(concat_6) 606 | 607 | # comditioned multi-tasking 608 | # first get distance 609 | distance_conv = Conv2D( 610 | filters=self.config.features_root, 611 | kernel_size=self.config.conv_size, 612 | strides=self.config.conv_stride, 613 | add_dropout=self.config.add_dropout, 614 | dropout_rate=dropout_rate, 615 | batch_normalization=self.config.add_batch_norm, 616 | num_repetitions=2, 617 | padding=self.config.padding)(concat_6) # in last layer we take the combined features 618 | logits_distance = tf.keras.layers.Conv2D(filters=self.config.n_classes, kernel_size=1)(distance_conv) 619 | logits_distance = tf.keras.layers.Softmax(name='distance')(logits_distance) 620 | 621 | # concatenate distance logits to features 622 | dcc = CropAndConcat()(ppm2, logits_distance) 623 | boundary_conv = Conv2D( 624 | filters=self.config.features_root, 625 | kernel_size=self.config.conv_size, 626 | strides=self.config.conv_stride, 627 | add_dropout=self.config.add_dropout, 628 | dropout_rate=dropout_rate, 629 | batch_normalization=self.config.add_batch_norm, 630 | num_repetitions=1, 631 | padding=self.config.padding)(dcc) 632 | logits_boundary = tf.keras.layers.Conv2D(filters=self.config.n_classes, kernel_size=1)(boundary_conv) 633 | logits_boundary = tf.keras.layers.Softmax(name='boundary')(logits_boundary) 634 | 635 | bdcc = CropAndConcat()(dcc, logits_boundary) 636 | extent_conv = Conv2D( 637 | filters=self.config.features_root, 638 | kernel_size=self.config.conv_size, 639 | strides=self.config.conv_stride, 640 | add_dropout=self.config.add_dropout, 641 | dropout_rate=dropout_rate, 642 | batch_normalization=self.config.add_batch_norm, 643 | num_repetitions=2, 644 | padding=self.config.padding)(bdcc) 645 | logits_extent = tf.keras.layers.Conv2D(filters=self.config.n_classes, kernel_size=1)(extent_conv) 646 | logits_extent = tf.keras.layers.Softmax(name='extent')(logits_extent) 647 | 648 | self.net = tf.keras.Model(inputs=x, outputs=[logits_extent, logits_boundary, logits_distance]) 649 | 650 | def call(self, inputs, training=True): 651 | return self.net(inputs, training) 652 | -------------------------------------------------------------------------------- /eoflow/models/transformer_encoder_layers.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | 4 | from tensorflow.keras.layers import Conv1D, LayerNormalization 5 | 6 | # This code is taken from the TF tutorial on transformers 7 | # https://www.tensorflow.org/tutorials/text/transformer 8 | def scaled_dot_product_attention(q, k, v, mask=None): 9 | """ Calculate the attention weights. 10 | q, k, v must have matching leading dimensions. 11 | k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v. 12 | The mask has different shapes depending on its type(padding or look ahead) 13 | but it must be broadcastable for addition. 14 | 15 | Args: 16 | q: query shape == (..., seq_len_q, depth) 17 | k: key shape == (..., seq_len_k, depth) 18 | v: value shape == (..., seq_len_v, depth_v) 19 | mask: Float tensor with shape broadcastable 20 | to (..., seq_len_q, seq_len_k). Defaults to None. 21 | 22 | Returns: 23 | output, attention_weights 24 | """ 25 | 26 | matmul_qk = tf.matmul(q, k, transpose_b=True) # (..., seq_len_q, seq_len_k) 27 | 28 | # scale matmul_qk 29 | dk = tf.cast(tf.shape(k)[-1], tf.float32) 30 | scaled_attention_logits = matmul_qk / tf.math.sqrt(dk) 31 | 32 | # add the mask to the scaled tensor. 33 | if mask is not None: 34 | scaled_attention_logits += (mask * -1e9) 35 | 36 | # softmax is normalized on the last axis (seq_len_k) so that the scores 37 | # add up to 1. 38 | attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1) # (..., seq_len_q, seq_len_k) 39 | 40 | output = tf.matmul(attention_weights, v) # (..., seq_len_q, depth_v) 41 | 42 | return output, attention_weights 43 | 44 | 45 | class MultiHeadAttention(tf.keras.layers.Layer): 46 | def __init__(self, d_model, num_heads): 47 | super(MultiHeadAttention, self).__init__() 48 | self.num_heads = num_heads 49 | self.d_model = d_model 50 | 51 | assert d_model % self.num_heads == 0 52 | 53 | self.depth = d_model // self.num_heads 54 | 55 | self.wq = tf.keras.layers.Dense(d_model) 56 | self.wk = tf.keras.layers.Dense(d_model) 57 | self.wv = tf.keras.layers.Dense(d_model) 58 | 59 | self.dense = tf.keras.layers.Dense(d_model) 60 | 61 | def split_heads(self, x, batch_size): 62 | """Split the last dimension into (num_heads, depth). 63 | Transpose the result such that the shape is (batch_size, num_heads, seq_len, depth) 64 | """ 65 | x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth)) 66 | return tf.transpose(x, perm=[0, 2, 1, 3]) 67 | 68 | def call(self, v, k, q, mask=None): 69 | batch_size = tf.shape(q)[0] 70 | 71 | q = self.wq(q) # (batch_size, seq_len, d_model) 72 | k = self.wk(k) # (batch_size, seq_len, d_model) 73 | v = self.wv(v) # (batch_size, seq_len, d_model) 74 | 75 | q = self.split_heads(q, batch_size) # (batch_size, num_heads, seq_len_q, depth) 76 | k = self.split_heads(k, batch_size) # (batch_size, num_heads, seq_len_k, depth) 77 | v = self.split_heads(v, batch_size) # (batch_size, num_heads, seq_len_v, depth) 78 | 79 | # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth) 80 | # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k) 81 | scaled_attention, attention_weights = scaled_dot_product_attention(q, k, v, mask) 82 | 83 | # (batch_size, seq_len_q, num_heads, depth) 84 | scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3]) 85 | 86 | # (batch_size, seq_len_q, d_model) 87 | concat_attention = tf.reshape(scaled_attention, (batch_size, -1, self.d_model)) 88 | 89 | # (batch_size, seq_len_q, d_model) 90 | output = self.dense(concat_attention) 91 | 92 | return output, attention_weights 93 | 94 | 95 | def point_wise_feed_forward_network(d_model, dff): 96 | return tf.keras.Sequential([ 97 | tf.keras.layers.Dense(dff, activation='relu'), # (batch_size, seq_len, dff) 98 | tf.keras.layers.Dense(d_model) # (batch_size, seq_len, d_model) 99 | ]) 100 | 101 | 102 | def positional_encoding(positions, d_model, T=10000): 103 | 104 | if isinstance(positions, int): 105 | positions = np.arange(positions) 106 | else: 107 | positions = np.array(positions) 108 | 109 | def _get_angles(pos, i, d_model): 110 | angle_rates = 1 / np.power(T, (2 * (i//2)) / np.float32(d_model)) 111 | return pos * angle_rates 112 | 113 | depths = np.arange(d_model) 114 | 115 | angle_rads = _get_angles(positions[:, np.newaxis], 116 | depths[np.newaxis, :], 117 | d_model) 118 | 119 | # apply sin to even indices in the array; 2i 120 | angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2]) 121 | 122 | # apply cos to odd indices in the array; 2i+1 123 | angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2]) 124 | 125 | pos_encoding = angle_rads[np.newaxis, ...] 126 | 127 | return tf.cast(pos_encoding, dtype=tf.float32) 128 | 129 | 130 | class EncoderLayer(tf.keras.layers.Layer): 131 | def __init__(self, d_model, num_heads, dff, rate=0.1): 132 | super(EncoderLayer, self).__init__() 133 | 134 | self.mha = MultiHeadAttention(d_model, num_heads) 135 | self.ffn = point_wise_feed_forward_network(d_model, dff) 136 | 137 | self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6) 138 | self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6) 139 | 140 | self.dropout1 = tf.keras.layers.Dropout(rate) 141 | self.dropout2 = tf.keras.layers.Dropout(rate) 142 | 143 | def call(self, x, training=None, mask=None): 144 | attn_output, _ = self.mha(x, x, x, mask) # (batch_size, input_seq_len, d_model) 145 | attn_output = self.dropout1(attn_output, training=training) 146 | out1 = self.layernorm1(x + attn_output) # (batch_size, input_seq_len, d_model) 147 | 148 | ffn_output = self.ffn(out1) # (batch_size, input_seq_len, d_model) 149 | ffn_output = self.dropout2(ffn_output, training=training) 150 | out2 = self.layernorm2(out1 + ffn_output) # (batch_size, input_seq_len, d_model) 151 | 152 | return out2 153 | 154 | 155 | class Encoder(tf.keras.layers.Layer): 156 | def __init__(self, num_layers, d_model, num_heads, dff, maximum_position_encoding, rate=0.1, layer_norm=False, T=10000): 157 | super(Encoder, self).__init__() 158 | 159 | self.d_model = d_model 160 | self.num_layers = num_layers 161 | 162 | self.lnorm_in = tf.keras.layers.LayerNormalization() if layer_norm else None 163 | self.lnorm_conv = tf.keras.layers.LayerNormalization() if layer_norm else None 164 | 165 | # replace embedding with 1d convolution 166 | self.conv_in = Conv1D(d_model, 1) 167 | # self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model) 168 | self.pos_encoding = positional_encoding(maximum_position_encoding, self.d_model, T=T) 169 | 170 | encoder_layers = [EncoderLayer(d_model, num_heads, dff, rate) 171 | for _ in range(num_layers)] 172 | self.encoder = tf.keras.Sequential(encoder_layers) 173 | 174 | self.dropout = tf.keras.layers.Dropout(rate) 175 | 176 | 177 | def call(self, x, training=None, mask=None): 178 | seq_len = tf.shape(x)[1] 179 | 180 | if self.lnorm_in: 181 | x = self.lnorm_in(x) 182 | 183 | # adding embedding and position encoding. 184 | x = self.conv_in(x, training=training) # (batch_size, input_seq_len, d_model) 185 | if self.lnorm_conv: 186 | x = self.lnorm_conv(x) 187 | 188 | x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32)) 189 | x += self.pos_encoding[:, :seq_len, :] 190 | 191 | x = self.dropout(x, training=training) 192 | 193 | x = self.encoder(x, training=training, mask=mask) 194 | 195 | return x # (batch_size, input_seq_len, d_model) 196 | -------------------------------------------------------------------------------- /eoflow/tasks/__init__.py: -------------------------------------------------------------------------------- 1 | from .train import TrainTask, TrainAndEvaluateTask 2 | from .predict import PredictTask 3 | from .evaluate import EvaluateTask 4 | -------------------------------------------------------------------------------- /eoflow/tasks/evaluate.py: -------------------------------------------------------------------------------- 1 | from marshmallow import Schema, fields 2 | 3 | from ..base import BaseTask 4 | from ..base.configuration import ObjectConfiguration 5 | 6 | 7 | class EvaluateTask(BaseTask): 8 | class EvaluateTaskConfig(Schema): 9 | model_directory = fields.String(required=True, description='Directory of the model', example='/tmp/model/') 10 | 11 | input_config = fields.Nested(nested=ObjectConfiguration, required=True, 12 | description="Input type and configuration.") 13 | 14 | def run(self): 15 | dataset = self.parse_input(self.config.input_config) 16 | 17 | self.model.prepare() 18 | self.model.load_latest(self.config.model_directory) 19 | 20 | values = self.model.evaluate(dataset) 21 | names = self.model.metrics_names 22 | 23 | metrics = {name:value for name,value in zip(names, values)} 24 | 25 | # Display metrics 26 | print("Evaluation results:") 27 | for metric_name in metrics: 28 | print("{}: {}".format(metric_name, metrics[metric_name])) 29 | -------------------------------------------------------------------------------- /eoflow/tasks/predict.py: -------------------------------------------------------------------------------- 1 | from marshmallow import Schema, fields 2 | 3 | from ..base import BaseTask 4 | from ..base.configuration import ObjectConfiguration 5 | 6 | 7 | class PredictTask(BaseTask): 8 | class PredictTaskConfig(Schema): 9 | model_directory = fields.String(required=True, description='Directory of the model', example='/tmp/model/') 10 | 11 | input_config = fields.Nested(nested=ObjectConfiguration, required=True, 12 | description="Input type and configuration.") 13 | 14 | def run(self): 15 | dataset_fn = self.parse_input(self.config.input_config) 16 | 17 | self.model.prepare() # TODO: find a way to initialize without compiling the model 18 | self.model.load_latest(self.config.model_directory) 19 | 20 | predictions_list = self.model.predict(dataset_fn) 21 | # TODO: something with predictions 22 | -------------------------------------------------------------------------------- /eoflow/tasks/train.py: -------------------------------------------------------------------------------- 1 | from marshmallow import Schema, fields 2 | 3 | from ..base import BaseTask 4 | from ..base.configuration import ObjectConfiguration 5 | 6 | 7 | class TrainTask(BaseTask): 8 | class TrainTaskConfig(Schema): 9 | num_epochs = fields.Int(required=True, description='Number of epochs used in training', example=50) 10 | iterations_per_epoch = fields.Int(required=True, description='Number of training steps per epoch', example=100) 11 | model_directory = fields.String(required=True, description='Directory of the model output', example='/tmp/model/') 12 | 13 | input_config = fields.Nested(nested=ObjectConfiguration, required=True, description="Input type and configuration.") 14 | 15 | save_steps = fields.Int(missing=100, description="Number of training steps between model checkpoints.") 16 | summary_steps = fields.Int(missing='epoch', description="Number of training steps between recording summaries.") 17 | 18 | def run(self): 19 | dataset = self.parse_input(self.config.input_config) 20 | 21 | self.model.prepare() 22 | 23 | self.model.train( 24 | dataset, 25 | num_epochs=self.config.num_epochs, 26 | iterations_per_epoch=self.config.iterations_per_epoch, 27 | model_directory=self.config.model_directory, 28 | save_steps=self.config.save_steps, 29 | summary_steps=self.config.summary_steps 30 | ) 31 | 32 | 33 | class TrainAndEvaluateTask(BaseTask): 34 | class TrainAndEvaluateTask(Schema): 35 | num_epochs = fields.Int(required=True, description='Number of epochs used in training', example=50) 36 | iterations_per_epoch = fields.Int(required=True, description='Number of training steps per epoch', example=100) 37 | model_directory = fields.String(required=True, description='Directory of the model output', 38 | example='/tmp/model/') 39 | 40 | train_input_config = fields.Nested(nested=ObjectConfiguration, required=True, 41 | description="Input type and configuration for training.") 42 | val_input_config = fields.Nested(nested=ObjectConfiguration, required=True, 43 | description="Input type and configuration for validation.") 44 | 45 | save_steps = fields.Int(missing=100, description="Number of training steps between model checkpoints.") 46 | summary_steps = fields.Int(missing='epoch', description="Number of training steps between recording summaries.") 47 | 48 | def run(self): 49 | train_dataset = self.parse_input(self.config.train_input_config) 50 | val_dataset = self.parse_input(self.config.val_input_config) 51 | 52 | self.model.prepare() 53 | 54 | self.model.train_and_evaluate( 55 | train_dataset, val_dataset, 56 | num_epochs=self.config.num_epochs, 57 | iterations_per_epoch=self.config.iterations_per_epoch, 58 | model_directory=self.config.model_directory, 59 | save_steps=self.config.save_steps, 60 | summary_steps=self.config.summary_steps 61 | ) 62 | -------------------------------------------------------------------------------- /eoflow/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from .utils import create_dirs, parse_classname, get_common_shape 2 | 3 | -------------------------------------------------------------------------------- /eoflow/utils/tf_utils.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import io 3 | import matplotlib.pyplot as plt 4 | 5 | 6 | def plot_to_image(figure): 7 | """ Converts the matplotlib plot specified by 'figure' to a PNG image and 8 | returns it. The supplied figure is closed and inaccessible after this call. """ 9 | 10 | # Save the plot to a PNG in memory. 11 | buf = io.BytesIO() 12 | plt.savefig(buf, format='png') 13 | # Closing the figure prevents it from being displayed directly inside 14 | # the notebook. 15 | plt.close(figure) 16 | buf.seek(0) 17 | # Convert PNG buffer to TF image 18 | image = tf.image.decode_png(buf.getvalue(), channels=4) 19 | # Add the batch dimension 20 | image = tf.expand_dims(image, 0) 21 | return image 22 | -------------------------------------------------------------------------------- /eoflow/utils/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | from pydoc import locate 3 | 4 | 5 | def parse_classname(classname): 6 | return locate(classname) 7 | 8 | 9 | def create_dirs(dirs): 10 | """ 11 | dirs - a list of directories to create if these directories are not found 12 | :param dirs: 13 | :return exit_code: 0:success -1:failed 14 | """ 15 | try: 16 | for dir_ in dirs: 17 | if not os.path.exists(dir_): 18 | os.makedirs(dir_) 19 | return 0 20 | except Exception as err: 21 | print("Creating directories error: {0}".format(err)) 22 | exit(-1) 23 | 24 | 25 | def get_common_shape(shape1, shape2): 26 | """ Get a common shape that fits both shapes. Dimensions that differ in size are set to None. 27 | Example: [None, 20, 100, 50], [None, 20, 200, 50] -> [None, 20, None, 50] 28 | """ 29 | if len(shape1) != len(shape2): 30 | raise ValueError("Can't compute common shape. Ndims is different.") 31 | 32 | common_shape = [dim1 if dim1==dim2 else None for dim1, dim2 in zip(shape1, shape2)] 33 | 34 | return common_shape 35 | -------------------------------------------------------------------------------- /examples/exporting_data.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Exporting data with numpy and h5py\n", 8 | "\n", 9 | "This notebook shows different ways to export the data for eoflow using numpy or h5py." 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import os\n", 19 | "import numpy as np\n", 20 | "import h5py" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "# Create temp dir\n", 30 | "os.makedirs('temp')" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "### Method 1: saving arrays using numpy" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "Let's create some numpy arrays to represent our features and labels." 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "features = np.random.random(size=(1024, 32, 32, 13))\n", 54 | "labels = np.random.randint(10, size=(1024,))\n", 55 | "\n", 56 | "features.shape, labels.shape" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "For numpy use the `np.savez` function to save multiple arrays into a single `.npz` file." 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "np.savez('temp/data.npz', features=features, labels=labels)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "Numpy reads and writes the whole file at the same time. Therefore the file size should be small to reduce the overhead.\n", 80 | "\n", 81 | "If the dataset size is large (can't fit into memory) it is better to split the dataset into multiple .npz files, or use the hdf5 format." 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "### Method 2: saving arrays using h5py" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": {}, 94 | "source": [ 95 | "Let's save the same data using the h5py library." 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "with h5py.File('temp/data.hdf5', 'w') as file:\n", 105 | " file.create_dataset('features', data=features)\n", 106 | " file.create_dataset('labels', data=labels)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": {}, 112 | "source": [ 113 | "The h5py allows us to create seperate datasets (and groups of datasets) and save the data to it. The format also allows for sequential reading. This means that only part of the data that is needed can be loaded. Therefore the spliting of the dataset into smaller pieces is not needed anymore.\n", 114 | "\n", 115 | "However, if the dataset we want to export is too big to fit into memory we cannot use this method to export the data. That's where the **Method 3** comes in." 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "### Method 3: saving arrays iteratively using h5py" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "The h5py allows us to write the data in parts (e.g. row by row). The datasets we create can be indexed and written to similarly to numpy arrays. Let's export a dataset produced from a generator." 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "def _generate_data(num_examples):\n", 139 | " \"\"\" Generates specified number of examples (example by example).\"\"\"\n", 140 | " \n", 141 | " for i in range(num_examples):\n", 142 | " features = np.random.random(size=(32, 32, 13))\n", 143 | " labels = np.random.randint(10, size=())\n", 144 | " \n", 145 | " yield features, labels" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "with h5py.File('temp/data_gen.hdf5', 'w') as file:\n", 155 | " num_examples = 1024\n", 156 | " \n", 157 | " # Define datasets (total shape)\n", 158 | " features_ds = file.create_dataset('features', (num_examples, 32, 32, 13), dtype=np.float32)\n", 159 | " labels_ds = file.create_dataset('labels', (num_examples,), dtype=np.int32)\n", 160 | " \n", 161 | " # Store the generated data into the datasets\n", 162 | " for i, (features, labels) in enumerate(_generate_data(num_examples)):\n", 163 | " features_ds[i] = features\n", 164 | " labels_ds[i] = labels" 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": {}, 170 | "source": [ 171 | "**NOTE**: the `data_gen.hdf5` size is smaller, because we specified the dtype of the features to be float32, while the original dtype of the array is float64." 172 | ] 173 | } 174 | ], 175 | "metadata": { 176 | "kernelspec": { 177 | "display_name": "tf2", 178 | "language": "python", 179 | "name": "tf2" 180 | }, 181 | "language_info": { 182 | "codemirror_mode": { 183 | "name": "ipython", 184 | "version": 3 185 | }, 186 | "file_extension": ".py", 187 | "mimetype": "text/x-python", 188 | "name": "python", 189 | "nbconvert_exporter": "python", 190 | "pygments_lexer": "ipython3", 191 | "version": "3.6.9" 192 | } 193 | }, 194 | "nbformat": 4, 195 | "nbformat_minor": 2 196 | } 197 | -------------------------------------------------------------------------------- /examples/input.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | from marshmallow import fields, Schema 6 | 7 | from eolearn.core import FeatureType 8 | from eoflow.base import BaseInput 9 | from eoflow.input.eopatch import eopatch_dataset, EOPatchSegmentationInput 10 | from eoflow.input.operations import extract_subpatches, augment_data, cache_dataset 11 | 12 | _valid_types = [t.value for t in FeatureType] 13 | 14 | 15 | class ExampleInput(BaseInput): 16 | """ A simple example of an Input class. Produces random data. """ 17 | 18 | class _Schema(Schema): 19 | input_shape = fields.List(fields.Int, description="Shape of a single input example.", required=True, example=[784]) 20 | num_classes = fields.Int(description="Number of classes.", required=True, example=10) 21 | 22 | batch_size = fields.Int(description="Number of examples in a batch.", required=True, example=20) 23 | batches_per_epoch = fields.Int(required=True, description='Number of batches in epoch', example=20) 24 | 25 | def _generate_batch(self): 26 | """ Generator that returns random features and labels. """ 27 | 28 | for i in range(self.config.batches_per_epoch): 29 | input_shape = [self.config.batch_size] + self.config.input_shape 30 | input_data = np.random.rand(*input_shape) 31 | 32 | onehot = np.eye(self.config.num_classes) 33 | output_shape = [self.config.batch_size] 34 | classes = np.random.randint(self.config.num_classes, size=output_shape) 35 | labels = onehot[classes] 36 | 37 | yield input_data, labels 38 | 39 | def get_dataset(self): 40 | input_shape = [self.config.batch_size] + self.config.input_shape 41 | output_shape = [self.config.batch_size, self.config.num_classes] 42 | 43 | # Create a tf dataset from a np.array generator 44 | dataset = tf.data.Dataset.from_generator( 45 | self._generate_batch, 46 | (tf.float32, tf.float32), 47 | (tf.TensorShape(input_shape), tf.TensorShape(output_shape)) 48 | ) 49 | 50 | return dataset 51 | 52 | 53 | class EOPatchInputExample(BaseInput): 54 | """ An example input method for EOPatches. Shows feature reading, subpatch extraction, data augmentation, 55 | caching, batching, etc. """ 56 | 57 | # Configuration schema (extended from EOPatchSegmentationInput) 58 | class _Schema(EOPatchSegmentationInput._Schema): 59 | # New fields 60 | patch_size = fields.List(fields.Int, description="Width and height of extracted patches.", required=True, example=[1,2]) 61 | num_subpatches = fields.Int(required=True, description="Number of subpatches extracted by random sampling.", example=5) 62 | 63 | interleave_size = fields.Int(description="Number of eopatches to interleave the subpatches from.", required=True, example=5) 64 | data_augmentation = fields.Bool(missing=False, description="Use data augmentation on images.") 65 | 66 | cache_file = fields.String( 67 | missing=None, description="A path to the file where the dataset will be cached. No caching if not provided.", example='/tmp/data') 68 | 69 | @staticmethod 70 | def _parse_shape(shape): 71 | shape = [None if s < 0 else s for s in shape] 72 | return shape 73 | 74 | def get_dataset(self): 75 | cfg = self.config 76 | print(json.dumps(cfg, indent=4)) 77 | 78 | # Create a tf.data.Dataset from EOPatches 79 | features_data = [ 80 | (cfg.input_feature_type, cfg.input_feature_name, 'features', np.float32, self._parse_shape(cfg.input_feature_shape)), 81 | (cfg.labels_feature_type, cfg.labels_feature_name, 'labels', np.int64, self._parse_shape(cfg.labels_feature_shape)) 82 | ] 83 | dataset = eopatch_dataset(self.config.data_dir, features_data, fill_na=-2) 84 | 85 | # Extract random subpatches 86 | extract_fn = extract_subpatches( 87 | self.config.patch_size, 88 | [('features', self.config.input_feature_axis), 89 | ('labels', self.config.labels_feature_axis)], 90 | random_sampling=True, 91 | num_random_samples=self.config.num_subpatches 92 | ) 93 | # Interleave patches extracted from multiple EOPatches 94 | dataset = dataset.interleave(extract_fn, self.config.interleave_size) 95 | 96 | # Cache the dataset so the patch extraction is done only once 97 | if self.config.cache_file is not None: 98 | dataset = cache_dataset(dataset, self.config.cache_file) 99 | 100 | # Data augmentation 101 | if cfg.data_augmentation: 102 | feature_augmentation = [ 103 | ('features', ['flip_left_right', 'rotate', 'brightness']), 104 | ('labels', ['flip_left_right', 'rotate']) 105 | ] 106 | dataset = dataset.map(augment_data(feature_augmentation)) 107 | 108 | # One-hot encode labels and return tuple 109 | def _prepare_data(data): 110 | features = data['features'] 111 | labels = data['labels'][..., 0] 112 | 113 | labels_oh = tf.one_hot(labels, depth=self.config.num_classes) 114 | 115 | return features, labels_oh 116 | 117 | dataset = dataset.map(_prepare_data) 118 | 119 | # Create batches 120 | dataset = dataset.batch(self.config.batch_size) 121 | 122 | return dataset 123 | -------------------------------------------------------------------------------- /examples/models.py: -------------------------------------------------------------------------------- 1 | from eoflow.base import BaseModel 2 | import tensorflow as tf 3 | 4 | from marshmallow import Schema, fields 5 | 6 | 7 | class ExampleModel(BaseModel): 8 | """ Example implementation of a model. Builds a fully connected net with a single hidden layer. """ 9 | 10 | class _Schema(Schema): 11 | output_size = fields.Int(required=True, description='Output size of the model', example=10) 12 | hidden_units = fields.Int(missing=512, description='Number of hidden units', example=512) 13 | learning_rate = fields.Float(missing=0.01, description='Learning rate for Adam optimizer', example=0.01) 14 | 15 | def init_model(self): 16 | l1 = tf.keras.layers.Dense(self.config.hidden_units, activation='relu') 17 | l2 = tf.keras.layers.Dense(self.config.output_size, activation='softmax') 18 | 19 | self.net = tf.keras.Sequential([l1, l2]) 20 | 21 | def call(self, inputs, training=False): 22 | x = self.net(inputs) 23 | 24 | return x 25 | 26 | def prepare(self, optimizer=None, loss=None, metrics=None, **kwargs): 27 | """ Prepares the model. Optimizer, loss and metrics are read using the following protocol: 28 | * If an argument is None, the default value is used from the configuration of the model. 29 | * If an argument is a key contained in segmentation specific losses/metrics, those are used. 30 | * Otherwise the argument is passed to `compile` as is. 31 | 32 | """ 33 | optimizer = tf.keras.optimizers.Adam(learning_rate=self.config.learning_rate) 34 | 35 | loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True) 36 | 37 | metrics = tf.keras.metrics.CategoricalAccuracy(name='accuracy') 38 | 39 | self.compile(optimizer=optimizer, loss=loss, metrics=[metrics], **kwargs) 40 | -------------------------------------------------------------------------------- /figures/fcn-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sentinel-hub/eo-flow/70a426fe3ab07d8ec096e06af4db7e445af1e740/figures/fcn-architecture.png -------------------------------------------------------------------------------- /figures/rfcn-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sentinel-hub/eo-flow/70a426fe3ab07d8ec096e06af4db7e445af1e740/figures/rfcn-architecture.png -------------------------------------------------------------------------------- /figures/tfcn-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sentinel-hub/eo-flow/70a426fe3ab07d8ec096e06af4db7e445af1e740/figures/tfcn-architecture.png -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | pytest>=4.0.0 2 | pytest-cov 3 | codecov 4 | pylint -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | eo-learn-core 2 | munch 3 | tensorflow>=2.1.0 4 | tensorflow-addons 5 | numpy 6 | marshmallow 7 | matplotlib 8 | h5py 9 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | from setuptools import setup, find_packages 3 | 4 | 5 | def parse_requirements(file): 6 | return sorted(set( 7 | line.partition('#')[0].strip() 8 | for line in open(os.path.join(os.path.dirname(__file__), file)) 9 | ) - set('')) 10 | 11 | 12 | setup( 13 | name='eo-flow', 14 | python_requires='>=3.5', 15 | version='1.2.0', 16 | description='Tensorflow wrapper built for prototyping and deploying earth observation deep models.', 17 | author='Sinergise EO research team', 18 | author_email='eoresearch@sinergise.com', 19 | packages=find_packages(), 20 | install_requires=parse_requirements('requirements.txt'), 21 | extras_require={ 22 | 'DEV': parse_requirements('requirements-dev.txt') 23 | } 24 | ) 25 | -------------------------------------------------------------------------------- /tests/test_layers.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | from eoflow.models.layers import ResConv2D, PyramidPoolingModule 6 | 7 | 8 | class TestLayers(unittest.TestCase): 9 | def test_res_conv_layer(self): 10 | input_shape = (4, 28, 28, 3) 11 | 12 | x = tf.ones(input_shape) 13 | 14 | for npar in range(1, 4): 15 | y = ResConv2D(3, kernel_size=[npar]*npar, num_parallel=npar, padding='SAME')(x) 16 | self.assertEqual(y.shape, input_shape) 17 | 18 | y = ResConv2D(3, kernel_size=[npar]*npar, dilation=[npar]*npar, num_parallel=npar, padding='SAME')(x) 19 | self.assertEqual(y.shape, input_shape) 20 | 21 | y = ResConv2D(3, dilation=[npar]*npar, num_parallel=npar, padding='SAME')(x) 22 | self.assertEqual(y.shape, input_shape) 23 | 24 | with self.assertRaises(ValueError): 25 | ResConv2D(filters=3, kernel_size=[3, 3], padding='SAME', num_parallel=3) 26 | ResConv2D(filters=3, dilation=[3, 3], padding='SAME', num_parallel=3) 27 | 28 | def test_ppm_layer(self): 29 | batch_size, height, width, nchannels = 1, 64, 64, 1 30 | input_shape = (batch_size, height, width, nchannels) 31 | filters = 4 32 | bins = (1, 2, 4, 8) 33 | 34 | x = np.arange(np.prod(input_shape)).reshape(input_shape).astype(np.float32) 35 | 36 | ppm = PyramidPoolingModule(filters=filters, bins=bins, interpolation='nearest') 37 | 38 | y = ppm(x) 39 | 40 | self.assertEqual(y.shape, (batch_size, height, width, filters+nchannels)) 41 | np.testing.assert_array_equal(y[..., 0], x[..., 0]) 42 | for nbin, bin_size in enumerate(bins): 43 | self.assertLessEqual(np.unique(y[..., nbin+1]).size, bin_size**2) 44 | 45 | 46 | if __name__ == '__main__': 47 | unittest.main() 48 | -------------------------------------------------------------------------------- /tests/test_losses.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import numpy as np 3 | 4 | from eoflow.models.losses import CategoricalCrossEntropy, CategoricalFocalLoss 5 | from eoflow.models.losses import JaccardDistanceLoss, TanimotoDistanceLoss 6 | 7 | 8 | class TestLosses(unittest.TestCase): 9 | def test_shapes(self): 10 | for loss_fn in [CategoricalFocalLoss(from_logits=True), CategoricalCrossEntropy(from_logits=True)]: 11 | 12 | ones_1 = np.ones((1, 1024, 2)) 13 | ones_2 = np.ones((1, 32, 32, 2)) 14 | 15 | val1 = loss_fn(ones_1, 1-ones_1).numpy() 16 | val2 = loss_fn(ones_2, 1-ones_2).numpy() 17 | 18 | # Values should be scalars 19 | self.assertEqual(val1.shape, ()) 20 | self.assertEqual(val2.shape, ()) 21 | 22 | # Loss values should be equal as they represent the same data, just in different shapes 23 | self.assertAlmostEqual(val1, val2, 10) 24 | 25 | def test_focal_loss_values(self): 26 | ones = np.ones((32, 32)) 27 | zeros = np.zeros((32, 32)) 28 | mixed = np.concatenate([ones[:16], zeros[:16]]) 29 | 30 | # Predict everything as class 1 31 | y_pred = np.stack([zeros, ones], axis=-1) 32 | 33 | y_true1 = np.stack([ones, zeros], axis=-1) # All class 0 34 | y_true2 = np.stack([zeros, ones], axis=-1) # All class 1 35 | y_true3 = np.stack([mixed, 1-mixed], axis=-1) # Half class 1, half class 0 36 | 37 | for loss_fn in [CategoricalFocalLoss(from_logits=False), 38 | CategoricalFocalLoss(from_logits=False, class_weights=np.array([0, 1]))]: 39 | 40 | # Compute loss values for different labels 41 | val1 = loss_fn(y_true1, y_pred).numpy() # Should be biggest (all are wrong) 42 | val2 = loss_fn(y_true2, y_pred).numpy() # Should be 0 (all are correct) 43 | val3 = loss_fn(y_true3, y_pred).numpy() # Should be in between (half are correct) 44 | 45 | self.assertAlmostEqual(val2, 0.0, 10) 46 | 47 | self.assertGreaterEqual(val3, val2) 48 | self.assertGreaterEqual(val1, val3) 49 | 50 | def test_jaccard_loss(self): 51 | loss_fn = JaccardDistanceLoss(from_logits=False, smooth=1) 52 | 53 | y_true = np.zeros([1, 32, 32, 3]) 54 | y_true[:, :16, :16, 0] = np.ones((1, 16, 16)) 55 | y_true[:, 16:, :16, 1] = np.ones((1, 16, 16)) 56 | y_true[:, :, 16:, 2] = np.ones((1, 32, 16)) 57 | 58 | y_pred = np.zeros([1, 32, 32, 3]) 59 | y_pred[..., 0] = 1 60 | 61 | val_1 = loss_fn(y_true, y_true).numpy() 62 | val_2 = loss_fn(y_true, y_pred).numpy() 63 | y_pred[..., 0] = 0 64 | y_pred[..., 1] = 1 65 | val_3 = loss_fn(y_true, y_pred).numpy() 66 | y_pred[..., 1] = 0 67 | y_pred[..., 2] = 1 68 | val_4 = loss_fn(y_true, y_pred).numpy() 69 | 70 | self.assertEqual(val_1, 0.0) 71 | self.assertAlmostEqual(val_2, 2.743428, 5) 72 | self.assertAlmostEqual(val_3, 2.743428, 5) 73 | self.assertAlmostEqual(val_4, 2.491730, 5) 74 | 75 | loss_fn = JaccardDistanceLoss(from_logits=False, smooth=1, class_weights=np.array([0, 1, 1])) 76 | 77 | val_1 = loss_fn(y_true, y_true).numpy() 78 | val_2 = loss_fn(y_true, y_pred).numpy() 79 | y_pred[..., 0] = 0 80 | y_pred[..., 1] = 1 81 | val_3 = loss_fn(y_true, y_pred).numpy() 82 | y_pred[..., 1] = 0 83 | y_pred[..., 2] = 1 84 | val_4 = loss_fn(y_true, y_pred).numpy() 85 | 86 | self.assertEqual(val_1, 0.0) 87 | self.assertAlmostEqual(val_2, 1.495621, 5) 88 | self.assertAlmostEqual(val_3, 1.248781, 5) 89 | self.assertAlmostEqual(val_4, 1.495621, 5) 90 | 91 | def test_tanimoto_loss(self): 92 | 93 | y_true = np.zeros([1, 32, 32, 2], dtype=np.float32) 94 | y_true[:, 16:, :16, 1] = np.ones((1, 16, 16)) 95 | y_true[..., 0] = np.ones([1, 32, 32]) - y_true[..., 1] 96 | 97 | y_pred = np.zeros([1, 32, 32, 2], dtype=np.float32) 98 | y_pred[..., 0] = 1 99 | 100 | self.assertEqual(TanimotoDistanceLoss(from_logits=False)(y_true, y_true).numpy(), 0.0) 101 | self.assertEqual(TanimotoDistanceLoss(from_logits=False)(y_pred, y_pred).numpy(), 0.0) 102 | self.assertAlmostEqual(TanimotoDistanceLoss(from_logits=False)(y_true, y_pred).numpy(), 1.25, 5) 103 | self.assertAlmostEqual(TanimotoDistanceLoss(from_logits=False, normalise=True)(y_true, y_pred).numpy(), 104 | 1.2460148, 5) 105 | self.assertAlmostEqual(TanimotoDistanceLoss(from_logits=False, class_weights=np.array([1, 0]))(y_true, 106 | y_pred).numpy(), 107 | 0.25, 5) 108 | 109 | y_true = np.zeros([1, 32, 32, 2], dtype=np.float32) 110 | y_true[..., 0] = np.ones([1, 32, 32]) - y_true[..., 1] 111 | self.assertEqual(TanimotoDistanceLoss(from_logits=False, normalise=True)(y_true, y_pred).numpy(), 0.) 112 | 113 | 114 | if __name__ == '__main__': 115 | unittest.main() -------------------------------------------------------------------------------- /tests/test_metrics.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import numpy as np 3 | 4 | from eoflow.models.metrics import MeanIoU, MCCMetric 5 | from eoflow.models.metrics import GeometricMetrics 6 | from scipy import ndimage 7 | 8 | class TestMeanIoU(unittest.TestCase): 9 | def test_not_initialized(self): 10 | metric = MeanIoU() 11 | 12 | y_true = np.zeros((1, 32, 32, 3)) 13 | y_pred = np.zeros((1, 32, 32, 3)) 14 | 15 | # Errors should be raised (because not initialized) 16 | self.assertRaises(ValueError, metric.update_state, y_true, y_pred) 17 | self.assertRaises(ValueError, metric.result) 18 | self.assertRaises(ValueError, metric.reset_states) 19 | self.assertRaises(ValueError, metric.get_config) 20 | 21 | metric.init_from_config() 22 | 23 | # Test that errors are not raised 24 | metric.update_state(y_true, y_pred) 25 | metric.result() 26 | metric.reset_states() 27 | metric.get_config() 28 | 29 | def test_iou_results(self): 30 | metric = MeanIoU() 31 | metric.init_from_config({'n_classes': 3}) 32 | 33 | ones = np.ones((32, 32)) 34 | zeros = np.zeros((32, 32)) 35 | mixed = np.concatenate([ones[:16], zeros[:16]]) 36 | 37 | # Predict everything as class 1 38 | y_pred = np.stack([zeros, ones], axis=-1) 39 | 40 | y_true1 = np.stack([ones, zeros], axis=-1) # All class 0 41 | y_true2 = np.stack([zeros, ones], axis=-1) # All class 1 42 | y_true3 = np.stack([mixed, 1 - mixed], axis=-1) # Half class 1, half class 0 43 | 44 | # Check each one seperately 45 | metric.update_state(y_true1, y_pred) 46 | self.assertAlmostEqual(metric.result().numpy(), 0.0, 10) 47 | 48 | metric.reset_states() 49 | metric.update_state(y_true2, y_pred) 50 | self.assertAlmostEqual(metric.result().numpy(), 1.0, 10) 51 | 52 | metric.reset_states() 53 | metric.update_state(y_true3, y_pred) 54 | self.assertAlmostEqual(metric.result().numpy(), 0.25, 10) # Class 1 IoU: 0.5, Class 2 IoU: 0.0 55 | 56 | # Check aggregation 57 | metric.reset_states() 58 | metric.update_state(y_true1, y_pred) 59 | metric.update_state(y_true2, y_pred) 60 | metric.update_state(y_true3, y_pred) 61 | self.assertAlmostEqual(metric.result().numpy(), 0.25, 10) # Class 1 IoU: 0.5, Class 2 IoU: 0.0 62 | 63 | 64 | class TestMCC(unittest.TestCase): 65 | def test_not_initialized(self): 66 | metric = MCCMetric() 67 | 68 | y_true = np.zeros((1, 32, 32, 3)) 69 | y_pred = np.zeros((1, 32, 32, 3)) 70 | 71 | # Errors should be raised (because not initialized) 72 | self.assertRaises(ValueError, metric.update_state, y_true, y_pred) 73 | self.assertRaises(ValueError, metric.result) 74 | self.assertRaises(ValueError, metric.reset_states) 75 | self.assertRaises(ValueError, metric.get_config) 76 | 77 | metric.init_from_config({'n_classes': 3}) 78 | 79 | # Test that errors are not raised 80 | metric.update_state(y_true, y_pred) 81 | metric.result() 82 | metric.reset_states() 83 | metric.get_config() 84 | 85 | def test_wrong_n_classes(self): 86 | metric = MCCMetric() 87 | 88 | n_classes = 3 89 | y_true = np.zeros((1, 32, 32, n_classes)) 90 | y_pred = np.zeros((1, 32, 32, n_classes)) 91 | 92 | metric.init_from_config({'n_classes': 1}) 93 | 94 | # Test that errors are raised 95 | with self.assertRaises(Exception) as context: 96 | metric.update_state(y_true, y_pred) 97 | self.assertTrue((f'Input to reshape is a tensor with {np.prod(y_true.shape)} values, ' 98 | f'but the requested shape has {np.prod(y_true.shape[:-1])}') in str(context.exception)) 99 | 100 | def test_mcc_results_binary_symmetry(self): 101 | metric = MCCMetric() 102 | metric.init_from_config({'n_classes': 2}) 103 | 104 | y_pred = np.random.randint(0, 2, (32, 32, 1)) 105 | y_pred = np.concatenate((y_pred, 1-y_pred), axis=-1) 106 | 107 | y_true = np.random.randint(0, 2, (32, 32, 1)) 108 | y_true = np.concatenate((y_true, 1 - y_true), axis=-1) 109 | 110 | metric.update_state(y_true, y_pred) 111 | results = metric.result().numpy() 112 | self.assertAlmostEqual(results[0], results[1], 7) 113 | 114 | def test_mcc_single_vs_binary(self): 115 | metric_single = MCCMetric() 116 | metric_single.init_from_config({'n_classes': 1}) 117 | 118 | y_pred = np.random.randint(0, 2, (32, 32, 1)) 119 | y_true = np.random.randint(0, 2, (32, 32, 1)) 120 | metric_single.update_state(y_true, y_pred) 121 | result_single = metric_single.result().numpy()[0] 122 | 123 | metric_binary = MCCMetric() 124 | metric_binary.init_from_config({'n_classes': 2}) 125 | 126 | y_pred = np.concatenate((y_pred, 1-y_pred), axis=-1) 127 | y_true = np.concatenate((y_true, 1 - y_true), axis=-1) 128 | metric_binary.update_state(y_true, y_pred) 129 | result_binary = metric_binary.result().numpy()[0] 130 | 131 | self.assertAlmostEqual(result_single, result_binary, 7) 132 | 133 | def test_mcc_results(self): 134 | # test is from an example of MCC in sklearn.metrics matthews_corrcoef 135 | y_true = np.array([1, 1, 1, 0])[..., np.newaxis] 136 | y_pred = np.array([1, 0, 1, 1])[..., np.newaxis] 137 | metric = MCCMetric() 138 | metric.init_from_config({'n_classes': 1}) 139 | metric.update_state(y_true, y_pred) 140 | self.assertAlmostEqual(metric.result().numpy()[0], -0.3333333, 7) 141 | 142 | def test_mcc_threshold(self): 143 | y_true = np.array([1, 1, 1, 0])[..., np.newaxis] 144 | y_pred = np.array([0.9, 0.6, 0.61, 0.7])[..., np.newaxis] 145 | metric = MCCMetric() 146 | metric.init_from_config({'n_classes': 1, 'mcc_threshold': 0.6}) 147 | metric.update_state(y_true, y_pred) 148 | self.assertAlmostEqual(metric.result().numpy()[0], -0.3333333, 7) 149 | 150 | 151 | class TestGeometricMetric(unittest.TestCase): 152 | 153 | def detect_edges(self, im, thr=0): 154 | sx = ndimage.sobel(im, axis=0, mode='constant') 155 | sy = ndimage.sobel(im, axis=1, mode='constant') 156 | sob = np.hypot(sx, sy) 157 | return sob > thr 158 | 159 | def test_equal_geometries(self): 160 | 161 | metric = GeometricMetrics(edge_func=self.detect_edges) 162 | 163 | y_true = np.zeros((2, 32, 32)) 164 | y_pred = np.zeros((2, 32, 32)) 165 | 166 | y_true[0, 10:20, 10:20] = 1 167 | y_pred[0, 10:20, 10:20] = 1 168 | 169 | y_true[1, 0:10, 0:10] = 1 170 | y_pred[1, 0:10, 0:10] = 1 171 | 172 | y_true[1, 15:20, 15:20] = 1 173 | y_pred[1, 15:20, 15:20] = 1 174 | 175 | metric.update_state(y_true, y_pred) 176 | overseg_err, underseg_err, border_err, fragmentation_err = metric.result() 177 | 178 | self.assertEqual(overseg_err, 0., "For equal geometries oversegmentation should be 0!") 179 | self.assertEqual(underseg_err, 0., "For equal geometries undersegmentation should be 0!") 180 | self.assertEqual(fragmentation_err, 0., "For equal geometries fragmentation error should be 0!") 181 | self.assertEqual(border_err, 0., "For equal geometries border error should be 0!") 182 | 183 | def test_empty_geometries(self): 184 | 185 | metric = GeometricMetrics(edge_func=self.detect_edges) 186 | 187 | y_true = np.ones((1, 32, 32)) 188 | y_pred = np.zeros((1, 32, 32)) 189 | 190 | metric.update_state(y_true, y_pred) 191 | overseg_err, underseg_err, border_err, fragmentation_err = metric.result() 192 | 193 | self.assertEqual(overseg_err, 1., "For empty geometries oversegmentation should be 1!") 194 | self.assertEqual(underseg_err, 1., "For empty geometries undersegmentation should be 1!") 195 | self.assertEqual(fragmentation_err, 0., "For empty geometries fragmentation error should be 0!") 196 | self.assertEqual(border_err, 1., "For empty geometries border error should be 1!") 197 | 198 | def test_quarter(self): 199 | metric = GeometricMetrics(edge_func=self.detect_edges) 200 | 201 | # A quarter of measurement covers a quarter of reference 202 | y_true = np.zeros((1, 200, 200)) 203 | y_pred = np.zeros((1, 200, 200)) 204 | 205 | y_true[0, :100, :100] = 1 206 | y_pred[0, 50:150, 50:150] = 1 207 | 208 | metric.update_state(y_true, y_pred) 209 | overseg_err, underseg_err, border_err, fragmentation_err = metric.result() 210 | 211 | self.assertEqual(overseg_err, 0.75) 212 | self.assertEqual(underseg_err, 0.75) 213 | self.assertEqual(fragmentation_err, 0.) 214 | self.assertAlmostEqual(border_err, 0.9949494949494949) 215 | 216 | def test_multiple(self): 217 | metric = GeometricMetrics(edge_func=self.detect_edges) 218 | 219 | # A quarter of measurement covers a quarter of reference 220 | y_true = np.zeros((1, 200, 200)) 221 | y_pred = np.zeros((1, 200, 200)) 222 | 223 | y_true[0, 10:20, 20:120] = 1 224 | y_true[0, 30:40, 20:120] = 1 225 | y_true[0, 50:60, 20:120] = 1 226 | 227 | y_pred[0, 15:33, 20:120] = 1 228 | y_pred[0, 36:65, 20:120] = 1 229 | 230 | metric.update_state(y_true, y_pred) 231 | 232 | overseg_err, underseg_err, border_err, fragmentation_err = metric.result() 233 | self.assertEqual(overseg_err, 0.3666666666666667) 234 | self.assertEqual(underseg_err, 0.7464878671775222) 235 | self.assertEqual(fragmentation_err, 0.000333667000333667) 236 | self.assertAlmostEqual(border_err, 0.9413580246913581) 237 | 238 | if __name__ == '__main__': 239 | unittest.main() 240 | --------------------------------------------------------------------------------