├── LICENSE
└── README.md
/LICENSE:
--------------------------------------------------------------------------------
1 | Apache License
2 | Version 2.0, January 2004
3 | http://www.apache.org/licenses/
4 |
5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6 |
7 | 1. Definitions.
8 |
9 | "License" shall mean the terms and conditions for use, reproduction,
10 | and distribution as defined by Sections 1 through 9 of this document.
11 |
12 | "Licensor" shall mean the copyright owner or entity authorized by
13 | the copyright owner that is granting the License.
14 |
15 | "Legal Entity" shall mean the union of the acting entity and all
16 | other entities that control, are controlled by, or are under common
17 | control with that entity. For the purposes of this definition,
18 | "control" means (i) the power, direct or indirect, to cause the
19 | direction or management of such entity, whether by contract or
20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
21 | outstanding shares, or (iii) beneficial ownership of such entity.
22 |
23 | "You" (or "Your") shall mean an individual or Legal Entity
24 | exercising permissions granted by this License.
25 |
26 | "Source" form shall mean the preferred form for making modifications,
27 | including but not limited to software source code, documentation
28 | source, and configuration files.
29 |
30 | "Object" form shall mean any form resulting from mechanical
31 | transformation or translation of a Source form, including but
32 | not limited to compiled object code, generated documentation,
33 | and conversions to other media types.
34 |
35 | "Work" shall mean the work of authorship, whether in Source or
36 | Object form, made available under the License, as indicated by a
37 | copyright notice that is included in or attached to the work
38 | (an example is provided in the Appendix below).
39 |
40 | "Derivative Works" shall mean any work, whether in Source or Object
41 | form, that is based on (or derived from) the Work and for which the
42 | editorial revisions, annotations, elaborations, or other modifications
43 | represent, as a whole, an original work of authorship. For the purposes
44 | of this License, Derivative Works shall not include works that remain
45 | separable from, or merely link (or bind by name) to the interfaces of,
46 | the Work and Derivative Works thereof.
47 |
48 | "Contribution" shall mean any work of authorship, including
49 | the original version of the Work and any modifications or additions
50 | to that Work or Derivative Works thereof, that is intentionally
51 | submitted to Licensor for inclusion in the Work by the copyright owner
52 | or by an individual or Legal Entity authorized to submit on behalf of
53 | the copyright owner. For the purposes of this definition, "submitted"
54 | means any form of electronic, verbal, or written communication sent
55 | to the Licensor or its representatives, including but not limited to
56 | communication on electronic mailing lists, source code control systems,
57 | and issue tracking systems that are managed by, or on behalf of, the
58 | Licensor for the purpose of discussing and improving the Work, but
59 | excluding communication that is conspicuously marked or otherwise
60 | designated in writing by the copyright owner as "Not a Contribution."
61 |
62 | "Contributor" shall mean Licensor and any individual or Legal Entity
63 | on behalf of whom a Contribution has been received by Licensor and
64 | subsequently incorporated within the Work.
65 |
66 | 2. Grant of Copyright License. Subject to the terms and conditions of
67 | this License, each Contributor hereby grants to You a perpetual,
68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69 | copyright license to reproduce, prepare Derivative Works of,
70 | publicly display, publicly perform, sublicense, and distribute the
71 | Work and such Derivative Works in Source or Object form.
72 |
73 | 3. Grant of Patent License. Subject to the terms and conditions of
74 | this License, each Contributor hereby grants to You a perpetual,
75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76 | (except as stated in this section) patent license to make, have made,
77 | use, offer to sell, sell, import, and otherwise transfer the Work,
78 | where such license applies only to those patent claims licensable
79 | by such Contributor that are necessarily infringed by their
80 | Contribution(s) alone or by combination of their Contribution(s)
81 | with the Work to which such Contribution(s) was submitted. If You
82 | institute patent litigation against any entity (including a
83 | cross-claim or counterclaim in a lawsuit) alleging that the Work
84 | or a Contribution incorporated within the Work constitutes direct
85 | or contributory patent infringement, then any patent licenses
86 | granted to You under this License for that Work shall terminate
87 | as of the date such litigation is filed.
88 |
89 | 4. Redistribution. You may reproduce and distribute copies of the
90 | Work or Derivative Works thereof in any medium, with or without
91 | modifications, and in Source or Object form, provided that You
92 | meet the following conditions:
93 |
94 | (a) You must give any other recipients of the Work or
95 | Derivative Works a copy of this License; and
96 |
97 | (b) You must cause any modified files to carry prominent notices
98 | stating that You changed the files; and
99 |
100 | (c) You must retain, in the Source form of any Derivative Works
101 | that You distribute, all copyright, patent, trademark, and
102 | attribution notices from the Source form of the Work,
103 | excluding those notices that do not pertain to any part of
104 | the Derivative Works; and
105 |
106 | (d) If the Work includes a "NOTICE" text file as part of its
107 | distribution, then any Derivative Works that You distribute must
108 | include a readable copy of the attribution notices contained
109 | within such NOTICE file, excluding those notices that do not
110 | pertain to any part of the Derivative Works, in at least one
111 | of the following places: within a NOTICE text file distributed
112 | as part of the Derivative Works; within the Source form or
113 | documentation, if provided along with the Derivative Works; or,
114 | within a display generated by the Derivative Works, if and
115 | wherever such third-party notices normally appear. The contents
116 | of the NOTICE file are for informational purposes only and
117 | do not modify the License. You may add Your own attribution
118 | notices within Derivative Works that You distribute, alongside
119 | or as an addendum to the NOTICE text from the Work, provided
120 | that such additional attribution notices cannot be construed
121 | as modifying the License.
122 |
123 | You may add Your own copyright statement to Your modifications and
124 | may provide additional or different license terms and conditions
125 | for use, reproduction, or distribution of Your modifications, or
126 | for any such Derivative Works as a whole, provided Your use,
127 | reproduction, and distribution of the Work otherwise complies with
128 | the conditions stated in this License.
129 |
130 | 5. Submission of Contributions. Unless You explicitly state otherwise,
131 | any Contribution intentionally submitted for inclusion in the Work
132 | by You to the Licensor shall be under the terms and conditions of
133 | this License, without any additional terms or conditions.
134 | Notwithstanding the above, nothing herein shall supersede or modify
135 | the terms of any separate license agreement you may have executed
136 | with Licensor regarding such Contributions.
137 |
138 | 6. Trademarks. This License does not grant permission to use the trade
139 | names, trademarks, service marks, or product names of the Licensor,
140 | except as required for reasonable and customary use in describing the
141 | origin of the Work and reproducing the content of the NOTICE file.
142 |
143 | 7. Disclaimer of Warranty. Unless required by applicable law or
144 | agreed to in writing, Licensor provides the Work (and each
145 | Contributor provides its Contributions) on an "AS IS" BASIS,
146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 | implied, including, without limitation, any warranties or conditions
148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 | PARTICULAR PURPOSE. You are solely responsible for determining the
150 | appropriateness of using or redistributing the Work and assume any
151 | risks associated with Your exercise of permissions under this License.
152 |
153 | 8. Limitation of Liability. In no event and under no legal theory,
154 | whether in tort (including negligence), contract, or otherwise,
155 | unless required by applicable law (such as deliberate and grossly
156 | negligent acts) or agreed to in writing, shall any Contributor be
157 | liable to You for damages, including any direct, indirect, special,
158 | incidental, or consequential damages of any character arising as a
159 | result of this License or out of the use or inability to use the
160 | Work (including but not limited to damages for loss of goodwill,
161 | work stoppage, computer failure or malfunction, or any and all
162 | other commercial damages or losses), even if such Contributor
163 | has been advised of the possibility of such damages.
164 |
165 | 9. Accepting Warranty or Additional Liability. While redistributing
166 | the Work or Derivative Works thereof, You may choose to offer,
167 | and charge a fee for, acceptance of support, warranty, indemnity,
168 | or other liability obligations and/or rights consistent with this
169 | License. However, in accepting such obligations, You may act only
170 | on Your own behalf and on Your sole responsibility, not on behalf
171 | of any other Contributor, and only if You agree to indemnify,
172 | defend, and hold each Contributor harmless for any liability
173 | incurred by, or claims asserted against, such Contributor by reason
174 | of your accepting any such warranty or additional liability.
175 |
176 | END OF TERMS AND CONDITIONS
177 |
178 | APPENDIX: How to apply the Apache License to your work.
179 |
180 | To apply the Apache License to your work, attach the following
181 | boilerplate notice, with the fields enclosed by brackets "[]"
182 | replaced with your own identifying information. (Don't include
183 | the brackets!) The text should be enclosed in the appropriate
184 | comment syntax for the file format. We also recommend that a
185 | file or class name and description of purpose be included on the
186 | same "printed page" as the copyright notice for easier
187 | identification within third-party archives.
188 |
189 | Copyright [yyyy] [name of copyright owner]
190 |
191 | Licensed under the Apache License, Version 2.0 (the "License");
192 | you may not use this file except in compliance with the License.
193 | You may obtain a copy of the License at
194 |
195 | http://www.apache.org/licenses/LICENSE-2.0
196 |
197 | Unless required by applicable law or agreed to in writing, software
198 | distributed under the License is distributed on an "AS IS" BASIS,
199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 | See the License for the specific language governing permissions and
201 | limitations under the License.
202 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | [](https://opensource.org/licenses/Apache-2.0)
2 | 
[](https://hits.seeyoufarm.com)
3 | # Loss-Functions-Package-Tensorflow-Keras-PyTorch
4 |
5 | This rope implements some popular Loass/Cost/Objective Functions that you can use to train your Deep Learning models.
6 |
7 | **With multi-class classification or segmentation, we sometimes use loss functions that calculate the average loss for each class, rather than calculating loss from the prediction tensor as a whole. This kernel is meant as a template reference for the basic code, so all examples calculate loss on the entire tensor, but it should be trivial for you to modify it for multi-class averaging.**
8 |
9 | I have provided the implementations in three popular libraries i.e. `tensorflow` `keras` and `pytorch`. Lets get started.
10 |
11 | These functions cannot simply be written in NumPy, as they must operate on tensors that also have gradient parameters which need to be calculated throughout the model during backpropagation. According, loss functions must be written using backend functions from the respective model library.
12 |
13 | With multi-class classification or segmentation, we sometimes use loss functions that calculate the average loss for each class, rather than calculating loss from the prediction tensor as a whole. This kernel is meant as a template reference for the basic code, so all examples calculate loss on the entire tensor, but it should be trivial for you to modify it for multi-class averaging.
14 | #### For [Learning-Rate-Schedulers-Packege-Tensorflow-PyTorch-Keras](https://github.com/Mr-TalhaIlyas/Learning-Rate-Schedulers-Packege-Tensorflow-PyTorch-Keras)
15 | #### For [Evaluation-Metrics-Package-Tensorflow-PyTorch-Keras](https://github.com/Mr-TalhaIlyas/Evaluation-Metrics-Package-Tensorflow-PyTorch-Keras/)
16 | ## Necessary Imports
17 | You can import some necessary packages as follows
18 | ```python
19 | # PyTorch
20 | import torch
21 | import torch.nn as nn
22 | import torch.nn.functional as F
23 | # Keras
24 | import keras
25 | import keras.backend as K
26 | # Tensorflow
27 | form tensorflow import keras
28 | from tensorflow import keras.backend as K
29 | ```
30 | ## Weighted Catagorical Cross Entropy Loss
31 |
32 | ```python
33 | # Tensorflow/Keras
34 | def weighted_categorical_crossentropy(weights):
35 | """
36 | A weighted version of keras.objectives.categorical_crossentropy
37 |
38 | weights: numpy array of shape (C,) where C is the number of classes
39 | np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.
40 | """
41 | weights = K.variable(weights)
42 |
43 | def loss(y_true, y_pred, from_logits=False):
44 | if from_logits:
45 | y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
46 | #y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
47 | # clip to prevent NaN's and Inf's
48 | y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
49 | # calc
50 | loss = y_true * K.log(y_pred) * weights
51 | loss = -K.sum(loss, -1)
52 | return loss
53 |
54 | return loss
55 | ```
56 | Or you can check the `weighted focal loss` below and set the `gamma=0` and `alpha=1` and it'll work same as `weighted catagorical cross entropy`
57 |
58 | ## Dice Loss
59 | The Dice coefficient, or Dice-Sørensen coefficient, is a common metric for pixel segmentation that can also be modified to act as a loss function:
60 |
61 | ```python
62 | #PyTorch
63 | class DiceLoss(nn.Module):
64 | def __init__(self, weight=None, size_average=True):
65 | super(DiceLoss, self).__init__()
66 |
67 | def forward(self, inputs, targets, smooth=1):
68 |
69 | #comment out if your model contains a sigmoid or equivalent activation layer
70 | inputs = F.sigmoid(inputs)
71 |
72 | #flatten label and prediction tensors
73 | inputs = inputs.view(-1)
74 | targets = targets.view(-1)
75 |
76 | intersection = (inputs * targets).sum()
77 | dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)
78 |
79 | return 1 - dice
80 | ```
81 | ```python
82 | def DiceLoss(y_true, y_pred, smooth=1e-6):
83 |
84 | # if you are using this loss for multi-class segmentation then uncomment
85 | # following lines
86 | # if y_pred.shape[-1] <= 1:
87 | # # activate logits
88 | # y_pred = tf.keras.activations.sigmoid(y_pred)
89 | # elif y_pred.shape[-1] >= 2:
90 | # # activate logits
91 | # y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
92 | # # convert the tensor to one-hot for multi-class segmentation
93 | # y_true = K.squeeze(y_true, 3)
94 | # y_true = tf.cast(y_true, "int32")
95 | # y_true = tf.one_hot(y_true, num_class, axis=-1)
96 |
97 | # cast to float32 datatype
98 | y_true = K.cast(y_true, 'float32')
99 | y_pred = K.cast(y_pred, 'float32')
100 |
101 | #flatten label and prediction tensors
102 | inputs = K.flatten(y_pred)
103 | targets = K.flatten(y_true)
104 |
105 | intersection = K.sum(K.dot(targets, inputs))
106 | dice = (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
107 | return 1 - dice
108 | ```
109 | ## BCE-Dice Loss
110 | This loss combines Dice loss with the standard binary cross-entropy (BCE) loss that is generally the default for segmentation models. Combining the two methods allows for some diversity in the loss, while benefitting from the stability of BCE. The equation for multi-class BCE by itself will be familiar to anyone who has studied logistic regression:
111 | ```python
112 | #PyTorch
113 | class DiceBCELoss(nn.Module):
114 | def __init__(self, weight=None, size_average=True):
115 | super(DiceBCELoss, self).__init__()
116 |
117 | def forward(self, inputs, targets, smooth=1):
118 |
119 | #comment out if your model contains a sigmoid or equivalent activation layer
120 | inputs = F.sigmoid(inputs)
121 |
122 | #flatten label and prediction tensors
123 | inputs = inputs.view(-1)
124 | targets = targets.view(-1)
125 |
126 | intersection = (inputs * targets).sum()
127 | dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)
128 | BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')
129 | Dice_BCE = BCE + dice_loss
130 |
131 | return Dice_BCE
132 | ```
133 | ```python
134 | class DiceLossMulticlass(nn.Module):
135 | def __init__(self, weights=None, size_average=False):
136 | super(mIoULoss, self).__init__()
137 |
138 | def forward(self, inputs, targets, smooth=1):
139 | if self.weights is not None:
140 | assert self.weights.shape == (targets.shape[1], )
141 |
142 | # make a copy not to change the default weights in the instance of DiceLossMulticlass
143 | weights = self.weights.copy()
144 |
145 | #comment out if your model contains a sigmoid or equivalent activation layer
146 | inputs = F.sigmoid(inputs)
147 |
148 | # flatten label and prediction images, leave BATCH and NUM_CLASSES
149 | # (BATCH, NUM_CLASSES, H, W) -> (BATCH, NUM_CLASSES, H * W)
150 | inputs = inputs.view(inputs.shape[0],inputs.shape[1],-1)
151 | targets = targets.view(targets.shape[0],targets.shape[1],-1)
152 |
153 | #intersection = (inputs * targets).sum()
154 | intersection = (inputs * targets).sum(0).sum(1)
155 | #dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)
156 | dice = (2.*intersection + smooth)/(inputs.sum(0).sum(1) + targets.sum(0).sum(1) + smooth)
157 |
158 | if (weights is None) and self.size_average==True:
159 | weights = (targets == 1).sum(0).sum(1)
160 | weights /= weights.sum() # so they sum up to 1
161 |
162 | if weights is not None:
163 | return 1 - (dice*weights).mean()
164 | else:
165 | return 1 - weights.mean()
166 | ```
167 |
168 | ```python
169 | #Tensorflow / Keras
170 | def DiceBCELoss(y_true, y_pred, smooth=1e-6):
171 |
172 | # if you are using this loss for multi-class segmentation then uncomment
173 | # following lines
174 | # if y_pred.shape[-1] <= 1:
175 | # # activate logits
176 | # y_pred = tf.keras.activations.sigmoid(y_pred)
177 | # elif y_pred.shape[-1] >= 2:
178 | # # activate logits
179 | # y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
180 | # # convert the tensor to one-hot for multi-class segmentation
181 | # y_true = K.squeeze(y_true, 3)
182 | # y_true = tf.cast(y_true, "int32")
183 | # y_true = tf.one_hot(y_true, num_class, axis=-1)
184 |
185 | # cast to float32 datatype
186 | y_true = K.cast(y_true, 'float32')
187 | y_pred = K.cast(y_pred, 'float32')
188 |
189 | #flatten label and prediction tensors
190 | inputs = K.flatten(y_pred)
191 | targets = K.flatten(y_true)
192 |
193 | BCE = binary_crossentropy(targets, inputs)
194 | intersection = K.sum(K.dot(targets, inputs))
195 | dice_loss = 1 - (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
196 | Dice_BCE = BCE + dice_loss
197 |
198 | return Dice_BCE
199 | ```
200 | ### Weighted BCE and Dice Loss
201 | Combines BCE and Dice loss
202 | ```python
203 | # Keras/ Tensorflow
204 | def Weighted_BCEnDice_loss(y_true, y_pred):
205 |
206 | # if you are using this loss for multi-class segmentation then uncomment
207 | # following lines
208 | # if y_pred.shape[-1] <= 1:
209 | # # activate logits
210 | # y_pred = tf.keras.activations.sigmoid(y_pred)
211 | # elif y_pred.shape[-1] >= 2:
212 | # # activate logits
213 | # y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
214 | # # convert the tensor to one-hot for multi-class segmentation
215 | # y_true = K.squeeze(y_true, 3)
216 | # y_true = tf.cast(y_true, "int32")
217 | # y_true = tf.one_hot(y_true, num_class, axis=-1)
218 |
219 |
220 | y_true = K.cast(y_true, 'float32')
221 | y_pred = K.cast(y_pred, 'float32')
222 | # if we want to get same size of output, kernel size must be odd number
223 | averaged_mask = K.pool2d(
224 | y_true, pool_size=(11, 11), strides=(1, 1), padding='same', pool_mode='avg')
225 | border = K.cast(K.greater(averaged_mask, 0.005), 'float32') * K.cast(K.less(averaged_mask, 0.995), 'float32')
226 | weight = K.ones_like(averaged_mask)
227 | w0 = K.sum(weight)
228 | weight += border * 2
229 | w1 = K.sum(weight)
230 | weight *= (w0 / w1)
231 | loss = weighted_dice_loss(y_true, y_pred, weight) + weighted_bce_loss(y_true, y_pred, weight)
232 | return loss
233 |
234 | def weighted_bce_loss(y_true, y_pred, weight):
235 | # avoiding overflow
236 | epsilon = 1e-7
237 | y_pred = K.clip(y_pred, epsilon, 1. - epsilon)
238 | logit_y_pred = K.log(y_pred / (1. - y_pred))
239 | #logit_y_pred = y_pred
240 |
241 | loss = (1. - y_true) * logit_y_pred + (1. + (weight - 1.) * y_true) * \
242 | (K.log(1. + K.exp(-K.abs(logit_y_pred))) + K.maximum(-logit_y_pred, 0.))
243 | return K.sum(loss) / K.sum(weight)
244 |
245 | def weighted_dice_loss(y_true, y_pred, weight):
246 | smooth = 1.
247 | w, m1, m2 = weight * weight, y_true, y_pred
248 | intersection = (m1 * m2)
249 | score = (2. * K.sum(w * intersection) + smooth) / (K.sum(w * (m1**2)) + K.sum(w * (m2**2)) + smooth) # Uptill here is Dice Loss with squared
250 | loss = 1. - K.sum(score) #Soft Dice Loss
251 | return loss
252 | ```
253 | ## HED Loss
254 | I was introduced in holistic edge detector to detect edges/boundaries of objects in https://arxiv.org/pdf/1504.06375.pdf.
255 | ```python
256 | # Keras/ Tensorflow
257 | def HED_loss(y_true, y_pred):
258 |
259 | #y_true = y_true * 255 # b/c keras generator normalizes images
260 | if y_pred.shape[-1] <= 1:
261 | y_true = y_true[:,:,:,0:1]
262 | elif y_pred.shape[-1] >= 2:
263 | y_true = K.squeeze(y_true, 3)
264 | y_true = tf.cast(y_true, "int32")
265 | y_true = tf.one_hot(y_true, num_class, axis=-1)
266 |
267 | y_true = K.cast(y_true, 'float32')
268 | y_pred = K.cast(y_pred, 'float32')
269 |
270 | loss = sigmoid_cross_entropy_balanced(y_pred, y_true)
271 | return loss
272 |
273 | def sigmoid_cross_entropy_balanced(logits, label, name='cross_entropy_loss'):
274 | """
275 | From:
276 |
277 | https://github.com/moabitcoin/holy-edge/blob/master/hed/losses.py
278 |
279 | Implements Equation [2] in https://arxiv.org/pdf/1504.06375.pdf
280 | Compute edge pixels for each training sample and set as pos_weights to
281 | tf.nn.weighted_cross_entropy_with_logits
282 | """
283 | y = tf.cast(label, tf.float32)
284 |
285 | count_neg = tf.reduce_sum(1. - y)
286 | count_pos = tf.reduce_sum(y)
287 |
288 | # Equation [2]
289 | beta = count_neg / (count_neg + count_pos)
290 |
291 | # Equation [2] divide by 1 - beta
292 | pos_weight = beta / (1 - beta)
293 | if int(str(tf.__version__)[0]) == 1:
294 | cost = tf.nn.weighted_cross_entropy_with_logits(logits=logits, targets=y, pos_weight=pos_weight)
295 | if int(str(tf.__version__)[0]) == 2:
296 | cost = tf.nn.weighted_cross_entropy_with_logits(logits=logits, labels=y, pos_weight=pos_weight)
297 |
298 | # Multiply by 1 - beta
299 | cost = tf.reduce_mean(cost * (1 - beta))
300 |
301 | # check if image has no edge pixels return 0 else return complete error function
302 | return tf.where(tf.equal(count_pos, 0.0), 0.0, cost, name=name)
303 | ```
304 | ## Jaccard/Intersection over Union (IoU) Loss
305 | The IoU metric, or Jaccard Index, is similar to the Dice metric and is calculated as the ratio between the overlap of the positive instances between two sets, and their mutual combined values:
306 |
307 | Like the Dice metric, it is a common means of evaluating the performance of pixel segmentation models.
308 |
309 | ```python
310 | #PyTorch
311 | class IoULoss(nn.Module):
312 | def __init__(self, weight=None, size_average=True):
313 | super(IoULoss, self).__init__()
314 |
315 | def forward(self, inputs, targets, smooth=1):
316 |
317 | #comment out if your model contains a sigmoid or equivalent activation layer
318 | inputs = F.sigmoid(inputs)
319 |
320 | #flatten label and prediction tensors
321 | inputs = inputs.view(-1)
322 | targets = targets.view(-1)
323 |
324 | #intersection is equivalent to True Positive count
325 | #union is the mutually inclusive area of all labels & predictions
326 | intersection = (inputs * targets).sum()
327 | total = (inputs + targets).sum()
328 | union = total - intersection
329 |
330 | IoU = (intersection + smooth)/(union + smooth)
331 |
332 | return 1 - IoU
333 | ```
334 | ```python
335 | #Tensorflow / Keras
336 | def IoULoss(y_true, y_pred, smooth=1e-6):
337 |
338 | # if you are using this loss for multi-class segmentation then uncomment
339 | # following lines
340 | # if y_pred.shape[-1] <= 1:
341 | # # activate logits
342 | # y_pred = tf.keras.activations.sigmoid(y_pred)
343 | # elif y_pred.shape[-1] >= 2:
344 | # # activate logits
345 | # y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
346 | # # convert the tensor to one-hot for multi-class segmentation
347 | # y_true = K.squeeze(y_true, 3)
348 | # y_true = tf.cast(y_true, "int32")
349 | # y_true = tf.one_hot(y_true, num_class, axis=-1)
350 |
351 | # cast to float32 datatype
352 | y_true = K.cast(y_true, 'float32')
353 | y_pred = K.cast(y_pred, 'float32')
354 |
355 | #flatten label and prediction tensors
356 | inputs = K.flatten(y_pred)
357 | targets = K.flatten(y_true)
358 |
359 | intersection = K.sum(K.dot(targets, inputs))
360 | total = K.sum(targets) + K.sum(inputs)
361 | union = total - intersection
362 |
363 | IoU = (intersection + smooth) / (union + smooth)
364 | return 1 - IoU
365 | ```
366 |
367 | ## Focal Loss
368 | Focal Loss was introduced by Lin et al of Facebook AI Research in 2017 as a means of combatting extremely imbalanced datasets where positive cases were relatively rare. Their paper "Focal Loss for Dense Object Detection" is retrievable here: https://arxiv.org/abs/1708.02002. In practice, the researchers used an alpha-modified version of the function so I have included it in this implementation.
369 |
370 | ```python
371 | #PyTorch
372 | ALPHA = 0.8
373 | GAMMA = 2
374 |
375 | class FocalLoss(nn.Module):
376 | def __init__(self, weight=None, size_average=True):
377 | super(FocalLoss, self).__init__()
378 |
379 | def forward(self, inputs, targets, alpha=ALPHA, gamma=GAMMA, smooth=1):
380 |
381 | #comment out if your model contains a sigmoid or equivalent activation layer
382 | inputs = F.sigmoid(inputs)
383 |
384 | #flatten label and prediction tensors
385 | inputs = inputs.view(-1)
386 | targets = targets.view(-1)
387 |
388 | #first compute binary cross-entropy
389 | BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')
390 | BCE_EXP = torch.exp(-BCE)
391 | focal_loss = alpha * (1-BCE_EXP)**gamma * BCE
392 |
393 | return focal_loss
394 | ```
395 |
396 | ```python
397 | #Tensorflow / Keras
398 |
399 | def FocalLoss(y_true, y_pred):
400 |
401 | alpha = 0.8
402 | gamma = 2
403 | # if you are using this loss for multi-class segmentation then uncomment
404 | # following lines
405 | # if y_pred.shape[-1] <= 1:
406 | # # activate logits
407 | # y_pred = tf.keras.activations.sigmoid(y_pred)
408 | # elif y_pred.shape[-1] >= 2:
409 | # # activate logits
410 | # y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
411 | # # convert the tensor to one-hot for multi-class segmentation
412 | # y_true = K.squeeze(y_true, 3)
413 | # y_true = tf.cast(y_true, "int32")
414 | # y_true = tf.one_hot(y_true, num_class, axis=-1)
415 |
416 | # cast to float32 datatype
417 | y_true = K.cast(y_true, 'float32')
418 | y_pred = K.cast(y_pred, 'float32')
419 |
420 | inputs = K.flatten(inputs)
421 | targets = K.flatten(targets)
422 |
423 | BCE = K.binary_crossentropy(targets, inputs)
424 | BCE_EXP = K.exp(-BCE)
425 | focal_loss = K.mean(alpha * K.pow((1-BCE_EXP), gamma) * BCE)
426 |
427 | return focal_loss
428 | ```
429 | ## Weighted Focal Loss
430 | *in developement*
431 | ```python
432 | # TensorFlow/Keras
433 | class WFL():
434 | '''
435 | Weighted Focal loss
436 | '''
437 | def __init__(self, alpha=0.25, gamma=2, class_weights=None, from_logits=False):
438 | self.class_weights = class_weights
439 | self.from_logits = from_logits
440 | self.alpha = alpha
441 | self.gamma = gamma
442 |
443 | def __call__(self, y_true, y_pred):
444 |
445 | if self.from_logits:
446 | y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
447 |
448 | y_pred = K.clip(y_pred, K.epsilon(), 1. - K.epsilon())
449 |
450 | # cast to float32 datatype
451 | y_true = K.cast(y_true, 'float32')
452 | y_pred = K.cast(y_pred, 'float32')
453 |
454 | WCCE = y_true * K.log(y_pred) * self.class_weights
455 | WFL = (self.alpha * K.pow((1-y_pred), self.gamma)) * WCCE
456 | # reduce sum -> reduces the loss over number of batches by simply taking sum over all samples
457 | # reduce mean -> reduces the loss ove number of batches by taking mean of all samples
458 | # if axis=-1 is given input batch is like B * C then loss will have shape B * 1
459 | # if axis is None then only 1 scaler value is output
460 |
461 | return -tf.math.reduce_sum(WFL, -1) #use this for custom training loop and dviding by global batch size. * (1/GB)
462 | #return -tf.reduce_mean(WFL, -1) # use this for complie fit keras API
463 | ```
464 | ## Tversky Loss
465 | This loss was introduced in "Tversky loss function for image segmentationusing 3D fully convolutional deep networks", retrievable here: https://arxiv.org/abs/1706.05721. It was designed to optimise segmentation on imbalanced medical datasets by utilising constants that can adjust how harshly different types of error are penalised in the loss function. From the paper:
466 | **... in the case of α=β=0.5 the Tversky index simplifies to be the same as the Dice coefficient, which is also equal to the F1 score. With α=β=1, Equation 2 produces Tanimoto coefficient, and setting α+β=1 produces the set of Fβ scores. Larger βs weigh recall higher than precision (by placing more emphasis on false negatives).**
467 | To summarise, this loss function is weighted by the constants 'alpha' and 'beta' that penalise false positives and false negatives respectively to a higher degree in the loss function as their value is increased. The beta constant in particular has applications in situations where models can obtain misleadingly positive performance via highly conservative prediction. You may want to experiment with different values to find the optimum. With alpha==beta==0.5, this loss becomes equivalent to Dice Loss.
468 |
469 | ```python
470 | #PyTorch
471 | ALPHA = 0.5
472 | BETA = 0.5
473 |
474 | class TverskyLoss(nn.Module):
475 | def __init__(self, weight=None, size_average=True):
476 | super(TverskyLoss, self).__init__()
477 |
478 | def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA):
479 |
480 | #comment out if your model contains a sigmoid or equivalent activation layer
481 | inputs = F.sigmoid(inputs)
482 |
483 | #flatten label and prediction tensors
484 | inputs = inputs.view(-1)
485 | targets = targets.view(-1)
486 |
487 | #True Positives, False Positives & False Negatives
488 | TP = (inputs * targets).sum()
489 | FP = ((1-targets) * inputs).sum()
490 | FN = (targets * (1-inputs)).sum()
491 |
492 | Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)
493 |
494 | return 1 - Tversky
495 | ```
496 |
497 | ```python
498 | #Tensorflow / Keras
499 | def TverskyLoss(y_true, y_pred, smooth=1e-6):
500 |
501 | if y_pred.shape[-1] <= 1:
502 | alpha = 0.3
503 | beta = 0.7
504 | gamma = 4/3 #5.
505 | y_pred = tf.keras.activations.sigmoid(y_pred)
506 | #y_true = y_true[:,:,:,0:1]
507 | elif y_pred.shape[-1] >= 2:
508 | alpha = 0.3
509 | beta = 0.7
510 | gamma = 4/3 #3.
511 | y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
512 | y_true = K.squeeze(y_true, 3)
513 | y_true = tf.cast(y_true, "int32")
514 | y_true = tf.one_hot(y_true, num_class, axis=-1)
515 |
516 | y_true = K.cast(y_true, 'float32')
517 | y_pred = K.cast(y_pred, 'float32')
518 | #flatten label and prediction tensors
519 | inputs = K.flatten(y_pred)
520 | targets = K.flatten(y_true)
521 |
522 | #True Positives, False Positives & False Negatives
523 | TP = K.sum((inputs * targets))
524 | FP = K.sum(((1-targets) * inputs))
525 | FN = K.sum((targets * (1-inputs)))
526 |
527 | Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)
528 |
529 | return 1 - Tversky
530 | ```
531 |
532 | ## Focal Tversky Loss
533 |
534 | A variant on the Tversky loss that also includes the gamma modifier from Focal Loss.
535 | ```python
536 | #PyTorch
537 | ALPHA = 0.5
538 | BETA = 0.5
539 | GAMMA = 1
540 |
541 | class FocalTverskyLoss(nn.Module):
542 | def __init__(self, weight=None, size_average=True):
543 | super(FocalTverskyLoss, self).__init__()
544 |
545 | def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA, gamma=GAMMA):
546 |
547 | #comment out if your model contains a sigmoid or equivalent activation layer
548 | inputs = F.sigmoid(inputs)
549 |
550 | #flatten label and prediction tensors
551 | inputs = inputs.view(-1)
552 | targets = targets.view(-1)
553 |
554 | #True Positives, False Positives & False Negatives
555 | TP = (inputs * targets).sum()
556 | FP = ((1-targets) * inputs).sum()
557 | FN = (targets * (1-inputs)).sum()
558 |
559 | Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)
560 | FocalTversky = (1 - Tversky)**gamma
561 |
562 | return FocalTversky
563 | ```
564 |
565 | ```python
566 | #Tensorflow / Keras
567 | def FocalTverskyLoss(y_true, y_pred, smooth=1e-6):
568 |
569 |
570 | if y_pred.shape[-1] <= 1:
571 | alpha = 0.3
572 | beta = 0.7
573 | gamma = 4/3 #5.
574 | y_pred = tf.keras.activations.sigmoid(y_pred)
575 | #y_true = y_true[:,:,:,0:1]
576 | elif y_pred.shape[-1] >= 2:
577 | alpha = 0.3
578 | beta = 0.7
579 | gamma = 4/3 #3.
580 | y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
581 | y_true = K.squeeze(y_true, 3)
582 | y_true = tf.cast(y_true, "int32")
583 | y_true = tf.one_hot(y_true, num_class, axis=-1)
584 |
585 |
586 | y_true = K.cast(y_true, 'float32')
587 | y_pred = K.cast(y_pred, 'float32')
588 | #flatten label and prediction tensors
589 | inputs = K.flatten(y_pred)
590 | targets = K.flatten(y_true)
591 |
592 | #True Positives, False Positives & False Negatives
593 | TP = K.sum((inputs * targets))
594 | FP = K.sum(((1-targets) * inputs))
595 | FN = K.sum((targets * (1-inputs)))
596 |
597 | Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)
598 | FocalTversky = K.pow((1 - Tversky), gamma)
599 |
600 | return FocalTversky
601 | ```
602 |
603 | ## Lovasz Hinge Loss
604 |
605 | This complex loss function was introduced by Berman, Triki and Blaschko in their paper "The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks", retrievable here: https://arxiv.org/abs/1705.08790. It is designed to optimise the Intersection over Union score for semantic segmentation, particularly for multi-class instances. Specifically, it sorts predictions by their error before calculating cumulatively how each error affects the IoU score. This gradient vector is then multiplied with the initial error vector to penalise most strongly the predictions that decreased the IoU score the most. This procedure is detailed by jeandebleu in his excellent summary here.
606 |
607 | This code is taken directly from the author's github repo here: https://github.com/bermanmaxim/LovaszSoftmax and all credit is to them.
608 |
609 | In this kernel I have implemented the flat variant that uses reshaped rank-1 tensors as inputs for PyTorch. You can modify it accordingly with the dimensions and class number of your data as needed. This code takes raw logits so ensure your model does not contain an activation layer prior to the loss calculation.
610 |
611 | I have hidden the researchers' own code below for brevity; simply load it into your kernel for the losses to function. In the case of their tensorflow implementation, I am still working to make it compatible with Keras. There are differences between the Tensorflow and Keras function libraries that complicate this.
612 |
613 | ```python
614 | #PyTorch
615 | class LovaszSoftmax(nn.Module):
616 | def __init__(self, classes='present', per_image=False, ignore=None):
617 | super(LovaszSoftmax, self).__init__()
618 | self.classes = classes
619 | self.per_image = per_image
620 | self.ignore = ignore
621 |
622 | def forward(self, inputs, targets):
623 | probas = F.softmax(inputs, dim=1) # B*C*H*W -> from logits to probabilities
624 | return lovasz_softmax(probas, targets, self.classes, self.per_image, self.ignore)
625 |
626 | def lovasz_softmax(probas, labels, classes='present', per_image=False, ignore=None):
627 | """
628 | Multi-class Lovasz-Softmax loss
629 | probas: [B, C, H, W] Variable, class probabilities at each prediction (between 0 and 1).
630 | Interpreted as binary (sigmoid) output with outputs of size [B, H, W].
631 | labels: [B, H, W] Tensor, ground truth labels (between 0 and C - 1)
632 | classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.
633 | per_image: compute the loss per image instead of per batch
634 | ignore: void class labels
635 | """
636 | if per_image:
637 | loss = mean(lovasz_softmax_flat(*flatten_probas(prob.unsqueeze(0), lab.unsqueeze(0), ignore), classes=classes)
638 | for prob, lab in zip(probas, labels))
639 | else:
640 | loss = lovasz_softmax_flat(*flatten_probas(probas, labels, ignore), classes=classes)
641 | return loss
642 |
643 |
644 | def lovasz_softmax_flat(probas, labels, classes='present'):
645 | """
646 | Multi-class Lovasz-Softmax loss
647 | probas: [P, C] Variable, class probabilities at each prediction (between 0 and 1)
648 | labels: [P] Tensor, ground truth labels (between 0 and C - 1)
649 | classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.
650 | """
651 | if probas.numel() == 0:
652 | # only void pixels, the gradients should be 0
653 | return probas * 0.
654 | C = probas.size(1)
655 | losses = []
656 | class_to_sum = list(range(C)) if classes in ['all', 'present'] else classes
657 | for c in class_to_sum:
658 | fg = (labels == c).float() # foreground for class c
659 | if (classes is 'present' and fg.sum() == 0):
660 | continue
661 | if C == 1:
662 | if len(classes) > 1:
663 | raise ValueError('Sigmoid output possible only with 1 class')
664 | class_pred = probas[:, 0]
665 | else:
666 | class_pred = probas[:, c]
667 | errors = (Variable(fg) - class_pred).abs()
668 | errors_sorted, perm = torch.sort(errors, 0, descending=True)
669 | perm = perm.data
670 | fg_sorted = fg[perm]
671 | losses.append(torch.dot(errors_sorted, Variable(lovasz_grad(fg_sorted))))
672 | return mean(losses)
673 |
674 |
675 | def flatten_probas(probas, labels, ignore=None):
676 | """
677 | Flattens predictions in the batch
678 | """
679 | if probas.dim() == 3:
680 | # assumes output of a sigmoid layer
681 | B, H, W = probas.size()
682 | probas = probas.view(B, 1, H, W)
683 | B, C, H, W = probas.size()
684 | probas = probas.permute(0, 2, 3, 1).contiguous().view(-1, C) # B * H * W, C = P, C
685 | labels = labels.view(-1)
686 | if ignore is None:
687 | return probas, labels
688 | valid = (labels != ignore)
689 | vprobas = probas[valid.nonzero().squeeze()]
690 | vlabels = labels[valid]
691 | return vprobas, vlabels
692 |
693 | def xloss(logits, labels, ignore=None):
694 | """
695 | Cross entropy loss
696 | """
697 | return F.cross_entropy(logits, Variable(labels), ignore_index=255)
698 |
699 | # --------------------------- HELPER FUNCTIONS ---------------------------
700 | def isnan(x):
701 | return x != x
702 |
703 |
704 | def mean(l, ignore_nan=False, empty=0):
705 | """
706 | nanmean compatible with generators.
707 | """
708 | l = iter(l)
709 | if ignore_nan:
710 | l = ifilterfalse(isnan, l)
711 | try:
712 | n = 1
713 | acc = next(l)
714 | except StopIteration:
715 | if empty == 'raise':
716 | raise ValueError('Empty mean')
717 | return empty
718 | for n, v in enumerate(l, 2):
719 | acc += v
720 | if n == 1:
721 | return acc
722 | return acc / n
723 |
724 |
725 | def lovasz_grad(gt_sorted):
726 | """
727 | Computes gradient of the Lovasz extension w.r.t sorted errors
728 | See Alg. 1 in paper
729 | """
730 | p = len(gt_sorted)
731 | gts = gt_sorted.sum()
732 | intersection = gts - gt_sorted.float().cumsum(0)
733 | union = gts + (1 - gt_sorted).float().cumsum(0)
734 | jaccard = 1. - intersection / union
735 | if p > 1: # cover 1-pixel case
736 | jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]
737 | return jaccard
738 | ```
739 |
740 | ```python
741 | #Keras
742 | # not working yet
743 | # def LovaszHingeLoss(inputs, targets):
744 | # return lovasz_hinge_loss(inputs, targets)
745 | ```
746 |
747 | ## Combo Loss
748 | This loss was introduced by Taghanaki et al in their paper "Combo loss: Handling input and output imbalance in multi-organ segmentation", retrievable here: https://arxiv.org/abs/1805.02798. Combo loss is a combination of Dice Loss and a modified Cross-Entropy function that, like Tversky loss, has additional constants which penalise either false positives or false negatives more respectively.
749 |
750 | ```python
751 | #PyTorch
752 | ALPHA = 0.5 # < 0.5 penalises FP more, > 0.5 penalises FN more
753 | CE_RATIO = 0.5 #weighted contribution of modified CE loss compared to Dice loss
754 |
755 | class ComboLoss(nn.Module):
756 | def __init__(self, weight=None, size_average=True):
757 | super(ComboLoss, self).__init__()
758 |
759 | def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA):
760 |
761 | #flatten label and prediction tensors
762 | inputs = inputs.view(-1)
763 | targets = targets.view(-1)
764 |
765 | #True Positives, False Positives & False Negatives
766 | intersection = (inputs * targets).sum()
767 | dice = (2. * intersection + smooth) / (inputs.sum() + targets.sum() + smooth)
768 |
769 | inputs = torch.clamp(inputs, e, 1.0 - e)
770 | out = - (ALPHA * ((targets * torch.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * torch.log(1.0 - inputs))))
771 | weighted_ce = out.mean(-1)
772 | combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)
773 |
774 | return combo
775 | ```
776 |
777 | ```python
778 | #Tensorflow / Keras
779 | def Combo_loss(y_true, y_pred, smooth=1):
780 |
781 | e = K.epsilon()
782 | if y_pred.shape[-1] <= 1:
783 | ALPHA = 0.8 # < 0.5 penalises FP more, > 0.5 penalises FN more
784 | CE_RATIO = 0.5 # weighted contribution of modified CE loss compared to Dice loss
785 | y_pred = tf.keras.activations.sigmoid(y_pred)
786 | elif y_pred.shape[-1] >= 2:
787 | ALPHA = 0.3 # < 0.5 penalises FP more, > 0.5 penalises FN more
788 | CE_RATIO = 0.7 # weighted contribution of modified CE loss compared to Dice loss
789 | y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
790 | y_true = K.squeeze(y_true, 3)
791 | y_true = tf.cast(y_true, "int32")
792 | y_true = tf.one_hot(y_true, num_class, axis=-1)
793 |
794 | # cast to float32 datatype
795 | y_true = K.cast(y_true, 'float32')
796 | y_pred = K.cast(y_pred, 'float32')
797 |
798 | targets = K.flatten(y_true)
799 | inputs = K.flatten(y_pred)
800 |
801 | intersection = K.sum(targets * inputs)
802 | dice = (2. * intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
803 | inputs = K.clip(inputs, e, 1.0 - e)
804 | out = - (ALPHA * ((targets * K.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * K.log(1.0 - inputs))))
805 | weighted_ce = K.mean(out, axis=-1)
806 | combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)
807 |
808 | return combo
809 | ```
810 |
811 | ### Usage
812 | Some tips
813 | * Tversky and Focal-Tversky loss benefit from very low learning rates, of the order 5e-5 to 1e-4. They would not see much improvement in my kernels until around 7-10 epochs, upon which performance would improve significantly.
814 |
815 | * In general, if a loss function does not appear to be working well (or at all), experiment with modifying the learning rate before moving on to other options.
816 |
817 | * You can easily create your own loss functions by combining any of the above with Binary Cross-Entropy or any combination of other losses. Bear in mind that loss is calculated for every batch, so more complex losses will increase runtime.
818 |
819 | * Care must be taken when writing loss functions for PyTorch. If you call a function to modify the inputs that doesn't entirely use PyTorch's numerical methods, the tensor will 'detach' from the the graph that maps it back through the neural network for the purposes of backpropagation, making the loss function unusable. Discussion of this is available [here](https://discuss.pytorch.org/t/some-problems-in-custom-loss-functions-and-so-on/36618).
820 |
821 |
822 | #### Refernces
823 |
824 | [RNA Kaggle](https://www.kaggle.com/bigironsphere/loss-function-library-keras-pytorch)
825 |
--------------------------------------------------------------------------------