├── README.md ├── LICENSE ├── stats_loss_testing_file.py ├── pytorch_stats_loss.py └── Pytorch_Statistical_Losses_Combined.py /README.md: -------------------------------------------------------------------------------- 1 | ## 1D WASSERSTEIN STATISTICAL DISTANCE LOSSES IN PYTORCH 2 | 3 |   4 | 5 | ### Introduction: 6 | This repository is created to provide a **Pytorch Wasserstein Statistical Loss** solution for a pair of 1D weight distributions. 7 | 8 |   9 | 10 | ### How To: 11 | All core functions of this repository are created in **pytorch_stats_loss.py**. To introduce the related Pytorch losses, just add this file into your project and import it at your wish. 12 | A group of dependent examples of related functionalities could be found in **stats_loss_testing_file.py**. 13 | **Pytorch_Statistical_Losses_Combined.py** makes a combination of the loss functions and their examples, and provides a "one click and run" program for the convinence of interested users. 14 | **pytorch_stats_loss.py** should be regarded as the center file of this project. 15 | 16 |   17 | 18 | ### Points of Background Information: 19 | Statistial Distances for 1D weight distributions 20 | Inspired by Scipy.Stats Statistial Distances for 1D distributions 21 | **Pytorch Version, supporting Autograd to make a valid Loss for deep learning** 22 | Supposing Inputs are Groups of Same-Length Weight Vectors 23 | Instead of (Points, Weight), full-length Weight Vectors are taken as Inputs 24 | **Losses are built up based on the result of CDF calculations** 25 | 26 |   27 | 28 | ### If you want to know more: 29 | Check Scipy.Stats module for more background knowledge. 30 | Check Pytorch to know more about deep learning. 31 | 32 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2020, Takara_Research 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /stats_loss_testing_file.py: -------------------------------------------------------------------------------- 1 | ############################################################# 2 | # Testing Script for Statistical Loss Function in Pytorch # 3 | ############################################################# 4 | #-----------------------------------------------------------# 5 | # This is a testing script for pytorch statistical loss function 6 | # pytorch_stats_loss.py should be in the same folder with this testing script 7 | 8 | import numpy as np 9 | from scipy import stats 10 | import torch 11 | from torch.autograd import Variable 12 | import pytorch_stats_loss as stats_loss 13 | # pytorch_stats_loss.py is the script for loss definitions 14 | 15 | # Testing Script for Statistical Loss Function in Pytorch 16 | 17 | BATCH = 16 18 | DIM = 32 19 | 20 | # Making Data for Testing 21 | vec_1 = np.random.random((BATCH,DIM)) 22 | vec_2 = np.random.random((BATCH,DIM)) 23 | vec_list = np.arange(DIM) 24 | 25 | # Making Scipy Numpy Results 26 | result_1=0 27 | result_2=0 28 | for i in range(BATCH): 29 | vec_dist_1 = stats.wasserstein_distance(vec_list, vec_list, vec_1[i], vec_2[i]) 30 | vec_dist_2 = stats.energy_distance(vec_list,vec_list,vec_1[i],vec_2[i]) 31 | result_1 += vec_dist_1 32 | result_2 += vec_dist_2 33 | print("Numpy-Based Scipy Results: \n", 34 | "Wasserstein distance",result_1/BATCH,"\n", 35 | "Energy distance",result_2/BATCH,"\n") 36 | 37 | # Making Pytorch Variable Calculations 38 | tensor_1=Variable(torch.from_numpy(vec_1)) 39 | tensor_2=Variable(torch.from_numpy(vec_2),requires_grad=True) 40 | tensor_3=Variable(torch.rand(BATCH+1,DIM)) 41 | 42 | # Show results 43 | print("Pytorch-Based Results:") 44 | print("Wasserstein loss",stats_loss.torch_wasserstein_loss(tensor_1,tensor_2).data,stats_loss.torch_wasserstein_loss(tensor_1,tensor_2).requires_grad) 45 | print("Energy loss",stats_loss.torch_energy_loss(tensor_1,tensor_2).data,stats_loss.torch_wasserstein_loss(tensor_1,tensor_2).requires_grad) 46 | print("p == 1.5 CDF loss", stats_loss.torch_cdf_loss(tensor_1,tensor_2,p=1.5).data) 47 | print("Validate Checking Errors:", stats_loss.torch_validate_distibution(tensor_1,tensor_2)) 48 | #torch_validate_distibution(tensor_1,tensor_3) -------------------------------------------------------------------------------- /pytorch_stats_loss.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.autograd import Variable 3 | 4 | 5 | ####################################################### 6 | # STATISTICAL DISTANCES(LOSSES) IN PYTORCH # 7 | ####################################################### 8 | 9 | ## Statistial Distances for 1D weight distributions 10 | ## Inspired by Scipy.Stats Statistial Distances for 1D 11 | ## Pytorch Version, supporting Autograd to make a valid Loss 12 | ## Supposing Inputs are Groups of Same-Length Weight Vectors 13 | ## Instead of (Points, Weight), full-length Weight Vectors are taken as Inputs 14 | ## Code Written by E.Bao, CASIA 15 | 16 | def torch_wasserstein_loss(tensor_a,tensor_b): 17 | #Compute the first Wasserstein distance between two 1D distributions. 18 | return(torch_cdf_loss(tensor_a,tensor_b,p=1)) 19 | 20 | def torch_energy_loss(tensor_a,tensor_b): 21 | # Compute the energy distance between two 1D distributions. 22 | return((2**0.5)*torch_cdf_loss(tensor_a,tensor_b,p=2)) 23 | 24 | def torch_cdf_loss(tensor_a,tensor_b,p=1): 25 | # last-dimension is weight distribution 26 | # p is the norm of the distance, p=1 --> First Wasserstein Distance 27 | # to get a positive weight with our normalized distribution 28 | # we recommend combining this loss with other difference-based losses like L1 29 | 30 | # normalize distribution, add 1e-14 to divisor to avoid 0/0 31 | tensor_a = tensor_a / (torch.sum(tensor_a, dim=-1, keepdim=True) + 1e-14) 32 | tensor_b = tensor_b / (torch.sum(tensor_b, dim=-1, keepdim=True) + 1e-14) 33 | # make cdf with cumsum 34 | cdf_tensor_a = torch.cumsum(tensor_a,dim=-1) 35 | cdf_tensor_b = torch.cumsum(tensor_b,dim=-1) 36 | 37 | # choose different formulas for different norm situations 38 | if p == 1: 39 | cdf_distance = torch.sum(torch.abs((cdf_tensor_a-cdf_tensor_b)),dim=-1) 40 | elif p == 2: 41 | cdf_distance = torch.sqrt(torch.sum(torch.pow((cdf_tensor_a-cdf_tensor_b),2),dim=-1)) 42 | else: 43 | cdf_distance = torch.pow(torch.sum(torch.pow(torch.abs(cdf_tensor_a-cdf_tensor_b),p),dim=-1),1/p) 44 | 45 | cdf_loss = cdf_distance.mean() 46 | return cdf_loss 47 | 48 | def torch_validate_distibution(tensor_a,tensor_b): 49 | # Zero sized dimension is not supported by pytorch, we suppose there is no empty inputs 50 | # Weights should be non-negetive, and with a positive and finite sum 51 | # We suppose all conditions will be corrected by network training 52 | # We only check the match of the size here 53 | if tensor_a.size() != tensor_b.size(): 54 | raise ValueError("Input weight tensors must be of the same size") 55 | -------------------------------------------------------------------------------- /Pytorch_Statistical_Losses_Combined.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy import stats 3 | import torch 4 | from torch.autograd import Variable 5 | 6 | #Testing Script for Statistical Loss Function 7 | 8 | BATCH = 4 9 | DIM = 6 10 | 11 | vec_1 = np.random.random((BATCH,DIM)) 12 | vec_2 = np.random.random((BATCH,DIM)) 13 | vec_list = np.arange(DIM) 14 | 15 | # Making Scipy Results 16 | result_1=0 17 | result_2=0 18 | for i in range(BATCH): 19 | vec_dist_1 = stats.wasserstein_distance(vec_list, vec_list, vec_1[i], vec_2[i]) 20 | vec_dist_2 = stats.energy_distance(vec_list,vec_list,vec_1[i],vec_2[i]) 21 | result_1 += vec_dist_1 22 | result_2 += vec_dist_2 23 | print("Numpy-Based Scipy Results: \n", 24 | "Wasserstein distance",result_1/BATCH,"\n", 25 | "Energy distance",result_2/BATCH,"\n") 26 | 27 | 28 | tensor_1=Variable(torch.from_numpy(vec_1)) 29 | tensor_2=Variable(torch.from_numpy(vec_2),requires_grad=True) 30 | tensor_3=Variable(torch.rand(BATCH+1,DIM)) 31 | 32 | 33 | ####################################################### 34 | # STATISTICAL DISTANCES(LOSSES) IN PYTORCH # 35 | ####################################################### 36 | 37 | ## Statistial Distances for 1D weight distributions 38 | ## Inspired by Scipy.Stats Statistial Distances for 1D 39 | ## Pytorch Version, supporting Autograd to make a valid Loss 40 | ## Supposing Inputs are Groups of Same-Length Weight Vectors 41 | ## Instead of (Points, Weight), full-length Weight Vectors are taken as Inputs 42 | ## Code Written by E.Bao, CASIA 43 | 44 | 45 | def torch_wasserstein_loss(tensor_a,tensor_b): 46 | #Compute the first Wasserstein distance between two 1D distributions. 47 | return(torch_cdf_loss(tensor_a,tensor_b,p=1)) 48 | 49 | def torch_energy_loss(tensor_a,tensor_b): 50 | # Compute the energy distance between two 1D distributions. 51 | return((2**0.5)*torch_cdf_loss(tensor_a,tensor_b,p=2)) 52 | 53 | def torch_cdf_loss(tensor_a,tensor_b,p=1): 54 | # last-dimension is weight distribution 55 | # p is the norm of the distance, p=1 --> First Wasserstein Distance 56 | # to get a positive weight with our normalized distribution 57 | # we recommend combining this loss with other difference-based losses like L1 58 | 59 | # normalize distribution, add 1e-14 to divisor to avoid 0/0 60 | tensor_a = tensor_a / (torch.sum(tensor_a, dim=-1, keepdim=True) + 1e-14) 61 | tensor_b = tensor_b / (torch.sum(tensor_b, dim=-1, keepdim=True) + 1e-14) 62 | # make cdf with cumsum 63 | cdf_tensor_a = torch.cumsum(tensor_a,dim=-1) 64 | cdf_tensor_b = torch.cumsum(tensor_b,dim=-1) 65 | 66 | # choose different formulas for different norm situations 67 | if p == 1: 68 | cdf_distance = torch.sum(torch.abs((cdf_tensor_a-cdf_tensor_b)),dim=-1) 69 | elif p == 2: 70 | cdf_distance = torch.sqrt(torch.sum(torch.pow((cdf_tensor_a-cdf_tensor_b),2),dim=-1)) 71 | else: 72 | cdf_distance = torch.pow(torch.sum(torch.pow(torch.abs(cdf_tensor_a-cdf_tensor_b),p),dim=-1),1/p) 73 | 74 | cdf_loss = cdf_distance.mean() 75 | return cdf_loss 76 | 77 | def torch_validate_distibution(tensor_a,tensor_b): 78 | # Zero sized dimension is not supported by pytorch, we suppose there is no empty inputs 79 | # Weights should be non-negetive, and with a positive and finite sum 80 | # We suppose all conditions will be corrected by network training 81 | # We only check the match of the size here 82 | if tensor_a.size() != tensor_b.size(): 83 | raise ValueError("Input weight tensors must be of the same size") 84 | 85 | print("Pytorch-Based Results:") 86 | print("Wasserstein loss",torch_wasserstein_loss(tensor_1,tensor_2).data,torch_wasserstein_loss(tensor_1,tensor_2).requires_grad) 87 | print("Energy loss",torch_energy_loss(tensor_1,tensor_2).data,torch_wasserstein_loss(tensor_1,tensor_2).requires_grad) 88 | print("p == 1.5 CDF loss", torch_cdf_loss(tensor_1,tensor_2,p=1.5).data) 89 | print("Validate Checking Errors:", torch_validate_distibution(tensor_1,tensor_2)) 90 | #torch_validate_distibution(tensor_1,tensor_3) 91 | 92 | 93 | --------------------------------------------------------------------------------