├── .github
    ├── ISSUE_TEMPLATE
    │   └── pr-checklist.md
    └── PULL_REQUEST_TEMPLATE
├── README.md
├── data_loader.py
├── functions.py
├── models.py
├── results
    ├── 43051.jpg
    ├── 43051_meteorshowers.png
    ├── 43051_ours.png
    ├── 8068.jpg
    ├── 8068_meteorshowers.png
    └── 8068_ours.png
├── sgd-0-log.txt
├── train_RCF.py
└── utils.py


/.github/ISSUE_TEMPLATE/pr-checklist.md:
--------------------------------------------------------------------------------
 1 | ## Things to verify before requesting a review
 2 | 
 3 | - [ ] Kindly run `make init` to ensure that the environment is updated.
 4 | - [ ] Kindly merge with the master branch or main branch and resolve conflicts.
 5 | - [ ] Kindly request review from team members whose step/functionality is impacted.
 6 |   - Ex: Ramya owns the Preprocessing step, it would be good to have her as part of the review process for the Preprocessing step.
 7 | - [ ] Kindly include me in all reviews.
 8 | - [ ] Kindly add a detailed description about what functionalities are newly added in the PR.
 9 | - [ ] Kindly add the steps that you did to execute this piece of work.
10 | 


--------------------------------------------------------------------------------
/.github/PULL_REQUEST_TEMPLATE:
--------------------------------------------------------------------------------
 1 | # Pull Request Template
 2 | 
 3 | ## Description
 4 | Please include a summary of the changes and the related issue. Provide any other relevant context.
 5 | 
 6 | ## Type of Change
 7 | - [ ] Bug fix
 8 | - [ ] New feature
 9 | - [ ] Breaking change
10 | - [ ] Documentation update
11 | 
12 | ## How Has This Been Tested?
13 | Describe the tests that you have conducted to verify your changes. Provide clear instructions to reproduce.
14 | 
15 | ## Checklist
16 | - [ ] Code follows the project style guidelines
17 | - [ ] Self-review of the code has been done
18 | - [ ] Changes have been documented
19 | - [ ] Testing has been performed
20 | - [ ] All necessary documentation is updated
21 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # RCF_Pytorch_Updated
  2 | Pytorch re-implementation of RCF. This PyTorch implementation is based on awesome implementation by XuanyiLi.
  3 | There were minor performance fixes made in this implementation.
  4 | 
  5 | Major Changes are as follows
  6 | 1. Fixing the weights of the upsampling layer [based on work by Weijian Xu for PyTorch HED  implementation]
  7 |    
  8 |         Initializing the value of upsampling layer each time during training would make the training less stable. In this repo, the upsampling layer weights get initialized only during the model initialization step.
  9 | 
 10 | 2. Setting the parameters of weight_deconv5 as shown below [based on Original RCF Implementation]
 11 |         
 12 |         # self.weight_deconv5 =  make_bilinear_weights(32, 1).cuda()   
 13 |         self.weight_deconv5 =  make_bilinear_weights(16, 1).cuda()
 14 |         
 15 | 3. Setting the initial value of fuse layer as 0.20 [based on Original RCF Implementation]
 16 | 4. Modified the crop layer [based on BDCN implementation]
 17 | 
 18 | The above mentioned changes boosted the overall ODS and OIS compared to implementation by XuanyiLi.
 19 | 
 20 | 
 21 | 
 22 | ### Results on BSDS Dataset alone
 23 | 
 24 | | Method |ODS F-score on BSDS500 dataset |OIS F-score on BSDS500 dataset|
 25 | |:---|:---:|:---:|
 26 | |ours| 0.795 | 0.812 |
 27 | |meteorshowers[4]| 0.790 | 0.809 |
 28 | | Reference[1]| 0.798 | 0.815  |
 29 | 
 30 | 
 31 | 
 32 | ### Sample Output
 33 | 
 34 | From Left to Right  : 
 35 | 
 36 |         1. Original image
 37 |         
 38 |         2. Image output from meteorshowers
 39 |         
 40 |         3. Image output from our repo
 41 | 
 42 | <p float="left">
 43 |   <img src="/results/43051.jpg" width="250" />
 44 |   <img src="/results/43051_meteorshowers.png" width="250" />
 45 |   <img src="/results/43051_ours.png" width="250" />
 46 | </p>
 47 | 
 48 | 
 49 | <p float="left">
 50 |   <img src="/results/8068.jpg" width="250" />
 51 |   <img src="/results/8068_meteorshowers.png" width="250" />
 52 |   <img src="/results/8068_ours.png" width="250" />
 53 | </p>
 54 | 
 55 | 
 56 | As can be seen above, strong edges are detected properly with high confidence and less noise.
 57 | 
 58 | ### Dataset
 59 | To download dataset, kindly follow the procedure as mentioned in [1]
 60 | 
 61 | 
 62 | ### Usage
 63 | 
 64 | #### Edge Detection Pipeline
 65 | 
 66 | To download the vgg16 pretrained backbone model. please click
 67 | 
 68 | https://drive.google.com/file/d/1lUhPKKj-BSOH7yQL0mOIavvrUbjydPp5/view?usp=sharing
 69 | 
 70 | To train a RCF model on BSDS500:
 71 | 
 72 |         python train_RCF.py
 73 | 
 74 | If you have multiple GPUs on your machine, you can also run the multi-GPU version training:
 75 | 
 76 |         CUDA_VISIBLE_DEVICES=0,1 python train_multi_gpu.py --num_gpus 2
 77 | 
 78 | After training, to evaluate:
 79 | 
 80 |         We have used the pipeline suggested by [2]. It's very fast compared to the approach suggested in [1]
 81 |         The results would vary by about 0.001 due to usage of approach [2], due to different parameter values used in NMS estimation
 82 | 
 83 | 
 84 | ### To do
 85 | To reach the ODS and OIS score mentioned in the paper
 86 | 
 87 | 
 88 | ### Notes
 89 | Run for 10 epochs. That would be sufficient. You could get the mentioned score within 3-8 epochs itself.
 90 | 
 91 | The log file of our training  [sgd-0-log.txt] can be viewed to check whether your training is proceeding in similar way.
 92 | 
 93 | ### Acknowledgements:
 94 | 
 95 | [1] <a href="https://github.com/yun-liu/rcf">Original RCF Implementation</a> 
 96 | 
 97 | [2] <a href="https://github.com/xwjabc/hed">HED Implementation</a> 
 98 | 
 99 | [3] <a href="https://github.com/pkuCactus/BDCN">BDCN Implementation</a> 
100 | 
101 | [4] <a href="https://github.com/meteorshowers/RCF-pytorch">RCF PyTorch Implementation</a>
102 | 
103 | 
104 | 
105 | ### Citations
106 | 
107 | If you are using the code/model/data provided here in a publication, please consider citing original paper:
108 | 
109 |     @article{liu2019richer,
110 |       title={Richer Convolutional Features for Edge Detection},
111 |       author={Liu, Yun and Cheng, Ming-Ming and Hu, Xiaowei and Bian, Jia-Wang and Zhang, Le and Bai, Xiang and Tang, Jinhui},
112 |       journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
113 |       volume={41},
114 |       number={8},
115 |       pages={1939--1946},
116 |       year={2019},
117 |       publisher={IEEE}
118 |     }
119 | 
120 |     @inproceedings{liu2017richer,
121 |       title={Richer Convolutional Features for Edge Detection},
122 |       author={Liu, Yun and Cheng, Ming-Ming and Hu, Xiaowei and Wang, Kai and Bai, Xiang},
123 |       booktitle={IEEE conference on computer vision and pattern recognition (CVPR)},
124 |       pages={3000--3009},
125 |       year={2017}
126 |     }
127 | 


--------------------------------------------------------------------------------
/data_loader.py:
--------------------------------------------------------------------------------
 1 | from torch.utils import data
 2 | import os
 3 | from os.path import join, abspath, splitext, split, isdir, isfile
 4 | from PIL import Image
 5 | import numpy as np
 6 | import cv2
 7 | 
 8 | def prepare_image_PIL(im):
 9 |     im = im[:,:,::-1] - np.zeros_like(im) # rgb to bgr
10 |     im -= np.array((104.00698793,116.66876762,122.67891434))
11 |     im = np.transpose(im, (2, 0, 1)) # (H x W x C) to (C x H x W)
12 |     return im
13 | 
14 | def prepare_image_cv2(im):
15 |     im -= np.array((104.00698793,116.66876762,122.67891434))
16 |     im = np.transpose(im, (2, 0, 1)) # (H x W x C) to (C x H x W)
17 |     return im
18 | 
19 | 
20 | class BSDS_RCFLoader(data.Dataset):
21 |     """
22 |     Dataloader BSDS500
23 |     """
24 |     def __init__(self, root='data/HED-BSDS_PASCAL', split='train', transform=False):
25 |         self.root = root
26 |         self.split = split
27 |         self.transform = transform
28 |         if self.split == 'train':
29 |             self.filelist = join(self.root, 'train_pair.lst')
30 |         elif self.split == 'test':
31 |             self.filelist = join(self.root, 'test.lst')
32 |         else:
33 |             raise ValueError("Invalid split type!")
34 |         with open(self.filelist, 'r') as f:
35 |             self.filelist = f.readlines()
36 | 
37 |     def __len__(self):
38 |         return len(self.filelist)
39 |     
40 |     def __getitem__(self, index):
41 |         if self.split == "train":
42 |             img_file, lb_file = self.filelist[index].split()
43 |             lb = np.array(Image.open(join(self.root, lb_file)), dtype=np.float32)
44 |             if lb.ndim == 3:
45 |                 lb = np.squeeze(lb[:, :, 0])
46 |             assert lb.ndim == 2
47 |             lb = lb[np.newaxis, :, :]
48 |             lb[lb == 0] = 0
49 |             lb[np.logical_and(lb>0, lb<127.5)] = 2
50 |             lb[lb >= 127.5] = 1
51 |         else:
52 |             img_file = self.filelist[index].rstrip()
53 | 
54 |         if self.split == "train":
55 |             img = np.array(cv2.imread(join(self.root, img_file)), dtype=np.float32)
56 |             img = prepare_image_cv2(img)
57 |             return img, lb
58 |         else:
59 |             img = np.array(Image.open(join(self.root, img_file)), dtype=np.float32)
60 |             img = prepare_image_PIL(img)
61 |             return img
62 | 
63 | 


--------------------------------------------------------------------------------
/functions.py:
--------------------------------------------------------------------------------
  1 | import numpy as np
  2 | import torch
  3 | from torch.optim.optimizer import Optimizer, required
  4 | 
  5 | def cross_entropy_loss_RCF(prediction, label):
  6 |     label = label.long()
  7 |     mask = label.float()
  8 |     num_positive = torch.sum((mask==1).float()).float()
  9 |     num_negative = torch.sum((mask==0).float()).float()
 10 | 
 11 |     mask[mask == 1] = 1.0 * num_negative / (num_positive + num_negative)
 12 |     mask[mask == 0] = 1.1 * num_positive / (num_positive + num_negative)
 13 |     mask[mask == 2] = 0
 14 |     cost = torch.nn.functional.binary_cross_entropy(
 15 |             prediction.float(),label.float(), weight=mask, reduce=False)
 16 |     return torch.sum(cost)    
 17 | 
 18 | class SGD_caffe(Optimizer):
 19 |     r"""Implements stochastic gradient descent (optionally with momentum).
 20 | 
 21 |     Nesterov momentum is based on the formula from
 22 |     `On the importance of initialization and momentum in deep learning`__.
 23 | 
 24 |     Args:
 25 |         params (iterable): iterable of parameters to optimize or dicts defining
 26 |             parameter groups
 27 |         lr (float): learning rate
 28 |         momentum (float, optional): momentum factor (default: 0)
 29 |         weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
 30 |         dampening (float, optional): dampening for momentum (default: 0)
 31 |         nesterov (bool, optional): enables Nesterov momentum (default: False)
 32 | 
 33 |     Example:
 34 |         >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
 35 |         >>> optimizer.zero_grad()
 36 |         >>> loss_fn(model(input), target).backward()
 37 |         >>> optimizer.step()
 38 | 
 39 |     __ http://www.cs.toronto.edu/%7Ehinton/absps/momentum.pdf
 40 | 
 41 |     .. note::
 42 |         The implementation of SGD with Momentum/Nesterov subtly differs from
 43 |         Sutskever et. al. and implementations in some other frameworks.
 44 | 
 45 |         Considering the specific case of Momentum, the update can be written as
 46 | 
 47 |         .. math::
 48 |                   v = \rho * v + g \\
 49 |                   p = p - lr * v
 50 | 
 51 |         where p, g, v and :math:`\rho` denote the parameters, gradient,
 52 |         velocity, and momentum respectively.
 53 | 
 54 |         This is in contrast to Sutskever et. al. and
 55 |         other frameworks which employ an update of the form
 56 | 
 57 |         .. math::
 58 |              v = \rho * v + lr * g \\
 59 |              p = p - v
 60 | 
 61 |         The Nesterov version is analogously modified.
 62 |     """
 63 | 
 64 |     def __init__(self, params, lr=required, momentum=0, dampening=0,
 65 |                  weight_decay=0, nesterov=False):
 66 |         if lr is not required and lr < 0.0:
 67 |             raise ValueError("Invalid learning rate: {}".format(lr))
 68 |         if momentum < 0.0:
 69 |             raise ValueError("Invalid momentum value: {}".format(momentum))
 70 |         if weight_decay < 0.0:
 71 |             raise ValueError("Invalid weight_decay value: {}".format(weight_decay))
 72 | 
 73 |         defaults = dict(lr=lr, momentum=momentum, dampening=dampening,
 74 |                         weight_decay=weight_decay, nesterov=nesterov)
 75 |         if nesterov and (momentum <= 0 or dampening != 0):
 76 |             raise ValueError("Nesterov momentum requires a momentum and zero dampening")
 77 |         super(SGD_caffe, self).__init__(params, defaults)
 78 | 
 79 |     def __setstate__(self, state):
 80 |         super(SGD_caffe, self).__setstate__(state)
 81 |         for group in self.param_groups:
 82 |             group.setdefault('nesterov', False)
 83 | 
 84 |     def step(self, closure=None):
 85 |         """Performs a single optimization step.
 86 | 
 87 |         Arguments:
 88 |             closure (callable, optional): A closure that reevaluates the model
 89 |                 and returns the loss.
 90 |         """
 91 |         loss = None
 92 |         if closure is not None:
 93 |             loss = closure()
 94 | 
 95 |         for group in self.param_groups:
 96 |             weight_decay = group['weight_decay']
 97 |             momentum = group['momentum']
 98 |             dampening = group['dampening']
 99 |             nesterov = group['nesterov']
100 | 
101 |             for p in group['params']:
102 |                 if p.grad is None:
103 |                     continue
104 |                 d_p = p.grad.data
105 |                 if weight_decay != 0:
106 |                     d_p.add_(weight_decay, p.data)
107 |                 if momentum != 0:
108 |                     param_state = self.state[p]
109 |                     if 'momentum_buffer' not in param_state:
110 |                         buf = param_state['momentum_buffer'] = torch.zeros_like(p.data)
111 |                         buf.mul_(momentum).add_(group['lr'], d_p)
112 |                     else:
113 |                         buf = param_state['momentum_buffer']
114 |                         buf.mul_(momentum).add_(group['lr'], d_p)
115 |                     if nesterov:
116 |                         d_p = d_p.add(momentum, buf)
117 |                     else:
118 |                         d_p = buf
119 | 
120 |                 p.data.add_(-1, d_p)
121 | 
122 |         return loss
123 | 


--------------------------------------------------------------------------------
/models.py:
--------------------------------------------------------------------------------
  1 | import os, sys
  2 | import torch
  3 | import torch.nn as nn
  4 | import torchvision.models as models
  5 | import torch.autograd.variable as Variable
  6 | import numpy as np
  7 | import scipy.io as sio
  8 | from torch.nn.modules.conv import _ConvNd
  9 | from torch.nn.modules.conv import _single, _pair, _triple
 10 | import torch.nn.functional as F
 11 | 
 12 | 
 13 | class RCF(nn.Module):
 14 |     def __init__(self):
 15 |         super(RCF, self).__init__()
 16 |         #lr 1 2 decay 1 0      
 17 |         self.conv1_1 = nn.Conv2d(3, 64, 3, padding=1)
 18 |         self.conv1_2 = nn.Conv2d(64, 64, 3, padding=1)
 19 | 
 20 |         self.conv2_1 = nn.Conv2d(64, 128, 3, padding=1)
 21 |         self.conv2_2 = nn.Conv2d(128, 128, 3, padding=1)
 22 | 
 23 |         self.conv3_1 = nn.Conv2d(128, 256, 3, padding=1)
 24 |         self.conv3_2 = nn.Conv2d(256, 256, 3, padding=1)
 25 |         self.conv3_3 = nn.Conv2d(256, 256, 3, padding=1)
 26 | 
 27 |         self.conv4_1 = nn.Conv2d(256, 512, 3, padding=1)
 28 |         self.conv4_2 = nn.Conv2d(512, 512, 3, padding=1)
 29 |         self.conv4_3 = nn.Conv2d(512, 512, 3, padding=1)
 30 | 
 31 |         self.conv5_1 = nn.Conv2d(512, 512, kernel_size=3,
 32 |                         stride=1, padding=2, dilation=2)
 33 |         self.conv5_2 = nn.Conv2d(512, 512, kernel_size=3,
 34 |                         stride=1, padding=2, dilation=2)
 35 |         self.conv5_3 = nn.Conv2d(512, 512, kernel_size=3,
 36 |                         stride=1, padding=2, dilation=2)
 37 | 
 38 |         self.activ =  nn.ReLU(inplace=True)
 39 | 
 40 |         self.maxpool4 = nn.MaxPool2d(2, stride=1, ceil_mode=True)
 41 |         self.maxpool_1  =  nn.MaxPool2d(2, stride=2, ceil_mode=True)
 42 |         self.maxpool_2  =  nn.MaxPool2d(2, stride=2, ceil_mode=True)
 43 |         self.maxpool_3  =  nn.MaxPool2d(2, stride=2, ceil_mode=True)
 44 | 
 45 |         #lr 0.1 0.2 decay 1 0
 46 |         self.conv1_1_down = nn.Conv2d(64, 21, 1, padding=0)
 47 |         self.conv1_2_down = nn.Conv2d(64, 21, 1, padding=0)
 48 | 
 49 |         self.conv2_1_down = nn.Conv2d(128, 21, 1, padding=0)
 50 |         self.conv2_2_down = nn.Conv2d(128, 21, 1, padding=0)
 51 | 
 52 |         self.conv3_1_down = nn.Conv2d(256, 21, 1, padding=0)
 53 |         self.conv3_2_down = nn.Conv2d(256, 21, 1, padding=0)
 54 |         self.conv3_3_down = nn.Conv2d(256, 21, 1, padding=0)
 55 | 
 56 |         self.conv4_1_down = nn.Conv2d(512, 21, 1, padding=0)
 57 |         self.conv4_2_down = nn.Conv2d(512, 21, 1, padding=0)
 58 |         self.conv4_3_down = nn.Conv2d(512, 21, 1, padding=0)
 59 |         
 60 |         self.conv5_1_down = nn.Conv2d(512, 21, 1, padding=0)
 61 |         self.conv5_2_down = nn.Conv2d(512, 21, 1, padding=0)
 62 |         self.conv5_3_down = nn.Conv2d(512, 21, 1, padding=0)
 63 | 
 64 |         #lr 0.01 0.02 decay 1 0
 65 |         self.score_dsn1 = nn.Conv2d(21, 1, 1)
 66 |         self.score_dsn2 = nn.Conv2d(21, 1, 1)
 67 |         self.score_dsn3 = nn.Conv2d(21, 1, 1)
 68 |         self.score_dsn4 = nn.Conv2d(21, 1, 1)
 69 |         self.score_dsn5 = nn.Conv2d(21, 1, 1)
 70 |         #lr 0.001 0.002 decay 1 0
 71 |         self.score_final = nn.Conv2d(5, 1, 1)
 72 | 
 73 |         ## Fixed the upsampling weights for the training process as per @https://github.com/xwjabc/hed
 74 |         self.weight_deconv2 =  make_bilinear_weights(4, 1).cuda()
 75 |         self.weight_deconv3 =  make_bilinear_weights(8, 1).cuda()
 76 |         self.weight_deconv4 =  make_bilinear_weights(16, 1).cuda()
 77 |         # Wrong Deconv Filter size. Updated from RCF yun_liu
 78 |         # self.weight_deconv5 =  make_bilinear_weights(32, 1).cuda()
 79 |         self.weight_deconv5 =  make_bilinear_weights(16, 1).cuda()
 80 | 
 81 |     def forward(self, x):
 82 |         # VGG
 83 |         img_H, img_W = x.shape[2], x.shape[3]
 84 |         conv1_1 = self.activ(self.conv1_1(x))
 85 |         conv1_2 = self.activ(self.conv1_2(conv1_1))
 86 |         pool1   = self.maxpool_1(conv1_2)
 87 | 
 88 |         conv2_1 = self.activ(self.conv2_1(pool1))
 89 |         conv2_2 = self.activ(self.conv2_2(conv2_1))
 90 |         pool2   = self.maxpool_2(conv2_2)
 91 | 
 92 |         conv3_1 = self.activ(self.conv3_1(pool2))
 93 |         conv3_2 = self.activ(self.conv3_2(conv3_1))
 94 |         conv3_3 = self.activ(self.conv3_3(conv3_2))
 95 |         pool3   = self.maxpool_3(conv3_3)
 96 | 
 97 |         conv4_1 = self.activ(self.conv4_1(pool3))
 98 |         conv4_2 = self.activ(self.conv4_2(conv4_1))
 99 |         conv4_3 = self.activ(self.conv4_3(conv4_2))
100 |         pool4   = self.maxpool4(conv4_3)
101 | 
102 |         conv5_1 = self.activ(self.conv5_1(pool4))
103 |         conv5_2 = self.activ(self.conv5_2(conv5_1))
104 |         conv5_3 = self.activ(self.conv5_3(conv5_2))
105 | 
106 |         conv1_1_down = self.conv1_1_down(conv1_1)
107 |         conv1_2_down = self.conv1_2_down(conv1_2)
108 |         conv2_1_down = self.conv2_1_down(conv2_1)
109 |         conv2_2_down = self.conv2_2_down(conv2_2)
110 |         conv3_1_down = self.conv3_1_down(conv3_1)
111 |         conv3_2_down = self.conv3_2_down(conv3_2)
112 |         conv3_3_down = self.conv3_3_down(conv3_3)
113 |         conv4_1_down = self.conv4_1_down(conv4_1)
114 |         conv4_2_down = self.conv4_2_down(conv4_2)
115 |         conv4_3_down = self.conv4_3_down(conv4_3)
116 |         conv5_1_down = self.conv5_1_down(conv5_1)
117 |         conv5_2_down = self.conv5_2_down(conv5_2)
118 |         conv5_3_down = self.conv5_3_down(conv5_3)
119 | 
120 |         so1_out = self.score_dsn1(conv1_1_down + conv1_2_down)
121 |         so2_out = self.score_dsn2(conv2_1_down + conv2_2_down)
122 |         so3_out = self.score_dsn3(conv3_1_down + conv3_2_down + conv3_3_down)
123 |         so4_out = self.score_dsn4(conv4_1_down + conv4_2_down + conv4_3_down)
124 |         so5_out = self.score_dsn5(conv5_1_down + conv5_2_down + conv5_3_down)
125 | 
126 |         upsample2 = torch.nn.functional.conv_transpose2d(so2_out, self.weight_deconv2, stride=2)
127 |         upsample3 = torch.nn.functional.conv_transpose2d(so3_out, self.weight_deconv3, stride=4)
128 |         upsample4 = torch.nn.functional.conv_transpose2d(so4_out, self.weight_deconv4, stride=8)
129 |         upsample5 = torch.nn.functional.conv_transpose2d(so5_out, self.weight_deconv5, stride=8)
130 |         
131 |         ### center crop
132 |         so1 = crop(so1_out, img_H, img_W, 0 , 0)
133 |         so2 = crop(upsample2, img_H, img_W , 1, 1 )
134 |         so3 = crop(upsample3, img_H, img_W , 2, 2 )
135 |         so4 = crop(upsample4, img_H, img_W , 4, 4)
136 |         so5 = crop(upsample5, img_H, img_W , 0, 0)
137 | 
138 |         fusecat = torch.cat((so1, so2, so3, so4, so5), dim=1)
139 |         fuse = self.score_final(fusecat)
140 |         results = [so1, so2, so3, so4, so5, fuse]
141 |         results = [torch.sigmoid(r) for r in results]
142 |         return results
143 | 
144 | # Based on BDCN Implementation @ https://github.com/pkuCactus/BDCN
145 | def crop(data1, h, w , crop_h, crop_w):
146 |     _, _, h1, w1 = data1.size()
147 |     assert(h <= h1 and w <= w1)
148 |     data = data1[:, :, crop_h:crop_h+h, crop_w:crop_w+w]
149 |     return data
150 | 
151 | def make_bilinear_weights(size, num_channels):
152 |     factor = (size + 1) // 2
153 |     if size % 2 == 1:
154 |         center = factor - 1
155 |     else:
156 |         center = factor - 0.5
157 |     og = np.ogrid[:size, :size]
158 |     filt = (1 - abs(og[0] - center) / factor) * (1 - abs(og[1] - center) / factor)
159 |     # print(filt)
160 |     filt = torch.from_numpy(filt)
161 |     w = torch.zeros(num_channels, num_channels, size, size)
162 |     w.requires_grad = False
163 |     for i in range(num_channels):
164 |         for j in range(num_channels):
165 |             if i == j:
166 |                 w[i, j] = filt
167 |     return w
168 | 
169 | def upsample(input, stride, num_channels=1):
170 |     kernel_size = stride * 2
171 |     kernel = make_bilinear_weights(kernel_size, num_channels).cuda()
172 |     return torch.nn.functional.conv_transpose2d(input, kernel, stride=stride)
173 | 


--------------------------------------------------------------------------------
/results/43051.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/43051.jpg


--------------------------------------------------------------------------------
/results/43051_meteorshowers.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/43051_meteorshowers.png


--------------------------------------------------------------------------------
/results/43051_ours.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/43051_ours.png


--------------------------------------------------------------------------------
/results/8068.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/8068.jpg


--------------------------------------------------------------------------------
/results/8068_meteorshowers.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/8068_meteorshowers.png


--------------------------------------------------------------------------------
/results/8068_ours.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/balajiselvaraj1601/RCF_Pytorch_Updated/6d21bc53fdda1e3ffcb5d51425a703c166827599/results/8068_ours.png


--------------------------------------------------------------------------------
/sgd-0-log.txt:
--------------------------------------------------------------------------------
  1 | Epoch: [0/30][0/28800] Time 0.242 (avg:0.242) Loss 659.617615 (avg:659.617615) 
  2 | Epoch: [0/30][1000/28800] Time 0.015 (avg:0.042) Loss 1.091052 (avg:247.256880) 
  3 | Epoch: [0/30][2000/28800] Time 0.032 (avg:0.043) Loss 213.841949 (avg:236.623477) 
  4 | Epoch: [0/30][3000/28800] Time 0.022 (avg:0.043) Loss 0.406745 (avg:225.637766) 
  5 | Epoch: [0/30][4000/28800] Time 0.032 (avg:0.043) Loss 114.511169 (avg:222.910042) 
  6 | Epoch: [0/30][5000/28800] Time 0.057 (avg:0.044) Loss 891.106079 (avg:219.486073) 
  7 | Epoch: [0/30][6000/28800] Time 0.035 (avg:0.044) Loss 138.956085 (avg:216.917924) 
  8 | Epoch: [0/30][7000/28800] Time 0.076 (avg:0.044) Loss 102.660843 (avg:215.089510) 
  9 | Epoch: [0/30][8000/28800] Time 0.033 (avg:0.044) Loss 129.993881 (avg:212.054536) 
 10 | Epoch: [0/30][9000/28800] Time 0.058 (avg:0.045) Loss 686.932800 (avg:210.090633) 
 11 | Epoch: [0/30][10000/28800] Time 0.014 (avg:0.045) Loss 12.060019 (avg:208.512276) 
 12 | Epoch: [0/30][11000/28800] Time 0.037 (avg:0.045) Loss 227.576279 (avg:207.665533) 
 13 | Epoch: [0/30][12000/28800] Time 0.026 (avg:0.045) Loss 0.428115 (avg:206.531110) 
 14 | Epoch: [0/30][13000/28800] Time 0.021 (avg:0.044) Loss 0.000000 (avg:204.154579) 
 15 | Epoch: [0/30][14000/28800] Time 0.075 (avg:0.044) Loss 247.017288 (avg:202.867508) 
 16 | Epoch: [0/30][15000/28800] Time 0.033 (avg:0.044) Loss 380.620453 (avg:201.577407) 
 17 | Epoch: [0/30][16000/28800] Time 0.063 (avg:0.044) Loss 222.053864 (avg:201.687163) 
 18 | Epoch: [0/30][17000/28800] Time 0.057 (avg:0.044) Loss 410.506165 (avg:201.101345) 
 19 | Epoch: [0/30][18000/28800] Time 0.015 (avg:0.044) Loss 17.961992 (avg:199.993459) 
 20 | Epoch: [0/30][19000/28800] Time 0.024 (avg:0.044) Loss 0.367837 (avg:199.087686) 
 21 | Epoch: [0/30][20000/28800] Time 0.119 (avg:0.044) Loss 423.319916 (avg:198.138197) 
 22 | Epoch: [0/30][21000/28800] Time 0.065 (avg:0.044) Loss 239.994736 (avg:197.415776) 
 23 | Epoch: [0/30][22000/28800] Time 0.013 (avg:0.044) Loss 1.937604 (avg:196.443025) 
 24 | Epoch: [0/30][23000/28800] Time 0.016 (avg:0.044) Loss 19.542868 (avg:195.861083) 
 25 | Epoch: [0/30][24000/28800] Time 0.032 (avg:0.044) Loss 266.476288 (avg:195.617260) 
 26 | Epoch: [0/30][25000/28800] Time 0.057 (avg:0.044) Loss 258.401489 (avg:194.828976) 
 27 | Epoch: [0/30][26000/28800] Time 0.031 (avg:0.044) Loss 301.813690 (avg:193.649964) 
 28 | Epoch: [0/30][27000/28800] Time 0.014 (avg:0.044) Loss 0.994663 (avg:192.888937) 
 29 | Epoch: [0/30][28000/28800] Time 0.014 (avg:0.044) Loss 3.026277 (avg:192.053858) 
 30 | Epoch: [1/30][0/28800] Time 0.177 (avg:0.177) Loss 1.066520 (avg:1.066520) 
 31 | Epoch: [1/30][1000/28800] Time 0.036 (avg:0.043) Loss 220.907104 (avg:180.973512) 
 32 | Epoch: [1/30][2000/28800] Time 0.064 (avg:0.042) Loss 68.850945 (avg:175.686026) 
 33 | Epoch: [1/30][3000/28800] Time 0.016 (avg:0.042) Loss 0.770612 (avg:172.737511) 
 34 | Epoch: [1/30][4000/28800] Time 0.075 (avg:0.043) Loss 116.654030 (avg:174.098601) 
 35 | Epoch: [1/30][5000/28800] Time 0.056 (avg:0.043) Loss 85.747810 (avg:175.288912) 
 36 | Epoch: [1/30][6000/28800] Time 0.041 (avg:0.042) Loss 200.032913 (avg:174.180242) 
 37 | Epoch: [1/30][7000/28800] Time 0.014 (avg:0.042) Loss 10.988256 (avg:171.730998) 
 38 | Epoch: [1/30][8000/28800] Time 0.016 (avg:0.042) Loss 7.046596 (avg:171.158601) 
 39 | Epoch: [1/30][9000/28800] Time 0.032 (avg:0.042) Loss 110.407410 (avg:172.798658) 
 40 | Epoch: [1/30][10000/28800] Time 0.015 (avg:0.042) Loss 13.129516 (avg:171.621092) 
 41 | Epoch: [1/30][11000/28800] Time 0.014 (avg:0.042) Loss 0.699045 (avg:173.037440) 
 42 | Epoch: [1/30][12000/28800] Time 0.031 (avg:0.042) Loss 376.635345 (avg:172.929699) 
 43 | Epoch: [1/30][13000/28800] Time 0.024 (avg:0.042) Loss 0.255425 (avg:172.912842) 
 44 | Epoch: [1/30][14000/28800] Time 0.036 (avg:0.042) Loss 113.670525 (avg:172.493613) 
 45 | Epoch: [1/30][15000/28800] Time 0.059 (avg:0.043) Loss 157.917328 (avg:173.737301) 
 46 | Epoch: [1/30][16000/28800] Time 0.015 (avg:0.042) Loss 1.583478 (avg:173.076851) 
 47 | Epoch: [1/30][17000/28800] Time 0.031 (avg:0.042) Loss 131.791168 (avg:172.213995) 
 48 | Epoch: [1/30][18000/28800] Time 0.015 (avg:0.042) Loss 1.970533 (avg:171.611332) 
 49 | Epoch: [1/30][19000/28800] Time 0.056 (avg:0.042) Loss 838.248047 (avg:171.386506) 
 50 | Epoch: [1/30][20000/28800] Time 0.065 (avg:0.042) Loss 125.642899 (avg:171.540818) 
 51 | Epoch: [1/30][21000/28800] Time 0.014 (avg:0.042) Loss 1.264588 (avg:170.795951) 
 52 | Epoch: [1/30][22000/28800] Time 0.064 (avg:0.043) Loss 83.850456 (avg:171.729029) 
 53 | Epoch: [1/30][23000/28800] Time 0.035 (avg:0.043) Loss 106.865639 (avg:171.322562) 
 54 | Epoch: [1/30][24000/28800] Time 0.036 (avg:0.043) Loss 413.928284 (avg:171.390660) 
 55 | Epoch: [1/30][25000/28800] Time 0.022 (avg:0.043) Loss 0.682184 (avg:171.142743) 
 56 | Epoch: [1/30][26000/28800] Time 0.022 (avg:0.043) Loss 0.812464 (avg:171.291137) 
 57 | Epoch: [1/30][27000/28800] Time 0.031 (avg:0.043) Loss 186.824127 (avg:171.136001) 
 58 | Epoch: [1/30][28000/28800] Time 0.014 (avg:0.043) Loss 0.411901 (avg:170.978973) 
 59 | Epoch: [2/30][0/28800] Time 0.172 (avg:0.172) Loss 0.000000 (avg:0.000000) 
 60 | Epoch: [2/30][1000/28800] Time 0.049 (avg:0.044) Loss 186.523163 (avg:168.629108) 
 61 | Epoch: [2/30][2000/28800] Time 0.049 (avg:0.042) Loss 101.177086 (avg:160.097500) 
 62 | Epoch: [2/30][3000/28800] Time 0.035 (avg:0.042) Loss 108.997643 (avg:157.235520) 
 63 | Epoch: [2/30][4000/28800] Time 0.042 (avg:0.042) Loss 321.805756 (avg:157.211134) 
 64 | Epoch: [2/30][5000/28800] Time 0.042 (avg:0.042) Loss 287.011597 (avg:158.730532) 
 65 | Epoch: [2/30][6000/28800] Time 0.120 (avg:0.042) Loss 255.364700 (avg:159.536984) 
 66 | Epoch: [2/30][7000/28800] Time 0.056 (avg:0.042) Loss 426.674316 (avg:161.047257) 
 67 | Epoch: [2/30][8000/28800] Time 0.023 (avg:0.042) Loss 0.000000 (avg:162.089389) 
 68 | Epoch: [2/30][9000/28800] Time 0.087 (avg:0.042) Loss 505.905609 (avg:163.107473) 
 69 | Epoch: [2/30][10000/28800] Time 0.036 (avg:0.043) Loss 202.474930 (avg:163.431300) 
 70 | Epoch: [2/30][11000/28800] Time 0.033 (avg:0.043) Loss 170.717346 (avg:162.605800) 
 71 | Epoch: [2/30][12000/28800] Time 0.037 (avg:0.043) Loss 281.022980 (avg:161.870717) 
 72 | Epoch: [2/30][13000/28800] Time 0.048 (avg:0.043) Loss 255.065308 (avg:161.907090) 
 73 | Epoch: [2/30][14000/28800] Time 0.087 (avg:0.043) Loss 176.589981 (avg:162.488952) 
 74 | Epoch: [2/30][15000/28800] Time 0.014 (avg:0.043) Loss 4.899400 (avg:163.677783) 
 75 | Epoch: [2/30][16000/28800] Time 0.013 (avg:0.043) Loss 4.343246 (avg:163.577525) 
 76 | Epoch: [2/30][17000/28800] Time 0.014 (avg:0.043) Loss 16.766474 (avg:163.104783) 
 77 | Epoch: [2/30][18000/28800] Time 0.035 (avg:0.043) Loss 37.025738 (avg:162.851683) 
 78 | Epoch: [2/30][19000/28800] Time 0.065 (avg:0.043) Loss 238.615555 (avg:163.307841) 
 79 | Epoch: [2/30][20000/28800] Time 0.052 (avg:0.043) Loss 578.762878 (avg:162.891985) 
 80 | Epoch: [2/30][21000/28800] Time 0.032 (avg:0.043) Loss 104.864639 (avg:163.079996) 
 81 | Epoch: [2/30][22000/28800] Time 0.056 (avg:0.043) Loss 241.037796 (avg:163.106993) 
 82 | Epoch: [2/30][23000/28800] Time 0.036 (avg:0.043) Loss 121.762352 (avg:162.683356) 
 83 | Epoch: [2/30][24000/28800] Time 0.031 (avg:0.043) Loss 159.760437 (avg:162.307596) 
 84 | Epoch: [2/30][25000/28800] Time 0.015 (avg:0.043) Loss 2.396573 (avg:161.930971) 
 85 | Epoch: [2/30][26000/28800] Time 0.016 (avg:0.043) Loss 17.902245 (avg:161.744600) 
 86 | Epoch: [2/30][27000/28800] Time 0.087 (avg:0.043) Loss 805.128540 (avg:161.729340) 
 87 | Epoch: [2/30][28000/28800] Time 0.032 (avg:0.043) Loss 214.010422 (avg:161.123785) 
 88 | Epoch: [3/30][0/28800] Time 0.173 (avg:0.173) Loss 4.829380 (avg:4.829380) 
 89 | Epoch: [3/30][1000/28800] Time 0.021 (avg:0.044) Loss 0.000000 (avg:170.511660) 
 90 | Epoch: [3/30][2000/28800] Time 0.059 (avg:0.043) Loss 87.426689 (avg:153.615781) 
 91 | Epoch: [3/30][3000/28800] Time 0.036 (avg:0.043) Loss 116.952538 (avg:152.688472) 
 92 | Epoch: [3/30][4000/28800] Time 0.056 (avg:0.043) Loss 59.627083 (avg:155.605617) 
 93 | Epoch: [3/30][5000/28800] Time 0.057 (avg:0.043) Loss 147.131653 (avg:154.940550) 
 94 | Epoch: [3/30][6000/28800] Time 0.065 (avg:0.043) Loss 121.371178 (avg:154.354279) 
 95 | Epoch: [3/30][7000/28800] Time 0.014 (avg:0.043) Loss 0.000000 (avg:153.737176) 
 96 | Epoch: [3/30][8000/28800] Time 0.052 (avg:0.043) Loss 391.691193 (avg:153.616634) 
 97 | Epoch: [3/30][9000/28800] Time 0.018 (avg:0.043) Loss 3.669242 (avg:153.054551) 
 98 | Epoch: [3/30][10000/28800] Time 0.026 (avg:0.043) Loss 0.229994 (avg:154.360975) 
 99 | Epoch: [3/30][11000/28800] Time 0.031 (avg:0.043) Loss 369.001587 (avg:154.175349) 
100 | Epoch: [3/30][12000/28800] Time 0.014 (avg:0.043) Loss 2.747590 (avg:153.254340) 
101 | Epoch: [3/30][13000/28800] Time 0.076 (avg:0.043) Loss 112.431763 (avg:153.048555) 
102 | Epoch: [3/30][14000/28800] Time 0.057 (avg:0.043) Loss 107.985619 (avg:152.856315) 
103 | Epoch: [3/30][15000/28800] Time 0.056 (avg:0.043) Loss 141.955063 (avg:152.888933) 
104 | Epoch: [3/30][16000/28800] Time 0.031 (avg:0.043) Loss 122.879433 (avg:153.183586) 
105 | Epoch: [3/30][17000/28800] Time 0.052 (avg:0.043) Loss 247.930786 (avg:152.994933) 
106 | Epoch: [3/30][18000/28800] Time 0.050 (avg:0.043) Loss 197.689056 (avg:153.294607) 
107 | Epoch: [3/30][19000/28800] Time 0.042 (avg:0.043) Loss 285.385590 (avg:153.621778) 
108 | Epoch: [3/30][20000/28800] Time 0.031 (avg:0.043) Loss 163.882462 (avg:153.253161) 
109 | Epoch: [3/30][21000/28800] Time 0.014 (avg:0.043) Loss 4.569673 (avg:153.152495) 
110 | Epoch: [3/30][22000/28800] Time 0.066 (avg:0.043) Loss 631.365356 (avg:153.163109) 
111 | Epoch: [3/30][23000/28800] Time 0.015 (avg:0.043) Loss 2.708856 (avg:153.106605) 
112 | Epoch: [3/30][24000/28800] Time 0.056 (avg:0.043) Loss 308.955902 (avg:152.335462) 
113 | Epoch: [3/30][25000/28800] Time 0.064 (avg:0.043) Loss 171.953278 (avg:152.573734) 
114 | Epoch: [3/30][26000/28800] Time 0.066 (avg:0.043) Loss 542.520935 (avg:152.579935) 
115 | Epoch: [3/30][27000/28800] Time 0.014 (avg:0.043) Loss 3.017154 (avg:152.847884) 
116 | Epoch: [3/30][28000/28800] Time 0.087 (avg:0.043) Loss 435.909668 (avg:152.583704) 
117 | Epoch: [4/30][0/28800] Time 0.174 (avg:0.174) Loss 0.982344 (avg:0.982344) 
118 | Epoch: [4/30][1000/28800] Time 0.016 (avg:0.041) Loss 9.418994 (avg:143.789303) 
119 | Epoch: [4/30][2000/28800] Time 0.049 (avg:0.042) Loss 193.380005 (avg:148.420133) 
120 | Epoch: [4/30][3000/28800] Time 0.031 (avg:0.043) Loss 87.747650 (avg:149.309974) 
121 | Epoch: [4/30][4000/28800] Time 0.051 (avg:0.043) Loss 527.788513 (avg:152.476399) 
122 | Epoch: [4/30][5000/28800] Time 0.017 (avg:0.043) Loss 1.715785 (avg:152.566318) 
123 | Epoch: [4/30][6000/28800] Time 0.032 (avg:0.043) Loss 227.813385 (avg:152.988506) 
124 | Epoch: [4/30][7000/28800] Time 0.052 (avg:0.043) Loss 421.555176 (avg:154.446202) 
125 | Epoch: [4/30][8000/28800] Time 0.021 (avg:0.043) Loss 0.169067 (avg:154.362194) 
126 | Epoch: [4/30][9000/28800] Time 0.056 (avg:0.043) Loss 70.824638 (avg:154.223116) 
127 | Epoch: [4/30][10000/28800] Time 0.056 (avg:0.043) Loss 68.818848 (avg:154.506226) 
128 | Epoch: [4/30][11000/28800] Time 0.014 (avg:0.043) Loss 13.975116 (avg:154.000145) 
129 | Epoch: [4/30][12000/28800] Time 0.066 (avg:0.043) Loss 405.785797 (avg:153.170884) 
130 | Epoch: [4/30][13000/28800] Time 0.058 (avg:0.043) Loss 229.945465 (avg:153.403125) 
131 | Epoch: [4/30][14000/28800] Time 0.014 (avg:0.043) Loss 0.775555 (avg:153.479509) 
132 | Epoch: [4/30][15000/28800] Time 0.027 (avg:0.043) Loss 1.350904 (avg:153.368176) 
133 | Epoch: [4/30][16000/28800] Time 0.036 (avg:0.043) Loss 142.193100 (avg:153.316420) 
134 | Epoch: [4/30][17000/28800] Time 0.066 (avg:0.043) Loss 156.336945 (avg:153.174035) 
135 | Epoch: [4/30][18000/28800] Time 0.025 (avg:0.043) Loss 1.343908 (avg:152.646319) 
136 | Epoch: [4/30][19000/28800] Time 0.056 (avg:0.043) Loss 382.316772 (avg:152.022305) 
137 | Epoch: [4/30][20000/28800] Time 0.026 (avg:0.043) Loss 0.000000 (avg:152.045801) 
138 | Epoch: [4/30][21000/28800] Time 0.025 (avg:0.043) Loss 0.549782 (avg:151.970350) 
139 | Epoch: [4/30][22000/28800] Time 0.016 (avg:0.043) Loss 2.803649 (avg:151.508970) 
140 | Epoch: [4/30][23000/28800] Time 0.017 (avg:0.043) Loss 5.213553 (avg:151.421002) 
141 | Epoch: [4/30][24000/28800] Time 0.060 (avg:0.043) Loss 52.445831 (avg:151.106472) 
142 | Epoch: [4/30][25000/28800] Time 0.036 (avg:0.043) Loss 292.474457 (avg:150.903903) 
143 | Epoch: [4/30][26000/28800] Time 0.021 (avg:0.043) Loss 0.985871 (avg:150.830614) 
144 | Epoch: [4/30][27000/28800] Time 0.055 (avg:0.043) Loss 735.160950 (avg:150.937013) 
145 | Epoch: [4/30][28000/28800] Time 0.032 (avg:0.043) Loss 229.996811 (avg:151.326135) 
146 | Epoch: [5/30][0/28800] Time 0.300 (avg:0.300) Loss 357.013367 (avg:357.013367) 
147 | Epoch: [5/30][1000/28800] Time 0.014 (avg:0.043) Loss 10.818632 (avg:143.126088) 
148 | Epoch: [5/30][2000/28800] Time 0.031 (avg:0.043) Loss 219.177399 (avg:145.537502) 
149 | Epoch: [5/30][3000/28800] Time 0.055 (avg:0.042) Loss 99.332481 (avg:143.547908) 
150 | Epoch: [5/30][4000/28800] Time 0.015 (avg:0.042) Loss 0.667727 (avg:146.247054) 
151 | Epoch: [5/30][5000/28800] Time 0.024 (avg:0.042) Loss 0.995017 (avg:145.486742) 
152 | Epoch: [5/30][6000/28800] Time 0.021 (avg:0.042) Loss 0.000000 (avg:147.337641) 
153 | Epoch: [5/30][7000/28800] Time 0.121 (avg:0.043) Loss 463.303711 (avg:148.907971) 
154 | Epoch: [5/30][8000/28800] Time 0.064 (avg:0.043) Loss 171.471710 (avg:149.453944) 
155 | Epoch: [5/30][9000/28800] Time 0.014 (avg:0.043) Loss 3.137382 (avg:149.537676) 
156 | Epoch: [5/30][10000/28800] Time 0.120 (avg:0.043) Loss 376.995422 (avg:148.413842) 
157 | Epoch: [5/30][11000/28800] Time 0.016 (avg:0.042) Loss 0.810440 (avg:147.441487) 
158 | Epoch: [5/30][12000/28800] Time 0.053 (avg:0.043) Loss 373.879761 (avg:148.163119) 
159 | Epoch: [5/30][13000/28800] Time 0.120 (avg:0.043) Loss 238.676987 (avg:147.656154) 
160 | Epoch: [5/30][14000/28800] Time 0.031 (avg:0.043) Loss 197.768082 (avg:148.050233) 
161 | Epoch: [5/30][15000/28800] Time 0.088 (avg:0.043) Loss 127.370949 (avg:148.540782) 
162 | Epoch: [5/30][16000/28800] Time 0.036 (avg:0.043) Loss 183.039719 (avg:149.350547) 
163 | Epoch: [5/30][17000/28800] Time 0.037 (avg:0.043) Loss 115.028748 (avg:149.831658) 
164 | Epoch: [5/30][18000/28800] Time 0.050 (avg:0.043) Loss 228.287033 (avg:150.064333) 
165 | Epoch: [5/30][19000/28800] Time 0.015 (avg:0.043) Loss 7.619267 (avg:150.181043) 
166 | Epoch: [5/30][20000/28800] Time 0.014 (avg:0.043) Loss 2.201378 (avg:150.604529) 
167 | Epoch: [5/30][21000/28800] Time 0.034 (avg:0.043) Loss 49.303505 (avg:150.471517) 
168 | Epoch: [5/30][22000/28800] Time 0.031 (avg:0.043) Loss 114.226479 (avg:150.306120) 
169 | Epoch: [5/30][23000/28800] Time 0.024 (avg:0.043) Loss 0.000000 (avg:150.143999) 
170 | Epoch: [5/30][24000/28800] Time 0.056 (avg:0.043) Loss 213.056931 (avg:150.022262) 
171 | Epoch: [5/30][25000/28800] Time 0.021 (avg:0.043) Loss 5.811708 (avg:149.701672) 
172 | Epoch: [5/30][26000/28800] Time 0.033 (avg:0.043) Loss 235.337570 (avg:149.613953) 
173 | Epoch: [5/30][27000/28800] Time 0.023 (avg:0.043) Loss 0.272518 (avg:149.789995) 
174 | Epoch: [5/30][28000/28800] Time 0.060 (avg:0.043) Loss 48.485909 (avg:150.344199) 
175 | Epoch: [6/30][0/28800] Time 0.176 (avg:0.176) Loss 11.454286 (avg:11.454286) 
176 | Epoch: [6/30][1000/28800] Time 0.041 (avg:0.043) Loss 113.318886 (avg:144.566371) 
177 | Epoch: [6/30][2000/28800] Time 0.056 (avg:0.042) Loss 251.665756 (avg:143.906497) 
178 | Epoch: [6/30][3000/28800] Time 0.016 (avg:0.043) Loss 4.139391 (avg:148.254686) 
179 | Epoch: [6/30][4000/28800] Time 0.036 (avg:0.043) Loss 83.818802 (avg:149.366215) 
180 | Epoch: [6/30][5000/28800] Time 0.017 (avg:0.043) Loss 1.322309 (avg:149.194338) 
181 | Epoch: [6/30][6000/28800] Time 0.086 (avg:0.043) Loss 108.437950 (avg:148.566274) 
182 | Epoch: [6/30][7000/28800] Time 0.064 (avg:0.043) Loss 523.318787 (avg:148.336735) 
183 | Epoch: [6/30][8000/28800] Time 0.014 (avg:0.043) Loss 13.146529 (avg:148.061799) 
184 | Epoch: [6/30][9000/28800] Time 0.037 (avg:0.043) Loss 123.153648 (avg:147.613101) 
185 | Epoch: [6/30][10000/28800] Time 0.058 (avg:0.043) Loss 118.205383 (avg:148.012432) 
186 | Epoch: [6/30][11000/28800] Time 0.013 (avg:0.043) Loss 9.626380 (avg:148.591682) 
187 | Epoch: [6/30][12000/28800] Time 0.048 (avg:0.043) Loss 69.196091 (avg:148.879621) 
188 | Epoch: [6/30][13000/28800] Time 0.031 (avg:0.043) Loss 204.407333 (avg:150.054826) 
189 | Epoch: [6/30][14000/28800] Time 0.037 (avg:0.043) Loss 137.181000 (avg:150.512994) 
190 | Epoch: [6/30][15000/28800] Time 0.031 (avg:0.043) Loss 47.051384 (avg:150.194018) 
191 | Epoch: [6/30][16000/28800] Time 0.064 (avg:0.043) Loss 351.758667 (avg:149.979025) 
192 | Epoch: [6/30][17000/28800] Time 0.016 (avg:0.043) Loss 2.316987 (avg:150.010203) 
193 | Epoch: [6/30][18000/28800] Time 0.057 (avg:0.043) Loss 433.867493 (avg:150.376057) 
194 | Epoch: [6/30][19000/28800] Time 0.015 (avg:0.043) Loss 9.728168 (avg:150.470717) 
195 | Epoch: [6/30][20000/28800] Time 0.053 (avg:0.043) Loss 60.350918 (avg:150.053637) 
196 | Epoch: [6/30][21000/28800] Time 0.055 (avg:0.043) Loss 159.700500 (avg:150.386004) 
197 | Epoch: [6/30][22000/28800] Time 0.017 (avg:0.043) Loss 9.343747 (avg:149.978825) 
198 | Epoch: [6/30][23000/28800] Time 0.031 (avg:0.043) Loss 119.258415 (avg:149.744076) 
199 | Epoch: [6/30][24000/28800] Time 0.087 (avg:0.043) Loss 555.917908 (avg:149.790668) 
200 | Epoch: [6/30][25000/28800] Time 0.017 (avg:0.043) Loss 8.051398 (avg:149.909079) 
201 | Epoch: [6/30][26000/28800] Time 0.060 (avg:0.043) Loss 200.840347 (avg:149.429950) 
202 | Epoch: [6/30][27000/28800] Time 0.014 (avg:0.043) Loss 18.660822 (avg:149.353568) 
203 | Epoch: [6/30][28000/28800] Time 0.066 (avg:0.043) Loss 113.890007 (avg:149.359197) 
204 | Epoch: [7/30][0/28800] Time 0.188 (avg:0.188) Loss 47.000946 (avg:47.000946) 
205 | Epoch: [7/30][1000/28800] Time 0.036 (avg:0.044) Loss 109.399780 (avg:155.597559) 
206 | Epoch: [7/30][2000/28800] Time 0.066 (avg:0.044) Loss 311.033630 (avg:155.615399) 
207 | Epoch: [7/30][3000/28800] Time 0.120 (avg:0.043) Loss 217.875778 (avg:153.910914) 
208 | Epoch: [7/30][4000/28800] Time 0.032 (avg:0.043) Loss 92.323051 (avg:152.223491) 
209 | Epoch: [7/30][5000/28800] Time 0.023 (avg:0.043) Loss 0.000000 (avg:150.333314) 
210 | Epoch: [7/30][6000/28800] Time 0.059 (avg:0.043) Loss 316.750305 (avg:150.561693) 
211 | Epoch: [7/30][7000/28800] Time 0.035 (avg:0.043) Loss 85.874321 (avg:149.592962) 
212 | Epoch: [7/30][8000/28800] Time 0.050 (avg:0.043) Loss 313.938110 (avg:149.604495) 
213 | Epoch: [7/30][9000/28800] Time 0.049 (avg:0.043) Loss 274.733704 (avg:150.232145) 
214 | Epoch: [7/30][10000/28800] Time 0.065 (avg:0.043) Loss 325.442047 (avg:150.308794) 
215 | Epoch: [7/30][11000/28800] Time 0.058 (avg:0.043) Loss 198.776764 (avg:149.735334) 
216 | Epoch: [7/30][12000/28800] Time 0.025 (avg:0.043) Loss 0.000000 (avg:149.169089) 
217 | Epoch: [7/30][13000/28800] Time 0.035 (avg:0.043) Loss 164.735352 (avg:149.303275) 
218 | Epoch: [7/30][14000/28800] Time 0.031 (avg:0.043) Loss 99.616020 (avg:150.146294) 
219 | Epoch: [7/30][15000/28800] Time 0.041 (avg:0.043) Loss 170.448166 (avg:150.354523) 
220 | Epoch: [7/30][16000/28800] Time 0.023 (avg:0.043) Loss 0.000000 (avg:150.583697) 
221 | Epoch: [7/30][17000/28800] Time 0.013 (avg:0.043) Loss 7.946659 (avg:150.129342) 
222 | Epoch: [7/30][18000/28800] Time 0.041 (avg:0.043) Loss 134.663483 (avg:150.416661) 
223 | Epoch: [7/30][19000/28800] Time 0.087 (avg:0.043) Loss 631.922913 (avg:150.344878) 
224 | Epoch: [7/30][20000/28800] Time 0.056 (avg:0.043) Loss 124.220764 (avg:149.831704) 
225 | Epoch: [7/30][21000/28800] Time 0.066 (avg:0.043) Loss 123.977226 (avg:149.654994) 
226 | Epoch: [7/30][22000/28800] Time 0.015 (avg:0.043) Loss 1.793114 (avg:149.078984) 
227 | Epoch: [7/30][23000/28800] Time 0.013 (avg:0.043) Loss 0.618930 (avg:149.149978) 
228 | Epoch: [7/30][24000/28800] Time 0.032 (avg:0.043) Loss 209.005844 (avg:148.853695) 
229 | Epoch: [7/30][25000/28800] Time 0.053 (avg:0.043) Loss 248.144562 (avg:148.782933) 
230 | Epoch: [7/30][26000/28800] Time 0.017 (avg:0.043) Loss 3.787706 (avg:149.209346) 
231 | Epoch: [7/30][27000/28800] Time 0.119 (avg:0.043) Loss 244.178818 (avg:149.209850) 
232 | Epoch: [7/30][28000/28800] Time 0.077 (avg:0.043) Loss 98.327141 (avg:149.041900) 
233 | Epoch: [8/30][0/28800] Time 0.221 (avg:0.221) Loss 101.535988 (avg:101.535988) 
234 | Epoch: [8/30][1000/28800] Time 0.064 (avg:0.044) Loss 113.325745 (avg:159.663731) 
235 | Epoch: [8/30][2000/28800] Time 0.121 (avg:0.043) Loss 229.542435 (avg:153.903375) 
236 | Epoch: [8/30][3000/28800] Time 0.016 (avg:0.043) Loss 0.000000 (avg:150.810412) 
237 | Epoch: [8/30][4000/28800] Time 0.036 (avg:0.043) Loss 161.251266 (avg:148.654262) 
238 | Epoch: [8/30][5000/28800] Time 0.033 (avg:0.042) Loss 23.068705 (avg:145.422255) 
239 | Epoch: [8/30][6000/28800] Time 0.048 (avg:0.042) Loss 176.203369 (avg:144.740235) 
240 | Epoch: [8/30][7000/28800] Time 0.016 (avg:0.042) Loss 2.953656 (avg:145.208722) 
241 | Epoch: [8/30][8000/28800] Time 0.015 (avg:0.042) Loss 0.622139 (avg:144.401334) 
242 | Epoch: [8/30][9000/28800] Time 0.031 (avg:0.042) Loss 183.014511 (avg:146.228131) 
243 | Epoch: [8/30][10000/28800] Time 0.021 (avg:0.042) Loss 0.000000 (avg:145.176724) 
244 | Epoch: [8/30][11000/28800] Time 0.065 (avg:0.042) Loss 149.465179 (avg:146.742756) 
245 | Epoch: [8/30][12000/28800] Time 0.067 (avg:0.042) Loss 518.663208 (avg:146.966415) 
246 | Epoch: [8/30][13000/28800] Time 0.043 (avg:0.042) Loss 231.457504 (avg:147.214689) 
247 | Epoch: [8/30][14000/28800] Time 0.021 (avg:0.042) Loss 2.098233 (avg:147.970444) 
248 | Epoch: [8/30][15000/28800] Time 0.056 (avg:0.042) Loss 121.353760 (avg:147.205202) 
249 | Epoch: [8/30][16000/28800] Time 0.034 (avg:0.042) Loss 82.504921 (avg:147.875784) 
250 | Epoch: [8/30][17000/28800] Time 0.035 (avg:0.043) Loss 104.069336 (avg:148.133180) 
251 | Epoch: [8/30][18000/28800] Time 0.017 (avg:0.043) Loss 11.897498 (avg:147.803061) 
252 | Epoch: [8/30][19000/28800] Time 0.031 (avg:0.043) Loss 171.998734 (avg:148.017983) 
253 | Epoch: [8/30][20000/28800] Time 0.024 (avg:0.043) Loss 1.471005 (avg:148.301192) 
254 | Epoch: [8/30][21000/28800] Time 0.013 (avg:0.043) Loss 0.200993 (avg:148.707936) 
255 | Epoch: [8/30][22000/28800] Time 0.015 (avg:0.043) Loss 1.265544 (avg:148.729777) 
256 | Epoch: [8/30][23000/28800] Time 0.016 (avg:0.043) Loss 1.530179 (avg:148.667655) 
257 | Epoch: [8/30][24000/28800] Time 0.052 (avg:0.043) Loss 341.857849 (avg:148.885802) 
258 | Epoch: [8/30][25000/28800] Time 0.034 (avg:0.043) Loss 205.130127 (avg:149.190406) 
259 | Epoch: [8/30][26000/28800] Time 0.031 (avg:0.043) Loss 115.957787 (avg:149.228337) 
260 | Epoch: [8/30][27000/28800] Time 0.031 (avg:0.043) Loss 227.187973 (avg:149.183487) 
261 | Epoch: [8/30][28000/28800] Time 0.016 (avg:0.043) Loss 17.262815 (avg:149.111567) 
262 | Epoch: [9/30][0/28800] Time 0.173 (avg:0.173) Loss 2.636274 (avg:2.636274) 
263 | Epoch: [9/30][1000/28800] Time 0.014 (avg:0.044) Loss 0.356181 (avg:155.741481) 
264 | Epoch: [9/30][2000/28800] Time 0.017 (avg:0.044) Loss 19.804243 (avg:154.124807) 
265 | Epoch: [9/30][3000/28800] Time 0.034 (avg:0.044) Loss 61.410793 (avg:156.661457) 
266 | Epoch: [9/30][4000/28800] Time 0.035 (avg:0.044) Loss 101.759712 (avg:153.957883) 
267 | Epoch: [9/30][5000/28800] Time 0.034 (avg:0.044) Loss 166.640549 (avg:153.067814) 
268 | Epoch: [9/30][6000/28800] Time 0.032 (avg:0.043) Loss 116.373306 (avg:151.880607) 
269 | Epoch: [9/30][7000/28800] Time 0.014 (avg:0.043) Loss 6.750780 (avg:151.774217) 
270 | Epoch: [9/30][8000/28800] Time 0.014 (avg:0.043) Loss 0.795588 (avg:152.533452) 
271 | Epoch: [9/30][9000/28800] Time 0.049 (avg:0.043) Loss 209.961044 (avg:151.804597) 
272 | Epoch: [9/30][10000/28800] Time 0.035 (avg:0.043) Loss 277.857391 (avg:150.778790) 
273 | Epoch: [9/30][11000/28800] Time 0.056 (avg:0.043) Loss 137.702133 (avg:151.634520) 
274 | Epoch: [9/30][12000/28800] Time 0.031 (avg:0.043) Loss 204.839966 (avg:151.977421) 
275 | Epoch: [9/30][13000/28800] Time 0.036 (avg:0.043) Loss 328.545532 (avg:152.349437) 
276 | Epoch: [9/30][14000/28800] Time 0.015 (avg:0.043) Loss 0.338463 (avg:151.910734) 
277 | Epoch: [9/30][15000/28800] Time 0.018 (avg:0.043) Loss 3.796898 (avg:151.623198) 
278 | Epoch: [9/30][16000/28800] Time 0.031 (avg:0.043) Loss 89.778404 (avg:151.707022) 
279 | Epoch: [9/30][17000/28800] Time 0.015 (avg:0.043) Loss 3.531024 (avg:151.771567) 
280 | Epoch: [9/30][18000/28800] Time 0.015 (avg:0.043) Loss 7.549531 (avg:152.181326) 
281 | Epoch: [9/30][19000/28800] Time 0.056 (avg:0.043) Loss 146.171997 (avg:151.751060) 
282 | Epoch: [9/30][20000/28800] Time 0.057 (avg:0.043) Loss 60.648987 (avg:151.503892) 
283 | Epoch: [9/30][21000/28800] Time 0.014 (avg:0.043) Loss 22.430092 (avg:151.132865) 
284 | Epoch: [9/30][22000/28800] Time 0.065 (avg:0.043) Loss 116.288330 (avg:150.948753) 
285 | Epoch: [9/30][23000/28800] Time 0.036 (avg:0.043) Loss 81.183922 (avg:150.851876) 
286 | Epoch: [9/30][24000/28800] Time 0.118 (avg:0.043) Loss 514.756714 (avg:150.416082) 
287 | Epoch: [9/30][25000/28800] Time 0.032 (avg:0.043) Loss 98.847267 (avg:150.245996) 
288 | Epoch: [9/30][26000/28800] Time 0.035 (avg:0.043) Loss 81.967842 (avg:149.587514) 
289 | Epoch: [9/30][27000/28800] Time 0.065 (avg:0.043) Loss 58.992493 (avg:149.458025) 
290 | Epoch: [9/30][28000/28800] Time 0.043 (avg:0.043) Loss 274.948883 (avg:149.667422) 
291 | Epoch: [10/30][0/28800] Time 0.241 (avg:0.241) Loss 77.426331 (avg:77.426331) 
292 | Epoch: [10/30][1000/28800] Time 0.086 (avg:0.043) Loss 541.057312 (avg:146.289830) 
293 | Epoch: [10/30][2000/28800] Time 0.088 (avg:0.045) Loss 566.700623 (avg:154.619401) 
294 | Epoch: [10/30][3000/28800] Time 0.043 (avg:0.044) Loss 310.052765 (avg:152.900838) 
295 | Epoch: [10/30][4000/28800] Time 0.026 (avg:0.044) Loss 3.139865 (avg:150.902947) 
296 | Epoch: [10/30][5000/28800] Time 0.031 (avg:0.044) Loss 153.873840 (avg:151.088008) 
297 | Epoch: [10/30][6000/28800] Time 0.035 (avg:0.044) Loss 129.857773 (avg:151.561480) 
298 | Epoch: [10/30][7000/28800] Time 0.060 (avg:0.043) Loss 88.978195 (avg:149.647995) 
299 | Epoch: [10/30][8000/28800] Time 0.058 (avg:0.043) Loss 156.103073 (avg:149.512486) 
300 | Epoch: [10/30][9000/28800] Time 0.076 (avg:0.043) Loss 75.693893 (avg:150.131682) 
301 | Epoch: [10/30][10000/28800] Time 0.031 (avg:0.043) Loss 181.444107 (avg:149.891751) 
302 | Epoch: [10/30][11000/28800] Time 0.058 (avg:0.043) Loss 63.069881 (avg:150.766156) 
303 | Epoch: [10/30][12000/28800] Time 0.065 (avg:0.043) Loss 331.661072 (avg:150.379574) 
304 | Epoch: [10/30][13000/28800] Time 0.038 (avg:0.043) Loss 143.435837 (avg:149.987278) 
305 | Epoch: [10/30][14000/28800] Time 0.016 (avg:0.043) Loss 3.048990 (avg:149.614775) 
306 | Epoch: [10/30][15000/28800] Time 0.067 (avg:0.043) Loss 612.927185 (avg:149.334372) 
307 | Epoch: [10/30][16000/28800] Time 0.032 (avg:0.043) Loss 276.802155 (avg:149.479103) 
308 | Epoch: [10/30][17000/28800] Time 0.057 (avg:0.043) Loss 115.825172 (avg:149.126751) 
309 | Epoch: [10/30][18000/28800] Time 0.076 (avg:0.043) Loss 296.448151 (avg:149.646170) 
310 | Epoch: [10/30][19000/28800] Time 0.066 (avg:0.043) Loss 95.479103 (avg:149.633840) 
311 | Epoch: [10/30][20000/28800] Time 0.035 (avg:0.043) Loss 238.221710 (avg:149.942485) 
312 | Epoch: [10/30][21000/28800] Time 0.053 (avg:0.043) Loss 216.167160 (avg:149.662724) 
313 | Epoch: [10/30][22000/28800] Time 0.021 (avg:0.043) Loss 0.000000 (avg:149.199628) 
314 | Epoch: [10/30][23000/28800] Time 0.057 (avg:0.043) Loss 108.968925 (avg:149.174092) 
315 | Epoch: [10/30][24000/28800] Time 0.015 (avg:0.043) Loss 0.000000 (avg:148.919737) 
316 | Epoch: [10/30][25000/28800] Time 0.036 (avg:0.043) Loss 117.421074 (avg:148.836655) 
317 | Epoch: [10/30][26000/28800] Time 0.036 (avg:0.043) Loss 141.395935 (avg:148.953416) 
318 | Epoch: [10/30][27000/28800] Time 0.068 (avg:0.043) Loss 632.304138 (avg:148.888402) 
319 | Epoch: [10/30][28000/28800] Time 0.033 (avg:0.043) Loss 247.537598 (avg:149.491076) 
320 | Epoch: [11/30][0/28800] Time 0.229 (avg:0.229) Loss 225.702591 (avg:225.702591) 
321 | Epoch: [11/30][1000/28800] Time 0.118 (avg:0.042) Loss 453.184387 (avg:150.178985) 
322 | Epoch: [11/30][2000/28800] Time 0.057 (avg:0.043) Loss 114.833168 (avg:148.832038) 
323 | Epoch: [11/30][3000/28800] Time 0.041 (avg:0.043) Loss 115.073891 (avg:147.832846) 
324 | Epoch: [11/30][4000/28800] Time 0.013 (avg:0.043) Loss 5.816532 (avg:149.744781) 
325 | Epoch: [11/30][5000/28800] Time 0.020 (avg:0.043) Loss 0.000000 (avg:149.266595) 
326 | Epoch: [11/30][6000/28800] Time 0.042 (avg:0.043) Loss 140.471954 (avg:149.773689) 
327 | Epoch: [11/30][7000/28800] Time 0.034 (avg:0.043) Loss 90.624474 (avg:148.948139) 
328 | Epoch: [11/30][8000/28800] Time 0.015 (avg:0.043) Loss 0.890632 (avg:148.991954) 
329 | Epoch: [11/30][9000/28800] Time 0.035 (avg:0.043) Loss 59.447144 (avg:149.317791) 
330 | Epoch: [11/30][10000/28800] Time 0.032 (avg:0.043) Loss 134.018906 (avg:149.526256) 
331 | Epoch: [11/30][11000/28800] Time 0.015 (avg:0.043) Loss 2.500348 (avg:149.457079) 
332 | Epoch: [11/30][12000/28800] Time 0.035 (avg:0.043) Loss 109.427567 (avg:150.429458) 
333 | Epoch: [11/30][13000/28800] Time 0.052 (avg:0.043) Loss 258.353790 (avg:150.431559) 
334 | Epoch: [11/30][14000/28800] Time 0.058 (avg:0.043) Loss 134.274170 (avg:150.117786) 
335 | Epoch: [11/30][15000/28800] Time 0.087 (avg:0.043) Loss 255.818970 (avg:149.864946) 
336 | Epoch: [11/30][16000/28800] Time 0.014 (avg:0.043) Loss 0.757100 (avg:149.736381) 
337 | Epoch: [11/30][17000/28800] Time 0.057 (avg:0.043) Loss 72.949860 (avg:149.617242) 
338 | Epoch: [11/30][18000/28800] Time 0.059 (avg:0.043) Loss 101.884720 (avg:149.843238) 
339 | Epoch: [11/30][19000/28800] Time 0.059 (avg:0.043) Loss 324.703308 (avg:149.435159) 
340 | Epoch: [11/30][20000/28800] Time 0.041 (avg:0.043) Loss 134.676605 (avg:149.595575) 
341 | Epoch: [11/30][21000/28800] Time 0.058 (avg:0.043) Loss 123.739639 (avg:149.645475) 
342 | Epoch: [11/30][22000/28800] Time 0.035 (avg:0.043) Loss 80.274994 (avg:149.835364) 
343 | Epoch: [11/30][23000/28800] Time 0.036 (avg:0.043) Loss 167.717957 (avg:149.809028) 
344 | Epoch: [11/30][24000/28800] Time 0.015 (avg:0.043) Loss 0.302080 (avg:149.743952) 
345 | Epoch: [11/30][25000/28800] Time 0.035 (avg:0.043) Loss 28.486526 (avg:149.619306) 
346 | Epoch: [11/30][26000/28800] Time 0.041 (avg:0.043) Loss 133.278275 (avg:149.844946) 
347 | Epoch: [11/30][27000/28800] Time 0.056 (avg:0.043) Loss 310.953949 (avg:149.387661) 
348 | Epoch: [11/30][28000/28800] Time 0.057 (avg:0.043) Loss 324.002899 (avg:149.301744) 
349 | Epoch: [12/30][0/28800] Time 0.191 (avg:0.191) Loss 89.533768 (avg:89.533768) 
350 | Epoch: [12/30][1000/28800] Time 0.014 (avg:0.043) Loss 1.941892 (avg:150.121681) 
351 | Epoch: [12/30][2000/28800] Time 0.041 (avg:0.043) Loss 129.917404 (avg:144.892997) 
352 | Epoch: [12/30][3000/28800] Time 0.014 (avg:0.043) Loss 2.411483 (avg:145.022915) 
353 | Epoch: [12/30][4000/28800] Time 0.041 (avg:0.043) Loss 91.227249 (avg:146.001627) 
354 | Epoch: [12/30][5000/28800] Time 0.064 (avg:0.043) Loss 127.429405 (avg:147.036641) 
355 | Epoch: [12/30][6000/28800] Time 0.050 (avg:0.043) Loss 463.735748 (avg:148.605932) 
356 | Epoch: [12/30][7000/28800] Time 0.057 (avg:0.043) Loss 318.984131 (avg:149.500261) 
357 | Epoch: [12/30][8000/28800] Time 0.017 (avg:0.043) Loss 5.169786 (avg:151.266501) 
358 | Epoch: [12/30][9000/28800] Time 0.086 (avg:0.043) Loss 172.012680 (avg:150.558944) 
359 | Epoch: [12/30][10000/28800] Time 0.015 (avg:0.043) Loss 2.213506 (avg:150.960486) 
360 | Epoch: [12/30][11000/28800] Time 0.051 (avg:0.043) Loss 114.340027 (avg:151.058558) 
361 | Epoch: [12/30][12000/28800] Time 0.065 (avg:0.043) Loss 134.839340 (avg:150.113945) 
362 | Epoch: [12/30][13000/28800] Time 0.015 (avg:0.043) Loss 7.169033 (avg:149.706226) 
363 | Epoch: [12/30][14000/28800] Time 0.016 (avg:0.043) Loss 1.493749 (avg:149.927077) 
364 | Epoch: [12/30][15000/28800] Time 0.016 (avg:0.043) Loss 6.803520 (avg:149.680870) 
365 | Epoch: [12/30][16000/28800] Time 0.014 (avg:0.043) Loss 2.606189 (avg:149.221207) 
366 | Epoch: [12/30][17000/28800] Time 0.058 (avg:0.043) Loss 98.175804 (avg:148.982270) 
367 | Epoch: [12/30][18000/28800] Time 0.060 (avg:0.043) Loss 57.917126 (avg:149.400569) 
368 | Epoch: [12/30][19000/28800] Time 0.065 (avg:0.043) Loss 227.986893 (avg:149.328563) 
369 | Epoch: [12/30][20000/28800] Time 0.014 (avg:0.043) Loss 1.579092 (avg:150.091091) 
370 | Epoch: [12/30][21000/28800] Time 0.057 (avg:0.043) Loss 205.653519 (avg:150.075293) 
371 | Epoch: [12/30][22000/28800] Time 0.031 (avg:0.043) Loss 65.008118 (avg:149.949040) 
372 | Epoch: [12/30][23000/28800] Time 0.031 (avg:0.043) Loss 161.290695 (avg:149.878696) 
373 | Epoch: [12/30][24000/28800] Time 0.056 (avg:0.043) Loss 179.389984 (avg:149.864973) 
374 | Epoch: [12/30][25000/28800] Time 0.058 (avg:0.043) Loss 204.869980 (avg:149.907154) 
375 | Epoch: [12/30][26000/28800] Time 0.021 (avg:0.043) Loss 5.072536 (avg:149.673270) 
376 | Epoch: [12/30][27000/28800] Time 0.032 (avg:0.043) Loss 115.516991 (avg:149.507297) 
377 | Epoch: [12/30][28000/28800] Time 0.032 (avg:0.043) Loss 345.981781 (avg:149.383403) 
378 | Epoch: [13/30][0/28800] Time 0.173 (avg:0.173) Loss 24.603485 (avg:24.603485) 
379 | Epoch: [13/30][1000/28800] Time 0.120 (avg:0.044) Loss 642.914001 (avg:149.717630) 
380 | Epoch: [13/30][2000/28800] Time 0.056 (avg:0.043) Loss 375.333649 (avg:145.989552) 
381 | Epoch: [13/30][3000/28800] Time 0.087 (avg:0.043) Loss 574.715515 (avg:147.839773) 
382 | Epoch: [13/30][4000/28800] Time 0.026 (avg:0.043) Loss 0.000000 (avg:146.032604) 
383 | Epoch: [13/30][5000/28800] Time 0.017 (avg:0.043) Loss 2.920690 (avg:146.280114) 
384 | Epoch: [13/30][6000/28800] Time 0.087 (avg:0.043) Loss 312.281250 (avg:145.881436) 
385 | Epoch: [13/30][7000/28800] Time 0.031 (avg:0.043) Loss 37.866436 (avg:146.512696) 
386 | Epoch: [13/30][8000/28800] Time 0.032 (avg:0.043) Loss 182.346695 (avg:147.447573) 
387 | Epoch: [13/30][9000/28800] Time 0.058 (avg:0.043) Loss 410.401367 (avg:147.301281) 
388 | Epoch: [13/30][10000/28800] Time 0.076 (avg:0.043) Loss 162.598465 (avg:146.672880) 
389 | Epoch: [13/30][11000/28800] Time 0.013 (avg:0.043) Loss 3.010171 (avg:146.362688) 
390 | Epoch: [13/30][12000/28800] Time 0.048 (avg:0.043) Loss 354.199554 (avg:146.878301) 
391 | Epoch: [13/30][13000/28800] Time 0.119 (avg:0.043) Loss 297.877319 (avg:147.697729) 
392 | Epoch: [13/30][14000/28800] Time 0.066 (avg:0.043) Loss 201.698822 (avg:147.687177) 
393 | Epoch: [13/30][15000/28800] Time 0.057 (avg:0.043) Loss 91.196304 (avg:148.261627) 
394 | Epoch: [13/30][16000/28800] Time 0.057 (avg:0.043) Loss 131.501160 (avg:149.220481) 
395 | Epoch: [13/30][17000/28800] Time 0.034 (avg:0.043) Loss 74.004677 (avg:148.997152) 
396 | Epoch: [13/30][18000/28800] Time 0.036 (avg:0.043) Loss 535.704041 (avg:149.445633) 
397 | Epoch: [13/30][19000/28800] Time 0.014 (avg:0.043) Loss 4.141502 (avg:150.119487) 
398 | Epoch: [13/30][20000/28800] Time 0.022 (avg:0.043) Loss 0.000000 (avg:149.642369) 
399 | Epoch: [13/30][21000/28800] Time 0.023 (avg:0.043) Loss 0.264379 (avg:149.256350) 
400 | Epoch: [13/30][22000/28800] Time 0.015 (avg:0.043) Loss 4.977058 (avg:149.155347) 
401 | Epoch: [13/30][23000/28800] Time 0.057 (avg:0.043) Loss 63.605152 (avg:149.191173) 
402 | Epoch: [13/30][24000/28800] Time 0.015 (avg:0.043) Loss 20.601055 (avg:149.317788) 
403 | Epoch: [13/30][25000/28800] Time 0.015 (avg:0.043) Loss 1.046671 (avg:149.321017) 
404 | Epoch: [13/30][26000/28800] Time 0.118 (avg:0.043) Loss 424.024384 (avg:149.160794) 
405 | Epoch: [13/30][27000/28800] Time 0.035 (avg:0.043) Loss 61.182838 (avg:148.869211) 
406 | Epoch: [13/30][28000/28800] Time 0.031 (avg:0.043) Loss 66.743797 (avg:148.957635) 
407 | Epoch: [14/30][0/28800] Time 0.183 (avg:0.183) Loss 2.824628 (avg:2.824628) 
408 | Epoch: [14/30][1000/28800] Time 0.015 (avg:0.044) Loss 5.336677 (avg:151.363891) 
409 | Epoch: [14/30][2000/28800] Time 0.034 (avg:0.044) Loss 87.345703 (avg:152.171130) 
410 | Epoch: [14/30][3000/28800] Time 0.032 (avg:0.044) Loss 140.771072 (avg:153.149748) 
411 | Epoch: [14/30][4000/28800] Time 0.053 (avg:0.044) Loss 216.249420 (avg:151.090204) 
412 | Epoch: [14/30][5000/28800] Time 0.068 (avg:0.043) Loss 174.264679 (avg:151.219120) 
413 | Epoch: [14/30][6000/28800] Time 0.034 (avg:0.044) Loss 203.193878 (avg:153.754233) 
414 | Epoch: [14/30][7000/28800] Time 0.021 (avg:0.044) Loss 0.000000 (avg:153.372044) 
415 | Epoch: [14/30][8000/28800] Time 0.023 (avg:0.044) Loss 0.647653 (avg:155.952148) 
416 | Epoch: [14/30][9000/28800] Time 0.014 (avg:0.043) Loss 1.403094 (avg:154.967719) 
417 | Epoch: [14/30][10000/28800] Time 0.015 (avg:0.043) Loss 0.468165 (avg:154.522694) 
418 | Epoch: [14/30][11000/28800] Time 0.015 (avg:0.043) Loss 13.377601 (avg:154.361028) 
419 | Epoch: [14/30][12000/28800] Time 0.042 (avg:0.043) Loss 206.876816 (avg:153.205744) 
420 | Epoch: [14/30][13000/28800] Time 0.056 (avg:0.043) Loss 95.686539 (avg:151.943427) 
421 | Epoch: [14/30][14000/28800] Time 0.032 (avg:0.043) Loss 145.181152 (avg:151.550659) 
422 | Epoch: [14/30][15000/28800] Time 0.016 (avg:0.043) Loss 1.520206 (avg:150.757183) 
423 | Epoch: [14/30][16000/28800] Time 0.118 (avg:0.043) Loss 242.276352 (avg:150.722365) 
424 | Epoch: [14/30][17000/28800] Time 0.020 (avg:0.043) Loss 0.000000 (avg:151.005617) 
425 | Epoch: [14/30][18000/28800] Time 0.061 (avg:0.043) Loss 228.211014 (avg:151.200162) 
426 | Epoch: [14/30][19000/28800] Time 0.057 (avg:0.043) Loss 348.007263 (avg:150.742235) 
427 | Epoch: [14/30][20000/28800] Time 0.023 (avg:0.043) Loss 0.000000 (avg:150.505724) 
428 | Epoch: [14/30][21000/28800] Time 0.064 (avg:0.043) Loss 159.896698 (avg:150.466550) 
429 | Epoch: [14/30][22000/28800] Time 0.056 (avg:0.043) Loss 100.875107 (avg:150.533661) 
430 | Epoch: [14/30][23000/28800] Time 0.037 (avg:0.043) Loss 310.369812 (avg:150.182297) 
431 | Epoch: [14/30][24000/28800] Time 0.035 (avg:0.043) Loss 281.694397 (avg:150.177136) 
432 | Epoch: [14/30][25000/28800] Time 0.058 (avg:0.043) Loss 107.658524 (avg:149.866300) 
433 | Epoch: [14/30][26000/28800] Time 0.013 (avg:0.043) Loss 3.438353 (avg:149.484192) 
434 | Epoch: [14/30][27000/28800] Time 0.087 (avg:0.043) Loss 259.404114 (avg:149.424071) 
435 | Epoch: [14/30][28000/28800] Time 0.032 (avg:0.043) Loss 150.586044 (avg:149.297173) 
436 | Epoch: [15/30][0/28800] Time 0.218 (avg:0.218) Loss 110.340721 (avg:110.340721) 
437 | Epoch: [15/30][1000/28800] Time 0.016 (avg:0.041) Loss 5.877908 (avg:143.623611) 
438 | Epoch: [15/30][2000/28800] Time 0.038 (avg:0.041) Loss 109.582481 (avg:142.121817) 
439 | Epoch: [15/30][3000/28800] Time 0.088 (avg:0.042) Loss 439.312653 (avg:145.839818) 
440 | Epoch: [15/30][4000/28800] Time 0.014 (avg:0.042) Loss 0.812953 (avg:143.457630) 
441 | Epoch: [15/30][5000/28800] Time 0.015 (avg:0.042) Loss 9.484548 (avg:145.707272) 
442 | Epoch: [15/30][6000/28800] Time 0.015 (avg:0.042) Loss 2.772706 (avg:146.259606) 
443 | Epoch: [15/30][7000/28800] Time 0.037 (avg:0.042) Loss 233.817581 (avg:148.237278) 
444 | Epoch: [15/30][8000/28800] Time 0.052 (avg:0.043) Loss 284.619934 (avg:148.934916) 
445 | Epoch: [15/30][9000/28800] Time 0.033 (avg:0.043) Loss 91.283257 (avg:149.682020) 
446 | Epoch: [15/30][10000/28800] Time 0.059 (avg:0.043) Loss 211.659180 (avg:149.947216) 
447 | Epoch: [15/30][11000/28800] Time 0.016 (avg:0.043) Loss 5.992701 (avg:150.089543) 
448 | Epoch: [15/30][12000/28800] Time 0.064 (avg:0.043) Loss 134.779205 (avg:149.517705) 
449 | Epoch: [15/30][13000/28800] Time 0.087 (avg:0.043) Loss 300.348602 (avg:149.170355) 
450 | Epoch: [15/30][14000/28800] Time 0.016 (avg:0.043) Loss 1.027260 (avg:149.015794) 
451 | Epoch: [15/30][15000/28800] Time 0.056 (avg:0.043) Loss 112.611389 (avg:149.609532) 
452 | Epoch: [15/30][16000/28800] Time 0.024 (avg:0.043) Loss 0.000000 (avg:149.215185) 
453 | Epoch: [15/30][17000/28800] Time 0.034 (avg:0.043) Loss 292.653778 (avg:149.448812) 
454 | Epoch: [15/30][18000/28800] Time 0.061 (avg:0.043) Loss 218.276199 (avg:149.076890) 
455 | Epoch: [15/30][19000/28800] Time 0.068 (avg:0.043) Loss 392.917114 (avg:149.279971) 
456 | Epoch: [15/30][20000/28800] Time 0.014 (avg:0.043) Loss 2.722612 (avg:149.321915) 
457 | Epoch: [15/30][21000/28800] Time 0.017 (avg:0.043) Loss 1.695252 (avg:149.146545) 
458 | Epoch: [15/30][22000/28800] Time 0.066 (avg:0.043) Loss 318.085785 (avg:148.885385) 
459 | Epoch: [15/30][23000/28800] Time 0.033 (avg:0.043) Loss 21.373547 (avg:148.596610) 
460 | Epoch: [15/30][24000/28800] Time 0.013 (avg:0.043) Loss 1.847978 (avg:148.720190) 
461 | Epoch: [15/30][25000/28800] Time 0.014 (avg:0.043) Loss 7.056276 (avg:148.661882) 
462 | Epoch: [15/30][26000/28800] Time 0.061 (avg:0.043) Loss 206.369507 (avg:149.009383) 
463 | Epoch: [15/30][27000/28800] Time 0.015 (avg:0.043) Loss 0.279301 (avg:148.810391) 
464 | Epoch: [15/30][28000/28800] Time 0.015 (avg:0.043) Loss 1.110056 (avg:149.015897) 
465 | Epoch: [16/30][0/28800] Time 0.226 (avg:0.226) Loss 671.215515 (avg:671.215515) 
466 | Epoch: [16/30][1000/28800] Time 0.015 (avg:0.044) Loss 20.914030 (avg:145.546152) 
467 | Epoch: [16/30][2000/28800] Time 0.031 (avg:0.043) Loss 121.345665 (avg:147.032497) 
468 | Epoch: [16/30][3000/28800] Time 0.015 (avg:0.043) Loss 4.872427 (avg:149.573991) 
469 | Epoch: [16/30][4000/28800] Time 0.031 (avg:0.043) Loss 158.159744 (avg:150.183174) 
470 | Epoch: [16/30][5000/28800] Time 0.015 (avg:0.043) Loss 8.147197 (avg:148.819803) 
471 | Epoch: [16/30][6000/28800] Time 0.024 (avg:0.043) Loss 1.339224 (avg:149.564594) 
472 | Epoch: [16/30][7000/28800] Time 0.059 (avg:0.044) Loss 130.993332 (avg:151.278459) 
473 | Epoch: [16/30][8000/28800] Time 0.055 (avg:0.043) Loss 169.917343 (avg:151.388666) 
474 | Epoch: [16/30][9000/28800] Time 0.031 (avg:0.043) Loss 122.935600 (avg:149.926203) 
475 | Epoch: [16/30][10000/28800] Time 0.024 (avg:0.043) Loss 1.170999 (avg:149.559853) 
476 | Epoch: [16/30][11000/28800] Time 0.014 (avg:0.043) Loss 16.858406 (avg:149.815508) 
477 | Epoch: [16/30][12000/28800] Time 0.014 (avg:0.043) Loss 7.670366 (avg:149.431645) 
478 | Epoch: [16/30][13000/28800] Time 0.036 (avg:0.043) Loss 263.990082 (avg:149.889285) 
479 | Epoch: [16/30][14000/28800] Time 0.014 (avg:0.043) Loss 0.000000 (avg:149.546263) 
480 | Epoch: [16/30][15000/28800] Time 0.061 (avg:0.043) Loss 160.233810 (avg:149.578994) 
481 | Epoch: [16/30][16000/28800] Time 0.017 (avg:0.043) Loss 0.930646 (avg:150.492666) 
482 | Epoch: [16/30][17000/28800] Time 0.014 (avg:0.043) Loss 13.371782 (avg:150.050203) 
483 | Epoch: [16/30][18000/28800] Time 0.065 (avg:0.043) Loss 111.896629 (avg:150.051056) 
484 | Epoch: [16/30][19000/28800] Time 0.015 (avg:0.043) Loss 2.250197 (avg:149.829804) 
485 | Epoch: [16/30][20000/28800] Time 0.060 (avg:0.043) Loss 362.018372 (avg:149.730061) 
486 | Epoch: [16/30][21000/28800] Time 0.015 (avg:0.043) Loss 3.196498 (avg:149.479888) 
487 | Epoch: [16/30][22000/28800] Time 0.052 (avg:0.043) Loss 67.704384 (avg:149.358235) 
488 | Epoch: [16/30][23000/28800] Time 0.016 (avg:0.043) Loss 4.419358 (avg:149.462815) 
489 | Epoch: [16/30][24000/28800] Time 0.036 (avg:0.043) Loss 29.255838 (avg:149.831112) 
490 | Epoch: [16/30][25000/28800] Time 0.049 (avg:0.043) Loss 248.166000 (avg:149.519026) 
491 | Epoch: [16/30][26000/28800] Time 0.014 (avg:0.043) Loss 2.954416 (avg:149.415140) 
492 | Epoch: [16/30][27000/28800] Time 0.017 (avg:0.043) Loss 1.304953 (avg:149.506085) 
493 | Epoch: [16/30][28000/28800] Time 0.056 (avg:0.043) Loss 172.498581 (avg:148.887061) 
494 | Epoch: [17/30][0/28800] Time 0.219 (avg:0.219) Loss 433.685791 (avg:433.685791) 
495 | Epoch: [17/30][1000/28800] Time 0.021 (avg:0.043) Loss 0.000000 (avg:149.590033) 
496 | Epoch: [17/30][2000/28800] Time 0.017 (avg:0.043) Loss 0.391808 (avg:147.924015) 
497 | Epoch: [17/30][3000/28800] Time 0.066 (avg:0.043) Loss 90.631943 (avg:145.513471) 
498 | Epoch: [17/30][4000/28800] Time 0.015 (avg:0.043) Loss 1.577089 (avg:146.374154) 
499 | Epoch: [17/30][5000/28800] Time 0.053 (avg:0.043) Loss 436.055634 (avg:148.527652) 
500 | Epoch: [17/30][6000/28800] Time 0.014 (avg:0.043) Loss 2.919078 (avg:148.774426) 
501 | Epoch: [17/30][7000/28800] Time 0.014 (avg:0.042) Loss 2.918633 (avg:149.524487) 
502 | Epoch: [17/30][8000/28800] Time 0.119 (avg:0.042) Loss 168.528259 (avg:150.128623) 
503 | Epoch: [17/30][9000/28800] Time 0.015 (avg:0.042) Loss 17.921728 (avg:151.183268) 
504 | Epoch: [17/30][10000/28800] Time 0.057 (avg:0.043) Loss 50.746464 (avg:151.440686) 
505 | Epoch: [17/30][11000/28800] Time 0.068 (avg:0.043) Loss 513.993591 (avg:151.253566) 
506 | Epoch: [17/30][12000/28800] Time 0.057 (avg:0.043) Loss 231.439117 (avg:151.066955) 
507 | Epoch: [17/30][13000/28800] Time 0.058 (avg:0.043) Loss 138.357162 (avg:150.819535) 
508 | Epoch: [17/30][14000/28800] Time 0.035 (avg:0.043) Loss 57.691143 (avg:149.954975) 
509 | Epoch: [17/30][15000/28800] Time 0.013 (avg:0.043) Loss 4.169206 (avg:150.113647) 
510 | Epoch: [17/30][16000/28800] Time 0.042 (avg:0.043) Loss 176.309265 (avg:150.157202) 
511 | Epoch: [17/30][17000/28800] Time 0.034 (avg:0.043) Loss 284.478363 (avg:149.695604) 
512 | Epoch: [17/30][18000/28800] Time 0.121 (avg:0.043) Loss 245.552521 (avg:149.232941) 
513 | Epoch: [17/30][19000/28800] Time 0.016 (avg:0.043) Loss 0.404261 (avg:148.865263) 
514 | Epoch: [17/30][20000/28800] Time 0.015 (avg:0.043) Loss 0.000000 (avg:149.032964) 
515 | Epoch: [17/30][21000/28800] Time 0.021 (avg:0.043) Loss 1.444697 (avg:148.914761) 
516 | Epoch: [17/30][22000/28800] Time 0.057 (avg:0.043) Loss 158.376785 (avg:148.668456) 
517 | Epoch: [17/30][23000/28800] Time 0.060 (avg:0.043) Loss 71.194725 (avg:148.910817) 
518 | Epoch: [17/30][24000/28800] Time 0.056 (avg:0.043) Loss 1003.486267 (avg:149.113526) 
519 | Epoch: [17/30][25000/28800] Time 0.120 (avg:0.043) Loss 324.671906 (avg:148.911154) 
520 | Epoch: [17/30][26000/28800] Time 0.119 (avg:0.043) Loss 446.737549 (avg:148.961842) 
521 | Epoch: [17/30][27000/28800] Time 0.120 (avg:0.043) Loss 357.484039 (avg:149.052661) 
522 | Epoch: [17/30][28000/28800] Time 0.036 (avg:0.043) Loss 66.054649 (avg:149.133197) 
523 | Epoch: [18/30][0/28800] Time 0.178 (avg:0.178) Loss 10.338355 (avg:10.338355) 
524 | Epoch: [18/30][1000/28800] Time 0.056 (avg:0.043) Loss 881.305786 (avg:150.526575) 
525 | Epoch: [18/30][2000/28800] Time 0.021 (avg:0.042) Loss 1.079805 (avg:147.091669) 
526 | Epoch: [18/30][3000/28800] Time 0.036 (avg:0.043) Loss 199.561737 (avg:148.704552) 
527 | Epoch: [18/30][4000/28800] Time 0.014 (avg:0.043) Loss 0.898035 (avg:148.796421) 
528 | Epoch: [18/30][5000/28800] Time 0.013 (avg:0.043) Loss 4.669255 (avg:147.692546) 
529 | Epoch: [18/30][6000/28800] Time 0.088 (avg:0.043) Loss 394.066223 (avg:147.178271) 
530 | Epoch: [18/30][7000/28800] Time 0.035 (avg:0.043) Loss 82.973267 (avg:146.574164) 
531 | Epoch: [18/30][8000/28800] Time 0.013 (avg:0.043) Loss 3.120333 (avg:147.261425) 
532 | Epoch: [18/30][9000/28800] Time 0.015 (avg:0.043) Loss 3.496347 (avg:148.319240) 
533 | Epoch: [18/30][10000/28800] Time 0.035 (avg:0.043) Loss 45.704197 (avg:148.313288) 
534 | Epoch: [18/30][11000/28800] Time 0.024 (avg:0.043) Loss 0.000000 (avg:148.214815) 
535 | Epoch: [18/30][12000/28800] Time 0.015 (avg:0.043) Loss 1.520120 (avg:147.521044) 
536 | Epoch: [18/30][13000/28800] Time 0.016 (avg:0.043) Loss 3.723933 (avg:148.050059) 
537 | Epoch: [18/30][14000/28800] Time 0.059 (avg:0.043) Loss 126.248062 (avg:148.196708) 
538 | Epoch: [18/30][15000/28800] Time 0.076 (avg:0.043) Loss 295.909393 (avg:148.855532) 
539 | Epoch: [18/30][16000/28800] Time 0.035 (avg:0.043) Loss 141.145172 (avg:148.606585) 
540 | Epoch: [18/30][17000/28800] Time 0.059 (avg:0.043) Loss 212.416824 (avg:148.940239) 
541 | Epoch: [18/30][18000/28800] Time 0.056 (avg:0.043) Loss 102.921684 (avg:149.123275) 
542 | Epoch: [18/30][19000/28800] Time 0.065 (avg:0.043) Loss 153.400146 (avg:149.314828) 
543 | Epoch: [18/30][20000/28800] Time 0.052 (avg:0.043) Loss 730.565918 (avg:148.919636) 
544 | Epoch: [18/30][21000/28800] Time 0.032 (avg:0.043) Loss 189.633575 (avg:149.187522) 
545 | Epoch: [18/30][22000/28800] Time 0.043 (avg:0.043) Loss 240.130768 (avg:149.015251) 
546 | Epoch: [18/30][23000/28800] Time 0.041 (avg:0.043) Loss 310.131470 (avg:149.363738) 
547 | Epoch: [18/30][24000/28800] Time 0.052 (avg:0.043) Loss 422.989014 (avg:148.998588) 
548 | Epoch: [18/30][25000/28800] Time 0.035 (avg:0.043) Loss 167.257019 (avg:149.258860) 
549 | Epoch: [18/30][26000/28800] Time 0.056 (avg:0.043) Loss 91.240395 (avg:149.506617) 
550 | Epoch: [18/30][27000/28800] Time 0.055 (avg:0.043) Loss 57.373524 (avg:149.439366) 
551 | Epoch: [18/30][28000/28800] Time 0.034 (avg:0.043) Loss 160.349274 (avg:149.429572) 
552 | Epoch: [19/30][0/28800] Time 0.210 (avg:0.210) Loss 246.803619 (avg:246.803619) 
553 | Epoch: [19/30][1000/28800] Time 0.031 (avg:0.044) Loss 235.392380 (avg:150.569040) 
554 | Epoch: [19/30][2000/28800] Time 0.014 (avg:0.043) Loss 3.332607 (avg:147.513552) 
555 | Epoch: [19/30][3000/28800] Time 0.034 (avg:0.043) Loss 140.558136 (avg:146.592808) 
556 | Epoch: [19/30][4000/28800] Time 0.031 (avg:0.043) Loss 167.244858 (avg:148.345372) 
557 | Epoch: [19/30][5000/28800] Time 0.056 (avg:0.043) Loss 211.912766 (avg:148.746069) 
558 | Epoch: [19/30][6000/28800] Time 0.057 (avg:0.043) Loss 119.468689 (avg:149.413090) 
559 | Epoch: [19/30][7000/28800] Time 0.014 (avg:0.043) Loss 0.226736 (avg:149.350989) 
560 | Epoch: [19/30][8000/28800] Time 0.060 (avg:0.043) Loss 198.435867 (avg:149.349823) 
561 | Epoch: [19/30][9000/28800] Time 0.014 (avg:0.043) Loss 7.972675 (avg:149.006888) 
562 | Epoch: [19/30][10000/28800] Time 0.041 (avg:0.043) Loss 338.198792 (avg:148.812672) 
563 | Epoch: [19/30][11000/28800] Time 0.032 (avg:0.043) Loss 125.306480 (avg:149.741373) 
564 | Epoch: [19/30][12000/28800] Time 0.067 (avg:0.043) Loss 287.105560 (avg:149.315806) 
565 | Epoch: [19/30][13000/28800] Time 0.053 (avg:0.043) Loss 131.106735 (avg:149.261265) 
566 | Epoch: [19/30][14000/28800] Time 0.051 (avg:0.043) Loss 371.791718 (avg:149.171644) 
567 | Epoch: [19/30][15000/28800] Time 0.076 (avg:0.043) Loss 166.591599 (avg:149.170208) 
568 | Epoch: [19/30][16000/28800] Time 0.022 (avg:0.043) Loss 2.624992 (avg:148.780867) 
569 | Epoch: [19/30][17000/28800] Time 0.035 (avg:0.043) Loss 55.513733 (avg:149.169482) 
570 | Epoch: [19/30][18000/28800] Time 0.013 (avg:0.043) Loss 5.356278 (avg:149.130267) 
571 | Epoch: [19/30][19000/28800] Time 0.086 (avg:0.043) Loss 308.586365 (avg:149.362688) 
572 | Epoch: [19/30][20000/28800] Time 0.035 (avg:0.043) Loss 211.519531 (avg:149.025000) 
573 | Epoch: [19/30][21000/28800] Time 0.053 (avg:0.043) Loss 512.314087 (avg:149.213640) 
574 | Epoch: [19/30][22000/28800] Time 0.042 (avg:0.043) Loss 68.617485 (avg:149.066847) 
575 | Epoch: [19/30][23000/28800] Time 0.015 (avg:0.043) Loss 10.850912 (avg:149.192369) 
576 | Epoch: [19/30][24000/28800] Time 0.057 (avg:0.043) Loss 439.280823 (avg:149.347608) 
577 | Epoch: [19/30][25000/28800] Time 0.087 (avg:0.043) Loss 216.802963 (avg:149.455544) 
578 | Epoch: [19/30][26000/28800] Time 0.087 (avg:0.043) Loss 690.820496 (avg:149.366252) 
579 | Epoch: [19/30][27000/28800] Time 0.048 (avg:0.043) Loss 172.209030 (avg:148.916945) 
580 | Epoch: [19/30][28000/28800] Time 0.031 (avg:0.043) Loss 179.754913 (avg:148.945992) 
581 | Epoch: [20/30][0/28800] Time 0.216 (avg:0.216) Loss 107.407448 (avg:107.407448) 
582 | Epoch: [20/30][1000/28800] Time 0.056 (avg:0.043) Loss 344.964020 (avg:147.360363) 
583 | Epoch: [20/30][2000/28800] Time 0.057 (avg:0.042) Loss 205.021072 (avg:146.391117) 
584 | Epoch: [20/30][3000/28800] Time 0.032 (avg:0.042) Loss 169.610641 (avg:151.486908) 
585 | Epoch: [20/30][4000/28800] Time 0.055 (avg:0.042) Loss 113.417313 (avg:149.020654) 
586 | Epoch: [20/30][5000/28800] Time 0.014 (avg:0.042) Loss 8.241750 (avg:149.517804) 
587 | Epoch: [20/30][6000/28800] Time 0.053 (avg:0.043) Loss 205.776611 (avg:149.949350) 
588 | Epoch: [20/30][7000/28800] Time 0.120 (avg:0.042) Loss 363.091034 (avg:149.897071) 
589 | Epoch: [20/30][8000/28800] Time 0.013 (avg:0.043) Loss 1.120608 (avg:151.118396) 
590 | Epoch: [20/30][9000/28800] Time 0.057 (avg:0.043) Loss 183.177505 (avg:151.255073) 
591 | Epoch: [20/30][10000/28800] Time 0.035 (avg:0.042) Loss 107.028435 (avg:149.829368) 
592 | Epoch: [20/30][11000/28800] Time 0.021 (avg:0.042) Loss 0.000000 (avg:149.246260) 
593 | Epoch: [20/30][12000/28800] Time 0.032 (avg:0.043) Loss 100.893860 (avg:148.909379) 
594 | Epoch: [20/30][13000/28800] Time 0.056 (avg:0.043) Loss 125.335930 (avg:149.035877) 
595 | Epoch: [20/30][14000/28800] Time 0.022 (avg:0.043) Loss 0.246172 (avg:149.487824) 
596 | Epoch: [20/30][15000/28800] Time 0.056 (avg:0.043) Loss 101.158554 (avg:148.888815) 
597 | Epoch: [20/30][16000/28800] Time 0.015 (avg:0.043) Loss 0.000000 (avg:149.236914) 
598 | Epoch: [20/30][17000/28800] Time 0.087 (avg:0.043) Loss 398.093842 (avg:149.528233) 
599 | Epoch: [20/30][18000/28800] Time 0.036 (avg:0.043) Loss 479.764069 (avg:149.608763) 
600 | Epoch: [20/30][19000/28800] Time 0.049 (avg:0.043) Loss 193.439682 (avg:149.797992) 
601 | Epoch: [20/30][20000/28800] Time 0.060 (avg:0.043) Loss 207.948120 (avg:149.713397) 
602 | Epoch: [20/30][21000/28800] Time 0.064 (avg:0.043) Loss 82.225029 (avg:149.463453) 
603 | Epoch: [20/30][22000/28800] Time 0.014 (avg:0.043) Loss 0.967872 (avg:150.158339) 
604 | Epoch: [20/30][23000/28800] Time 0.066 (avg:0.043) Loss 201.533463 (avg:150.170680) 
605 | Epoch: [20/30][24000/28800] Time 0.076 (avg:0.043) Loss 86.098869 (avg:149.599373) 
606 | Epoch: [20/30][25000/28800] Time 0.041 (avg:0.043) Loss 182.117172 (avg:149.583253) 
607 | Epoch: [20/30][26000/28800] Time 0.060 (avg:0.043) Loss 47.854000 (avg:149.096851) 
608 | Epoch: [20/30][27000/28800] Time 0.036 (avg:0.043) Loss 117.499535 (avg:149.223991) 
609 | Epoch: [20/30][28000/28800] Time 0.120 (avg:0.043) Loss 357.211517 (avg:149.433956) 
610 | Epoch: [21/30][0/28800] Time 0.200 (avg:0.200) Loss 118.970215 (avg:118.970215) 
611 | Epoch: [21/30][1000/28800] Time 0.120 (avg:0.043) Loss 480.871002 (avg:143.392771) 
612 | Epoch: [21/30][2000/28800] Time 0.015 (avg:0.043) Loss 4.497313 (avg:146.721037) 
613 | Epoch: [21/30][3000/28800] Time 0.015 (avg:0.043) Loss 2.155743 (avg:147.733330) 
614 | Epoch: [21/30][4000/28800] Time 0.056 (avg:0.043) Loss 107.958214 (avg:147.526738) 
615 | Epoch: [21/30][5000/28800] Time 0.060 (avg:0.043) Loss 54.024616 (avg:148.598680) 
616 | Epoch: [21/30][6000/28800] Time 0.036 (avg:0.043) Loss 120.271080 (avg:150.997901) 
617 | Epoch: [21/30][7000/28800] Time 0.057 (avg:0.043) Loss 169.361282 (avg:149.430651) 
618 | Epoch: [21/30][8000/28800] Time 0.035 (avg:0.043) Loss 130.362488 (avg:148.613799) 
619 | Epoch: [21/30][9000/28800] Time 0.035 (avg:0.043) Loss 216.493973 (avg:149.498820) 
620 | Epoch: [21/30][10000/28800] Time 0.014 (avg:0.043) Loss 11.609165 (avg:149.974797) 
621 | Epoch: [21/30][11000/28800] Time 0.022 (avg:0.043) Loss 2.118447 (avg:150.479914) 
622 | Epoch: [21/30][12000/28800] Time 0.058 (avg:0.043) Loss 69.830467 (avg:150.868062) 
623 | Epoch: [21/30][13000/28800] Time 0.038 (avg:0.043) Loss 119.114418 (avg:150.539479) 
624 | Epoch: [21/30][14000/28800] Time 0.021 (avg:0.043) Loss 1.809788 (avg:150.559219) 
625 | Epoch: [21/30][15000/28800] Time 0.015 (avg:0.043) Loss 0.368142 (avg:150.162758) 
626 | Epoch: [21/30][16000/28800] Time 0.015 (avg:0.043) Loss 1.706194 (avg:149.764545) 
627 | Epoch: [21/30][17000/28800] Time 0.032 (avg:0.043) Loss 117.496727 (avg:149.767038) 
628 | Epoch: [21/30][18000/28800] Time 0.014 (avg:0.043) Loss 7.778365 (avg:149.474725) 
629 | Epoch: [21/30][19000/28800] Time 0.059 (avg:0.043) Loss 68.693336 (avg:149.596356) 
630 | Epoch: [21/30][20000/28800] Time 0.031 (avg:0.043) Loss 205.716217 (avg:150.057185) 
631 | Epoch: [21/30][21000/28800] Time 0.061 (avg:0.043) Loss 147.039108 (avg:150.026911) 
632 | Epoch: [21/30][22000/28800] Time 0.120 (avg:0.043) Loss 388.183075 (avg:149.991479) 
633 | Epoch: [21/30][23000/28800] Time 0.016 (avg:0.043) Loss 1.401194 (avg:149.944101) 
634 | Epoch: [21/30][24000/28800] Time 0.014 (avg:0.043) Loss 17.503042 (avg:149.280045) 
635 | Epoch: [21/30][25000/28800] Time 0.120 (avg:0.043) Loss 146.974442 (avg:149.182391) 
636 | Epoch: [21/30][26000/28800] Time 0.068 (avg:0.043) Loss 292.239014 (avg:149.229033) 
637 | Epoch: [21/30][27000/28800] Time 0.118 (avg:0.043) Loss 240.509567 (avg:149.185252) 
638 | Epoch: [21/30][28000/28800] Time 0.087 (avg:0.043) Loss 94.346939 (avg:148.976161) 
639 | Epoch: [22/30][0/28800] Time 0.175 (avg:0.175) Loss 4.760528 (avg:4.760528) 
640 | Epoch: [22/30][1000/28800] Time 0.048 (avg:0.042) Loss 283.565552 (avg:135.388302) 
641 | Epoch: [22/30][2000/28800] Time 0.017 (avg:0.042) Loss 10.881489 (avg:138.752100) 
642 | Epoch: [22/30][3000/28800] Time 0.031 (avg:0.043) Loss 120.392227 (avg:143.819047) 
643 | Epoch: [22/30][4000/28800] Time 0.032 (avg:0.043) Loss 150.585449 (avg:147.795407) 
644 | Epoch: [22/30][5000/28800] Time 0.032 (avg:0.043) Loss 215.986984 (avg:148.350603) 
645 | Epoch: [22/30][6000/28800] Time 0.017 (avg:0.043) Loss 1.403387 (avg:148.784589) 
646 | Epoch: [22/30][7000/28800] Time 0.020 (avg:0.043) Loss 0.000000 (avg:149.678451) 
647 | Epoch: [22/30][8000/28800] Time 0.057 (avg:0.043) Loss 89.189407 (avg:149.172122) 
648 | Epoch: [22/30][9000/28800] Time 0.041 (avg:0.043) Loss 224.439621 (avg:148.764843) 
649 | Epoch: [22/30][10000/28800] Time 0.033 (avg:0.043) Loss 430.544678 (avg:149.284726) 
650 | Epoch: [22/30][11000/28800] Time 0.013 (avg:0.043) Loss 6.371672 (avg:149.660028) 
651 | Epoch: [22/30][12000/28800] Time 0.025 (avg:0.043) Loss 0.402078 (avg:149.405764) 
652 | Epoch: [22/30][13000/28800] Time 0.015 (avg:0.043) Loss 0.862700 (avg:148.938308) 
653 | Epoch: [22/30][14000/28800] Time 0.119 (avg:0.043) Loss 136.415771 (avg:149.151116) 
654 | Epoch: [22/30][15000/28800] Time 0.059 (avg:0.043) Loss 335.088348 (avg:149.314180) 
655 | Epoch: [22/30][16000/28800] Time 0.087 (avg:0.043) Loss 448.889496 (avg:149.783322) 
656 | Epoch: [22/30][17000/28800] Time 0.087 (avg:0.043) Loss 644.767273 (avg:149.291382) 
657 | Epoch: [22/30][18000/28800] Time 0.041 (avg:0.043) Loss 441.781158 (avg:149.353771) 
658 | Epoch: [22/30][19000/28800] Time 0.076 (avg:0.043) Loss 103.629883 (avg:149.117981) 
659 | Epoch: [22/30][20000/28800] Time 0.021 (avg:0.043) Loss 1.128998 (avg:148.853174) 
660 | Epoch: [22/30][21000/28800] Time 0.055 (avg:0.043) Loss 208.876999 (avg:148.523212) 
661 | Epoch: [22/30][22000/28800] Time 0.067 (avg:0.043) Loss 1007.094360 (avg:148.529710) 
662 | Epoch: [22/30][23000/28800] Time 0.016 (avg:0.043) Loss 1.115340 (avg:148.797812) 
663 | Epoch: [22/30][24000/28800] Time 0.056 (avg:0.043) Loss 334.537201 (avg:148.416437) 
664 | Epoch: [22/30][25000/28800] Time 0.031 (avg:0.043) Loss 123.783897 (avg:148.503173) 
665 | Epoch: [22/30][26000/28800] Time 0.016 (avg:0.043) Loss 2.293593 (avg:148.953647) 
666 | Epoch: [22/30][27000/28800] Time 0.031 (avg:0.043) Loss 120.085327 (avg:149.276342) 
667 | Epoch: [22/30][28000/28800] Time 0.013 (avg:0.043) Loss 0.970752 (avg:149.205467) 
668 | Epoch: [23/30][0/28800] Time 0.224 (avg:0.224) Loss 131.415970 (avg:131.415970) 
669 | Epoch: [23/30][1000/28800] Time 0.014 (avg:0.043) Loss 1.478835 (avg:154.665372) 
670 | Epoch: [23/30][2000/28800] Time 0.120 (avg:0.043) Loss 942.165039 (avg:151.552555) 
671 | Epoch: [23/30][3000/28800] Time 0.065 (avg:0.043) Loss 184.050446 (avg:152.925562) 
672 | Epoch: [23/30][4000/28800] Time 0.015 (avg:0.043) Loss 0.000000 (avg:153.363033) 
673 | Epoch: [23/30][5000/28800] Time 0.060 (avg:0.043) Loss 138.617081 (avg:152.534742) 
674 | Epoch: [23/30][6000/28800] Time 0.117 (avg:0.043) Loss 109.212730 (avg:153.104382) 
675 | Epoch: [23/30][7000/28800] Time 0.119 (avg:0.043) Loss 775.232544 (avg:152.066539) 
676 | Epoch: [23/30][8000/28800] Time 0.057 (avg:0.043) Loss 209.698227 (avg:151.622978) 
677 | Epoch: [23/30][9000/28800] Time 0.035 (avg:0.043) Loss 234.169662 (avg:151.235880) 
678 | Epoch: [23/30][10000/28800] Time 0.059 (avg:0.043) Loss 115.958092 (avg:151.509580) 
679 | Epoch: [23/30][11000/28800] Time 0.031 (avg:0.043) Loss 176.018295 (avg:150.919528) 
680 | Epoch: [23/30][12000/28800] Time 0.015 (avg:0.043) Loss 3.595459 (avg:150.336690) 
681 | Epoch: [23/30][13000/28800] Time 0.058 (avg:0.043) Loss 102.948013 (avg:149.942870) 
682 | Epoch: [23/30][14000/28800] Time 0.037 (avg:0.043) Loss 133.953659 (avg:150.174359) 
683 | Epoch: [23/30][15000/28800] Time 0.037 (avg:0.043) Loss 221.227570 (avg:150.145656) 
684 | Epoch: [23/30][16000/28800] Time 0.031 (avg:0.043) Loss 136.110184 (avg:150.240130) 
685 | Epoch: [23/30][17000/28800] Time 0.087 (avg:0.043) Loss 383.356628 (avg:150.245673) 
686 | Epoch: [23/30][18000/28800] Time 0.025 (avg:0.043) Loss 0.000000 (avg:150.252380) 
687 | Epoch: [23/30][19000/28800] Time 0.036 (avg:0.043) Loss 208.791382 (avg:150.088542) 
688 | Epoch: [23/30][20000/28800] Time 0.119 (avg:0.043) Loss 377.034332 (avg:149.745585) 
689 | Epoch: [23/30][21000/28800] Time 0.013 (avg:0.043) Loss 0.704159 (avg:149.790040) 
690 | Epoch: [23/30][22000/28800] Time 0.036 (avg:0.043) Loss 234.518097 (avg:149.802888) 
691 | Epoch: [23/30][23000/28800] Time 0.036 (avg:0.043) Loss 507.807434 (avg:150.004180) 
692 | Epoch: [23/30][24000/28800] Time 0.066 (avg:0.043) Loss 295.031586 (avg:149.890180) 
693 | Epoch: [23/30][25000/28800] Time 0.057 (avg:0.043) Loss 198.713928 (avg:149.619124) 
694 | Epoch: [23/30][26000/28800] Time 0.056 (avg:0.043) Loss 99.089882 (avg:149.362288) 
695 | Epoch: [23/30][27000/28800] Time 0.022 (avg:0.043) Loss 0.601080 (avg:149.551139) 
696 | Epoch: [23/30][28000/28800] Time 0.023 (avg:0.043) Loss 0.638336 (avg:149.128980) 
697 | Epoch: [24/30][0/28800] Time 0.195 (avg:0.195) Loss 111.291679 (avg:111.291679) 
698 | Epoch: [24/30][1000/28800] Time 0.031 (avg:0.042) Loss 182.334290 (avg:147.209422) 
699 | Epoch: [24/30][2000/28800] Time 0.015 (avg:0.042) Loss 6.970479 (avg:145.505152) 
700 | Epoch: [24/30][3000/28800] Time 0.014 (avg:0.042) Loss 3.860879 (avg:142.027710) 
701 | Epoch: [24/30][4000/28800] Time 0.032 (avg:0.042) Loss 112.538834 (avg:145.812182) 
702 | Epoch: [24/30][5000/28800] Time 0.088 (avg:0.043) Loss 175.011108 (avg:146.174726) 
703 | Epoch: [24/30][6000/28800] Time 0.065 (avg:0.043) Loss 81.685280 (avg:146.641511) 
704 | Epoch: [24/30][7000/28800] Time 0.121 (avg:0.043) Loss 267.269318 (avg:147.489474) 
705 | Epoch: [24/30][8000/28800] Time 0.064 (avg:0.043) Loss 55.361664 (avg:146.454476) 
706 | Epoch: [24/30][9000/28800] Time 0.014 (avg:0.043) Loss 1.000793 (avg:147.134351) 
707 | Epoch: [24/30][10000/28800] Time 0.053 (avg:0.043) Loss 580.897156 (avg:147.461301) 
708 | Epoch: [24/30][11000/28800] Time 0.087 (avg:0.043) Loss 293.398987 (avg:147.632810) 
709 | Epoch: [24/30][12000/28800] Time 0.014 (avg:0.043) Loss 1.468499 (avg:148.035659) 
710 | Epoch: [24/30][13000/28800] Time 0.087 (avg:0.043) Loss 29.149906 (avg:148.248390) 
711 | Epoch: [24/30][14000/28800] Time 0.014 (avg:0.043) Loss 0.214933 (avg:148.195395) 
712 | Epoch: [24/30][15000/28800] Time 0.014 (avg:0.043) Loss 21.709730 (avg:148.641297) 
713 | Epoch: [24/30][16000/28800] Time 0.087 (avg:0.043) Loss 405.346344 (avg:148.848320) 
714 | Epoch: [24/30][17000/28800] Time 0.016 (avg:0.043) Loss 0.000000 (avg:149.073321) 
715 | Epoch: [24/30][18000/28800] Time 0.049 (avg:0.043) Loss 141.822525 (avg:149.199575) 
716 | Epoch: [24/30][19000/28800] Time 0.035 (avg:0.043) Loss 179.633896 (avg:149.275753) 
717 | Epoch: [24/30][20000/28800] Time 0.013 (avg:0.043) Loss 2.712296 (avg:149.267018) 
718 | Epoch: [24/30][21000/28800] Time 0.017 (avg:0.043) Loss 2.198340 (avg:149.074473) 
719 | Epoch: [24/30][22000/28800] Time 0.067 (avg:0.043) Loss 510.505676 (avg:149.268708) 
720 | Epoch: [24/30][23000/28800] Time 0.053 (avg:0.043) Loss 61.104168 (avg:149.187952) 
721 | Epoch: [24/30][24000/28800] Time 0.058 (avg:0.043) Loss 110.898293 (avg:149.204440) 
722 | Epoch: [24/30][25000/28800] Time 0.031 (avg:0.043) Loss 47.998241 (avg:149.258660) 
723 | Epoch: [24/30][26000/28800] Time 0.041 (avg:0.043) Loss 264.782410 (avg:149.005872) 
724 | Epoch: [24/30][27000/28800] Time 0.034 (avg:0.043) Loss 94.903969 (avg:149.081733) 
725 | Epoch: [24/30][28000/28800] Time 0.088 (avg:0.043) Loss 218.725296 (avg:148.984558) 
726 | Epoch: [25/30][0/28800] Time 0.175 (avg:0.175) Loss 10.831248 (avg:10.831248) 
727 | Epoch: [25/30][1000/28800] Time 0.014 (avg:0.042) Loss 7.103641 (avg:148.511615) 
728 | Epoch: [25/30][2000/28800] Time 0.037 (avg:0.042) Loss 308.517395 (avg:144.446865) 
729 | Epoch: [25/30][3000/28800] Time 0.035 (avg:0.043) Loss 92.446342 (avg:145.684144) 
730 | Epoch: [25/30][4000/28800] Time 0.057 (avg:0.043) Loss 124.058861 (avg:149.211535) 
731 | Epoch: [25/30][5000/28800] Time 0.017 (avg:0.043) Loss 27.884817 (avg:150.656034) 
732 | Epoch: [25/30][6000/28800] Time 0.017 (avg:0.043) Loss 1.743552 (avg:151.619193) 
733 | Epoch: [25/30][7000/28800] Time 0.087 (avg:0.043) Loss 567.399963 (avg:150.513554) 
734 | Epoch: [25/30][8000/28800] Time 0.034 (avg:0.043) Loss 23.400953 (avg:149.790880) 
735 | Epoch: [25/30][9000/28800] Time 0.032 (avg:0.043) Loss 22.243116 (avg:150.190889) 
736 | Epoch: [25/30][10000/28800] Time 0.016 (avg:0.043) Loss 2.760412 (avg:149.856931) 
737 | Epoch: [25/30][11000/28800] Time 0.021 (avg:0.043) Loss 3.058153 (avg:150.403269) 
738 | Epoch: [25/30][12000/28800] Time 0.053 (avg:0.043) Loss 373.866669 (avg:150.997378) 
739 | Epoch: [25/30][13000/28800] Time 0.015 (avg:0.043) Loss 5.624683 (avg:150.168829) 
740 | Epoch: [25/30][14000/28800] Time 0.048 (avg:0.043) Loss 111.384636 (avg:150.347030) 
741 | Epoch: [25/30][15000/28800] Time 0.088 (avg:0.043) Loss 105.318687 (avg:150.134614) 
742 | Epoch: [25/30][16000/28800] Time 0.055 (avg:0.043) Loss 125.381859 (avg:150.079944) 
743 | Epoch: [25/30][17000/28800] Time 0.041 (avg:0.043) Loss 181.474335 (avg:149.344605) 
744 | Epoch: [25/30][18000/28800] Time 0.057 (avg:0.043) Loss 123.567863 (avg:149.326319) 
745 | Epoch: [25/30][19000/28800] Time 0.059 (avg:0.043) Loss 49.640408 (avg:149.515937) 
746 | Epoch: [25/30][20000/28800] Time 0.023 (avg:0.043) Loss 0.235879 (avg:149.746284) 
747 | Epoch: [25/30][21000/28800] Time 0.034 (avg:0.043) Loss 201.556870 (avg:149.984252) 
748 | Epoch: [25/30][22000/28800] Time 0.014 (avg:0.043) Loss 1.200578 (avg:149.567142) 
749 | Epoch: [25/30][23000/28800] Time 0.088 (avg:0.043) Loss 123.516235 (avg:149.191853) 
750 | Epoch: [25/30][24000/28800] Time 0.087 (avg:0.043) Loss 523.713928 (avg:149.295729) 
751 | Epoch: [25/30][25000/28800] Time 0.066 (avg:0.043) Loss 313.669159 (avg:149.336069) 
752 | Epoch: [25/30][26000/28800] Time 0.052 (avg:0.043) Loss 91.705742 (avg:149.069425) 
753 | Epoch: [25/30][27000/28800] Time 0.035 (avg:0.043) Loss 118.508133 (avg:149.262314) 
754 | Epoch: [25/30][28000/28800] Time 0.057 (avg:0.043) Loss 239.043732 (avg:149.126499) 
755 | Epoch: [26/30][0/28800] Time 0.219 (avg:0.219) Loss 215.222168 (avg:215.222168) 
756 | Epoch: [26/30][1000/28800] Time 0.120 (avg:0.043) Loss 634.191650 (avg:157.117546) 
757 | Epoch: [26/30][2000/28800] Time 0.065 (avg:0.043) Loss 92.290855 (avg:156.794116) 
758 | Epoch: [26/30][3000/28800] Time 0.014 (avg:0.043) Loss 10.865592 (avg:153.978159) 
759 | Epoch: [26/30][4000/28800] Time 0.031 (avg:0.043) Loss 75.002548 (avg:152.476988) 
760 | Epoch: [26/30][5000/28800] Time 0.056 (avg:0.043) Loss 543.644226 (avg:152.714593) 
761 | Epoch: [26/30][6000/28800] Time 0.014 (avg:0.043) Loss 5.739184 (avg:151.529391) 
762 | Epoch: [26/30][7000/28800] Time 0.075 (avg:0.043) Loss 528.365112 (avg:150.733255) 
763 | Epoch: [26/30][8000/28800] Time 0.058 (avg:0.043) Loss 193.268845 (avg:150.615450) 
764 | Epoch: [26/30][9000/28800] Time 0.036 (avg:0.043) Loss 78.802345 (avg:150.570471) 
765 | Epoch: [26/30][10000/28800] Time 0.024 (avg:0.043) Loss 1.389069 (avg:150.068337) 
766 | Epoch: [26/30][11000/28800] Time 0.021 (avg:0.043) Loss 0.000000 (avg:150.448533) 
767 | Epoch: [26/30][12000/28800] Time 0.015 (avg:0.043) Loss 1.107135 (avg:150.387751) 
768 | Epoch: [26/30][13000/28800] Time 0.023 (avg:0.043) Loss 1.920595 (avg:150.497753) 
769 | Epoch: [26/30][14000/28800] Time 0.036 (avg:0.043) Loss 141.362701 (avg:150.091197) 
770 | Epoch: [26/30][15000/28800] Time 0.016 (avg:0.043) Loss 0.438279 (avg:149.931643) 
771 | Epoch: [26/30][16000/28800] Time 0.015 (avg:0.043) Loss 1.723150 (avg:149.728990) 
772 | Epoch: [26/30][17000/28800] Time 0.036 (avg:0.043) Loss 283.819550 (avg:149.956353) 
773 | Epoch: [26/30][18000/28800] Time 0.058 (avg:0.043) Loss 91.234550 (avg:149.745183) 
774 | Epoch: [26/30][19000/28800] Time 0.014 (avg:0.043) Loss 2.279078 (avg:149.883435) 
775 | Epoch: [26/30][20000/28800] Time 0.022 (avg:0.043) Loss 0.444712 (avg:149.559982) 
776 | Epoch: [26/30][21000/28800] Time 0.036 (avg:0.043) Loss 147.165451 (avg:149.647210) 
777 | Epoch: [26/30][22000/28800] Time 0.120 (avg:0.043) Loss 670.018738 (avg:149.428462) 
778 | Epoch: [26/30][23000/28800] Time 0.052 (avg:0.043) Loss 230.591599 (avg:149.262948) 
779 | Epoch: [26/30][24000/28800] Time 0.024 (avg:0.043) Loss 1.182779 (avg:149.554594) 
780 | Epoch: [26/30][25000/28800] Time 0.026 (avg:0.043) Loss 0.000000 (avg:149.412492) 
781 | Epoch: [26/30][26000/28800] Time 0.016 (avg:0.043) Loss 0.438509 (avg:149.101953) 
782 | Epoch: [26/30][27000/28800] Time 0.035 (avg:0.043) Loss 166.995773 (avg:149.136349) 
783 | Epoch: [26/30][28000/28800] Time 0.052 (avg:0.043) Loss 116.836258 (avg:149.230606) 
784 | Epoch: [27/30][0/28800] Time 0.179 (avg:0.179) Loss 3.177790 (avg:3.177790) 
785 | Epoch: [27/30][1000/28800] Time 0.035 (avg:0.042) Loss 80.900963 (avg:157.881069) 
786 | Epoch: [27/30][2000/28800] Time 0.053 (avg:0.043) Loss 1056.255493 (avg:159.034633) 
787 | Epoch: [27/30][3000/28800] Time 0.068 (avg:0.043) Loss 855.093445 (avg:153.524522) 
788 | Epoch: [27/30][4000/28800] Time 0.038 (avg:0.043) Loss 186.452881 (avg:151.856342) 
789 | Epoch: [27/30][5000/28800] Time 0.087 (avg:0.043) Loss 185.667999 (avg:152.001410) 
790 | Epoch: [27/30][6000/28800] Time 0.015 (avg:0.043) Loss 3.939406 (avg:152.267972) 
791 | Epoch: [27/30][7000/28800] Time 0.015 (avg:0.043) Loss 0.000000 (avg:150.511898) 
792 | Epoch: [27/30][8000/28800] Time 0.060 (avg:0.043) Loss 167.103622 (avg:149.350501) 
793 | Epoch: [27/30][9000/28800] Time 0.041 (avg:0.043) Loss 204.215042 (avg:149.406578) 
794 | Epoch: [27/30][10000/28800] Time 0.014 (avg:0.043) Loss 3.245559 (avg:149.009037) 
795 | Epoch: [27/30][11000/28800] Time 0.059 (avg:0.043) Loss 89.114098 (avg:148.414715) 
796 | Epoch: [27/30][12000/28800] Time 0.014 (avg:0.043) Loss 11.377410 (avg:148.346364) 
797 | Epoch: [27/30][13000/28800] Time 0.035 (avg:0.043) Loss 186.753464 (avg:148.069328) 
798 | Epoch: [27/30][14000/28800] Time 0.036 (avg:0.043) Loss 148.857254 (avg:148.470518) 
799 | Epoch: [27/30][15000/28800] Time 0.017 (avg:0.043) Loss 1.154489 (avg:149.096584) 
800 | Epoch: [27/30][16000/28800] Time 0.057 (avg:0.043) Loss 200.208862 (avg:149.243129) 
801 | Epoch: [27/30][17000/28800] Time 0.035 (avg:0.043) Loss 78.971550 (avg:149.096855) 
802 | Epoch: [27/30][18000/28800] Time 0.015 (avg:0.043) Loss 2.976479 (avg:149.260364) 
803 | Epoch: [27/30][19000/28800] Time 0.032 (avg:0.043) Loss 207.190170 (avg:148.783703) 
804 | Epoch: [27/30][20000/28800] Time 0.014 (avg:0.043) Loss 4.042011 (avg:149.145078) 
805 | Epoch: [27/30][21000/28800] Time 0.054 (avg:0.043) Loss 771.988464 (avg:149.167834) 
806 | Epoch: [27/30][22000/28800] Time 0.014 (avg:0.043) Loss 17.552999 (avg:149.239263) 
807 | Epoch: [27/30][23000/28800] Time 0.042 (avg:0.043) Loss 207.976547 (avg:149.193955) 
808 | Epoch: [27/30][24000/28800] Time 0.056 (avg:0.043) Loss 386.131989 (avg:149.282886) 
809 | Epoch: [27/30][25000/28800] Time 0.031 (avg:0.043) Loss 64.499779 (avg:149.423354) 
810 | Epoch: [27/30][26000/28800] Time 0.031 (avg:0.043) Loss 148.045563 (avg:149.488556) 
811 | Epoch: [27/30][27000/28800] Time 0.034 (avg:0.043) Loss 98.131142 (avg:149.148514) 
812 | Epoch: [27/30][28000/28800] Time 0.053 (avg:0.043) Loss 293.210632 (avg:148.845854) 
813 | Epoch: [28/30][0/28800] Time 0.195 (avg:0.195) Loss 157.956894 (avg:157.956894) 
814 | Epoch: [28/30][1000/28800] Time 0.087 (avg:0.042) Loss 284.013519 (avg:146.209407) 
815 | Epoch: [28/30][2000/28800] Time 0.036 (avg:0.043) Loss 83.232277 (avg:148.624030) 
816 | Epoch: [28/30][3000/28800] Time 0.065 (avg:0.043) Loss 275.582275 (avg:149.412474) 
817 | Epoch: [28/30][4000/28800] Time 0.014 (avg:0.043) Loss 5.917217 (avg:148.553249) 
818 | Epoch: [28/30][5000/28800] Time 0.035 (avg:0.043) Loss 511.284821 (avg:148.191663) 
819 | Epoch: [28/30][6000/28800] Time 0.015 (avg:0.043) Loss 12.480091 (avg:148.287585) 
820 | Epoch: [28/30][7000/28800] Time 0.014 (avg:0.043) Loss 12.241472 (avg:148.732730) 
821 | Epoch: [28/30][8000/28800] Time 0.013 (avg:0.043) Loss 0.805185 (avg:149.879485) 
822 | Epoch: [28/30][9000/28800] Time 0.120 (avg:0.043) Loss 497.105560 (avg:150.253129) 
823 | Epoch: [28/30][10000/28800] Time 0.014 (avg:0.043) Loss 5.857354 (avg:150.172959) 
824 | Epoch: [28/30][11000/28800] Time 0.050 (avg:0.043) Loss 76.284889 (avg:149.963219) 
825 | Epoch: [28/30][12000/28800] Time 0.014 (avg:0.043) Loss 2.679338 (avg:149.640858) 
826 | Epoch: [28/30][13000/28800] Time 0.014 (avg:0.043) Loss 9.553287 (avg:149.179460) 
827 | Epoch: [28/30][14000/28800] Time 0.064 (avg:0.043) Loss 195.665070 (avg:149.017658) 
828 | Epoch: [28/30][15000/28800] Time 0.057 (avg:0.043) Loss 218.007721 (avg:149.083534) 
829 | Epoch: [28/30][16000/28800] Time 0.034 (avg:0.043) Loss 150.842514 (avg:150.047922) 
830 | Epoch: [28/30][17000/28800] Time 0.120 (avg:0.043) Loss 464.293701 (avg:150.406022) 
831 | Epoch: [28/30][18000/28800] Time 0.058 (avg:0.043) Loss 404.287109 (avg:150.299572) 
832 | Epoch: [28/30][19000/28800] Time 0.076 (avg:0.043) Loss 241.744995 (avg:149.916714) 
833 | Epoch: [28/30][20000/28800] Time 0.077 (avg:0.043) Loss 247.945633 (avg:149.878051) 
834 | Epoch: [28/30][21000/28800] Time 0.034 (avg:0.043) Loss 123.545532 (avg:149.633936) 
835 | Epoch: [28/30][22000/28800] Time 0.014 (avg:0.043) Loss 13.726707 (avg:149.547020) 
836 | Epoch: [28/30][23000/28800] Time 0.022 (avg:0.043) Loss 6.546249 (avg:149.447623) 
837 | Epoch: [28/30][24000/28800] Time 0.055 (avg:0.043) Loss 98.425690 (avg:149.432681) 
838 | Epoch: [28/30][25000/28800] Time 0.049 (avg:0.043) Loss 53.510403 (avg:149.468621) 
839 | Epoch: [28/30][26000/28800] Time 0.075 (avg:0.043) Loss 191.380127 (avg:149.554588) 
840 | Epoch: [28/30][27000/28800] Time 0.026 (avg:0.043) Loss 3.927336 (avg:149.272727) 
841 | Epoch: [28/30][28000/28800] Time 0.088 (avg:0.043) Loss 367.425507 (avg:149.194273) 
842 | Epoch: [29/30][0/28800] Time 0.240 (avg:0.240) Loss 636.043335 (avg:636.043335) 
843 | Epoch: [29/30][1000/28800] Time 0.088 (avg:0.043) Loss 196.375793 (avg:155.615315) 
844 | Epoch: [29/30][2000/28800] Time 0.056 (avg:0.043) Loss 237.561020 (avg:157.300380) 
845 | Epoch: [29/30][3000/28800] Time 0.033 (avg:0.043) Loss 201.309067 (avg:153.655429) 
846 | Epoch: [29/30][4000/28800] Time 0.031 (avg:0.043) Loss 94.737465 (avg:152.609412) 
847 | Epoch: [29/30][5000/28800] Time 0.017 (avg:0.043) Loss 1.069966 (avg:154.688529) 
848 | Epoch: [29/30][6000/28800] Time 0.076 (avg:0.043) Loss 90.349754 (avg:153.460752) 
849 | Epoch: [29/30][7000/28800] Time 0.014 (avg:0.043) Loss 1.463422 (avg:152.072707) 
850 | Epoch: [29/30][8000/28800] Time 0.021 (avg:0.043) Loss 3.517265 (avg:150.702455) 
851 | Epoch: [29/30][9000/28800] Time 0.026 (avg:0.043) Loss 1.216110 (avg:150.430634) 
852 | Epoch: [29/30][10000/28800] Time 0.052 (avg:0.043) Loss 225.364609 (avg:150.299208) 
853 | Epoch: [29/30][11000/28800] Time 0.076 (avg:0.043) Loss 290.018646 (avg:149.570841) 
854 | Epoch: [29/30][12000/28800] Time 0.035 (avg:0.043) Loss 105.741722 (avg:149.469815) 
855 | Epoch: [29/30][13000/28800] Time 0.027 (avg:0.043) Loss 0.000000 (avg:149.128173) 
856 | Epoch: [29/30][14000/28800] Time 0.031 (avg:0.043) Loss 92.920456 (avg:148.991338) 
857 | Epoch: [29/30][15000/28800] Time 0.057 (avg:0.043) Loss 63.408314 (avg:148.976461) 
858 | Epoch: [29/30][16000/28800] Time 0.050 (avg:0.043) Loss 274.618622 (avg:149.379635) 
859 | Epoch: [29/30][17000/28800] Time 0.015 (avg:0.043) Loss 1.350945 (avg:149.285152) 
860 | Epoch: [29/30][18000/28800] Time 0.119 (avg:0.043) Loss 547.746277 (avg:149.040805) 
861 | Epoch: [29/30][19000/28800] Time 0.120 (avg:0.043) Loss 257.943237 (avg:149.366009) 
862 | Epoch: [29/30][20000/28800] Time 0.036 (avg:0.043) Loss 117.075768 (avg:149.249146) 
863 | Epoch: [29/30][21000/28800] Time 0.014 (avg:0.043) Loss 28.177923 (avg:148.604479) 
864 | Epoch: [29/30][22000/28800] Time 0.036 (avg:0.043) Loss 25.938597 (avg:148.440296) 
865 | Epoch: [29/30][23000/28800] Time 0.032 (avg:0.043) Loss 269.534515 (avg:148.506315) 
866 | Epoch: [29/30][24000/28800] Time 0.056 (avg:0.043) Loss 198.760223 (avg:148.397196) 
867 | Epoch: [29/30][25000/28800] Time 0.067 (avg:0.043) Loss 1074.802368 (avg:148.973048) 
868 | Epoch: [29/30][26000/28800] Time 0.031 (avg:0.043) Loss 220.346024 (avg:148.798300) 
869 | Epoch: [29/30][27000/28800] Time 0.015 (avg:0.043) Loss 1.910797 (avg:149.015462) 
870 | Epoch: [29/30][28000/28800] Time 0.064 (avg:0.043) Loss 117.319336 (avg:149.199048) 
871 | 


--------------------------------------------------------------------------------
/train_RCF.py:
--------------------------------------------------------------------------------
  1 | #!/user/bin/python
  2 | # coding=utf-8
  3 | import os, sys
  4 | import numpy as np
  5 | from PIL import Image
  6 | import cv2
  7 | import shutil
  8 | import argparse
  9 | import time
 10 | import datetime
 11 | import torch
 12 | import torch.nn as nn
 13 | import torch.nn.functional as F
 14 | from torch.optim import lr_scheduler
 15 | import torchvision
 16 | import torchvision.transforms as transforms
 17 | import matplotlib
 18 | matplotlib.use('Agg')
 19 | import matplotlib.pyplot as plt
 20 | from data_loader import BSDS_RCFLoader
 21 | from models import RCF
 22 | from functions import  cross_entropy_loss_RCF, SGD_caffe
 23 | from torch.utils.data import DataLoader, sampler
 24 | from utils import Logger, Averagvalue, save_checkpoint, load_vgg16pretrain
 25 | from os.path import join, split, isdir, isfile, splitext, split, abspath, dirname
 26 | from torchvision import models
 27 | from torchsummary import summary
 28 | 
 29 | parser = argparse.ArgumentParser(description='PyTorch Training')
 30 | parser.add_argument('--batch_size', default=1, type=int, metavar='BT',
 31 |                     help='batch size')
 32 | # =============== optimizer
 33 | parser.add_argument('--lr', '--learning_rate', default=1e-6, type=float,
 34 |                     metavar='LR', help='initial learning rate')
 35 | parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
 36 |                     help='momentum')
 37 | parser.add_argument('--weight_decay', '--wd', default=2e-4, type=float,
 38 |                     metavar='W', help='default weight decay')
 39 | parser.add_argument('--stepsize', default=3, type=int, 
 40 |                     metavar='SS', help='learning rate step size')
 41 | parser.add_argument('--gamma', '--gm', default=0.1, type=float,
 42 |                     help='learning rate decay parameter: Gamma')
 43 | parser.add_argument('--maxepoch', default=30, type=int, metavar='N',
 44 |                     help='number of total epochs to run')
 45 | parser.add_argument('--itersize', default=10, type=int,
 46 |                     metavar='IS', help='iter size')
 47 | # =============== misc
 48 | parser.add_argument('--start_epoch', default=0, type=int, metavar='N',
 49 |                     help='manual epoch number (useful on restarts)')
 50 | parser.add_argument('--print_freq', '-p', default=1000, type=int,
 51 |                     metavar='N', help='print frequency (default: 50)')
 52 | parser.add_argument('--gpu', default='0', type=str,
 53 |                     help='GPU ID')
 54 | parser.add_argument('--resume', default='', type=str, metavar='PATH',
 55 |                     help='path to latest checkpoint (default: none)')
 56 | parser.add_argument('--tmp', help='tmp folder', default='tmp/RCF')
 57 | # ================ dataset
 58 | # parser.add_argument('--dataset', help='root folder of dataset', default='data/HED-BSDS_PASCAL')
 59 | parser.add_argument('--dataset', help='root folder of dataset', default='/home/itlchennai/Edge_Analysis/pack_1/pack_1/data/HED-BSDS/')
 60 | 
 61 | args = parser.parse_args()
 62 | 
 63 | os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"   # see issue #152
 64 | os.environ["CUDA_VISIBLE_DEVICES"] = args.gpu
 65 | 
 66 | THIS_DIR = abspath(dirname(__file__))
 67 | TMP_DIR = join(THIS_DIR, args.tmp)
 68 | if not isdir(TMP_DIR):
 69 |   os.makedirs(TMP_DIR)
 70 | # print('***', args.lr)
 71 | def main():
 72 |     args.cuda = True
 73 |     # dataset
 74 |     # train_dataset = BSDS_RCFLoader(root=args.dataset, lst= "train_pair.lst")
 75 |     # test_dataset = BSDS_RCFLoader(root=args.dataset,  lst= "test.lst")
 76 |     train_dataset = BSDS_RCFLoader(root=args.dataset, split= "train")
 77 |     test_dataset = BSDS_RCFLoader(root=args.dataset,  split= "test")
 78 |     train_loader = DataLoader(
 79 |         train_dataset, batch_size=args.batch_size,
 80 |         num_workers=8, drop_last=True,shuffle=True)
 81 |     test_loader = DataLoader(
 82 |         test_dataset, batch_size=args.batch_size,
 83 |         num_workers=8, drop_last=True,shuffle=False)
 84 |     with open('/home/itlchennai/Edge_Analysis/pack_1/pack_1/data/HED-BSDS/test.lst', 'r') as f:
 85 |         test_list = f.readlines()
 86 |     test_list = [split(i.rstrip())[1] for i in test_list]
 87 |     assert len(test_list) == len(test_loader), "%d vs %d" % (len(test_list), len(test_loader))
 88 | 
 89 |     # model
 90 | 
 91 |     model = RCF()
 92 |     # print(model)
 93 | 
 94 |     model.cuda()
 95 |     model.apply(weights_init)
 96 |     load_vgg16pretrain(model)
 97 | 
 98 |     if args.resume:
 99 |         if isfile(args.resume): 
100 |             # print("=> loading checkpoint '{}'".format(args.resume))
101 |             checkpoint = torch.load(args.resume)
102 |             model.load_state_dict(checkpoint['state_dict'])
103 |             # print("=> loaded checkpoint '{}'".format(args.resume))
104 |         # else:
105 |             # print("=> no checkpoint found at '{}'".format(args.resume))
106 |     
107 |     #tune lr
108 |     net_parameters_id = {}
109 |     net = model
110 |     for pname, p in net.named_parameters():
111 |         if pname in ['conv1_1.weight','conv1_2.weight',
112 |                      'conv2_1.weight','conv2_2.weight',
113 |                      'conv3_1.weight','conv3_2.weight','conv3_3.weight',
114 |                      'conv4_1.weight','conv4_2.weight','conv4_3.weight']:
115 |             # print(pname, 'lr:1 de:1')
116 |             if 'conv1-4.weight' not in net_parameters_id:
117 |                 net_parameters_id['conv1-4.weight'] = []
118 |             net_parameters_id['conv1-4.weight'].append(p)
119 |         elif pname in ['conv1_1.bias','conv1_2.bias',
120 |                        'conv2_1.bias','conv2_2.bias',
121 |                        'conv3_1.bias','conv3_2.bias','conv3_3.bias',
122 |                        'conv4_1.bias','conv4_2.bias','conv4_3.bias']:
123 |             # print(pname, 'lr:2 de:0')
124 |             if 'conv1-4.bias' not in net_parameters_id:
125 |                 net_parameters_id['conv1-4.bias'] = []
126 |             net_parameters_id['conv1-4.bias'].append(p)
127 |         elif pname in ['conv5_1.weight','conv5_2.weight','conv5_3.weight']:
128 |             # print(pname, 'lr:100 de:1')
129 |             if 'conv5.weight' not in net_parameters_id:
130 |                 net_parameters_id['conv5.weight'] = []
131 |             net_parameters_id['conv5.weight'].append(p)
132 |         elif pname in ['conv5_1.bias','conv5_2.bias','conv5_3.bias'] :
133 |             # print(pname, 'lr:200 de:0')
134 |             if 'conv5.bias' not in net_parameters_id:
135 |                 net_parameters_id['conv5.bias'] = []
136 |             net_parameters_id['conv5.bias'].append(p)
137 |         elif pname in ['conv1_1_down.weight','conv1_2_down.weight',
138 |                        'conv2_1_down.weight','conv2_2_down.weight',
139 |                        'conv3_1_down.weight','conv3_2_down.weight','conv3_3_down.weight',
140 |                        'conv4_1_down.weight','conv4_2_down.weight','conv4_3_down.weight',
141 |                        'conv5_1_down.weight','conv5_2_down.weight','conv5_3_down.weight']:
142 |             # print(pname, 'lr:0.1 de:1')
143 |             if 'conv_down_1-5.weight' not in net_parameters_id:
144 |                 net_parameters_id['conv_down_1-5.weight'] = []
145 |             net_parameters_id['conv_down_1-5.weight'].append(p)
146 |         elif pname in ['conv1_1_down.bias','conv1_2_down.bias',
147 |                        'conv2_1_down.bias','conv2_2_down.bias',
148 |                        'conv3_1_down.bias','conv3_2_down.bias','conv3_3_down.bias',
149 |                        'conv4_1_down.bias','conv4_2_down.bias','conv4_3_down.bias',
150 |                        'conv5_1_down.bias','conv5_2_down.bias','conv5_3_down.bias']:
151 |             # print(pname, 'lr:0.2 de:0')
152 |             if 'conv_down_1-5.bias' not in net_parameters_id:
153 |                 net_parameters_id['conv_down_1-5.bias'] = []
154 |             net_parameters_id['conv_down_1-5.bias'].append(p)
155 |         elif pname in ['score_dsn1.weight','score_dsn2.weight','score_dsn3.weight',
156 |                        'score_dsn4.weight','score_dsn5.weight']:
157 |             # print(pname, 'lr:0.01 de:1')
158 |             if 'score_dsn_1-5.weight' not in net_parameters_id:
159 |                 net_parameters_id['score_dsn_1-5.weight'] = []
160 |             net_parameters_id['score_dsn_1-5.weight'].append(p)
161 |         elif pname in ['score_dsn1.bias','score_dsn2.bias','score_dsn3.bias',
162 |                        'score_dsn4.bias','score_dsn5.bias']:
163 |             # print(pname, 'lr:0.02 de:0')
164 |             if 'score_dsn_1-5.bias' not in net_parameters_id:
165 |                 net_parameters_id['score_dsn_1-5.bias'] = []
166 |             net_parameters_id['score_dsn_1-5.bias'].append(p)
167 |         elif pname in ['score_final.weight']:
168 |             # print(pname, 'lr:0.001 de:1')
169 |             if 'score_final.weight' not in net_parameters_id:
170 |                 net_parameters_id['score_final.weight'] = []
171 |             net_parameters_id['score_final.weight'].append(p)
172 |         elif pname in ['score_final.bias']:
173 |             # print(pname, 'lr:0.002 de:0')
174 |             if 'score_final.bias' not in net_parameters_id:
175 |                 net_parameters_id['score_final.bias'] = []
176 |             net_parameters_id['score_final.bias'].append(p)
177 | 
178 |     optimizer = torch.optim.SGD([
179 |             {'params': net_parameters_id['conv1-4.weight']      , 'lr': args.lr*1    , 'weight_decay': args.weight_decay},
180 |             {'params': net_parameters_id['conv1-4.bias']        , 'lr': args.lr*2    , 'weight_decay': 0.},
181 |             {'params': net_parameters_id['conv5.weight']        , 'lr': args.lr*100  , 'weight_decay': args.weight_decay},
182 |             {'params': net_parameters_id['conv5.bias']          , 'lr': args.lr*200  , 'weight_decay': 0.},
183 |             {'params': net_parameters_id['conv_down_1-5.weight'], 'lr': args.lr*0.1  , 'weight_decay': args.weight_decay},
184 |             {'params': net_parameters_id['conv_down_1-5.bias']  , 'lr': args.lr*0.2  , 'weight_decay': 0.},
185 |             {'params': net_parameters_id['score_dsn_1-5.weight'], 'lr': args.lr*0.01 , 'weight_decay': args.weight_decay},
186 |             {'params': net_parameters_id['score_dsn_1-5.bias']  , 'lr': args.lr*0.02 , 'weight_decay': 0.},
187 |             {'params': net_parameters_id['score_final.weight']  , 'lr': args.lr*0.001, 'weight_decay': args.weight_decay},
188 |             {'params': net_parameters_id['score_final.bias']    , 'lr': args.lr*0.002, 'weight_decay': 0.},
189 |         ], lr=args.lr, momentum=args.momentum, weight_decay=args.weight_decay)
190 |     scheduler = lr_scheduler.StepLR(optimizer, step_size=args.stepsize, gamma=args.gamma)
191 | 
192 | 
193 |     # optimizer = torch.optim.Adam([
194 |     #         {'params': net_parameters_id['conv1-4.weight']      , 'lr': args.lr*1    , 'weight_decay': args.weight_decay},
195 |     #         {'params': net_parameters_id['conv1-4.bias']        , 'lr': args.lr*2    , 'weight_decay': 0.},
196 |     #         {'params': net_parameters_id['conv5.weight']        , 'lr': args.lr*100  , 'weight_decay': args.weight_decay},
197 |     #         {'params': net_parameters_id['conv5.bias']          , 'lr': args.lr*200  , 'weight_decay': 0.},
198 |     #         {'params': net_parameters_id['conv_down_1-5.weight'], 'lr': args.lr*0.1  , 'weight_decay': args.weight_decay},
199 |     #         {'params': net_parameters_id['conv_down_1-5.bias']  , 'lr': args.lr*0.2  , 'weight_decay': 0.},
200 |     #         {'params': net_parameters_id['score_dsn_1-5.weight'], 'lr': args.lr*0.01 , 'weight_decay': args.weight_decay},
201 |     #         {'params': net_parameters_id['score_dsn_1-5.bias']  , 'lr': args.lr*0.02 , 'weight_decay': 0.},
202 |     #         {'params': net_parameters_id['score_final.weight']  , 'lr': args.lr*0.001, 'weight_decay': args.weight_decay},
203 |     #         {'params': net_parameters_id['score_final.bias']    , 'lr': args.lr*0.002, 'weight_decay': 0.},
204 |     #     ], lr=args.lr, betas=(0.9, 0.99), weight_decay=args.weight_decay)
205 |     # scheduler = lr_scheduler.StepLR(optimizer, step_size=args.stepsize, gamma=args.gamma)
206 |     
207 |     # log
208 |     log = Logger(join(TMP_DIR, '%s-%d-log.txt' %('sgd',args.lr)))
209 |     sys.stdout = log
210 | 
211 |     train_loss = []
212 |     train_loss_detail = []
213 |     for epoch in range(args.start_epoch, args.maxepoch):
214 |         if epoch == 0:
215 |             # print("Performing initial testing...")
216 |             multiscale_test(model, test_loader, epoch=epoch, test_list=test_list,
217 |                  save_dir = join(TMP_DIR, 'initial-testing-record'))
218 | 
219 |         tr_avg_loss, tr_detail_loss = train(
220 |             train_loader, model, optimizer, epoch,
221 |             save_dir = join(TMP_DIR, 'epoch-%d-training-record' % epoch))
222 |         test(model, test_loader, epoch=epoch, test_list=test_list,
223 |             save_dir = join(TMP_DIR, 'epoch-%d-testing-record-view' % epoch))
224 |         multiscale_test(model, test_loader, epoch=epoch, test_list=test_list,
225 |             save_dir = join(TMP_DIR, 'epoch-%d-testing-record' % epoch))
226 |         log.flush() # write log
227 |         # Save checkpoint
228 |         save_file = os.path.join(TMP_DIR, 'checkpoint_epoch{}.pth'.format(epoch))
229 |         save_checkpoint({
230 |             'epoch': epoch,
231 |             'state_dict': model.state_dict(),
232 |             'optimizer': optimizer.state_dict()
233 |                          }, filename=save_file)
234 |         scheduler.step() # will adjust learning rate
235 |         # save train/val loss/accuracy, save every epoch in case of early stop
236 |         train_loss.append(tr_avg_loss)
237 |         train_loss_detail += tr_detail_loss
238 | 
239 | def train(train_loader, model, optimizer, epoch, save_dir):
240 |     batch_time = Averagvalue()
241 |     data_time = Averagvalue()
242 |     losses = Averagvalue()
243 |     # switch to train mode
244 |     model.train()
245 |     end = time.time()
246 |     epoch_loss = []
247 |     counter = 0
248 |     for i, (image, label) in enumerate(train_loader):
249 |         # measure data loading time
250 |         data_time.update(time.time() - end)
251 |         image, label = image.cuda(), label.cuda()
252 |         outputs = model(image)
253 |         loss = torch.zeros(1).cuda()
254 |         for o in outputs:
255 |             loss = loss + cross_entropy_loss_RCF(o, label)
256 |         counter += 1
257 |         loss = loss / args.itersize
258 |         loss.backward()
259 |         if counter == args.itersize:
260 |             optimizer.step()
261 |             optimizer.zero_grad()
262 |             counter = 0
263 |         # measure accuracy and record loss
264 |         losses.update(loss.item(), image.size(0))
265 |         epoch_loss.append(loss.item())
266 |         batch_time.update(time.time() - end)
267 |         end = time.time()
268 |         # display and logging
269 |         if not isdir(save_dir):
270 |             os.makedirs(save_dir)
271 |         if i % args.print_freq == 0:
272 |             info = 'Epoch: [{0}/{1}][{2}/{3}] '.format(epoch, args.maxepoch, i, len(train_loader)) + \
273 |                    'Time {batch_time.val:.3f} (avg:{batch_time.avg:.3f}) '.format(batch_time=batch_time) + \
274 |                    'Loss {loss.val:f} (avg:{loss.avg:f}) '.format(
275 |                        loss=losses)
276 |             print(info)
277 |             label_out = torch.eq(label, 1).float()
278 |             outputs.append(label_out)
279 |             _, _, H, W = outputs[0].shape
280 |             all_results = torch.zeros((len(outputs), 1, H, W))
281 |             for j in range(len(outputs)):
282 |                 all_results[j, 0, :, :] = outputs[j][0, 0, :, :]
283 |             torchvision.utils.save_image(1-all_results, join(save_dir, "iter-%d.jpg" % i))
284 |         # save checkpoint
285 |     save_checkpoint({
286 |         'epoch': epoch,
287 |         'state_dict': model.state_dict(),
288 |         'optimizer': optimizer.state_dict()
289 |             }, filename=join(save_dir, "epoch-%d-checkpoint.pth" % epoch))
290 | 
291 |     return losses.avg, epoch_loss
292 | 
293 | def test(model, test_loader, epoch, test_list, save_dir):
294 |     model.eval()
295 |     if not isdir(save_dir):
296 |         os.makedirs(save_dir)
297 |     for idx, image in enumerate(test_loader):
298 |         image = image.cuda()
299 |         _, _, H, W = image.shape
300 |         results = model(image)
301 |         result = torch.squeeze(results[-1].detach()).cpu().numpy()
302 |         results_all = torch.zeros((len(results), 1, H, W))
303 |         for i in range(len(results)):
304 |           results_all[i, 0, :, :] = results[i]
305 |         filename = splitext(test_list[idx])[0]
306 |         torchvision.utils.save_image(1-results_all, join(save_dir, "%s.jpg" % filename))
307 |         result = Image.fromarray((result * 255).astype(np.uint8))
308 |         result.save(join(save_dir, "%s.png" % filename))
309 |         # print("Running test [%d/%d]" % (idx + 1, len(test_loader)))
310 | # torch.nn.functional.upsample(input, size=None, scale_factor=None, mode='nearest', align_corners=None)
311 | 
312 | def multiscale_test(model, test_loader, epoch, test_list, save_dir):
313 |     model.eval()
314 |     if not isdir(save_dir):
315 |         os.makedirs(save_dir)
316 |     scale = [0.5, 1, 1.5]
317 |     for idx, image in enumerate(test_loader):
318 |         image = image[0]
319 |         image_in = image.numpy().transpose((1,2,0))
320 |         _, H, W = image.shape
321 |         multi_fuse = np.zeros((H, W), np.float32)
322 |         for k in range(0, len(scale)):
323 |             im_ = cv2.resize(image_in, None, fx=scale[k], fy=scale[k], interpolation=cv2.INTER_LINEAR)
324 |             im_ = im_.transpose((2,0,1))
325 |             results = model(torch.unsqueeze(torch.from_numpy(im_).cuda(), 0))
326 |             result = torch.squeeze(results[-1].detach()).cpu().numpy()
327 |             fuse = cv2.resize(result, (W, H), interpolation=cv2.INTER_LINEAR)
328 |             multi_fuse += fuse
329 |         multi_fuse = multi_fuse / len(scale)
330 |         ### rescale trick suggested by jiangjiang
331 |         # multi_fuse = (multi_fuse - multi_fuse.min()) / (multi_fuse.max() - multi_fuse.min())
332 |         filename = splitext(test_list[idx])[0]
333 |         result_out = Image.fromarray(((1-multi_fuse) * 255).astype(np.uint8))
334 |         result_out.save(join(save_dir, "%s.jpg" % filename))
335 |         result_out_test = Image.fromarray((multi_fuse * 255).astype(np.uint8))
336 |         result_out_test.save(join(save_dir, "%s.png" % filename))
337 |         # print("Running test [%d/%d]" % (idx + 1, len(test_loader)))
338 | 
339 | def weights_init(m):
340 |     if isinstance(m, nn.Conv2d):
341 |         # xavier(m.weight.data)
342 |         m.weight.data.normal_(0, 0.01)
343 |         if m.weight.data.shape == torch.Size([1, 5, 1, 1]):
344 |             # for new_score_weight
345 |             torch.nn.init.constant_(m.weight, 0.2) # as per https://github.com/yun-liu/rcf
346 |         if m.bias is not None:
347 |             m.bias.data.zero_()
348 | 
349 | if __name__ == '__main__':
350 |     startinitial_time = time.time()
351 |     main()
352 |     # print("--- %s seconds ---" % (time.time() - startinitial_time))
353 | 


--------------------------------------------------------------------------------
/utils.py:
--------------------------------------------------------------------------------
  1 | import os, sys
  2 | import torch
  3 | import torch.nn as nn
  4 | import torch.nn.functional as F
  5 | import torchvision.models as models
  6 | import torch.autograd.variable as Variable
  7 | import numpy as np
  8 | import scipy.io as sio
  9 | from os.path import join as pjoin
 10 | #from skimage.transform import resize
 11 | #from models import HiFi1Edge
 12 | import skimage.io as io
 13 | import time
 14 | import skimage
 15 | import warnings
 16 | from PIL import Image
 17 | 
 18 | class Logger(object):
 19 |   def __init__(self, fpath=None):
 20 |     self.console = sys.stdout
 21 |     self.file = None
 22 |     if fpath is not None:
 23 |       self.file = open(fpath, 'w')
 24 | 
 25 |   def __del__(self):
 26 |     self.close()
 27 | 
 28 |   def __enter__(self):
 29 |     pass
 30 | 
 31 |   def __exit__(self, *args):
 32 |     self.close()
 33 | 
 34 |   def write(self, msg):
 35 |     self.console.write(msg)
 36 |     if self.file is not None:
 37 |         self.file.write(msg)
 38 | 
 39 |   def flush(self):
 40 |     self.console.flush()
 41 |     if self.file is not None:
 42 |         self.file.flush()
 43 |         os.fsync(self.file.fileno())
 44 | 
 45 |   def close(self):
 46 |     self.console.close()
 47 |     if self.file is not None:
 48 |         self.file.close()
 49 | 
 50 | class Averagvalue(object):
 51 |     """Computes and stores the average and current value"""
 52 | 
 53 |     def __init__(self):
 54 |         self.reset()
 55 | 
 56 |     def reset(self):
 57 |         self.val = 0
 58 |         self.avg = 0
 59 |         self.sum = 0
 60 |         self.count = 0
 61 | 
 62 |     def update(self, val, n=1):
 63 |         self.val = val
 64 |         self.sum += val * n
 65 |         self.count += n
 66 |         self.avg = self.sum / self.count
 67 | 
 68 | def save_checkpoint(state, filename='checkpoint.pth'):
 69 |     torch.save(state, filename)
 70 | 
 71 | def load_pretrained(model, fname, optimizer=None):
 72 |     """
 73 |     resume training from previous checkpoint
 74 |     :param fname: filename(with path) of checkpoint file
 75 |     :return: model, optimizer, checkpoint epoch
 76 |     """
 77 |     if os.path.isfile(fname):
 78 |         print("=> loading checkpoint '{}'".format(fname))
 79 |         checkpoint = torch.load(fname)
 80 |         model.load_state_dict(checkpoint['state_dict'])
 81 |         if optimizer is not None:
 82 |             optimizer.load_state_dict(checkpoint['optimizer'])
 83 |             return model, optimizer, checkpoint['epoch']
 84 |         else:
 85 |             return model, checkpoint['epoch']
 86 |     else:
 87 |         print("=> no checkpoint found at '{}'".format(fname))
 88 | 
 89 | def load_vgg16pretrain(model, vggmodel='vgg16convs.mat'):
 90 |     vgg16 = sio.loadmat(vggmodel)
 91 |     torch_params =  model.state_dict()
 92 |     
 93 |     for k in vgg16.keys():
 94 |         name_par = k.split('-')
 95 |         size = len(name_par)
 96 |         if size  == 2:
 97 |             name_space = name_par[0] + '.' + name_par[1]            
 98 |             data = np.squeeze(vgg16[k])
 99 |             torch_params[name_space] = torch.from_numpy(data)    
100 |     model.load_state_dict(torch_params)
101 | 
102 | def load_vgg16pretrain_half(model, vggmodel='vgg16convs.mat'):
103 |     vgg16 = sio.loadmat(vggmodel)
104 |     torch_params =  model.state_dict()
105 |     for k in vgg16.keys():
106 |         name_par = k.split('-')
107 |         size = len(name_par)
108 |         if size  == 2:
109 |             name_space = name_par[0] + '.' + name_par[1]
110 |             data = np.squeeze(vgg16[k])
111 |             shape = data.shape
112 |             index = int(shape[0]/2)
113 |             if len(shape) == 1:
114 |                 data = data[:index]
115 |             else:
116 |                 data = data[:index,:,:,:]
117 |             torch_params[name_space] = torch.from_numpy(data)
118 |     model.load_state_dict(torch_params)
119 | 
120 | def load_fsds_caffe(model, fsdsmodel='caffe-fsds.mat'):
121 |     fsds = sio.loadmat(fsdsmodel)
122 |     torch_params =  model.state_dict()
123 |     for k in fsds.keys():
124 |         name_par = k.split('-')
125 | 
126 |         size = len(name_par)
127 | 
128 |         data = np.squeeze(fsds[k])
129 | 
130 | 
131 |         if 'upsample' in name_par:
132 |            # print('skip upsample')
133 |             continue 
134 | 
135 | 
136 |         if size  == 2:
137 |             name_space = name_par[0] + '.' + name_par[1]
138 |             data = np.squeeze(fsds[k])
139 |             if data.ndim==2:
140 |                 data = np.reshape(data, (data.shape[0], data.shape[1]))
141 | 
142 |             torch_params[name_space] = torch.from_numpy(data)
143 | 
144 |         if size  == 3:
145 |            # if 'bias' in name_par:
146 |             #    continue
147 | 
148 |             name_space = name_par[0] + '_' + name_par[1]+ '.' + name_par[2]
149 |             data = np.squeeze(fsds[k])
150 |            # print(data.shape)
151 |             if data.ndim==2:
152 |                # print (data.shape[0])
153 |                 data = np.reshape(data,(data.shape[0], data.shape[1]))
154 |             if data.ndim==1 :                
155 |                 data = np.reshape(data, (1, len(data), 1, 1))
156 |             if data.ndim==0:
157 |                 data = np.reshape(data, (1))
158 | 
159 |             torch_params[name_space] = torch.from_numpy(data)
160 | 
161 |         if size == 4:
162 |            # if 'bias' in name_par:
163 |             #    continue
164 |             data = np.squeeze(fsds[k])
165 |             name_space = name_par[0] + '_' + name_par[1] + name_par[2] + '.' + name_par[3]
166 |             if data.ndim==2:
167 |                 data = np.reshape(data,(data.shape[0], data.shape[1], 1, 1))
168 | 
169 |             torch_params[name_space] = torch.from_numpy(data)
170 | 
171 |     model.load_state_dict(torch_params)
172 |     print('loaded')
173 | 


--------------------------------------------------------------------------------