├── .gitignore ├── LICENSE ├── README.md ├── carla_loader.py ├── carla_net.py ├── docker ├── carla_cil_compose │ └── docker-compose.yml └── docker_build │ └── Dockerfile ├── helper.py ├── main.py └── unit_test.py /.gitignore: -------------------------------------------------------------------------------- 1 | *ipynb_checkpoints 2 | *.pyc 3 | __pycache__ 4 | bk 5 | logs 6 | test 7 | runs 8 | save_models 9 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2018, Lei Tai 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # carla_cil_pytorch 2 | 3 | 4 | The pytorch implementation to train the uncertain aware imitation learning policy in "Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective". 5 | 6 | ## Requirements 7 | python 3.6 8 | pytorch > 0.4.0 9 | tensorboardX 10 | opencv 11 | imagaug 12 | h5py 13 | 14 | please check ***docker/docker_build/Dockerfile*** for details. 15 | 16 | ## Train 17 | **train-dir** and **eval-dir** should point to where the [Carla dataset](https://github.com/carla-simulator/imitation-learning/blob/master/README.md) located. 18 | Please check our [paper](https://arxiv.org/abs/1903.00821) that how we split the train and eval dataset. 19 | ``` 20 | $ python main.py 21 | --batch-size 1000 22 | --workers 24 23 | --speed-weight 1 24 | --learning-rate 0.0001 25 | --lr-gamma 0.5 26 | --lr-step 10 27 | --train-dir "/path/to/the/training_data" 28 | --eval-dir "/path/to/the/eval_data" 29 | --net-structure 2 30 | --gpu 0 31 | --id name_of_policy 32 | ``` 33 | Check the training log through tensorboard. 34 | ``` 35 | $ tensorboard --logdir runs 36 | ``` 37 | 38 | ## Docker 39 | Revise the path of the dataset and this repo in ***docker/carla_cil_compose/docker-compose.yml***. 40 | docker-compose 2.3 and nvidia-docker 2 are required. 41 | 42 | ``` 43 | $ cd docker/carla_cil_compose 44 | $ docker-compose up -d 45 | ``` 46 | We can still use tensorboard to check the log out of the docker. 47 | 48 | ## Dataset 49 | Please check the original [dataset](https://github.com/carla-simulator/imitation-learning/blob/master/README.md) of Carla Imitation Learning. 50 | Please check this [issue](https://github.com/carla-simulator/imitation-learning/issues/1) for data augmentation. 51 | 52 | ## Benchmark 53 | Please reference [carla_cil_pytorch_eval](https://github.com/onlytailei/carla_cil_pytorch_eval/blob/pytorch_eval/README.md). 54 | For the benchmark results, please check our paper [Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective](https://arxiv.org/abs/1903.00821). 55 | 56 | ## Reference 57 | [carla-simulator/imitation-learning](https://github.com/carla-simulator/imitation-learning) 58 | [mvpcom/carlaILTrainer](https://github.com/mvpcom/carlaILTrainer) 59 | [End-to-end Driving via Conditional Imitation Learning](https://arxiv.org/abs/1710.02410) 60 | [CARLA: An Open Urban Driving Simulator](http://proceedings.mlr.press/v78/dosovitskiy17a/dosovitskiy17a.pdf) 61 | [VR-Goggles for Robots: Real-to-sim Domain Adaptation for Visual Control](https://ram-lab.com/file/tailei/vr_goggles/index.html) 62 | [Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective](https://arxiv.org/abs/1903.00821) 63 | 64 | The code for original "End-to-end Driving via Conditional Imitation Learning" and "CARLA: An Open Urban Driving Simulator" is in the [master branch](https://github.com/onlytailei/carla_cil_pytorch/tree/master). In the paper VR-Goggles, we also used the original setup to train the policy. 65 | 66 | Please consider to cite our paper if thie repo helps: 67 | ``` 68 | @inproceedings{tai2019visual, 69 | author={Tai, Lei and Yun, Peng and Chen, Yuying and Liu, Congcong and Ye, Haoyang and Liu, Ming}, 70 | title={Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective}, 71 | booktitle={2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 72 | year={2019} 73 | } 74 | @ARTICLE{zhang2019vrgoggles, 75 | author={Zhang, Jingwei and Tai, Lei and Yun, Peng and Xiong, Yufeng and Liu, Ming and Boedecker, Joschka and Burgard, Wolfram}, 76 | journal={IEEE Robotics and Automation Letters}, 77 | title={VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control}, 78 | year={2019}, 79 | volume={4}, 80 | number={2}, 81 | pages={1148-1155}, 82 | keywords={Visualization;Training;Robots;Adaptation models;Semantics;Task analysis;Navigation;Deep learning in robotics and automation;visual-based navigation;model learning for control}, 83 | doi={10.1109/LRA.2019.2894216}, 84 | ISSN={2377-3766}, 85 | month={April},} 86 | ``` 87 | -------------------------------------------------------------------------------- /carla_loader.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | ''' 4 | Author:Tai Lei 5 | Date:Thu Nov 22 12:09:27 2018 6 | Info: 7 | ''' 8 | 9 | import glob 10 | 11 | import numpy as np 12 | import h5py 13 | import torch 14 | from torchvision import transforms 15 | from torch.utils.data import Dataset 16 | 17 | from imgaug import augmenters as iaa 18 | from helper import RandomTransWrapper 19 | 20 | 21 | class CarlaH5Data(): 22 | def __init__(self, 23 | train_folder, 24 | eval_folder, 25 | batch_size=4, num_workers=4, distributed=False): 26 | 27 | self.loaders = { 28 | "train": torch.utils.data.DataLoader( 29 | CarlaH5Dataset( 30 | data_dir=train_folder, 31 | train_eval_flag="train"), 32 | batch_size=batch_size, 33 | num_workers=num_workers, 34 | pin_memory=True, 35 | shuffle=True 36 | ), 37 | "eval": torch.utils.data.DataLoader( 38 | CarlaH5Dataset( 39 | data_dir=eval_folder, 40 | train_eval_flag="eval"), 41 | batch_size=batch_size, 42 | num_workers=num_workers, 43 | pin_memory=True, 44 | shuffle=False 45 | )} 46 | 47 | 48 | class CarlaH5Dataset(Dataset): 49 | def __init__(self, data_dir, 50 | train_eval_flag="train", sequence_len=200): 51 | self.data_dir = data_dir 52 | self.data_list = glob.glob(data_dir+'*.h5') 53 | self.data_list.sort() 54 | self.sequnece_len = sequence_len 55 | self.train_eval_flag = train_eval_flag 56 | 57 | self.build_transform() 58 | 59 | def build_transform(self): 60 | if self.train_eval_flag == "train": 61 | self.transform = transforms.Compose([ 62 | transforms.RandomOrder([ 63 | RandomTransWrapper( 64 | seq=iaa.GaussianBlur( 65 | (0, 1.5)), 66 | p=0.09), 67 | RandomTransWrapper( 68 | seq=iaa.AdditiveGaussianNoise( 69 | loc=0, 70 | scale=(0.0, 0.05), 71 | per_channel=0.5), 72 | p=0.09), 73 | RandomTransWrapper( 74 | seq=iaa.Dropout( 75 | (0.0, 0.10), 76 | per_channel=0.5), 77 | p=0.3), 78 | RandomTransWrapper( 79 | seq=iaa.CoarseDropout( 80 | (0.0, 0.10), 81 | size_percent=(0.08, 0.2), 82 | per_channel=0.5), 83 | p=0.3), 84 | RandomTransWrapper( 85 | seq=iaa.Add( 86 | (-20, 20), 87 | per_channel=0.5), 88 | p=0.3), 89 | RandomTransWrapper( 90 | seq=iaa.Multiply( 91 | (0.9, 1.1), 92 | per_channel=0.2), 93 | p=0.4), 94 | RandomTransWrapper( 95 | seq=iaa.ContrastNormalization( 96 | (0.8, 1.2), 97 | per_channel=0.5), 98 | p=0.09), 99 | ]), 100 | transforms.ToTensor()]) 101 | else: 102 | self.transform = transforms.Compose([ 103 | transforms.ToTensor(), 104 | ]) 105 | 106 | def __len__(self): 107 | return self.sequnece_len * len(self.data_list) 108 | 109 | def __getitem__(self, idx): 110 | data_idx = idx // self.sequnece_len 111 | file_idx = idx % self.sequnece_len 112 | file_name = self.data_list[data_idx] 113 | 114 | with h5py.File(file_name, 'r') as h5_file: 115 | img = np.array(h5_file['rgb'])[file_idx] 116 | img = self.transform(img) 117 | target = np.array(h5_file['targets'])[file_idx] 118 | target = target.astype(np.float32) 119 | # 2 Follow lane, 3 Left, 4 Right, 5 Straight 120 | # -> 0 Follow lane, 1 Left, 2 Right, 3 Straight 121 | command = int(target[24])-2 122 | # Steer, Gas, Brake (0,1, focus on steer loss) 123 | target_vec = np.zeros((4, 3), dtype=np.float32) 124 | target_vec[command, :] = target[:3] 125 | # in km/h, <90 126 | speed = np.array([target[10]/90, ]).astype(np.float32) 127 | mask_vec = np.zeros((4, 3), dtype=np.float32) 128 | mask_vec[command, :] = 1 129 | 130 | # TODO 131 | # add preprocess 132 | return img, speed, target_vec.reshape(-1), \ 133 | mask_vec.reshape(-1), 134 | -------------------------------------------------------------------------------- /carla_net.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | ''' 4 | Author:Tai Lei 5 | Date:Thu Nov 22 12:09:33 2018 6 | Info: 7 | ''' 8 | 9 | import torch 10 | import torch.nn as nn 11 | # from torch.nn import functional as F 12 | 13 | 14 | class CarlaNet(nn.Module): 15 | def __init__(self, dropout_vec=None): 16 | super(CarlaNet, self).__init__() 17 | self.conv_block = nn.Sequential( 18 | nn.Conv2d(3, 32, kernel_size=5, stride=2), 19 | nn.BatchNorm2d(32), 20 | # nn.Dropout(self.dropout_vec[0]), 21 | nn.ReLU(), 22 | nn.Conv2d(32, 32, kernel_size=3, stride=1), 23 | nn.BatchNorm2d(32), 24 | # nn.Dropout(self.dropout_vec[1]), 25 | nn.ReLU(), 26 | nn.Conv2d(32, 64, kernel_size=3, stride=2), 27 | nn.BatchNorm2d(64), 28 | # nn.Dropout(self.dropout_vec[2]), 29 | nn.ReLU(), 30 | nn.Conv2d(64, 64, kernel_size=3, stride=1), 31 | nn.BatchNorm2d(64), 32 | # nn.Dropout(self.dropout_vec[3]), 33 | nn.ReLU(), 34 | nn.Conv2d(64, 128, kernel_size=3, stride=2), 35 | nn.BatchNorm2d(128), 36 | # nn.Dropout(self.dropout_vec[4]), 37 | nn.ReLU(), 38 | nn.Conv2d(128, 128, kernel_size=3, stride=1), 39 | nn.BatchNorm2d(128), 40 | # nn.Dropout(self.dropout_vec[5]), 41 | nn.ReLU(), 42 | nn.Conv2d(128, 256, kernel_size=3, stride=1), 43 | nn.BatchNorm2d(256), 44 | # nn.Dropout(self.dropout_vec[6]), 45 | nn.ReLU(), 46 | nn.Conv2d(256, 256, kernel_size=3, stride=1), 47 | nn.BatchNorm2d(256), 48 | # nn.Dropout(self.dropout_vec[7]), 49 | nn.ReLU(), 50 | ) 51 | 52 | self.img_fc = nn.Sequential( 53 | nn.Linear(8192, 512), 54 | nn.Dropout(0.3), 55 | nn.ReLU(), 56 | nn.Linear(512, 512), 57 | nn.Dropout(0.3), 58 | nn.ReLU(), 59 | ) 60 | 61 | self.speed_fc = nn.Sequential( 62 | nn.Linear(1, 128), 63 | nn.Dropout(0.5), 64 | nn.ReLU(), 65 | nn.Linear(128, 128), 66 | nn.Dropout(0.5), 67 | nn.ReLU(), 68 | ) 69 | 70 | self.emb_fc = nn.Sequential( 71 | nn.Linear(512+128, 512), 72 | nn.Dropout(0.5), 73 | nn.ReLU(), 74 | ) 75 | 76 | self.control_branches = nn.ModuleList([ 77 | nn.Sequential( 78 | nn.Linear(512, 256), 79 | nn.Dropout(0.5), 80 | nn.ReLU(), 81 | nn.Linear(256, 256), 82 | # nn.Dropout(self.dropout_vec[i*2+14]), 83 | nn.ReLU(), 84 | nn.Linear(256, 3), 85 | ) for i in range(4) 86 | ]) 87 | 88 | self.speed_branch = nn.Sequential( 89 | nn.Linear(512, 256), 90 | nn.Dropout(0.5), 91 | nn.ReLU(), 92 | nn.Linear(256, 256), 93 | # nn.Dropout(self.dropout_vec[1]), 94 | nn.ReLU(), 95 | nn.Linear(256, 1), 96 | ) 97 | 98 | def forward(self, img, speed): 99 | img = self.conv_block(img) 100 | img = img.view(-1, 8192) 101 | img = self.img_fc(img) 102 | 103 | speed = self.speed_fc(speed) 104 | emb = torch.cat([img, speed], dim=1) 105 | emb = self.emb_fc(emb) 106 | 107 | pred_control = torch.cat( 108 | [out(emb) for out in self.control_branches], dim=1) 109 | pred_speed = self.speed_branch(img) 110 | return pred_control, pred_speed, img, emb 111 | 112 | 113 | class UncertainNet(nn.Module): 114 | def __init__(self, structure=2, dropout_vec=None): 115 | super(UncertainNet, self).__init__() 116 | self.structure = structure 117 | 118 | if (self.structure < 2 or self.structure > 3): 119 | raise("Structure must be one of 2|3") 120 | 121 | self.uncert_speed_branch = nn.Sequential( 122 | nn.Linear(512, 256), 123 | nn.ReLU(), 124 | nn.Linear(256, 256), 125 | nn.ReLU(), 126 | nn.Linear(256, 1), 127 | ) 128 | if self.structure == 2: 129 | self.uncert_control_branches = nn.ModuleList([ 130 | nn.Sequential( 131 | nn.Linear(512, 256), 132 | nn.ReLU(), 133 | nn.Linear(256, 256), 134 | nn.ReLU(), 135 | nn.Linear(256, 3), 136 | ) for i in range(4) 137 | ]) 138 | 139 | if self.structure == 3: 140 | self.uncert_control_branches = nn.Sequential( 141 | nn.Linear(512, 256), 142 | nn.ReLU(), 143 | nn.Linear(256, 256), 144 | nn.ReLU(), 145 | nn.Linear(256, 3), 146 | ) 147 | 148 | def forward(self, img_emb, emb): 149 | if self.structure == 2: 150 | log_var_control = torch.cat( 151 | [un(emb) for un in self.uncert_control_branches], dim=1) 152 | if self.structure == 3: 153 | log_var_control = self.uncert_control_branches(emb) 154 | log_var_control = torch.cat([log_var_control for _ in range(4)], 155 | dim=1) 156 | 157 | log_var_speed = self.uncert_speed_branch(img_emb) 158 | 159 | return log_var_control, log_var_speed 160 | 161 | 162 | class FinalNet(nn.Module): 163 | def __init__(self, structure=2, dropout_vec=None): 164 | super(FinalNet, self).__init__() 165 | self.structure = structure 166 | 167 | self.carla_net = CarlaNet(dropout_vec=dropout_vec) 168 | self.uncertain_net = UncertainNet(structure) 169 | 170 | for m in self.modules(): 171 | if isinstance(m, nn.Conv2d): 172 | nn.init.kaiming_normal_( 173 | m.weight, mode='fan_out', nonlinearity='relu') 174 | elif isinstance(m, nn.BatchNorm2d): 175 | nn.init.constant_(m.weight, 1) 176 | nn.init.constant_(m.bias, 0) 177 | elif isinstance(m, nn.Linear): 178 | nn.init.xavier_uniform_( 179 | m.weight) 180 | m.bias.data.fill_(0.01) 181 | 182 | def forward(self, img, speed): 183 | pred_control, pred_speed, img_emb, emb = self.carla_net(img, speed) 184 | log_var_control, log_var_speed = self.uncertain_net(img_emb, emb) 185 | 186 | return pred_control, pred_speed, log_var_control, log_var_speed 187 | -------------------------------------------------------------------------------- /docker/carla_cil_compose/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '2.3' 2 | services: 3 | threeweather: 4 | image: onlytailei/carla_cil:pytorch1.0-cuda10-oepcv3.4.3-imgaug 5 | runtime: nvidia 6 | environment: 7 | - NVIDIA_VISIBLE_DEVICES=3 8 | volumes: 9 | - /path/to/dataset:/home/dataset 10 | - /path/to/this_repo/carla_cil_training:/home/ws/carla_cil_training 11 | working_dir: /home/ws/carla_cil_training 12 | cpu_count: 24 13 | mem_reservation: 10000M 14 | shm_size: 10000M 15 | command: 16 | python main.py 17 | --batch-size 1000 18 | --workers 24 19 | --speed-weight 1 20 | --learning-rate 0.0001 21 | --lr-gamma 0.5 22 | --lr-step 10 23 | --train-dir "/home/dataset/carla_cil/chosen_weather_train/three_train/" 24 | --eval-dir "/home/dataset/carla_cil/chosen_weather_test/three_test/" 25 | --net-structure 2 26 | --gpu 0 27 | --id three_policy_0137 28 | 29 | -------------------------------------------------------------------------------- /docker/docker_build/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM pytorch/pytorch:1.0-cuda10.0-cudnn7-runtime 2 | 3 | RUN conda install -y -c conda-forge opencv 4 | RUN conda clean -y -p -s 5 | RUN pip install tensorboardX imgaug h5py 6 | RUN apt-get update 7 | RUN apt-get -y install libgl1-mesa-glx 8 | 9 | 10 | RUN rm -rf /var/lib/apt/lists/* 11 | RUN rm -rf /tmp/* 12 | 13 | WORKDIR /home 14 | -------------------------------------------------------------------------------- /helper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | ''' 4 | Author:Tai Lei 5 | Date:Thu Nov 22 12:09:27 2018 6 | Info: 7 | ''' 8 | 9 | import os 10 | import random 11 | import shutil 12 | 13 | import torch 14 | 15 | # original transformations 16 | # check: https://github.com/carla-simulator/imitation-learning/issues/1 17 | 18 | # from imgaug import augmenters as iaa 19 | # st = lambda aug: iaa.Sometimes(0.4, aug) 20 | # oc = lambda aug: iaa.Sometimes(0.3, aug) 21 | # rl = lambda aug: iaa.Sometimes(0.09, aug) 22 | # seq = iaa.SomeOf((4, None), [ 23 | # # blur images with a sigma between 0 and 1.5 24 | # rl(iaa.GaussianBlur((0, 1.5))), 25 | # # add gaussian noise to images 26 | # rl(iaa.AdditiveGaussianNoise( 27 | # loc=0, 28 | # scale=(0.0, 0.05), 29 | # per_channel=0.5)), 30 | # # randomly remove up to X% of the pixels 31 | # oc(iaa.Dropout((0.0, 0.10), per_channel=0.5)), 32 | # # randomly remove up to X% of the pixels 33 | # oc(iaa.CoarseDropout( 34 | # (0.0, 0.10), size_percent=(0.08, 0.2), per_channel=0.5)), 35 | # # change brightness of images (by -X to Y of original value) 36 | # oc(iaa.Add((-40, 40), per_channel=0.5)), 37 | # # change brightness of images (X-Y% of original value) 38 | # st(iaa.Multiply((0.10, 2.5), per_channel=0.2)), 39 | # # improve or worsen the contrast 40 | # rl(iaa.ContrastNormalization((0.5, 1.5), per_channel=0.5)), 41 | # # rl(iaa.Grayscale((0.0, 1))), # put grayscale 42 | # ], random_order=True) 43 | 44 | 45 | class TransWrapper(object): 46 | def __init__(self, seq): 47 | self.seq = seq 48 | 49 | def __call__(self, img): 50 | return self.seq.augment_image(img) 51 | 52 | 53 | class RandomTransWrapper(object): 54 | def __init__(self, seq, p=0.5): 55 | self.seq = seq 56 | self.p = p 57 | 58 | def __call__(self, img): 59 | if self.p < random.random(): 60 | return img 61 | return self.seq.augment_image(img) 62 | 63 | 64 | class AverageMeter(object): 65 | """Computes and stores the average and current value""" 66 | 67 | def __init__(self): 68 | self.reset() 69 | 70 | def reset(self): 71 | self.val = 0 72 | self.avg = 0 73 | self.sum = 0 74 | self.count = 0 75 | 76 | def update(self, val, n=1): 77 | self.val = val 78 | self.sum += val * n 79 | self.count += n 80 | self.avg = self.sum / self.count 81 | 82 | 83 | def save_checkpoint(state, id_, is_best, filename='checkpoint.pth'): 84 | torch.save(state, filename) 85 | if is_best: 86 | shutil.copyfile( 87 | filename, 88 | os.path.join("save_models", "{}_best.pth".format(id_)) 89 | ) 90 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | ''' 4 | Author:Tai Lei 5 | Date:Wed Sep 19 20:30:48 2018 6 | Info: 7 | References: https://github.com/pytorch/examples/tree/master/imagenet 8 | ''' 9 | import argparse 10 | import os 11 | import random 12 | import time 13 | import datetime 14 | import math 15 | import logging 16 | 17 | import torch 18 | import torch.nn as nn 19 | import torch.nn.parallel 20 | import torch.backends.cudnn as cudnn 21 | import torch.distributed as dist 22 | import torch.optim as optim 23 | from tensorboardX import SummaryWriter 24 | 25 | from carla_net import CarlaNet, FinalNet 26 | from carla_loader import CarlaH5Data 27 | from helper import AverageMeter, save_checkpoint 28 | 29 | parser = argparse.ArgumentParser(description='Carla CIL training') 30 | parser.add_argument('-j', '--workers', default=4, type=int, metavar='N', 31 | help='number of data loading workers (default: 4)') 32 | parser.add_argument('--speed-weight', default=0.1, type=float, 33 | help='speed weight') 34 | parser.add_argument('--branch-weight', default=1, type=float, 35 | help='branch weight') 36 | parser.add_argument('--id', default="test", type=str) 37 | parser.add_argument('--train-dir', 38 | default="/home/tai/ws/ijrr_2018/carla_cil_dataset/AgentHuman/chosen_weather_train/clearnoon_h5/", 39 | type=str, metavar='PATH', 40 | help='training dataset') 41 | parser.add_argument('--eval-dir', 42 | default="/home/tai/ws/ijrr_2018/carla_cil_dataset/AgentHuman/chosen_weather_test/clearnoon_h5/", 43 | type=str, metavar='PATH', 44 | help='evaluation dataset') 45 | parser.add_argument('--epochs', default=90, type=int, metavar='N', 46 | help='number of total epochs to run') 47 | parser.add_argument('--start-epoch', default=0, type=int, metavar='N', 48 | help='manual epoch number (useful on restarts)') 49 | parser.add_argument('-b', '--batch-size', default=1, type=int, 50 | metavar='N', help='mini-batch size (default: 256)') 51 | parser.add_argument('--lr', '--learning-rate', default=1e-4, type=float, 52 | metavar='LR', help='initial learning rate') 53 | parser.add_argument('--lr-step', default=10, type=int, 54 | help='learning rate step size') 55 | parser.add_argument('--lr-gamma', default=0.5, type=float, 56 | help='learning rate gamma') 57 | parser.add_argument('--weight-decay', '--wd', default=1e-4, type=float, 58 | metavar='W', help='weight decay (default: 1e-4)') 59 | parser.add_argument('--print-freq', '-p', default=10, type=int, 60 | metavar='N', help='print frequency (default: 10)') 61 | parser.add_argument('--resume', default='', type=str, metavar='PATH', 62 | help='path to latest checkpoint (default: none)') 63 | parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true', 64 | help='evaluate model on validation set') 65 | parser.add_argument('--evaluate-log', default="", 66 | type=str, metavar='PATH', 67 | help='path to log evaluation results (default: none)') 68 | parser.add_argument('--world-size', default=1, type=int, 69 | help='number of distributed processes') 70 | parser.add_argument('--seed', default=None, type=int, 71 | help='seed for initializing training. ') 72 | parser.add_argument('--gpu', default=None, type=int, 73 | help='GPU id to use.') 74 | parser.add_argument('--net-structure', default=2, type=int, 75 | help='Network structure 1|2|3|4.') 76 | # 1 pure regression 77 | # 2 uncertainty separate branch 78 | # 3 uncertainty unify 79 | # 4 uncertainty under branch 80 | 81 | 82 | def output_log(output_str, logger=None): 83 | """ 84 | standard output and logging 85 | """ 86 | print("[{}]: {}".format(datetime.datetime.now(), output_str)) 87 | if logger is not None: 88 | logger.critical("[{}]: {}".format(datetime.datetime.now(), output_str)) 89 | 90 | 91 | def log_args(logger): 92 | ''' 93 | log args 94 | ''' 95 | attrs = [(p, getattr(args, p)) for p in dir(args) if not p.startswith('_')] 96 | for key, value in attrs: 97 | output_log("{}: {}".format(key, value), logger=logger) 98 | 99 | 100 | def main(): 101 | global args 102 | args = parser.parse_args() 103 | log_dir = os.path.join("./", "logs", args.id) 104 | run_dir = os.path.join("./", "runs", args.id) 105 | save_weight_dir = os.path.join("./save_models", args.id) 106 | os.makedirs(log_dir, exist_ok=True) 107 | os.makedirs(save_weight_dir, exist_ok=True) 108 | 109 | logging.basicConfig(filename=os.path.join(log_dir, "carla_training.log"), 110 | level=logging.ERROR) 111 | tsbd = SummaryWriter(log_dir=run_dir) 112 | log_args(logging) 113 | if args.seed is not None: 114 | random.seed(args.seed) 115 | torch.manual_seed(args.seed) 116 | cudnn.deterministic = True 117 | output_log( 118 | 'You have chosen to seed training. ' 119 | 'This will turn on the CUDNN deterministic setting, ' 120 | 'which can slow down your training considerably! ' 121 | 'You may see unexpected behavior when restarting ' 122 | 'from checkpoints.', logger=logging) 123 | 124 | if args.gpu is not None: 125 | output_log('You have chosen a specific GPU. This will completely ' 126 | 'disable data parallelism.', logger=logging) 127 | 128 | args.distributed = args.world_size > 1 129 | 130 | if args.distributed: 131 | dist.init_process_group(backend=args.dist_backend, 132 | init_method=args.dist_url, 133 | world_size=args.world_size, 134 | rank=0) 135 | 136 | model = FinalNet(args.net_structure) 137 | criterion = nn.MSELoss() 138 | 139 | model.carla_net.load_state_dict( 140 | torch.load("./save_models/new_structure_best.pth")['state_dict']) 141 | 142 | tsbd.add_graph(model, 143 | (torch.zeros(1, 3, 88, 200), 144 | torch.zeros(1, 1))) 145 | 146 | if args.gpu is not None: 147 | model = model.cuda(args.gpu) 148 | else: 149 | model = torch.nn.DataParallel(model).cuda() 150 | 151 | optimizer = optim.Adam( 152 | model.uncertain_net.parameters(), args.lr, betas=(0.7, 0.85)) 153 | lr_scheduler = optim.lr_scheduler.StepLR( 154 | optimizer, step_size=args.lr_step, gamma=args.lr_gamma) 155 | best_prec = math.inf 156 | 157 | # optionally resume from a checkpoint 158 | if args.resume: 159 | args.resume = os.path.join(save_weight_dir, args.resume) 160 | if os.path.isfile(args.resume): 161 | output_log("=> loading checkpoint '{}'".format(args.resume), 162 | logging) 163 | checkpoint = torch.load(args.resume) 164 | args.start_epoch = checkpoint['epoch'] 165 | model.load_state_dict(checkpoint['state_dict']) 166 | optimizer.load_state_dict(checkpoint['optimizer']) 167 | lr_scheduler.load_state_dict(checkpoint['scheduler']) 168 | best_prec = checkpoint['best_prec'] 169 | output_log("=> loaded checkpoint '{}' (epoch {})" 170 | .format(args.resume, checkpoint['epoch']), logging) 171 | else: 172 | output_log("=> no checkpoint found at '{}'".format(args.resume), 173 | logging) 174 | 175 | cudnn.benchmark = True 176 | 177 | carla_data = CarlaH5Data( 178 | train_folder=args.train_dir, 179 | eval_folder=args.eval_dir, 180 | batch_size=args.batch_size, 181 | num_workers=args.workers) 182 | 183 | train_loader = carla_data.loaders["train"] 184 | eval_loader = carla_data.loaders["eval"] 185 | 186 | if args.evaluate: 187 | args.id = args.id+"_test" 188 | if not os.path.isfile(args.resume): 189 | output_log("=> no checkpoint found at '{}'" 190 | .format(args.resume), logging) 191 | return 192 | if args.evaluate_log == "": 193 | output_log("=> please set evaluate log path with --evaluate-log ") 194 | 195 | evaluate(eval_loader, model, criterion, 0, tsbd) 196 | return 197 | 198 | for epoch in range(args.start_epoch, args.epochs): 199 | lr_scheduler.step() 200 | branch_losses, speed_losses, losses = \ 201 | train(train_loader, model, criterion, optimizer, epoch, tsbd) 202 | 203 | prec = evaluate(eval_loader, model, criterion, epoch, tsbd) 204 | 205 | # remember best prec@1 and save checkpoint 206 | is_best = prec < best_prec 207 | best_prec = min(prec, best_prec) 208 | save_checkpoint( 209 | {'epoch': epoch + 1, 210 | 'state_dict': model.state_dict(), 211 | 'best_prec': best_prec, 212 | 'scheduler': lr_scheduler.state_dict(), 213 | 'optimizer': optimizer.state_dict()}, 214 | args.id, 215 | is_best, 216 | os.path.join( 217 | save_weight_dir, 218 | "{}_{}.pth".format(epoch+1, args.id)) 219 | ) 220 | 221 | 222 | def train(loader, model, criterion, optimizer, epoch, writer): 223 | batch_time = AverageMeter() 224 | data_time = AverageMeter() 225 | uncertain_losses = AverageMeter() 226 | ori_losses = AverageMeter() 227 | branch_losses = AverageMeter() 228 | speed_losses = AverageMeter() 229 | uncertain_control_means = AverageMeter() 230 | uncertain_speed_means = AverageMeter() 231 | 232 | # switch to train mode 233 | model.train() 234 | end = time.time() 235 | step = epoch * len(loader) 236 | for i, (img, speed, target, mask) in enumerate(loader): 237 | data_time.update(time.time() - end) 238 | 239 | # if args.gpu is not None: 240 | img = img.cuda(args.gpu, non_blocking=True) 241 | speed = speed.cuda(args.gpu, non_blocking=True) 242 | target = target.cuda(args.gpu, non_blocking=True) 243 | mask = mask.cuda(args.gpu, non_blocking=True) 244 | 245 | if args.net_structure != 1: 246 | branches_out, pred_speed, log_var_control, log_var_speed = model(img, 247 | speed) 248 | 249 | branch_square = torch.pow((branches_out - target), 2) 250 | branch_loss = torch.mean((torch.exp(-log_var_control) 251 | * branch_square 252 | + log_var_control) * 0.5 * mask) * 4 253 | 254 | speed_square = torch.pow((pred_speed - speed), 2) 255 | speed_loss = torch.mean((torch.exp(-log_var_speed) 256 | * speed_square 257 | + log_var_speed) * 0.5) 258 | 259 | uncertain_loss = args.branch_weight*branch_loss+args.speed_weight*speed_loss 260 | with torch.no_grad(): 261 | ori_loss = args.branch_weight * torch.mean(branch_square*mask*4) \ 262 | + args.speed_weight * torch.mean(speed_square) 263 | uncertain_control_mean = torch.mean(torch.exp(log_var_control) * mask * 4) 264 | uncertain_speed_mean = torch.mean(torch.exp(log_var_speed)) 265 | 266 | ori_losses.update(ori_loss.item(), args.batch_size) 267 | uncertain_control_means.update(uncertain_control_mean.item(), 268 | args.batch_size) 269 | uncertain_speed_means.update(uncertain_speed_mean.item(), 270 | args.batch_size) 271 | 272 | else: 273 | branches_out, pred_speed = model(img, speed) 274 | branch_loss = criterion(branches_out * mask, target) * 4 275 | speed_loss = criterion(pred_speed, speed) 276 | uncertain_loss = args.branch_weight * branch_loss \ 277 | + args.speed_weight * speed_loss 278 | 279 | uncertain_losses.update(uncertain_loss.item(), args.batch_size) 280 | branch_losses.update(branch_loss.item(), args.batch_size) 281 | speed_losses.update(speed_loss.item(), args.batch_size) 282 | 283 | # compute gradient and do SGD step 284 | optimizer.zero_grad() 285 | model.zero_grad() 286 | uncertain_loss.backward() 287 | optimizer.step() 288 | 289 | # measure elapsed time 290 | batch_time.update(time.time() - end) 291 | end = time.time() 292 | 293 | if i % args.print_freq == 0 or i+1 == len(loader): 294 | writer.add_scalar('train/branch_loss', branch_losses.val, step+i) 295 | writer.add_scalar('train/speed_loss', speed_losses.val, step+i) 296 | writer.add_scalar('train/uncertain_loss', uncertain_losses.val, step+i) 297 | writer.add_scalar('train/ori_loss', ori_losses.val, step+i) 298 | writer.add_scalar('train/control_uncertain', 299 | uncertain_control_means.val, step+i) 300 | writer.add_scalar('train/speed_uncertain', 301 | uncertain_speed_means.val, step+i) 302 | output_log( 303 | 'Epoch: [{0}][{1}/{2}]\t' 304 | 'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t' 305 | 'Data {data_time.val:.3f} ({data_time.avg:.3f})\t' 306 | 'Branch loss {branch_loss.val:.3f} ({branch_loss.avg:.3f})\t' 307 | 'Speed loss {speed_loss.val:.3f} ({speed_loss.avg:.3f})\t' 308 | 'Uncertain Loss {uncertain_loss.val:.4f} ({uncertain_loss.avg:.4f})\t' 309 | 'Ori Loss {ori_loss.val:.4f} ({ori_loss.avg:.4f})\t' 310 | 'Control Uncertain {control_uncertain.val:.4f} ({control_uncertain.avg:.4f})\t' 311 | 'Speed Uncertain {speed_uncertain.val:.4f} ({speed_uncertain.avg:.4f})\t' 312 | .format( 313 | epoch+1, i, len(loader), batch_time=batch_time, 314 | data_time=data_time, 315 | branch_loss=branch_losses, 316 | speed_loss=speed_losses, 317 | uncertain_loss=uncertain_losses, 318 | ori_loss=ori_losses, 319 | control_uncertain=uncertain_control_means, 320 | speed_uncertain=uncertain_speed_means 321 | ), logging) 322 | 323 | return branch_losses.avg, speed_losses.avg, uncertain_losses.avg 324 | 325 | 326 | def evaluate(loader, model, criterion, epoch, writer): 327 | batch_time = AverageMeter() 328 | uncertain_losses = AverageMeter() 329 | ori_losses = AverageMeter() 330 | uncertain_control_means = AverageMeter() 331 | uncertain_speed_means = AverageMeter() 332 | 333 | # switch to evaluate mode 334 | model.eval() 335 | with torch.no_grad(): 336 | end = time.time() 337 | for i, (img, speed, target, mask) in enumerate(loader): 338 | img = img.cuda(args.gpu, non_blocking=True) 339 | speed = speed.cuda(args.gpu, non_blocking=True) 340 | target = target.cuda(args.gpu, non_blocking=True) 341 | mask = mask.cuda(args.gpu, non_blocking=True) 342 | 343 | branches_out, pred_speed, log_var_control, log_var_speed = model(img, speed) 344 | 345 | mask_out = branches_out * mask 346 | ori_branch_loss = criterion(mask_out, target) * 4 347 | ori_speed_loss = criterion(pred_speed, speed) 348 | 349 | branch_loss = torch.mean((torch.exp(-log_var_control) 350 | * torch.pow((branches_out - target), 2) 351 | + log_var_control) * 0.5 * mask) * 4 352 | 353 | speed_loss = torch.mean((torch.exp(-log_var_speed) 354 | * torch.pow((pred_speed - speed), 2) 355 | + log_var_speed) * 0.5) 356 | 357 | uncertain_loss = args.branch_weight*branch_loss + \ 358 | args.speed_weight*speed_loss 359 | ori_loss = args.branch_weight*ori_branch_loss + \ 360 | args.speed_weight*ori_speed_loss 361 | 362 | uncertain_control_mean = torch.mean(torch.exp(log_var_control) * mask * 4) 363 | uncertain_speed_mean = torch.mean(torch.exp(log_var_speed)) 364 | 365 | uncertain_losses.update(uncertain_loss.item(), args.batch_size) 366 | ori_losses.update(ori_loss.item(), args.batch_size) 367 | uncertain_control_means.update(uncertain_control_mean.item(), 368 | args.batch_size) 369 | uncertain_speed_means.update(uncertain_speed_mean.item(), 370 | args.batch_size) 371 | 372 | # measure elapsed time 373 | batch_time.update(time.time() - end) 374 | end = time.time() 375 | 376 | writer.add_scalar('eval/uncertain_loss', uncertain_losses.avg, epoch+1) 377 | writer.add_scalar('eval/origin_loss', ori_losses.avg, epoch+1) 378 | writer.add_scalar('eval/control_uncertain', 379 | uncertain_control_means.avg, epoch+1) 380 | writer.add_scalar('eval/speed_uncertain', 381 | uncertain_speed_means.avg, epoch+1) 382 | output_log( 383 | 'Epoch Test: [{0}]\t' 384 | 'Time {batch_time.avg:.3f}\t' 385 | 'Uncertain Loss {uncertain_loss.avg:.4f}\t' 386 | 'Original Loss {ori_loss.avg:.4f}\t' 387 | 'Control Uncertain {control_uncertain.avg:.4f}\t' 388 | 'Speed Uncertain {speed_uncertain.avg:.4f}\t' 389 | .format( 390 | epoch+1, batch_time=batch_time, 391 | uncertain_loss=uncertain_losses, 392 | ori_loss=ori_losses, 393 | control_uncertain=uncertain_control_means, 394 | speed_uncertain=uncertain_speed_means, 395 | ), logging) 396 | return uncertain_losses.avg 397 | 398 | 399 | if __name__ == '__main__': 400 | main() 401 | -------------------------------------------------------------------------------- /unit_test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding=utf-8 3 | ''' 4 | Author:Tai Lei 5 | Date:Thu Sep 20 10:48:02 2018 6 | Info: 7 | ''' 8 | 9 | import os 10 | 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | 14 | DataLoaderTest = True 15 | 16 | if DataLoaderTest: 17 | print("======CarlaH5Data test start======") 18 | from carla_loader import CarlaH5Data 19 | from torchvision.utils import make_grid 20 | 21 | base_path = "/home/tai/ws/ijrr_2018/carla_cil_dataset/AgentHuman" 22 | data = CarlaH5Data( 23 | train_folder=os.path.join( 24 | base_path, 25 | "chosen_weather_train/clearnoon_h5/"), 26 | eval_folder=os.path.join( 27 | base_path, 28 | "chosen_weather_test/clearnoon_h5/"), 29 | batch_size=128, 30 | num_workers=10) 31 | 32 | train_loader = data.loaders["train"] 33 | eval_loader = data.loaders["eval"] 34 | 35 | print(len(train_loader)) 36 | print(len(eval_loader)) 37 | 38 | for i, (img, speed, _, _) in enumerate(train_loader): 39 | show_img = make_grid(img) 40 | plt.imshow((np.transpose( 41 | show_img.numpy(), 42 | (1, 2, 0))*255).astype(np.uint8)) 43 | plt.show() 44 | #print(one_hot) 45 | if i == 60: 46 | break 47 | 48 | # for i, (img, speed, command, predict) in enumerate(eval_loader): 49 | # print(img.size()) 50 | # if i == 15: 51 | # break 52 | --------------------------------------------------------------------------------