├── project.nod.xml ├── project.sumocfg ├── project.netccfg ├── project.con.xml ├── project.edg.xml ├── project.add.xml ├── README.md ├── project.net.xml ├── AI_Traffic_Light.py └── project.rou.xml /project.nod.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | -------------------------------------------------------------------------------- /project.sumocfg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | -------------------------------------------------------------------------------- /project.netccfg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | -------------------------------------------------------------------------------- /project.con.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /project.edg.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /project.add.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AI-Traffic-Control-System 2 | This project was completed in partial fulfilment of my engineering degree. 3 | 4 | ## **Outline:** 5 | 6 | The system utilises SUMO traffic simulator and Python 3 with TensorFlow. The system is developed for a minor arterial road intersection 7 | with traffic lights with turning signals. Utilising the reinforcement learning algorithm called Deep Q-learning, the system tries to choose the optimal traffic duration and phase to minimise traffic waiting 8 | times around the intersection. 9 | A 7 second yellow interval was employed, and green durations were adjusted between 10 s, 15 s and 20 s, depending on the vehicle demand. 10 | This system is a modified and adapted system developed by [1] as well as extracts from work done by [2, 3]. 11 | A more realistic approach was undertaken when developing this system with regards to real world sensors and data. 12 | Induction loop sensors were considered and thus the data from these sensors is what is used to drive the system. 13 | 14 | ## **The Agent:** 15 | 16 | **Framework:** Deep Q-Learning 17 | 18 | **Environment:** A 4-way intersection with 2 incoming lanes and 2 outgoing lanes per arm. The traffic signals include a turning signal. 19 | 500 cars were used for the simulation 20 | 21 | **Sensors:** 3 induction loop sensors were placed 21 m apart per approaching lane to the intersection. Totalling 12 sensors used for 22 | the intersection. These sensors are used to obtain the waiting times, vehicle occupancy, vehicle position and vehicle velocity 23 | used by the agent to obtain the reward and develop the state. 24 | 25 | **State:** The state is made up of the position and velocity data received from the induction loop sensors as well as the 26 | current traffic phase and current duration. 27 | 28 | **Action:** The actions are the traffic phases. Since a turning signal is included, there are 4 possible phases for the intersection. 29 | However, the phase durations were as consider. 10 s, 15 s and 20 s green phase durations were used in this system. 30 | Therefore, the total number of actions were 12. 31 | 32 | **Training:** Q-learning was used with an Adaptive Momemnt Estimation (ADAM) optimiser. Additionally, Experience Replay techniques were also used so that the system 33 | could learn over time based on its previous memory. 34 | 35 | **Reward:** The reward is based on the vehicle occupancy on the induction loop sensors. These sensors given the time the 36 | vehicle was on the sensor thus, this is used to determine the reward. The waiting time of the vehicles on each sensor is 37 | summed to obtain the total waiting time. The goal of the system was to decrease the total waiting time as much as possible. 38 | 39 | **Action Policy:** Since the system is developed to learn over time, an epsilon greedy action possible is applied. 40 | 41 | 42 | ## **Requirements to run the code:** 43 | • Python 44 | 45 | • TensorFlow 46 | 47 | • SUMO Traffic simulator 48 | 49 | • Traffic Control Interface (TraCI) – this is included with SUMO 50 | 51 | ## **Additional files for the traffic generation and intersection layout:** 52 | • Add.xml – This file for the induction loops and initialling the traffic light phases. 53 | 54 | • Rou.xml – This file is created when running the code. It is for the vehicle routes and the paths in the simulation. 55 | 56 | • Con.xml – This file is for the round connections in the simulations. 57 | 58 | • Edg.xml – This is for the lanes. 59 | 60 | • Nod.xml – This is for the state and end points for the roads. 61 | 62 | • Net.xml – This is a configuration file to combine all the above files and create the road network. 63 | 64 | • Netccfg – This is a sumo network configuration. 65 | 66 | • Sumocfg – This is GUI file for the simulation 67 | 68 | 69 | ## **References:** 70 | 1. Vidali A, Crociani L, Vizzari G, Bandini,S, (2019). Bandini. A Deep Reinforcement Learning Approach to Adaptive Traffic Lights Management [cited 23 August 2019]. Available from: http://ceur-ws.org/Vol-2404/paper07.pdf 71 | 2. Gao J, Shen Y, Liu J, Ito M and Shiratori N. Adaptive Traffic Signal Control: Deep Reinforcement Learning Algorithm with Experience Replay and Target Network. [Internet]. Arxiv.org. 2019 [cited 28 June 2019]. Available from: https://arxiv.org/pdf/1705.02755.pdf 72 | 3. Liang X, Du X, Wang G, Han Z. (2018). Deep Reinforcement Learning for Traffic Light Control in Vehicular Networks. [cited 10 July 2019]. Available from: https://www.researchgate.net/publication/324104685_Deep_Reinforcement_Learning_for_Traffic_Light_Control_in_Vehicular_Networks 73 | 4. DLR - Institute of Transportation Systems - Eclipse SUMO – Simulation of Urban MObility [Internet]. Dlr.de. 2019 [cited 10 July 2019]. Available from: https://www.dlr.de/ts/en/desktopdefault.aspx/tabid-9883/16931_read-41000/ 74 | 75 | 76 | 77 | 78 | 79 | -------------------------------------------------------------------------------- /project.net.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | -------------------------------------------------------------------------------- /AI_Traffic_Light.py: -------------------------------------------------------------------------------- 1 | #Sanil Lala 2 | 3 | from __future__ import absolute_import 4 | from __future__ import print_function 5 | import os 6 | import sys 7 | import matplotlib.pyplot as plt 8 | import datetime 9 | import tensorflow as tf 10 | import numpy as np 11 | import math 12 | import timeit 13 | import random 14 | 15 | 16 | #Set SUMO environment path and import SUMO library and Traci 17 | tools = os.path.join(os.environ['SUMO_HOME'], 'tools') 18 | sys.path.append(tools) 19 | import sumolib 20 | from sumolib import checkBinary 21 | import traci 22 | 23 | #Class for traffic generation 24 | class TrafficGenerator: 25 | def __init__(self, Max_Steps): 26 | self.Total_Number_Cars = 500 #Number of cars used in the simulation 27 | self._max_steps = Max_Steps 28 | def generate_routefile(self, seed): 29 | np.random.seed(seed) 30 | Timing = np.random.poisson(2, self.Total_Number_Cars) #Poisson distribution for the car approach rate to teh intersection 31 | Timing = np.sort(Timing) 32 | Car_Generation_Steps = [] 33 | min_old = math.floor(Timing[1]) 34 | max_old = math.ceil(Timing[-1]) 35 | min_new = 0 36 | max_new = self._max_steps 37 | 38 | #Create .xml file for SUMO simulation 39 | for value in Timing: 40 | Car_Generation_Steps = np.append(Car_Generation_Steps, ((max_new - min_new) / (max_old - min_old)) * (value - max_old) + max_new) 41 | 42 | Car_Generation_Steps = np.rint(Car_Generation_Steps) 43 | with open("project.rou.xml", "w") as routes: 44 | 45 | #Generate Routes 46 | print(""" 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | """, file=routes) 61 | 62 | #Generate cars to follow routes 63 | for car_counter, step in enumerate(Car_Generation_Steps): 64 | Straight_or_Turn = np.random.uniform() 65 | if Straight_or_Turn < 0.25: 66 | Straight = np.random.randint(1, 9) 67 | if Straight == 1: 68 | print(' ' % (car_counter, step), file=routes) 69 | elif Straight == 2: 70 | print(' ' % (car_counter, step), file=routes) 71 | elif Straight == 3: 72 | print(' ' % (car_counter, step), file=routes) 73 | elif Straight == 4: 74 | print(' ' % (car_counter, step), file=routes) 75 | elif Straight == 5: 76 | print(' ' % (car_counter, step), file=routes) 77 | elif Straight == 6: 78 | print(' ' % (car_counter, step), file=routes) 79 | elif Straight == 7: 80 | print(' ' % (car_counter, step), file=routes) 81 | else: 82 | print(' ' % (car_counter, step), file=routes) 83 | else: 84 | Turn = np.random.randint(1, 5) 85 | if Turn == 1: 86 | print(' ' % (car_counter, step), file=routes) 87 | 88 | elif Turn == 2: 89 | print(' ' % (car_counter, step), file=routes) 90 | 91 | elif Turn == 3: 92 | print(' ' % (car_counter, step), file=routes) 93 | else: 94 | print(' ' % (car_counter, step), file=routes) 95 | print("", file=routes) 96 | 97 | #Main class for running the simulation 98 | class RunSimulation: 99 | def __init__(self, SimSession, model, memory, traffic_gen, total_episodes, gamma, Max_Steps, Green_Duration, Yellow_Duration, SUMO_Command): 100 | self.Session = SimSession 101 | self._Model = model 102 | self._memory = memory 103 | self._traffic_gen = traffic_gen 104 | self._total_episodes = total_episodes 105 | self._gamma = gamma 106 | self._epsilongreedy = 0 107 | self._steps = 0 108 | self._waiting_times = {} 109 | self._SUMO_Command = SUMO_Command 110 | self._max_steps = Max_Steps 111 | self._green_duration = Green_Duration 112 | self._yellow_duration = Yellow_Duration 113 | self._sum_intersection_queue = 0 114 | self._StoreReward = [] 115 | self._cumulative_wait_store = [] 116 | self._avg_intersection_queue_store = [] 117 | 118 | 119 | #Defining the initial conditions and running the simulation 120 | def run(self, episode): 121 | self._traffic_gen.generate_routefile(episode) 122 | traci.start(self._SUMO_Command) 123 | self._epsilongreedy = 1.0 - (episode / self._total_episodes) #Epsilon Greedy action policy 124 | self._steps = 0 125 | tot_neg_reward = 0 126 | old_total_wait = 0 127 | intersection_queue = 0 128 | self._waiting_times = {} 129 | self._sum_intersection_queue = 0 130 | 131 | while self._steps < self._max_steps: 132 | current_state = self._get_state() 133 | current_total_wait = self._get_waiting_times() 134 | reward = old_total_wait - current_total_wait 135 | 136 | #Add previous state, action, reward and current state to memory 137 | if self._steps != 0: 138 | self._memory.Add_Sample((old_state, old_action, reward, current_state)) 139 | action = self._choose_action(current_state) 140 | 141 | 142 | #Set yellow phase if traffic signal is different from previous signal 143 | if self._steps != 0 and old_action != action: 144 | self._Set_YellowPhase(old_action) 145 | self._simulate(self._yellow_duration) 146 | self._Set_GreenPhaseandDuration(action) 147 | self._simulate(self._green_duration) 148 | old_state = current_state 149 | old_action = action 150 | old_total_wait = current_total_wait 151 | if reward < 0: 152 | tot_neg_reward += reward 153 | 154 | #Save stats and print current eposide results 155 | self._save_stats(tot_neg_reward) 156 | print("Total reward: {}, Eps: {}".format(tot_neg_reward, self._epsilongreedy)) 157 | traci.close() 158 | 159 | 160 | def _simulate(self, steps_todo): 161 | if (self._steps + steps_todo) >= self._max_steps: 162 | steps_todo = self._max_steps - self._steps 163 | self._steps = self._steps + steps_todo 164 | while steps_todo > 0: 165 | traci.simulationStep() 166 | self._replay() 167 | steps_todo -= 1 168 | intersection_queue = self._get_stats() 169 | self._sum_intersection_queue += intersection_queue 170 | 171 | 172 | #Obtain vehicle waiting times from simulation 173 | def _get_waiting_times(self): 174 | incoming_roads = ["1i", "4i", "2i", "3i"] 175 | halt_N = traci.inductionloop.getLastStepOccupancy("Loop4i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop4i_0_2") + traci.inductionloop.getLastStepOccupancy( 176 | "Loop4i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop4i_1_1") + traci.inductionloop.getLastStepOccupancy( 177 | "Loop4i_1_2")+ traci.inductionloop.getLastStepOccupancy("Loop4i_1_3") 178 | halt_S = traci.inductionloop.getLastStepOccupancy("Loop3i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop3i_0_2") + traci.inductionloop.getLastStepOccupancy( 179 | "Loop3i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop3i_1_1") + traci.inductionloop.getLastStepOccupancy( 180 | "Loop3i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop3i_1_3") 181 | halt_E = traci.inductionloop.getLastStepOccupancy("Loop1i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop1i_0_2") + traci.inductionloop.getLastStepOccupancy( 182 | "Loop1i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop1i_1_1") + traci.inductionloop.getLastStepOccupancy( 183 | "Loop1i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop1i_1_3") 184 | halt_W = traci.inductionloop.getLastStepOccupancy("Loop2i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop2i_0_2") + traci.inductionloop.getLastStepOccupancy( 185 | "Loop2i_1_3") + traci.inductionloop.getLastStepOccupancy("Loop2i_1_1") + traci.inductionloop.getLastStepOccupancy( 186 | "Loop2i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop2i_1_3") 187 | wait = halt_N + halt_S + halt_E + halt_W 188 | total_waiting_time =wait 189 | return total_waiting_time 190 | 191 | #choose action 192 | def _choose_action(self, state): 193 | if random.uniform(0.0,1.0) < self._epsilongreedy: 194 | return random.randint(0, self._Model.Number_Actions - 3) #Random action 195 | else: 196 | return np.argmax(self._Model.predict_one(state, self.Session) ) #Use memory 197 | 198 | #set yellow phase 199 | def _Set_YellowPhase(self, old_action): 200 | if old_action == 0 or old_action == 1 or old_action == 2: 201 | yellow_phase = 1 202 | elif old_action == 3 or old_action == 4 or old_action == 5: 203 | yellow_phase = 3 204 | elif old_action == 6 or old_action == 7 or old_action == 8: 205 | yellow_phase = 5 206 | elif old_action == 9 or old_action == 10 or old_action == 11: 207 | yellow_phase = 7 208 | traci.trafficlight.setPhase("0",yellow_phase) 209 | 210 | #set green phase duration 211 | def _Set_GreenPhaseandDuration(self, action): 212 | if action == 0: 213 | traci.trafficlight.setPhase("0", 0) 214 | traci.trafficlight.setPhaseDuration("0", 15) 215 | self._green_duration = 15 216 | elif action == 1: 217 | traci.trafficlight.setPhase("0", 0) 218 | traci.trafficlight.setPhaseDuration("0", 10) 219 | self._green_duration = 10 220 | elif action == 2: 221 | traci.trafficlight.setPhase("0", 0) 222 | traci.trafficlight.setPhaseDuration("0", 20) 223 | self._green_duration = 20 224 | elif action == 3: 225 | traci.trafficlight.setPhase("0", 2) 226 | traci.trafficlight.setPhaseDuration("0", 15) 227 | self._green_duration = 15 228 | elif action == 4: 229 | traci.trafficlight.setPhase("0", 2) 230 | traci.trafficlight.setPhaseDuration("0", 10) 231 | self._green_duration = 10 232 | elif action == 5: 233 | traci.trafficlight.setPhase("0", 2) 234 | traci.trafficlight.setPhaseDuration("0", 20) 235 | self._green_duration = 20 236 | elif action == 5: 237 | traci.trafficlight.setPhase("0", 4) 238 | traci.trafficlight.setPhaseDuration("0", 15) 239 | self._green_duration = 15 240 | elif action == 7: 241 | traci.trafficlight.setPhase("0", 4) 242 | traci.trafficlight.setPhaseDuration("0", 10) 243 | self._green_duration = 10 244 | elif action == 8: 245 | traci.trafficlight.setPhase("0", 4) 246 | traci.trafficlight.setPhaseDuration("0", 20) 247 | self._green_duration = 20 248 | elif action == 9: 249 | traci.trafficlight.setPhase("0", 6) 250 | traci.trafficlight.setPhaseDuration("0", 15) 251 | self._green_duration = 15 252 | elif action == 10: 253 | traci.trafficlight.setPhase("0", 6) 254 | traci.trafficlight.setPhaseDuration("0", 10) 255 | self._green_duration = 10 256 | elif action == 11: 257 | traci.trafficlight.setPhase("0", 6) 258 | traci.trafficlight.setPhaseDuration("0", 20) 259 | self._green_duration = 20 260 | 261 | #obtain queue stats from simulation 262 | def _get_stats(self): 263 | intersection_queue = 0 264 | halt_N = traci.inductionloop.getLastStepOccupancy("Loop4i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop4i_0_2") + traci.inductionloop.getLastStepOccupancy( 265 | "Loop4i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop4i_1_1") + traci.inductionloop.getLastStepOccupancy( 266 | "Loop4i_1_2")+ traci.inductionloop.getLastStepOccupancy("Loop4i_1_3") 267 | halt_S = traci.inductionloop.getLastStepOccupancy("Loop3i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop3i_0_2") + traci.inductionloop.getLastStepOccupancy( 268 | "Loop3i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop3i_1_1") + traci.inductionloop.getLastStepOccupancy( 269 | "Loop3i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop3i_1_3") 270 | halt_E = traci.inductionloop.getLastStepOccupancy("Loop1i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop1i_0_2") + traci.inductionloop.getLastStepOccupancy( 271 | "Loop1i_0_3") + traci.inductionloop.getLastStepOccupancy("Loop1i_1_1") + traci.inductionloop.getLastStepOccupancy( 272 | "Loop1i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop1i_1_3") 273 | halt_W = traci.inductionloop.getLastStepOccupancy("Loop2i_0_1") + traci.inductionloop.getLastStepOccupancy("Loop2i_0_2") + traci.inductionloop.getLastStepOccupancy( 274 | "Loop2i_1_3") + traci.inductionloop.getLastStepOccupancy("Loop2i_1_1") + traci.inductionloop.getLastStepOccupancy( 275 | "Loop2i_1_2") + traci.inductionloop.getLastStepOccupancy("Loop2i_1_3") 276 | intersection_queue = halt_N + halt_S + halt_E + halt_W 277 | return intersection_queue 278 | 279 | #ibtain state after action 280 | def _get_state(self): 281 | #Create 12 x 3 Matrix for Vehicle Positions and Velocities 282 | Position_Matrix = [] 283 | Velocity_Matrix = [] 284 | for i in range(8): 285 | Position_Matrix.append([]) 286 | Velocity_Matrix.append([]) 287 | for j in range(3): 288 | Position_Matrix[i].append(0) 289 | Velocity_Matrix[i].append(0) 290 | Position_Matrix = np.array(Position_Matrix) 291 | Velocity_Matrix = np.array(Velocity_Matrix) 292 | 293 | Loop1i_0_1 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_0_1" ) 294 | Loop1i_0_2 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_0_2" ) 295 | Loop1i_0_3 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_0_3" ) 296 | Loop1i_1_1 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_1_1" ) 297 | Loop1i_1_2 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_1_2" ) 298 | Loop1i_1_3 = traci.inductionloop.getLastStepVehicleIDs("Loop1i_1_3" ) 299 | 300 | 301 | if len(Loop1i_0_1) != 0: 302 | Velocity_Matrix[0,0] = traci.vehicle.getSpeed(Loop1i_0_1[0]) 303 | Loop1i_0_1 = 1 304 | else: 305 | Loop1i_0_1 = 0 306 | 307 | if len(Loop1i_0_2) != 0: 308 | Velocity_Matrix[0,1] = traci.vehicle.getSpeed(Loop1i_0_2[0]) 309 | Loop1i_0_2 = 1 310 | else: 311 | Loop1i_0_2 = 0 312 | 313 | if len(Loop1i_0_3) != 0: 314 | Velocity_Matrix[0,2] = traci.vehicle.getSpeed(Loop1i_0_3[0]) 315 | Loop1i_0_3 = 1 316 | else: 317 | Loop1i_0_3 = 0 318 | 319 | if len(Loop1i_1_1) != 0: 320 | Velocity_Matrix[1,0] = traci.vehicle.getSpeed(Loop1i_1_1[0]) 321 | Loop1i_1_1 = 1 322 | else: 323 | Loop1i_1_1 = 0 324 | 325 | if len(Loop1i_1_2) != 0: 326 | Velocity_Matrix[1,1] = traci.vehicle.getSpeed(Loop1i_1_2[0]) 327 | Loop1i_1_2 = 1 328 | else: 329 | Loop1i_1_2 = 0 330 | 331 | if len(Loop1i_1_3) != 0: 332 | Velocity_Matrix[1,2] = traci.vehicle.getSpeed(Loop1i_1_3[0]) 333 | Loop1i_1_3 = 1 334 | else: 335 | Loop1i_1_3 = 0 336 | 337 | Position_Matrix[0,0] = Loop1i_0_1 338 | Position_Matrix[0,1] = Loop1i_0_2 339 | Position_Matrix[0,2] = Loop1i_0_3 340 | Position_Matrix[1,0] = Loop1i_1_1 341 | Position_Matrix[1,1] = Loop1i_1_2 342 | Position_Matrix[1,2] = Loop1i_1_3 343 | 344 | Loop2i_0_1 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_0_1" ) 345 | Loop2i_0_2 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_0_2" ) 346 | Loop2i_0_3 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_0_3" ) 347 | Loop2i_1_1 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_1_1" ) 348 | Loop2i_1_2 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_1_2" ) 349 | Loop2i_1_3 = traci.inductionloop.getLastStepVehicleIDs("Loop2i_1_3" ) 350 | 351 | 352 | if len(Loop2i_0_1) != 0: 353 | Velocity_Matrix[2,0] = traci.vehicle.getSpeed(Loop2i_0_1[0]) 354 | Loop2i_0_1 = 1 355 | else: 356 | Loop2i_0_1 = 0 357 | 358 | if len(Loop2i_0_2) != 0: 359 | Velocity_Matrix[2,1] = traci.vehicle.getSpeed(Loop2i_0_2[0]) 360 | Loop2i_0_2 = 1 361 | else: 362 | Loop2i_0_2 = 0 363 | 364 | if len(Loop2i_0_3) != 0: 365 | Velocity_Matrix[2,1] = traci.vehicle.getSpeed(Loop2i_0_3[0]) 366 | Loop2i_0_3 = 1 367 | else: 368 | Loop2i_0_3 = 0 369 | 370 | if len(Loop2i_1_1) != 0: 371 | Velocity_Matrix[3,0] = traci.vehicle.getSpeed(Loop2i_1_1[0]) 372 | Loop2i_1_1 = 1 373 | else: 374 | Loop2i_1_1 = 0 375 | 376 | if len(Loop2i_1_2) != 0: 377 | Velocity_Matrix[3,1] = traci.vehicle.getSpeed(Loop2i_1_2[0]) 378 | Loop2i_1_2 = 1 379 | else: 380 | Loop2i_1_2 = 0 381 | 382 | if len(Loop2i_1_3) != 0: 383 | Velocity_Matrix[3,2] = traci.vehicle.getSpeed(Loop2i_1_3[0]) 384 | Loop2i_1_3 = 1 385 | else: 386 | Loop2i_1_3 = 0 387 | 388 | Position_Matrix[2,0] = Loop2i_0_1 389 | Position_Matrix[2,1] = Loop2i_0_2 390 | Position_Matrix[2,2] = Loop2i_0_3 391 | Position_Matrix[3,0] = Loop2i_1_1 392 | Position_Matrix[3,1] = Loop2i_1_2 393 | Position_Matrix[3,2] = Loop2i_1_3 394 | 395 | Loop3i_0_1 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_0_1" ) 396 | Loop3i_0_2 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_0_2" ) 397 | Loop3i_0_3 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_0_3" ) 398 | Loop3i_1_1 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_1_1" ) 399 | Loop3i_1_2 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_1_2" ) 400 | Loop3i_1_3 = traci.inductionloop.getLastStepVehicleIDs("Loop3i_1_3" ) 401 | 402 | 403 | if len(Loop3i_0_1) != 0: 404 | Velocity_Matrix[4,0] = traci.vehicle.getSpeed(Loop3i_0_1[0]) 405 | Loop3i_0_1 = 1 406 | else: 407 | Loop3i_0_1 = 0 408 | 409 | if len(Loop3i_0_2) != 0: 410 | Velocity_Matrix[4,1] = traci.vehicle.getSpeed(Loop3i_0_2[0]) 411 | Loop3i_0_2 = 1 412 | else: 413 | Loop3i_0_2 = 0 414 | 415 | if len(Loop3i_0_3) != 0: 416 | Velocity_Matrix[4,2] = traci.vehicle.getSpeed(Loop3i_0_3[0]) 417 | Loop3i_0_3 = 1 418 | else: 419 | Loop3i_0_3 = 0 420 | 421 | if len(Loop3i_1_1) != 0: 422 | Velocity_Matrix[5,0] = traci.vehicle.getSpeed(Loop3i_1_1[0]) 423 | Loop3i_1_1 = 1 424 | else: 425 | Loop3i_1_1 = 0 426 | 427 | if len(Loop3i_1_2) != 0: 428 | Velocity_Matrix[5,1] = traci.vehicle.getSpeed(Loop3i_1_2[0]) 429 | Loop3i_1_2 = 1 430 | else: 431 | Loop3i_1_2 = 0 432 | 433 | if len(Loop3i_1_3) != 0: 434 | Velocity_Matrix[5,2] = traci.vehicle.getSpeed(Loop3i_1_3[0]) 435 | Loop3i_1_3 = 1 436 | else: 437 | Loop3i_1_3 = 0 438 | 439 | Position_Matrix[4,0] = Loop3i_0_1 440 | Position_Matrix[4,1] = Loop3i_0_2 441 | Position_Matrix[4,2] = Loop3i_0_3 442 | Position_Matrix[5,0] = Loop3i_1_1 443 | Position_Matrix[5,1] = Loop3i_1_2 444 | Position_Matrix[5,2] = Loop3i_1_3 445 | 446 | Loop4i_0_1 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_0_1" ) 447 | Loop4i_0_2 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_0_2" ) 448 | Loop4i_0_3 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_0_3" ) 449 | Loop4i_1_1 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_1_1" ) 450 | Loop4i_1_2 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_1_2" ) 451 | Loop4i_1_3 = traci.inductionloop.getLastStepVehicleIDs("Loop4i_1_3" ) 452 | 453 | if len(Loop4i_0_1) != 0: 454 | Velocity_Matrix[6,0] = traci.vehicle.getSpeed(Loop4i_0_1[0]) 455 | Loop4i_0_1 = 1 456 | else: 457 | Loop4i_0_1 = 0 458 | 459 | if len(Loop4i_0_2) != 0: 460 | Velocity_Matrix[6,1] = traci.vehicle.getSpeed(Loop4i_0_2[0]) 461 | Loop4i_0_2 = 1 462 | else: 463 | Loop4i_0_2 = 0 464 | 465 | if len(Loop4i_0_3) != 0: 466 | Velocity_Matrix[6,2] = traci.vehicle.getSpeed(Loop4i_0_3[0]) 467 | Loop4i_0_3 = 1 468 | else: 469 | Loop4i_0_3 = 0 470 | 471 | if len(Loop4i_1_1) != 0: 472 | Velocity_Matrix[7,0] = traci.vehicle.getSpeed(Loop4i_1_1[0]) 473 | Loop4i_1_1 = 1 474 | else: 475 | Loop4i_1_1 = 0 476 | 477 | if len(Loop4i_1_2) != 0: 478 | Velocity_Matrix[7,1] = traci.vehicle.getSpeed(Loop4i_1_2[0]) 479 | Loop4i_1_2 = 1 480 | else: 481 | Loop4i_1_2 = 0 482 | 483 | if len(Loop4i_1_3) != 0: 484 | Velocity_Matrix[7,2] = traci.vehicle.getSpeed(Loop4i_1_3[0]) 485 | Loop4i_1_3 = 1 486 | else: 487 | Loop4i_1_3 = 0 488 | 489 | Position_Matrix[6,0] = Loop4i_0_1 490 | Position_Matrix[6,1] = Loop4i_0_2 491 | Position_Matrix[6,2] = Loop4i_0_3 492 | Position_Matrix[7,0] = Loop4i_1_1 493 | Position_Matrix[7,1] = Loop4i_1_2 494 | Position_Matrix[7,2] = Loop4i_1_3 495 | 496 | 497 | 498 | #Create 4 x 1 matrix for phase state 499 | Phase = [] 500 | if traci.trafficlight.getPhase('0') == 0 or traci.trafficlight.getPhase('0') == 1 or traci.trafficlight.getPhase('0') == 2: 501 | Phase = [1, 0, 0, 0] 502 | elif traci.trafficlight.getPhase('0') == 3 or traci.trafficlight.getPhase('0') == 4 or traci.trafficlight.getPhase('0') == 5: 503 | Phase = [0, 1, 0, 0] 504 | elif traci.trafficlight.getPhase('0') == 6 or traci.trafficlight.getPhase('0') == 7 or traci.trafficlight.getPhase('0') == 8: 505 | Phase = [0, 0, 1, 0] 506 | elif traci.trafficlight.getPhase('0') == 9 or traci.trafficlight.getPhase('0') == 10 or traci.trafficlight.getPhase('0') == 11: 507 | Phase = [0, 0, 0, 1] 508 | 509 | Phase = np.array(Phase) 510 | Phase = Phase.flatten() 511 | 512 | state = np.concatenate((Position_Matrix,Velocity_Matrix), axis=0) 513 | state = state.flatten() 514 | state = np.concatenate((state,Phase), axis=0) 515 | 516 | #Create matrix for duration 517 | Duration_Matrix = [traci.trafficlight.getPhaseDuration('0')] 518 | 519 | Duration_Matrix = np.array(Duration_Matrix) 520 | Duration_Matrix = Duration_Matrix.flatten() 521 | state = np.concatenate((state,Duration_Matrix), axis=0) 522 | 523 | 524 | return state 525 | 526 | 527 | #Replay memory 528 | def _replay(self): 529 | Batch = self._memory.Get_Samples(self._Model.batch_size) 530 | if len(Batch) > 0: 531 | states = np.array([val[0] for val in Batch]) 532 | next_states = np.array([val[3] for val in Batch]) 533 | QSA = self._Model.predict_batch(states, self.Session) 534 | QSATarget = self._Model.predict_batch(next_states, self.Session) 535 | x = np.zeros((len(Batch), self._Model.Number_States)) 536 | y = np.zeros((len(Batch), self._Model.Number_Actions)) 537 | for i, b in enumerate(Batch): 538 | state, action, reward, next_state = b[0], b[1], b[2], b[3] 539 | Current_Q = QSA[i] 540 | Current_Q[action] = reward + self._gamma * np.amax(QSATarget[i]) 541 | x[i] = state 542 | y[i] = Current_Q 543 | self._Model.train_batch(self.Session, x, y) 544 | 545 | 546 | def _save_stats(self, tot_neg_reward): 547 | self._StoreReward.append(tot_neg_reward) 548 | self._cumulative_wait_store.append(self._sum_intersection_queue) 549 | 550 | @property 551 | def reward_store(self): 552 | return self._StoreReward 553 | 554 | @property 555 | def cumulative_wait_store(self): 556 | return self._cumulative_wait_store 557 | 558 | @property 559 | def avg_intersection_queue_store(self): 560 | return self._avg_intersection_queue_store 561 | 562 | #Reinforcement learning model 563 | class Model: 564 | def __init__(self, Number_States, Number_Actions, batch_size): 565 | self._Number_States = Number_States 566 | self._Number_Actions = Number_Actions 567 | self._batch_size = batch_size 568 | 569 | self._states = None 570 | self._actions = None 571 | 572 | self._logits = None 573 | self._optimizer = None 574 | self._var_init = None 575 | 576 | self._define_model() 577 | 578 | #Create neural network 579 | def _define_model(self): 580 | 581 | self._states = tf.placeholder(shape=[None, self._Number_States], dtype=tf.float32) 582 | self._q_s_a = tf.placeholder(shape=[None, self._Number_Actions], dtype=tf.float32) 583 | 584 | fc1 = tf.layers.dense(self._states, 33, activation=tf.nn.relu) 585 | fc2 = tf.layers.dense(fc1, 33, activation=tf.nn.relu) 586 | fc3 = tf.layers.dense(fc2, 33, activation=tf.nn.relu) 587 | self._logits = tf.layers.dense(fc3, self._Number_Actions) 588 | 589 | loss = tf.losses.mean_squared_error(self._q_s_a, self._logits) 590 | self._optimizer = tf.train.AdamOptimizer().minimize(loss) 591 | self._var_init = tf.global_variables_initializer() 592 | 593 | def predict_one(self, state, SimSession): 594 | return SimSession.run(self._logits, feed_dict={self._states: state.reshape(1, self.Number_States)}) 595 | 596 | def predict_batch(self, states, SimSession): 597 | return SimSession.run(self._logits, feed_dict={self._states: states}) 598 | 599 | def train_batch(self, SimSession, x_batch, y_batch): 600 | SimSession.run(self._optimizer, feed_dict={self._states: x_batch, self._q_s_a: y_batch}) 601 | 602 | @property 603 | def Number_States(self): 604 | return self._Number_States 605 | 606 | @property 607 | def Number_Actions(self): 608 | return self._Number_Actions 609 | 610 | @property 611 | def batch_size(self): 612 | return self._batch_size 613 | 614 | @property 615 | def var_init(self): 616 | return self._var_init 617 | 618 | #Class for storying and receiving memory 619 | class Memory: 620 | def __init__(self, Memory_Size): 621 | self._Memory_Size = Memory_Size 622 | self.Samples = [] 623 | 624 | def Get_Samples(self, Number_Samples): 625 | if Number_Samples > len(self.Samples): 626 | return random.sample(self.Samples, len(self.Samples)) 627 | else: 628 | return random.sample(self.Samples, Number_Samples) 629 | 630 | def Add_Sample(self, sample): 631 | self.Samples.append(sample) 632 | if len(self.Samples) > self._Memory_Size: 633 | self.Samples.pop(0) 634 | 635 | 636 | #Saving and ploting graphs 637 | def save_graphs(sim_runner, total_episodes, plot_path): 638 | 639 | plt.rcParams.update({'font.size': 24}) 640 | 641 | # reward 642 | data = sim_runner.reward_store 643 | plt.plot(data) 644 | plt.ylabel("Cumulative negative reward") 645 | plt.xlabel("Episode") 646 | plt.margins(0) 647 | min_val = min(data) 648 | max_val = max(data) 649 | plt.ylim(min_val + 0.05 * min_val, max_val - 0.05 * max_val) 650 | fig = plt.gcf() 651 | fig.set_size_inches(20, 11.25) 652 | fig.savefig(plot_path + 'reward.png', dpi=96) 653 | plt.close("all") 654 | with open(plot_path + 'reward_data.txt', "w") as file: 655 | for item in data: 656 | file.write("%s\n" % item) 657 | 658 | data = sim_runner.cumulative_wait_store 659 | plt.plot(data) 660 | plt.ylabel("Cumulative Qccupancy (s)") 661 | plt.xlabel("Episode") 662 | plt.margins(0) 663 | min_val = min(data) 664 | max_val = max(data) 665 | plt.ylim(min_val - 0.05 * min_val, max_val + 0.05 * max_val) 666 | fig = plt.gcf() 667 | fig.set_size_inches(20, 11.25) 668 | fig.savefig(plot_path + 'delay.png', dpi=96) 669 | plt.close("all") 670 | with open(plot_path + 'delay_data.txt', "w") as file: 671 | for item in data: 672 | file.write("%s\n" % item) 673 | 674 | 675 | if __name__ == "__main__": 676 | 677 | gui = True 678 | total_episodes = 100 679 | gamma = 0.75 680 | batch_size = 32 681 | Memory_Size = 3200 682 | path = "./model/model_1g/" 683 | # ---------------------- 684 | 685 | Number_States = 53 686 | Number_Actions = 12 687 | Max_Steps = 3600 688 | Green_Duration = 10 689 | Yellow_Duration = 7 690 | 691 | #Change to False if Simulation GUI must be shown 692 | if gui == True: 693 | sumoBinary = checkBinary('sumo') 694 | else: 695 | sumoBinary = checkBinary('sumo-gui') 696 | 697 | model = Model(Number_States, Number_Actions, batch_size) 698 | memory = Memory(Memory_Size) 699 | traffic_gen = TrafficGenerator(Max_Steps) 700 | SUMO_Command = [sumoBinary, "-c", "project.sumocfg", '--start', "--no-step-log", "true", "--waiting-time-memory", str(Max_Steps)] 701 | saver = tf.train.Saver() 702 | 703 | with tf.Session() as SimSession: 704 | print("PATH:", path) 705 | print("----- Start time:", datetime.datetime.now()) 706 | SimSession.run(model.var_init) 707 | sim_runner = RunSimulation(SimSession, model, memory, traffic_gen, total_episodes, gamma, Max_Steps, Green_Duration, Yellow_Duration, SUMO_Command) 708 | episode = 0 709 | 710 | while episode < total_episodes: 711 | print('----- Episode {} of {}'.format(episode+1, total_episodes)) 712 | start = timeit.default_timer() 713 | sim_runner.run(episode) # run the simulation 714 | stop = timeit.default_timer() 715 | print('Time: ', round(stop - start, 1)) 716 | episode += 1 717 | 718 | os.makedirs(os.path.dirname(path), exist_ok=True) 719 | saver.save(SimSession, path + "my_tlcs_model.ckpt") 720 | print("----- End time:", datetime.datetime.now()) 721 | print("PATH:", path) 722 | save_graphs(sim_runner, total_episodes, path) 723 | -------------------------------------------------------------------------------- /project.rou.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 | 349 | 350 | 351 | 352 | 353 | 354 | 355 | 356 | 357 | 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | 386 | 387 | 388 | 389 | 390 | 391 | 392 | 393 | 394 | 395 | 396 | 397 | 398 | 399 | 400 | 401 | 402 | 517 | --------------------------------------------------------------------------------