├── .gitignore
├── README.md
├── animation.gif
├── requirements.txt
├── src
    ├── animate.py
    ├── confidence_bound.py
    ├── custom_components.py
    ├── helpers.py
    ├── images
    │   ├── cloud-black.png
    │   ├── cloud-red.png
    │   ├── rsu-black-transparent.png
    │   └── rsu-red-transparent.png
    ├── movement.py
    ├── plot_figures.py
    ├── results
    │   ├── out-10-Cloud.txt
    │   ├── out-10-Greedy.txt
    │   ├── out-10-RL.txt
    │   ├── out-10-RSU.txt
    │   ├── out-10-Random.txt
    │   ├── out-20-Cloud.txt
    │   ├── out-20-Greedy.txt
    │   ├── out-20-RL.txt
    │   ├── out-20-Random.txt
    │   ├── out-5-Cloud.txt
    │   ├── out-5-Greedy.txt
    │   ├── out-5-RL.txt
    │   ├── out-5-RSU.txt
    │   └── out-5-Random.txt
    ├── trained_models
    │   ├── env10-300000.zip
    │   ├── env20-300000.zip
    │   └── env5-300000.zip
    ├── vfcenv.py
    ├── vfcenv10.py
    └── vfcenv20.py
└── system-model-visual.png


/.gitignore:
--------------------------------------------------------------------------------
 1 | ### macOS ###
 2 | *.DS_Store
 3 | .AppleDouble
 4 | .LSOverride
 5 | 
 6 | ### Python ###
 7 | __pycache__/
 8 | env/
 9 | build/
10 | dist/
11 | eggs/
12 | lib/
13 | lib64/
14 | parts/
15 | sdist/
16 | var/
17 | wheels/
18 | *.egg-info/
19 | .installed.cfg
20 | *.egg
21 | 
22 | # dotenv
23 | .env
24 | 
25 | # virtualenv
26 | .venv
27 | venv/
28 | ENV/


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Vehicular Fog Computing Task Offloading RL Environment
 2 | 
 3 | This repository contains the source code of the following paper: <em>"Deep Reinforcement Learning for Delay-Optimized Task Offloading in Vehicular Fog Computing".</em> The pre-print version can be view here:
 4 | https://arxiv.org/abs/2410.03472. 
 5 | 
 6 | We appreciate citing the following IEEE conference version of the paper if you found this repository to be useful: https://ieeexplore.ieee.org/document/10967415
 7 | 
 8 | ![system-model-visual](system-model-visual.png)
 9 | 
10 | 
11 | Paper abstract:
12 | > The imminent rise of autonomous vehicles (AVs) is revolutionizing the future of transport. The Vehicular Fog Computing (VFC) paradigm has emerged to alleviate the load of compute-intensive and delay-sensitive AV programs via task of- floading to nearby vehicles. Effective VFC requires an intelligent and dynamic offloading algorithm. As a result, this paper adapts Deep Reinforcement Learning (DRL) for VFC offloading. First, a simulation environment utilizing realistic hardware and task specifications, in addition to a novel vehicular movement model based on grid-planned cities, is created. Afterward, a DRL-based algorithm is trained and tested on the environment with the goal of minimizing global task delay. The DRL model displays impressive results, outperforming other greedy and conventional methods. The findings further demonstrate the effectiveness of the DRL model in minimizing queue congestion, especially when compared to traditional cloud computing methods that struggle to handle the demands of a large fleet of vehicles. This is corroborated by queuing theory, highlighting the self-scalability of the VFC-based DRL approach.
13 | 
14 | Here, you can find the custom gymnasium environment used in the paper, as well as the code for training and comparing the RL model with other conventional offloading methods. For more info, feel free to contact us via [ptoopchinezhad@gmail.com].
15 | 
16 | 
17 | ## Gym Environments
18 | The core of the code focuses on the 3 custom VFC task-offloading gymnasium environments. Each environment corresponds to one of the 3 scenarios considered in the paper:
19 | - Scenario 1: 5 Client Vehicles & 2 Service Vehicles --> `vfcenv.py`
20 | - Scenario 2: 10 Client Vehicles & 4 Service Vehicles --> `vfcenv10.py`
21 | - Scenario 3: 20 Client Vehicles & 8 Service Vehicles --> `vfcenv20.py`
22 |   
23 | Modeling of the central queuing network present in all environments was implemented with help of [Ciw](https://ciw.readthedocs.io/en/latest/). For those interested with creating a new offloading environment, I recommend digging into `custom_components.py`.
24 | 
25 | ## Downloading & Testing
26 | First, install the latest version via git:
27 | ```
28 | git clone https://github.com/Procedurally-Generated-Human/VFC-Offloading-RL.git
29 | ```
30 | Afterward, install the libraries (preferably in a virtual environment):
31 | ```
32 | pip install -r requirements.txt
33 | ```
34 | Now, you can test an environment by executing its file, for example:
35 | ```
36 | python vfcenv.py
37 | ```
38 | Setting `render_mode="human"` will create an animation:
39 | 
40 | ![animation](animation.gif)
41 | 
42 | Note: Creating an animation takes up a lot of RAM. For a larger environment, it may run into problems.
43 | 


--------------------------------------------------------------------------------
/animation.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/animation.gif


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
 1 | Ciw==3.2.0
 2 | networkx==3.3
 3 | numpy==1.26.4
 4 | tqdm==4.66.4
 5 | celluloid==0.2.0
 6 | Ciw==3.2.0
 7 | cloudpickle==3.0.0
 8 | contourpy==1.2.1
 9 | cycler==0.12.1
10 | Farama-Notifications==0.0.4
11 | filelock==3.14.0
12 | fonttools==4.51.0
13 | fsspec==2024.3.1
14 | gymnasium==0.29.1
15 | Jinja2==3.1.4
16 | kiwisolver==1.4.5
17 | MarkupSafe==2.1.5
18 | matplotlib==3.8.4
19 | mpmath==1.3.0
20 | networkx==3.3
21 | numpy==1.26.4
22 | packaging==24.0
23 | pandas==2.2.2
24 | pillow==10.3.0
25 | pyparsing==3.1.2
26 | python-dateutil==2.9.0.post0
27 | pytz==2024.1
28 | scipy==1.14.0
29 | six==1.16.0
30 | stable_baselines3==2.3.2
31 | sympy==1.12
32 | torch==2.3.0
33 | tqdm==4.66.4
34 | typing_extensions==4.11.0
35 | tzdata==2024.1
36 | 


--------------------------------------------------------------------------------
/src/animate.py:
--------------------------------------------------------------------------------
 1 | import matplotlib.pyplot as plt
 2 | from celluloid import Camera
 3 | import numpy as np
 4 | from matplotlib.patches import Rectangle
 5 | import math
 6 | np.set_printoptions(suppress=True)
 7 | 
 8 | 
 9 | AREA_SIZE = 1000
10 | RSU_IMAGE_SIZE = 100
11 | RSU_IMAGE_POSITION = [500,500]
12 | CLOUD_IMAGE_SIZE = 40
13 | CLOUD_IMAGE_POSITION = [550,550]
14 | FRAME_INTERVAL = 200
15 | 
16 | 
17 | class Animator(object):
18 |     def __init__(self, walks, parked, parked_hardware, rsu_hardware, cloud_hardware):
19 |         self.walks = np.array(walks)
20 |         self.parked = np.array(parked)
21 |         self.parked_hardware = parked_hardware
22 |         self.rsu_hardware = rsu_hardware
23 |         self.cloud_hardware = cloud_hardware
24 |         self.fig = plt.figure("Vehicular Fog Computing Simulation")
25 |         self.camera = Camera(self.fig)
26 |         self.rsu_image_black = plt.imread('images/rsu-black-transparent.png')
27 |         self.rsu_image_red = plt.imread('images/rsu-red-transparent.png')
28 |         self.cloud_image_black = plt.imread('images/cloud-black.png')
29 |         self.cloud_image_red = plt.imread('images/cloud-red.png')
30 |         plt.xlim(-10,AREA_SIZE+10)
31 |         plt.ylim(-10,AREA_SIZE+10)
32 |         plt.gca().xaxis.set_major_locator(plt.MultipleLocator(200))
33 |         plt.gca().yaxis.set_major_locator(plt.MultipleLocator(200))
34 | 
35 |     def add_frame(self, t, emitted_node, action, obs):
36 |         emitted_node -= 1
37 |         # plot vehicles
38 |         plt.scatter(self.walks[:,math.trunc(t),0],self.walks[:,math.trunc(t),1],c="blue")
39 |         plt.scatter(self.parked[:,0],self.parked[:,1],c="red")
40 |         # plot serivce vehicle hardware data
41 |         for i,txt in enumerate(self.parked):
42 |             plt.annotate(text=str(self.parked_hardware[i]),xy=self.parked[i])
43 |         # plot line from emmited node to RSU
44 |         plt.plot([self.walks[emitted_node][math.trunc(t)][0],RSU_IMAGE_POSITION[0]], [self.walks[emitted_node][math.trunc(t)][1],RSU_IMAGE_POSITION[1]], marker='', color='cornflowerblue')
45 |         # plot city blocks
46 |         for i in range(5):
47 |             for j in range(5):
48 |                 plt.gca().add_patch(Rectangle((200*i+10, 200*j+10), 170, 170, facecolor="silver", zorder=-100.0))
49 |         # plot RSU and cloud (black)
50 |         plt.imshow(self.rsu_image_black, extent=[(RSU_IMAGE_POSITION[0]-RSU_IMAGE_SIZE/2), (RSU_IMAGE_POSITION[0]+RSU_IMAGE_SIZE/2), RSU_IMAGE_POSITION[1]-RSU_IMAGE_SIZE/2, RSU_IMAGE_POSITION[1]+RSU_IMAGE_SIZE/2])
51 |         plt.imshow(self.cloud_image_black, extent=[(CLOUD_IMAGE_POSITION[0]-CLOUD_IMAGE_SIZE/2), (CLOUD_IMAGE_POSITION[0]+CLOUD_IMAGE_SIZE/2), CLOUD_IMAGE_POSITION[1]-CLOUD_IMAGE_SIZE/2, CLOUD_IMAGE_POSITION[1]+CLOUD_IMAGE_SIZE/2])
52 |         # plot action taken: 0:red rsu - 1:red cloud - 2,3: draw line to service car
53 |         if action == 0:
54 |             plt.imshow(self.rsu_image_red, extent=[(RSU_IMAGE_POSITION[0]-RSU_IMAGE_SIZE/2), (RSU_IMAGE_POSITION[0]+RSU_IMAGE_SIZE/2), RSU_IMAGE_POSITION[1]-RSU_IMAGE_SIZE/2, RSU_IMAGE_POSITION[1]+RSU_IMAGE_SIZE/2])
55 |         elif action == 1:
56 |             plt.imshow(self.cloud_image_red, extent=[(CLOUD_IMAGE_POSITION[0]-CLOUD_IMAGE_SIZE/2), (CLOUD_IMAGE_POSITION[0]+CLOUD_IMAGE_SIZE/2), CLOUD_IMAGE_POSITION[1]-CLOUD_IMAGE_SIZE/2, CLOUD_IMAGE_POSITION[1]+CLOUD_IMAGE_SIZE/2])
57 |             plt.plot([CLOUD_IMAGE_POSITION[0],RSU_IMAGE_POSITION[0]], [CLOUD_IMAGE_POSITION[1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
58 |         elif action == 2:
59 |             plt.plot([self.parked[0][0],RSU_IMAGE_POSITION[0]], [self.parked[0][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
60 |         elif action == 3:
61 |             plt.plot([self.parked[1][0],RSU_IMAGE_POSITION[0]], [self.parked[1][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
62 |         elif action == 4:
63 |             plt.plot([self.parked[2][0],RSU_IMAGE_POSITION[0]], [self.parked[2][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
64 |         elif action == 5:
65 |             plt.plot([self.parked[3][0],RSU_IMAGE_POSITION[0]], [self.parked[3][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
66 |         elif action == 6:
67 |             plt.plot([self.parked[4][0],RSU_IMAGE_POSITION[0]], [self.parked[4][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
68 |         elif action == 7:
69 |             plt.plot([self.parked[5][0],RSU_IMAGE_POSITION[0]], [self.parked[5][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
70 |         elif action == 8:
71 |             plt.plot([self.parked[6][0],RSU_IMAGE_POSITION[0]], [self.parked[6][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
72 |         elif action == 9:
73 |             plt.plot([self.parked[7][0],RSU_IMAGE_POSITION[0]], [self.parked[7][1],RSU_IMAGE_POSITION[1]], marker='', color='pink')
74 |         # place simulation details text
75 |         props = dict(boxstyle='round', facecolor='grey', alpha=0.15)
76 |         details = "-- Simulation Details --"+"\nTime: "+str(round(t,5))+"\nOrigin: "+str(emitted_node)+"\nAction: "+str(action)+"\nRSU Cpu: "+str(self.rsu_hardware[0])+"\nCloud CPU: "+str(self.cloud_hardware[1])+"\nTrans Queues: "+str(obs[0])+"\nCPU Queues: "+str(obs[1])+"\nTask CU: "+str(obs[2])+"\nTask SZ: "+str(obs[3])
77 |         plt.text(1050, 1000, details, fontsize=12, verticalalignment='top', bbox=props)
78 |         self.camera.snap()
79 | 
80 |     def show_animation(self):
81 |         animation = self.camera.animate(interval = FRAME_INTERVAL)
82 |         plt.show()


--------------------------------------------------------------------------------
/src/confidence_bound.py:
--------------------------------------------------------------------------------
 1 | import numpy as np
 2 | import scipy.stats
 3 | 
 4 | def mean_confidence_interval(filename, confidence=0.95):
 5 |     with open(filename+".txt", 'r') as file:
 6 |         numbers = [float(line.strip()) for line in file]
 7 |     numbers_array = np.array(numbers)
 8 |     a = 1.0 * np.array(numbers_array)
 9 |     n = len(a)
10 |     m, se = np.mean(a), scipy.stats.sem(a)
11 |     h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)
12 |     print(filename,round(m,2),"+-",round(h,2))
13 | 
14 | 
15 | 
16 | mean_confidence_interval("results/out-5-RL")
17 | mean_confidence_interval("results/out-10-RL")
18 | mean_confidence_interval("results/out-20-RL")
19 | mean_confidence_interval("results/out-5-Greedy")
20 | mean_confidence_interval("results/out-10-Greedy")
21 | mean_confidence_interval("results/out-20-Greedy")
22 | mean_confidence_interval("results/out-5-Cloud")
23 | mean_confidence_interval("results/out-10-Cloud")
24 | mean_confidence_interval("results/out-20-Cloud")
25 | mean_confidence_interval("results/out-5-Random")
26 | mean_confidence_interval("results/out-10-Random")
27 | mean_confidence_interval("results/out-20-Random")


--------------------------------------------------------------------------------
/src/custom_components.py:
--------------------------------------------------------------------------------
  1 | import ciw
  2 | import numpy as np
  3 | import math
  4 | import random
  5 | 
  6 | 
  7 | class CustomSimulation(ciw.Simulation):
  8 |     def __init__(self, network):
  9 |         super().__init__(network=network, arrival_node_class=CustomArrival, tracker=ciw.trackers.NodePopulation(), individual_class=CustomIndividual,  node_class=[ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, CustomNode, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node])
 10 |     def simulate_until_decision(self, max_simulation_time):
 11 |         next_active_node = self.find_next_active_node()
 12 |         self.current_time = next_active_node.next_event_date
 13 |         while self.current_time < max_simulation_time:
 14 |             next_active_node = self.event_and_return_nextnode(next_active_node)
 15 |             self.statetracker.timestamp()
 16 |             self.current_time = next_active_node.next_event_date
 17 |             if next_active_node == self.nodes[6]:
 18 |                 break
 19 |         return self.current_time
 20 |     
 21 | 
 22 | class CustomSimulation10(ciw.Simulation):
 23 |     def __init__(self, network):
 24 |         super().__init__(network=network, arrival_node_class=CustomArrival, tracker=ciw.trackers.NodePopulation(), individual_class=CustomIndividual,  node_class=[ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, CustomNode10, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node])
 25 |     def simulate_until_decision(self, max_simulation_time):
 26 |         next_active_node = self.find_next_active_node()
 27 |         self.current_time = next_active_node.next_event_date
 28 |         while self.current_time < max_simulation_time:
 29 |             next_active_node = self.event_and_return_nextnode(next_active_node)
 30 |             self.statetracker.timestamp()
 31 |             self.current_time = next_active_node.next_event_date
 32 |             if next_active_node == self.nodes[11]: #CGD
 33 |                 break
 34 |         return self.current_time
 35 |     
 36 | 
 37 | class CustomSimulation20(ciw.Simulation):
 38 |     def __init__(self, network):
 39 |         super().__init__(network=network, arrival_node_class=CustomArrival, tracker=ciw.trackers.NodePopulation(), individual_class=CustomIndividual,  node_class=[ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, CustomNode20, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node, ciw.Node])
 40 |     def simulate_until_decision(self, max_simulation_time):
 41 |         next_active_node = self.find_next_active_node()
 42 |         self.current_time = next_active_node.next_event_date
 43 |         while self.current_time < max_simulation_time:
 44 |             next_active_node = self.event_and_return_nextnode(next_active_node)
 45 |             self.statetracker.timestamp()
 46 |             self.current_time = next_active_node.next_event_date
 47 |             if next_active_node == self.nodes[21]: #CGD
 48 |                 break
 49 |         return self.current_time
 50 | 
 51 | 
 52 | class CustomNode(ciw.Node):
 53 |     # 7-8-10-12
 54 |     def __init__(self, *args, **kwargs):
 55 |         super().__init__(*args, **kwargs)
 56 |         self.decision = 0
 57 |         self.number_of_serviced_individuals = 0
 58 |         self.stop = 0
 59 |     def next_node(self, ind):
 60 |         self.number_of_serviced_individuals += 1
 61 |         self.stop = 1
 62 |         if self.decision == 0: #rsu
 63 |             return self.simulation.nodes[7]
 64 |         elif self.decision == 1: #cloud
 65 |             return self.simulation.nodes[8]
 66 |         elif self.decision == 2: #service1
 67 |             return self.simulation.nodes[10]
 68 |         elif self.decision == 3: #service2
 69 |             return self.simulation.nodes[12]
 70 |         
 71 | 
 72 | class CustomNode10(ciw.Node):
 73 |     #12-13-15-17-19-21
 74 |     def __init__(self, *args, **kwargs):
 75 |         super().__init__(*args, **kwargs)
 76 |         self.decision = 0
 77 |         self.number_of_serviced_individuals = 0
 78 |         self.stop = 0
 79 |     def next_node(self, ind):
 80 |         self.number_of_serviced_individuals += 1
 81 |         self.stop = 1
 82 |         if self.decision == 0: #rsu
 83 |             return self.simulation.nodes[12]
 84 |         elif self.decision == 1: #cloud
 85 |             return self.simulation.nodes[13]
 86 |         elif self.decision == 2: #service1
 87 |             return self.simulation.nodes[15]
 88 |         elif self.decision == 3: #service2
 89 |             return self.simulation.nodes[17]
 90 |         elif self.decision == 4: #service3
 91 |             return self.simulation.nodes[19]
 92 |         elif self.decision == 5: #service4
 93 |             return self.simulation.nodes[21]
 94 |         
 95 | 
 96 | class CustomNode20(ciw.Node):
 97 |     #22-23-25-27-29-31-33-35-37-39
 98 |     def __init__(self, *args, **kwargs):
 99 |         super().__init__(*args, **kwargs)
100 |         self.decision = 0
101 |         self.number_of_serviced_individuals = 0
102 |         self.stop = 0
103 |     def next_node(self, ind):
104 |         self.number_of_serviced_individuals += 1
105 |         self.stop = 1
106 |         if self.decision == 0: #rsu
107 |             return self.simulation.nodes[22]
108 |         elif self.decision == 1: #cloud
109 |             return self.simulation.nodes[23]
110 |         elif self.decision == 2: #service1
111 |             return self.simulation.nodes[25]
112 |         elif self.decision == 3: #service2
113 |             return self.simulation.nodes[27]
114 |         elif self.decision == 4: #service3
115 |             return self.simulation.nodes[29]
116 |         elif self.decision == 5: #service4
117 |             return self.simulation.nodes[31]
118 |         elif self.decision == 6: #service5
119 |             return self.simulation.nodes[33]
120 |         elif self.decision == 7: #service6
121 |             return self.simulation.nodes[35]
122 |         elif self.decision == 8: #service7
123 |             return self.simulation.nodes[37]
124 |         elif self.decision == 9: #service8
125 |             return self.simulation.nodes[39]
126 | 
127 | 
128 | class CustomIndividual(object):
129 |     def __init__(self, id_number, cu, sz, dl, customer_class='Customer', priority_class=0, simulation=False):
130 |         """
131 |         Initialise an individual.
132 |         """
133 |         self.arrival_date = False
134 |         self.service_start_date = False
135 |         self.service_time = False
136 |         self.service_end_date = False
137 |         self.exit_date = False
138 |         self.id_number = id_number
139 |         self.data_records = []
140 |         self.customer_class = customer_class
141 |         self.previous_class = customer_class
142 |         self.priority_class = priority_class
143 |         self.prev_priority_class = priority_class
144 |         self.original_class = customer_class
145 |         self.is_blocked = False
146 |         self.server = False
147 |         self.queue_size_at_arrival = False
148 |         self.queue_size_at_departure = False
149 |         self.destination = False
150 |         self.interrupted = False
151 |         self.node = False
152 |         self.simulation = simulation
153 |         self.cu = cu
154 |         self.sz = sz
155 |         self.dl = dl
156 | 
157 |     def __repr__(self):
158 |         """Represents an Individual instance as a string.
159 |         """
160 |         return f"Individual {self.id_number}"
161 | 
162 | 
163 | class CustomArrival(ciw.ArrivalNode):
164 |     def have_event(self):
165 |         """
166 |         Finds a batch size. Creates that many Individuals and send
167 |         them to the relevent node. Then updates the event_dates_dict.
168 |         """
169 |         batch = self.batch_size(self.next_node, self.next_class)
170 |         for _ in range(batch):
171 |             self.number_of_individuals += 1
172 |             self.number_of_individuals_per_class[self.next_class] += 1
173 |             priority_class = self.simulation.network.priority_class_mapping[self.next_class]
174 |             next_individual = self.simulation.IndividualType(
175 |                 self.number_of_individuals,
176 |                 random.choice([18200,75600,69300,41400,1100,16800,30000,44000,30100,43700,56400,91900,33000,28900,26300,520400,116000])/40, #cu
177 |                 random.randint(20,40), #sz
178 |                 random.random(), #dl
179 |                 self.next_class,
180 |                 priority_class,
181 |                 simulation=self.simulation,
182 |             )
183 | 
184 |             next_node = self.simulation.transitive_nodes[self.next_node - 1]
185 |             self.release_individual(next_node, next_individual)
186 | 
187 |         self.event_dates_dict[self.next_node][self.next_class] = self.increment_time(
188 |             self.event_dates_dict[self.next_node][self.next_class],
189 |             self.inter_arrival(self.next_node, self.next_class),
190 |         )
191 |         self.find_next_event_date()
192 | 
193 | 
194 | class ComputationDist(ciw.dists.Distribution):
195 |     def __init__(self, mips):
196 |         self.mips = mips
197 |     def sample(self, t=None, ind=None):
198 |         return ind.cu/self.mips
199 | 
200 | 
201 | class CloudCompDelay(ciw.dists.Distribution):
202 |     def __init__(self, mips, min, max):
203 |         self.mips = mips
204 |         self.min = min
205 |         self.max = max
206 |     def sample(self, t=None, ind=None):
207 |         return ind.cu/self.mips + np.random.uniform(self.min,self.max)
208 | 
209 | 
210 | class CloudTransDelay(ciw.dists.Distribution):
211 |     def __init__(self, bw):
212 |         self.bw = bw
213 |     def sample(self, t=None, ind=None):
214 |         return ind.sz/self.bw
215 | 
216 | 
217 | class StationaryTransmissionDistNew(ciw.dists.Distribution):
218 |     def __init__(self, x, y):
219 |         self.x = x
220 |         self.y = y
221 |     def sample(self, t=None, ind=None):
222 |         distance = np.linalg.norm((self.x,self.y) - np.array([500,500]))
223 |         r_div_t = 2*5 - 32.44 - 20*math.log10(distance*0.001) - 20*math.log10(5900)
224 |         r = 40*(10**6)*math.log2(1+((1*(10**(r_div_t/10)))/(40*(10**6)*(10**(-17.4)))))
225 |         return (ind.sz/(r/1_000_000))
226 |     
227 |     
228 | class MovingTransmissionDistNew(ciw.dists.Distribution):
229 |     def __init__(self, coords):
230 |         self.coords = coords
231 |     def sample(self, t=None, ind=None):
232 |         distance = np.linalg.norm(self.coords[math.trunc(t)] - np.array([500,500]))
233 |         r_div_t = 2*5 - 32.44 - 20*math.log10(distance*0.001) - 20*math.log10(5900)
234 |         r = 40*(10**6)*math.log2(1+((1*(10**(r_div_t/10)))/(40*(10**6)*(10**(-17.4)))))
235 |         return (ind.sz/(r/1_000_000))


--------------------------------------------------------------------------------
/src/helpers.py:
--------------------------------------------------------------------------------
 1 | import numpy as np
 2 | 
 3 | 
 4 | def generate_service_cpu(n):
 5 |     hardware_data = np.zeros((n))
 6 |     for i in range(n):
 7 |         hardware_data[i] = np.random.choice([71120,42820,18375])
 8 |     return hardware_data
 9 | 
10 | def generate_cloud_hardware():
11 |     hardware_data = np.zeros((2))
12 |     hardware_data[0] = 0
13 |     hardware_data[1] = np.random.choice([100_000])
14 |     return hardware_data
15 | 
16 | def generate_rsu_hardware():
17 |     hardware_data = np.zeros((1))
18 |     hardware_data[0] = np.random.choice([18375])
19 |     return hardware_data
20 | 
21 | def shortest_queue(obs):
22 |     new_queues = np.array([obs[3], obs[0]+obs[4], obs[1]+obs[5], obs[2]+obs[6]])
23 |     action = np.argmin(new_queues)
24 |     return action
25 | 
26 | def shortest_queue10(obs):
27 |     new_queues = np.array([obs[5], obs[0]+obs[6], obs[7]+obs[1], obs[8]+obs[2], obs[9]+obs[3], obs[10]+obs[4]])
28 |     action = np.argmin(new_queues)
29 |     return action
30 | 
31 | def shortest_queue20(obs):
32 |     new_queues = np.array([obs[9], obs[0]+obs[10], obs[1]+obs[11], obs[2]+obs[12], obs[3]+obs[13], obs[14]+obs[4], obs[5]+obs[15], obs[6]+obs[16], obs[7]+obs[17], obs[8]+obs[18]])
33 |     action = np.argmin(new_queues)
34 |     return action


--------------------------------------------------------------------------------
/src/images/cloud-black.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/images/cloud-black.png


--------------------------------------------------------------------------------
/src/images/cloud-red.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/images/cloud-red.png


--------------------------------------------------------------------------------
/src/images/rsu-black-transparent.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/images/rsu-black-transparent.png


--------------------------------------------------------------------------------
/src/images/rsu-red-transparent.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/images/rsu-red-transparent.png


--------------------------------------------------------------------------------
/src/movement.py:
--------------------------------------------------------------------------------
 1 | import numpy as np
 2 | import random
 3 | 
 4 | 
 5 | def create_parked_coords():
 6 |     blocks = [[20,20,170,170],[220,20,370,170],[420,20,570,170],[620,20,770,170],[820,20,970,170],
 7 |               [20,220,170,370],[220,220,370,370],[420,220,570,370],[620,220,770,370],[820,220,970,370],
 8 |               [20,420,170,570],[220,420,370,570],[420,420,570,570],[620,420,770,570],[820,420,970,570],
 9 |               [20,620,170,770],[220,620,370,770],[420,620,570,770],[620,620,770,770],[820,620,970,770],
10 |               [20,820,170,970],[220,820,370,970],[420,820,570,970],[620,820,770,970],[820,820,970,970],
11 |               ]
12 | 
13 |     block = blocks[np.random.choice(len(blocks))]
14 |     x_min = block[0]
15 |     y_min = block[1]
16 |     x_max = block[2]
17 |     y_max = block[3]
18 |     width = abs(x_max - x_min)
19 |     height = abs(y_max - y_min)
20 |     
21 |     # Calculate the total length of the rectangle's perimeter
22 |     perimeter = 2 * (width + height)
23 |     
24 |     # Generate a random number between 0 and the perimeter
25 |     rand_num = random.uniform(0, perimeter)
26 |     
27 |     # Determine which side of the rectangle the point is on
28 |     if rand_num < width:
29 |         return (x_min + rand_num, y_min)
30 |     elif rand_num < width + height:
31 |         return (x_max, y_min + rand_num - width)
32 |     elif rand_num < 2 * width + height:
33 |         return (x_max - rand_num + width + height, y_max)
34 |     else:
35 |         return (x_min, y_max - rand_num + 2 * width + height)
36 | 
37 | 
38 | def create_random_walk(timesteps):
39 |     timesteps += 100
40 |     x = random.choice([200,400,600,800])
41 |     y = random.choice([200,400,600,800])
42 |     speed = random.choice([10,20,25,40])
43 |     dirs = ["up","right","down","left"]
44 |     dir = random.choice(dirs)
45 |     coords = np.zeros((timesteps,2))
46 |     coords[0][0] = x
47 |     coords[0][1] = y
48 |     for t in range(1,timesteps):
49 |         dirs = ["up","right","down","left"]
50 |         if x%200==0 and y%200==0:
51 |             if x==0: 
52 |                 dirs.remove("left")
53 |             if x==1000:
54 |                 dirs.remove("right")
55 |             if y==0:
56 |                 dirs.remove("down")
57 |             if y==1000:
58 |                 dirs.remove("up")
59 |             dir = random.choice(dirs)
60 |         if dir=="up":
61 |             y = y+speed
62 |         elif dir=="right":
63 |             x = x+speed
64 |         elif dir=="down":
65 |             y = y-speed
66 |         elif dir=="left":
67 |             x = x-speed
68 |         else:
69 |             print("something went wrong")
70 |         coords[t][0] = x
71 |         coords[t][1] = y
72 |     return coords


--------------------------------------------------------------------------------
/src/plot_figures.py:
--------------------------------------------------------------------------------
 1 | import pandas as pd
 2 | import matplotlib.pyplot as plt
 3 | 
 4 | 
 5 | def read_and_average_results(file_name):
 6 |     with open(file_name+".txt") as f:
 7 |         data = [float(line.rstrip('\n')) for line in f]
 8 |         average = sum(data)/len(data)
 9 |         return average
10 | 
11 | rl_5 = read_and_average_results("results/out-5-RL")
12 | rl_10 = read_and_average_results("results/out-10-RL")
13 | rl_20 = read_and_average_results("results/out-20-RL")
14 | gr_5 = read_and_average_results("results/out-5-Greedy")
15 | gr_10 = read_and_average_results("results/out-10-Greedy")
16 | gr_20 = read_and_average_results("results/out-20-Greedy")
17 | cl_5 = read_and_average_results("results/out-5-Cloud")
18 | cl_10 = read_and_average_results("results/out-10-Cloud")
19 | cl_20 = read_and_average_results("results/out-20-Cloud")
20 | rn_5 = read_and_average_results("results/out-5-Random")
21 | rn_10 = read_and_average_results("results/out-10-Random")
22 | rn_20 = read_and_average_results("results/out-20-Random")
23 | print(cl_10)
24 | 
25 | rl_values = [rl_5, rl_10, rl_20]
26 | greedy_values = [gr_5, gr_10, gr_20]
27 | cloud_values = [cl_5, cl_10, cl_20]
28 | random_values = [rn_5, rn_10, rn_20]
29 | 
30 | 
31 | index = ['Scenario 1', 'Scenario 2', 'Scenario 3',]
32 | df = pd.DataFrame({'DRL': rl_values,
33 |                     'Greedy': greedy_values,
34 |                     "Cloud-Only":cloud_values,
35 |                     "Random":random_values,
36 |                     }, index=index)
37 | ax = df.plot.bar(rot=0, width=0.8, color={"DRL": "blue", "Greedy": "green", "Cloud-Only": "black", "Random": "red"})
38 | ax.set_axisbelow(True)
39 | bars = ax.patches
40 | hatches = ["\\\\","\\\\","\\\\","//","//","//","..","..","..","..","..",".."]
41 | for bar, hatch in zip(bars, hatches):
42 |     bar.set_hatch(hatch)
43 | ax.yaxis.grid(color='gray', linestyle='dashed')
44 | plt.ylabel("Avg. Task Delay (s)")
45 | plt.legend(fontsize="large")
46 | plt.show()
47 | 


--------------------------------------------------------------------------------
/src/results/out-10-Cloud.txt:
--------------------------------------------------------------------------------
  1 | 1.210842335311103
  2 | 1.3040298708468379
  3 | 1.3229666794520247
  4 | 1.7142996254331053
  5 | 2.216743203842245
  6 | 1.3683827355839087
  7 | 1.0779480468041462
  8 | 1.4449326772144833
  9 | 1.9289687017887378
 10 | 1.7897751803514994
 11 | 1.6833993336411834
 12 | 2.06148000527844
 13 | 0.8331080055537393
 14 | 1.3168318220012927
 15 | 1.5276683415278027
 16 | 1.2228579219420084
 17 | 1.0639472058271715
 18 | 1.148831546133851
 19 | 1.4722632723470948
 20 | 0.820604135046948
 21 | 1.4043019122720886
 22 | 1.940814559691989
 23 | 1.486509324870378
 24 | 1.0682881839336948
 25 | 1.091255441667391
 26 | 2.2915191489202127
 27 | 1.747223458654122
 28 | 1.700771990485004
 29 | 0.9978783603514878
 30 | 2.0391214208496953
 31 | 1.998212379524565
 32 | 2.1231338049385338
 33 | 1.5635335437033921
 34 | 2.1418372774969825
 35 | 1.599439236017964
 36 | 0.8948193602355724
 37 | 1.5958230468931687
 38 | 1.7712476662069583
 39 | 0.8274482683698536
 40 | 1.0096131064297778
 41 | 0.8838775818267114
 42 | 1.3139163261008697
 43 | 1.4504969866543787
 44 | 1.3144558639240704
 45 | 1.545602353758041
 46 | 1.1704158844223735
 47 | 1.1115349234011938
 48 | 1.3444753119812884
 49 | 1.0992007645787651
 50 | 0.8953955799110503
 51 | 1.094405069570689
 52 | 1.4530471933593447
 53 | 0.7042127741488907
 54 | 2.2935406996935583
 55 | 0.9748841094250311
 56 | 0.9466709291607466
 57 | 1.2816769929567828
 58 | 1.1066767031777867
 59 | 1.2147660540114091
 60 | 1.2423176505177802
 61 | 1.3117498791484319
 62 | 1.0584872277224104
 63 | 1.3077631427499787
 64 | 1.879733505693456
 65 | 1.2036757850311295
 66 | 2.051511548887039
 67 | 1.7975827956924029
 68 | 1.6714832801940114
 69 | 0.9325782869052383
 70 | 1.1979070703162469
 71 | 1.0292585888732484
 72 | 1.1185760672371672
 73 | 1.3036446570248006
 74 | 1.0348430734990297
 75 | 0.9736780897931495
 76 | 0.75088283627283
 77 | 1.202641842797849
 78 | 1.3864748750476203
 79 | 1.594384625752017
 80 | 1.154578257429501
 81 | 1.136708580827945
 82 | 1.0864946340430042
 83 | 1.2097401436660786
 84 | 2.231625187628934
 85 | 0.7308217870721231
 86 | 2.907187268288952
 87 | 1.8296621098702541
 88 | 1.1042544315405602
 89 | 0.8296366626723563
 90 | 1.1452764788936602
 91 | 1.3353226022352367
 92 | 1.0263084168349887
 93 | 2.7026946123765065
 94 | 1.0771518189197764
 95 | 1.048547209839003
 96 | 1.6176477619476628
 97 | 1.2487872903169912
 98 | 1.1204799810333403
 99 | 1.566726507479184
100 | 1.5535316243937074
101 | 


--------------------------------------------------------------------------------
/src/results/out-10-Greedy.txt:
--------------------------------------------------------------------------------
  1 | 2.212613527500728
  2 | 1.2834989792948823
  3 | 0.9318218912683669
  4 | 1.0275160816699445
  5 | 1.7202798086963091
  6 | 1.7686351886026792
  7 | 1.3503028739823133
  8 | 1.2177160695565241
  9 | 1.0724843562801225
 10 | 1.2068380504214289
 11 | 1.4902244545839887
 12 | 1.1835373135965146
 13 | 2.637113181675275
 14 | 1.2807947058906552
 15 | 2.447559861817013
 16 | 1.868382877554831
 17 | 1.0993084866394525
 18 | 1.3378002432274942
 19 | 1.197712706303824
 20 | 0.9413922625904645
 21 | 1.6868492736543006
 22 | 1.1198478671866439
 23 | 1.173718939831774
 24 | 1.5400460752889737
 25 | 1.3519421232387383
 26 | 1.7205147849642886
 27 | 1.6818458770975648
 28 | 1.1660202326188807
 29 | 1.9444511510927633
 30 | 1.9331479990324238
 31 | 1.479565139940845
 32 | 1.9147245012404097
 33 | 1.760617343301855
 34 | 1.382789072980663
 35 | 1.4466831094399215
 36 | 1.454276189203392
 37 | 2.3837600705152306
 38 | 1.6048211507527625
 39 | 1.5407839915741743
 40 | 1.393039505551522
 41 | 0.9455026326647177
 42 | 2.1412793944083957
 43 | 1.6663334917510806
 44 | 2.200404988595726
 45 | 1.0797436392662134
 46 | 1.3372259624032203
 47 | 1.0411907943255585
 48 | 1.7013423119639985
 49 | 2.1627469144086375
 50 | 1.444245960566767
 51 | 1.260744744403911
 52 | 1.7314029085832705
 53 | 1.1440523354217504
 54 | 0.8996066520052695
 55 | 1.235215077576272
 56 | 1.1785419803359076
 57 | 1.0950481768323164
 58 | 2.053354200985313
 59 | 1.4293826124408704
 60 | 1.0107828774197773
 61 | 0.9394109274633731
 62 | 1.1636463756275226
 63 | 1.214532075752453
 64 | 0.9366102561961908
 65 | 2.087714266622318
 66 | 1.9378594188452691
 67 | 1.5886614887847508
 68 | 1.1787858570032557
 69 | 1.0086916949173272
 70 | 1.4159388777108906
 71 | 1.0494512222275463
 72 | 1.482865731230459
 73 | 1.624692600747422
 74 | 2.165258798410511
 75 | 1.3462727639483227
 76 | 2.2785712270400373
 77 | 1.6515731861107517
 78 | 1.2187352181771491
 79 | 1.1559629168663679
 80 | 1.0982637074717143
 81 | 1.6462859197727566
 82 | 2.202708542581737
 83 | 1.2524760412575195
 84 | 1.5044406599681317
 85 | 2.3098256021040973
 86 | 2.0580074472644694
 87 | 1.5953611565718515
 88 | 1.237818467700775
 89 | 1.1218625915342955
 90 | 1.1778168581478623
 91 | 1.0661933506436008
 92 | 1.247101608912464
 93 | 2.1719274392059127
 94 | 1.409555598132571
 95 | 1.265513389671558
 96 | 1.3787601331712847
 97 | 1.251138388487759
 98 | 1.2052842912393367
 99 | 1.4251482347539453
100 | 0.8243242818362857
101 | 


--------------------------------------------------------------------------------
/src/results/out-10-RL.txt:
--------------------------------------------------------------------------------
  1 | 1.5299737693741933
  2 | 1.2629407112994424
  3 | 1.0394877383177588
  4 | 1.0300460240802458
  5 | 1.0800464893910044
  6 | 1.5390878216557062
  7 | 1.376162543137374
  8 | 1.6131175498458523
  9 | 1.0100348533869319
 10 | 1.438075587666025
 11 | 1.0477013961111474
 12 | 0.7512015551929249
 13 | 1.3178776682773001
 14 | 1.2497163844655998
 15 | 0.9715867393378249
 16 | 0.8919184637500509
 17 | 1.7197773971889692
 18 | 0.8109060849754858
 19 | 0.9992265362764152
 20 | 1.625519241787048
 21 | 1.4408013241330675
 22 | 1.3721582335949003
 23 | 1.849325198745333
 24 | 0.7136534380541115
 25 | 1.0948699788216574
 26 | 1.6567931288736069
 27 | 1.7657528048188555
 28 | 1.676073980939743
 29 | 1.623357436960503
 30 | 0.8943260166232295
 31 | 1.2901216717151223
 32 | 0.8471274957667475
 33 | 2.036310754852531
 34 | 0.8141924305328999
 35 | 1.7202198828595463
 36 | 1.2760926905636572
 37 | 1.2526552663540123
 38 | 1.3605504766614251
 39 | 0.8731182321038216
 40 | 1.1374360395641632
 41 | 0.8523879445064227
 42 | 0.9155350144295611
 43 | 1.3046851321975939
 44 | 0.866824990593194
 45 | 1.48942617020851
 46 | 1.4663243198726823
 47 | 1.5387421176222356
 48 | 1.194680306425705
 49 | 1.2004953093084296
 50 | 1.692870424071292
 51 | 0.7280111764363567
 52 | 1.9545917846051433
 53 | 1.6003788629068199
 54 | 1.1032336631559265
 55 | 2.330806651289188
 56 | 1.2167468086638829
 57 | 0.9267391511386569
 58 | 1.2704937596900023
 59 | 1.4574715349514125
 60 | 1.499275992021094
 61 | 1.1364061294437249
 62 | 0.5402772231504417
 63 | 1.5063769250566494
 64 | 0.9801932206989061
 65 | 0.7910331396798216
 66 | 0.7081221677462801
 67 | 1.721736338354217
 68 | 1.6619378008417391
 69 | 1.0219457190919883
 70 | 1.2600007588497197
 71 | 2.83018272429442
 72 | 0.8028952800229211
 73 | 1.0693332200041399
 74 | 0.7465068578378449
 75 | 1.5064785224162962
 76 | 1.1589367517066906
 77 | 1.2819973947654315
 78 | 1.8159298982612442
 79 | 1.9974960749909467
 80 | 1.4962908599370472
 81 | 1.1272882102708468
 82 | 1.2408858928600204
 83 | 1.4432220353850211
 84 | 1.131928215838453
 85 | 1.3457683983529725
 86 | 1.0425689777561113
 87 | 0.6414010799953178
 88 | 1.3525646293755138
 89 | 1.4358379384313271
 90 | 0.9615430526325931
 91 | 1.1404870307051207
 92 | 0.902256653166331
 93 | 1.7961051200506473
 94 | 1.0835208692363962
 95 | 0.7191603046630558
 96 | 1.0041658320826452
 97 | 1.298272127379224
 98 | 1.1327575528537517
 99 | 1.643406613236219
100 | 1.0702222796067873
101 | 


--------------------------------------------------------------------------------
/src/results/out-10-RSU.txt:
--------------------------------------------------------------------------------
  1 | 17.065632451089662
  2 | 20.248412394778825
  3 | 23.036811251365066
  4 | 20.462180331338093
  5 | 19.107371021365413
  6 | 19.35026629086668
  7 | 18.708889054286193
  8 | 19.80902602415987
  9 | 19.232271242452153
 10 | 18.702088323258927
 11 | 19.33224298401536
 12 | 19.510295737488
 13 | 19.47364631065216
 14 | 17.60585335577222
 15 | 22.667990506949884
 16 | 17.24951054445001
 17 | 19.406144926357726
 18 | 19.87775692524115
 19 | 21.061848918033206
 20 | 19.292112969957973
 21 | 20.659715297834435
 22 | 20.452661763335396
 23 | 17.36308898583685
 24 | 20.942044317127234
 25 | 18.83147535525496
 26 | 20.422733181407324
 27 | 19.221852519329172
 28 | 20.73301314861747
 29 | 19.876506959158533
 30 | 19.525031629227225
 31 | 20.31524223815491
 32 | 21.81609722014428
 33 | 20.524606397159374
 34 | 19.155719308787663
 35 | 19.055571181676974
 36 | 19.8297680895359
 37 | 19.435531747341123
 38 | 21.549694991354126
 39 | 18.8146643053541
 40 | 19.43634670282401
 41 | 18.498359220047206
 42 | 18.902524383713974
 43 | 19.139455335603298
 44 | 20.084190596484785
 45 | 19.77525753493155
 46 | 18.521544114137075
 47 | 20.51718319720076
 48 | 21.161097945907635
 49 | 20.985380964405017
 50 | 23.005814951854582
 51 | 18.331079763924354
 52 | 18.89570066618607
 53 | 20.28873189958921
 54 | 20.339805081618255
 55 | 20.081660216910144
 56 | 18.43116847073236
 57 | 19.51938868844431
 58 | 21.23727941032468
 59 | 18.148641185368188
 60 | 19.121517851888427
 61 | 21.819007530028752
 62 | 19.252200322022766
 63 | 20.017283223863092
 64 | 20.967027739195323
 65 | 17.890055273718424
 66 | 24.10081112761324
 67 | 22.62134036002549
 68 | 19.791875088763216
 69 | 21.046668820998963
 70 | 18.632333885195045
 71 | 20.046698450100724
 72 | 21.37564168705052
 73 | 19.6691515227941
 74 | 18.139392743913795
 75 | 18.465374809661668
 76 | 19.600461025950583
 77 | 20.117678910723185
 78 | 19.706962750425646
 79 | 24.326851148531148
 80 | 17.43867789702864
 81 | 19.088552272709023
 82 | 21.359981701061425
 83 | 18.4745437401023
 84 | 20.8578701152477
 85 | 20.322783821718502
 86 | 19.08362748571019
 87 | 21.165558351500977
 88 | 22.15598548490074
 89 | 19.275390629705182
 90 | 18.104449991828684
 91 | 19.54835034666834
 92 | 19.120011573580282
 93 | 20.533128887503228
 94 | 20.010708596123862
 95 | 19.777745733315072
 96 | 21.078141898019005
 97 | 19.984545894722398
 98 | 19.252973182896763
 99 | 21.17639312161539
100 | 20.551445286854467
101 | 


--------------------------------------------------------------------------------
/src/results/out-10-Random.txt:
--------------------------------------------------------------------------------
  1 | 4.8428394655168345
  2 | 4.961526399838135
  3 | 5.359950104611352
  4 | 4.100306585578054
  5 | 3.8435512807546517
  6 | 7.127807599962768
  7 | 6.052909777670816
  8 | 4.0902612225051405
  9 | 3.086920759512588
 10 | 3.360278876011317
 11 | 5.427556655730236
 12 | 4.889922758981146
 13 | 5.2660560811994
 14 | 4.110419168868219
 15 | 5.139586403389791
 16 | 2.2957311371656326
 17 | 4.409510277767957
 18 | 6.393920118697182
 19 | 3.8174079032261523
 20 | 6.329611526563412
 21 | 2.485860239493526
 22 | 4.392191331756464
 23 | 5.63603557658604
 24 | 3.0693411452790196
 25 | 2.1628079889455036
 26 | 4.990505730984063
 27 | 4.82706804676624
 28 | 5.6997523502433545
 29 | 6.128864001886134
 30 | 6.438025984561578
 31 | 5.094447663853479
 32 | 4.4562236002381646
 33 | 5.024980274287038
 34 | 5.372449806607662
 35 | 3.7975262014680737
 36 | 3.855337994035406
 37 | 4.2153809362839345
 38 | 4.210246087343543
 39 | 4.08826472238342
 40 | 4.6910489698626
 41 | 4.257066349186509
 42 | 5.659703927717353
 43 | 1.9228500773591601
 44 | 5.720578519781301
 45 | 4.6670964065939
 46 | 6.036410902011197
 47 | 8.08047887906324
 48 | 5.100618307629179
 49 | 4.25956292544274
 50 | 4.344366132904882
 51 | 3.540943519262084
 52 | 4.42367521493672
 53 | 2.2747070544517314
 54 | 3.1715993270357568
 55 | 4.82110849244779
 56 | 6.973630448496776
 57 | 5.560439549705957
 58 | 6.046421195079026
 59 | 4.216775821082124
 60 | 5.336185869774703
 61 | 4.966451255633966
 62 | 3.191117896056842
 63 | 6.401973548076884
 64 | 3.77046075792721
 65 | 3.361485759091516
 66 | 5.139360604024937
 67 | 4.8928147511948605
 68 | 4.127280524233974
 69 | 3.0225945057480232
 70 | 4.4002667819018715
 71 | 3.0708933753835312
 72 | 4.477087162644275
 73 | 3.4019153965225373
 74 | 4.29644499614614
 75 | 3.3711133742755157
 76 | 5.003350099456192
 77 | 3.0416423457793185
 78 | 3.1257314975375246
 79 | 4.920743334890082
 80 | 3.169663585764452
 81 | 4.369515507611869
 82 | 3.3440791495140556
 83 | 6.898132831108215
 84 | 1.3894772203600185
 85 | 5.921001023159594
 86 | 5.478291722136387
 87 | 2.8578122052760433
 88 | 5.165650536503362
 89 | 4.412896470341185
 90 | 5.085473536945859
 91 | 4.511966036578265
 92 | 3.9745138059037894
 93 | 4.863970899129374
 94 | 3.4376046608108877
 95 | 5.243834392894127
 96 | 3.34449852452972
 97 | 5.091730837927014
 98 | 5.560180445943294
 99 | 5.680502198493618
100 | 4.3052695455285335
101 | 


--------------------------------------------------------------------------------
/src/results/out-20-Cloud.txt:
--------------------------------------------------------------------------------
  1 | 13.676074275147887
  2 | 13.065806337467707
  3 | 14.144675344072855
  4 | 13.658883391850047
  5 | 13.66102169196612
  6 | 13.45564619554276
  7 | 13.96133574334208
  8 | 12.864462784086262
  9 | 14.46600315960911
 10 | 13.60367635918139
 11 | 12.61504553944189
 12 | 13.025988956231387
 13 | 13.745783576408696
 14 | 13.410596855303647
 15 | 13.78145032498425
 16 | 13.29377719244688
 17 | 13.301614756863982
 18 | 13.064836239159607
 19 | 13.183835251498877
 20 | 13.321879900224177
 21 | 13.865508905557375
 22 | 13.620744713892899
 23 | 13.16832085028406
 24 | 13.377866873027845
 25 | 13.423250806948083
 26 | 12.984051239969068
 27 | 13.762946822602895
 28 | 13.638384205347172
 29 | 12.655941231291115
 30 | 14.344230255719458
 31 | 13.417824316120749
 32 | 13.798877630912823
 33 | 13.459189895854463
 34 | 14.019294011563636
 35 | 13.716319297619103
 36 | 13.580288626056277
 37 | 13.593509697652525
 38 | 13.204358802899089
 39 | 14.215941496465554
 40 | 13.51306452538429
 41 | 13.54698277756058
 42 | 14.903716315867058
 43 | 13.346185655710963
 44 | 13.841321429687799
 45 | 13.742955957746846
 46 | 14.281701986460071
 47 | 13.653435810172029
 48 | 14.654260885349649
 49 | 13.378905020628435
 50 | 13.161959464606412
 51 | 13.828532788390099
 52 | 13.110387556120212
 53 | 12.6288596736596
 54 | 13.538132716246537
 55 | 13.454301893617012
 56 | 13.78319911827217
 57 | 13.305580145269714
 58 | 13.580707921257146
 59 | 14.30317784059983
 60 | 13.543711959049807
 61 | 14.587660747776619
 62 | 13.204192581781568
 63 | 14.087508896147943
 64 | 13.683265704392356
 65 | 13.5206800542843
 66 | 13.065215762031409
 67 | 12.977259910720482
 68 | 13.546584392449745
 69 | 13.839351235565474
 70 | 14.01536169960051
 71 | 13.006727990110608
 72 | 13.816733244648812
 73 | 13.596800910726781
 74 | 14.040935378867909
 75 | 14.274719795082303
 76 | 12.998285125024688
 77 | 14.007423974099579
 78 | 13.793080116909307
 79 | 12.972759985737845
 80 | 13.156453595043741
 81 | 13.31840630087526
 82 | 13.91242074462071
 83 | 14.179450878671313
 84 | 13.919389359171713
 85 | 12.726161920772514
 86 | 13.193709835926818
 87 | 13.111113237586526
 88 | 14.074539593560134
 89 | 14.089936875490409
 90 | 13.812984714595093
 91 | 13.19003084328275
 92 | 12.887558734346662
 93 | 13.8909211110277
 94 | 12.7839697415299
 95 | 13.82698710449058
 96 | 13.725578645671634
 97 | 13.392973356501924
 98 | 14.153589798442201
 99 | 14.17253695201981
100 | 13.900242725145969
101 | 


--------------------------------------------------------------------------------
/src/results/out-20-Greedy.txt:
--------------------------------------------------------------------------------
  1 | 3.0380671701902626
  2 | 3.3744130554256695
  3 | 1.0958514006377387
  4 | 3.2951858893387858
  5 | 1.5903405129656338
  6 | 2.984343217321559
  7 | 2.9673169262851378
  8 | 2.1382727383580344
  9 | 2.966133492281772
 10 | 3.1191301557642963
 11 | 1.5517878491203059
 12 | 2.4985402398779586
 13 | 2.5442267105116065
 14 | 2.8661350362302582
 15 | 3.404172544504751
 16 | 3.3581413610940376
 17 | 1.7025698910579
 18 | 2.7422298399189016
 19 | 2.325248088940345
 20 | 1.6251376294022024
 21 | 2.9953110514939385
 22 | 3.4959593069154526
 23 | 2.529629001165505
 24 | 2.5264693960897837
 25 | 2.9245716866069693
 26 | 1.5414510205631713
 27 | 1.2605997300309526
 28 | 2.6433967089920847
 29 | 3.237111347029492
 30 | 2.6336665515720057
 31 | 3.0443963971766608
 32 | 2.9964615677228434
 33 | 2.2516054710474376
 34 | 2.4150303760891307
 35 | 2.962448488085793
 36 | 2.6944219667870355
 37 | 2.143762839949266
 38 | 3.1034340609470084
 39 | 2.7843932628434507
 40 | 2.5076933732718896
 41 | 1.7431877720444096
 42 | 1.966661213756864
 43 | 2.9844864074149875
 44 | 2.401264870572144
 45 | 3.0508062395787277
 46 | 1.9392890364738034
 47 | 2.4358405391170623
 48 | 2.6034231136689834
 49 | 3.1887777574162883
 50 | 2.3401695439474692
 51 | 2.4348707599185784
 52 | 3.1250078200888853
 53 | 1.5600072395805737
 54 | 2.4073268884626766
 55 | 2.903983077833093
 56 | 3.3197022515904533
 57 | 3.1496024273680696
 58 | 2.588456209249943
 59 | 2.687612192623536
 60 | 3.045624537510005
 61 | 3.3429547158816977
 62 | 2.189422239329314
 63 | 3.1342530007128353
 64 | 1.7692116086388983
 65 | 2.3656243926052487
 66 | 2.632472702630665
 67 | 3.2839276828841752
 68 | 2.528962939721342
 69 | 2.882489253513283
 70 | 3.20800399426055
 71 | 2.4835223625542433
 72 | 2.7007141501324465
 73 | 3.453030151079384
 74 | 2.3690199818186017
 75 | 1.7102335660744308
 76 | 3.013674194544758
 77 | 2.3491820313500353
 78 | 2.2887693285855226
 79 | 2.4807486843598183
 80 | 2.5635485173931496
 81 | 1.6307876926816496
 82 | 2.1644933300849676
 83 | 1.867435896372873
 84 | 1.7700221760376396
 85 | 2.6187330068744554
 86 | 1.5389734953109924
 87 | 1.8680955019031862
 88 | 1.9296536797965023
 89 | 1.9797609792358588
 90 | 2.76207898702907
 91 | 2.3189496009595394
 92 | 2.4469601089514894
 93 | 3.5904734983363555
 94 | 1.1901068127688008
 95 | 2.398446995224247
 96 | 1.249970737226575
 97 | 3.2899789991793225
 98 | 2.8557028177532593
 99 | 2.6550741328016176
100 | 2.829982450512081
101 | 


--------------------------------------------------------------------------------
/src/results/out-20-RL.txt:
--------------------------------------------------------------------------------
  1 | 1.4704852081078819
  2 | 1.5458806476041327
  3 | 1.1196008852532102
  4 | 1.931924608071708
  5 | 1.5118907892582873
  6 | 1.7214989077110066
  7 | 1.5672227756547623
  8 | 1.6855114204522315
  9 | 1.3593198841563008
 10 | 1.4329939210266756
 11 | 1.279789825715351
 12 | 2.002673847902525
 13 | 1.7914883055834825
 14 | 1.5009008391533933
 15 | 1.298431696868292
 16 | 1.7098702097036822
 17 | 1.8530526097474351
 18 | 1.4463957919393178
 19 | 1.4245279572608556
 20 | 1.872140619168779
 21 | 1.9754588381427987
 22 | 1.4744016865448148
 23 | 1.6544797618774143
 24 | 1.715637722858126
 25 | 1.3933066219631167
 26 | 1.7272649797622759
 27 | 1.6779842973552899
 28 | 2.4104498079673284
 29 | 1.6299504597797179
 30 | 1.4898693693233833
 31 | 1.3994257995260102
 32 | 1.8564164802945327
 33 | 1.5293186791583846
 34 | 1.5799648095975762
 35 | 2.129107562640119
 36 | 1.6435616592452231
 37 | 1.3809559777420677
 38 | 1.3491024324662149
 39 | 1.7240022047889427
 40 | 1.6355901241756376
 41 | 1.3326828445285022
 42 | 1.972981080168591
 43 | 1.321072599060113
 44 | 1.92697311428678
 45 | 1.9430575556628917
 46 | 1.4099699828370862
 47 | 1.3645440402727347
 48 | 1.7509423510048376
 49 | 1.241486902077449
 50 | 1.4817851824398722
 51 | 1.69695463158355
 52 | 1.673230668996704
 53 | 1.8079722356325216
 54 | 1.3526130828453928
 55 | 1.6710389832109802
 56 | 1.3172950231814327
 57 | 1.5765508366233978
 58 | 1.3455716099698862
 59 | 1.7260113363509513
 60 | 1.608428166611367
 61 | 1.6362914530671977
 62 | 1.1591100667837564
 63 | 1.0343311832068258
 64 | 2.1973791402741756
 65 | 1.9966577373816785
 66 | 1.488599987249759
 67 | 1.3444403953451254
 68 | 1.2889462557259326
 69 | 1.6428339145448643
 70 | 1.3889298899247233
 71 | 1.2925217264902458
 72 | 1.9001868124365873
 73 | 1.1307523481022546
 74 | 1.654025281623975
 75 | 1.682058348875113
 76 | 1.6735706619558997
 77 | 1.6403430442654292
 78 | 1.4609022231929107
 79 | 1.5357106594280063
 80 | 1.6950246927784312
 81 | 2.1943202021239596
 82 | 2.1937449129266913
 83 | 1.4348666295664454
 84 | 1.5186829471124261
 85 | 1.3646010417382723
 86 | 1.34442717121847
 87 | 1.4718575395982936
 88 | 1.4161017257567545
 89 | 1.7077733754303475
 90 | 1.7249996558857237
 91 | 1.9355686272534034
 92 | 1.6423723611045864
 93 | 1.558182595749276
 94 | 1.3861873888432235
 95 | 1.1871004544823522
 96 | 1.5427610317431282
 97 | 1.7359442491621064
 98 | 1.6597096798580706
 99 | 1.7453688835923566
100 | 1.4064916303130994
101 | 


--------------------------------------------------------------------------------
/src/results/out-20-Random.txt:
--------------------------------------------------------------------------------
  1 | 5.653340390526367
  2 | 5.6492508227511316
  3 | 9.777241456586392
  4 | 5.689586389082215
  5 | 5.769882985687358
  6 | 9.03466799058714
  7 | 6.335753803612622
  8 | 7.170734733519449
  9 | 7.6013653447169895
 10 | 8.410858735791397
 11 | 6.559951595443244
 12 | 6.983572516455989
 13 | 3.492057440599093
 14 | 8.789913700179293
 15 | 8.68092937444533
 16 | 9.752198549722575
 17 | 7.21164355657913
 18 | 8.003773538340337
 19 | 8.149507690579052
 20 | 7.04657849399066
 21 | 5.73282669967066
 22 | 7.830665325252804
 23 | 7.004606478202165
 24 | 5.354206705059021
 25 | 10.109341837302162
 26 | 6.034763843444703
 27 | 5.9947780021330725
 28 | 6.781575123384998
 29 | 5.575316758720577
 30 | 5.905984680067971
 31 | 8.285813521098905
 32 | 8.95877443541617
 33 | 10.445851051812664
 34 | 5.686138768405667
 35 | 4.100260120875567
 36 | 5.511578761219653
 37 | 5.746644888020132
 38 | 6.511533153667859
 39 | 6.675632543146097
 40 | 8.328432297512355
 41 | 6.648520484048901
 42 | 8.334947589257348
 43 | 5.593170127134809
 44 | 7.278460570528617
 45 | 5.653585180713182
 46 | 5.405246161649961
 47 | 6.381259446971203
 48 | 7.805690210398313
 49 | 7.602786941078831
 50 | 5.422739998855651
 51 | 8.559918926957916
 52 | 6.274633344440984
 53 | 6.177131817892599
 54 | 8.146591198646304
 55 | 3.8603323617154612
 56 | 6.519878559970858
 57 | 10.694297877903745
 58 | 7.3262915382193485
 59 | 6.3306409074520085
 60 | 5.714410362224479
 61 | 8.409228321849495
 62 | 8.41740177898664
 63 | 7.310574688794482
 64 | 6.124747691138075
 65 | 5.435901674494733
 66 | 4.7929279943177265
 67 | 7.475290049769943
 68 | 7.207094109253458
 69 | 6.414670450470761
 70 | 8.04687847970684
 71 | 3.826652747145445
 72 | 4.6910273806854486
 73 | 8.222578676805053
 74 | 7.347289657607155
 75 | 5.8684236628474125
 76 | 4.712248455332383
 77 | 6.0758044782386085
 78 | 8.887432484684908
 79 | 7.030760877934075
 80 | 5.388787626046079
 81 | 6.14396462099324
 82 | 5.849112392650995
 83 | 6.563779197185351
 84 | 7.335026842955124
 85 | 6.368751408978875
 86 | 6.354736555038488
 87 | 5.937985820683475
 88 | 6.730630996371443
 89 | 5.941545848568976
 90 | 8.820444096896678
 91 | 7.00152799170448
 92 | 5.302837808954404
 93 | 5.262715773285251
 94 | 4.383057755342128
 95 | 4.180313007795046
 96 | 6.334248798370669
 97 | 7.602605126211633
 98 | 7.142822544481897
 99 | 5.942503332837669
100 | 6.369548855702308
101 | 


--------------------------------------------------------------------------------
/src/results/out-5-Cloud.txt:
--------------------------------------------------------------------------------
  1 | 0.8945179922142744
  2 | 1.6103087125390572
  3 | 1.9971346935176828
  4 | 3.5186320049506534
  5 | 0.7933665140720727
  6 | 1.346023797194635
  7 | 1.3284168596113526
  8 | 1.4564869606719932
  9 | 1.171076340517287
 10 | 0.9241780662982986
 11 | 0.6466590279101403
 12 | 1.2517977664212683
 13 | 1.862355942471485
 14 | 3.1638906575097954
 15 | 1.6688512991321345
 16 | 0.9400623427002895
 17 | 1.100300009044609
 18 | 1.7756702900584482
 19 | 0.8760684398847838
 20 | 1.5903261501657617
 21 | 2.5345885634800718
 22 | 0.906058577300561
 23 | 1.9421885371312377
 24 | 1.8538335880993115
 25 | 1.1244437907173856
 26 | 0.7963266682428696
 27 | 1.4572397435424904
 28 | 0.629101481855549
 29 | 1.484917752297528
 30 | 1.1185935226294406
 31 | 1.498664554347842
 32 | 1.0391707037505402
 33 | 0.9057013681821605
 34 | 1.5834685717246733
 35 | 2.258486037721142
 36 | 2.296284719755798
 37 | 1.637154269536391
 38 | 1.6611268884145276
 39 | 0.9498168290740546
 40 | 1.7865046795923332
 41 | 1.1830642016250639
 42 | 1.7719536323244656
 43 | 0.5425383329119081
 44 | 1.172264244605549
 45 | 1.636067482903395
 46 | 1.2724578943573162
 47 | 1.6454880053405305
 48 | 1.5385976512036876
 49 | 1.058739376970911
 50 | 1.5631183042789625
 51 | 1.3373704114845795
 52 | 2.1284646339642515
 53 | 0.9649265311741106
 54 | 1.8952184915467298
 55 | 0.7768102394061134
 56 | 1.5661848165899424
 57 | 1.0257799248263757
 58 | 0.8817296243661383
 59 | 1.1051339897476853
 60 | 0.8221447376866234
 61 | 1.2829739517371301
 62 | 1.5202344400571755
 63 | 0.9308951191404182
 64 | 1.1043296436320136
 65 | 0.9398350032603106
 66 | 1.114702609943632
 67 | 1.462520059661807
 68 | 1.805585576749709
 69 | 1.4231855166176584
 70 | 1.1881188211106144
 71 | 2.1459513411218505
 72 | 1.8912074491054855
 73 | 1.1222399714201445
 74 | 1.4691584735717667
 75 | 0.9748765459569648
 76 | 1.6676682441567596
 77 | 1.4377954432668372
 78 | 1.3911877878421588
 79 | 0.8807521998998156
 80 | 1.7190159406732046
 81 | 1.5412948806340216
 82 | 1.049655598747156
 83 | 1.65779410447511
 84 | 0.8543803159548705
 85 | 1.348448362925951
 86 | 0.7132997254405323
 87 | 1.8340947224135504
 88 | 2.0515318414580146
 89 | 1.3398172704925981
 90 | 1.3043968899469887
 91 | 1.467189447290601
 92 | 1.7166401261855166
 93 | 1.6764218208141868
 94 | 0.8510623622439952
 95 | 2.494007801321547
 96 | 2.7618991794038212
 97 | 1.0710761806984117
 98 | 1.7082317574981138
 99 | 1.0317768249275547
100 | 1.1946507470620014
101 | 


--------------------------------------------------------------------------------
/src/results/out-5-Greedy.txt:
--------------------------------------------------------------------------------
  1 | 0.9156705842952589
  2 | 2.5994005200635937
  3 | 1.3880697163769502
  4 | 1.339620497966184
  5 | 1.2181258070960395
  6 | 1.1002840358302564
  7 | 0.6732700443949979
  8 | 2.3028321776612204
  9 | 1.7757900880396844
 10 | 1.3502058074432666
 11 | 0.509028192823362
 12 | 0.4977059568040803
 13 | 1.2585462087033397
 14 | 1.2323030883914627
 15 | 1.0009800955186643
 16 | 1.4810481873415928
 17 | 1.9553539310411767
 18 | 1.64216371886693
 19 | 0.850191401399287
 20 | 1.4092773644742136
 21 | 2.3626708958650506
 22 | 1.0891863478183212
 23 | 2.6349633038086844
 24 | 1.3335885347783818
 25 | 0.7516504211911776
 26 | 0.7004297059497611
 27 | 2.347801304685708
 28 | 0.894823501228759
 29 | 1.54286781529947
 30 | 0.8816298986585276
 31 | 0.8050042848828959
 32 | 1.180499268962072
 33 | 1.3633644986304854
 34 | 0.8275404299834027
 35 | 1.992615085073302
 36 | 1.562434958386473
 37 | 0.9040950551435664
 38 | 0.8336846495953937
 39 | 2.190812426016757
 40 | 0.7914431868298155
 41 | 2.268538783100497
 42 | 0.791950279435054
 43 | 2.287326736132529
 44 | 1.1592881215674018
 45 | 2.5990955127664477
 46 | 1.2533979508493642
 47 | 0.6676012489565696
 48 | 0.6115179153201936
 49 | 1.5735178749902259
 50 | 1.986054796218505
 51 | 2.3279788879442194
 52 | 0.7182868135677472
 53 | 0.9895146756048172
 54 | 0.9119840916985548
 55 | 1.166794142795532
 56 | 1.851050978476185
 57 | 1.505135207071253
 58 | 1.8000037610618917
 59 | 0.7388067779479474
 60 | 1.5190794456798922
 61 | 2.204619446626005
 62 | 1.5124730575615186
 63 | 1.0591707456848505
 64 | 1.6594451234217034
 65 | 0.7276546314245539
 66 | 0.536751052300169
 67 | 1.6197601005153164
 68 | 0.6127375193700806
 69 | 1.6119164447063339
 70 | 1.1338324012016694
 71 | 1.3348385350527203
 72 | 1.1147945515508442
 73 | 1.584885455782848
 74 | 1.2497945703215725
 75 | 0.9891063150358952
 76 | 0.84028668443424
 77 | 1.7071946038038437
 78 | 2.2049332997197246
 79 | 1.7891144946475297
 80 | 1.5490669549860732
 81 | 1.2664377254497288
 82 | 1.3000939849342976
 83 | 1.7503768499282977
 84 | 0.9192673920938133
 85 | 1.135304649351127
 86 | 0.8404684275651816
 87 | 1.1172528625434799
 88 | 1.3891352956290928
 89 | 1.07961930776871
 90 | 1.7476998793831706
 91 | 0.834920796123282
 92 | 1.7154573107528763
 93 | 0.8456641669557964
 94 | 2.744555128259109
 95 | 1.6461686136052072
 96 | 1.2345571251115008
 97 | 0.6090576515748816
 98 | 0.9352377814970646
 99 | 1.237551975708482
100 | 1.4343694257042188
101 | 


--------------------------------------------------------------------------------
/src/results/out-5-RL.txt:
--------------------------------------------------------------------------------
  1 | 1.4643861026494482
  2 | 0.8867062053732474
  3 | 1.442143736723275
  4 | 0.871541365145
  5 | 1.971913985048624
  6 | 1.5995508480588585
  7 | 0.9558040973100154
  8 | 0.8678038878762715
  9 | 1.6021555430605354
 10 | 1.0371750642855646
 11 | 2.1060357197235278
 12 | 0.9920399763973315
 13 | 0.9324161256062737
 14 | 1.5862785707014235
 15 | 2.075345509399013
 16 | 1.631599894982732
 17 | 1.002886160675287
 18 | 0.7185808411203345
 19 | 1.2547985593992568
 20 | 0.8640037554156305
 21 | 1.5803723282398414
 22 | 0.6417000814100527
 23 | 1.1839114778712805
 24 | 1.2468544416668663
 25 | 0.9069170742848062
 26 | 0.8612785402564191
 27 | 0.9535874188191718
 28 | 0.6403363139777312
 29 | 2.694223001828734
 30 | 0.7320302197126712
 31 | 0.8462433704937221
 32 | 1.0329997458860605
 33 | 1.270309945305063
 34 | 0.7241909667598425
 35 | 2.631997918232195
 36 | 2.5169105547425414
 37 | 0.71617369224451
 38 | 0.8401096132253574
 39 | 0.9318111580465958
 40 | 1.5641928567629702
 41 | 1.281243219232639
 42 | 2.0138527200032468
 43 | 1.015201404758857
 44 | 1.139112440615116
 45 | 0.5533790528653448
 46 | 1.0946924574861674
 47 | 1.1376750088797438
 48 | 0.9376220066937446
 49 | 0.7219054265727182
 50 | 1.2075032379372619
 51 | 2.1281759869417027
 52 | 1.720953157274756
 53 | 1.5765996349080198
 54 | 1.1351633548700126
 55 | 1.0939839493429717
 56 | 0.710713251253699
 57 | 1.040536844022095
 58 | 1.8993669709830092
 59 | 1.1625686873821244
 60 | 1.2464350961486381
 61 | 0.718830049456743
 62 | 0.9999672871600365
 63 | 1.4404318608535256
 64 | 1.2493075177868131
 65 | 1.3430504445450708
 66 | 0.8934000373454651
 67 | 0.6328450844916566
 68 | 1.0804574900566883
 69 | 0.6892427192253076
 70 | 2.8902679911301643
 71 | 1.2373895582568597
 72 | 0.6253058199058217
 73 | 1.1830134365945102
 74 | 1.0319923096799064
 75 | 0.6797996169870035
 76 | 1.2729801310733833
 77 | 0.485035334927565
 78 | 2.108548315415647
 79 | 0.7267216168613096
 80 | 0.7842003959248378
 81 | 1.9954202787133428
 82 | 0.8604924630371296
 83 | 1.9399302440973387
 84 | 0.7451065588622852
 85 | 1.3690048183301928
 86 | 0.6281853300527974
 87 | 1.3519376352809984
 88 | 0.9592802956565587
 89 | 1.2446823028176006
 90 | 3.3344610118419795
 91 | 0.8009942676304032
 92 | 1.0372458115225411
 93 | 0.8577273279745516
 94 | 0.4999336847068753
 95 | 0.9898743364967975
 96 | 2.1869465297915607
 97 | 0.7110593824867388
 98 | 1.1779955335749643
 99 | 0.5996563268000527
100 | 1.4221927389916331
101 | 


--------------------------------------------------------------------------------
/src/results/out-5-RSU.txt:
--------------------------------------------------------------------------------
  1 | 8.0979709508878
  2 | 10.200269739080845
  3 | 11.967863051758737
  4 | 10.56158509124621
  5 | 10.221696766077008
  6 | 9.435410351696763
  7 | 9.120770343756485
  8 | 12.234950922699927
  9 | 7.541567151219471
 10 | 11.32249190869579
 11 | 9.682783321335416
 12 | 10.419857271910292
 13 | 8.208523280794491
 14 | 10.328834772641413
 15 | 12.1795066664342
 16 | 7.899437599226423
 17 | 10.299806591827545
 18 | 10.442677166930444
 19 | 9.551163556294748
 20 | 12.261505246133229
 21 | 12.62206720333551
 22 | 11.268985084650916
 23 | 8.930356719596015
 24 | 12.202532406528714
 25 | 8.700866447363117
 26 | 6.857603498779904
 27 | 10.58671564559774
 28 | 7.147479206143945
 29 | 10.99252522645408
 30 | 8.445094955766455
 31 | 11.948890273381618
 32 | 10.369991972943208
 33 | 8.977142196849064
 34 | 9.816353539749013
 35 | 9.018629896071522
 36 | 12.404973724436129
 37 | 10.134793511072282
 38 | 9.558734146926946
 39 | 11.43652053832197
 40 | 11.2733423611365
 41 | 12.286062002935038
 42 | 9.889036427022912
 43 | 7.850559289220326
 44 | 9.417901437511807
 45 | 12.552996023257402
 46 | 8.755108070750852
 47 | 9.36330336173696
 48 | 8.037321225642096
 49 | 8.372666467310358
 50 | 11.02016254448879
 51 | 8.589649976387822
 52 | 13.959999655344667
 53 | 10.669333335829585
 54 | 8.377648030960726
 55 | 8.867970183024882
 56 | 11.462766972254748
 57 | 10.192185571866476
 58 | 7.199931963485378
 59 | 10.37078310307884
 60 | 14.299953020920661
 61 | 10.857877109423208
 62 | 10.306399896562674
 63 | 13.011655898228257
 64 | 10.191845749113249
 65 | 8.544542500372664
 66 | 8.85843756312081
 67 | 9.544972291482997
 68 | 13.966252329216188
 69 | 9.122761938004386
 70 | 8.185660364545072
 71 | 9.052441607787124
 72 | 13.149650906565869
 73 | 11.179649485645138
 74 | 11.097228674205576
 75 | 10.932365125781134
 76 | 10.364633976451369
 77 | 10.363519475461116
 78 | 11.456466799385504
 79 | 10.867840144138963
 80 | 7.465184951110314
 81 | 9.239248155588886
 82 | 9.291945831922519
 83 | 10.225972278262079
 84 | 11.188738873342132
 85 | 13.500009228222211
 86 | 10.995259905741943
 87 | 7.573387814512562
 88 | 10.35755515603392
 89 | 12.817774050657151
 90 | 13.46532340891095
 91 | 10.981452511283647
 92 | 10.612211559483795
 93 | 9.157489579792005
 94 | 12.562163715024665
 95 | 10.410320973295047
 96 | 8.475981155511311
 97 | 8.862236644881289
 98 | 9.205345346528313
 99 | 8.247055372969664
100 | 9.544338258482528
101 | 


--------------------------------------------------------------------------------
/src/results/out-5-Random.txt:
--------------------------------------------------------------------------------
  1 | 2.551746296220199
  2 | 2.8695843026150087
  3 | 3.231612097451105
  4 | 2.35694518277583
  5 | 2.7063243062399764
  6 | 1.852919837134224
  7 | 2.666963991497889
  8 | 2.240860477459341
  9 | 3.460222545760133
 10 | 1.5819787661385287
 11 | 2.517247584870364
 12 | 3.547761731933289
 13 | 1.893631412621366
 14 | 2.476374238113229
 15 | 3.011937708523886
 16 | 1.115996393073862
 17 | 2.6153951100314248
 18 | 2.0954444409955935
 19 | 3.44058681243381
 20 | 1.8768558955159214
 21 | 1.9451787105334386
 22 | 1.6854928569776888
 23 | 1.9106975399983948
 24 | 1.807933142946009
 25 | 2.5042294223895296
 26 | 3.132100308900452
 27 | 2.0117982540864974
 28 | 2.098322260723014
 29 | 1.3789816745166557
 30 | 1.0091329120411103
 31 | 2.0090775952167412
 32 | 2.616566008371085
 33 | 2.6293146867140194
 34 | 3.0757545503227774
 35 | 1.070104749705247
 36 | 3.1137702995875043
 37 | 1.447903187761607
 38 | 3.4334654997512577
 39 | 3.382960630786226
 40 | 2.0352112189656046
 41 | 5.288914248887915
 42 | 1.155028343230762
 43 | 4.631559247601403
 44 | 4.525298773944414
 45 | 2.5575578266422045
 46 | 1.2944144201909145
 47 | 3.08519147697868
 48 | 2.702625446201505
 49 | 4.8447620664074105
 50 | 3.868827167790038
 51 | 1.5562665760732777
 52 | 3.5621012095131155
 53 | 1.347567094740458
 54 | 1.7543789076829985
 55 | 1.062008542030719
 56 | 2.588804226880277
 57 | 4.459370722941695
 58 | 0.8334612442113017
 59 | 1.4175149978640362
 60 | 3.7051141162082266
 61 | 3.2306302328686978
 62 | 1.362220189356107
 63 | 1.484125823614841
 64 | 2.5782742886929926
 65 | 1.0803831897921095
 66 | 4.030295925204565
 67 | 2.470536224880742
 68 | 2.2993850884229183
 69 | 1.2656275013459117
 70 | 2.6509633717357635
 71 | 3.1363415237150276
 72 | 3.05839124245638
 73 | 2.953852807202369
 74 | 3.8707170090016474
 75 | 1.2262689427556
 76 | 2.541276896857949
 77 | 2.763400886637814
 78 | 3.1924093208519992
 79 | 2.563575335663672
 80 | 2.8287125057771005
 81 | 1.4311233570551478
 82 | 2.822256516809167
 83 | 2.558769258926954
 84 | 2.3732603326377193
 85 | 2.958604520012827
 86 | 2.8706550732225713
 87 | 1.4241701890691993
 88 | 3.139056511166849
 89 | 1.9456412974927333
 90 | 4.082320849730411
 91 | 1.7606721045912015
 92 | 1.9506219596773842
 93 | 2.9164774652548435
 94 | 1.9748245844851504
 95 | 2.7397310518308235
 96 | 1.8064746139920904
 97 | 1.49764338520308
 98 | 4.12181577467994
 99 | 1.637746919492333
100 | 2.253397741947691
101 | 


--------------------------------------------------------------------------------
/src/trained_models/env10-300000.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/trained_models/env10-300000.zip


--------------------------------------------------------------------------------
/src/trained_models/env20-300000.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/trained_models/env20-300000.zip


--------------------------------------------------------------------------------
/src/trained_models/env5-300000.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/src/trained_models/env5-300000.zip


--------------------------------------------------------------------------------
/src/vfcenv.py:
--------------------------------------------------------------------------------
  1 | import ciw.dists
  2 | import gymnasium as gym
  3 | import ciw
  4 | import numpy as np
  5 | from movement import create_random_walk, create_parked_coords
  6 | from custom_components import MovingTransmissionDistNew, ComputationDist, StationaryTransmissionDistNew, CustomSimulation, CloudCompDelay, CloudTransDelay
  7 | from animate import Animator
  8 | from helpers import generate_service_cpu, generate_rsu_hardware, generate_cloud_hardware
  9 | np.set_printoptions(suppress=True)
 10 | 
 11 | 
 12 | class VFCOffloadingEnv(gym.Env):
 13 | 
 14 |     def __init__(self, n_timesteps, render_mode=None) -> None:
 15 |         self.action_space = gym.spaces.Discrete(4)
 16 |         self.observation_space =  gym.spaces.Box(0,10,(15,))
 17 |         self.n_timesteps = n_timesteps
 18 |         self.current_timestep = 0
 19 |         self.render_mode = render_mode
 20 |         super().__init__()
 21 | 
 22 |     def reset(self, seed=None):
 23 |         ciw.seed(seed)
 24 |         walk_1 = create_random_walk(self.n_timesteps)
 25 |         walk_2 = create_random_walk(self.n_timesteps)
 26 |         walk_3 = create_random_walk(self.n_timesteps)
 27 |         walk_4 = create_random_walk(self.n_timesteps)
 28 |         walk_5 = create_random_walk(self.n_timesteps)
 29 |         self.parked_1 = create_parked_coords()
 30 |         self.parked_2 = create_parked_coords()
 31 |         self.service_vehicles_cpu = generate_service_cpu(2)
 32 |         self.rsu_hardware = generate_rsu_hardware()
 33 |         self.cloud_hardware = generate_cloud_hardware()
 34 |         if self.render_mode == "human":
 35 |             self.anim = Animator([walk_1,walk_2,walk_3,walk_4,walk_5],[self.parked_1,self.parked_2], self.service_vehicles_cpu, self.rsu_hardware, self.cloud_hardware)
 36 |         self.N = ciw.create_network(
 37 |             arrival_distributions=[ciw.dists.Exponential(rate=3),   #client-trns-1       1
 38 |                            ciw.dists.Exponential(rate=3),           #client-trns-2       2
 39 |                            ciw.dists.Exponential(rate=3),           #client-trns-3       3
 40 |                            ciw.dists.Exponential(rate=3),           #client-trns-4       4
 41 |                            ciw.dists.Exponential(rate=3),           #client-trns-5       5
 42 |                            None,                                    #rsu-trns            6
 43 |                            None,                                    #rsu-cpu             7
 44 |                            None,                                    #trns-to-cloud       8
 45 |                            None,                                    #cloud-cpu           9
 46 |                            None,                                    #trns-to-service-1  10
 47 |                            None,                                    #service-cpu-1      11
 48 |                            None,                                    #trns-to-service-2  12
 49 |                            None,                                    #service-cpu-2      13
 50 |                            ],                                       ######################
 51 |             service_distributions=[MovingTransmissionDistNew(coords=walk_1),#client-trns-1       1
 52 |                            MovingTransmissionDistNew(coords=walk_2),        #client-trns-2       2
 53 |                            MovingTransmissionDistNew(coords=walk_3),        #client-trns-3       3
 54 |                            MovingTransmissionDistNew(coords=walk_4),        #client-trns-4       4
 55 |                            MovingTransmissionDistNew(coords=walk_5),        #client-trns-5       5
 56 |                            ciw.dists.Deterministic(value=0.0000001),        #rsu-trns            6
 57 |                            ComputationDist(mips=self.rsu_hardware[0]),      #rsu-cpu             7
 58 |                            CloudTransDelay(bw=1000),                        #trns-to-cloud       8
 59 |                            CloudCompDelay(self.cloud_hardware[1],0.05,0.2), #cloud-cpu           9
 60 |                            StationaryTransmissionDistNew(x=self.parked_1[0],y=self.parked_1[1]),        #trns-to-service-1  10
 61 |                            ComputationDist(mips=self.service_vehicles_cpu[0]),                        #service-cpu-1      11
 62 |                            StationaryTransmissionDistNew(x=self.parked_2[0],y=self.parked_2[1]),        #trns-to-service-2  12
 63 |                            ComputationDist(mips=self.service_vehicles_cpu[1]),                        #service-cpu-2      13
 64 |                            ],
 65 |                        #1,2,3,4,5,6,7,8,9,0,1,2,3           
 66 |             routing = [[0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 67 |                [0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 68 |                [0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 69 |                [0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 70 |                [0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 71 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 72 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 73 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0],
 74 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 75 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0],
 76 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
 77 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0],
 78 |                [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]],
 79 |             number_of_servers=[1,1,1,1,1,1,1,1,float("Inf"),1,1,1,1]
 80 |             )
 81 |         self.Q = CustomSimulation(self.N)
 82 |         self.Q.simulate_until_decision(self.n_timesteps)
 83 |         rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[7].all_individuals))
 84 |         cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[8].all_individuals))
 85 |         cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[9].all_individuals))
 86 |         service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[10].all_individuals))
 87 |         service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[11].all_individuals))
 88 |         service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[12].all_individuals))
 89 |         service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[13].all_individuals))
 90 |         obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500]))])/707
 91 |         obs_queues_cpu = np.array([rsu_cpu_queue, cloud_cpu_queue, service_1_cpu_queue, service_2_cpu_queue], dtype=np.float32)/1000000
 92 |         obs_queues_trans = np.array([cloud_trans_queue, service_1_trans_queue, service_2_trans_queue], dtype=np.float32)/100
 93 |         obs_task_cu = (np.array([self.Q.nodes[6].all_individuals[0].cu])-27.5)/(13010-27.5) #1
 94 |         obs_task_sz = (np.array([self.Q.nodes[6].all_individuals[0].sz])-20)/(40-20) #1
 95 |         obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375) #4
 96 |         obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
 97 |         info = {}
 98 |         self.calculated_inds = []
 99 |         return obs, info
100 |     
101 |     def step(self, action):
102 |         self.Q.nodes[6].decision = action
103 |         time = self.Q.simulate_until_decision(self.n_timesteps)
104 |         exitnode = self.Q.nodes[-1]
105 |         rew = 0
106 |         for ind in exitnode.all_individuals:
107 |             if ind not in self.calculated_inds:
108 |                 ind_recs = ind.data_records
109 |                 arrival_date = ind_recs[0].arrival_date
110 |                 exit_date = ind_recs[-1].exit_date
111 |                 total_delay = exit_date - arrival_date
112 |                 rew += 5-total_delay
113 |                 self.calculated_inds.append(ind)
114 |         info = {}
115 |         ter = False
116 |         tur = False
117 |         if self.Q.current_time >= self.n_timesteps:
118 |             ter = True
119 |         if not ter:
120 |             rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[7].all_individuals))
121 |             cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[8].all_individuals))
122 |             cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[9].all_individuals))
123 |             service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[10].all_individuals))
124 |             service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[11].all_individuals))
125 |             service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[12].all_individuals))
126 |             service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[13].all_individuals))
127 |             obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500]))])/707
128 |             obs_queues_cpu = np.array([rsu_cpu_queue,cloud_cpu_queue,service_1_cpu_queue,service_2_cpu_queue], dtype=np.float32)/1000000
129 |             obs_queues_trans = np.array([cloud_trans_queue,service_1_trans_queue,service_2_trans_queue], dtype=np.float32)/100
130 |             obs_task_cu = (np.array([self.Q.nodes[6].all_individuals[0].cu])-27.5)/(13010-27.5) #1
131 |             obs_task_sz = (np.array([self.Q.nodes[6].all_individuals[0].sz])-20)/(40-20) #1
132 |             obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375) #4
133 |             obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
134 |         else:
135 |             obs = np.zeros((15))
136 |         if self.render_mode == "human":
137 |             if not ter:
138 |                 emitted_node = self.Q.nodes[6].all_individuals[0].data_records[0].node
139 |                 self.anim.add_frame(time, emitted_node, action, [obs_queues_trans*100, obs_queues_cpu*1000000, (obs_task_cu*(52040-110))+110, (obs_task_sz*(15-5))+5])
140 |             else:
141 |                 self.anim.show_animation()
142 |         return obs, rew, ter, tur, info
143 | 
144 | 
145 | 
146 | from stable_baselines3 import PPO
147 | from helpers import shortest_queue
148 | number_of_trials = 100
149 | train_env = VFCOffloadingEnv(60)
150 | #model = PPO("MlpPolicy", train_env, verbose=1, gamma=0.85).learn(300000)
151 | #model.save("trained_models/env5-new")
152 | model = PPO("MlpPolicy", train_env, verbose=1).load("trained_models/env5-300000")
153 | 
154 | 
155 | def test_offloading_method(n, method_name):
156 |     total_rew = 0
157 |     total_delay = 0
158 |     total_num_tasks = 0
159 |     test_delays = []
160 |     for i in range(n):
161 |         env = VFCOffloadingEnv(60, render_mode=None)
162 |         obs,_ = env.reset()
163 |         ter = False
164 |         tot_rew = 0
165 |         while not ter:
166 |                 if method_name == "RL":
167 |                     action = model.predict(obs, deterministic=True)[0]
168 |                 elif method_name == "Greedy":
169 |                     action = shortest_queue(obs)
170 |                 elif method_name == "RSU":
171 |                     action = 0
172 |                 elif method_name == "Cloud":
173 |                     action = 1
174 |                 elif method_name == "Random":
175 |                     action = env.action_space.sample()
176 |                 else:
177 |                     print("INVALID METHOD")
178 |                 obs,rew,ter,_,_ = env.step(action)
179 |                 tot_rew += rew
180 |         total_rew += tot_rew
181 |         finished_tasks = env.Q.nodes[-1].all_individuals
182 |         finished_tasks_details = [r.data_records for r in finished_tasks]
183 |         delay = []
184 |         for task in finished_tasks_details:
185 |             delay.append(task[-1].exit_date - task[0].arrival_date)
186 |         total_delay += sum(delay)/len(delay)
187 |         total_num_tasks += len(delay)
188 |         test_delays.append(sum(delay)/len(delay))
189 |     print("--------------------------------------")
190 |     print(method_name, "Average Reward:",total_rew/n)
191 |     print(method_name, "Average Task Delay:",total_delay/n)
192 |     print(method_name, "Average #Tasks Completed:",total_num_tasks/n)
193 |     print(method_name, "Test Delays:", test_delays)
194 |     with open("results/out-5-"+method_name+".txt", 'w') as f:
195 |         for line in test_delays:
196 |             f.write(f"{line}\n")
197 | 
198 | 
199 | test_offloading_method(10, "RL")
200 | test_offloading_method(10, "Random")
201 | test_offloading_method(10, "Cloud")
202 | test_offloading_method(10, "RSU")
203 | test_offloading_method(10, "Greedy")


--------------------------------------------------------------------------------
/src/vfcenv10.py:
--------------------------------------------------------------------------------
  1 | import ciw.dists
  2 | import gymnasium as gym
  3 | import ciw
  4 | import numpy as np
  5 | from movement import create_random_walk, create_parked_coords
  6 | from custom_components import MovingTransmissionDistNew, ComputationDist, StationaryTransmissionDistNew, CustomSimulation10, CloudTransDelay, CloudCompDelay #CNG
  7 | from animate import Animator
  8 | from helpers import generate_service_cpu, generate_rsu_hardware, generate_cloud_hardware
  9 | np.set_printoptions(suppress=True)
 10 | 
 11 | 
 12 | class VFCOffloadingEnv10(gym.Env):
 13 | 
 14 |     def __init__(self, n_timesteps, render_mode=None) -> None:
 15 |         self.action_space = gym.spaces.Discrete(6) #CNG
 16 |         self.observation_space =  gym.spaces.Box(0,10,(23,)) #CNG
 17 |         self.n_timesteps = n_timesteps
 18 |         self.current_timestep = 0
 19 |         self.render_mode = render_mode
 20 |         super().__init__()
 21 | 
 22 |     def reset(self, seed=None):
 23 |         ciw.seed(seed)
 24 |         walk_1 = create_random_walk(self.n_timesteps)
 25 |         walk_2 = create_random_walk(self.n_timesteps)
 26 |         walk_3 = create_random_walk(self.n_timesteps)
 27 |         walk_4 = create_random_walk(self.n_timesteps)
 28 |         walk_5 = create_random_walk(self.n_timesteps) #CNG
 29 |         walk_6 = create_random_walk(self.n_timesteps)
 30 |         walk_7 = create_random_walk(self.n_timesteps)
 31 |         walk_8 = create_random_walk(self.n_timesteps)
 32 |         walk_9 = create_random_walk(self.n_timesteps)
 33 |         walk_10 = create_random_walk(self.n_timesteps)
 34 |         self.parked_1 = create_parked_coords()
 35 |         self.parked_2 = create_parked_coords()
 36 |         self.parked_3 = create_parked_coords() #CNG
 37 |         self.parked_4 = create_parked_coords()
 38 |         self.service_vehicles_cpu = generate_service_cpu(4) #CNG
 39 |         self.rsu_hardware = generate_rsu_hardware()
 40 |         self.cloud_hardware = generate_cloud_hardware()
 41 |         if self.render_mode == "human":
 42 |             self.anim = Animator([walk_1,walk_2,walk_3,walk_4,walk_5,walk_6,walk_7,walk_8,walk_9,walk_10],[self.parked_1,self.parked_2,self.parked_3,self.parked_4], self.service_vehicles_cpu, self.rsu_hardware, self.cloud_hardware) #CNG
 43 |         self.N = ciw.create_network(
 44 |             arrival_distributions=[ciw.dists.Exponential(rate=3),   #client-trns-1       1
 45 |                            ciw.dists.Exponential(rate=3),           #client-trns-2       2
 46 |                            ciw.dists.Exponential(rate=3),           #client-trns-3       3
 47 |                            ciw.dists.Exponential(rate=3),           #client-trns-4       4
 48 |                            ciw.dists.Exponential(rate=3),           #client-trns-5       5
 49 |                            ciw.dists.Exponential(rate=3),           #client-trns-6       6 #CNG
 50 |                            ciw.dists.Exponential(rate=3),           #client-trns-7       7
 51 |                            ciw.dists.Exponential(rate=3),           #client-trns-8       8
 52 |                            ciw.dists.Exponential(rate=3),           #client-trns-9       9
 53 |                            ciw.dists.Exponential(rate=3),           #client-trns-10     10
 54 |                            None,                                    #rsu-trns           11
 55 |                            None,                                    #rsu-cpu            12
 56 |                            None,                                    #trns-to-cloud      13
 57 |                            None,                                    #cloud-cpu          14
 58 |                            None,                                    #trns-to-service-1  15
 59 |                            None,                                    #service-cpu-1      16
 60 |                            None,                                    #trns-to-service-2  17
 61 |                            None,                                    #service-cpu-2      18
 62 |                            None,                                    #trns-to-service-3  19 #CNG
 63 |                            None,                                    #service-cpu-3      20
 64 |                            None,                                    #trns-to-service-4  21
 65 |                            None,                                    #service-cpu-4      22
 66 |                            ],                                       ######################
 67 |             service_distributions=[MovingTransmissionDistNew(coords=walk_1),#client-trns-1       1
 68 |                            MovingTransmissionDistNew(coords=walk_2),        #client-trns-2       2
 69 |                            MovingTransmissionDistNew(coords=walk_3),        #client-trns-3       3
 70 |                            MovingTransmissionDistNew(coords=walk_4),        #client-trns-4       4
 71 |                            MovingTransmissionDistNew(coords=walk_5),        #client-trns-5       5
 72 |                            MovingTransmissionDistNew(coords=walk_6),        #client-trns-6       6 #CNG
 73 |                            MovingTransmissionDistNew(coords=walk_7),        #client-trns-7       7
 74 |                            MovingTransmissionDistNew(coords=walk_8),        #client-trns-8       8
 75 |                            MovingTransmissionDistNew(coords=walk_9),        #client-trns-9       9
 76 |                            MovingTransmissionDistNew(coords=walk_10),       #client-trns-10     10
 77 |                            ciw.dists.Deterministic(value=0.0000001),        #rsu-trns           11
 78 |                            ComputationDist(mips=self.rsu_hardware[0]),      #rsu-cpu            12
 79 |                            CloudTransDelay(bw=1000),                        #trns-to-cloud      13
 80 |                            CloudCompDelay(self.cloud_hardware[1],0.05,0.2), #cloud-cpu          14
 81 |                            StationaryTransmissionDistNew(x=self.parked_1[0],y=self.parked_1[1]),      #trns-to-service-1    15
 82 |                            ComputationDist(mips=self.service_vehicles_cpu[0]),                        #service-cpu-1        16
 83 |                            StationaryTransmissionDistNew(x=self.parked_2[0],y=self.parked_2[1]),      #trns-to-service-2    17
 84 |                            ComputationDist(mips=self.service_vehicles_cpu[1]),                        #service-cpu-2        18
 85 |                            StationaryTransmissionDistNew(x=self.parked_3[0],y=self.parked_3[1]),      #trns-to-service-3    19 #CNG
 86 |                            ComputationDist(mips=self.service_vehicles_cpu[2]),                        #service-cpu-3        20
 87 |                            StationaryTransmissionDistNew(x=self.parked_4[0],y=self.parked_4[1]),      #trns-to-service-4    21
 88 |                            ComputationDist(mips=self.service_vehicles_cpu[3]),                        #service-cpu-4        22
 89 |                            ],
 90 |                                   
 91 |             routing = [ #CNG
 92 |                                #1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 
 93 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #1
 94 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #2
 95 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #3
 96 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #4
 97 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #5
 98 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #6
 99 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #7
100 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #8
101 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #9
102 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0])), #10
103 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #11
104 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #12
105 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0])), #13
106 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #14
107 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0])), #15
108 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #16
109 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0])), #17
110 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #18
111 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0])), #19
112 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #20
113 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1])), #21
114 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #22
115 |                ],
116 |             number_of_servers=[1,1,1,1,1,1,1,1,1,1,1,1,1,float("Inf"),1,1,1,1,1,1,1,1] #CNG
117 |             )
118 |         self.Q = CustomSimulation10(self.N) #CNG
119 |         self.Q.simulate_until_decision(self.n_timesteps)
120 |         rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[12].all_individuals)) #CNG
121 |         cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[13].all_individuals)) #CNG
122 |         cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[14].all_individuals)) #CNG
123 |         service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[15].all_individuals)) #CNG
124 |         service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[16].all_individuals)) #CNG
125 |         service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[17].all_individuals)) #CNG
126 |         service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[18].all_individuals)) #CNG
127 |         service_3_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[19].all_individuals)) #CNG
128 |         service_3_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[20].all_individuals)) #CNG
129 |         service_4_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[21].all_individuals)) #CNG
130 |         service_4_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[22].all_individuals)) #CNG
131 |         obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500])), np.linalg.norm((self.parked_3[0],self.parked_3[1]) - np.array([500,500])), np.linalg.norm((self.parked_4[0],self.parked_4[1]) - np.array([500,500]))])/707 #CNG
132 |         obs_queues_cpu = np.array([rsu_cpu_queue, cloud_cpu_queue, service_1_cpu_queue, service_2_cpu_queue, service_3_cpu_queue, service_4_cpu_queue], dtype=np.float32)/1000000 #CNG
133 |         obs_queues_trans = np.array([cloud_trans_queue, service_1_trans_queue, service_2_trans_queue, service_3_trans_queue, service_4_trans_queue], dtype=np.float32)/100 #CNG
134 |         obs_task_cu = (np.array([self.Q.nodes[11].all_individuals[0].cu])-27.5)/(13010-27.5) #CGD
135 |         obs_task_sz = (np.array([self.Q.nodes[11].all_individuals[0].sz])-20)/(40-20) #CGD
136 |         obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375) 
137 |         obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
138 |         info = {}
139 |         self.calculated_inds = []
140 |         return obs, info
141 |     
142 |     def step(self, action):
143 |         self.Q.nodes[11].decision = action #CGD
144 |         time = self.Q.simulate_until_decision(self.n_timesteps)
145 |         exitnode = self.Q.nodes[-1]
146 |         rew = 0
147 |         for ind in exitnode.all_individuals:
148 |             if ind not in self.calculated_inds:
149 |                 ind_recs = ind.data_records
150 |                 arrival_date = ind_recs[0].arrival_date
151 |                 exit_date = ind_recs[-1].exit_date
152 |                 total_delay = exit_date - arrival_date
153 |                 rew += 5 - total_delay
154 |                 self.calculated_inds.append(ind)
155 |         info = {}
156 |         ter = False
157 |         tur = False
158 |         if self.Q.current_time >= self.n_timesteps:
159 |             ter = True
160 |         if not ter:
161 |             rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[12].all_individuals)) #CNG
162 |             cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[13].all_individuals)) #CNG
163 |             cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[14].all_individuals)) #CNG
164 |             service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[15].all_individuals)) #CNG
165 |             service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[16].all_individuals)) #CNG
166 |             service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[17].all_individuals)) #CNG
167 |             service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[18].all_individuals)) #CNG
168 |             service_3_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[19].all_individuals)) #CNG
169 |             service_3_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[20].all_individuals)) #CNG
170 |             service_4_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[21].all_individuals)) #CNG
171 |             service_4_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[22].all_individuals)) #CNG
172 |             obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500])), np.linalg.norm((self.parked_3[0],self.parked_3[1]) - np.array([500,500])), np.linalg.norm((self.parked_4[0],self.parked_4[1]) - np.array([500,500]))])/707 #CNG
173 |             obs_queues_cpu = np.array([rsu_cpu_queue, cloud_cpu_queue, service_1_cpu_queue, service_2_cpu_queue, service_3_cpu_queue, service_4_cpu_queue], dtype=np.float32)/1000000 #CNG
174 |             obs_queues_trans = np.array([cloud_trans_queue, service_1_trans_queue, service_2_trans_queue, service_3_trans_queue, service_4_trans_queue], dtype=np.float32)/100 #CNG
175 |             obs_task_cu = (np.array([self.Q.nodes[11].all_individuals[0].cu])-27.5)/(13010-27.5) #CGD
176 |             obs_task_sz = (np.array([self.Q.nodes[11].all_individuals[0].sz])-20)/(40-20) #CGD
177 |             obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375)
178 |             obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
179 |         else:
180 |             obs = np.zeros((23)) #CND
181 |         if self.render_mode == "human":
182 |             if not ter:
183 |                 emitted_node = self.Q.nodes[11].all_individuals[0].data_records[0].node #CGD
184 |                 self.anim.add_frame(time, emitted_node, action, [obs_queues_trans*100, obs_queues_cpu*1000000, (obs_task_cu*(54600-2690))+2690, (obs_task_sz*(30-5))+5])
185 |             else:
186 |                 self.anim.show_animation()
187 |         return obs, rew, ter, tur, info
188 | 
189 | 
190 | from stable_baselines3 import PPO
191 | from helpers import shortest_queue10
192 | number_of_trials = 100
193 | train_env = VFCOffloadingEnv10(60)
194 | #model = PPO("MlpPolicy", train_env, verbose=1, gamma=0.85).learn(300000)
195 | #model.save("trained_models/env10-new")
196 | model = PPO("MlpPolicy", train_env, verbose=1).load("trained_models/env10-300000")
197 | 
198 | 
199 | def test_offloading_method(n, method_name):
200 |     total_rew = 0
201 |     total_delay = 0
202 |     total_num_tasks = 0
203 |     test_delays = []
204 |     for i in range(n):
205 |         env = VFCOffloadingEnv10(60, render_mode=None)
206 |         obs,_ = env.reset()
207 |         ter = False
208 |         tot_rew = 0
209 |         while not ter:
210 |                 if method_name == "RL":
211 |                     action = model.predict(obs, deterministic=True)[0]
212 |                 elif method_name == "Greedy":
213 |                     action = shortest_queue10(obs)
214 |                 elif method_name == "RSU":
215 |                     action = 0
216 |                 elif method_name == "Cloud":
217 |                     action = 1
218 |                 elif method_name == "Random":
219 |                     action = env.action_space.sample()
220 |                 else:
221 |                     print("INVALID METHOD")
222 |                 obs,rew,ter,_,_ = env.step(action)
223 |                 tot_rew += rew
224 |         total_rew += tot_rew
225 |         finished_tasks = env.Q.nodes[-1].all_individuals
226 |         finished_tasks_details = [r.data_records for r in finished_tasks]
227 |         delay = []
228 |         for task in finished_tasks_details:
229 |             delay.append(task[-1].exit_date - task[0].arrival_date)
230 |         total_delay += sum(delay)/len(delay)
231 |         total_num_tasks += len(delay)
232 |         test_delays.append(sum(delay)/len(delay))
233 |     print("--------------------------------------")
234 |     print(method_name, "Average Reward:",total_rew/n)
235 |     print(method_name, "Average Task Delay:",total_delay/n)
236 |     print(method_name, "Average #Tasks Completed:",total_num_tasks/n)
237 |     print(method_name, "Test Delays:", test_delays)
238 |     with open("results/out-10-"+method_name+".txt", 'w') as f:
239 |         for line in test_delays:
240 |             f.write(f"{line}\n")
241 | 
242 | test_offloading_method(10, "RL")
243 | test_offloading_method(10, "Random")
244 | test_offloading_method(10, "Cloud")
245 | test_offloading_method(10, "RSU")
246 | test_offloading_method(10, "Greedy")
247 | 


--------------------------------------------------------------------------------
/src/vfcenv20.py:
--------------------------------------------------------------------------------
  1 | import ciw.dists
  2 | import gymnasium as gym
  3 | import ciw
  4 | import numpy as np 
  5 | from movement import create_random_walk, create_parked_coords
  6 | from custom_components import MovingTransmissionDistNew, ComputationDist, StationaryTransmissionDistNew, CustomSimulation20, CloudCompDelay, CloudTransDelay #CNG
  7 | from animate import Animator
  8 | from helpers import generate_service_cpu, generate_rsu_hardware, generate_cloud_hardware
  9 | np.set_printoptions(suppress=True)
 10 | 
 11 | 
 12 | class VFCOffloadingEnv20(gym.Env):
 13 | 
 14 |     def __init__(self, n_timesteps, render_mode=None) -> None:
 15 |         self.action_space = gym.spaces.Discrete(10) #CNG
 16 |         self.observation_space =  gym.spaces.Box(0,10,(39,)) #CNG
 17 |         self.n_timesteps = n_timesteps
 18 |         self.current_timestep = 0
 19 |         self.render_mode = render_mode
 20 |         super().__init__()
 21 | 
 22 |     def reset(self, seed=None):
 23 |         ciw.seed(seed)
 24 |         walk_1 = create_random_walk(self.n_timesteps)
 25 |         walk_2 = create_random_walk(self.n_timesteps)
 26 |         walk_3 = create_random_walk(self.n_timesteps)
 27 |         walk_4 = create_random_walk(self.n_timesteps)
 28 |         walk_5 = create_random_walk(self.n_timesteps) #CNG
 29 |         walk_6 = create_random_walk(self.n_timesteps)
 30 |         walk_7 = create_random_walk(self.n_timesteps)
 31 |         walk_8 = create_random_walk(self.n_timesteps)
 32 |         walk_9 = create_random_walk(self.n_timesteps)
 33 |         walk_10 = create_random_walk(self.n_timesteps)
 34 |         walk_11 = create_random_walk(self.n_timesteps)
 35 |         walk_12 = create_random_walk(self.n_timesteps)
 36 |         walk_13 = create_random_walk(self.n_timesteps)
 37 |         walk_14 = create_random_walk(self.n_timesteps)
 38 |         walk_15 = create_random_walk(self.n_timesteps)
 39 |         walk_16 = create_random_walk(self.n_timesteps)
 40 |         walk_17 = create_random_walk(self.n_timesteps)
 41 |         walk_18 = create_random_walk(self.n_timesteps)
 42 |         walk_19 = create_random_walk(self.n_timesteps)
 43 |         walk_20 = create_random_walk(self.n_timesteps)
 44 |         self.parked_1 = create_parked_coords()
 45 |         self.parked_2 = create_parked_coords()
 46 |         self.parked_3 = create_parked_coords() #CNG
 47 |         self.parked_4 = create_parked_coords()
 48 |         self.parked_5 = create_parked_coords()
 49 |         self.parked_6 = create_parked_coords()
 50 |         self.parked_7 = create_parked_coords()
 51 |         self.parked_8 = create_parked_coords()
 52 |         self.service_vehicles_cpu = generate_service_cpu(8) #CNG
 53 |         self.rsu_hardware = generate_rsu_hardware()
 54 |         self.cloud_hardware = generate_cloud_hardware()
 55 |         if self.render_mode == "human":
 56 |             self.anim = Animator([walk_1,walk_2,walk_3,walk_4,walk_5,walk_6,walk_7,walk_8,walk_9,walk_10,walk_11,walk_12,walk_13,walk_14,walk_15,walk_16,walk_17,walk_18,walk_19,walk_20],[self.parked_1,self.parked_2,self.parked_3,self.parked_4,self.parked_5,self.parked_6,self.parked_7,self.parked_8], self.service_vehicles_cpu, self.rsu_hardware, self.cloud_hardware) #CNG
 57 |         self.N = ciw.create_network(
 58 |             arrival_distributions=[ciw.dists.Exponential(rate=3),   #client-trns-1       1
 59 |                            ciw.dists.Exponential(rate=3),           #client-trns-2       2
 60 |                            ciw.dists.Exponential(rate=3),           #client-trns-3       3
 61 |                            ciw.dists.Exponential(rate=3),           #client-trns-4       4
 62 |                            ciw.dists.Exponential(rate=3),           #client-trns-5       5
 63 |                            ciw.dists.Exponential(rate=3),           #client-trns-6       6 
 64 |                            ciw.dists.Exponential(rate=3),           #client-trns-7       7
 65 |                            ciw.dists.Exponential(rate=3),           #client-trns-8       8
 66 |                            ciw.dists.Exponential(rate=3),           #client-trns-9       9
 67 |                            ciw.dists.Exponential(rate=3),           #client-trns-10     10
 68 |                            ciw.dists.Exponential(rate=3),           #client-trns-11     11
 69 |                            ciw.dists.Exponential(rate=3),           #client-trns-12     12
 70 |                            ciw.dists.Exponential(rate=3),           #client-trns-13     13
 71 |                            ciw.dists.Exponential(rate=3),           #client-trns-14     14
 72 |                            ciw.dists.Exponential(rate=3),           #client-trns-15     15
 73 |                            ciw.dists.Exponential(rate=3),           #client-trns-16     16 
 74 |                            ciw.dists.Exponential(rate=3),           #client-trns-17     17
 75 |                            ciw.dists.Exponential(rate=3),           #client-trns-18     18
 76 |                            ciw.dists.Exponential(rate=3),           #client-trns-19     19
 77 |                            ciw.dists.Exponential(rate=3),           #client-trns-20     20
 78 |                            None,                                    #rsu-trns           21
 79 |                            None,                                    #rsu-cpu            22
 80 |                            None,                                    #trns-to-cloud      23
 81 |                            None,                                    #cloud-cpu          24
 82 |                            None,                                    #trns-to-service-1  25
 83 |                            None,                                    #service-cpu-1      26
 84 |                            None,                                    #trns-to-service-2  27
 85 |                            None,                                    #service-cpu-2      28
 86 |                            None,                                    #trns-to-service-3  29 
 87 |                            None,                                    #service-cpu-3      30
 88 |                            None,                                    #trns-to-service-4  31
 89 |                            None,                                    #service-cpu-4      32
 90 |                            None,                                    #trns-to-service-5  33
 91 |                            None,                                    #service-cpu-5      34
 92 |                            None,                                    #trns-to-service-6  35
 93 |                            None,                                    #service-cpu-6      36
 94 |                            None,                                    #trns-to-service-7  37 
 95 |                            None,                                    #service-cpu-7      38
 96 |                            None,                                    #trns-to-service-8  39
 97 |                            None,                                    #service-cpu-8      40
 98 |                            ],                                       ######################
 99 |             service_distributions=[MovingTransmissionDistNew(coords=walk_1),#client-trns-1       1
100 |                            MovingTransmissionDistNew(coords=walk_2),        #client-trns-2       2
101 |                            MovingTransmissionDistNew(coords=walk_3),        #client-trns-3       3
102 |                            MovingTransmissionDistNew(coords=walk_4),        #client-trns-4       4
103 |                            MovingTransmissionDistNew(coords=walk_5),        #client-trns-5       5
104 |                            MovingTransmissionDistNew(coords=walk_6),        #client-trns-6       6 
105 |                            MovingTransmissionDistNew(coords=walk_7),        #client-trns-7       7
106 |                            MovingTransmissionDistNew(coords=walk_8),        #client-trns-8       8
107 |                            MovingTransmissionDistNew(coords=walk_9),        #client-trns-9       9
108 |                            MovingTransmissionDistNew(coords=walk_10),       #client-trns-10     10
109 |                            MovingTransmissionDistNew(coords=walk_11),       #client-trns-11     11
110 |                            MovingTransmissionDistNew(coords=walk_12),       #client-trns-12     12
111 |                            MovingTransmissionDistNew(coords=walk_13),       #client-trns-13     13
112 |                            MovingTransmissionDistNew(coords=walk_14),       #client-trns-14     14
113 |                            MovingTransmissionDistNew(coords=walk_15),       #client-trns-15     15
114 |                            MovingTransmissionDistNew(coords=walk_16),       #client-trns-16     16
115 |                            MovingTransmissionDistNew(coords=walk_17),       #client-trns-17     17
116 |                            MovingTransmissionDistNew(coords=walk_18),       #client-trns-18     18
117 |                            MovingTransmissionDistNew(coords=walk_19),       #client-trns-19     19
118 |                            MovingTransmissionDistNew(coords=walk_20),       #client-trns-20     20
119 |                            ciw.dists.Deterministic(value=0.0000001),        #rsu-trns           21
120 |                            ComputationDist(mips=self.rsu_hardware[0]),      #rsu-cpu            22
121 |                            CloudTransDelay(bw=1000),                        #trns-to-cloud      23
122 |                            CloudCompDelay(self.cloud_hardware[1],0.05,0.2), #cloud-cpu          24
123 |                            StationaryTransmissionDistNew(x=self.parked_1[0],y=self.parked_1[1]),      #trns-to-service-1    25
124 |                            ComputationDist(mips=self.service_vehicles_cpu[0]),                        #service-cpu-1        26
125 |                            StationaryTransmissionDistNew(x=self.parked_2[0],y=self.parked_2[1]),      #trns-to-service-2    27
126 |                            ComputationDist(mips=self.service_vehicles_cpu[1]),                        #service-cpu-2        28
127 |                            StationaryTransmissionDistNew(x=self.parked_3[0],y=self.parked_3[1]),      #trns-to-service-3    29 
128 |                            ComputationDist(mips=self.service_vehicles_cpu[2]),                        #service-cpu-3        30
129 |                            StationaryTransmissionDistNew(x=self.parked_4[0],y=self.parked_4[1]),      #trns-to-service-4    31
130 |                            ComputationDist(mips=self.service_vehicles_cpu[3]),                        #service-cpu-4        32
131 |                            StationaryTransmissionDistNew(x=self.parked_5[0],y=self.parked_5[1]),      #trns-to-service-5    33
132 |                            ComputationDist(mips=self.service_vehicles_cpu[4]),                        #service-cpu-5        34
133 |                            StationaryTransmissionDistNew(x=self.parked_6[0],y=self.parked_6[1]),      #trns-to-service-6    35
134 |                            ComputationDist(mips=self.service_vehicles_cpu[5]),                        #service-cpu-6        36
135 |                            StationaryTransmissionDistNew(x=self.parked_8[0],y=self.parked_7[1]),      #trns-to-service-7    37 
136 |                            ComputationDist(mips=self.service_vehicles_cpu[6]),                        #service-cpu-7        38
137 |                            StationaryTransmissionDistNew(x=self.parked_8[0],y=self.parked_8[1]),      #trns-to-service-8    39
138 |                            ComputationDist(mips=self.service_vehicles_cpu[7]),                        #service-cpu-8        40
139 |                            ],
140 |             routing = [ #CNG
141 |                                #1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
142 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #1
143 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #2
144 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #3
145 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #4
146 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #5
147 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #6
148 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #7
149 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #8
150 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #9
151 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #10
152 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #11
153 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #12
154 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #13
155 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #14
156 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #15
157 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #16
158 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #17
159 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #18
160 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #19
161 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #20
162 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #21
163 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #22
164 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #23
165 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #24
166 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #25
167 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #26
168 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0])), #27
169 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #28
170 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0])), #29
171 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #30
172 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0])), #31
173 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #32
174 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0])), #33
175 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #34
176 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0])), #35
177 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #36
178 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0])), #37
179 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #38
180 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1])), #39
181 |                 list(np.float_([0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])), #40
182 |                ],
183 |             number_of_servers= [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,float("Inf"),1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] #CNG
184 |             )
185 |         self.Q = CustomSimulation20(self.N) #CNG
186 |         self.Q.simulate_until_decision(self.n_timesteps)
187 |         rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[22].all_individuals)) #CNG
188 |         cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[23].all_individuals)) #CNG
189 |         cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[24].all_individuals)) #CNG
190 |         service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[25].all_individuals)) #CNG
191 |         service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[26].all_individuals)) #CNG
192 |         service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[27].all_individuals)) #CNG
193 |         service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[28].all_individuals)) #CNG
194 |         service_3_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[29].all_individuals)) #CNG
195 |         service_3_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[30].all_individuals)) #CNG
196 |         service_4_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[31].all_individuals)) #CNG
197 |         service_4_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[32].all_individuals)) #CNG
198 |         service_5_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[33].all_individuals)) #CNG
199 |         service_5_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[34].all_individuals)) #CNG
200 |         service_6_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[35].all_individuals)) #CNG
201 |         service_6_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[36].all_individuals)) #CNG
202 |         service_7_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[37].all_individuals)) #CNG
203 |         service_7_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[38].all_individuals)) #CNG
204 |         service_8_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[39].all_individuals)) #CNG
205 |         service_8_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[40].all_individuals)) #CNG
206 |         obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500])), np.linalg.norm((self.parked_3[0],self.parked_3[1]) - np.array([500,500])), np.linalg.norm((self.parked_4[0],self.parked_4[1]) - np.array([500,500])), np.linalg.norm((self.parked_5[0],self.parked_5[1]) - np.array([500,500])),np.linalg.norm((self.parked_6[0],self.parked_6[1]) - np.array([500,500])), np.linalg.norm((self.parked_7[0],self.parked_7[1]) - np.array([500,500])), np.linalg.norm((self.parked_8[0],self.parked_8[1]) - np.array([500,500]))])/707 #CNG
207 |         obs_queues_cpu = np.array([rsu_cpu_queue, cloud_cpu_queue, service_1_cpu_queue, service_2_cpu_queue, service_3_cpu_queue, service_4_cpu_queue, service_5_cpu_queue, service_6_cpu_queue, service_7_cpu_queue, service_8_cpu_queue], dtype=np.float32)/1000000 #CNG
208 |         obs_queues_trans = np.array([cloud_trans_queue, service_1_trans_queue, service_2_trans_queue, service_3_trans_queue, service_4_trans_queue, service_5_trans_queue, service_6_trans_queue, service_7_trans_queue, service_8_trans_queue], dtype=np.float32)/100 #CNG
209 |         obs_task_cu = (np.array([self.Q.nodes[21].all_individuals[0].cu])-27.5)/(13010-27.5) #CGD
210 |         obs_task_sz = (np.array([self.Q.nodes[21].all_individuals[0].sz])-10)/(30-10) #CGD
211 |         obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375) 
212 |         obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
213 |         info = {}
214 |         self.calculated_inds = []
215 |         return obs, info
216 |     
217 |     def step(self, action):
218 |         self.Q.nodes[21].decision = action #CGD
219 |         time = self.Q.simulate_until_decision(self.n_timesteps)
220 |         exitnode = self.Q.nodes[-1]
221 |         rew = 0
222 |         for ind in exitnode.all_individuals:
223 |             if ind not in self.calculated_inds:
224 |                 ind_recs = ind.data_records
225 |                 arrival_date = ind_recs[0].arrival_date
226 |                 exit_date = ind_recs[-1].exit_date
227 |                 total_delay = exit_date - arrival_date
228 |                 rew += 5 - total_delay
229 |                 self.calculated_inds.append(ind)
230 |         info = {}
231 |         ter = False
232 |         tur = False
233 |         if self.Q.current_time >= self.n_timesteps:
234 |             ter = True
235 |         if not ter:
236 |             rsu_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[22].all_individuals)) #CNG
237 |             cloud_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[23].all_individuals)) #CNG
238 |             cloud_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[24].all_individuals)) #CNG
239 |             service_1_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[25].all_individuals)) #CNG
240 |             service_1_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[26].all_individuals)) #CNG
241 |             service_2_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[27].all_individuals)) #CNG
242 |             service_2_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[28].all_individuals)) #CNG
243 |             service_3_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[29].all_individuals)) #CNG
244 |             service_3_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[30].all_individuals)) #CNG
245 |             service_4_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[31].all_individuals)) #CNG
246 |             service_4_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[32].all_individuals)) #CNG
247 |             service_5_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[33].all_individuals)) #CNG
248 |             service_5_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[34].all_individuals)) #CNG
249 |             service_6_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[35].all_individuals)) #CNG
250 |             service_6_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[36].all_individuals)) #CNG
251 |             service_7_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[37].all_individuals)) #CNG
252 |             service_7_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[38].all_individuals)) #CNG
253 |             service_8_trans_queue = sum(tuple(x.sz for x in self.Q.nodes[39].all_individuals)) #CNG
254 |             service_8_cpu_queue = sum(tuple(x.cu for x in self.Q.nodes[40].all_individuals)) #CNG
255 |             obs_service_distance = np.array([np.linalg.norm((self.parked_1[0],self.parked_1[1]) - np.array([500,500])), np.linalg.norm((self.parked_2[0],self.parked_2[1]) - np.array([500,500])), np.linalg.norm((self.parked_3[0],self.parked_3[1]) - np.array([500,500])), np.linalg.norm((self.parked_4[0],self.parked_4[1]) - np.array([500,500])), np.linalg.norm((self.parked_5[0],self.parked_5[1]) - np.array([500,500])),np.linalg.norm((self.parked_6[0],self.parked_6[1]) - np.array([500,500])), np.linalg.norm((self.parked_7[0],self.parked_7[1]) - np.array([500,500])), np.linalg.norm((self.parked_8[0],self.parked_8[1]) - np.array([500,500]))])/707 #CNG
256 |             obs_queues_cpu = np.array([rsu_cpu_queue, cloud_cpu_queue, service_1_cpu_queue, service_2_cpu_queue, service_3_cpu_queue, service_4_cpu_queue, service_5_cpu_queue, service_6_cpu_queue, service_7_cpu_queue, service_8_cpu_queue], dtype=np.float32)/1000000 #CNG
257 |             obs_queues_trans = np.array([cloud_trans_queue, service_1_trans_queue, service_2_trans_queue, service_3_trans_queue, service_4_trans_queue, service_5_trans_queue, service_6_trans_queue, service_7_trans_queue, service_8_trans_queue], dtype=np.float32)/100 #CNG
258 |             obs_task_cu = (np.array([self.Q.nodes[21].all_individuals[0].cu])-27.5)/(13010-27.5) #CGD
259 |             obs_task_sz = (np.array([self.Q.nodes[21].all_individuals[0].sz])-10)/(30-10) #CGD
260 |             obs_cpu = (np.concatenate(([self.cloud_hardware[1]],[self.rsu_hardware[0]],self.service_vehicles_cpu))-18375)/(71120-18375) 
261 |             obs = np.concatenate((obs_queues_trans, obs_queues_cpu, obs_service_distance, obs_task_cu, obs_task_sz, obs_cpu),dtype=np.float32)
262 |         else:
263 |             obs = np.zeros((39))
264 |         if self.render_mode == "human":
265 |             if not ter:
266 |                 emitted_node = self.Q.nodes[21].all_individuals[0].data_records[0].node #CGD
267 |                 self.anim.add_frame(time, emitted_node, action, [obs_queues_trans*100, obs_queues_cpu*1000000, (obs_task_cu*(54600-2690))+2690, (obs_task_sz*(30-5))+5])
268 |             else:
269 |                 self.anim.show_animation()
270 |         return obs, rew, ter, tur, info
271 | 
272 | 
273 | from stable_baselines3 import PPO
274 | from helpers import shortest_queue20
275 | train_env = VFCOffloadingEnv20(60)
276 | #model = PPO("MlpPolicy", train_env, verbose=1, gamma=0.85).learn(300000)
277 | #model.save("trained_models/env20-new")
278 | model = PPO("MlpPolicy", train_env, verbose=1).load("trained_models/env20-300000")
279 | 
280 | 
281 | def test_offloading_method(n, method_name):
282 |     total_rew = 0
283 |     total_delay = 0
284 |     total_num_tasks = 0
285 |     test_delays = []
286 |     for i in range(n):
287 |         print(i)
288 |         env = VFCOffloadingEnv20(60, render_mode=None)
289 |         obs,_ = env.reset()
290 |         ter = False
291 |         tot_rew = 0
292 |         #zorder = 0
293 |         while not ter:
294 |                 if method_name == "RL":
295 |                     # color = "blue"
296 |                     # zorder = 10
297 |                     action = model.predict(obs)[0]
298 |                     #action_store.append(action)
299 |                 elif method_name == "Greedy":
300 |                     # color = "green"
301 |                     action = shortest_queue20(obs)
302 |                 elif method_name == "RSU":
303 |                     # color = "black"
304 |                     action = 0
305 |                 elif method_name == "Cloud":
306 |                     # color = "black"
307 |                     action = 1
308 |                 elif method_name == "Random":
309 |                     # color = "red"
310 |                     action = env.action_space.sample()
311 |                 else:
312 |                     print("INVALID METHOD")
313 |                 obs,rew,ter,_,_ = env.step(action)
314 |                 tot_rew += rew
315 |         total_rew += tot_rew
316 |         finished_tasks = env.Q.nodes[-1].all_individuals
317 |         finished_tasks_details = [r.data_records for r in finished_tasks]
318 |         delay = []
319 |         for task in finished_tasks_details:
320 |             delay.append(task[-1].exit_date - task[0].arrival_date)
321 |         total_delay += sum(delay)/len(delay)
322 |         total_num_tasks += len(delay)
323 |         test_delays.append(sum(delay)/len(delay))
324 |     print("--------------------------------------")
325 |     print(method_name, "Average Reward:",total_rew/n)
326 |     print(method_name, "Average Task Delay:",total_delay/n)
327 |     print(method_name, "Average #Tasks Completed:",total_num_tasks/n)
328 |     print(method_name, "Test Delays:", test_delays)
329 |     with open("results/out-20-"+method_name+".txt", 'w') as f:
330 |         for line in test_delays:
331 |             f.write(f"{line}\n")
332 |     # ts = [ts[0] for ts in env.Q.statetracker.history]
333 |     # cld_trns = [ts[1][22] for ts in env.Q.statetracker.history]
334 |     # s1_trns = [ts[1][24] for ts in env.Q.statetracker.history]
335 |     # s2_trns =[ts[1][26] for ts in env.Q.statetracker.history]
336 |     # s3_trns =[ts[1][28] for ts in env.Q.statetracker.history]
337 |     # s4_trns =[ts[1][30] for ts in env.Q.statetracker.history]
338 |     # s5_trns = [ts[1][32] for ts in env.Q.statetracker.history]
339 |     # s6_trns =[ts[1][34] for ts in env.Q.statetracker.history]
340 |     # s7_trns =[ts[1][36] for ts in env.Q.statetracker.history]
341 |     # s8_trns =[ts[1][38] for ts in env.Q.statetracker.history]
342 |     # plt.plot(ts, np.array(cld_trns)+np.array(s1_trns)+np.array(s2_trns)+np.array(s3_trns)+np.array(s4_trns)+np.array(s5_trns)+np.array(s6_trns)+np.array(s7_trns)+np.array(s8_trns), label=method_name, linewidth=3, color=color, zorder=zorder)
343 | 
344 | test_offloading_method(10, "RL")
345 | test_offloading_method(10, "Greedy")
346 | test_offloading_method(10, "Random")
347 | test_offloading_method(10, "Cloud")
348 | test_offloading_method(10, "RSU")
349 | 
350 | 
351 | # plt.legend(loc='best')
352 | # plt.xlabel("Simulation Timestep (s)")
353 | # plt.ylabel("Transmisson Queue Length")
354 | # plt.legend(fontsize="large")
355 | # plt.show()


--------------------------------------------------------------------------------
/system-model-visual.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Procedurally-Generated-Human/VFC-Offloading-RL/49933b925406957b418979cffa6305efed0b8477/system-model-visual.png


--------------------------------------------------------------------------------